Build from best practices, but use your judgement too

Sometimes best practices avoid past mistakes, and sometimes they create new ones

Almost as long as we've built software and IT infrastructures, we've been caught up in best practices. There are real, tangible reasons why they're called "best" practices and not "mostly OK" practices or "occasionally useful" practices. The idea is that these are pearls of wisdom collected from direct experience, and they should be exercised to prevent us from repeating the mistakes and unwise decisions made by others in the past. This is good, wholesome, useful stuff that allows us to simultaneously bolster the robustness of our design while reducing the time to implementation.

However, one important consideration is often overlooked when discussing best practices, and that's the question of scale. This, in and of itself, should be a best practice.

When we document a project that breaks new ground or offers new functions and capabilities, we may include a best practice document that describes the trials and tribulations of the design and build process and illustrates why key decisions were made and the results of those decisions. However, these best practices are typically written within the context of the scale of the project and will not necessarily translate to significantly smaller or significantly larger implementations.

As an example, when building a smaller back-end app that we expect to serve at most two dozen requests per second, we may pass data through a Redis queue between worker processes because there's no concern about that data changing before it's acted upon. This reduces development time and resource consumption, and it results in an app that works as expected.

However, if that app were to scale to tens or hundreds of thousands of requests per second, passing data through Redis might not work, due to the sheer size of the data we'd have to queue and because we'd have to build access to the queues at scale. We'd be better off with a different approach (such as passing pointers to the data through Redis), or we might use another datastore altogether. All of these approaches could be considered best practices for the size of the implementation they describe.

An oft-heard argument about best practices is that you should design and build to a higher scale than you actually expect; thus, using best practices for a huge project in a small project will reap rewards later on. This is sometimes true, but in most cases, it's actually detrimental to the overall project to overspec and overbuild. You introduce useless moving parts that become critical to core functionality, but actually do very little. These overbuilt parts become maintenance issues and problem areas without contributing anything really useful.

You'll see these monstrosities in many places. It's the custom-built internal app that has multiple replicated databases, piles of server VMs, dedicated caching systems, and load balancing that has a grand total of a few dozen users. It's the $20,000 router on the remote end of a T1. It's the hideously complex Windows infrastructure design with every conceivable server role split out into separate VMs (or physical hardware!) from AD to DHCP to NPS ... for a network of 200 users. It's the purchase of six racks full of identical, ultra-high-end server hardware for a project that could easily fit on only a few.

In these planning meetings -- whether they're about hardware requirements and purchases or software design -- there will always be those who wish to err on the side of caution and go big on critical components to fend off the specter of capacity problems down the line. In and of itself, this is not a bad thing, and we need to plan that way. However, we don't need to go too far with those plans. You wouldn't put military spec blast doors in a condo, for instance, though that's a best practice for a military installation, right?

Instead, we need to plan for slightly more than we actually think we'll need, but leave the door open in the design for specific components to be broken out when and if we do need to scale. This is tricky, and it can be a delicate balancing act at times, but it's a necessary part of every app and infrastructure plan.

This is why we need to read and digest best practices, then apply them judiciously to our actual project. Understand why these methods were developed and the situation within which they were developed, then adapt them to our needs. From there, we can write another set of best practices and the cycle continues.