And when implementing network monitoring, be sure to leave no stone unturned. Monitor the CPU utilization of your routers and switches; watch the error rates on Ethernet interfaces; have your routers and switches log to central syslog servers and implement some form of logfile analysis to alert you when there are reports of anything from IP conflicts to circuits going down. Careful, conscientious implementation and tweaking of your monitoring framework will save enormous amounts of time and energy, especially when it counts the most.
Performance tip No. 6: Know your apps
Infrastructure performance monitoring will only get you so far. All the computing and storage resources that you are offering up on your network are being consumed by your applications. For too many of us, those applications form something akin to a black hole -- we can easily observe their effects on our infrastructure, but it's often difficult to see inside them to know what's going on.
Many IT shops are content to let software vendors install and implement the applications on their networks; after all, that's less work for IT. But be careful -- you're on the hook when the network later slows to a crawl.
Spend time testing your apps with an eye to uncovering their soft spots. Whether it's a particularly expensive stored database procedure that gets called when users log in or a massive performance slowdown during third shift when backups kick off, you need to know ahead of time where your likely performance drains reside.
To accomplish this, insist on testing new applications in your infrastructure before they are purchased. Pay close attention to the amount of resources used as you test and project how much performance the application will require under real-life production loads. This kind of testing can uncover severe architectural flaws in the application that may make it inappropriate for your environment. Better to know that in advance than to find yourself fending off users armed with torches and pitchforks.
Performance tip No. 7: Terabytes and spindle counts, oh my
The past few years have seen explosive growth in disk capacity. With the advent of 2TB SATA disks, it's now possible to jam more than 10TB into a single two-rack-unit server. And that's great -- because now you need fewer disks, right? Not so fast.
It's crucial to understand that today's SATA disks share an important trait with their smaller predecessors: They're fast. While it may be possible to fit 2TB of data onto a single 7,200-rpm SATA disk, you'll still be limited to an average randomized transactional throughput of perhaps 80 IOPS (I/O operations per second) per disk. Unless you're storing a mostly static data bone yard, be prepared to be thoroughly unhappy with the performance you'll get out of these new drives as compared to twice the number of 1TB disks.
If your applications require a lot of randomized reads and writes -- database and email servers commonly fit this bill -- you'll need a lot of individual disks to obtain the necessary transactional performance. While huge disks are great for storing less frequently used data, your most prized data must still sit on disk arrays made up of faster and smaller disks.
Performance tip No. 8: Beware the 10-pound server in the 5-pound bag
Virtualization has to be just about the coolest thing to happen to the enterprise datacenter in a long time. It offers a multitude of manageability and monitoring benefits, scales cleanly, makes disaster recovery simpler than ever before, and dramatically decreases the number of physical servers you need chewing up power and spewing out heat.