Bulletproof your infrastructure tip No. 3: Establish backup links wherever and whenever possible
If at all possible, there should be multiple paths to every data center and remote office. Back in the day, this was very expensive, but now you can probably get a business-class DSL or cable connection to most of your locations. For less than $100 a month in many cases, you have an alternate access method to that site for use in emergencies -- or for sensitive remote configurations of the production routers and firewalls. It might even be feasible to split your traffic in those sites, pushing business traffic over leased lines and Internet browsing traffic over the DSL or cable circuit.
If cost is the ultimate issue, you can take a page from the first item in this list and procure a used firewall from eBay for this circuit. Because it's not production, you have less concern over the reliability of the device, so a used piece of gear is a good fit for a tight budget.
Bulletproof your infrastructure tip No. 4: Bet on a big box
This one really applies to virtualized infrastructures only. Say you have a virtualization farm of a dozen 1U servers running a few hundred virtual machines. If something goes wrong with the production system, you can probably get away with running some subset of those VMs to maintain critical line-of-business applications. If that's the case, you don't need to maintain a duplicate virtualization farm. Instead, you can invest in a single four-CPU server with a bunch of RAM that can take the production load for some length of time.
This server wouldn't necessarily play in the farm itself (though it could), but would instead be installed and ready to handle a load if the situation calls for it. In some cases, you may even be able to game the virtualization vendor's evaluation period to avoid paying for licenses on a dormant server, but your mileage may vary.
The size of this emergency server should correspond to your infrastructure needs and the number and weight of the virtual machines you expect it to run. Generally speaking, you can get an awful lot of emergency processing power in a virtualized environment for under $10,000. Is that too much for peace of mind?
Bulletproof your infrastructure tip No. 5: Learn Linux
Even if you're a Windows shop, learning enough about Linux can open up a huge number of valuable, low-cost options. You may not feel comfortable running critical business applications on Linux for whatever reason, but the plethora of open source network and systems monitoring and maintenance tools available on a Linux or Unix is incredible. There are Windows versions of many of these tools, but they are natively Unix-based.
I've been accused of being overbearing in my advocacy of full-scale monitoring and maintenance packages like Nagios, Zenoss, Cacti, RANCID, and so forth, but the truth is that these tools make an enormous difference in both day-to-day IT operation and in times of trouble. The benefit of learning Linux and running these tools is twofold: You gain Linux skills, and you enrich your network with a raft of supporting players that makes everyone's life simpler.
It's easier to preach about being proactive than to actually make these measures happen in the topsy-turvy, break-fix world of IT. But to paraphrase an old saying, if you're too busy mopping the floor to turn off the faucet, you probably need to rethink your approach.