I've spent a great deal of the past 15 years doing two things: building new server, storage, and network infrastructures -- and fixing them when they fall flat on their face for one reason or another. Over that time, I've seen one common theme emerge: There are very easy, seemingly unimportant things you can do when you build and maintain infrastructure that will save your bacon later when things go pear shaped.
Most of them involve writing stuff down -- you know, documentation.
I realize if there's one word that can cause a room full of IT folks to roll their eyes, it's "documentation." Usually, when you tell IT pros they need to document what they're doing, the thought of writing a book-length, screenshot-laden tome that would allow a monkey to manage a complex system comes to mind. Nothing could be more horrifying.
We don't need no stinking documentation
But guess what? As it turns out, a monster reference manual is the least useful type of documentation to have in an emergency, because it's so verbose you can never find what you're looking for. Plus, it's a pain to update, people won't do it, and it will quickly become inaccurate. If there's one thing that's worse than having no documentation at all, it's having inaccurate documentation.