The revelation didn't happen immediately, but a day or two later he thought to run a packet capture on the network traffic departing that dev system and collected several packets of this odd ICMP ping traffic. Peeling the packets apart in Wireshark, he noted that there appeared to be a custom payload pad in the packet. A few minutes later, he'd peeled out the 16 bytes in that pad and translated them from hex to ASCII. A minute after that, he successfully logged into the core switches.
If nothing else, the fired network admin was telling the truth -- the password was definitely on the network, but only where a jackass might put it.
Jackass IT stunt No. 4: Stick an eraser on the motherboard
Note to all jackass IT practitioners: Document your hacks. After all, you never know when you might need to repeat your feat of asinine brilliance -- or prevent it from being undone.
It was a real head-scratcher: a simple RAM upgrade to a production server that left the server unable to power up. There were no POST or beeps from the mainboard -- just the silence surrounding a very big problem.
Troubleshooting step No. 1: Put the original RAM back in -- no effect. Pull and reseat every interface card -- all cards snug in their slots, and no change in behavior.
There are certainly instances when computers give up the ghost for no apparent reason, but this was a relatively new server that had never shown issues in the past, and a process as benign as RAM replacement shouldn't have caused such a major problem.
Anyone who's been in this kind of situation knows the sort of questions that were asked: Was there anything you noticed about the server when you opened it up? Did you take anything else out of it or put anything else back in? Were you drunk?
The answers to all the questions were no -- except one. Upon reflection, the admin who performed the RAM upgrade suddenly remembered having seen a rubber eraser on the mainboard when he opened the cover, and that he'd removed it before putting the server back in the rack. This caused a stir as everyone contemplated how a rubber eraser could have contributed to the well-being of the server or its downfall.
Then another admin asked where the eraser was. It was found on a desk and inspected. It was discovered that it was indeed a rubber eraser, with a crease down the middle. The same admin walked over to the server, opened the cover, and set the eraser on top of the SCSI adapter, where the crease seemed to fit perfectly, and replaced the cover. He hit the power switch and the server booted immediately.
It seems that an unknown admin (at least, no one would cop to it) had solved the problem of a popping SCSI adapter by using the eraser as a shim between the case and the card and hadn't mentioned that fact to anyone else. Naturally, the long-term fix for this problem was to tape the eraser to the underside of the server cover, next to a note saying, "DO NOT REMOVE ERASER."
Jackass IT stunt No. 5: Flat-line your heartbeat server
Customized safeguards that keep critical systems running are fertile playgrounds for daredevil IT hacks. Built into systems to prevent major problems, all too often these safeguards become major problems themselves, and sometimes it takes the IT equivalent of open-heart surgery to keep vital systems running -- with little more than a laptop and a few lines of Perl.