Guess who finally quit smoking
An IT team saves the day when a sales department's key server calls it quits
Follow @ifw_otrI worked for a sales organization in a large company. In this environment the sales group had their own servers for reporting development. When reporting systems matured enough, they were turned over to the IT group, which used a formal change management methodology.
In 2007 we were reviewing systems and architecture and noted that one of our key servers was a good candidate to be replaced by upgraded hardware located in a formal datacenter under IT control. At the time it lived in a closet in a sales center -- certainly a far cry from a state-of-the-art datacenter. Replacement parts were no longer available.
[ Do you have an IT tale when something went right, a war story, or a lesson learned? Submit it to InfoWorld's Off the Record. If we publish your story, we'll send you a $50 American Express gift card.]
This server, seven years old, had started out as a report development convenience for sales -- it pulled information from several phone management servers and kept an archive of the consolidated data. This archive strategy allowed the phone servers to purge data older than one month. Eventually more phone servers were added until they numbered at least three dozen. Yet all of their information was consolidated on this one server for convenience of reporting. With the increase in volume system, performance had begun to suffer, hence the need to upgrade. Often the daily backup was taking more than 24 hours to complete.
In December 2007 the sales organization engaged the IT organization to develop a migration plan. Sales took a backup of the database and provided it to the IT group so they could look things over, size the requirements, and develop a plan for the changeover. IT was doing a number of these migrations so there were already plenty of people in the queue ahead of us, so we mutually agreed that doing the changeover in July of 2008 would work best. We had monthly meetings to ensure everything was on track. At the end of April 2008, IT took another backup of the server to capture any schema changes. The purchase orders had been filled out and were due for on-time delivery of the new hardware. Software licensing was all in order.
On a sultry summer day in early June 2008, the automated data pulls from this server started timing out. It was impossible to connect to the server. Other groups were experiencing the same symptom. A conference call was set up and we all dialed in. One of our team members, who worked in the site that housed the server, came on the line and said, "Hey! Guess who finally quit smoking!" We all asked who, and he said the name of the server. The group let out a collective shudder.
