Sometimes in the course of IT events, a project comes to a group and it quickly becomes evident that there are redundant skill sets. At times, this redundancy can lead to a cohesive team that works shoulder to shoulder, bringing out the best in each other and beating project time lines. At other times, a redundant skill set can lead to territorial behavior and clashes over "who knows best."
While managing helpdesk for a midsize company, I learned about a new ERP system that we would be rolling out. Two of our developers would be point of contact. The first, "Taj," had been involved in a similar project prior to coming on board and was going to take lead. The second developer, "Opie," had been assigned some front-end integration tasks on the rollout.
[ Got amazing IT tales, real-life experiences, lessons learned the hard way, or war stories from the trenches? Submit it to InfoWorld's Off the Record blog. If we publish your story, we'll send you a $50 American Express gift card. | Join or start an IT conversation in InfoWorld's Adventures in IT discussion area ]
Part of my responsibility in help desk was managing backups -- a truly thankless job. Opie, who'd been with the company for many years, dropped by my office late one day and requested that I add one of the development servers to the backup jobs. (My "office," for the record, was a converted, windowless room with a giant workbench that was covered with laptops and desktops in need of repair or waiting to be deployed.)
Opie wanted to make sure he had a copy of the database backed up. He said he didn't trust the aging hardware and it wouldn't fit on a floppy. (This was well before the days of thumb drives and CD burners.) It was a straightforward request, and as we had several extra license keys, I added an agent to the server and began doing backup of the directory to tape every night.
The next day, Opie came down and asked me to restore the database backups to the server. I asked him what happened and he shrugged his shoulders, saying that he had somehow corrupted the database and wanted to restore from backup. He said that some of the integration with the ERP system was getting tricky and he might need to go back to tape daily. As we only archived tape off-site weekly, the tapes from the last night's backup were in the library. I said that it wasn't a problem, just to drop me an e-mail and I'd pull it back from tape.
This went on for a couple of weeks, with Opie dropping by late in the afternoon and asking for the file back.
One day I received a call from Taj, asking if I could restore a directory from tape. He'd gone to restore a database and it looked like his local backups were screwed up. I said, "No problem," and, as I'd been doing for weeks, restored the file.
Minutes later, I received another call from Taj, asking if I had done the restore. I said that I had. There was a moment of silence, and he said, "I'll be right down."
He walked into my office a few minutes later, clearly irate. "When did Opie ask you to start backing up this server? It was Opie, right?" I told him that the request had been made a few weeks before. He looked at me for a moment, then spun on his heel and headed back to the developer pen. Sensing something might be about to go down (and that my name could be involved), I followed him.
"I know you've been screwing with my database for weeks," Taj literally pointed his finger at Opie, "and now I've lost all of my work." Opie leaned back in his chair, arms folded, and didn't say a word, just rocked in his chair. Taj looked at me, as if for assistance, and I shrugged my shoulders and said, "I just do the backups."
The director of IT was called in to settle the dispute. Over the course of a long, intense conversation, the details came to light.
Apparently, Opie felt like his input wasn't being acted upon and had been working out of his silo. Opie was trying to shoulder the job that Taj had been hired to do, along with the tasks he'd been assigned, because he felt he could do a better job.
Taj would come to work early and would leave before Opie. After Taj left for the day, Opie would back up Taj's files and then restore his own database from the tape backup. When Opie was finished working, he would restore the database from the local backup he'd created of Taj's work and then save his own work as the backup file. The backup job would take his data to tape, so in the event that Taj overwrote the backups, Opie would have a copy of his work.
If you're following who has a backup of their work and who doesn't, what happens next should be a foregone conclusion. The problem came when Taj hit an issue and, rather than work around it, decided it would be quicker to just restore from backup. When he completed the restore, he was looking at a database that he hadn't designed -- it was Opie's project that sat on the screen, staring back at him.
I'm guessing it was due to the fact that he had written a core program used by the company (and he was the only one who really understood it) that Opie wasn't fired. He was removed from the project, and in the rest of my time with the company, I don't think he was ever put in charge of anything else.
What I learned from this experience is that your data doesn't exist unless you have it in three places -- and at least one of those places should be under your direct control.