It's a fact that the help desk works on a priority system. For example, if the HR department determines a file is missing, you drop everything and attend to them. Otherwise, it can escalate into a Big Deal, even when it turns out to be a minor incident.
Thus, I jumped into service when a panicked and angry HR contacted us with a major issue. Not only was an important employee file missing, but the whole subdirectory containing several of this employee’s files had “mysteriously” vanished. They demanded to know where it would be and for us to find it -- yesterday.
I asked questions, and more information came out: This was a standard directory created upon each person’s employment with their benefits and other such information. The files, therefore, were of a nature that didn’t need to be accessed very often.
Where oh where did the files go?
I arrived at the HR department manager’s office to see for myself and verified that the files were, indeed, missing. They must have been deleted somehow, which neither of us was happy to hear. It was time to put our backup and disaster recovery (BDR) plan to the test.
Under the watchful eye of the HR department manager (always good to have supervision, right?), I began the task of loading various backups from prior months, as nobody in the department was sure when the directory was deleted. This was important enough that pretty much the whole HR department was involved at that point. All they were sure of was the name of the directory and the files that “should” exist in it.
We began working our way back through the prior three years’ data that resided on our BDR server. Alas, the directory didn’t exist in any of those backups. I could begin to see concern in the manager’s face as to how the files could have been deleted so long ago. I was requested to go back even further. This would require accessing “off-site archived data.”
I obtained the necessary storage material, loaded it, and called the HR manager to come back and go through it with me. Working our way back through another two years of data (this was the original hire date of the employee) also proved futile.
Now the blame was focused on IT's backup procedure and the question of whether we were backing up all subdirectories. I quickly laid that charge to rest by accessing several other employees in that five-year-old backup, showing that the subdirectory and files existed under their names. Every subdirectory and file the HR manager threw at me to check was there -- except the ones that initially prompted this visit.
The HR manager was finally mollified and had to admit that the HR staff must have been negligent and never created the directory or electronically recorded the data. Only one conclusion could be drawn: HR scanned the paper documents in for storage years ago, and someone missed adding these into the computer and shredded the hard copies.
I was glad to vindicate my own department, but I felt for the HR personnel who had to go back to the employee and rectify this mess of their own making. That is never fun.
I was thankful for our BDR plan and learned it's also wise to have the head of a department or whoever is at the top of the chain of command next to you when doing recovery work. That way, they can view your procedures and witness the care we give to storage. Chalk up a win for IT.