Our goal was to minimize user logins at the end of the day, not to get rid of them entirely; we didn't want to antagonize anyone either. Therefore, I was determined to make the job killer easy to circumvent. If someone was dead-set on staying logged into the system by running a program, that was fine. But I've seen users go to great lengths to circumvent this kind of tool -- for example, by creating a completely useless and continual subprocess loop that would prevent their session from ever going idle, in direct opposition to the goal of freeing up system resources.
As for system reboots, we always gave advance warning and would place a courtesy call to anyone still logged in before cut off transmissions. As expected, the job killer reduced the number of users in the system -- and our hair.
Catching the culprit ... or not
We ran into a snag. Once this utility was in place, overall it seemed to work well. However, on a few occasions it mysteriously died. Everything checked out fine as I narrowed down the possibilities. Eventually I came to a conclusion: In all likelihood, someone with privileged access had shut down the process killer.
To find the wrongdoer, I used a trick I'd heard about at a recent computer symposium. I had learned that one could dump the command buffer for any current user on the system to see what commands were issued. Often you could see every command the user issued on their terminal since logging into the system.
It was limited only by the total size of the collection of commands because the size of the command history area in system memory for each user was fixed. If the user logged out, the command history would be gone. For once, it worked to our advantage that some users did not want to log off. The next time my process died, I looked at the command buffers for several users and found the command that stopped my process. I had caught the person red-handed and presented a printout of these commands to my boss.
We confronted the user and asked him not to do this, but he said he had never entered those commands and someone else must have used his account to do the deed. It wasn't worth pressing the issue any further at that point, so my boss gave the user the benefit of the doubt and gently asked him to keep his terminal and password secure. Whether or not that user was guilty, we didn't see the problem again.
Besides being a fun project, this endeavor taught me a lesson: Just because I could show an action originated from a particular user account does not always prove it was done by that particular user. More evidence may be required to tie an event to an actual person. Proceed cautiously -- it could be their job on the line.
Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to firstname.lastname@example.org. If we publish it, you'll receive a $50 American Express gift cheque.
This story, "Watch out, rogue users — the data detective is on the case," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.