I don't like automation -- I love it. I whisper sweet nothings, come 'round with flowers, and buy milkshakes for automation. I've even stood outside the window with a boombox for automation. I will go out of my way to automate tasks that, while they are not terribly tedious, I don't want to have to remember exactly how to do them somewhere down the road, when months have gone by since the last time I had to relearn them.
Scripting such an event becomes its own documentation -- and it's way easier than grepping through command history to figure out how the hell you did it last year. Me and automation, we're solid.
But as the lines of love and hate are so very close to each other, I completely abhor bad automation. There's plenty of it to go around.
The two most common forms of automation in deep IT are scripting common tasks across multiple systems on a regular basis or, as mentioned, scripting tasks that may not be common but are tricky to navigate. The latter are kicked off as part of a manual process by an admin or are triggered by the actions of a higher-level application. Regardless, they operate at low levels and will usually even perform tasks that the requesting user does not have privileges to do manually.
As a blindingly simple example, core admins don't want to bother with common user issues such as account creation, account lockout lifts, renaming, and other tedious tasks that may not be possible through self-service portals. Thus, there are tools that allow helpdesk folks to perform these functions. A deeper example may be a developer or engineer kicking off a server process to go through a series of databases and purge all data prior to a particular date. All the dev knows is that when they run db-purge <projectname> —before 2014-08-01, stuff happens and all is well.
The idea is that whether we're automating tedious tasks at root privilege level or giving unprivileged users the ability to help themselves, we're saving our own cycles. We are gaining time by writing these things. However, that's not always how it works out.
The problem starts when someone starts escalating the automation and piles automation on top of automation until no man may name the original tool. There are many cases when adding on to existing tools is good and laudable, but there's a slippery slope where cobbling together a variety of automation scripts in an effort to lighten the workload actually winds up achieving the opposite, and horrible things happen.
Perhaps that database archive code was meant to be used sparingly at first, at the outset of a project, and by one developer. As weeks turned to months and the project grew, that script was tied into further automation efforts by other developers who used it in their prep scripts to make sure that they were only working with recent data. Then, those prep scripts were pulled into use elsewhere, then linked by another group that was using the same data for a different project, which started adjusting some parameters. It then gets lost in the mix as the project moves ahead, and nobody notices right away.
Eventually, the day comes when someone's asking why there's only a week's worth of data in the database and nobody knows how that happened. Now a full investigation is launched to figure out exactly how that happened, and who broke it. Turns out it was the new guy who thought he was merely requesting a week's worth of data, not incinerating everything before that, because he was working with a new tool that obfuscated the actual function.
If you ever start automating by calling existing, home-grown automation scripts, you might want to take a step back and consider the bigger picture. It may be that you would be better off working at a lower level to begin with, where you're less likely to build a Rube Goldberg machine of unintended consequences. If left alone in the wild, that's exactly what some systems look like -- an indecipherable series of scripts that is essentially magic -- until it all inevitably falls apart.