The story of a 'System frickin' Administrator, not developer, you hear me?'

Traditional system administrators are feeling pressure to up their development skills to keep up with business demand. Can Joe look past the trees of his IT past and see the DevOps forest?

fence looking over

This is the start of a new series entitled: System Administrator to Developer. It will be told as a fictional story of Joe Schmoe and will cover Joe's realization that DevOps is where his career needs to go. This is the first post in this series that introduces our friend Joe and how he first gets introduced to DevOps the hard way.

Joe Schmoe is a system administrator at Acme Corp. Acme is an SMB that cranks out widgets as fast as it can. It has a small IT team with Joe heading it up which naturally forces Joe to wear lots of hats. He's the storage guy, server guy, VMware guy and the occasional desktop guy when his tech is on vacation. And he's damn good at it, if you ask him.

Joe strolled into the office one day after a busy weekend of the usual server maintenance to find that no one could access their mailbox. He thought this might happen because his team was doing some upgrade to their Exchange environment but thought he had tested things sufficiently. Apparently not. Now everyone from the janitor to the CEO is upset and clamoring to get access to their email ASAP.

"What's going on?!" thought Joe. He was sure he followed the step-by-step testing routine in the Word doc that his team had created ahead of time to ensure something like this did not happen when everyone returned to work on Monday. How could this happen?

A few hours into the incident, Joe found the issue. It ended up being a forgotten setting that was left out of the testing procedure. Apparently, the test environment the team was using was slightly different than the production environment, which prevented an important item to be added to the test procedure. Ouch!

After email had been restored, Joe was told to head into Mel's office immediately. Mel Stewart is the IT director and doesn't put up with any crap. He took a lot of heat from the CIO for this mishap and wanted answers ASAP. "What the hell happened, Joe?" Mel fumed. Joe was sure he was fired.

"We got 99% of the upgrade right but failed on that crucial 1%," Joe said. "It's unfortunate that everyone sees this as a 1% failure instead of a 99% success. It was one of the mailbox role servers. We took a snapshot of production a couple of months ago and thought that was a good representation of production, so we used that as a test. We had forgotten that we had to put an emergency exception in place on SERVER1 a few weeks ago bringing our snapshot and production settings out of sync. As a result, we didn't have proper testing procedures in place and forgot to check a particular configuration item that had some dangerous ramifications, as you saw. I'm sorry, Mel," Joe apologized.

Luckily, Mel saw this was a simple mistake and let it slide ... this time. "Why did you guys forget about this setting," asked Mel. "Couldn't you have confirmed you were working with the production copy before you ran the upgrade?"

"Yeah," said Joe, "but clicking through each setting in production again would have taken days! It took us forever to confirm all those settings in the first place. It wasn't feasible to do that right before the migration."

Mel thought for a second and said, "I've heard that Exchange Server has great PowerShell scripting support and you can do a lot with scripts. Could that be used to automate some of that stuff?"

"I'm not sure," Joe remarked. "I've heard about PowerShell before and all that scripting stuff, but neither I nor anyone on my team is a developer. That coding stuff is not what we do. We're system administrators and techs. Writing code is foreign to us and frankly, not our job."

Mel stared blankly at Joe. Joe's "not my job" comment was the wrong thing to say to Mel today after this fiasco. "I pay you to make our IT systems run efficiently to help the business run and to produce value for our company! I don't pay you to put your blinders on when faced with unfamiliar concepts! I expect you to pursue every option available to you to come up with a better solution!" Mel exclaimed.

Mel was angry. Why? All this time Joe had heard of an opportunity to make this process more efficient and had purposefully ignored it because he thought that was pie-in-the-sky developer stuff. "That's crap," Mel thought. Joe's not capable of coming out of his comfort zone a little bit and thinking about the bigger picture here.

Mel had an ultimatum. "Joe, the next time you come in here, I want you or at least someone on your team to get some training on automation and PowerShell. I want you to show me your team is capable of more than just clicking a mouse and doing things how you've always done them. Show me that this problem and any in the future can be prevented with some automation. If you can't do that, I'll find someone who can."

Joe was floored. Never in his 20-year career as a support person, helpdesk tech, and Windows system administrator has he ever been told he essentially wasn't good enough. It was a major blow to Joe's ego. Joe had no idea at the time but little did he know that that this encounter with Mel would be the spark that he desperately needed to light a fire under him to bring him into a more DevOps-focused mindset.


Copyright © 2016 IDG Communications, Inc.

How to choose a low-code development platform