I value the experience gained through on-the-job trial and error, but as my career continues, I find it both difficult and maddening to stand back and watch as younger techies encounter the process. On the one hand, I want to swoop in and help them head off problems and not see them struggle or waste time -- I've been there. But on the other hand, if they don't listen to the advice, I'm often the one called by the boss to clean up the problem. It's a lose-lose scenario.
One company where I worked had a high number of these eager yet overzealous employees. They were competent in their jobs. But they -- and the work -- would have benefited if they had slowed down a bit and thought issues through ahead of time.
Case in point: Our company was expanding, which was good, but it meant a lot of extra projects for the IT department. Along with the mergers came more locations to set up and support, especially since some of the buildings had not been constituted to house the tech equipment we'd need.
On site and off track
First up were inspections of the new sites. We needed to know the layout, figure out how to set up the network and server rooms, and take care of all the small but important details that go along with such an information-gathering expedition.
A couple of these employees were eager to handle the task and travel to these locations, so the boss agreed. I've done plenty of these trips, and the timing was miserable: It was late fall and very cold. I didn't mind staying home.
The crew returned with pictures of existing computer equipment and ideas about how to set up the server rooms, and they compiled a list of additional servers, switches, and PCs required. After we received and configured new equipment at HQ, we shipped the appropriate pieces to the three locations. This included toolboxes and cabling equipment because we'd have to run wires.
Once we received notice of the packages' arrival at their destinations, it was time to schedule a whirlwind tour to do the installs. I was concerned that not enough time was allotted to do the job and advised adding another day to the mix, but the suggestion was vetoed.
For the installs, I again manned the fort while the small crew traveled. I didn't mind -- it was even colder now in the dead of winter.
The trip was completed in the allotted time, although they admitted when they got back that they only got three hours sleep one night and should have had another day to do a better job. I suppressed the "I told you so" comment that so very much wanted to escape.
The next months passed with only minor problems at the locations. Most issues were fixed remotely or through vendor services.
A prickly problem
Fast-forward to a summer of blazing heat.
One Sunday night I was on call and my phone started receiving notices from one of these remote locations. The server was overheating, then going to shut down, but reporting all was OK. This cycle repeated every minute. There was nothing I could do since no one in the tech department was on site. The office personnel would be arriving in a few hours, so I decided to let them sleep and prayed the server would make it.
The next morning I pinged the server and found it was up. I logged into it, and all indications showed it was performing fine. One of the techs who had done the install arrived, and I questioned him about what could be going on. His answer told me all I needed to know: "That server room doesn't have A/C."
I asked why. He replied that they had found out during the install that they couldn't squeeze a server into the location's air conditioned offices and, thus, had set it up in a small adjacent building. It was cold when they were there, so they figured it would be fine. Plus, they were running out of time during the install and needed to get done.
I called the location. An employee went to investigate and discovered that a thermometer hanging over the server read well over 100 degrees. A fan was deployed to keep it "cool."
I tried to find a good balance of watching the ensuing discussions about "what to do now" from the sidelines, and giving advice if asked but not enough so that I would be assigned the cleanup job. You never stop learning or making mistakes during the course of a career. But the nature of what lessons you need to learn certainly changes.