We all like to think we learn from mistakes, whether our own or others’. So in theory, the more serious bloopers you know about, the less likely you are to be under the bright light of interrogation, explaining how you managed to screw up big-time. That’s why we put out an all-points bulletin to IT managers and vendors everywhere: For the good of humanity, tell us about the gotchas that have gotten you, so others can avoid them.
As it turns out, our many contributors to this article had a lot to say -- but precious little to say on record. Names may be withheld, but the lessons are still potent. We’ve distilled this glut of information down to the top 20 mistakes -- instances in which wrong decisions can lead to costly project overruns, business disasters, and in the worst cases, lost jobs. Read on, takes notes, and avoid.
1. Botching your outsourcing strategy
Mistakes relating to outsourcing could easily fill our top 20 list on their own. There are two different flavors. The first is the sin of commission: outsourcing important IT functions to avoid the hard work of understanding them. Relinquishing those functions can make it hard to get simple things done.
The other mistake is to hold on to functions that could easily and effectively be outsourced, such as running your own messaging environment. IT organizations with an overt bias against outsourcing could be courting disaster. For example, one CTO we interviewed took over operations for a Manhattan-based online services company, only to discover that the Web-hosting infrastructure for all mission-critical and revenue-producing applications was in-house because the IT staff didn’t trust third-party operations. When the great blackout of August 2003 darkened parts of Manhattan for as long as 28 hours, the company’s UPS systems kept everything running for only a relatively short time -- while competitors at well-provisioned Web-hosting companies experienced no downtime.
2. Dismissing open source -- or bowing before it
For better or worse, many IT shops are susceptible to “religious” behavior -- a blind, unyielding devotion to a particular technology or platform. Nowhere is that more true than with open source.
On the one hand, the most conservative IT shops dismiss open source solutions as a matter of policy. That’s a big mistake: Taking an indefinite wait-and-see attitude toward open source means passing up proven, stable, and scalable low-cost solutions such as Linux, Apache, MySQL, and PHP. On the other hand, insisting on open source purity in your IT operation can delay progress, as developers are forced to cobble together inferior or unwieldy open source solutions when more appropriate commercial software solutions already exist.
Open source software is not inherently better than commercial software; it all depends on the problem to be solved and the maturity of the solution being considered.
3. Offshoring with blinders on
Any list of IT mistakes would be incomplete without a mention of offshoring. The experience of one vice president of operations provides an instructive cautionary tale. At his previous employer, the vice president opened a branch office in India for software development and encountered numerous surprises, many counter to conventional offshoring wisdom.
At the time, India had been experiencing an IT employment boom similar to that of Silicon Valley in the late ’90s. According to the vice president, the workforce was not stable as a result. Transportation difficulties and the importance of time with family in Indian culture meant that employees generally worked eight-hour days -- the concept of the Silicon Valley engineer who goes sleepless at release time was, well, foreign.
In the end, the cost of offshoring the branch office was only 20 percent less than the going rate in the United States, and for cultural reasons, far more face time than initially expected was needed to ensure the commitment U.S. management demanded -- which resulted in trips to India at least once per quarter. The vice president emphasized that offshoring can indeed work but said it’s a mistake to assume that managing offshore IT is in any way equivalent to managing local IT or that cost savings will be as dramatic as you might expect.
4. Discounting internal security threats
IT managers focusing on external threats can easily lull themselves into a sense of false security. According to Gartner, 70 percent of security incidents that incur actual losses are inside jobs, making the insider threat arguably the most critical one facing the enterprise.
Of course, not all insider threats are born of malicious intent. In September 2004, HFC Bank, one of the United Kingdom’s largest banks, sent to 2,600 customers an e-mail that, due to an internal operator error, made recipients’ e-mail addresses visible to everyone else on the list. The problem was compounded when customers’ out-of-office messages -- containing home and mobile phone numbers -- responded to the mailing.
Even malicious acts are often carried out using very little technical sophistication. In a joint study released this year by CERT and the Secret Service, 87 percent of insider security breaches were found to have been achieved using simple, legitimate user commands, suggesting that IT needs to be vigilant about granting only necessary privileges to end-users. Identity management with specific permissions can help.
5. Failing to secure a fluid perimeter
IT’s responsibility now extends to Starbucks and beyond. The increasing mobility of workers, combined with the proliferation of public wireless hotspots and broadband in the home, means that IT is now responsible for securing systems on networks it does not control. In this environment, solid security means implementing host-based firewalls that will provide some level of protection on an unsecured broadband connection at home or at sites with public Wi-Fi access.
If you’re an experienced IT manager, you might feel comfortable with the top-of-the-line firewall you purchased three years ago. You configure it to block all incoming traffic except port 25 for inbound e-mail, and your employees generally make outbound WAN connections to the Web via ports 80 and 443. This is a common approach, but in a more decentralized IT environment, centralized approaches to network security are no longer sufficient. By encrypting traffic on your internal LAN, you will better protect your network from insider threats and from intruders who might have hopped onto your network via rogue wireless access points.
6. Ignoring security for handhelds
Although even inexperienced IT managers recognize the need for username/password authentication on network resources and desktop and laptop PCs, most IT shops still seem to be in a “wild West” phase when it comes to handheld devices.
A CTO of a wireless software company tells us about a venture capitalist who lost his BlackBerry on a business trip while he was in the middle of closing a highly sensitive, confidential deal. The BlackBerry wasn’t password-protected, so even after the panicked venture capitalist contacted his IT department to have e-mail delivery to the device stopped, anyone who happened to pick up the lost BlackBerry could read e-mails already received.
In this case, the minor convenience of not requiring a password had major implications. Ignoring the security of easily lost devices, particularly those belonging to key executives that traffic in confidential information, is a recipe for disaster.
7. Promoting the wrong people
As CTO or CIO, rewarding your top technologist with a promotion to a management position might seem like the right thing to do. But when a technologist is not ready to give up constant, hands-on technology work in favor of more people-oriented management duties, it could be a mistake you’ll regret on many levels.
One vice president of IT painted a grim picture of such a decision: The promoted employee could be resented by former peers and might not like the new management duties, which could lead to poor performance. Even worse, the new manager might feel compelled to cling to the ill-fitting position because the old position might no longer be available.
Just such an experience put this particular vice president in the tough position of having to deal with a new manager’s performance problems, which led to a double whammy: A top technologist left the company, and the new manager still had to be fired.
Management training can help avoid such disasters. But use your gut. Either the aptitude is there, or it isn’t.
8. Mishandling change management
The former CTO of a computer equipment manufacturer describes one situation in which a talented, but perhaps overly ambitious, systems administrator decided to make seemingly simple changes to a set of critical servers during routine maintenance.
While this individual was making the changes, all of which had been agreed on and planned in advance, he decided on his own to upgrade BIND (Berkeley Internet Name Domain), the open source server software that powers mission-critical local DNS for many companies.
A few hours later, the entire business was at a standstill, as all DNS functions failed. Reversing the “one small change” took hours, and millions of dollars in revenue were likely lost as a result. The lesson is that even talented employees can cause major problems when they don’t follow change management procedures.
Remember, change management is cultural. It all starts at the top: If IT management cuts corners, so will IT staff.
9. Mismanaging software development
In his seminal book The Mythical Man-Month, Frederick Brooks posited that planning software-development projects based on per-unit “man-months” ultimately does not work due to the unique nature of software development.
Even if the building of software could be broken into easily managed, interchangeable time units, the vast productivity difference between the best coders and merely average ones means IT managers might get their best work out of fewer, but more talented, programmers doing their work in less time.
Henri Asseily, CTO of BizRate, tells us via e-mail, “The right individual will always create better and faster core software than a group of people [will]. Everyone in every industry talks the usual talk of, ‘We invest in people,’ or, ‘Our people are our greatest asset,’ but nowhere is it more important than in IT. Simply put, a great programmer is 100 times more valuable than a regular programmer.”
The mythical man-month has been part of software lore since Brooks’ book came out 30 years ago, but many IT managers still plan projects and staff them based on this disproved paradigm. Holding on to this method might lead a naïve IT manager to staff a project with the right number of people for a defined amount of work, but CTOs such as Asseily insist that getting quality people is most important.
“IT managers should devote most of their free time to [finding] the best people. Almost nothing else matters, really,” Asseily says.
10. Letting engineers do their own QA
Not allowing engineers to do their own QA is an axiom of software development, but for small software development teams, there is always the temptation to cut corners. In fact, sometimes management colludes with developers to enable the practice. One CTO relates a situation in which a software development project was running significantly behind schedule and the lead developer had begun to do his own QA to try to speed up the release process. To make matters worse, the lead developer had planned a vacation that was approaching rapidly. A day before the vacation commenced, the developer pronounced all outstanding bugs resolved, and the system was released into production. By the time the developer arrived at his tropical destination, the system was crashing to the point of being unusable. Many of the existing bugs had not been corrected because the developer had not tested thoroughly or formally. Allowing engineers to perform their own QA is akin to allowing defendants to be the judges and juries for their own trials.
11. Developing Web apps for IE only
Despite the fact that mission-critical applications continue their march onto the Web browser and that Windows continues to dominate the corporate desktop, Web developers should avoid the temptation to develop applications only for bug-ridden IE. IT shops that insist on using IE for Web applications should be prepared to deal with malicious code attacks such as JS.Scob.
First discovered in June 2004, JS.Scob was distributed via compromised IIS Web servers. The code itself quietly redirects customers of compromised sites to sites controlled by a Russian hacking group. There, unwitting IE users download a Trojan horse program that captures keystrokes and personal data. Although this might not sound like a threat to corporate IT, keep in mind that employees often use the same passwords across corporate and personal assets.
Many enterprises may not be able to avoid using IE. But if you make sure your key Web applications don’t depend on IE-only functionality, you’ll have an easier time switching to an alternative, such as Mozilla Firefox, if ongoing IE security holes become too burdensome and risky for your IT environment.
12. Relying on a single network performance