My story on Terry Childs and the San Francisco City network "hostage situation" ran late in the day on Friday. Since then I've received numerous emails, some offering more information, some thanking me for providing some clarity to what is still a murky subject, and some asking more questions than I have answers for.
In order to keep things together in my mind, and perhaps for those looking for information on this story, I'm going to attempt to relay what I know so far, including some new information, some hypotheses, and -- unfortunately -- some more questions.
[ Follow the Terry Childs saga with InfoWorld special report: Terry Childs: Admin gone rogue. ]
Now that we have some background on Terry Childs, he seems somewhat familiar. In my life as an enterprise consultant, I've seen more than a few IT admins that fit his mold -- extremely intelligent individuals that lack the social tools or the will to conform within a highly political environment. The best thing that could possibly happen for people of this nature is to be left alone, trusted implicitly, and basically protected from the political machinations that seem to consume some organizations. This type of employee can be extremely valuable, but also tend to have short fuses, very little patience for the mundane, and are a high risk for burnout.
In fact, I must admit that I may share some of his traits, including the reluctance to allow marginal admins access to core network resources. While some may think that this is hubris or arrogance, in my case, it's based on experience. There are few things more frustrating than to complete a very complex network implementation only to have another admin blow it up through ignorance or incompetence. It's happened to me many times.
It's quite difficult to accurately convey the stress and effort required to build and maintain large complex networks to those with no real frame of reference. I've done it for years, building networks for city governments, universities, hospitals, and private companies. At some point, a network moves beyond "straightforward" complexity, and almost becomes a work of art. Whether it's a clever iBGP VPN failover for a large MPLS-based WAN, an OSPF-based ISDN dialback configuration, or a novel method of route injection through a third-party cloud, there are instances where network architects and admins need to color outside the lines to provide a needed service or measure of redundancy. It's at this point that the proverbial wheat is separated from the chaff in terms of network administration.
I get the feeling that Terry Childs subscribed to this theory, and at some point felt that there was nobody else that could be trusted to understand what he had built. As the lone CCIE, he was apparently well beyond his peers in skills and knowledge, and felt that he, and only he, could handle what he had created. Of course, this should have been noticed by his colleagues and dealt with by his superiors, but it obviously wasn't. If it was widely known that he was the only one with the logins to the FiberWAN network, then I find it very disturbing that this particular problem hadn't been solved well before now.
Also at issue is the way that the city dealt with the problem once it finally noticed. Generally speaking, you don't have your highest-level network administrator jailed for computer tampering if no actual damage has been done. Even a civil proceeding seems harsh given the circumstances, but to bring criminal charges? Unless there are large parts of this story still to be told, that seems like a very extreme measure to take for what appears to be simple insubordination. I would also think it unadvisable to make public statements claiming "millions of dollars" in damages and consistently confusing the network with the services, applications and data that ride on top of it.
To reiterate the technical details as I understand them, the city's network is functioning normally. There are some number of network devices that cannot be accessed by administrators, but all of the applications, data, and services that the network supports are fully operational. I want to make this perfectly clear: The actions of Terry Childs do not appear to have caused any disruption in normal operation of any city resources. Obviously the lack of access is a significant problem, but it is not impairing the normal functions of the city at this time.
However, I have received some more information that may change a few minds. It may be the case that Childs did in fact modify the logins for other routers and switches within the city network, not just those under his direct purview. If this is true, and he did this in order to prove a point or in retribution for some perceived wrong, then the case may take on a different light. I have no hard proof of this one way or the other, and my original source has been silent, presumably due to the publicity surrounding this case and this story.
This brings me to my questions. I have quite a few:
Who's investigating this case? Has the network been declared a crime scene? Can a Cisco switch configuration be entered into evidence? If so, who is providing the required information to the police? If the FiberWAN network is as complex as it appears to be, are there CCIE-level forensic networking experts employed or contracted by the San Francisco police department? Who's inspecting ACLs for backdoors? Who's poring through routing tables looking for clues, or evidence of wrongdoing? How extensive was Childs' reach?
There are only around sixteen thousand CCIEs in the world. Since Cisco split the certification into different technologies, we might assume that there are maybe ten thousand individuals in the world that could match Terry Childs' switching and routing skills. Are any of them actively involved in the investigation of this case? If Childs has in fact been offering to give the city access to the network since last Tuesday, as his lawyer claims, why hasn't this happened by now?
In short, how on Earth does the city of San Francisco plan on prosecuting Terry Childs? How much will it cost for them to get rid of their most advanced networking expert?