Why San Francisco's network admin went rogue

Last Sunday, Terry Childs, a network administrator employed by the City of San Francisco, was arrested and taken into custody, charged with four counts of computer tampering. He remains in jail, held on $5 million bail. News reports have depicted a rogue admin taking a network hostage for reasons unknown, but new information from a source close to the situation presents a different picture.

In posts to my blog, I postulated about what might have occurred. Based on the small amount of public information, I guessed that the situation revolved around the network itself, not the data or the servers. A quote from a city official that Cisco was getting involved seemed to back that up, so I assumed that Childs must have locked down the routers and switches that form the FiberWAN network, and nobody but Childs knew the logins. If this were true, then regaining control over those network components would cause some service disruption, but would hardly constitute the "millions of dollars in damages" that city representatives feared.

Apparently, I wasn’t far off the mark. In response to one of by blog posts, a source with direct knowledge of the City of San Francisco’s IT infrastructure and of Childs himself offered to tell me everything he knew about the situation, under condition that he remain anonymous. I agreed, and within an hour, a long e-mail arrived in my in box, painting a very detailed picture of the events. Based on this information, the case of Terry Childs appears to be much more – and much less – than previously reported.

A man and his network
It seems that Terry Childs is a very intelligent man. According to my source, Childs holds a Cisco Certified Internetwork Expert certification, the highest level of certification offered by Cisco. He has worked in the city’s IT department for five years, and during that time has become simply indispensable.

Although Childs was not the head architect for the city’s FiberWAN network, he is the one -- and only one -- that built the network, and was tasked with handling most of the implementation, including the acquisition, configuration, and installation of all the routers and switches that comprise the network. According to my source's e-mail, his purview extended only to the network and had nothing to do with servers, databases, or applications:

“Terry's area of responsibility was purely network. As far as I know (which admittedly is not very far), he did not work on servers, except maybe VoIP servers, AAA servers, and similar things directly related to the administration of the network. My suspicion is that you are right about how he was 'monitoring e-mail'; it was probably via a sniffer, IPS, or possibly a spam-filtering/antivirus appliance. But that's just conjecture on my part.”

Like many network administrators who work in the rarified air of enterprise network architecture and administration, Childs apparently trusted no one but himself with the details of the network, including routing configuration and login information. Again, from the source's e-mail:

“The routing configuration of the FiberWAN is extremely complex. Probably more so than it ought to be; I sometimes got the feeling that, in order to maintain more centralized control over the routing structure, [Childs] bent some of the rules of MPLS networks and caused problems for himself in terms of maintaining the routing.