It's been nearly three weeks since Terry Childs was arrested on four counts of computer tampering and sent to jail on $5 million bail. In those three weeks, this event has taken turns to the strange, and wound up firmly in the land of the absurd. From bombastic claims in the press to midnight visits by San Francisco Mayor Gavin Newsom to pages of functional usernames and passwords entered into the public record, this case has certainly proven engaging.
Lost in all the drama is what actually happened. How could a large city government apparently lose control of its network, and how could its own characterizations of the system be so questionable?
[ Follow the Terry Childs saga blow by blow in InfoWorld's special report. | Read the actual court documents. ]
I've been covering this case in my blog almost since day one, and have been trying to figure out exactly what happened, reading between the lines of published articles, and reading court documents until the wee hours of the morning. Here's what seems to be true, what is clearly open for question, and what lessons business IT should draw from this saga.
First, despite the many news reports claiming that Childs had shut down all or part of the city and county of San Francisco's network, what actually happened was that Childs refused to provide his superiors the passwords to the city's core FiberWAN network, effectively preventing them from administering the network. The network continued to function, and no city applications, data, or resources were lost, unavailable, or otherwise inaccessible.
Just who is Terry Childs, and why was he so powerful?
Terry Childs, a Cisco Certified Internetworking Engineer (certification number 14018), was a member of the San Francisco DTIS, the city's IT department, for the past five years. As a CCIE, Childs shares this distinction with only 16,000 or so others across the globe. He was part of the group that built and managed the city's networks, and in the past several years had been tasked with bringing together the many disparate networks that ran the city. As the city's most experienced and advanced network administrator, he essentially single-handedly designed and built the FiberWAN, a city-wide network built on fiber interconnects and MPLS. This network is complex, and forms the core of all city services.
Following the completion of the FiberWAN, Childs looked upon his creation as art -- so much so that he applied and was granted a copyright for the network design as technical artistry. Skeptical of his colleagues' abilities, Childs became the sole administrator of the FiberWAN, and the only person with the passwords to the routers and switches that comprised the network. This state of affairs was widely known throughout DTIS, and Childs was the only point of contact for changes, troubleshooting, and overall management of this network.
Sources have stated that not only was Childs the only admin, he was always on call, 24 hours a day, 7 days a week, 365 days a year. As the only admin with the knowledge and access to the FiberWAN, he had no help. During the past few years, the DTIS staff has been significantly reduced due to budget cuts, keeping the city dependent on a sole admin for its core network.
The confrontation that started the standoff
On Friday, June 20, there was an altercation between Childs and Jeana Pieralde, the new DTIS security manager at the 1 Market Street datacenter in San Francisco. Until her promotion, she had been a city network engineer who worked with Childs. The city's court filings claimed that Childs harassed Pieralde, confronted her, and took photos of her with his cell phone. Fearing for her safety, Pieralde retreated to a room in the building, locked herself in, and called the DTIS CIO for help. The DTIS CIO then called Childs and the two had words. Childs subsequently left the premises. Why was Childs so upset? According to the city, no one had told him or others that Pieralde was auditing his network, and he perceived it as a threat or intrusion.
Childs disputed this interpretation of events, claiming in court documents that Pieralde was conducting clandestine searches of DTIS employee workspaces and had removed a hard drive from an office when he confronted her. He also denied taking photos of Pieralde.
What occurred over the next two weeks remains a mystery, but at some point, DTIS officials demanded that Childs relinquish the usernames and passwords used to access the FiberWAN network devices, and Childs refused to do so. He was suspended for insubordination on July 9.
In the court documents, the city stated that Childs was placed under surveillance and was arrested on the evening of July 12 as he was parking his vehicle near his home in the suburb of Pittsburg. At the time of his arrest, he was found to have $10,000 cash on his person and receipts showing that he had traveled to Sparks, Nevada, where he had looked at renting storage units. Following his arrest, police searched his house and workspaces. Police turned up 9mm and .45 caliber bullets, but apparently no weapons.
The possession of ammunition may have raised flags with the police, because 25 years ago, at the age of 17, Childs was arrested and convicted of aggravated burglary, and spent four years in a Kansas prison. In 1995, prosecutors said, Childs was again arrested in Kansas and charged with aggravated assault and carrying a concealed weapon. The case was reduced to misdemeanor weapons possession.
In addition to the bullets, police found documentation of the city network, including configurations, maps, and diagrams of the FiberWAN and possibly other networks. This was hardly surprising, considering that Childs was the lead network admin for the city and was on constant support duty for it.
Even following his arrest, Childs refused to divulge the passwords to the network. He offered to give them only to Mayor Newsom. Late on Monday, July 21, Newsom paid Childs a visit in jail, met with Childs for 15 minutes, and received the passwords. Newsom then gave this information to DTIS officials, and -- following a clarifying call to Childs -- DTIS was finally able to log into the routers and switches of the FiberWAN.
After relinquishing the passwords to Newsom, Childs' attorney filed a motion with the court for reduced bail. Considering that normal bail for a murder case is $1 million -- one fifth of what Childs' bail was set at -- this filing was unexpected. The motion was heard on Wednesday, July 23, and was denied. Childs would remain in jail in lieu of bail.
Where San Francisco seems to have made dubious claims
Once Childs was arrested, the claims against him began to mushroom. Many don't hold up to scrutiny.
Exaggerated claims of jeopardized systems. The city's legal justification for the arrest was the fact that Childs refused to give the passwords to DTIS officials, which effectively locked them out of administering their own network. But in the press reports that soon surfaced, statements ascribed to city officials made it appear that some or all of the data on the network was in jeopardy, including e-mail, 311 service (the one-stop phone number for residents to get help on city services), and law-enforcement applications. But these services do not appear to have ever been jeopardized. And Childs' influence over this network did not appear to extend to these services, only to the network itself.
Also during the bail motion proceedings, the city provided new documents that it claimed showed Childs was a threat to others and the city network. To back up these claims, the city offered evidence collected from Childs' computers, including a document labeled Exhibit A, which was an unredacted list of 150 VPN groupnames and passwords.
Access to VPN data portrayed as malicious. The portrayal of the VPN information suggested that Childs should not have had this documentation, even though he was the city's lead network admin and apparently had to maintain these lists as part of his job. But entering the VPN information into the court records made them public -- the San Francisco district attorney's office committed a significant security breach, opening up VPN access to anyone who cared to look at the document. Although the passwords alone were not enough to provide complete access to the city networks, they did constitute one part of the VPN's two-phase authentication configuration.
Nearly two days after the DA's office divulged these passwords to the public, DTIS changed all the passwords, locking everyone out of the city VPN services until they had reconfigured their client to the new passwords. Ironically, this was the first time the city network failed since Childs' arrest.
Contradictions over FiberWAN device access. Also, until these court filings on the bail issue, the city had claimed it could not access the FiberWAN network's devices. But four days before that bail hearing, the city claimed it had scheduled a power outage at the 1 Market Street datacenter. That power outage would have affected routers and switches running the FiberWAN network. In the court filing four days later, the city contended that Childs had "booby-trapped" the network to collapse during this power outage by not writing the device configurations to flash on some number of routers. A local news report stated that "experts caught the problem in time and transferred data to permanent files, [Assistant DA Conrad] del Rosario said."
This statement contradicts the city's stance that it had no access to these routers, as there is no way it could have written those configurations to flash, or save them anywhere, on July 19 if it could not access the devices. By the city's own admission, it did not have that access until after midnight on July 21, two days after this shutdown was scheduled.
Other news reports have stated that the city cancelled the shutdown when it learned that the network had been "booby-trapped." But again, without the passwords, the city could not have known the state of those routers, nor could it have known whether the configurations were saved to flash memory.
The city also highlighted the fact that Childs had a copy of the datacenter shutdown memo in his workspace, and presented this as evidence that he had planned to cause the network to fail. Given Childs' 24/7 support responsibilities, it’s far more likely that he had the memo because it was sent to him because he had those duties.
Common practices portrayed as nefarious. The documents filed by the city in opposition to Childs' bail reduction contained many vague references and claims of nefarious actions. But to those with experience in network administration, many of these activities seem like common practice.
For example, the documents portrayed the fact that Childs had configured some number of routers to disable password recovery as a subversive action, when it's common to use that function to secure routers and switches that cannot be physically secured.
They also stated that Childs had several modems in his workspace, hooked up to computers, and that Childs used these modems to access the network remotely without logging or auditing. It seems much more likely, however, that they were used as dialup/dial-back access for Childs to perform emergency work during off-hours.
The documents claimed that he had installed sniffers on the network, but of course there are sniffers on most large networks, installed by their administrators. For years, Cisco has manufactured sniffers specifically designed to be placed within network cores, and the use of such devices is not just common, they’re almost mandatory as a best practice.
One statement made in the original affidavit for Childs' arrest warrant claimed that Childs' pager went off after he had surrendered it to DTIS officials, and that the page was "sent from one of the routers on the network." This was portrayed as proof that Childs had remote access to the network and was thus a danger. This was a key fact in the arrest warrant, even though it's far more likely that this page was from the network monitoring application What's Up Gold, which Childs stated that he used to keep tabs on the network. In fact, Childs states that at least one of the modems found in his workspace existed for just that purpose. This is an extremely common form of network monitoring, and not a subversive action.
Throughout the court documents, the city offers very little of technical substance relative to Childs' actions. To those unfamiliar with the intricacies of network architecture and administration, many of their claims would seem to be clear evidence of wrongdoing, but in reality, are common practice in networks the world over.
A claim that could backfire. In another twist to this case, the city may have undermined its case against Childs. In the court document opposing Childs' bail motion, the city claims that Childs had "installed three modems that were connected to the FiberWAN networks, two in the locked room he maintained and a third in a locked cabinet near his cubicle. Cisco engineers have indicated that the types of modems the Defendant installed bypass logging, auditing, and security measures of a secured network. Further, anyone can gain access to the network by dialing into these unsecured modems, risking the security of the network." The city also claimed that Childs could have access to more than 1,100 other devices, including routers, switches, and modems, and possibly wireless access points.