An outside scoop on San Francisco's network lockout

If you haven't heard, San Francisco is being held hostage. At least, the city's new network is being held hostage. It seems that Terry Childs, a disgruntled network admin took it upon himself to lock out all the other admins from "the city's new FiberWAN network," and is currently hanging out in jail, holding the keys to San Francisco's kingdom. There have been many articles written about this event, and they al

If you haven't heard, San Francisco is being held hostage. At least, the city's new network is being held hostage. It seems that Terry Childs, a disgruntled network admin took it upon himself to lock out all the other admins from "the city's new FiberWAN network," and is currently hanging out in jail, holding the keys to San Francisco's kingdom.

There have been many articles written about this event, and they all share an obscene lack of detail. The "network" as used in these pieces could be interpreted as just about anything from one or more servers, the network switches and routers, some storage servers, or any combination thereof. This quote from an IDG news item unfortunately doesn't offer much clarity: "The new FiberWAN handles city payroll files, jail bookings, law enforcement documents, and official e-mail for San Francisco." There are an awful lot of moving parts in that description. We obviously don't know what part(s) they're talking about. Thus, it's terribly difficult to draw a clear picture of what's actually transpired.

A clue as to the actual nature of the lockout has come from statements that it might cost the city "millions of dollars" to unlock the system.

Millions? Really?

Unless Childs managed to install BIOS or kernel-level disk encryption on all the servers or stuff M80s in the server drive bays, there's no way that the cost of "unlocking" the network would run into the millions of dollars. Since officials are talking publicly about bringing in Cisco experts to undo the damage, it may be safe to assume that what Childs did was change the login to some or all of the routers and switches running the network. Now, being a veteran Cisco network architect, I can tell you that there's no way that a network of this size should have been built with only local passwords on the switches and routers. Using TACACS+ or RADIUS to control admin logins isn't just a good idea, it's the only way to handle authentication on a network of this size. Perhaps one of these methods was in use, but Childs modified the configurations to use only local logins. We can't know for sure, but we can speculate.

In any event, if we're talking about resetting the passwords on Cisco switches and routers that are physically accessible, then we're talking about a much, much smaller problem. It takes a few minutes to powercycle a Cisco router or switch, break the boot, change the configuration register (0x2142), reboot the switch and restore the configuration (don't forget to reset the confreg!). I could probably write an expect script to do just that in 20 minutes. Multiply the time required to either do it manually or run a script by the number of switches and routers affected, and you have an estimate of the hours required to undo the damage. My guess would be that with a few knowledgeable folks with laptops and a brief set of instructions, the whole network could be "fixed" in a matter of a day or so. There would be downtime, of course, but it shouldn't be terribly significant.

As to the claims that he might have installed a back door into the system, I can't speculate. There are too many unknowns, and apparently too many clueless people talking to the press to get any real idea of what that might mean. And the claims of him installing a "tracing system to monitor communications related to his personnel case", well, that could be as simple as a keylogger or a SNORT box in the right place, or as complex as custom code running on a server somewhere. My guess is the former.

Granted, there are a lot of assumptions leading me to the conclusion that city officials are playing Chicken Little, but based on everything I've read so far, they're certainly making a mountain out of a molehill. Heck, I'd fly out there and fix the whole thing myself for only $500,000. What a bargain!

From CIO: 8 Free Online Courses to Grow Your Tech Skills
Join the discussion
Be the first to comment on this article. Our Commenting Policies