On Sunday, I wrote a blog post titled "Distillation" in which I said:
"It's quite difficult to accurately convey the stress and effort required to build and maintain large complex networks to those with no real frame of reference. I've done it for years, building networks for city governments, universities, hospitals, and private companies. At some point, a network moves beyond "straightforward" complexity, and almost becomes a work of art. Whether it's a clever iBGP VPN failover for a large MPLS-based WAN, an OSPF-based ISDN dialback configuration, or a novel method of route injection through a third-party cloud, there are instances where network architects and admins need to color outside the lines to provide a needed service or measure of redundancy. It's at this point that the proverbial wheat is separated from the chaff in terms of network administration."
I've felt this way about several of the networks that I've built in the past -- they transcended the mundane and became basically a work of art. Terry Childs also felt this way, because he applied for and received a copyright in June 2007 on the configuration of the FiberWAN as technical artistry. This would back up my contention that Childs' felt that what he had created couldn't be understood or maintained by anyone else. After all, would Picasso let anyone else work on one of his paintings?
[ Follow the Terry Childs saga with InfoWorld special report: Terry Childs: Admin gone rogue. ]
More information coming to light shows just how in the dark his managers really are. In the arrest warrant, several key details are presented as evidence of malfeasance on the part of Childs. These include a detailed description of an analog modem and a DSL modem that were discovered in a network cabinet that he had built, and another analog modem attached to a desktop PC that he had installed. The description in the arrest warrant introduces these devices as evidence that Childs had added backdoors to the FiberWAN. Further on, the inspector describes an event in which Childs' pager was taken from him, and shortly thereafter, the pager went off with a message described as having come "from one of the routers on the network". This event was presented as evidence that Childs "still had administrative access to the network", and was probably a very important "fact" that helped convince the judge to sign the warrant.
The fact that this information is in the arrest warrant underscores the fact that the city truly doesn't understand anything about this case. According to Childs, one of the modems was in place to perform dialback services to provide him with emergency access to the network, and was installed following an outage event that was extended due to the lack of such access. Further, it was installed with the full knowledge of his managers. Also according to Childs, the other analog modem was hooked up to a desktop system running What's Up Gold, a network monitoring tool. This modem was used solely to send warning messages to Childs' pager when problems occurred on the network -- it's more than likely that this is the same modem that called Childs' pager after he had surrendered it to his management.
I find it deeply disturbing that both the inspector that prepared the arrest warrant and affidavit, and the "expert" brought in to help the city with this situation did not understand the actual purpose of these items, and yet are apparently still involved in the investigation of this case. I find Childs' description of these two modems and their purpose to be far more realistic than the description in the arrest warrant affidavit.
The DSL modem is slightly more curious. If it was connected to a raw pair (or a BANA circuit), where's the other end? If it was connected to an ISP, providing Internet access or a path through which to access the network, I find it hard to believe that nobody else knew about it. After all, unless Childs was paying for that circuit from his own pocket, the bills had to go somewhere, and ostensibly, somebody had to sign off on it. More information is needed on that one.
Unfortunately, it appears I might have an answer to some of my questions from my post on Sunday, namely "If the FiberWAN network is as complex as it appears to be, are there CCIE-level forensic networking experts employed or contracted by the San Francisco police department?" The answer would be that no, there aren't. The people tasked with investigating this case appear to be woefully ignorant, and lack basic understanding of how enterprise networks are constructed and maintained. This isn't necessarily a knock on the SFPD -- there's no realistic expectation that they should have this level of expertise on staff. They should, however, contract with skilled engineers that can provide that service for them.
