San Francisco hunts for mystery device on city network

With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network.

The device, referred to as a "terminal server" in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services (DTIS) isn't even certain where the device is located, court filings state.

[ Follow the Terry Childs saga blow by blow in InfoWorld's special report | Read the actual court documents ]

The router was discovered on Aug. 28. When investigators attempted to log in to the device, they were greeted with what appears to be a router login prompt and a warning message saying "This system is the personal property of Terry S. Childs," according to a screenshot of the prompt filed by the prosecution.

The disclosure is the latest turn in a bizarre story that has made headlines in San Francisco for the past two months. Childs, a network administrator with DTIS, was arrested July 12 on charges of network tampering after he refused to provide his superiors with administrative access to the city of San Francisco's network, which he had managed for the past five years.

Initially Childs refused to hand over administrative passwords to the city's routers, which had been configured to wipe out all configuration information if they were reset.

After a dramatic jailhouse meeting with San Francisco's mayor one week after his arrest, Childs handed over the data, but DTIS Chief Administrative Officer Ron Vinson said Wednesday that the city now expects to spend more than $1 million to clean up the mess. To date, DTIS has paid out $182,000 to Cisco contractors and $15,000 in overtime costs, he said.

The city has also set aside a further $800,000 to address the problem. Vinson did not specify what the additional money was expected to cover, but if the city has to hire network consultants to remap, reconfigure, and lock down its network, this would not be an unreasonable estimate. The city has also retained a security consulting firm called Secure DNA to conduct a vulnerability assessment of its network.

Meanwhile, Childs remains in county jail, held on a $5 million bond. His supporters say he is a dedicated city employee who was pushed too far by incompetent management, while the county's district attorney argues that he concealed a violent criminal past when hired by the city and remains a threat to the city's network. Childs served prison time following a 1983 robbery conviction, a fact he concealed in his city job application forms.

In court filings, prosecutors say Childs has not provided passwords to city-owned encrypted hard drives or access to two Corsair Flash Survivor USB drives that may contain sensitive information.

In a report filed before the city disclosed the hidden router, a court-appointed expert witness for the defense wrote that DTIS could easily prevent Childs from accessing the networks. "I have seen no evidence that Mr. Childs is a 'computer hacker,' and by taking a number of simple steps, DTIS could block access by Mr. Childs to San Francisco networks," wrote Doug Tygar, a University of California, Berkeley computer science professor.

Childs' next court appearance is set for Sept. 24. If convicted, he faces up to seven years in prison.

Correction: This story as originally posted misreported the date of Terry Childs' arrest. The article has been amended.