I once worked in the IT department of a tech manufacturing company where we ran into an interesting problem: A large portion of the office network would go offline for several minutes to an hour or more at random intervals on random days.
The network administrators ran diagnostic after diagnostic, probed logs, and interrogated routers, yet couldn't pinpoint the problem. However, most of the time the network had started working again before they could find the issue.
[ Read InfoWorld Paul Venezia's take on ridiculous IT job postings: "Wanted: Brilliant technology slave." | Want to find out when the latest tale of IT terror hits? Get a new tech tale in your inbox every week in InfoWorld's Off the Record newsletter. | Follow InfoWorld's Off the Record on Twitter. ]
After weeks of this random, intermittent behavior, the network eventually went down and stayed down. Finally, the network admins had a repeatable problem to locate.
They discovered that there were two nodes on the network with the same IP address. One of these was the default gateway for an on-site office networks. The other was a mystery.
They traced the port to the cubicle of an engineer who had two PCs at his desk. One PC was his regular office PC; the other was a PC set up for development work.
This second PC was connected to a small, unmanaged switch. These switches were fairly common, since they allowed the engineers to develop machine control code at their desks, rather than inside the clean rooms were they had to wear bulky clothing. Also connected to the switch was a petite industrial camera used as part of a machine vision system that inspected the products with an Ethernet port at the back.
The unmanaged switch had originally been set up as a local network between the camera and the development PC, but someone had later uplinked the switch to the office network.