Later, I went back to Color Printer, crestfallen, staring at the same pop-up authorization dialog and unable to get in -- until I realized I was using a stale DNS cache and thus visiting whatever was on the old IP address. Finally, we had proof that something was indeed there.
Then it hit me: The meaningless string of letters and numbers was a product model number. A Google search showed it was a wireless print server -- of a model we've never purchased.
The mystery deepened. We could ping it and check its MAC address, but we couldn't tell where, physically, it was in the building. We spent a while looking behind doors and above ceiling tiles, scouring the warehouse, crawling under desks, and especially looking near printers, but we found no sign of the interloper.
Eventually I looked up the default username and password for that model and was able to log into the device's Web interface (more proof we hadn't set it up -- we never use default passwords), and by luck stumbled upon another clue: the name of the Wi-Fi network it connected to, which told me it was in our warehouse.
I printed out a picture of the device and walked around the warehouse asking, "Have you seen this?" as if it were a mug shot. (In fact, later, I made a Wanted Dead or Alive poster out of it.)
Finally, someone said, "It's right there," and pointed above me. There it was, mounted by zip-ties to the girders holding up a conveyor belt, hooked not to a printer but to a bunch of bar code scanners.
It turns out that three years ago the vendor putting in those scanners as part of a warehouse automation solution, for reasons that are still unclear, put a wireless print server into the system despite there being no printers involved. They chose an IP address, didn't bother to tell us, and installed it. It had been up there ever since, supporting a system we don't use -- the conveyer-mounted scanners proved too unreliable and we never bothered to remove them.
In hindsight, everything we'd first dismissed had been a key clue, and the postmortem resembled the parlor scene in a mystery novel as we retraced the events. For instance:
- The DHCP reservation didn't work because the IP was already there, but didn't have a DNS entry for us to see.
- Most print jobs worked because the Wi-Fi server was farther out in network round-trip times, allowing hard-wired Color Printer to get first dibs on traffic for that IP address.
- The print server we'd check showed all jobs accepted even if Color Printer never saw them because the device that happened to be using the IP address was ... another print server, one not attached to a printer to mysteriously spit out paper and get our attention.
- Color Printer lost more jobs when the network was quiet because at those times the warehouse Wi-Fi wasn't as clogged by handheld scanners, allowing the interloping print server a better chance at getting first dibs on the network traffic.
- There were no error messages about an IP conflict because both devices happened to be printers, not computers that would display a message in that situation.
Color Printer is now happy and healthy, as reliable as Bulk Printer, and the offending print server is in protective custody to make sure it doesn't become the victim of frustration-motivated revenge. There aren't words for the sense of satisfaction and triumph at cracking this mystery -- if only my boss understood it, so we could get the credit we deserve.
Do you have a tech story to share? Send it to firstname.lastname@example.org. If we publish it, you'll receive a $50 American Express gift cheque.
This story, "In plain sight: On the trail of a problematic printer" was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.