For the most part, our blind trust in those devices is well placed. These situations are rare, and based on the sheer quantity of similar parts in our infrastructures, their reliability and stability is quite impressive. We work with operating systems of various flavors across our servers, hypervisors, switches, routers, and firewalls. We push bits around at a high level to force the lowest-level code to do our bidding. When you muck about with the configuration of a core switch, you're manipulating the levers that lead to the code running on those ASICs. If that baseline code is not tight and bug-free, problems occur and reality begins to warp.
That, right there, is the crux of the issue. We've all come across problems that appear to defy physics and threaten our very understanding of network construction and behavior. Problems that evade the pathways of our fundamental understanding of how things work are the worst of all. They might be as challenging as the one Kristian faced and require days or weeks of sleuthing to uncover, or they may be as simple and maddening as introduced licensing restrictions that go unnoticed, causing a network device to behave in bizarre, yet purposefully crafted ways. (If you ever run into a problem where you have intermittent host connectivity through a firewall that follows no discernable pattern, yet seemingly resolves after hours when you have time to work on it, do yourself a favor and check if the firewall has internal host count restrictions.)
When all is said and done, it pays to remember if you're facing a problem that defies all known laws of networking, you might be looking at a problem that lies in a place you can't access or repair. All you can do is work to identify the culprit and hold the manufacturer's feet to the fire until you receive a fix.
Sherlock Holmes said it best: "When you have eliminated the impossible, whatever remains, however improbable, must be the truth."
This story, "'Packets of death' reveal road to enlightenment," was originally published at InfoWorld.com. Read more of Paul Venezia's The Deep End blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.