What the days of gonzo IT taught us

There's real value in figuring out how to fix things yourself instead of buying the latest off-the-shelf solution

Only in hindsight do I realize how crazy IT was during the last ten years of the 20th century -- and how lucky I am to have cut my IT chops back then. In most shops, the rules surrounding x86 computing were scant, in contrast to endless procedures protecting AS/400s and mainframes. So all kinds of inventive, homegrown solutions emerged to address the brand-new problems brought about by the Internet boom.

Think back to 1995: Windows 95 launched in August of that year, Windows NT 4 launched in 1996, Red Hat Linux 2.0 was released, FreeBSD 2.0.5 appeared, and many of the thousands of small dial-up ISPs in the United States ran on BSDi 2.0.1. Google didn't exist; everyone sharp used Alta Vista. Usenet was massive and the concept of firewall-plus-NAT was foreign to most organizations.

[ Paul Venezia knows networking. Check out his Networking Deep Dive Report. | Also see InfoWorld's 10 tips for boosting network performance. ]

Those were the days of gonzo IT. Back then we were writing the rules as we went. We were developing the foundations of what we now think of as IT using duct tape and bailing wire. It was a period of constant innovation at fundamental levels. So many problems and requirements had no official solution that we had to build those solutions from whole cloth.

That type of thinking seldom arises in modern IT -- the determination to fix a problem yourself, rather than just purchase a solution that (sort of) addresses the issue. Back then, when problems emerged or functionality beckoned, most often no commercial product would do much good, so we developed solutions using whatever tools were available. An awful lot of Perl was written during those years.

Of course, plenty of those homegrown fixes were slipshod, but that's not really the point. The fact that those solutions were developed internally gave admins and developers an opportunity to grow as they worked without a template or an instruction manual. They were improvising, not just reading the music, and that led to better IT as a whole.

These days, IT is less about developing solutions to problems and more about adapting problems to fit purchased solutions. We're coloring by numbers most of the time.

The fates must be watching me write. As I wrote the previous paragraph, Nagios sent me an email saying that the load on my primary FreeBSD mailserver had suddenly spiked to 10.53 and that there were over 1,000 processes. I SSH'd in and quickly determined that I was on the receiving end of a DDoS attack from a botnet, with thousands of connections coming in every second to TCP/25 with no data -- a classic flood attack. This was causing the box to spawn thousands of sendmail processes and everything was grinding to a halt.

I fixed the problem in a minute or two with this one-liner, broken into three lines for display purposes:

for i in `grep "did not issue MAIL/" /var/log/maillog | \

awk -F '[][]' '{print$4}' | sort | uniq`; do \

pfctl -t smtpblock -T add $i; done

This single line found all the IP addresses of the offending remote systems in the mail log and added them to a table called "smtpblock" in the FreeBSD pf packet filter. This table is referenced in a rule to block all access to TCP/25. The DDoS is still in progress, but the brunt of the attack is now handled at the kernel level by pf, not by sendmail spawning a new process for every incoming connection. The load on the box dropped from around 12 to 0.35 within a minute or so. With some tinkering, this one-liner could be consolidated even more, but I was in a rush.

If I hadn't grown up dealing with issues like these over the course of 15 years, I'm not sure I'd have known a fix like this was even possible, much less how to implement it. If this was a Microsoft Exchange server, a fix like this wouldn't even be realistically possible unless it was running on Windows Server 2008, and even then, working with netsh advfirewall is nowhere near as fluid. Left with no other choices, I'd have to look elsewhere -- say, a separate hardware firewall or a mail filtering appliance -- to deal with the problem. No matter what the ultimate solution might have been, it certainly wouldn't have been a quick fix.

Don't get me wrong. Many times it's the better part of valor to pony up for a pre-packaged solution. But in plenty of situations, what you really need is a firm foundation, the ability to conjure out-of-the-box solutions, and the experience to know how to implement them.

The days of gonzo IT may be behind us, but the lessons we learned then are more valuable than ever in the era of prepackaged IT.

This story, "What the days of gonzo IT taught us," was originally published at InfoWorld.com. Read more of Paul Venezia's The Deep End blog at InfoWorld.com.