Bitten by the Red Hat Perl bug

Is it realistic to expect stem-to-stern application platform support from any one vendor?

As my colleague Martin Heller recently observed, smart coders always optimize the slowest thing. Trying to optimize every trivial performance issue in your code is just chasing your own tail. You should find the one problem that's causing the biggest performance hit and fix that first.

But what if you go over and over your code and you can't find what's wrong? In fact, what if it turns out that there isn't anything wrong with your code? What if "the slowest thing" is actually the code supplied by your vendor?

That's exactly the situation that Vipul Ved Prakash discovered when he started tinkering with one of his company's Linux boxes. For some reason, Perl code on the server seemed to be running at least 100 times slower than expected. To his surprise, Prakash found that the parts of his application that were eating up the most CPU involved "bless" and "overload" -- core Perl language functions that are used in the process of instantiating new objects.

Hold your flames, please. This isn't a story about how object-oriented Perl is slow and inefficient, because it isn't -- at least, not normally. Prakash had never encountered a problem like this before. Identical code ran on his MacBook without a hitch.

So what was the problem? Simply put, the low-performing instance of the code was running on a CentOS Linux server, using Perl packages built from code maintained by Red Hat.

This Perl is a lemon

Prakash has posted a detailed description of the problem on his blog. To make a long story short, he got rid of the Perl executable that came with his CentOS installation, compiled a new one from stock source code, and the bug disappeared. Clearly, the Perl hackers are blameless in this case. The fault lies squarely with Red Hat for distributing a buggy version of the interpreter.

What's more disturbing, however, is that it turns out that this Red Hat Perl performance issue is a known bug. It was documented and verified long before Prakash ever raised a stink about it. How long? Try 2006, according to Red Hat's own Bugzilla database.

The current bug has been open since late last year. That's bad enough, but similar slowdowns were reported in 2007 and 2006, in versions of Red Hat's code base dating back to Fedora Core 7. Nonetheless, despite being characterized as "medium severity," the priority of fixing the current bug is listed as "low."

That's unfortunate, because the impact of this bug could be far-reaching. Red Hat Enterprise Linux customers are undoubtedly susceptible, as are users of Fedora (Red Hat's community-maintained development branch) and CentOS (a free Linux distribution built from Red Hat's source code).

Fish or cut bait?

The immediate lesson to be learned from this incident is the importance of profiling your code. Prakash discovered the bug in the CentOS Perl interpreter using the Devel::NYTProf source code profiler. Had he not done so, he might have been hard at work chasing his own tail even now, fruitlessly trying to optimize away the bottlenecks in his own code -- as doubtless many other CentOS and Red Hat customers are still doing.

If your organization is affected by this bug, you have two options. First, you can always sit tight. The bug will have no significant impact on many Perl scripts, especially those that don't instantiate a lot of objects (which might be most of them). And the more people hear about the problem, the more likely it is that Red Hat will expedite a fix.

If that won't work, you could compile the Perl interpreter from source, as Prakash did. The process is straightforward and should be painless for anyone with basic Unix experience.

Doing so raises some issues, however. First, if you substitute homegrown binaries for the ones that shipped with your Linux distribution, you effectively waive any support contract for that particular tool. That hardly makes sense if you rely upon the tool for a mission-critical system or application.

Furthermore, who's to say that the version you build won't have some hidden bug or performance problem of its own? If Red Hat can screw up, so can you. It's a brave IT manager who would be willing to take responsibility for a custom built development tool on an enterprise-wide basis.

A better LAMP stack

Perhaps the best solution would be to find a software distributor that is willing to provide enterprise-class support for the open source application development platforms that you value the most. At the end of the day, maybe it's unrealistic to expect a single company -- a Linux vendor -- to provide comprehensive maintenance and support for the entire universe of open source software projects.

A number of companies have offered support contacts specifically for Perl. Most recently, Sun unveiled an enterprise-ready version of the LAMP stack at OSCON. In the past, however, such offerings have met with lukewarm response. Having chosen open source software on the basis of cost savings, the majority of customers choose to go it alone, rather than shell out for additional support contracts for software they can otherwise download for free.

Vipul Ved Prakash's case is proof enough that this attitude must be curbed. For companies that rely on Perl for mission-critical applications, a robust, reliable Perl interpreter is every bit as important as a well-tuned Linux kernel. Unfortunately, realizing this is one thing; committing funds to it is another.