How the Linux kernel gets built

The Linux kernel development process is akin to a game of leapfrog. Even-numbered kernels (v2.0, v2.2, v2.4) are stable kernels, and odd-numbered kernels are unstable, or development, kernels. As soon as a kernel is released as an even number, the next odd numbered kernel is born, and historically, the previous even-numbered kernel is placed into maintenance mode.

Thus, with the advent of v2.6, Linux kernel v.2.4 will be maintained with security patches and bug fixes, but official releases of the v2.4 kernel will not contain new features; major updates will be handled in the v2.6 branch. Also following the official release of v2.6, the v2.7 kernel will soon be born and become the rough framework for the as-yet-unspecified v2.8 kernel.

During the development process, the LKML (Linux Kernel Mailing List) is the main sounding board for the kernel developers, and in fact, anyone developing core Linux code. The list is not restricted, so any Linux user may post questions or commentary regarding any Linux kernel. Because any question for the list may have been addressed previously, the LKML archives are a great resource to those working with the internals of the Linux kernel or trying to debug a problem.

During the course of my kernel testing, I ran into a few interesting anomalies with the v2.6-test kernel series and posted my findings and thoughts to the list. The first post detailed an issue regarding poor per-character performance of the v2.6.0-test8 kernel. Less than 30 minutes later, Linus Torvalds had responded to my post and offered some suggestions, followed by Bill Rugolsky Jr. and Andrew Morton adding insight. Within six hours, the problem had been reliably reproduced by a number of core Linux kernel developers, and traced to an issue with a library linked by the NPTL (Native POSIX Thread Library) code. The problem was fixed by Ulrich Drepper of Red Hat and incorporated into the next release.

A few days later, I noticed another performance issue. The Samba tests under the v2.6 kernel were ridiculously slow. The numbers I was seeing on the Intel Xeon system were truly terrible. I had previously tested the lower-level kernel subsystems on this server and seen no problems with performance, but the higher-level tests showed a significant bottleneck somewhere. Another post to the LKML, followed by responses from Torvalds and Morton, helped me identify the source of the problem.

I had originally configured my array controller, LSI Logic’s Fusion-MPT, without RAID so that each disk was viewed as a stand-alone SCSI drive. I did the I/O subsystem tests with this configuration and noted very high performance results. During a rebuild of the server, however, I reconfigured the disks into a RAID1 array. I then performed the Samba tests, which produced abysmal performance numbers. Naturally, that didn’t make sense.

The problem was with the Fusion-MPT drivers. With a RAID array built on the controller, the MPT drivers simply don’t function properly, rendering the overall system very sluggish. Reconfiguring the controller to use stand-alone drives fixed the problem, and the performance was stellar. This issue was relayed to the developers responsible for the Fusion-MPT drivers, but a fix has yet to be incorporated into the official kernel, and v2.6 was released with this bug intact.