Over the next week or so I'm going to be discussing what's in my test lab, both the good and bad, and what I use on a daily basis vs what gets shelved. In any working lab -- especially one testing all sorts of new gear -- there's a foundation. That foundation includes racks, UPSes, KVMs, switches, servers, and basically everything that a standard datacenter would have, but all geared to be modified and changed constantly to facilitate hardware and software testing.
In essence, this means that the foundation lab gear gets much more abuse than standard datacenter gear -- for instance, the racks are constantly changing, with new gear getting racked and older hardware re-racked, switch configurations change all the time, and servers are built with multiple operating systems more times than I can count. In short, over and above the hardware and software that gets reviewed, there's some really rough real-world testing occurring constantly, and that part of things rarely gets any press. So, I'm going to start that right now, starting with servers, storage, and operating systems.
There's an old lady of the lab. Actually, there's two, but the other one is a switch that I'll get to in a later post. The elderly yet rock-solid HP ML370 sitting in the corner of the lab has proven to be a wise investment. It's a G2 model, two 2.8Ghz Intel XEON CPUs, a few gigabit NICs and 4GB of RAM. There's no RAID controller, just a mix of 18GB, 36GB, 72GB, and 147GB drives in the six U320 drive bays. It began life running Red Hat AS 2.1, and was then upgraded to RHEL3 quite some time ago. It hasn't been upgraded since due largely to lack of time and need. Among other things, it was running GSX Server for over a year, just recently upgraded to VMware Server 1.0. There are only a few VMs running on it, but they include the lab's main domain controller, and currently a VirtualCenter 2.0 server. Otherwise, this server is used to generate loads of varying types to hardware under test, from Web loads to SMB torture testing, network throughput testing, and on, and on. I'm probably jinxing it now, but in the several years it's been running, it hasn't hiccuped once, nor lost a disk. Also, it's never been backed up.
Hmmm. I'll be right back.
Okay. I feel slightly better now that it's backing up to nearline storage.
The other lab stalwarts aren't quite as aged as the ML370, but are put under heavy load constantly and have again proven to be 100% reliable no matter what the conditions. These servers include a Newisys N4300 with four dual-core Operons, an HP DL585 G1, also with four dual-core Opterons, and a SunFire T2000 running the Sun SPARC T1 eight-core processor. Each of these servers has proven their worth over and over throughout the time they've been chained to the proverbial lab plow. The DL585 in particular has been a champ, reassuring me that the Server class Technology of the Year award I gave it last year was well earned. That server has run nearly every modern OS and functioned as a virtualization platform under VMware and Virtuozzo, and served very well under extremely heavy load and focused attempts to break either it, or the software running on it. There are other servers in the lab, of course, but these three stand out as true workhorses. If I'd written this post yesterday, I would have included my Apple XServe in this list, but after spending a month without being run, it suddenly refuses to power up.
And, oddly, a workstation is in the mix. The lab's DNS, DHCP, and PXE infrastructure is all centered around a six-year-old HP Karak XU800 with a single PIII 866Mhz CPU, a pair of software mirrored 40GB PATA drives and 512MB of RAMBUS RAM running Fedora Core 3. It's simply a case of ain't broke, don't fix it. By serving as the PXE TFTP host, that box has launched a thousand OS builds, if not many more.
The lab is built mainly on three OSes. Four, if you count VMware. In no particular order, they are Linux, FreeBSD and Mac OS X. The Macs are the workstations in the form of several laptops and a few G4 PowerMacs, and Linux and FreeBSD power just about everything else, from storage, mail, DNS, DHCP, testing platforms, and so forth. Windows boxes are built as needed for testing, but none remain as permanent fixtures other than two that run under VMware, providing AD services for a few other systems. There are disk sets with Windows x64 builds for the Opteron servers, but they're only called into play when the test calls for it. Otherwise, it's Linux or FreeBSD.
The main point of storage in the lab at the moment is a SNAPServer 18000 with around 1.5TB of SATA storage. It's not the fastest storage device out there, nor the most elegant, but it works, and it works well. Accompanying it in the lab right now are a NetApp StoreVault with around 3.5TB of SATA storage, and a EqualLogic PS3800XV with a few TB in SAS drives. Ever since they posted the best numbers in the six-way iSCSI test I ran last year, EqualLogic has impressed me, and this unit seems to share the high points of the PS200E I tested with the added bonus of more space and faster disks. The full review of this unit will be in an upcoming InfoWorld issue.
Also driving some disk in the lab is a custom-built server running a 3Ware 9500S-8 SATA controller with around one TB of storage. That's running Fedora Core 3 and currently holds all the OS installer roots, updates, various testing tools, and more ISO images than you can shake a stick at.
So there you have a little insight into the inner workings of my test lab. Still to come are my notes on racks, UPSes, KVMs, switching, routing, firewalls, and more. Stay tuned.