The partition had been 99% full for months. I'd deleted, moved, tucked, and shuffled, but there it remains... 2GB free on a 224GB partition.
After over two years of faithful service, the Promise SuperTrak6000 and three Maxtor 120GB disks in my main fileserver had to go. It had been a peaceful, if aloof relationship. I didn't open the server, and the array didn't break. The server was running RedHat Linux 7.3, and had been built just following the release of 7.3. It runs a multitude of applications and MySQL databases, mostly driving media functions. It's also the main NFS/SMB fileserver, PXE installation host, and functions as a backup DNS server for the network. Subsequently, the cruft of the ages abounded. This server holds all A/V content for my Xbox-and-TiVo driven media centers. All the music, digital pictures, and videos of an active geek lifestyle. Also thrown into the mix was 100GB of WAV and MP3 recordings of various bands I'd played with, and both mixed and raw studio tracks. It was simply so valuable, yet so restricting, I had no choice. I had to act swiftly.
This server was pieced together awhile back, and had been the pinnacle of reliability since. I decided to keep everything but the disk subsystem. I also decided to truly invest in this array and tapped 3Ware's 9500S-8 as the SATA RAID controller, and Maxtor's 300GB SATA disks. I tied these together with 3Ware's RDC-400-SATA hot-swap SATA internal inclosure. I also picked up two Seagate 120GB SATA drives to function as new system disks. The total cost was just about $1,500, or the list price of a 146GB U320 SCSI drive from the vendor.
I was expecting somewhat of a fight since the plan was to do a fresh installation of Fedora Core 3 on the new 120GB RAID1 system drives, create the ~850GB RAID5 partition, then rsync the data back to the new array from a 250GB FireWire drive holding zone. The plan was solid, but I had some problems with normal SATA cables and the RDC-400-SATA enclosure. In fact, they were somewhat distressing, as drives would magically disconnect from the controller without any warning. Reseating the cable on the enclosure would then bring it back. The RDC-400-SATA is essentially an internal disk cage with shared power and distinct SATA connectors for each disk. Why this isn't available using SATA Multi-lane, I don't know, but maybe it will be soon. The relative fragility of the straight SATA cables was high, and it actually appeared that the fans in the case moved enough air to cause intermittent cable issues. I picked up four right-angle to straight SATA cables, and haven't seen a recurrence of this issue.
I threw everything into the case at once. Six new drives, a new array controller, and a new CD-ROM drive. I hit power, accessed the 3Ware BIOS-level configuration utility, built the two arrays, and rebooted into the FC3 installer. 3Ware has had good Linux driver support for years, and this was no expection. The 3ware-9xxx module loaded without issue. Installing over the LAN brought me to a login prompt in about 12 minutes. Quite slick, all around. The only downside was that the mainboard didn't have a 66Mhz slot, so I can't really get all the performance from the card.
The 128MB SODIMM cache on the controller helps, as does increasing the read-ahead cache with
blockdev --setra 16384 /dev/sdx. Currently, I'm seeing 55MB/s reads and writes from the RAID1 mirrorset, and 70MB/s reads and 62MB/s writes on the four-spindle 850GB RAID5 array. It's not staggeringly fast, but the mainboard is rather old, it's on a 33Mhz bus, and the AMD XP2000 CPU isn't exactly an Opteron.
I spent about an hour rebuilding the application and database structure, borrowing heavily from backed-up configs. This was actually much simpler than I'd feared. There's something beautiful about simply copying a PHP application from a 2.5 year old Linux install running Apache 1.3.58 to a brand-new Fedora Core 3 install running Apache 2.0.52 and everything instantly Just Works. The MySQL databases moved with equal simplicity.
An rsync of the A/V data took about 2 hours, and I now have 530GB free. It seems as though it's time to take some more pictures.