For the past few months, I've been running a Sun Ultra 40 M2 coupled with a 3Ware 9650SE SATA RAID controller. It would seem that this is a marriage made in heaven.
As I've remarked before, the Ultra 40 M2 is simply the most powerful workstation available from a mainstream vendor today. Armed with two AMD Opteron 2218 dual-core CPUs, up to 16GB RAM, eight hot-swap SAS or SATA drive bays, two PCI-X slots, built-in 7.1 sound, S/PDIF optical input and output, a dual-layer DVD burner, and (in my case) an nVidia Quadro 5500 graphics card, this system is the creme de la creme of the workstation world. The only downside is the relatively anemic nVidia SATA RAID controller built into the mainboard. The performance of this controller isn't terrible, but the Linux driver support simply isn't there. Enter the 3Ware 9650SE.
The 3Ware 9650SE-8LPML I have running in this system is a full-on 8-port SATA RAID controller with 256MB RAM and an optional battery-backup unit. There are two four-port SATA multilane connectors on the card, which can be used to marry the 9650SE to a multilane-capable disk array, or to individual SATA drives with multi-lane to discrete cabling. In the case of the Ultra 40 M2, however, multilane to SAS cabling is needed. Fortunately, the built-in nVidia controller uses multilane connectors to feed the disk backplanes within the Ultra 40 chassis, but the included cables aren't long enough to reach the 9650SE. Sun can supply cables of appropriate length to reach the card, however.
Once I had the right cables, it was simply a matter of cable routing to each backplane connector and then back into the 9650SE. The fan tray that sits to the left of the disk bays can get in the way here, but some creative cable management within the case made everything fit and look like it was meant to be there. I placed eight 250GB SATA drives in the disk cages, and powered the system on. The 9650SE posted, found all the drives, and all was well.
I configured the eight drives into a RAID50 set, giving me high throughput on 1.36TB of usable space while providing significant fault-tolerance. The configuration through the 3Ware BIOS tools is quick and easy. Unfortunately, installing and running Fedora Core 6 (or any reasonably recent distro) on the 3Ware 9650SE isn't as simple. The 9650SE and the more recent cards from 3Ware aren't supported in the included 3w-9xxx driver found in stock 2.6 kernels. Historically, 3Ware has been extremely good at providing support for Linux and FreeBSD, so I would think that this problem will be rectified shortly, but in the interim, there are a few steps involved in getting everything working right on Fedora and RedHat. The first is to download the right install disk from 3Ware. You can find the files for just about every major distro on their site. These are just zipfiles with driver sets. Format a floppy with mformat (
mformat a:), and unzip the installdisk file to the floppy. Then, boot the system as you would for a normal installation. At the boot: line, enter
linux dd and the installer will prompt for a boot disk. Select the floppy drive, and it should load the appropriate driver. Continue the installation normally. On the Ultra 40 M2, I had to use a USB floppy drive, which appears as /dev/sda.
Following the initial boot, the system needs to be updated. Be aware that updating the kernel may result in a non-bootable system since the new kernel will not have the right driver for the 3650SE. Fortunately, it's easy to remedy this problem. Run the yum update to pull in all the new packages, including a new kernel and kernel-devel package. Then, download the upstream driver for the 2.6.19+ kernels from 3Ware's download site. Extract the driver source into a new directory, such as /usr/local/src/3ware, (
mkdir -p /usr/local/src/3w-9xxx; cd /usr/local/src/3w-9xxx; tar zxf /path/to/source.tgz; tar zxf ./3w-9xxx.tgz) move into the driver directory, and edit the Makefile to pull in the right kernel path. In my case, the SRC:= line at the top of the Makefile should be modified to
SRC := /lib/modules/2.6.20-1.2948.fc6/source/. This will tell the compiler to build the driver with the source tree of the new kernel, not the running kernel. Then, simply run
/lib/modules/2.6.20-1.2948.fc6/updates and you should be all set.
Once this was done, I rsync'd 190GB to the fresh install (yes, my home directory is 190GB), and saw write throughput to the RAID50 set at around 100MB/s. Reads were slightly higher than that at 110MB/s. I've been beating up the 9650SE and the Ultra 40 M2 with my normal brand of workstation torture -- cyclic MD5 sums on multi-gigabit files, kernel recompilations, DVD ripping, MP3 encoding, and two virtual systems running under VMware Workstation 6, all while playing movies from NFS shares and running Beryl with all the widgets enabled. Between the stellar performance of the 9650SE and the calm and steady power of the Ultra 40 M2, all of these tasks were handled with aplomb. Suffice it to say, you'd be hard-pressed to equal or surpass the performance of this box with any computing hardware available today.
As far as longevity and survivability goes, the 9650SE has been running for a few months without a problem, and my several-year-old 9500 8-port SATA RAID controller has been driving a four-disk RAID5 set without a hiccup. If history is any indicator, reliability isn't an issue with 3Ware cards. I'll be posting more on this power duo as time and events warrant, but for right now, I'm a very happy guy.