OS X Security: How I became a spam kingpin, went legit and turned detective

I'm picking up the first-person account of a Leopard Server root exploit where it left off from my preceding post.

I did slam the door on the ClamAV exploiter, and close observation for a couple of weeks allayed my concerns that any lasting hole had been blown in my OS X Leopard Server's security. I felt quite pleased with myself. My mail server was back on-line and healthy, and days of backed-up e-mails, including the requisite quantity of spam, started streaming in. I was a happy camper in April.

Then came May, which has been an unkind month thus far. Walking past my Xserve one night, I noticed that the CPU activity lights were pegged when the server should have been idle. Eight cores, burning rubber doing nothing? I went to the Activity Monitor GUI, and then to top from the command line. Neither identified a process responsible for sucking up all of my Xserve's CPU cores. I rebooted, and the problem seemed to go away. But after about ten minutes, CPU utilization began climbing. I disabled Postfix and rebooted again with the same pattern.

It was at this point that I knew I was under attack. It's at this point that a sensible person like you would pull his WAN cable. But I, in addition to using Xserve as my 24/7 server, use it to unravel mysteries that might make for interesting copy, even if it means feeding my limbs to wolves in the process. I care that much.

Looking at Activity Monitor, top and ps again, I noticed that there were five sshd (secure shell daemon) processes running whose CPU times (total time a process spends occupying a CPU) nearly kept pace with the system's uptime. I often keep multiple ssh sessions going at once from a couple of machines, so except for the CPU time, it was hard to see five sshd instances as unusual. Until, that is, I used lsof to find out which files each of the sshd processes had open. I found socket connections open to Russia, China, Poland, Sweden, and Italy. My Xserve seemed to be on a promiscuous world tour.

I blew away the sshd processes, including my own (coming in through Remote Desktop instead), and I used Server Admin to disable ssh. The CPU VU meters dropped to minimal. Well, that was easy. "Too easy," I said in my noir gumshoe manner.

Every so often, ps would show launchproxy, sshd -i or both in the process list, but they'd vanish before I could lsof them. This was a job for Little Snitch. Little Snitch traces outbound socket connections by application, source IP and destination IP. Its killer feature is that it keeps a list of the last several connections. Watching Little Snitch, I could see launchproxy, then sshd pop up from connections from mostly offshore domains. The sockets would close within a couple of seconds of opening. However, Little Snitch couldn't tell me what these programs were doing. Was spam still getting out?