In this instance, I spent a few days helping the client deploy some honeypots. Most customers ask me how we are going attract hackers to the honeypots. I always laugh and answer the same way: "Don't worry, they will come." Indeed, every honeypot I've ever set up has detected nefarious activity within a day or two. These new honeypots were no different.
We detected network logon attempts coming from multiple workstations, none of which had a legitimate reason to be logging on. We pulled a few of these workstations and forensically examined their hard drives. We found that the APT had placed a remote-access Trojan on each of them. The Trojan's name? The same as the anti-APT detection software. The bad guys had someone replace the legitimate anti-APT software with a Trojan, and it turns out they did it on nearly every computer.
This explained a few things, like why no APT had been detected. But the bigger question was how did it get installed in the first place. It turned out the customer's "gold build" had been compromised in its build environment, and this Trojan was part of the build.
Lessons: First, verify the integrity of your builds; prevent unauthorized modification or invent some way to detect it. Second, honeypots are a great way to detect malicious activity. Third, always look for and investigate strange network connections from unexpected places.
APT war story No. 4: All your PKI base belong to us
APT attacks on Public Key Infrastructure (PKI) servers used to be somewhat rare. In fact, until two years ago, I never personally ran across an APT where PKI servers had been involved. Now, it's fairly common. But the most relevant story is the one where the PKI turned into physical access in a sensitive area.
This particular customer used its internal PKI servers to create employee smartcards. These smartcards were used to not only log on to computers but to physically access company buildings and other infrastructure.
The customer's root CA (certification authority) server was a virtual instance sitting, disabled, on a VMware host server. The bad guys had found it, copied it offsite, cracked the local (weak) administrator password, and generated their own trusted subordinate CA. They used this CA to issue themselves PKI access to everything they could.
What surprised me most was the video my client showed me of two unknown men posing as employees. Using the fake smartcards they created, they had parked their cars inside the secured company parking lot, walked into the building through the employee entrance, and onto a floor that stored highly sensitive data.
My customer couldn't tell me what happened after that or what was taken, but I knew they were not happy. There was a very serious mood in the room. I was invited to help them create a new PKI and to migrate the company into the better-secured PKI environment.
Lesson: Protect your PKI CA servers. Offline CAs should be just that: offline! They should not be disabled or sitting on the network with their network cards disabled, but off the network, stored in a safe, and not so easy to compromise. CA private keys should be protected by a Hardware Storage Module appliance, and all related passwords should be very long (15 characters or more) and complex. Plus, it can't hurt to look for and monitor if other unauthorized CAs get added as trusted CAs.