Lesson: Protect your PKI CA servers. Offline CAs should be just that: offline! They should not be disabled or sitting on the network with their network cards disabled, but off the network, stored in a safe, and not so easy to compromise. CA private keys should be protected by a Hardware Storage Module appliance, and all related passwords should be very long (15 characters or more) and complex. Plus, it can't hurt to look for and monitor if other unauthorized CAs get added as trusted CAs.
APT war story No. 5: Don't forget the accounts you're not supposed to touch
As mentioned above, most APT recovery events involve resetting passwords. If you're going to reset passwords, reset all accounts -- though it's easier said than done. All my customers start out doe-eyed, ready to reset all passwords, but when they discover how much it will disrupt the business, they quickly scale back their goals. It's far easier to get fired for causing a significant business interruption than it is for not getting all the hackers out.
This particular customer was ready and incredibly thorough. The plan was not only to reset all user and service accounts, but computer accounts as well. Almost no companies do this, especially when it comes to resetting service and computer accounts. Heck, I'm giddy if they reset all elevated user accounts, because it's hard to get that little bit done thoroughly. Laugh only if you haven't been through this drill.
Password reset day came and went. There were significant service disruptions, some of which were painful enough that we had to tell the CEO. By the end of the week, however, we had reset all the passwords.
Within a few days, the APT owned everything again, picking up all email, controlling all the elevated accounts, including IT security accounts. It was like the password reset never happened. We were perplexed. As best we knew, we had removed the easy holes, educated employees, and couldn't see any evidence of Trojan backdoors.
Alas, there's a built-in Windows account called krbtgt that is used for Kerberos authentication. You shouldn't touch it, remove it, or as far as we previously knew, change its password. It really shouldn't be a user account that shows up in user account management tools, and this APT team knew it.
As I've learned on successive engagements, krbtgt is a go-to technique. After an APT crew compromises an environment, they add the krbtgt account to other elevated groups. Because customers usually leave it alone, even during a password reset, it can be exploited as a go-to backdoor account. Great idea -- if you're a malicious hacker.
My customer reset the passwords of its krbtgt accounts and everything else (again). As far as I know, it has not had another detected problem. Be aware that resetting krbtgt accounts will absolutely cause authentication problems. It's a pain. But if you have to do this, you too will get through it.
Lesson: if you're going to reset all accounts, make sure you know what "all" means.
APT war story No. 6: Information overload is spurring APT innovation, too
My last story isn't about a single client, and it shows the evolution of APT over the years. Early APT practitioners would immediately collect everything they could as soon as they broke in. They would siphon out all old emails and install bots to get every new email sent. Many times they would install Trojans to monitor the network and databases, and if new content was created, they would copy it.
In other words, many companies have online backup services they aren't paying for.
Those were the old days. In the world where terabyte databases are no longer even close to surprising, APT has a problem. When they get complete access to a network and learn where all the information is stored, they have to be more selective. Whereas they used to grab everything, what we see now are very discrete selections. The more advanced APTs these days build their own search engines, sometimes with their own APIs or borrowing the APIs of other well-known search engines, to search for specific data. They may still only leave with gigabytes of data a day, but what they have is highly selective.
Lesson: APT has the same issues finding and managing data just like you do. Don't let them index your data better than you do.
- 11 sure signs you've been hacked
- 7 sneak attacks used by today's most devious hackers
- True tales of (mostly) white-hat hacking
- 14 dirty IT tricks, security pros edition
- IT's 9 biggest security threats
- 9 popular IT security practices that just don't work
- 10 crazy IT security tricks that actually work
- Malware Deep Dive Report
- Data Loss Prevention Deep Dive Report
- Insider Threat Deep Dive Report
- Malware IQ test: Round 1
- Malware IQ test: Round 2
- Malware IQ test: Round 3
This story, "6 lessons learned about the scariest security threats," was originally published at InfoWorld.com. Follow the latest developments in security at InfoWorld.com. For the latest developments in business technology news, follow InfoWorld.com on Twitter.