This week, I was called into an office to look at an odd situation with a Hyper-V server. The server has been up and running for years, with Windows Server 2008 as the Hyper-V parent and three VM servers running Active Directory, SharePoint, and various other server applications. For some reason, none of the VMs was running. Upon investigation, it was easy to see why: Hyper-V was no longer enabled as a server role. Somehow it had uninstalled that role and wouldn't reinstall itself.
That was a new issue for me. I've seen Hyper-V hiccup before, but never uninstall itself. I put that concern aside for the moment and pulled the three VMs over to another system running Hyper-V. They all mounted without issue. Phew! Good news.
[ Don't look now, but your antivirus may be killing your virtualization infrastructure. InfoWorld's Matt Prigge shows you how to detect the warning signs. | Stay atop key Microsoft technologies in our Technology: Microsoft newsletter. ]
I thought, "We fix the Hyper-V server, we restore these VMs, and we're good to go." Not so fast -- the SharePoint server wasn't serving sites, as it had faithfully done for five years. Upon further investigation, I found that the SQL databases were still in place -- but not SharePoint. It was as if it had been uninstalled from the system. Only three people have access to these servers, and none of them uninstalled SharePoint or disabled Hyper-V.
The mystery remains as I write this, and we're poring through event logs to see if we can get to the bottom of it.
An instance of the fabled escape attack?
Someone suggested the possibility of a hack going through SharePoint (which is Internet-facing) and into the Hyper-V parent. That particular hack is known as an escape attack. A few years ago, it was just a theoretical possibility, but recent news reports indicate that these vulnerabilities do in fact exist and can be exploited. My colleague David Marshall recently relayed a security flaw in which someone can cause a system exception in virtualized code and escape from the guest OS into the host environment with elevated privileges.
Although many people are skeptical about the existence of attacks such as the escape attack, Gartner Fellow Neil MacDonald says, "It's just a matter of time before a widespread publicly disclosed enterprise breach is tied back to a hypervisor vulnerability." He recommends you take a look at the National Institute of Standards and Technology's Guide to Security for Full Virtualization Technologies.
I'm left wondering if the Hyper-V server in my client's case was hit by an attack. It feels purposeful, not like a simple virus. If it was an attack, it's absolutely the result of a cavalier attitude on the administrator's part. Every rule in the book was ignored here. You always think, "What are the odds?" until you become part of the statistics. Thankfully, there are valid backups of the VMs, so only a little data will be lost.