Azure customer saves Microsoft from an RHEL disaster

Were it not for an alert customer, attackers could have compromised every RHEL instance on Microsoft Azure due to improper configuration of Red Hat Update Appliances

Microsoft Azure plugs huge security hole in its RHEL config

Broken authentication, improperly secured configuration files, and poor certificate management: Attackers could have exploited these issues to compromise any RHEL (Red Hat Enterprise Linux) instance on Microsoft Azure.

Ian Duffy, an Irish software engineer with the e-commerce company Zalando, discovered these flaws when creating a machine image of RHEL that was compliant with the Security Technical Implementation Guide defined by the Department of Defense. Microsoft has since fixed these problems, but they offer an object lesson in the hazards of poorly implemented cloud security.

The client configuration files in Azure's Red Hat Package Manager contain build host information that could be used to discover all of Azure's Red Hat Update Appliances, Duffy said. The Red Hat Update Appliance is part of the Red Hat Update Infrastructure, which lets cloud platforms like Microsoft Azure and Amazon Web Services run local yum repositories, instead of having individual RHEL instances connect to Red Hat servers every time they need to update a package or install a new application.

Both Azure and AWS manage a Red Hat Update Appliance for each region, and each RHEL instance connects to the region's appliance when running yum to install or update packages. Duffy found it was possible to discover all of Azure's Red Hat Update Appliances and gain administrator access in order to upload compromised packages to the servers. Attackers could gain control over all Azure RHEL instances that executed yum against the compromised appliance and received the tampered files.

"In theory ... one could have gained root access to all virtual machines consuming the repositories by releasing an updated version of a common package and waiting for virtual machines to execute yum update," said Duffy.

Exposed update servers

Duffy was trying to create a secure RHEL image that could be used on Azure and AWS when he uncovered the configuration files and the Red Hat Update Appliances. All the servers were exposing their REST APIs over HTTPS. An application called running on port 8080 ( is included in the PrepareRHUI package that runs on all Azure RHEL instances) let him look into the archives containing logs, more configuration files, and the SSL certificates granting full administrative access to the Red Hat Update Appliances.

Though the application required username and password authentication, Duffy found it was possible to run a back-end log collector and obtain the URLs to the archives without any credentials.

The SSL certificates were the same for every instance, and it was possible to copy the SSL certificates from one RHEL instance to another on Azure and still authenticate on the appliance, Duffy found. This wasn't the case with AWS, as the instance also needed to boot from a correctly configured Amazon Machine Image (AMI) before it could use the SSL certificate.

Microsoft immediately took steps to prevent public access to and the Red Hat Update Appliances. The company told Duffy it has rotated all secrets, so even if the certificates had been maliciously copied, the attackers would no longer be able to access the appliances in this manner.

The appliances were exposing their REST APIs over HTTPS, which meant attackers with full administrator access could upload their own packages into the appliance. Any RHEL instance on Azure running yum to obtain a package with that same name would automatically receive that modified package, not the official one served by Red Hat. All Azure RHEL images are configured without GPG validation checks, so clients built off that image would not be able to tell the package had been tampered with.

Since the packages frequently execute as root, the attacker would gain full control over the RHEL instance. A specially crafted package capable of encrypting the entire virtual machine could lead to a ransomware attack on a massive scale. Or vandals could take over every RHEL instance for fun.

"[Compromising updates] would just be a case of bumping the version number and releasing a package under the same name," Duffy said.

Exposed storage accounts

The issue with the update appliances would have given attackers access to every compromised RHEL instance on Azure, but a different vulnerability within the mandatory Microsoft Azure Linux Agent (WaLinuxAgent) would have had a "much more widespread" impact, Duffy said.

The Red Hat Enterprise Linux image available on the Microsoft Azure Marketplace had a vulnerable version of WaLinuxAgent that exposed the administrator API keys for the storage account used by the virtual machine. With the API key, the attacker could download virtual hard disks for any RHEL instances using that storage account. Since multiple virtual machines shared a single storage account, an attacker could download multiple virtual hard disks at a time.

Azure administrators should check to make sure they aren't using RHEL image with the vulnerable agent, WaLinuxAgent 2.0.16.

Shared responsibility on the cloud

Microsoft has clearly spelled out the expectations for securing its cloud platforms in the Shared Responsibilities for Cloud Computing and Microsoft Azure Security Response in the Cloud whitepapers. Microsoft will take care of all the security for its buildings, servers, networking hardware, and the hypervisor for organizations using Azure for IaaS. The operating system, network configuration, applications, identity management, client security, and data remain under IT control.

In this case, because the issue lay with the fact that the appliances and applications were publicly accessible, the fix was Microsoft's responsibility. However, that doesn't absolve IT administrators from regularly monitoring the instances for unusual activity or checking what packages are being installed on their machines. Just because the provider is responsible for that portion of cloud security doesn't mean IT administrators can ease their vigilance.

Copyright © 2016 IDG Communications, Inc.

InfoWorld Technology of the Year Awards 2023. Now open for entries!