Knowlton, a principal group program manager for Office, gave Microsoft's most detailed account yet for the September screw-ups in a Friday message to the listserv.
"Both of these errors are anomalies in our release operation," said Knowlton. "The XML config[uration] entries had to be hand-authored due to some product code changes. We rarely do this; they are typically machine-generated. In [the blank folder pane in Outlook 2013], a late change to the list of things we intended to ship resulted in a specific configuration not executing as expected."
Knowlton argued that the quality level for Office updates is "very high" considering the volume of updates issued and the number of customers who apply them. He also promised that the quality of patches would improve -- a message Microsoft has used before -- saying, "We are as concerned as any of our customers about these issues and we will come back in October better than we were before September."
Another Microsoft manager, however, sounded peeved that Bradley had emailed the CEO.
"We are following up with the people who published those updates. And no, it's not because Mr. Ballmer intervened," wrote Ben Herila, who identified himself as the program manager for WSUS (Windows Server Update Services), the widely used enterprise patch management service Microsoft runs. "Rather, it's because Susan so kindly let us (the WSUS product team) know about her problem."
Dustin Childs, a group manager of Microsoft's Trustworthy Computing group, also alluded to doing something -- he did not specify what -- to put a stop to the mistakes. "The quality of security updates is critical to our customers, and it is a high priority for us, too," Childs said. "We are actively looking at where improvements can be made with the goal of reducing implementation issues, and we will remain transparent with our customers about security threats, protections and update issue resolution."
It may take a lot more than words to calm the roiled waters.
"Not only are the end users suffering by these bad patches, the IT administrators are suffering even more because they have to hear all of the complaints from the end users and they have to spend time troubleshooting the issues and get things fixed," wrote John Hallis on the same mailing list thread. "You would think a company that has received billions of dollars from us would actually listen to what we are telling them about patching issues and get right on it."
And Bradley saw the problem as endemic at Microsoft.
"I think that releasing 80 non-security updates on an already busy patch month is releasing way too much code at one time," she said via in an email to Computerworld today. "You are going to get stuff missed."
Like other patch and security professionals, she cited the advantage baked into the cloud when compared to on-premise software. "Cloud gets a build to build deployment and thus when Exchange 2013 got its first security update, their cloud servers were fine, [but] on-premise servers barfed," she said, referring to the August update gaffe involving Exchange.
But she also blamed overstretch for the slide in quality.
"My rant wasn't just about the quality of security updates -- but the quality of patching as a whole," Bradley said. "Documentation is lacking, quality of updates -- especially in certain categories of updates -- is clearly lacking.