Virtualization and private clouds have delivered hardware savings in a big way for most enterprises. Software savings, however, are proving more elusive, and that's been particularly true when it comes to virtualizing Oracle databases, according to several sources.
While virtualization has enabled server consolidation ratios of 3:1 or more, businesses may see little or no reduction in associated software costs. In some cases enterprise software licensing can be so expensive that it overshadows the cost of the rest of the system stack combined, says David Welch, chief technology officer at House of Brick Technologies, an integrator with expertise in Oracle software licensing issues.
[ Also on InfoWorld: Microsoft Hyper-V 2012 vs. VMware vSphere 5.1 face off -- see which virtualization client prevails. | Doing server virtualization right is not so simple. InfoWorld's expert contributors show you how to get it right in this 24-page "Server Virtualization Deep Dive" PDF guide. | Track the latest trends in virtualization in InfoWorld's Virtualization Report newsletter. ]
(Editor's note: During the months of reporting this story, we contacted Oracle 13 times, and the vendor's spokespeople declined to comment on this story on three separate occasions.)
Some consultants and a handful of IT executives say they face licensing obstacles with many enterprise software vendors, but Oracle agreements can present some of the most confusing compliance issues, especially for customers that fall under Oracle's processor-based licensing models. "Using Oracle on VMware provides no licensing savings, just operational cost savings," says R Wang, principal analyst and CEO at Constellation Research.
"Virtualization is an area where customers get creamed all the time," says Craig Guarente, CEO of software auditing and compliance consultancy Palisade Compliance. Both Guarente and Welch say that, all too often, organizations end up overpaying for new Oracle licenses as well as software maintenance after consolidating servers or adding new licenses in a virtual environment.
Several IT executives contacted for this story declined to speak on the record. One executive, "Tom," has dealt with Oracle as the CIO for several Fortune 500 corporations, and agreed to speak only on condition that his name and company affiliation not be used. "Part of the challenge is knowing what others are paying," he says, and that information is considered confidential. But professional negotiators can be helpful because they know what the average company is paying for any given product, he adds.
Some Oracle customers are "p.o.'d" about virtualization, Guarente says.
One client called Palisade after installing Oracle database software on two servers in an eight-server vSphere cluster. Oracle discovered the configuration during a software audit and demanded that the customer buy a license for every processor and server in the cluster.
We haven't noticed other enterprise software vendors that assert the full-cluster licensing claim. David Welch, chief technology officer, House of Brick Technologies
That's unusual, says Welch. "We haven't noticed other enterprise software vendors that assert the full-cluster licensing claim," he says. Oracle's per-processor licensing fees start at $47,500, so charges can add up quickly. To make matters worse, the customer had also unknowingly installed the more expensive Enterprise Edition of the Oracle database instead of the Standard Edition.
Before the audit, the company had been paying Oracle $50,000 per year. The list price for licensing the new configuration: Just over $1 million. The customer ended up reconfiguring the vSphere infrastructure to limit the number of servers in the cluster, and bought more Oracle licenses. Guarente wouldn't say exactly how much the customer saved, but did say the customer paid a fraction of that original $1 million bill.
The company could have achieved the same goal without paying anything additional had it reconfigured the cluster to run Standard Edition before the audit took place. Or it could have gone back to Oracle and bought more licenses, Guarente says. "The first option would have resulted in no additional fees to Oracle. The second would have resulted in some fees, but far less than what they ended up paying."
Bob Clarke, manager of enterprise services for the Office of Technology in the State of Indiana, is considering his options for saving money. Like the customer described above, he also faces the requirement to license every core and server in the cluster.
"It gets very pricey," he says, but he hopes to make the numbers work by consolidating some of the existing licenses for Oracle software used by separate departments. "We will create a vSphere cluster that's dedicated to Oracle. We will comply with their licensing by doing it that way," he says.
But user experiences vary widely. "Bill," a CIO at another Fortune 500 company who also asked not to be named, says he has never been asked to pay to license every core in a server used in a virtual environment when not all are running Oracle, nor has he been asked to buy Oracle licenses for every server in a vSphere or any other virtual server cluster.
The key is in the negotiations and the customer's relationship with Oracle. "It's battle. You have to work hard," he says, but he's always been able to negotiate a satisfactory contract. Then again, he admits, he's never been through an audit.
Being asked to limit vSphere clusters just to Oracle isn't always efficient, especially for large data centers, and it's not necessary, professional negotiators say. "There is nothing in the contracts that say you have to pay for all the servers in a cluster," Guarente says. Although the software vendor has published separate policy documents that attempt to address this issue, Welch says there's nothing in black and white to back up the assertion that every server in a vSphere cluster must be licensed if Oracle software is installed or running on any one of those servers.
"The biggest issue for me is that these documents are not part of the contract the customer signed," says Guarente. "Oracle has very interesting policies around virtualization, and none of those policies are located in the contracts."
The problem with soft partitioning
One of the most relevant policy documents on the matter, the Oracle Partitioning Policy (PDF), distinguishes between hard partitioning and soft partitioning technologies with respect to licensing. With hard partitioning, customers can limit the number of processor licenses needed to those running Oracle. With soft partitioning, they cannot.
Oracle defines Microsoft Hyper-V and VMware as soft partitioning, although many technologists will argue that the features and methods used in VMware are no different than those that appear in Oracle's list of hard partitioning technologies, Guarente says. "Given that this is a competitor's offering, Oracle will always consider this as a form of soft partitioning," he explains.
The one exception, he adds, is under Microsoft's Windows Azure cloud offering, which uses Hyper-V. In that case, Oracle can be deployed within a virtual machine and the customer pays only for the resources used. "The reliance is on the provider to control the size of the virtual machines," he says.
Even if the customer's software license refers to the Partitioning Policy, that document includes a note stating that the policy "is for informational purposes only," and "does not constitute a contract or a commitment to any specific terms."
So does the policy apply or not?
Guarente says when he challenged one contract based on published policy material for an organization Oracle was auditing, he received that very same response from License Management Services, Oracle's auditing group. "They came back to us and said the documents were for educational purposes only and not binding. Fascinating." That business was able to successfully defend its position with Oracle, he adds.
"It's the art of working with Oracle versus the science of compliance. It's all very muddy," Guarente says. Furthermore, the list of approved hard partitioning technologies typically does not appear within the list of documents referenced in the Oracle Software Licensing Agreement (OSLA) that customers sign, says David Blake, CEO at UpperEdge, an IT sourcing consultancy.
Nonetheless, if one follows the Oracle Partitioning Policy, every processor and core within a single physical server must be licensed for Oracle in a VMware environment, even if the Oracle software is installed or running on just one core. House of Brick's Welch tells his clients to pay to license every processor and core in that situation. "We tell everyone that Oracle is right. That policy has been there since 2002," he says.
The subcluster debate
Running Oracle on a dedicated group of servers within a vSphere cluster, sometimes referred to as a subcluster, is a different matter, however. "Oracle likes to tell prospects and customers that if they are running Oracle on any physical server within a large vSphere cluster they have to license the entire cluster for Oracle. Nothing could be further from the truth," Welch says.
"The contract states that you must license any physical server on which you have installed or are running Oracle binaries. But you don't have to license other servers in that cluster," he insists. "Amazingly, many organizations don't know that."
Welch says he has negotiated on this point many times for enterprise clients. "When we have asked Oracle auditors if they could show us where it says in writing that every server in a vSphere cluster must be licensed for Oracle they said it was an unwritten policy."
Blake concurs. "Overall, the policy does not specifically call out the cluster scenario, so we advise clients to challenge Oracle" when a request is made to license every server in a vSphere cluster, he says.
In some cases, Welch adds, Oracle has asked some customers to license failover nodes in the cluster as well, but you're entitled to leave one node unlicensed, even when every physical server in a vSphere cluster is dedicated to Oracle, he says. Oracle's Software Investment Guide includes what's commonly known as the 10-day rule. "It says that you can run Oracle on an unlicensed physical server for up to 10 cumulative days per year as long as it's a named, designated node, and you fail back."
So why is this happening? Oracle is such a large organization that "people just make the assumption that there's no way that an Oracle account representative could be representing something to an organization that would not be contractual or binding," Welch says.
But he cautions, "Corporations need to examine their licensing agreements rather than just listening to Oracle. And that can include case law searches should a licensee care to check if the largest relational database vendor in the world has ever made any legal filings on this topic."
As to the issue of licensing failover nodes, Welch elaborates: "VMware High Availability accomplishes the same thing as its active/passive HA predecessors such as IBM HACMP, HP Service Guard or Veritas Cluster Server. We never saw Oracle assert [that] the passive node had to be licensed in those legacy clustering technologies. But suddenly it is making a fuss about vSphere clusters," Welch says.
One potential reason for the discrepancy, Welch asserts, could be that those legacy HA failover technologies don't present a threat to Oracle's processor-based license revenue in the way VMware does by enabling massive server consolidations.
Overspending on Oracle
The costs of over-licensing can be substantial, and Welch thinks many customers make unnecessary "donations" to Oracle when it comes to virtualization. "When buying licenses for a new project on a vSphere cluster, we see 150 percent to 300 percent overspend on Oracle licenses alone," he says. "This overspend dynamic with respect to VMware is unique to Oracle, in our observation," Welch says.
That's not to say that other enterprise software doesn't present challenges. For example, Microsoft's licensing policies for SQL Server on VMware took a significant turn for the worse with SQL Server 2012. But in that case the heavier licensing requirement was "unambiguously contractual," he says.
One House of Brick client had hundreds of vSphere hosts, only wanted to run Oracle on a few of them and didn't want to divide up the cluster to dedicate one to Oracle. "For that corporation, a national leader in their industry, the ramifications would have scaled into the tens of millions of dollars of unnecessary additional spend with Oracle," Welch says. Executives there are still trying to decide what course to pursue.
Another company, this one in the transportation industry, came to Palisade after finding itself in a disagreement with Oracle over how to count processor licenses in its virtual environment. "In this case we were talking about 60 processors. It was almost a $3 million judgment call," Guarente says. Ultimately the client prevailed. But, he adds, their legal counsel did so in a manner that did not destroy the business' relationship with Oracle.