OK, class, time for a pop quiz. Who suffers the most from the censorship clauses many software companies include in their
shrink-wrap agreements? Is the answer A) the computer press, B) independent labs, C) the software companies themselves, D)
software customers, or E) all of the above?
In case you’re having trouble with your answer, allow me to provide a crib sheet with this recent example of how censorship
clauses are inhibiting open public discourse. Chris Merrill, an engineer and partner at Raleigh, N.C.-based Web Performance,
was trying last fall to put together a benchmark comparison of Java servers using his company’s Web Performance Trainer Web-load
testing tool. But he encountered a roadblock because several of the commercial products he wished to include had a provision
in their license agreements prohibiting publication of benchmark results without the company’s permission. In particular,
he tried to obtain permission from BEA Systems to include results for the single-server version of WebLogic, only to be told that BEA has a blanket policy against granting permission for publication of third-party benchmarks.
“One question we get asked a lot is how the performance of commercial and open-source J2EE servlet applications compare; so we were just trying to provide some objective information in our research report,” Merrill says.
“None of the commercial vendors that had a benchmark restriction in their license agreement would grant us permission, but
we were able to determine in several cases that the restriction didn’t apply to what we were doing.”
In an e-mail exchange with Merrill, however, BEA officials said they would strictly enforce the company’s benchmark policy.
“At first they seemed to be saying they could not allow anyone who is not an expert with their product to publish performance
measurements, but they never asked for my credentials or for my testing methodology,” Merrill says. “Later they suggested
they might grant me permission if they could preview the results — and if the results favored BEA — but I wasn’t willing to
work that way.”
So when Web Performance posted its benchmark comparison in November, WebLogic wasn't included. Merrill was aware that the censorship clauses are of very dubious legal standing, but sensibly enough, he
wasn’t about to get into a legal fight with a company as big as BEA. “For our company, which is very small, we would be out
of business long before the case made it to a judge. Legal enforceability does not matter — we would never make it that far.”
For their part, BEA officials are refreshingly up front about the reasons for their policies regarding publishing of benchmark
results. “I wouldn’t say our policy is no benchmarks. But the bar is pretty high for what we want to see for the person executing
the benchmark, the other vendors participating, etc., if we’re going to invest all the time to participate,” says Eric Stahl,
director of WebLogic Server marketing at San Jose, Calif.-based BEA.
“Performance-tuning of an application server is a very important thing, and it has to be done correctly if we are going to
help people see the differences between products. And we just don’t have the resources to send out our engineers to participate
in every opportunity that comes our way. So unless it’s a really high-visibility scenario, I’m sorry, but we cannot give our
permission to test,” Stahl says.
Stahl was one of the BEA officials Merrill corresponded with, and he deserves credit for not stonewalling these requests the
way we’ve seen certain other companies do in this situation. He didn’t recall Merrill’s specific case, however, because he
gets many similar requests. “Probably not a week goes by that we don’t have to deal with a request like this,” Stahl says.
“I can understand the frustration that those making these requests feel, but we invest many millions of dollars into the product.
So we don’t want anyone publishing our benchmarks unless we agree on how it’s being tested, because if it’s done wrong, it
could have a devastating impact on our business.”
Stahl thinks the only answer is for companies to get together and settle on industry-standard benchmarks similar to the Transaction
Processing Performance Council tests that database vendors use. “That approach lets each vendor put forth their best representation
and alleviate all this problem of letting people publish anything they want,” he says. But Stahl admits that such tests don’t
always reflect real-world results and are also subject to abuse by competitors who find ways to sidestep the censorship clauses.
Specifically, he points to Microsoft-sponsored J2EE vs. .Net benchmarks published in November (about the same time Merrill’s
BEA-less results were published) as a “sham of the highest order” and “fraudulent misrepresentation of our product” that are
still to be found on Microsoft’s site in spite of having been debunked.
If this sounds vaguely familiar, it’s because Microsoft officials have had similar things to say on how J2EE competitors have
misrepresented .Net performance. In spite of all these vendors having censorship clauses that are supposed to protect them
from publication of biased or badly conducted performance tests, they all claim to be victimized anyway. So what good are
these benchmark restrictions, other than keeping the small-fry like Merrill from collectively providing a more balanced and
objective view? And, by the way, there’s also the little detail that “people publishing what they want” isn’t supposed to
be a problem in this country but a right.
So the answer to our quiz is …. Sorry, it was a trick question. Not even E) really adequately covers who suffers from the
fact that open discussion about a crucial technology is being thwarted by a handful of companies on an almost daily basis.
Our whole society, and our future, is suffering incalculable damage. It’s time we wake up to that fact.
E-mail
Printer Friendly
Reprints
(InfoWorld) - In the litigious world we live in, deploying a unified communications platform in your enterprise could...
