Why did progress slow?
There's a simple explanation for Java falling off the pace: Sun was not a healthy company. Java was born in the dot-com boom when Sun was selling Sparc boxes left and right. Meanwhile, the cost of Intel/AMD came down, but the price of Sparc didn't. Despite the excitement around the T1000 and later platforms, the economics weren't there. (Sure, the late-model Sparc is more efficient, but the price is so freaking high that I might as well power my data center on mahogany and endangered species rather than pay the premium for the hardware. Even if our legislators passed cap and trade, I could buy a small rainforest worth of carbon credits on the cost difference and theoretically make up for my excess power usage.)
Then the dot-com boom went bust, and Sun decided to pursue "commodity" computing in its hardware business right as the big iron clusters were being built. In short, Sun made the wrong bets in its hardware business.
Sun was great at creating ecosystems; it just couldn't create products that businesses wanted to buy. Its eventual suitor, Oracle, is great at burning ecosystems to the ground, eating or destroying all of the companies that play in them, and creating highly profitable products to replace them.
Oracle acknowledged some of the business and political issues that delayed Java 7 in one of its classically terse public statements: "We all know for various business and political reasons that this release has taken some time."
However, we have to look beyond Sun's financial troubles and examine the system surrounding Java. Because Sun went back on its word to submit Java for standardization, it created its own "standards" committee called the Java Community Process. Originally it was a sort of smoky backroom star chamber. Over time, it opened up to some degree, but Sun and now Oracle, with absolute veto power, ignored the rules and did whatever they wanted.
What's been holding back the JCP? It's not openness -- it's competing interests. Although I sat on the sideline and ate popcorn, I remember the three-letter vendors participating in EJB3 back in the day. They contributed delays and procedural concerns. Why? They needed to buy or develop a product to integrate into their app servers. If the next JavaEE spec came out, they didn't want to be late to market if they could help it.
It's difficult to coordinate a product release in one company, let alone several. Luckily, with consolidation this may get easier. I don't think the JCP contributes the largest part of the problem for Java.
Separate the Saucer Section
Today, Sun is a memory and Oracle is the boss. Yet why do Java releases still take so long? The simplest explanation is that Java is too big. Big projects tend to go slow and are fraught with risk. To solve this problem, let's look at how Java could be smaller.
First, Oracle has to get over its crush on client-side technologies. Sure, there isn't much in Swing or Oracle's anointed successor for it, JavaFX, that can't be done in a modern Web browser with fewer problems, but Oracle needs you to be tied to its platform -- at least it did.