Berners-Lee blames the browsers for not requiring well-formed code, but his colleague Håkon Wium Lie, CTO of Opera Software and inventor of CSS, believes there's more to it than that. Lie suspects that XHTML is unpopular because it tends to "punish the good guys" by being too rigid and unforgiving in its syntax. Writing good XHTML is laborious, a pursuit better suited to engineers or library science majors than Web designers. What Web publishers care about is producing exciting content, not standards compliance.
So it's back to the drawing board. In his post, Berners-Lee announced a brand-new working group within the W3C that would once again try to address the challenges and shortcomings of HTML, while working on the XHTML standards in parallel. The new group will take input from engineers, browser vendors, and Web developers, and make incremental improvements to the standards, taking into account the needs of diverse audiences.
It's a good step. But it does make me wonder about the future of Berners-Lee's vision of the Semantic Web. The lesson learned from XHTML is that, when it comes to standards, just because you build it doesn't mean they will come. And yet, XHTML is only the beginning of the standards compliance that the Semantic Web would require. If the Semantic Web is to succeed, it will have to find ways to accommodate human nature, and not just good engineering -- or I suspect the work Mr. Berners-Lee has ahead of him will be very hard, indeed.