Even Consumer Reports finds smartphone ratings are tough

Rates iPhone 3G S on top, but not by much, in latest evaluation

When it comes to rating new smartphones, even one of the toughest and most objective rating sources, Consumer Reports, concedes the task is difficult.

In its latest ratings available to subscribers, Consumer Reports listed the new iPhone 3G S on top, but the magazine noted that other smartphones were close.

[ Stay ahead of advances in mobile technology with InfoWorld's Mobile Report blog and Mobilize newsletter. ]

In making those ratings, however, Consumer Reports appears to have gone through plenty of late-night soul-searching about how to rate complex products like smartphones, recognizing that personal taste about what features matter the most to users comes into play, as well as testing methodology.

"Smartphones are among the most complex products we test, and perhaps most subject to personal preference," Consumer Reports blogger Paul Reynolds wrote about the smartphone ratings.

Reynolds also said the magazine recently revamped its ratings methods and scrapped its previous tests of wireless reception because the tests could "no longer adequately replicate real-life reception experience."

Again, the complexity of the smartphone technology came into play in that decision, he added. "Creating reliable reception tests has become steadily more challenging as smartphones add more tasks and use a growing range of reception technologies," he wrote. Consumers Reports is developing tests to more accurately evaluate reception, and is "wrestling with ways to evaluate a smartphone's speed and versatility - growing concerns as the devices acquire more sophisticated operating systems and a host of third-party applications."

As for the actual ratings, the iPhone 3G S, according to the CR blog, has "strong performance in everything but voice quality."

The magazine had earlier conducted a head-to-head review of the iPhone 3G S and the Palm Pre, and found that the Pre and the BlackBerry Storm bested the iPhone 3G S in messaging, while the Pre is considered a "superior multitasker" with its card deck handling of multiple applications.

On its 100-point scale, the iPhone 3G S ranked a 73, besting the iPhone 3G, the second generation model, which got a 70. The BlackBerry Storm got 69 points, along with the T-Mobile G1, BlackBerry Bold and Samsung Epix. The Palm Pre and BlackBerry Curve were both rated 67.

Some of the smartphone problems mentioned in recent days on user forums apparently don't come into account in the Consumer Reports ratings, which looked at display, navigation, voice quality, phoning, messaging, Web browsing, multimedia and battery life. For example, the iPhone 3G S got an "excellent" rating on battery life, even though users are seeing iPhone 3.0 software as a drain on batteries.

And the Pre has recently been rapped by users in forums for hardware glitches. Hardware quality isn't a major factor in the Consumer Reports ratings.

Consumer Reports rated voice quality "fair" on nearly all the top-rated smartphones, but commented in its blog that it had recently changed its ratings attributes to reduce the value of voice quality "in part to reflect the growing importance of non-voice use of smartphones." As a consequence, Consumer Reports put more emphasis on display, ease of navigation and multimedia and messaging prowess.

Like any reviewer, critic or ratings group, Consumer Reports recognizes that one rap on a particular brand of smartphone will bring out objections from its fans in force on Internet forums who don't hold their punches. Past smartphone ratings "have prompted debate and sometimes disagreement," Reynolds wrote in classic understatement.

Consumer Reports seems willing to leave those comments to its blogs, although the tone is more civil than comments on nearly any other smartphone forums.