1. someoneinwa's Avatar
    In January I posted an "open letter" here to Consumer Reports magazine about the blatant Apple love they had displayed in two consecutive issues. The most recent issue of the magazine contains a large section on smartphones and a separate article on "chameleon computers" such as the Surface. And again, I have trouble with the magazine's standards for judging these devices as well as some of their editorial decisions.

    First, the improvements. In one of the issues I criticized in January, they referred to the Surface RT as "bulky" which made me wonder if they had even touched it. In the new issue, the Surface Pro is ranked highest (tied with the Samsung Ativ) among the group of 10 to 12 inch detachable laptops. Despite this ranking, the story is accompanied by a photo of the 3rd ranked HP Envy and there is absolutely no text description of the Pro beyond the ranked list. The Surface RT is ranked 7th in a category of tablets with 9 to 12 inch screens (tied with the Samsung Ativ and Galaxy). The RT even gets its own callout on one page under the sub-headline of "Best for Work" in which it is compared to the Ativ. The formerly "bulky" Surface RT is now "excellent" and is praised for its kick-stand and the quality of its display. The only knock is the lack of a GPS. Unlike the Pro, the RT scores a photo in its section. Interestingly, the magazine chose to show the RT in portrait position, which is not its best side in my opinion. Overall though a fair bit of work.

    On the other hand, Windows Phones are shunted off to a box with the Blackberry. The magazine ranks phones via carrier, so there are no Windows phones listed for Sprint or T-Mobile, only the HTC 8X for Verizon and the HTC and the Nokia Lumia 920 for AT&T. These two phones don't measure up well here. The HTC ranks 12th on Verizon and 11th on AT&T, while the Lumia ranks 13th. In an otherwise balanced article about changes occurring in smartphone features, the editors refer to Microsoft's products as "Windows Mobile" while offering the back-handed compliment that "even recommended phones that use the new Windows Mobile and revamped Blackberry platforms show the kind of innovation that was once Apple's hallmark." Apple fans need not despair at even this whiff of criticism of Apple however. The very next paragraph describes the "elegant iOS operating system". Sadly, the editors fail to tell the reader just how they have discovered the way to measure elegance in their testing.

    And that gets to my biggest complaint. The article acknowledges that Consumer Reports' research has revealed that most of their subscribers use iOS or MacOS products. They also reveal that their ranking of phones and tablets comes not from their testing but from satisfaction surveys of their subscribers. I've received these surveys as well as a subscriber. The bizarre result of this technique of ranking products, is that the rankings are first, not objective; and second, vague. For example. The rankings of phones for AT&T included the HTC 8X in 11th place with an overall score of 76. The 920 was in 13th place, also with a score of 76. Also at 76 points was the LG Escape. Six phones were tied at a score of 77, one at 78 and two at 79. The Galaxy S4 stood out with a score of 81. What in the world do these numbers mean? Often when ranking products, Consumer Reports will mention (in tiny type near the charts) that differences of a few points are meaningless. They don't do that here, so are we to believe that from a survey of Apple favoring Consumer Reports subscribers that the HTC One (ranked 3rd with a score of 79) is significantly better than the HTC 8X (way down the list in 11th place with a score of 76). The positions on the list, which is what people will notice, would indicate a significant difference between these phones. The scores may or may not be significant. We can't know because we aren't told.

    Producing reports based solely on customer satisfaction scores like this is very problematic. First, the fanboy syndrome affects Consumer Reports subscribers just like it does other people. Humans tend to be defensive of our choices. Second, people all grade differently. Just look at Amazon customer reviews. Even people who love a product can score it very differently. Third, editorial biases skew conclusions. In my January post, I criticized Consumer Reports for the photos it used in two consecutive months. In the most recent issue, the photo selection was better balanced, but the editorial content continues to have issues. In addition to the "elegant" problem I mentioned above, in another part of the article the writer says "Apple all but invented the smart phone and the tablet at least in their current incarnations." Even if that were true, how is it relevant to product testing?

    I think there is a real value for Consumer Reports to review and recommend smartphones, tablets, computers and other electronics. I am disturbed that far too much of their work in this category comes from surveys and unexplained rankings, and not the kind of research they do with other consumer products. Surveys have a place, but I'd have a lot more respect for a story that told me that a popular phone model has battery life issues or is touting a feature that doesn't really do anything. Tech magazines are doing a far better job in many cases of evaluating phones and tablets than the scientists and editors at Consumer Reports, yet given their large circulation, their survey based pseudo research is having a greater impact.
    Last edited by someoneinwa; 07-06-2013 at 04:58 PM.
    07-06-2013 01:17 PM
  2. Laura Knotek's Avatar
    I haven't found anything yet when searching for demographics of Consumer Reports subscribers. However, my guess if I kept digging or if someone else found results, would be that Consumer Reports subscribers tend to be wealthier and more educated than the average person in the US. Hence, the same people that fit the demographics of Apple product owners (especially Mac owners) would also fit the demographics of Consumer Reports subscribers.

    I haven't found anything so far to substantiate my claim, but I believe that Consumer Reports subscribers own more Macs (and other Apple products) than the general US population.
    Kevin Rush likes this.
    07-06-2013 02:24 PM
  3. Kevin Rush's Avatar
    I have noticed the bias favoring Apple and the slighting of Microsoft in the Consumer Reports magazine. It is disgraceful if Consumer Reports doesn't properly test and analyze the phones they report on. This is what we assume and expect.

    We receive the Consumer Reports surveys too, and find them poorly worded. I always feel like there is an answer that they want? I never thought this information would substitute entirely for actual testing and analysis.

    That all said, we've subscribed for more than 20 years and have made many purchases based on their recommendations. We assumed rigorous analysis.
    07-06-2013 02:54 PM
  4. mase123987's Avatar
    Consumer Reports is easily one of the worst sources for tech analysis and rankings. It has consistently shown a lack of knowledge and understand. I caught many of the same things as the OP.

    Back when the original Asus Transformer tablet came out, it was either the only or few Android tablets that had a keyboard. The keyboard itself had extra features that made it a very good product at the time. Consumer Reports called it one of the least versatile tablets. At this point, I knew they had no clue about the products.

    BTW, some people are going to like the look and form factor or the 8x over the 920, but there is no way it is a better phone. There is also plenty of phones above those two that had no reason in being there.

    Another thing, they ranked Atom-based tablets alone side the Surface Pro. The Atom-based tablets got dinged hard for performance. While I understand that the Atom processors don't perform as near as well, the two sets of tablets aren't competing against each other. $500 price range vs $1000 price range is two whole different segments.
    Laura Knotek and Kevin Rush like this.
    07-06-2013 03:39 PM
  5. Laura Knotek's Avatar
    When I was searching, I found that many car manufacturers complain about similar bias. It's not just with computers and smartphones.
    mase123987 likes this.
    07-06-2013 04:17 PM
  6. mase123987's Avatar
    For as much crud we give sites like Engadget or The Verge, at least they know it is WP and not Windows Mobile.
    Laura Knotek and Kevin Rush like this.
    07-06-2013 05:02 PM
  7. someoneinwa's Avatar
    I know Consumer Reports can do effective technology research. I should have noted in my post that the same issue I was criticizing has an article about ink jet printers and the result of actual tests of the products the magazine did to measure how much ink printers use in their maintenance modes and the vast differences in ink consumption based on whether you print XX number of pages all at once or over an extended period. The story includes a listing of printers based on the results. There was no survey of Consumer Reports subscribers about their satisfaction with their printers. This was helpful research and the kind of thing the magazine ought to be doing when it comes to other consumer electronics as well.
    Laura Knotek likes this.
    07-06-2013 05:07 PM
  8. mase123987's Avatar
    All I will say is that lazy analyses on their part isn't likely to be strapped to one segment. I just notice phones and tablets the most since that is where my knowledge is greatest.
    Laura Knotek and Kevin Rush like this.
    07-06-2013 05:13 PM
  9. Laura Knotek's Avatar
    All I will say is that lazy analyses on their part isn't likely to be strapped to one segment. I just notice phones and tablets the most since that is where my knowledge is greatest.
    I wonder if Consumer Reports failure is because they are a "jack of all trades but master of none." At least a site that specializes in one area, or in several closely related areas, will not have that problem. You won't see Motor Trend reviewing refrigerators or smartphones, nor will you see WPCentral reviewing refrigerators or cars. Sites that limit their focus probably have more knowledge of the products they review, and are better able to give quality reviews.
    07-06-2013 06:21 PM
  10. mase123987's Avatar
    That is a good point Laura. I think that they also try to boil results down to bite size information so people can look at it at a glance.
    Laura Knotek likes this.
    07-06-2013 06:24 PM

Similar Threads

  1. Now runs perfect on Orange
    By Coolknight1968 in forum Windows Phones
    Replies: 3
    Last Post: 02-13-2014, 09:30 AM
  2. Exclusive Samsung ChatON Messenger On Lumia 820?
    By RahulRana in forum The "Off Topic" Lounge
    Replies: 5
    Last Post: 07-21-2013, 02:39 AM
  3. Angry Birds Star Wars on HTC 8s
    By Mairsh in forum Windows Central Archive
    Replies: 2
    Last Post: 07-08-2013, 04:13 PM
  4. Navigating Forums on WPCental App
    By IamJonso in forum General Phone Discussion
    Replies: 6
    Last Post: 07-07-2013, 07:31 AM
  5. Replies: 1
    Last Post: 07-06-2013, 01:17 PM