Microsoft vs Consumer Reports: A closer look at Microsoft's Surface defence

Microsoft, in response to a Consumer Reports report about the Surface and reliability, has penned a blog-post defending the honor of the Surface.

While I am a fan of Surface, I’m also a fan of consumer rights over manufacturer feelings. A news headline which reads “Microsoft’s Surface is a reliable product according to Microsoft”, has the same impact as “water is wet” to me. So, let’s take a look at Microsoft’s Surface defence piece and whether it does as much defence as it appears to on the surface.

The Surface Team’s mission is and has always been to make devices that deliver great experiences to our customers and fans. It’s the motivation for everything we do, and we are proud of the Surface devices we have built.

This is a great intro, but it doesn’t say much that a reasonable person wouldn’t have expected from a company defending its product.

This is why today’s Consumer Reports survey is disappointing. While we respect Consumer Reports, we disagree with their findings.

This is actually a strangely vague paragraph here. Microsoft is taking issue with a survey and views it as “disappointing”. What exactly does Microsoft disagree with? The research methodology? If so, the firm should disagree with the findings not only for Surface but for other WIndows powered PCs as well in the interest of fairness. If the methodology isn’t being fair to Surface, then it shouldn’t be fair to Apple, Samsung and HP as well. Is it the outcome that is disagreeable? All the PCs are tested on the same scale, so if we were to charitably make it 1 in 10 Surfaces which are faulty, then the other PCs should still improve in reliability proportionately and Microsoft would still score significantly lower than the other OEMs, moving us back to square one.

Surface has had quite a journey over the last few years, and we’ve learned a lot. In the Surface team we track quality constantly, using metrics that include failure and return rates – both our predicted 1-2-year failure and actual return rates for Surface Pro 4 and Surface Book are significantly lower than 25%. Additionally, we track other indicators of quality such as incidents per unit (IPU), which have improved from generation to generation and are now at record lows of well below 1%.

Once more, this is vague. Consumer Reports does qualify what it means by incidents in its report. The firm lists as freezing, start up issues etc. as examples of issues which cause the demerits. In comparison, Microsoft does not address what it ranks as issues. Do issues resolved by customer support count as issues? Or is it only those which lead to returns which are resolved as issues? When the firm says record IPU lows of well below 1%, what time-frame are they referring to? Is it “in August 2017, we re-evaluated our devices, and the IPUs are at record lows of below 1%”. Or is it “on the 11th of August 2017 between 11:00 PM and 11:01 PM, we checked and IPUs for are below 1%”? Or, are they counting only the Surface Laptop and new Surface Pro (Since they make reference to generational improvements), which are recent devices and therefore not yet candidates for long-term stability?

Surface also ranks highly in customer satisfaction. 98% of Surface Pro 4 users and Surface Book users say they are satisfied with their device*, and our Surface Laptop and new Surface Pro continue to get rave reviews.

This is also vague. Microsoft bases this “on a January -June 2017 study of consumers in the US, UK, Germany, France, China, Japan, and Australia, conducted by research company IPSOS, commissioned by Microsoft.”

The firm does not say how many customers were surveyed at this time, and it doesn’t do wonders for the perception that Google’s first auto suggestion for the term IPSOS Survey is  “IPSOS Survey Scam.”   In addition, I speak from experience when I say “rave reviews” which take up two to three weeks of use at most cannot be used as an indicator of any form of long term quality unless said reviewers continue to use Surfaces regularly post-review period. Most reviewers do not, and prefer to use traditional laptops for work. At most, a review can only tell you whether said device is worth buying over other devices, long-term, it’s a dice roll.

In my anecdotal use of a Surface Pro device over three years (Replaced twice for hardware issues), sold when it worked, it worked, When it didn’t work, you wanted to toss it out the window.

If Microsoft intends to fight statistics with statistics, the firm is doing a poor job of it so far.

A part of being product makers is listening to our customers and pushing ourselves to evolve and improve with every generation of devices we create. We’re forever seeking innovation, encouraging positive change. It’s what we do, and what our customers do, but I can promise you one thing that will never change is our commitment to our customers and our dedication to ensuring your Surface experience only gets better.

We are proud of our products and the amazing things our customers are doing with them. We stand firmly behind the quality and reliability of the Surface family of devices, and I can confidently tell you there has never been a better time to buy a Surface.

This is the best iPhone yet. Sorry, wrong company.

I believe Microsoft was between a rock and a hard place here, while the firm would in no way come out and say that “Yes, our devices are unreliable”, its defence of the Surface is a non-defense. It does not provide any numbers to counter the Consumer Reports report. When it does, it is too vague to be useful, or too specific to be helpful.

It’s worth noting that Apple and Consumer Reports also had a “scrap” about device ratings last year, with Consumer Reports testing demonstrating that the Macbook Pro offered significantly less battery life than advertised. Apple did work with consumer reports to fix the issue, leading them to revise their rating, but its unclear whether Microsoft can do the same, especially in a survey which is long-term in nature.

The best way for Microsoft to fix its Surface perception problem is simply to work harder and maintain reliability. After all, with Microsoft controlling everything about this device, it has no one to blame when things go wrong, but itself.