Microsoft explains why the October 2018 issue should never have happened

After the disastrous initial roll-out of the Windows 10 October 2018 Update Microsoft is trying again today, and in a blog post, they explained their approach to releasing high-quality Windows 10 updates.

Calling last month’s problems “small but serious”, Microsoft said the issue belied a trend of quality improvements with each successive Windows 10 release.

Michael Fortin, Corporate Vice President for Windows, said there was a real and steady drop in the key indicators of product quality – customer service call and chat volumes, even while the number of PCs running Windows 10 increased. He also claimed Windows 10 April 2018 Update had the highest Net Promoter rating of any version of Windows 10.

Fortin said Microsoft invests in customer and partner planning feedback to help them shape and frame product specifications, including defining both required testing and success metrics and employ a wide variety of automated testing processes as they develop features, allowing them to detect and correct issues quickly. He insisted that testing was a key part of delivering Windows and that features needed to pass fundamental tests to be integrated into Windows.

He very briefly addressed the issue of Microsoft firing SDET testers in 2014, noting:

We shifted the responsibility for base functional testing to our development teams in order to deliver higher quality code from the start.

He also said Microsoft had a strong culture of running the latest internal version of Windows on their own machines, and then to make sure issues are pursued and fixed.

He also notes that Microsoft partners had their own labs where they tested Windows which included testing with:

• External testing labs with global, continuous coverage for application compatibility, hardware and peripherals
• ISVs for a range of apps including Anti-Virus (AV)
• OEMs partner with Microsoft to test and ensure quality across a vast array of systems, devices and drivers

He also called out the Windows Insider Program for providing fresh and valuable insights, with similar programs available for other commercial customers such as the Windows Insider Program for Business which allows organizations to access Insider Preview builds to validate apps and infrastructure ahead of the next public Windows release.

There is also the invitation-only program for large enterprise customers, the Technology Adoption Preview (TAP) program, that lets enterprise customers provide early feedback on product updates via real-world product testing to help Microsoft identify issues during the development process.

When it comes time for actual releases, Michael noted that the company used data-driven decision making using detailed dashboards and metrics to evaluate the builds.

He notes that Microsoft uses their real-time detection and response capabilities to roll out Windows in a careful and data-driven way, and that their first principle was to only update devices that their data shows will have a good experience, currently using a machine learning model to select the devices that are offered updates first. He says that if Microsoft detects that a device might have an issue, they will not offer the update until that issue is resolved.

The roll-out also starts slow and is first only available to “seekers”, users who take action to get the updates early, while Microsoft carefully monitors telemetry, including feedback from their customer service teams, carefully to learn about new issues. Microsoft also analyzing feedback logs and screenshots directly through the Feedback Hub, and listening to signals sent through social media channels.

If Microsoft finds a combination of factors that result in a bad experience, they create a block that prevents similar devices from receiving an update until a full resolution occurs.

Microsoft says they continue to look at ways to improve their ability to detect issues, especially high-impact issues where there are low volume and potentially weak signals, including recently added the ability for users to indicate the impact or severity of issues when they provide feedback.

Fortin said that despite the positive trends, they also hear clearly the voices of users who are facing frustrating issues, and pledges to do more and focus on new quality-focused innovation across product design, development, validation and delivery.

What Fortin did not deliver was a real autopsy of why the issues with the October 2018 update took place, and besides the work at amplifying “weak signals”, why such issues will not happen again.

Read Fortin’s full and lengthy blog post here, and let us know if you are satisfied with Microsoft approach to Windows testing.