Microsoft and Volkswagen Collaboration: The Story Behind HoloLens 2’s Moving Platform Feature

Mixed reality headsets can be truly useful, but their ability faces limitations when used in one specific scenario: a moving environment. Microsoft and Volkswagen have found a solution for this through a joint effort that brought us the “moving platform” mode for HoloLens 2.

Volkswagen envisions augmented reality as a major component of future mobility concepts. So, it started different steps to explore the technology. In 2015, it conducted a research project using self-piloted vehicles and augmented reality for its Race Trainer system to teach driving on a racing circuit. It also introduced the augmented reality head-up display in its ID. family of electric cars in 2020, wherein different information about the environment in front of the vehicle (like navigation arrows and lane markings) is projected onto the windshield.

“We think mixed reality information is the most intuitive information we could provide to enhance our customers’ user experience,” Dr. Volkswagen Group Innovation’s Data Science Team Head Andro Kleen told Microsoft. “Because what you see there, and what you need to process, is very close to what humans normally see and process. It’s not so abstract.”

With an aspiration to get more out of technology, the German motor vehicle manufacturer used the HoloLens for another research project, where it was put in a moving vehicle. Things didn’t go well, unfortunately. The sensors were losing their tracking ability, and holograms were disappearing. The problem pointed out to the two motion-related sensors of the device experiencing confusion in the signals they were picking up while the vehicle was moving. With this, Volkswagen asked Microsoft for help, and the journey of the companies started in 2018.

“We had extensive discussions,” says Marc Pollefeys, Microsoft director of science and an expert in 3D computer vision and machine learning and now the director of Microsoft’s Mixed Reality and AI Lab in Zurich, Switzerland. “They presented their use cases and what they were hoping to enable. They were eager to work with us to find a solution and be able to use HoloLens in those situations.”

With the pandemic causing havoc around the globe, the research apparently faced various problems. The Covid-19 restrictions in Switzerland prevented the team from testing the tech inside a vehicle with other passengers, so Joshua Elsdon – a Microsoft senior software engineer part of the project – had to work on it and perform tests within his Zurich apartment.

“We had to do a lot of testing in my apartment. These aren’t ideal development conditions,” says Elsdon. “All of this stuff was done remotely and distributed across different countries, which was interesting.”

Later, the team tested the prototype on Puget Sound by riding recreational boats, then with Volkswagen inside vehicles.

“We connected a positioning system that tracks the location of the vehicle,” says Michael Wittkämper, Volkswagen’s augmented reality expert. “This way we were able to also place 3D elements such as information on point of interests outside of the car. This opens up completely new possibilities to not only display holograms within the driver’s forward-facing field of view, but also wherever the user wearing the glasses is looking.”

Everything became a success, allowing Microsoft to roll out the moving platform feature. Currently, the technology is a big attraction for maritime companies, which use HoloLens to connect maintenance crews on the ship to consult experts from other locations. Microsoft has bigger plans for the new feature in the future, specifically where moving environments are involved. Pollefeys also said that the company hopes to make it functional in daily life through portable smart glasses. However, with the current price tags of HoloLens 2 ($3,500 to $5,199), the plan might still be far from reality, especially if Microsoft wants the device to reach ordinary consumers in the market.