Microsoft Research develop cheap AR headsets using passthrough VR

Real Augmented Reality headsets are very complicated, and even at their best, they offer a compromise experience, with issues such as small field of view, sensitivity to bright light and of course costing thousands of dollars.

Simply passing an external video signal through a VR headset has long been a proposed solution, but suffers from issues such as latency, which delivers a poor experience to users.

Now Microsoft Research has given the idea some attention and reports some surprisingly good results with a solution which only costs $40 more than a standard VR headset.

In development since 2016, Project Prism adds standard stereo cameras to a virtual reality (VR) system. The user sees the real world through the video feed from the cameras. Synthetic imagery is composited with the video to create a low-cost AR system.

This form of AR has several advantages over optical see-through AR:

1. Low-cost camera modules add roughly $40 parts cost to a VR headset
2. Wide field of view, achieving more than 100+ degrees horizontally and vertically
3. Works in full sunlight, with truly opaque synthetic imagery. This improves colour rendition and contrast of synthetic imagery even indoors.

Their first prototypes used off-the-shelf machine vision cameras attached to a commercial VR headset with a display resolution of approximately 1M pixel/eye at 60Hz or 90Hz. Microsoft wrote their own real-time image signal processing pipeline to demosaic the Bayer image, correct for camera lens distortion, and correct colour.

The resulting system had photon-to-photon latency of only approximately 50ms. The good results encouraged Microsoft to continue the project and in their newest system corrects the biggest weaknesses of the first generation, which were the bulk and weight of the large form factor-machine vision cameras, and the low resolution of the displays.

Their 2017 system has a custom camera control circuit board designed and built by the MSR Labs hardware lab. This uses a rolling-shutter image sensor which can capture 4M pixels at 90Hz, within a cell phone form factor. The cameras are synchronized to the VR headset displays by tapping the sync signal from the display controller board. The new camera module is much smaller and lighter than the smallest off-the-shelf machine vision system of equivalent resolution and frame rate and image quality is also surprisingly good given the small 2-micron-square pixels.

The MSR Labs Central Engineering team wrote custom camera-controller firmware so the camera frame rate, exposure, and other sensor parameters can be set in software on the host PC. They also wrote custom USB drivers to handle the high-data-rate video coming over the USB 3.0 cables. Photon-to-photon latency was also approximately 50 ms,  corrected with a late-stage reprojection homography.

The higher resolution significantly improved the view of the real world and the lightweight camera modules made it far more comfortable than the first-generation prototypes. In addition, the inside out tracking of the Mixed Reality headset did away with the mobility limitations of the earlier prototypes. This new system uses a backpack computer and is fully mobile.

Approximately 40 of these headsets were produced, and are being used for research purposes in various groups inside Microsoft.

Microsoft says they are actively developing future prototypes with latency in the 5-10 ms range. Higher resolution prototypes will be made as the resolution of off-the shelf HMDs increases.

With most of us not having $2000 to $5000 to spend on a real Augmented Reality headset, I am sure many of us would welcome an intermediate solution which is much more affordable.

Read more at Microsoft here.

Via WalkingCat