Microsoft's SIGMA is made for real-world tasks. This is how AI enters your eyes through HoloLens

Me for real-world tasks for HoloLens 2

Reading time icon 2 min. read

Readers help support MSpoweruser. We may get a commission if you buy through our links. Tooltip Icon

Read our disclosure page to find out how can you help MSPoweruser sustain the editorial team Read more

Key notes

  • Microsoft introduces SIGMA, an open-source platform for real-world tasks
  • SIGMA blends mixed reality and AI, offering step-by-step guidance through HoloLens 2
  • Tasks can be pre-defined or dynamic, with language and vision models aiding object detection and user queries

Microsoft researchers have just announced on Monday that it launched SIGMA. Short for Situated Interactive Guidance, Monitoring, and Assistance, it’s an “open-source research platform and testbed prototype” made for real-world tasks. 

Now available on GitHub, SIGMA is an interactive tool designed for HoloLens 2 that blends mixed reality and AI tech. It assists users in performing step-by-step tasks by providing guidance and answering questions, or in simpler words, it lets AI enter your eyes. 

Tasks can then be pre-defined or generated dynamically, and SIGMA uses language and vision models to detect and highlighting relevant objects in the environment. It can also answer user questions and detect task-relevant objects in the environment using vision models like Detic and SEEM.

“SIGMA was designed to serve as a research platform. Our goal in open-sourcing the system is to help other researchers leapfrog the basic engineering challenges of putting together a full-stack interactive application,” Microsoft says.

Built on the Platform for Situated Intelligence framework, SIGMA enables fast prototyping and provides tools for visualization, debugging, and maintenance. The system is designed with a client-server architecture, allowing data streams from HoloLens 2 to be processed on a desktop server, bypassing device limitations. 

Microsoft has been at the forefront of the AI race for quite some time. The Redmond-based company’s billions of dollars in investment in OpenAI, its numbers of AI models, and the Copilot AI tool in Windows 11/10 and mobile devices are just to name a few.