Microsoft's SIGMA is made for real-world tasks. This is how AI enters your eyes through HoloLens

Microsoft researchers have just announced on Monday that it launched SIGMA. Short for Situated Interactive Guidance, Monitoring, and Assistance, it’s an “open-source research platform and testbed prototype” made for real-world tasks. 

Now available on GitHub, SIGMA is an interactive tool designed for HoloLens 2 that blends mixed reality and AI tech. It assists users in performing step-by-step tasks by providing guidance and answering questions, or in simpler words, it lets AI enter your eyes. 

Tasks can then be pre-defined or generated dynamically, and SIGMA uses language and vision models to detect and highlighting relevant objects in the environment. It can also answer user questions and detect task-relevant objects in the environment using vision models like Detic and SEEM.

“SIGMA was designed to serve as a research platform. Our goal in open-sourcing the system is to help other researchers leapfrog the basic engineering challenges of putting together a full-stack interactive application,” Microsoft says.

Built on the Platform for Situated Intelligence framework, SIGMA enables fast prototyping and provides tools for visualization, debugging, and maintenance. The system is designed with a client-server architecture, allowing data streams from HoloLens 2 to be processed on a desktop server, bypassing device limitations. 

Microsoft has been at the forefront of the AI race for quite some time. The Redmond-based company’s billions of dollars in investment in OpenAI, its numbers of AI models, and the Copilot AI tool in Windows 11/10 and mobile devices are just to name a few.