Google explains how its SIMA AI model can play video games

The researchers at Google DeepMind have published a new report on the Scalable, Instructable, Multiworld Agent (SIMA) AI model that they have developed. This new model is all about video games, but not in the winning way you think.

The firm went into some pretty deep details about how its researchers trained the model to play video games naturally just like you and I might. This AI model, per the company, isn’t really about individual games, but rather an instructable game-playing AI agent. Or, as you might think of it, another “player” in your game.

Google partnered with leading game developers on the project to train SIMA in different types of gaming worlds so it can learn how to follow instructions properly. It worked with the folks behind No Man’s Sky, Teardown, and also six other developers. Google even built four research environments with Unity, and a special one called the Construction Lab to aid in its research and how the AI model can understand things like manipulating objects and navigating the world. SIMA doesn’t need an API to work or even access to the game’s source code.

Some other goals of SIMA include learning how language ties in with gameplay, perceiving and understanding a variety of environments, and then taking action to achieve an instructed goal. SIMA uses the images on the screen and user instructions as well as keyboard and mouse inputs for this task. Google trained SIMA on 600 skills like navigation, climbing ladders, and opening maps, all of which can be done in 10 seconds. These include things like “finding water” or “chopping trees” in Valaheim.

There’s still work to be done, though. Google is planning to train SIMA on higher-level tasks that need more planning.