GPT4 plays DOOM, forgets about in-game enemies, and is happy to grab a gun and start blasting

A recent research paper by Adrian de Wynter, a researcher at the University of York in England, has explored the capabilities of GPT4 in playing the classic first-person shooter game DOOM.

The paper, titled “Will GPT-4 Run DOOM?” finds out whether GPT4 can play the game without prior training. To test this, de Wynter designed a system that uses GPT-4V, a multimodal variant of GPT4 that can accept images as input, to capture screenshots from the game engine and return structured descriptions of the game state. 

This information is then fed into an Agent model that makes decisions based on the visual input and previous history, translating its responses into keystroke commands that the game engine can understand. 

In easier words, the system takes screenshots of the game and uses them to figure out what’s going on. Then, it uses another part of the system, called the Agent model, to make decisions about what to do next in the game. The Agent model turns its decisions into commands that the game can understand, like which keys to press.

While the system showed promise in playing DOOM, it also revealed some limitations. One of the main issues was 

• GPT4’s lack of object permanence means it forgets about in-game enemies when they go off-screen. 
• Additionally, GPT4 struggled to reason and provide clear explanations for its actions, often providing incorrect information.

Despite these limitations, de Wynter considers it remarkable that GPT4 is capable of playing DOOM without prior training.