AI Generates Games without Game Engines

Google research scientists have published a paper detailing how their artificial intelligence (AI) "GameNGen" creates a gaming experience similar to Doom through neural networks.

These four scientists - Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter - combine Stability AI's image generation AI "Stable Diffusion" with input detection to process previously generated frames and produce new frames (i.e., parts of the gameplay/levels) to ensure coherent content.

This means that you are not playing actual levels of Doom on GameNGen. The model uses its understanding of Doom to create similar game levels, much like fan-made mods for other games. As players play, the model dynamically constructs levels around them.

GameNGen's Doom can be played like a "proper" video game, with weapons, enemies, locked doors, and health pickups, but it is not perfect. In a demo, the protagonist of Doom jumps into a toxic pool, but their health remains at 4% without dying. Nevertheless, it is still very impressive. According to the paper, "human raters performed only slightly better than random guessing in distinguishing between actual game segments and simulated ones."

How does GameNGen do it?

To make AI create Doom, a large amount of data is needed for effective prediction and meaningful content generation. The scientists realized that collecting enough game data from human players to train the model is impractical, so they first trained an AI agent to play Doom at different difficulty levels, simulating players of varying skill levels to capture the gameplay of Doom as comprehensively as possible.

Gaussian noise loop and image coherence

To address defects in image generation that cause accuracy and coherence to decline over time, the scientists go beyond typical single-frame conditioning by incorporating previous frames as feedback into the training of new frames, combining user input. They then introduce Gaussian noise (a type of random electronic noise) to perturb these frames and use another network to restore these perturbed frames. This ensures a self-correcting loop that allows gameplay to continue for longer periods without this approach.

Currently, the generator has only about three seconds of memory, so it does not know what you did a minute ago.

This means that managing things with persistent knowledge in the game, such as acquired keys, is difficult and a key obstacle that scientists will continue to overcome.

Although still imperfect, GameNGen represents a significant advancement in the field of AI-generated games and engines. Additionally, it contributes to the vast category of games where "someone put Doom on [blank]," which has recently expanded to include lighthearted and humorous Doom clones like "someone put Doom in Fortnite."