AI Learns to Simulate Super Mario Bros. with a Single RTX 4090
A new research project has demonstrated the potential of AI invideo game development by using a single RTX 4090 graphics card to simulate the classic game Super Mario Bros. using video generation techniques.
The project,known as MarioVGG, was led by GitHub users Ernie Chew and Brian Lim and published by Virtuals Protocol, a cryptocurrency-focused AI company. It utilizes machine learning toanalyze game video and input data, inferring the game’s physical rules and dynamics.
The researchers trained their model on a dataset of 280 levels, encompassing over 737,000 frames of gameplay. They focusedon two potential inputs: running right and running right while jumping. After 48 hours of training, the model could generate new video frame sequences from a static initial game image and text input.
MarioVGG employs a text-to-video generation approach, learning from game footage to produce corresponding game sequences based on player text inputs like run or jump. The model receives an initial frame and a text description of the desired action, then generates a series of frames visually depicting the action.
To improve efficiency, the researchers reduced the output frameresolution from the original NES’s 256×240 to 64×48 or 128×96. They also compressed 35 frames of video time into just seven generated frames, spaced evenly, resulting in a rougher video quality than the actual game.
Despite these methods,MarioVGG faces challenges in achieving near real-time video generation. With a single RTX 4090, it takes approximately six seconds to generate a six-frame video sequence, lasting just over half a second, even at an extremely limited frame rate.
The researchers acknowledge this is neither practical nor user-friendlyfor interactive video games, but they hope future optimizations will improve the speed.
Despite its limitations, MarioVGG exhibits impressive features. It learns the game’s physics, including Mario’s gravity effect when falling from a cliff and stopping when encountering obstacles, solely from training data.
While primarily focused on simulating Mario’sactions, MarioVGG can also effectively generate new obstacles for Mario that are consistent with the game’s graphical language. It can even create unique Mario levels, with only the first frame input coming from a real game state.
However, MarioVGG faces significant challenges. The researchers observed that the input action textisn’t always followed by the model. Visual issues like Mario passing through obstacles and enemies, color flickering, shrinking/enlarging between frames, or completely disappearing and reappearing across multiple frames can occur.
The model lacks game logic, continuing to render video frames even after Mario dies instead of returning to the last checkpoint. Additionally, level and object generation is uncontrolled, relying solely on procedural generation.
The MarioVGG project opens new possibilities for future game development, showcasing how AI can learn from video data to understand and recreate complex game mechanics. The researchers hope this represents a first step towards producing and showcasing reliable and controllable video game generators,potentially even replacing game development and game engines entirely with video generation models in the future.
However, achieving this vision requires addressing challenges like real-time performance, visual quality, and game logic consistency. For example, determining the termination state in a generated game remains an open research question.
Nevertheless, considering MarioVGG’s impressive results with a relatively small training dataset and a single consumer-grade GPU, the project is a significant achievement. Virtuals Protocols optimistically tweeted, The age of infinite interactive worlds is here.
While video generation for game creation cannot fully replace traditional game development methods yet, it injects newenergy into the gaming industry, offering a glimpse into an AI-driven gaming future. These new projects provide fresh perspectives on game design and development, demonstrating how AI can learn from video data to understand and recreate complex game mechanics.
In the future, we may see more AI-assisted game development tools, or even entirely AI-generated game worlds, as technology advances.
Views: 0