NitroGen

NitroGen is an open-source unified vision-to-action foundation model developed by NVIDIA researchers in collaboration with Stanford, Caltech, UChicago, UT Austin, and others.

It enables generalist gaming agents to play a wide variety of commercial video games directly from raw RGB video frames by predicting gamepad actions through large-scale behavior cloning.

Trained on 40,000 hours of internet-scale gameplay videos spanning more than 1,000 diverse titles, it learns to handle action, platformer, racing, RPG, battle royale, 2D/3D games, and more without game-specific APIs or reward signals.

The model takes 256×256 RGB inputs and outputs a 21×16 action tensor (two continuous 2D joystick vectors plus 17 binary buttons).

Architecture combines a SigLip2 Vision Transformer backbone with a Diffusion Matching Transformer (DiT) for action prediction.

It demonstrates strong zero-shot competence in complex tasks like combat, navigation, high-precision platforming, and exploration, with up to 52 percent relative improvement in task success rates when fine-tuned on unseen games compared to scratch-trained models.

Released in December 2025 with full weights on Hugging Face, dataset, universal Gymnasium API simulator for commercial games, evaluation suite, and code on GitHub under NVIDIA’s non-commercial license (backbone Apache 2.0).

Ideal for advancing embodied AI research, game testing automation, NPC behavior training, and transferring skills to robotics or real-world simulation.

As an open model, it promotes progress in generalist agents beyond narrow RL-trained bots.