World–Action Modeling for Games

GameWAM

A World Action Model for Video Games

Joint future-video and native-action training, with action-only online control from realized visual context.

Yuncheng Guo1 Zhanqiu Zhang2,† Yiwen Guo3,† Weijia Li4,†
1Fudan University 2LIGHTSPEED 3Independent Researcher 4Tsinghua SIGS †Corresponding authors
View paper View code Watch demo
GameWAM demo reel Minecraft + ViZDoom · 06:18
01 — Overview

Joint learning for native control.

Future-video prediction provides training supervision; online inference generates native actions without future-video denoising.

Overview of GameWAM's joint world–action training, action-only online inference, block-cycle control, and evaluation on gameplay and GUI tasks.
02 — Method

Two training streams.
One action path online.

GameWAM architecture with parallel Video and Action DiTs for joint training, a timestep-wise interaction router, hierarchical visual history, and action-only online inference.
GameWAM architecture. Joint world–action training (left) and action-only closed-loop inference with cached realized visual context (right); future video is not generated online.
Architecture

Parallel generative processes

Parallel Video and Action DiTs use modality-specific prediction paths.

Interaction

Mode-specific action heads

Gameplay and GUI interaction use separate action-flow predictions and normalization statistics.

Context

Hierarchical visual history

Cycle-local context and compressed cross-cycle visual history span replanning cycles.

Block-cycle control

Look further.
Commit less.

Execute the first 8 actions of each 16-action plan, discard the suffix, observe the new state, and replan.

16predict
8execute
Block-cycle schedule showing overlapping prediction windows, short execution prefixes, cycle-level memory, and hierarchical history.
03 — Results

Closed-loop evaluation in Minecraft and ViZDoom.

Closed-loop evaluation covers more than 800 MCU tasks and a four-map ViZDoom suite.

50.7% MCU Mini
Average ASR 10 runs per task
46.6% MCU All
Average ASR 5 runs per task
12.51 Hz Online execution
frequency Mean on one NVIDIA H200
Minecraft MCU benchmark

Embodied, GUI, and combat task comparison.

Average native interaction steps over successful episodes and ASR by MCU task category
Model Game PT. Embodied tasks GUI tasks Combat tasks Average
Steps ↓Mini ↑All ↑ Steps ↓Mini ↑All ↑ Steps ↓Mini ↑All ↑ Mini ↑All ↑
VPT×37710.16.03980.70.83963.63.64.83.5
STEVE-1×3848.48.03910.03.23954.93.94.45.0
ROCKET-1×39219.218.9—0.00.032029.827.916.315.6
JARVIS-VLA×30531.030.033925.325.135218.318.524.924.5
LatentHA×36327.324.43933.53.03718.28.513.012.0
MotionHA×33631.627.4—0.00.03929.14.313.610.6
GroundingHA×29039.737.13803.76.734628.226.523.923.4
SkillHA×36513.811.33973.46.33933.46.56.98.0
TextVLA×32123.927.029114.025.831727.110.021.720.9
OpenHA×28737.030.131433.332.531640.031.936.831.5
Game-TARS✓373—50.4406—39.1372—38.1—42.5
GameWAM×13870.047.515543.060.020339.032.250.746.6

Steps ↓ Average native interaction steps over successful episodes.

ASR Mini / All ↑ Average success rate using 10 / 5 runs per task.

Game PT. Large-scale policy or continual pretraining on interaction data spanning many game environments; single-game training and generic foundation-model pretraining are excluded.

— Not reported.

Average-reward comparison on four ViZDoom maps, with 50 episodes evaluated per map.
ViZDoom evaluation. Average episode reward across four maps, with 50 episodes evaluated per map.
Design comparison

Modality-decoupled masking.

In the matched comparison, overall MCU Mini/All ASR is 50.7/46.6 versus 46.3/39.6, with mean execution frequency of 12.51 versus 8.12 Hz.

Matched comparison of modality-decoupled and joint current-block masking
MaskMCU Mini ↑MCU All ↑Frequency ↑
Modality-decoupled50.746.612.51 Hz
Joint video–action46.339.68.12 Hz
04 — Rollouts

Representative closed-loop rollouts.

Selected Minecraft and ViZDoom trajectories are shown as qualitative diagnostics rather than additional benchmark results.

Minecraft rollout in which the white bed is absent from the initial view; GameWAM explores nearby, enters a structure, observes the target, and completes the mining interaction.
Minecraft · Representative rollout

Mine the white bed

The white bed is absent from the initial observation. GameWAM explores the nearby environment through native movement and viewpoint adjustment, enters a structure where the target becomes visible, and then completes the mining interaction. This example is interpreted as closed-loop local exploration rather than evidence of an explicit search algorithm or symbolic planner.

Minecraft rollout showing GameWAM aligning with an obsidian block and maintaining mining across successive replanning steps until it breaks.
Minecraft · Representative rollout

Collect obsidian for crafting

Obtaining obsidian requires a sustained mining action rather than a brief click. After approaching and aligning with the target block, GameWAM maintains the required behavior across successive replanning steps until the block breaks. The sequence illustrates temporal consistency of native-action control during a long interaction.

Minecraft crafting rollout in which GameWAM makes an intermediate placement error, observes the resulting GUI state, adjusts the grid, and completes the dispenser.
Minecraft · Representative rollout

Create a dispenser

The initial sequence includes an intermediate placement error that leaves the crafting grid incorrect. After the resulting GUI state is observed, subsequent actions modify the placement and construct the valid recipe. The rollout illustrates recovery through repeated closed-loop observation and replanning, not a perfectly correct open-loop sequence.

ViZDoom Battle 1 rollout showing navigation, reactions to newly visible enemies, view adjustment, and repeated engagement.
ViZDoom · Representative rollout 01

Battle 1

The rollout alternates between navigation and combat as enemies enter the field of view. GameWAM changes orientation to bring nearby threats into the firing direction, engages them, and continues moving through the environment. Movement, target acquisition, and firing are repeatedly recomputed from newly realized observations.

ViZDoom Battle 2 rollout in which GameWAM navigates and fights through the maze, collects a health pack after taking damage, and resumes combat.
ViZDoom · Representative rollout 02

Battle 2

Walls and corridors repeatedly change which enemies and resources are visible. After taking damage during combat, GameWAM approaches and collects a health pack, recovers health, then continues through the maze and resumes combat. Subsequent actions are conditioned on the realized change in agent state.

ViZDoom Defend the Center rollout showing repeated orientation changes and engagement of threats approaching from different directions.
ViZDoom · Representative rollout 03

Defend the center

Threats are not restricted to one frontal direction. From the central position, GameWAM repeatedly changes view orientation, redirects its aim between different parts of the arena, and fires as enemies enter actionable views. The example emphasizes target switching rather than pursuit through the environment.

ViZDoom Defend the Line rollout showing repeated acquisition of approaching enemies, horizontal aim adjustment, and firing as the active threat changes.
ViZDoom · Representative rollout 04

Defend the line

Enemies approach from the front under constrained movement. GameWAM repeatedly acquires approaching enemies, adjusts horizontal aim, and fires as the active threat changes. The sequence illustrates reactive closed-loop control under the frontal-defense objective.

05 — Analysis

Low-Frequency Action
Source Imprinting.

Under fixed conditioning, interventions on low-temporal-frequency action-source components causally change corresponding coarse generated camera-motion components.

When one sampled source is reused across replanning steps, its low-frequency bias can accumulate into persistent turning. Resampling at each step largely removes the episode-level failure pattern, but not the underlying source sensitivity.

0.890Yaw-DCT0 within-condition
source/action correlation
94.8%Yaw-DCT0 donor-follow rate
after replacing modes 0–2
99.25%Yaw-DCT0 variance removed
after zeroing modes 0–2
Controlled LASI experiments measuring fixed-condition association, low-frequency source replacement, and source zeroing.
Controlled evidence. Within-condition association, donor replacement, and zeroing quantify the effect of sampled low-frequency source components on generated camera-action coefficients.
LASI frequency-response and amplification analyses comparing low- and high-frequency source effects.
Frequency selectivity and amplification. Across the tested perturbations, low-frequency modes produce larger responses, and source gain increases along the iterative model path.
Closed-loop LASI analysis measuring alignment between action-source and executed-action low-frequency DCT coefficients across replanning steps.
Closed-loop alignment. Episode-adjusted source/action association is strongest at the correct source–replanning alignment across the analyzed trajectories.

World modeling meets
native action.

Explore the code