Two things live in this repository, and honesty requires introducing them separately.
The first is a perception stack for playing NES games the way a person does: seeing the screen, hearing the sound, and nothing else. The emulator's memory is closed to the policy — it is read only by training and evaluation, as ground truth for checking what the agent read off the screen.
The second is a measurement journal — and the journal's verdict, as of now, is that the strongest player in the repository is not that stack. It is a planner that openly cheats: it uses the emulator itself as a world model. That verdict was reached by paired experiments, survived every attempt to overturn it, and is the most interesting thing here.
Both of those are means. The end, written down here before it is reached: the model has never seen the game, and before it starts it is handed the documentation that shipped in the box. Manual-assisted zero-shot play. Not a prior tuned by whoever is running the experiment, not addresses read out of the emulator's memory — the manual, the way a person gets it, in the box with the cartridge.
That is a fair description of how anyone meets a game they have not played, and it is a harder test than anything below. Every number in this file was measured on a game that had already been watched for weeks.
Where that stands, so the target is not mistaken for a result: the mechanism
exists and is worth something measurable — assets/priors/ is read by the
planner, and the base's first room opens 17 times in 32 with it. Since
16 September the prior's source names a scan of the actual manual rather than
recollection of it, and the scope of each fact came from the manual's own
control pages rather than from a guess (docs/manuals.md). What has not
happened is the test itself: a game nobody here has tuned for, its manual, and
a number. Until that runs, this is a direction, not an achievement.
scripts/experiments/oracle_mpc.py: at each decision the planner saves the
console state, plays out a handful of 48-frame button templates plus the
policy's own plan, continues each with the reactive policy for 96 frames,
averages 2–4 such futures, commits 16 frames of the winner, and repeats.
Progress comes from the game's own position counter; a plan that dies is not
compared on distance.
The same planner, the same handful of templates, no per-game tuning beyond a position address and a boot sequence — and, since 3 September, not even the address:
Privileged track — the planner reads the emulator and rewinds it. These numbers are what a search with a perfect world model can reach; they are not what a policy plays.
| environment | paired gain over the policy | seeds |
|---|---|---|
| SMB 1-1 | +3828 (and 20/32 full clears at 4 draws) | 32 |
| SMB 2-1 … 6-1 | +509 … +2483, every CI clear of zero | 32 each |
| Contra (J) | +2270; level 1 cleared from power-on in 12/32 with five composed templates and a damage term on the game's object tables | 32 |
| Contra (U) | +2134 | 32 |
| Super C | +1210 | 32 |
| Rush'n Attack | +145 [+124, +166], 32/32 — position and templates found by scanning, zero hand-written addresses | 32 |
| Contra (J), the base | first room opened in 17/32 from a room state, with the manual's two notes (what to destroy, where the door is) and every other input from scans | 32 |
Every per-game input the planner uses in Contra is now found by a scan
rather than typed: which buttons do what and whether fire is tapped or held
(controllability.py), the templates built from that (scan_templates),
the camera or an 8-bit scroll (find_camera.py), the object tables that
hold hit points (object_tables.py), the player's own sprites
(hero_tiles.py), and the stage's section counter from the fades between
rooms (section_scan.py). What stays human is the manual: a small file per
game saying what the game wants destroyed and where the exit is
(assets/priors/), which is what a person reads before playing too.
Read the caveats before the numbers impress you: the planner is privileged by construction — it looks at futures instead of predicting them; on Contra the learned policy used to be inert, so the planner's gain there is carried by the template prefixes (a clone trained on the planner's own clears does play — see below — but it is not what the planner is measured against); the journal records four retractions from these two days, each caught by a frame or a scripted check rather than by thought (a "hits" count that was a savestate's constant, a vertical term that was measuring enemy fire, a "horizon problem" and an "arithmetic problem" that were one term returning zero); and the base result is one room, not the stage — in the next room the soldier parks in a corner again.
The obvious next step — distil the planner into a network and stop paying for rollouts — was tried to the end. Distilling its choice failed with measured causes. A learner was priced in five roles (value estimator, action chooser, binary gate, DAgger student, compute allocator) and lost all five; a battery of seven input representations, from two scalars to the full privileged state, captured the same ~20% of the planner's per-decision gain; the practically useful remainder was reachable only by simulating the future. Two structural findings survived every control: monolithic 96-frame plans are toxic (half the progress, double the deaths) and a few hand-written compositions buy most of it back; and at 32 paired seeds the online noise floor certifies only planner-sized effects — finer economics is measured offline, on stored rollout matrices, under common random numbers. Distilling its successful trajectories is a different target, and that one works in part. Once the planner began clearing Contra's first level, twelve of those clears were recorded as ordinary episodes and a clone was trained on them with the same recipe as before — only the data changed. It plays from pixels and sound alone, with no planner and no rewinding.
Honest track — pixels and sound, no emulator state, no rewinding. The teacher is listed in italics for scale, not as a peer: it searches the future, the clone looks at a screen.
| Contra (J), 32 seeds | median progress | clears |
|---|---|---|
| clone trained on 57 exploration episodes | 0 | 0 / 32 |
| clone trained on 12 planner clears | 1952 | 0 / 32 |
| its teacher, the privileged planner | 3072 | 12 / 32 |
The clone's gain over the inert one is +1719 [+1516, +1895], winning every seed; it runs the bridge, jumps the water and shoots, and it clears nothing. Two honest notes belong with it: the knowledge is the planner's, and validation accuracy predicted none of it — the clone that plays beats its majority baseline by 5.5 points, the inert one beat its own by 56.
How many clears does that take? Eighty more planner runs produced fifty-nine, and clones trained on nested subsets of 6, 12, 24 and 59 of them are indistinguishable: −182 [−415, +34] between the ends of the ladder. The whole distance from zero to about 1900 is bought by the first handful, and the next fifty buy nothing.
Recording demonstrations where the clone actually fails is the obvious next move, and measuring it turned out to be the hard part. Letting the planner take the wheel wherever the clone stalls and training on the rescues looked worth +278 px on one training run, then −240 px on three. Playing every epoch of every run explained why: the second and third epochs add no progress at all and quadruple the spread between runs, from 340 px to 1527 px. One clone that scored 1885 after a single epoch was trained down to 563 by the third, while its loss fell the whole way. At one epoch, where the instrument is steady, six training runs put the rescues at +169 [−107, +445], positive in five of six. Zero is inside, so that stays a lead rather than a result.
Measuring properly finally paid for itself on the next question. A policy with one weight per pixel per action and nothing in between — the conv net's problem with the representation removed — reaches 985 px against the conv net's 1641, so the representation is worth +656 [+282, +1031] over six training runs each. But the arm that isolates it also settles an older question: switching off the attention loss, which supervises where the network looks, costs −353 [−526, −180], losing all six. That loss was added long ago, measured once, and never checked across training runs. It is also the steadiest arm, so the hint about where to look removes most of the run-to-run lottery as well. Meanwhile the arm that fits the demonstrations best plays worst, for the sixth time running.
The methodological lesson cost more than the experiment. The first three seeds agreed on +221, +277 and +239, and that agreement is exactly what made it look certain; three fresh seeds gave −355, +382 and +252. Three numbers that agree are a coincidence that happens often. The unit of observation here is a training run, and six of them are not enough to publish a few hundred pixels. The 0 → 1952 result clears that bar by a wide margin.
The full chronology, retractions included, is docs/experiments.md.
The original point stands as a goal: skills that carry over to games the agent has never seen, from pixels and sound alone.
Left: the game with Grad-CAM showing where the network is looking, boxes around the objects the motion tracker found, and rings where a sound was predicted to come from. The gamepad below lights up the buttons actually being pressed — here B and RIGHT, matching the 0.68 the model gives that action. Right: the action distribution, uncertainty over time, tracked objects, live conv features, remembered sprites, the sounds heard, and the agent's own running commentary.
hud read 300 1 1 1 391 is score, coins, world and timer, read off the picture
with no labels and no memory access. The game's own HUD in the same frame reads
000300, ×01, 1-1, TIME 391.
- Finds objects by motion, corrected for camera scroll, and works out which one it controls by correlating world-space velocity with button presses. Subtracting the scroll is the part that matters: when the camera follows the hero his screen velocity is nearly zero, and without the correction the agent cannot recognise itself at all.
- Reads the screen. Digits are learned with no labels whatsoever: in any counter the lowest digit's transitions form a ring 0→9, and zero is found topologically, as the glyph with a closed hole. Checked against RAM, the timer reads with correlation 1.000 and 95.8% exact matches.
- Listens. A hit, damage or a scene change is often audible before it is visible. Adding audio raises action-prediction accuracy by 7 points on 4 of 4 seeds.
- Builds a world model around the object it controls and plans 16 steps ahead over behaviour templates.
| First contact with an unseen game (Castlevania) | 700 points from instincts alone; a random agent and a transferred network both score 0 |
| Trained games | 7, action-prediction accuracy 0.70–0.94 |
| Few-shot on unseen games, 3 episodes | Gradius 0.707 → 0.787, Battletoads 0.594 → 0.671 |
| Attention inside object boxes | 12.5% → 21.5% (38.0% multi-game) against 13.8% for a uniform gaze |
| Whole model | 16.5 MB — vision, hearing and decision together |
Negative results are published beside the positive ones. Zero-shot policy transfer does not work: on an unseen game the multi-game base covers 9.7 units of distance against 126 for a random agent. Features transfer; behaviour does not. Slot attention failed to decompose NES frames into objects, because a tiled background is described just as well by positional blobs. The full log is in docs/experiments.md.
No demonstrations are required. An instinct policy — no training, just control calibration and a handful of exploration rules — plays the game itself, headless at about a thousand frames per second, and behavioural cloning learns from its episodes. Collecting a dataset for a new game takes minutes.
macOS on Apple Silicon and Linux x86_64 work out of the box; Windows goes
through WSL2, since stable-retro ships no native Windows build. A GPU is
optional — CUDA, Metal or CPU, selected automatically. Everything runs
through uv, Astral's Python package and
project manager — install it first (curl -LsSf https://astral.sh/uv/install.sh | sh
on macOS and Linux, or see its README), then:
uv sync
uv run pytest -qNo ROM images are distributed with this project. Import your own:
uv run python -m retro.import /path/to/romsTrained weights are release assets, not repository contents — seven checkpoints at 16 MB each is not what git is for:
uv run python scripts/fetch_models.py --list # see what there is
uv run python scripts/fetch_models.py # fetch it, about 105 MBYou can skip that entirely. The instinct policy needs no model at all, and training one for a new game takes about half an hour; see docs/models.md.
Then, either the launcher:
./start.shor directly:
# Watch a trained model play
uv run nes-player play --game SuperMarioBros-Nes-v0 --checkpoint runs/bc_smb_attn3 \
--window --realtime --hd --auto-start
# Let the instincts explore a new game and record what they do
uv run nes-player explore --game Gradius-Nes-v0 \
--record datasets/explore_gradius --loop --max-frames 3600
# Train on those episodes
uv run nes-player train-bc --episode datasets/explore_gradius \
--out runs/bc_gradius --audio --attn 1.0Everything is in docs/: the
specification,
architecture,
command line,
dashboard,
perception,
training,
emulation cores,
models,
ROMs,
where the manuals come from,
what a comparison of this size can resolve and the
experiment log.
Every published number lives in docs/claims.jsonl as one
machine-readable line, and recheck_claims.py recomputes all of them from the
logs in about a tenth of a second. Four claims on the experiment log have been
retracted, each caught by somebody noticing in time; the registry is so the
fifth does not depend on that.
MIT, © 2026 Ruslan Semov — see LICENSE.
Two things carry their own terms and are not distributed here: the libretro emulation cores (fceumm, nestopia, quicknes) are GPL-2.0 and are downloaded from the official libretro builds onto your machine at first use, and ROM images belong to their rights holders and are yours to supply.