Case study / reinforcement learning / NES

Teaching PPO to finish 1942

I wanted to know whether a policy could finish the entire NES game from a fresh power-on—not whether I could disguise a controller recording as AI.

32/32stages

64,178policy decisions

0overrides

71:18uninterrupted

01 — ENVIRONMENT

The policy sees memory, not pixels.

Stable-Retro runs the NES core. Each observation contains four frames of the full 2 KB system RAM plus the previously executed action. That exposes the real game state without asking a vision model to rediscover object tables, stage progress, lives, and bullets from a 240 × 224 image.

The policy chooses from 12 controller actions on every decision: movement combined with fire, roll, no-op, and START. START matters. In the final evaluator, even beginning the game is the policy’s responsibility.

RuntimeStable-Retro + PufferLib

Observation4 × full 2 KB RAM + action history

PolicyPyTorch PPO on CUDA

Action space12 learned controller actions

02 — REWARD DESIGN

Progress is the objective. Score is a tie-breaker.

Early reward experiments made it too easy to look busy. An agent could collect score, shoot enemies, or farm a checkpoint without solving the course. The final reward puts forward stage progress first, gives weak credit for score and kills, penalizes death, and reserves a large terminal reward for the game’s real ending path.

Course progress primary Score delta weak Kill delta weak Death −20 Full completion +10,000

Completion is not inferred from a score threshold. Reverse-engineered RAM state tracks the stage-active latch and final scroll state so a game-over screen cannot be mistaken for finishing the game.

03 — TRAINING

The long horizon was only half the problem.

A full run takes more than sixty-four thousand decisions. Fresh-start PPO alone had a poor learning signal that far from the ending, so I used a completed planner trajectory as training data: imitation initialization and reverse-curriculum snapshots, never as a controller during evaluation.

Failure 01

Starting only from power-on

The useful reward was too far away. Curriculum snapshots moved the training window backward from later stages toward a fresh start.

Failure 02

Forgetting earlier behavior

PPO improvement in one part of the game could damage another. Demonstration rehearsal reduced catastrophic forgetting during training.

Failure 03

One actor for every situation

The shared actor struggled with a specific late-game recovery. A small gated recovery branch handled that narrow state without rewriting the base policy.

The final strict checkpoint then received 524,288 PPO environment steps and 4,096 optimizer updates for every policy component with imitation losses disabled. A parameter-delta audit confirms that the base and recovery components were actually updated during that PPO-only phase.

04 — STRICT PROOF

The evaluator is designed to reject convenient explanations.

Strict mode starts at a real power-on state and loads only the trained checkpoint. It rejects curriculum starts, startup jitter, action perturbations, action overrides, missing PPO update records, a START press not selected by the policy, and anything short of all 32 stages.

✓ Fresh power-on
✓ Policy selected START
✓ No planner or demonstration loaded
✓ No action tape or recovery controller
✓ Zero action perturbations
✓ Zero action overrides

Checkpoint SHA-256 292b456c90d66a3fa80fa15312955aaaf3b677a912c8f0a4043a91be25c67991

Policy trace042d754e749d…

Final score660,050

The repository is private for now. The full run and evaluation details above are public.

Back to projects