Teaching a small, open LLM to play 1v1 Settlers of Catan with reinforcement learning — discover its weaknesses, build verifiable environments, and GRPO-train it to play measurably better.
Headline · head-to-head win rate
100 games, trained vs. the untrained baseline — a coin-flip baseline is 50%.
Trained = the placement-RL checkpoint (our best). A coin-flip baseline wins ~50% by definition. Full-chain (placement+maritime+build) lands ~70% — the later envs sharpened isolated decisions but slightly hurt end-to-end play, so the placement-only checkpoint is the one we ship. (300-turn cap.)
Watch the games
Step through any game on a rendered board — the trained model beating the base, or the base model playing itself. Click a game to open its replay.
Loading games…
Fully autonomous · recursive self-improvement
No human in the loop. The system plays, finds its own weaknesses, builds the reward that fixes them, trains, and verifies on unseen boards — then starts over on a stronger model. Each pass compounds: recursive self-improvement that just keeps running.
Run 1v1 self-play with the base open model to generate full game transcripts.
Mine the transcripts for systematic mistakes: poor openings, hoarding, aimless trades.
Turn each weakness into a small RL env with a deterministic, programmatic reward.
Group-relative policy optimization on Qwen3.5 8B (thinking off) via HUD / Tinker, our graders as reward.
Score before vs after on held-out, unseen boards to confirm real generalization.
We left it running
Left the fully-autonomous loop running ~2 hours (12 rounds, no human in the loop). It played, mined its own weaknesses, wrote new reward envs, and kept only checkpoints that beat the last — lifting win-rate vs base from a coin flip to 56%, with held-out decision quality climbing too.
Headline results
Each environment evaluated on a disjoint set of unseen boards. The base model vs. the GRPO-trained checkpoint, both forced to answer directly so we measure pick quality, not formatting.
Held-out = scored on boards the model never trained on. Placement and build generalize cleanly; maritime learned to stop over-trading (lower trade-rate is the intended direction here).
Training
Per-step reward on the training boards for each environment during the final GRPO run. All three show a clear climb.
Session
The whole overnight GRPO session across all three environments.
Explore
Interactive static viewers — open in a new tab.
Step through trained-model-vs-base games on rendered Catan boards.
Open ↗📊Head-to-head win rates and per-decision held-out scores.
Open ↗📍How opening settlement spots are scored against the optimal pick.
Open ↗📖The exact 1v1 Catan ruleset the engine and model play under.
Open ↗Live play isn't deployed on this site — it needs the inference gateway and a local game server. To play against the trained model yourself, run it locally from the repo: