ai-plays-pokemon

Put an AI in front of Pokémon on a Nintendo DS emulator and watch it play, live (and take back the controls whenever you want). The same game, the same "eyes" and the same controller for every AI, so they can be compared:

• Jev, TypeSafe's "System One" decision model: it doesn't generate text, it picks one option from a closed list and returns a calibrated probability for every option, in ~150 ms;
• LLMs through Koog: OpenAI, Anthropic, OpenRouter (Gemini, Mistral, DeepSeek, Llama...) or a local model with Ollama. They reason before each choice and can keep notes;
• Claude with your Claude Code login (e.g. a Claude subscription, no API key), through claude -p;
• any external agent through MCP: the app runs an MCP server and the agent (e.g. Claude Code) plays with tools.

What every AI gets:

• eyes: we read the game's RAM (map around the player as text, position, dialogue, menus, party, battle...) and describe it as JSON: no AI sees pixels;
• memory: what each of its actions changed, where it has been, what it read, the map it explored, its own notes;
• hands, depending on the mode you pick:
  • Pure: the controller, nothing more: "which button do I press?" among the 12 DS buttons (+ wait);
  • Assisted: the buttons plus actions carried out by code ("walk to the stairs and take them", "talk to Mom", "choose YES", "go to Route 30", "surf across"): the AI still decides what to do, code does the walking (across floors and maps, with Surf, Cut, Strength, Waterfall... used on the way);
  • Hybrid: assisted actions picked by a fast decision model (Jev), with an LLM planner called when it hesitates, loops, or periodically, whose goal the fast model then follows.

Every variant is an option in the app (mode, knowledge level, timing, sequences, reasoning, notes, explored map...), and each run records milestones (new places, story progress) with decisions, presses, time, tokens and cost in ~/.ai-plays-pokemon/runs/, so configurations can be compared.

Quick start

You need Java 21 (for example Temurin 21) and a Pokémon HeartGold (USA) ROM that you dumped yourself (no ROM is provided).

1. Download ai-plays-pokemon-<version>-universal.jar from the latest release. The same jar runs on macOS, Linux and Windows (x86-64 and Apple Silicon).

2. Run it, with the path to your ROM (or without it: the app asks for the ROM with a file picker and remembers it):

   java -jar ai-plays-pokemon-<version>-universal.jar "/path/to/Pokemon - HeartGold Version (USA).nds"
   

   On first launch it downloads the DeSmuME emulator core for your platform (see below).

3. Let an AI play:

   • an agent through MCP (e.g. Claude Code or Codex): keep the MCP tab selected (the server starts on port 3333), then connect the agent, e.g. for Claude Code claude mcp add --transport http pokemon http://localhost:3333/mcp, and ask it to play, for example: "Play Pokémon HeartGold through the pokemon MCP tools. Start a new game and beat the Pokémon League. Keep notes of your progress in NOTES.md." For any other MCP client, the server's URL is http://localhost:3333/mcp (streamable HTTP);
   • an LLM or Jev from the app: choose it in the panel on the right, paste its API key if needed, and click ▶ Let … play.

Running from source

Requirements: JDK 21 and the ROM as above. Then an API key for the AI you want to try (or Ollama for a local model, e.g. ollama pull gemma4:26b: a mixture-of-experts model, 4B parameters active, so fast enough on a laptop; small local models play poorly, cloud models play much better).

./gradlew run --args="/path/to/Pokemon - HeartGold Version (USA).nds"

Gradle downloads the libraries from Maven Central (libretro-kmp and pokemon-client included). While the app depends on a -SNAPSHOT of pokemon-client (see gradle/libs.versions.toml), clone that repository too and run ./gradlew publishToMavenLocal in it first. To build the jar that runs on every platform yourself: ./gradlew packageUberJarForCurrentOS -Puniversal, written to build/compose/jars/ (the name carries the platform it was built on, but -Puniversal adds the other platforms' natives).

Files and the emulator core

Bring the ROM yourself. Its file name names the save: heartgold-us.nds → heartgold-us.sav.

The emulator core is the only other file. It is downloaded on first launch from the libretro buildbot. Or copy it yourself, built for the target platform, into the data directory's cores/ folder. The data directory is ~/.ai-plays-pokemon, i.e. %USERPROFILE%\.ai-plays-pokemon on Windows.

| Platform | Core file (DeSmuME) | Buildbot folder | |-------------------------------|--------------------------|----------------------------------------| | macOS (Apple Silicon / Intel) | desmume_libretro.dylib | apple/osx/arm64 / apple/osx/x86_64 | | Linux x86-64 | desmume_libretro.so | linux/x86_64 | | Windows x86-64 | desmume_libretro.dll | windows/x86_64 |

The URL is https://buildbot.libretro.com/nightly/<folder>/latest/desmume_libretro.<ext>.zip. No BIOS or firmware is needed.

The core is checked against a pinned SHA-256 where one is pinned. Today that is only macOS Apple Silicon, the build we tested; other platforms take whatever build is "latest". If the buildbot has moved on, the app stops with "Unexpected … build". Then either:

• copy the core from a computer where it works;
• or test the new build and update the hash in LibretroCoreSpec.

To carry a game over, copy saves/<rom name>.sav from the data directory. It is converted for the core automatically.

Using the app

In the window: the game on the left; on the right, choose the AI (Jev, or an LLM provider + model id), paste its API key if needed, pick the mode and options, and ▶ Let … play starts the autonomous loop. You can see exactly what the AI receives ("What the AI sees" → Show JSON), and for each decision Jev's probability for every option, or the LLM's reasoning and note.

Claude without an API key: choose LLM → Claude (Claude Code login) with a model like sonnet or opus. Each decision runs claude -p (a few seconds of overhead) and counts towards your Claude plan's usage.

An external agent through MCP: select the MCP tab (it starts the server; the port can be changed there), then connect the agent (see the quick start) and ask it to play. It gets four tools, always the same:

• get_state: the screen (with the ids of its entries), messages and events since the last call, the team, the battle (with the estimated effectiveness of each move at the Pokédex knowledge level), a text map of the surroundings with the ids of exits, people and signs, and the actions possible now with their valid values;
• act: one typed action ({"type": "buy", "items": [{"item": "Poke Ball", "quantity": 5}]}, {"type": "go_to", "map": "Route 30"}, {"type": "attack", "move": "move:85"}, pc, step, set_options, soft_reset...), optionally followed by up to 8 more (then), a note to itself and a short reasoning; a refused action always says why (typed error). The answer is compact by default (only what changed: messages, screen, position, team or "team unchanged"); detail: "full" returns the whole state like get_state;
• lookup: game knowledge from the ROM (species, moves, items, types, learnsets, TMs), within the knowledge level;
• screenshot: both screens, for anything the state doesn't describe.

The agent keeps its own context between calls, and the game is frozen while it thinks.

| Keys | Action | Keys | Action | |-----------|----------------|-------|--------------------------------| | Arrows | D-pad | Space | AI play / pause | | X / Z | A / B | P | Pause / resume the emulator | | S / A | X / Y | F | Fast forward | | Q / W | L / R | M | Mute | | Enter / ⌫ | Start / Select | F1–F4 | Load state (Shift+F1–F4: save) | | Mouse | Touch screen | F12 | Save a RAM snapshot (debug) |

You can play at the same time as the AI: the emulator merges both inputs.

Configuration

Everything can be set in the app; settings are saved in ~/.ai-plays-pokemon/config.properties (next to the downloaded core, in-game saves and save states). Environment variables take precedence:

| Environment variable | Config key | Default | |-------------------------------------------------------------------------------|-------------------------|----------------------------------------| | POKEMON_ROM (or first argument) | rom | asked with a file picker | | DECISION_BACKEND (mcp / llm / jev) | backend | mcp | | TYPESAFE_API_KEY | typesafe.apiKey | entered in the app | | JEV_MODEL | typesafe.model | jev-latest | | JEV_ENDPOINT | typesafe.endpoint | https://api.typesafe.ai/v1/systemone | | LLM_PROVIDER (openai, anthropic, openrouter, ollama, claude_code) | llm.provider | ollama | | LLM_MODEL | llm.<provider>.model | a default per provider | | LLM_THINKING (Ollama reasoning models think first) | llm.thinking | false | | MCP_PORT | mcp.port | 3333 | | EMULATOR_CORE (desmume / melonds) | emulator.core | desmume | | START_MUTED | start.muted | false | | OPENAI_API_KEY, ANTHROPIC_API_KEY, OPENROUTER_API_KEY | llm.<provider>.apiKey | entered in the app | | AI_PLAYS_POKEMON_DATA_DIR | — | ~/.ai-plays-pokemon |

Jev without the cloud. Jev itself is only served by TypeSafe (no public weights). JEV_ENDPOINT can point to any server implementing the same POST /v1/systemone protocol, e.g. an open-weight reproduction of Jev served locally.

How it works

 ┌──────────────── app (me.nathanfallet.aiplayspokemon) ─────────────────┐
 │ ui: EmulatorScreen · ControlPanel        mcp: GameMcpServer (4 tools)  │
 │ agent: GameSession · PokemonPlayer       decision: Jev · LLMs · claude │
 │ emulator: ConsoleHost (one console thread, drivers: idle/human/agent)  │
 │   └ LibretroConsole ── libretro-kmp ── DeSmuME / melonDS core          │
 └───────────────┬───────────────────────────────────────────────────────┘
                 │ ConsolePort (step, read RAM, frames, save states)
 ┌───────────────▼──── pokemon-cl