ai-jam-sessions

<p align="center"> <a href="README.ja.md">日本語</a> | <a href="README.zh.md">中文</a> | <a href="README.es.md">Español</a> | <a href="README.fr.md">Français</a> | <a href="README.hi.md">हिन्दी</a> | <a href="README.it.md">Italiano</a> | <a href="README.pt-BR.md">Português (BR)</a> </p> <p align="center"> <img src="logo-banner.png" alt="AI Jam Sessions" width="520" /> </p> <p align="center"> <em>Machine Learning the Old Fashioned Way</em> </p> <p align="center"> An MCP server that teaches AI to play piano and guitar — and sing.<br/> 120 songs across 12 genres. Six sound engines. Interactive guitar tablature.<br/> A browser cockpit with vocal synthesizer. A practice journal that remembers everything. </p> <p align="center"> <a href="https://github.com/mcp-tool-shop-org/ai-jam-sessions/actions/workflows/ci.yml"><img src="https://github.com/mcp-tool-shop-org/ai-jam-sessions/actions/workflows/ci.yml/badge.svg" alt="CI"></a> <a href="https://www.npmjs.com/package/@mcptoolshop/ai-jam-sessions"><img src="https://img.shields.io/npm/v/@mcptoolshop/ai-jam-sessions" alt="npm"></a> <a href="https://github.com/mcp-tool-shop-org/ai-jam-sessions"><img src="https://img.shields.io/badge/songs-120_across_12_genres-blue" alt="Songs"></a> <a href="https://github.com/mcp-tool-shop-org/ai-jam-sessions"><img src="https://img.shields.io/badge/annotated-120%2F120-green" alt="Ready"></a> <a href="datasets/jam-actions-v0-public/README.md"><img src="https://img.shields.io/badge/dataset-jam--actions--v0%20(115_records)-8b5cf6" alt="Training dataset"></a> <a href="https://doi.org/10.5281/zenodo.20279918"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.20279918.svg" alt="DOI"></a> </p>

---

What is this?

A piano and guitar that AI learns to play. Not a synthesizer, not a MIDI library — a teaching instrument.

An LLM can read and write text, but it can't experience music the way we do. No ears, no fingers, no muscle memory. AI Jam Sessions closes that gap by giving the model senses it can actually use:

• Reading — real MIDI sheet music with deep musical annotations. Not hand-written approximations — parsed, analyzed, and explained.
• Hearing — six audio engines (oscillator piano, sample piano, vocal samples, physical vocal tract, additive vocal synth, physically-modeled guitar) that play through your speakers, so the humans in the room become the AI's ears.
• Seeing — a piano roll that renders what was played as SVG the model can read back and verify. An interactive guitar tablature editor. A browser cockpit with a visual keyboard, dual-mode note editor, and tuning lab.
• Remembering — a practice journal that persists across sessions, so learning compounds over time.
• Singing — vocal tract synthesis with 20 voice presets, from operatic soprano to electronic choir. Sing-along mode with solfege, contour, and syllable narration.

Every one of the 120 songs is now fully annotated — historical context, bar-by-bar structural analysis, key moments, teaching goals, and performance tips, in all 12 genres. An earlier version of this README said the raw songs were "waiting for the AI to absorb the patterns, play the music, and write its own annotations." That is exactly what happened: the annotations were written by AI against a deterministic per-song analysis (chords, repetition structure, section boundaries, content-verified keys), gated by a quality rubric, and adversarially fact-checked claim by claim — measure numbers, chord windows, and structural counts all verified against the actual MIDI before anything shipped.

Out of this same work, we also publish jam-actions-v0 — a public dataset of 115 multi-turn MCP tool-use traces over real classical piano. It teaches LLMs to do grounded tool-use over symbolic music, not just text generation, and ships with a 7-axis release gate that distinguishes "passing on evidence" from "passing because the task is trivial." See Training Dataset below for the full story.

The Piano Roll

The piano roll is how the AI sees music. It renders any song as SVG — blue for right hand, coral for left, with beat grids, dynamics, and measure boundaries:

<p align="center"> <img src="docs/fur-elise-m1-8.svg" alt="Piano roll of Fur Elise measures 1-8, showing right hand (blue) and left hand (coral) notes" width="100%" /> </p> <p align="center"><em>Für Elise, measures 1–8 — the E5-D#5 trill in blue, bass accompaniment in coral</em></p>

Two color modes: hand (blue/coral) or pitch-class (chromatic rainbow — every C is red, every F# is cyan). The SVG format means the model can both see the image and read the markup to verify pitch, rhythm, and hand independence.

The Cockpit

A browser-based composition studio that lives in this repo at apps/cockpit — and runs live at mcp-tool-shop-org.github.io/ai-jam-sessions/cockpit. No plugins, no DAW, no install; everything stays in your browser (your work autosaves locally). Prefer to hack on it?

cd apps/cockpit && npm install && npm run dev   # Vite dev server, opens in your browser

• Beat-accurate transport — notes live in musical time, so the BPM control actually retimes playback; a click-to-seek time-ruler with drag-to-set loop regions; auto-scroll that follows the playhead
• Record-arm capture — play the QWERTY keys, on-screen keyboard, or a Web MIDI device and it lands in the score: 1-bar count-in, looper-style overdub across loop cycles (or replace mode), raw performance timing preserved under a quantized view, each pass one undoable unit
• Full undo/redo — every edit including Clear and Import is reversible (Ctrl+Z), with drag gestures coalescing the way real editors do
• Multi-select + clipboard — marquee selection under a Select/Draw tool toggle, platform-standard modifier clicks, copy/cut/paste-at-playhead, Duplicate
• Touch + accessibility — pointer events with capture on every surface, tap-to-relocate as a non-drag alternative, keyboard note editing, colorblind-safe scored overlays
• Dual-mode piano roll — switch between Instrument mode (chromatic pitch-class colors) and Vocal mode (notes colored by vowel shape: /a/ /e/ /i/ /o/ /u/)
• Visual keyboard — two octaves from C4, mapped to your QWERTY keyboard. Click or type.
• 20 voice presets — 15 Kokoro-mapped voices (Aoede, Heart, Jessica, Sky, Eric, Fenrir, Liam, Onyx, Alice, Emma, Isabella, George, Lewis, plus choir and synth-vox), 4 tract-mapped voices, and a synthetic choir section
• 10 instrument presets — the 6 server-side piano voices plus synth-pad, organ, bell, and strings
• Note inspector — click any note to edit velocity, vowel, and breathiness
• 7 tuning systems — Equal temperament, just intonation (major/minor), Pythagorean, quarter-comma meantone, Werckmeister III, or custom cent offsets. Adjustable A4 reference (392–494 Hz).
• Tuning audit — frequency table, interval tester with beat-frequency analysis, and tuning export/import
• Score import/export — serialize the entire score as JSON and load it back
• LLM-facing API — window.__cockpit exposes exportScore(), importScore(), addNote(), play(), stop(), panic(), setMode(), and getScore() so an LLM can compose, arrange, and play back programmatically

The Learning Loop

<p align="center"> <img src="docs/learning-loop.svg" alt="The learning loop: Read (MIDI + annotations) → Play (six sound engines) → See (piano roll · guitar tab) → Reflect (practice journal), with the journal persisting so the next session picks up where the last left off" width="100%" /> </p>

The Song Library

120 songs across 12 genres, built from real MIDI files. Each genre has one deeply annotated exemplar — with historical context, bar-by-bar harmonic analysis, key moments, teaching goals, and performance tips (including vocal guidance). These exemplars serve as templates: the AI studies one, then annotates the rest.

| Genre | Exemplar | Key | What it teaches | |-------|----------|-----|-----------------| | Blues | The Thrill Is Gone (B.B. King) | B minor | Minor blues form, call-and-response, playing behind the beat | | Classical | Für Elise (Beethoven) | A minor | Rondo form, touch differentiation, pedaling discipline | | Film | Comptine d'un autre été (Tiersen) | E minor | Arpeggiated textures, dynamic architecture without harmonic change | | Folk | Greensleeves | E minor | 3/4 waltz feel, modal mixture, Renaissance vocal style | | Jazz | Autumn Leaves (Kosma) | G minor | ii-V-I progressions, guide tones, swing eighths, rootless voicings | | Latin | The Girl from Ipanema (Jobim) | F major | Bossa nova rhythm, chromatic modulation, vocal restraint | | New-Age | River Flows in You (Yiruma) | A major | I-V-vi-IV recognition, flowing arpeggios, rubato | | Pop | Imagine (Lennon) | C major | Arpeggiated accompaniment, restraint, vocal sincerity | | Ragtime | The Entertainer (Joplin) | C major | Oom-pah bass, syncopation, multi-strain form, tempo discipline | | R&B | Superstition (Stevie Wonder) | Eb minor | 16th-note funk, percussive keyboard, ghost notes | | Rock | Your Song (Elton John) | Eb major | Piano ballad voice-leading, inversions, conversational singing | | Soul | Lean on Me (Bill Withers) | C major | Diatonic melody, gospel accompaniment, call-and-response |

Songs progress from raw (MIDI only) → annotated → ready (fully playable with musical language). The AI promotes songs by studying them and writing annotations with annotate_song.

Sound Engines

Six engines, plus a layered combinator that runs any two simultaneously:

| Engine | Type | What it sounds like | |--------|------|---------------------| | Oscillator Piano | Additive synthesis | Multi-harmonic piano with hammer noise, inharmonicity, 48-voice polyphony, stereo imaging. Zero dependencies. | | Sample Piano | WAV playback | Salamander Grand Piano — 480 samples, 16 velocity layers, 88 keys. The real thing. Programmatic API only: samples are not shipped (you supply the Salamander download); not yet wired into the CLI/MCP engine lists. | | Vocal (Sample) | Pitch-shifted samples | Sustained vowel tones with portamento and legato mode. | | Vocal Tract | Physical model | Pink Trombone — LF glottal waveform through a 44-cell digital waveguide. Four presets: soprano, alto, tenor, bass. | | Vocal Synth | Additive synthesis | 15 Kokoro voice presets with formant shaping, breathiness, vibrato. Deterministic (seeded RNG). | | Guitar | Additive synthesis | Physically-modeled plucked string — 4 presets (steel dreadnought, nylon classical, jazz archtop, twelve-string), 8 tunings, 17 tunable parameters. | | Layered | Combinator | Wraps two engines and dispatches every MIDI event to both — piano+synth, vocal+synth, etc. |

Keyboard Voices

Six tunable piano voices, each adjustable per-parameter (brightness, decay, hammer hardness, detune, stereo width, and more):

| Voice | Character | |-------|-----------| | Concert Grand | Rich, full, classical | | Upright | Warm, intimate, folk | | Electric Piano | Silky, jazzy, Fender Rhodes feel | | Honky-Tonk | Detuned, ragtime, saloon | | Music Box | Crystalline, ethereal | | Bright Grand | Cutting, contemporary, pop |

Guitar Voices

Four guitar voice presets with physically-modeled string synthesis, each with 17 tunable parameters (brightness, body resonance, pluck position, string damping, and more):

| Voice | Character | |-------|-----------| | Steel Dreadnought | Bright, balanced, classic acoustic | | Nylon Classical | Warm, soft, rounded | | Jazz Archtop | Mellow, woody, clean | | Twelve-String | Shimmering, doubled, chorus-like |

The Practice Journal

After every session, the server captures what happened — which song, what speed, how m