PersonalJarvis
Self-hosted voice assistant. Speak to your computer: it answers aloud, types your dictation, and operates your apps.
Install / Use
npx skills add PersonalJarvis/PersonalJarvisInstalls into whichever agent you are using.
Other
Other agent config
Quality Score
Category
AI & Machine LearningSupported Platforms
Skill content
View source on GitHubInstall it in one command
Windows (PowerShell)
irm https://raw.githubusercontent.com/PersonalJarvis/PersonalJarvis/main/install/install.ps1 | iex
macOS and Linux
curl -fsSL https://raw.githubusercontent.com/PersonalJarvis/PersonalJarvis/main/install/install.sh | bash
Python 3.11+ and Git, nothing else. The installer asks nothing in the terminal, and the app walks you through language, wake word, and keys once. Full install notes below.
<p align="center"> <a href="https://pypi.org/project/personal-jarvis/"><img alt="PyPI: personal-jarvis" src="https://img.shields.io/pypi/v/personal-jarvis?style=for-the-badge&labelColor=242424&color=e7c46e" /></a> <a href="https://github.com/PersonalJarvis/PersonalJarvis/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-e7c46e?style=for-the-badge&labelColor=242424" /></a> <a href="https://discord.gg/x7USduHxbc"><img alt="Discord" src="https://img.shields.io/badge/Discord-Join-5865F2?style=for-the-badge&logo=discord&logoColor=white&labelColor=242424" /></a> <a href="https://personaljarvis.ai/"><img alt="Personal Jarvis website" src="https://img.shields.io/badge/Website-personaljarvis.ai-e7c46e?style=for-the-badge&labelColor=242424" /></a> </p>A typical voice assistant talks back. Personal Jarvis does the thing. At the center of every voice conversation sits a tool model: it decides how much a request actually needs, runs shell commands, takes the mouse and keyboard, and reaches any service that speaks MCP. The short stuff it handles itself. Anything heavier goes to a coding-agent worker (Claude Code, Codex CLI, Gemini CLI, or an in-process worker on whatever API key you already have), which runs in isolation, gets checked by a critic, and reports back in the language you spoke.
Every tier has a keyless local option, so the whole assistant can run on your own hardware with no cloud account anywhere in the chain — details under Runs on your own hardware below. If you would rather use a hosted model, you pick the provider per tier: Gemini, Claude, OpenAI, or OpenRouter, one setting for each. It can rewrite its own configuration, and it runs on a headless server just as well as on a desktop with a microphone.
<p align="center"> <img src="https://github.com/PersonalJarvis/PersonalJarvis/raw/main/assets/screenshots/app-chat.webp" alt="The desktop app's chat view: voice history in the sidebar, the ghost mascot over a golden wave wallpaper, and the Ready for commands prompt" width="900" /> </p> <p align="center"> <sub>The desktop app, ready for commands. The assistant takes whatever name you pick as your wake word — this install answers to George.</sub> </p>What you can say
| You say | What happens | |---|---| | "Research vector databases." | An isolated agent does the research. The finished report lands in Outputs as a file you can download. | | "Call the clinic and book the next open appointment." | A real outbound phone call goes out over the optional Twilio line. | | "Remember: Alex prefers Signal over email." | Written to the Knowledge Wiki, and still known in every later session. | | "Switch the voice over to Cartesia." | The speech provider changes while you talk, and Jarvis reads the change back to you, old then new. | | "Tell T1 to run the tests." | The instruction lands in terminal 1 of the Agentic IDE workspace. | | "Open the browser and pull up the weather." | Jarvis takes the mouse and keyboard and does it on your screen. |
All six work today; none of this is a roadmap item. Two need extra setup: the phone call
needs the optional [telephony] extra plus your own Twilio account, a number, and a
publicly reachable HTTPS URL for the webhooks, and computer use needs a desktop install
with a screen, not the headless one.
Which model answers you
Three models sit behind that table, and you never choose between them: the handover happens mid-sentence, based on what the request needs.
The realtime model carries the conversation itself. It hears you and answers in under a second, built for talking, not for thinking hard. The moment a request needs an actual tool, that turn hands off to a second model, slower and noticeably smarter, the one that reads your wiki, changes a setting, places the call, or takes the screen. It answers in the same voice, so from where you're sitting it never stopped being one conversation. Real work, the kind that takes minutes, goes to a third: a coding agent running in its own isolated copy of the workspace, reviewed by a critic, that comes back with a file instead of just an answer.
In the app this lives on one screen: API Keys has one tab per tier, each with its own provider, and you only need keys for the tiers you use. Every provider says how it bills: a subscription login you already have, or an API key charged per token.
Runs on your own hardware
Hosted providers are a choice here, not a requirement. Every layer that could reach for a cloud account has a local option that needs no API key and no signup, so a complete install can keep your voice, your screen and your files on the machine they started on.
| Layer | Keyless local option | What it costs you |
|---|---|---|
| Conversation (realtime) | A self-hosted server speaking the OpenAI Realtime protocol. The install panel checks the machine and sets up a managed one in a click, or you point it at any address of your own. | About 12 GB of GPU or unified memory for a good experience. Still marked experimental. |
| Brain (decisions, tools) | Ollama, found automatically at http://localhost:11434, or any OpenAI-compatible server. | Pull a tools-capable model, e.g. ollama pull qwen3.5. |
| Speech to text | Whisper large-v3 on device, or NVIDIA's Nemotron 3.5 streaming model. | Whisper is a one-time 3 GB download and wants a graphics card; Nemotron is ~690 MB, covers 40 languages, and runs several times faster than real time on a plain CPU. |
| Text to speech | Piper on this machine, plus Kokoro and Qwen3-TTS voice profiles. | A voice download in the hundreds of megabytes. |
Mixing is normal and expected: a local recognizer with a hosted brain, or a local brain with a hosted voice. Nothing forces the whole chain one way.
Two capabilities stay outside this promise, and it would be dishonest to imply otherwise. The outbound phone call goes over Twilio, which is a hosted service by definition, and the coding-agent workers run on whichever agent CLI or API key you point them at. Everything in the table above is genuinely local.
Demo
The clip at the top is the short version. Watch the full demo on YouTube, where one voice command takes the screen and changes a Windows setting live, or browse the channel for the rest.
What it does differently
The router itself stays small. It works out what you said, picks a tool or a worker, and gets out of the way; there is no single giant prompt trying to be everything. Anything non-trivial runs as a mission in an isolated worktree and gets a critic's review before you ever hear the result. You are not left listening to silence while that happens, either: the moment the router picks an action, Jarvis says one line about that specific action, not a generic "working on it".
Providers are interchangeable, and that matters most on the day one of them fails. If the configured provider is unreachable or out of quota, Jarvis crosses to a different provider family instead of leaving you stuck. Workers run on a subscription login or a pay-per-token key, whichever you have. Speech and voice providers can be switched by voice mid- conversation; the brain provider cannot, on purpose, that one stays yours to change from the app or the CLI.
It also remembers: a Knowledge Wiki of plain Markdown files, plus an awareness layer, build up a picture of you across sessions. And it can change its own settings through a guarded, audited pipeline, the full mechanics are under Self-modification below.
How it works
<img src="https://github.com/PersonalJarvis/PersonalJarvis/raw/main/assets/brand/how-personal-jarvis-works.png" width="1064" height="568" alt="How Personal Jarvis works: routing voice and chat through safe actions or reviewed missions" />
Higher layers can only reach lower ones through protocols; everything else talks over a typed, immutable EventBus. That's the strict seam that makes harnesses, providers, and plugins swappable in the first place.
<details> <summary><b>The 8-layer map</b></summary>L7 UI/UX Desktop app (FastAPI + React + pywebview), tray, Orb overlay
L6 Orchestrator State machine, Router, BrainManager, Mission-Manager + workers, Controller
L5 Harness adapter python-script, computer-use (coding agents are L6 mission workers)
L4 Brain Gemini · Claude · OpenAI · Grok · OpenRouter + sub-second Ack-Brain
L3 Intent / Risk Classifier, four-tier risk policy, approval, rate-limit tracking
L2 Speech Wake → VAD → STT → TTS (cloud or local, your choice)
L1 Audio I/O Device routing, chime feedback
L0 OS / Hardware Mic, speakers, global hotkeys, optional GPU
A deeper engineering map, with anti-patterns, bug classes, and phase status down to
file:line, lives in docs/LLM-CONTEXT.md.
Install
One command on Windows, macOS, or Linux. You need Python 3.11 or newer and Git; the installer checks for both and stops with a download link if one is missing. It asks nothing in the terminal. It launches the app, and the app walks you through a one-time setup for language, wake word, and API keys.
What it costs: nothing to us. Personal Jarvis is MIT-licensed software you run on your own machine. There is no subscription for it, no paid tier, no marketplace cut, and no referral link behind any provider named on this page. What you do need is access to a model, and that is billed by whoever provides it, straight to you. An AI subscription you already pay for works, and so does a pay-per-token API key. The same goes for the optional pieces: a phone call runs on your own Twilio account at Twilio's prices.
Windows (PowerShell)
irm https://raw.githubusercontent.com/PersonalJarvis/PersonalJarvis/main/install/install.ps1 | iex
macOS and Linux
curl -fsSL https://raw.githubusercontent.com/PersonalJarvis/PersonalJarvis/main/install/install.sh | bash
This is open source, so read the installer before you run it. It creates a venv, installs dependencies, prefetches the voice models, and launches the app. Your keys land in your operating system's credential manager
Truncated for display — read the full file on GitHub.
Related Skills
momen-cursurrules-prompt-file
40.6kCursor rules for building custom frontends with Momen.app as headless BaaS with GraphQL API, actionflows, AI agents, and Stripe integration.
semiotic-react-dataviz-cursorrules-prompt-file
40.6kCursor rules for Semiotic data visualization library with 30+ chart types, MCP server, and AI-assisted chart generation.
claude-mem
91.5kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
Understand-Anything
80.1kGraphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
