small-language-model-gate

[!NOTE] Related project — Tech-Lead-Stack: An agent-agnostic library of Markdown "skills" plus an MCP server that turns Claude, Gemini, or GPT into a full software-delivery team (planning, building, review, security, release), organized around a nine-phase lifecycle. Its self-correcting Reflexion loop grades implementation plans against four engineering pillars before any code is written.

<a href="https://github.com/bronz3beard/ai.tech-lead-stack" target="_blank" rel="noopener noreferrer">Explore tech-lead-stack on GitHub →</a>

---

Table of Contents

1. What is slm-gate?
2. Prerequisites & Hardware Sizing
3. How It Operates: Choosing Your Integration Layer
4. Step-by-Step Setup Walkthrough
5. Configuration Setup Guide
6. Verification & Day-to-Day Use
7. Architecture & Advanced Features

---

1. What is slm-gate?

small-language-model-gate (CLI shortname: slm-gate) is a local AI pre-processing and routing layer that sits in front of your AI coding assistant and quietly does two very useful things before anything reaches the paid cloud:

It compresses the noise. Your editor constantly packages up huge files, long logs, and sprawling system instructions and sends them to the cloud AI with every single message. Most of that is content the AI skims past. slm-gate intercepts this, runs a small, fast, free AI on your own computer, and strips it down to what actually matters — sending a fraction of the original text to the cloud.

It answers easy questions locally. Many conversations open with something a small model can answer — a quick fact, a short explanation, a greeting. slm-gate lets your local model try the first message of each conversation; if its answer passes a check, that request never reaches your paid plan at all. Anything else, and every slash command, goes to the cloud as normal.

The result: your paid AI plan lasts dramatically longer. Whether you're on a subscription (Claude Pro, Cursor Pro, Gemini Advanced, ChatGPT Plus) or paying per-token via an API key, you spend far less on the same amount of real work.

Why You Need This

Every AI subscription comes with rate limits. Claude Pro's five-hour windows, Cursor's monthly turn caps, ChatGPT Plus's hourly message limits — these aren't just numbers. Hit them mid-project and you're waiting hours to continue. slm-gate acts as a buffer. By intercepting routine traffic and compressing what does go to the cloud, your effective quota stretches much further.

On pay-per-token API plans (like gpt-4o or Claude Sonnet via API key), every token costs money. Sending a 500-line file when the model only needed 30 lines of context is a direct waste of budget. slm-gate eliminates that waste automatically.

Key Benefits

| Benefit | What It Means For You | | :--- | :--- | | Quota Protection | Fewer turns and tokens consumed means your subscription window lasts longer | | $0 Local Execution | Small, repetitive queries answered free by your local machine | | Context Compression | Large files, logs, and tool responses trimmed intelligently before hitting the cloud | | Privacy | Less of your actual code and data leaves your machine | | Provider-Agnostic | Works with Claude, GPT, Gemini, or any OpenAI-compatible endpoint |

What Gets Measured: Your Savings Dashboard

slm-gate records every decision it makes in a local database file on your computer (a SQLite database — think of it as a simple, fast spreadsheet that lives on your machine). For every request, it logs:

• How many tokens (units of text) were in the original payload
• How many tokens remained after compression
• Whether the request was answered locally (free) or forwarded to the cloud (answering locally is llm-gate only; see Section 3)
• How long the local processing took
• The simulated dollar cost saved (for API key users)

Checking your savings is one command:

pnpm run slm-gate metrics

This prints a clean, offline summary showing tokens saved, compression ratio, requests handled locally, and how many extra minutes of subscription headroom you've gained — no API keys, no internet connection required.

Want a visual dashboard? Run pnpm run dashboard for the built-in one-page dashboard — per-cycle window time returned per provider, tokens saved, weekly charts and routing split, straight from the local ledger, publishable to GitHub Pages for free (see Section 6). If you also set up Langfuse (a free, open-source observability tool), slm-gate will send traces there for session-by-session analysis. Both are optional — the local metrics command always works regardless.

How It Works: Visual

<img width="825" height="768" alt="Screenshot 2026-09-10 at 12 30 05 pm" src="https://github.com/user-attachments/assets/81d09734-234e-441f-9213-881ef219bedd" />

---

2. Prerequisites & Hardware Sizing

Required Software

Before you start, you need three things installed on your computer:

1. Node.js (version 22 or later)

Node.js is the runtime that slm-gate itself runs on. Check if you have it:

node -v
# Should print: v22.x.x or higher (22, 24 and 26 are all supported)

If not, download it from nodejs.org (choose the LTS version).

No compiler or Xcode Command Line Tools are needed. The one native dependency (better-sqlite3) ships prebuilt binaries for macOS, Linux and Windows, and pnpm install uses them as-is. Node 20 reached end-of-life and is no longer supported: pnpm install prints an Unsupported engine warning on it and doctor reports an issue.

2. pnpm (package manager)

pnpm is the tool used to install slm-gate's dependencies. Install it once:

npm install -g pnpm

3. Ollama (the local AI runner)

Ollama is a free program that downloads and runs small AI models on your machine — no cloud account needed. Install it from ollama.com and make sure it's running. You can verify with:

ollama list
# Should print a list of downloaded models (empty is fine initially)

Ollama should be reachable at http://localhost:11434 (its default address).

Optional Prerequisites

These are not required to get started, but unlock additional capabilities:

• Langfuse — A free, open-source web dashboard for visualising your AI usage and token savings. Self-hostable or use the cloud version. You only need this if you want richer visual analytics beyond the CLI metrics command.
• Cloud API Key — An ANTHROPIC_API_KEY, OPENAI_API_KEY, or similar. Only needed if you plan to use Layer 2 (llm-gate for model endpoint routing) or run the offline benchmarking harness. For Layer 1 subscription users, no API key is needed.
• Tech-Lead-Stack — An optional companion MCP server that slm-gate can sit in front of. See Section 3 for details.

---

Machine Sizing & Hardware Recommendations

<a id="appendix-c-ram-by-machine-model-table"></a>

Choosing the right local AI model for your machine is the single most important setup decision. Too large a model and your computer will slow to a crawl; too small and the compression quality suffers.

⚡ Built-In Preflight Diagnostic Tools

Before downloading any models, let slm-gate's built-in tools tell you exactly what your machine can handle:

# 1. Inspect your pulled models and get two personalised preset recommendations:
npm run models:check
# or: node dist/cli.js models:check

# 2. Run a full hardware + connectivity check:
pnpm run dev doctor
# or: node dist/cli.js doctor

• models:check reads your installed Ollama models, calculates their total memory footprint, and recommends two configurations tailored to your machine:
  • Option A (Dedicated AI Node): Maximises model quality when slm-gate is the main workload on this machine.
  • Option B (Primary Workhorse): Leaves enough memory free for your OS, browser, and IDE to avoid slowdowns and disk swapping.
• doctor detects your CPU architecture (Apple Silicon, Intel, CUDA), measures your available RAM, verifies Ollama is reachable, checks whether your context window setting (NUM_CTX) fits your hardware, and automatically writes a safe fallback configuration if it detects potential memory issues.

[!TIP] Not sure which models run well on your exact machine? Use llmfit.

AlexsJones/llmfit is an open-source terminal tool (with both a CLI and an interactive terminal UI) that right-sizes local AI models to your specific machine — measuring your RAM, CPU, GPU/VRAM, and estimating real tokens-per-second performance. It scores models across quality, speed, fit, and context dimensions, and supports Ollama, llama.cpp, MLX, and Docker Model Runner.

# macOS / Linux (Homebrew)
brew install AlexsJones/llmfit/llmfit

# Windows (Scoop)
scoop install llmfit

RAM-by-Machine Model Table

Use the table below to pick your starting models. The RAM Preset column maps directly to the preset .env files in the configs/ folder — you can copy the matching preset instead of configuring from scratch.

| RAM | RAM Preset | Brain Model (Smarter, Heavier Tasks) | Gate Model (Fast Router, Quick Decisions) | | :--------- | :---------- | :--------------------------------- | :---------------------------- | | 16 GB | ram-16 | qwen2.5-coder:3b, tinyllama | qwen2.5-coder:0.5b | | 24 GB | ram-24 | qwen3.5:4b, llama3.2:3b | qwen2.5-coder:3b, phi3:mini | | 32 GB | ram-32 | qwen2.5:7b, mistral:7b | qwen2.5-coder:3b, phi3:mini | | 64 GB | ram-64 | qwen3.5:9b, llama3:8b | qwen3.5:4b, llama3.2:3b | | 128 GB | ram-128 | qwen3:14b, llama3:70b (Q4) | qwen3:7b, mistral:7b |

What is the "Brain" vs. "Gate" model? slm-gate uses two local models:

• The Gate model is tiny and fast. Its only job is to make split-second routing decisions: "Can I handle this locally, or does it need to go to the cloud?" Speed is everything here.
• The Brain model is slightly larger and smarter. It does the actual compression, summarisation, and local resolution when the Gate decides a request can be handled on-device.

Dual-Model Setup: Keeping Both Models Ready at Once

When running both a Gate model and a Brain model, you want both loaded in memory at the same time — otherwise Ollama has to swap one out every time it switches, adding several seconds of latency to each request.

# macOS (set permanently via launchctl)
launchctl setenv OLLAMA_MAX_LOADED_MODELS 2

# Linux / any terminal session
export OLLAMA_MAX_LOADED_MODELS=2

Note on Memory Limits: Loading two models simultaneously means