MLX-VLM

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) and Omni Models (VLMs with audio and video support) on your Mac using MLX.

Table of Contents

• Installation
• Usage
  • Command Line Interface (CLI)
    • Thinking Budget
  • Speculative Decoding
    • DFlash (Qwen3.5)
    • Gemma 4 MTP
    • Gemma 4 EAGLE-3
    • MiniMax M3 EAGLE-3
  • Chat UI with Gradio
  • Python Script
  • Server (FastAPI)
    • Continuous Batching
    • Automatic Prefix Caching (APC)
    • KV Cache Quantization
• 1-bit Affine Inference
• Activation Quantization (CUDA)
• Multi-Image Chat Support
  • Supported Models
  • Usage Examples
• Model-Specific Documentation
• Vision Feature Caching
• TurboQuant KV Cache
• Distributed Inference
• Fine-tuning

Model-Specific Documentation

Some models have detailed documentation with prompt formats, examples, and best practices:

| Model | Documentation | |-------|---------------| | DeepSeek-OCR | Docs | | DeepSeek-OCR-2 | Docs | | Unlimited-OCR | Docs | | DOTS-OCR | Docs | | DOTS-MOCR | Docs | | ERNIE 4.5 VL | Docs | | GLM-OCR | Docs | | Phi-4 Reasoning Vision | Docs | | MiniCPM-o | Docs | | PaddleOCR-VL | Docs | | Phi-4 Multimodal | Docs | | MolmoPoint | Docs | | LocateAnything | Docs | | Moondream2 | Docs | | Moondream3 | Docs | | Gemma 4 | Docs | | MiniMax M3 | Docs | | Falcon-OCR | Docs | | Granite Vision 3.2 | Docs | | Granite 4.0 Vision | Docs | | MiniCPM-V 4.6 | Docs |

Installation

The easiest way to get started is to install the mlx-vlm package using pip:

pip install -U mlx-vlm

The Gradio chat UI needs an extra dependency that is not part of the base install:

pip install -U 'mlx-vlm[ui]'

Quote the package name so that shells which expand square brackets, such as zsh, do not treat [ui] as a glob pattern.

Agent Skills

This repo ships an agent-skills bundle under skills/ for common MLX-VLM workflows — usage, conversion, development, and support. Skills load into a coding agent (Claude Code, Codex, Gemini) so it follows the right project conventions instead of guessing.

| Skill | Description | |-------|-------------| | cli-inference | Run and debug command-line inference (mlx_vlm.generate) — text/image/audio inputs and image-generation flags. | | server-inference | Run and debug the local server across the models, chat, responses, messages, audio, image, cache, and metrics endpoints. | | convert-quantize | Convert and quantize Hugging Face models to MLX (mlx_vlm.convert) — bits/group size, quant modes, RTN/AWQ, mixed recipes. | | add-new-model | Port a new architecture into mlx_vlm/models — config, weight-name mapping, reuse a similar model, add a test class. | | benchmarking | Produce credible, reproducible perf numbers and fork-vs-main A/B tables for PRs. | | contributing | Shape a change to pass review — code/config/test placement, pre-commit hooks, and PR expectations. | | hf-cache-models | List MLX-VLM-supported (and, with --check-arch, loadable) models in the local Hugging Face cache. | | reproducible-github-issues | Turn CLI or server failures into concise, reproducible GitHub issues. |

Validate the bundle at any time:

python3 skills/scripts/validate_skills.py

Install from a local checkout:

# Claude Code
/plugin marketplace add /path/to/mlx-vlm
/plugin install mlx-vlm-skills@mlx-vlm

# Codex CLI
codex plugin marketplace add /path/to/mlx-vlm
codex plugin add mlx-vlm-skills@mlx-vlm

# Gemini CLI
gemini extensions install /path/to/mlx-vlm/skills

Usage

Command Line Interface (CLI)

Generate output from a model using the CLI:

# Text generation
mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Hello, how are you?"

# Image generation
mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --temperature 0.0 --image http://images.cocodataset.org/val2017/000000039769.jpg

# Audio generation (New)
mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you hear" --audio /path/to/audio.wav

# Multi-modal generation (Image + Audio)
mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you see and hear" --image /path/to/image.jpg --audio /path/to/audio.wav

Thinking Budget

For thinking models (e.g., Qwen3.5), you can limit the number of tokens spent in the thinking block:

mlx_vlm.generate --model mlx-community/Qwen3.5-2B-4bit \
  --thinking-budget 50 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>" \
  --enable-thinking \
  --prompt "Solve 2+2"

| Flag | Description | |------|-------------| | --enable-thinking | Activate thinking mode in the chat template | | --thinking-budget | Max tokens allowed inside the thinking block | | --thinking-start-token | Token that opens a thinking block (default: <think>) | | --thinking-end-token | Token that closes a thinking block (default: </think>) |

When the budget is exceeded, the model is forced to emit \n</think> and transition to the answer. If --enable-thinking is passed but the model's chat template does not support it, the budget is applied only if the model generates the start token on its own.

On the server, thinking mode is disabled by default. Start the server with --enable-thinking to make thinking mode the default for requests that do not specify it:

mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking

You can also set server defaults for the thinking budget and delimiter tokens:

mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --enable-thinking \
  --thinking-budget 512 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>"

Requests can override the server defaults with enable_thinking, thinking_budget, thinking_start_token, or thinking_end_token.

Speculative Decoding

Speed up generation by drafting several candidate tokens with a small "drafter" model and verifying them in a single target forward pass. Three drafter families are supported.

| Flag | Description | |------|-------------| | --draft-model | HuggingFace repo or local path for the drafter | | --draft-kind | Drafter family — dflash (default), eagle3, or mtp (native/assistant MTP) | | --draft-block-size | Override the drafter's configured block size |

See docs/usage.md for Python API examples including batch generation.

DFlash (Qwen3.5)

A lightweight block-diffusion drafter that predicts multiple tokens per round, typically 2–3× faster.

# Text generation with speculative decoding
mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0 --enable-thinking

# Also works with images
mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --image examples/images/cats.jpg \
  --prompt "Describe this image." \
  --max-tokens 256 --temperature 0 --enable-thinking

# Server with speculative decoding
mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash

DFlash draft-cache windowing is available from the Python API. During speculative decoding the target model still verifies every proposed token with its full KV cache; this knob only changes the