Meeting Agent

A full-stack RAG application that ingests meeting recordings (video/audio), documents (PDF, PPTX, DOCX, XLSX, CSV, TXT), and images (PNG, JPG, WebP, TIFF, BMP), transcribes and parses them, indexes the content into a vector database, and exposes Q&A with conversation memory, knowledge-graph entity tracking, and optional web-search augmentation. Also ships an MCP server for external tool integration.

Demo Videos

Walk-throughs and feature demos are on the project's YouTube channel:

Click a thumbnail to play on YouTube — GitHub-flavored Markdown does not embed live players inline.

| Full Demo | Invoke Skills | |:---:|:---:| | | | | End-to-end overview of upload → ingest → chat with citations | Custom skills registered via the API, fired either by direct invocation or by intent matching from chat | | Step by Step | Memory & Knowledge Graph | | | | | Chat at three scope levels — unscoped (all meetings), meeting-scoped, and file-scoped — showing how retrieval narrows with each pick | Long-term memory, knowledge-graph entities, and cross-session recall |

Highlights

• Multi-modal ingestion — video / audio / PDF / PPTX / DOCX / XLSX / CSV / TXT / images, with magic-byte validation and streaming size limits.
• Cloud-native parsing cascade — content-aware routing across Marker, MinerU, and PaddleOCR APIs with a local PyMuPDF / python-pptx fast path and a guaranteed last-resort fallback so ingestion never hard-fails.
• Speaker-diarized ASR — AssemblyAI with editable speaker → real-name mapping that re-indexes the affected file's vectors and per-file summary.
• Hybrid retrieval — semantic (Chroma) + BM25 lexical with Reciprocal Rank Fusion, fair per-file allocation, anchor-based eviction for session continuity, and Cohere / BGE rerank.
• Unified citations — chunks, file summaries, and meeting summaries share one [N] numbering, all clickable from the chat UI to jump back to source page / slide / timestamp.
• Long-term memory — auto-extracted facts with TTL and decay, knowledge-graph entities + relations with alias merging, and episodic cross-session summaries with semantic search.
• Streaming everywhere — chat, summary generation, and rebuild operations stream tokens via SSE with per-step trace events.
• Multi-provider LLM / embedding — OpenAI, Azure OpenAI, Anthropic, DeepSeek, OpenRouter, Groq, Together, Mistral, Ollama, LM Studio, vLLM, llama.cpp; embeddings additionally support Jina, Cohere, Hugging Face, Google Vertex AI.
• Hardened API — versioned /api/v1, per-endpoint rate limiting, idempotency keys with AES-GCM-encrypted response storage, HMAC-signed file-download tokens, and structured JSON logs.
• MCP server — six tools exposed over stdio (and optional HTTP), so Claude / other agents can drive the system as a backend.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                       Frontend (React 19)                        │
│   chat · materials · memory · history · generate · settings      │
└──────────────────────────────┬──────────────────────────────────┘
                                │ /api/v1 (proxied via Vite/nginx)
┌──────────────────────────────▼──────────────────────────────────┐
│                   FastAPI · LangChain LCEL                       │
│                                                                  │
│  Upload  →  parse / transcribe  →  chunk  →  embed  →  Chroma   │
│                                              ↓                   │
│                                          BM25 + FTS5             │
│                                                                  │
│  Query   →  scope routing  →  retrieve  →  rerank  →  context   │
│                                                       ↓          │
│                                                      LLM         │
│                                                       ↓          │
│                                            answer + citations    │
│                                                                  │
│  Background: KG entities · memory decay · session summaries      │
└──────────────────────────────────────────────────────────────────┘

Backend — FastAPI 0.115+ · LangChain LCEL · ChromaDB · SQLite (WAL, alembic-managed) · slowapi · pydantic v2. Frontend — React 19 · TypeScript · Vite 6 · Ant Design 6 · react-router v7 · framer-motion. Infra — Docker Compose · Helm chart · Prometheus + Grafana + Loki + Promtail (optional observability stack).

Quick Start

Docker (recommended)

cp backend/.env.example backend/.env
# Edit backend/.env: set LLM_API_KEY (required) and ASSEMBLYAI_API_KEY (for video).
docker compose up --build

Port mapping (host → container): backend 7008 → 8000, frontend 8307 → 80. Override in docker-compose.yaml if needed.

• Frontend: http://localhost:8307
• Backend API: http://localhost:7008
• API docs: http://localhost:7008/docs
• WebSocket: ws://localhost:7008/api/v1/ws

Manual setup

# Backend (Python 3.12+)
cd backend
uv sync --dev                          # recommended; uses uv.lock for reproducible installs
# pip install -e ".[dev]"              # alternative

# Optional extras (only install if you actually need the provider):
#   uv sync --dev --extra multimodal   # RAGAnything bridge (RAGANYTHING_ENABLED=true)
#   uv sync --dev --extra google       # Vertex AI embeddings
#   uv sync --dev --extra huggingface  # local HF embeddings + BGE reranker
#   uv sync --dev --extra local        # huggingface + llama-cpp-python (fully offline)
#   uv sync --dev --extra observability # Sentry + OpenTelemetry

cp .env.example .env                   # set LLM_API_KEY and ASSEMBLYAI_API_KEY
uv run python -m uvicorn src.main:app --reload    # http://localhost:8000

# Frontend (Node 22+; separate terminal)
cd frontend
npm install
npm run dev                            # http://localhost:5173, proxies /api → :8000

Project-level shortcuts

make dev          # backend + frontend concurrently
make dev-be       # backend only
make dev-fe       # frontend only
make cli          # interactive terminal frontend (scripts.cli_agent)
make lint         # lint everything
make test         # run all tests
make qa           # full QA: lint + tests + Playwright E2E against Docker
make clean        # remove generated files

Pre-commit hooks

pip install pre-commit && pre-commit install

Hooks in .pre-commit-config.yaml: ruff, eslint, prettier, bandit, gitleaks, detect-secrets.

Configuration

Three-tier override (highest priority last):

1. backend/config/main.yaml — non-secret defaults (model names, RAG knobs, upload limits).
2. backend/.env — secrets and per-environment overrides.
3. Environment variables — for Docker / CI.

Settings are merged via pydantic-settings; see backend/.env.example for the full surface.

Key settings

| Setting | Default | Description | |---|---|---| | LLM_BINDING | openai | openai, azure_openai, anthropic, deepseek, openrouter, groq, together, mistral, ollama, lm_studio, vllm, llama_cpp | | LLM_MODEL | gpt-4o-mini | Any chat model the binding supports | | LLM_API_KEY | (required) | API key for the chosen LLM provider | | LLM_BASE_URL / LLM_HOST | (empty) | Custom endpoint for OpenAI-compatible / local providers | | EMBEDDING_BINDING | openai | + jina, cohere, huggingface, google (Vertex AI), and the LLM-shared bindings | | EMBEDDING_MODEL | text-embedding-3-small | Embedding model | | EMBEDDING_DIMENSION | 1536 | Must match the model's vector size | | ASR_PROVIDER | assemblyai | Only assemblyai is supported | | ASSEMBLYAI_API_KEY | (required for AV) | env-only, never in YAML | | OCR_PROVIDER | marker | Routing hint: marker, mineru, paddle | | RAG_RETRIEVER_PROVIDER | native | native, hybrid, multimodal, hybrid_multimodal | | RAGANYTHING_ENABLED | false | Multimodal dual-index branch (requires multimodal extra) | | SEARCH_BINDING | (empty) | Web search: duckduckgo, serpapi, tavily, bing, exa | | MEMORY_AUTO_EXTRACT | true | Auto-extract facts from each Q&A turn | | KNOWLEDGE_GRAPH_ENABLED | true | Index entities + relations into the KG | | ENVIRONMENT | dev | dev, staging, prod (non-dev requires API_KEY) | | API_KEY | (empty) | Empty = dev mode (auth bypassed); non-empty for staging/prod | | LOG_FORMAT | text | Set to json for structured logs |

Helm deployment notes (SQLite)

• Run backend as a single replica (backend.replicaCount=1) — SQLite cannot share writers.
• Provide secrets via Kubernetes Secret and set backend.secretName.
• HPA / PDB templates are intentionally absent for SQLite safety.

Using Dashscope (Qwen)

LLM_MODEL=qwen-plus
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
LLM_API_KEY=sk-your-dashscope-key

MCP Server

uv run python -m src.mcp                  # stdio transport
MCP_HTTP_PORT=9000 uv run python -m src.mcp   # HTTP transport (requires API_KEY)

Tools: list_meetings, search_meetings, ask_about_meetings, manage_memory, list_skills, invoke_skill.

API Endpoints

All routes are versioned at /api/v1. Authentication is via the X-API-Key header (empty API_KEY = dev mode). Rate limits are per-endpoint (upload / chat 20 / min, settings 5 / min, reads 60 / min). Errors share a unified ErrorResponse envelope (code, message, request_id, details).

Meetings

| Method | Path | Description | |---|---|---| | POST | /meetings/upload | Upload a file (creates new meeting if no meeting_id) | | POST | /meetings | Create a new empty meeting | | GET | /meetings | List meetings (filterable by status) | | GET | /meetings/{id} | Meeting detail with file list | | PUT | /meetings/{id} | Update meeting metadata | | DELETE | /meetings/{id} | Delete a meeting and all files | | GET | /meetings/{id}/files | List files for a meeting | | GET | /meetings/{id}/files/{fid} | Download a file (header X-API-Key or ?token=) | | POST | /meetings/file-token | Issue a short-lived global file token | | POST | /meetings/{id}/files/{fid}/signed-url | Issue a file-scoped HMAC-signed URL | | GET | /meetings/assets | Fetch a meeting asset by relative path | | GET | /meetings/{id}/files/{fid}/timeline | File timeline (segments / pages / captions / text) | | GET | /meetings/{id}/files/{fid}/speakers | List speaker mappings | | PUT | /meetings/{id}/files/{fid}/speakers | Update speaker → real-name mappings | | GET | /meetings/{id}/files/{fid}/speakers/{code}/audio | Sample audio clip for a speaker | | DELETE | /meetings/{id}/files/{fid} | Delete a single file from a meeting | | POST | `/meetings/{id}/su