ctxlayer

Agent context layer — an MCP service on Cloudflare that:

• serves curated org docs (Markdown, with Vectorize-backed RAG search) as MCP resources and a search_docs tool — and speaks the Open Knowledge Format (OKF) in and out, so an org's library round-trips with any OKF bundle;
• proxies other MCP servers (HTTP/SSE natively; a stdio MCP server is supported via bring-your-own-bridge — run your own stdio↔HTTP bridge and register its URL as a streamable_http upstream), centralising per-user credentials sealed at rest;
• exposes a React + Vite SPA for self-onboarding, BlockNote + Yjs collaborative markdown editing, admin upstream management, and usage analytics.

Screenshots

| | | | --- | --- | | | | | Curate & gate upstream MCP servers — cache their tools, attach org playbooks + docs per tool | Usage analytics — calls, tokens, and top tools per user / upstream | | | | | Self-service onboarding — connect upstreams via OAuth or a personal token, sealed at rest | Curated skills — procedural playbooks agents load on demand |

<p align="center"> <img src="docs/screenshots/upstream-config.png" alt="Per-upstream configuration drawer" width="680"><br> <em>Per-upstream config — transport, auth strategy, resilience caps, and team/product visibility</em> </p>

Pointers: CONTRIBUTING.md for the contributor on-ramp (the Quickstart below is the copy-pasteable version), docs/PLAN.md for the architecture & data-model reference, CLAUDE.md / AGENTS.md for AI agents working in this repo, and docs/plan/G-conventions.md for conventions and gotchas.

Integration surfaces. The supported, stable contract for external clients is the MCP surface (/mcp, /sse) plus the OAuth provider. The /api/* REST endpoints are an internal contract for the bundled SPA — not versioned and subject to change between releases. Build agents and scripts against MCP, not /api.

What it does

• Sign-in & admission — GitHub / Google OAuth with org/domain/user allowlists, plus request (admin-approval queue) and invite / join-code admission policies; per-request user lifecycle (suspend cuts live sessions).
• Doc library — BlockNote editor with realtime Yjs collaboration (DocRoomDO), revision history with autosave coalescing, folders, per-doc locks, team/product/free-form tags; open-read org-wide.
• RAG search — hybrid retrieval (dense + lexical over Vectorize, LLM query rewrite, cross-encoder rerank) behind the search_docs tool and /api/search.
• MCP server (/mcp, /sse) — built-ins (search_docs, get_doc, list_upstreams, describe_upstream, list_skills / get_skill, draft_skill, poll_task / list_tasks, reload_upstreams, …), doc + skill resources, and an OAuth provider for MCP clients.
• Upstream proxy — aggregate other MCP servers (Streamable HTTP / SSE; stdio via bring-your-own-bridge) under namespaced tools, with per-user or shared credentials sealed AES-GCM at rest, static or DCR OAuth, per-team/ product/role visibility, per-tool ACLs, timeouts + response-size caps, an async submit→poll path for slow tools, and a cached tool catalogue with a degraded-response shrink guard.
• Skills — curated procedural playbooks served over MCP, attachable per upstream or per tool, authorable by any user (private drafts → org sharing), in-app AI drafting, SKILL.md export.
• Git sync — mirror Markdown from GitHub / GitLab / Azure DevOps repos (PAT or OAuth) into the library on a cron, and propose edits back as PRs (round-trip-safe normalisation, HTML + frontmatter guards).
• Usage & audit — per-user/upstream call + token analytics with error detail, and an admin audit log covering role, credential, visibility and content mutations.

Open Knowledge Format (OKF) — early adopter

ctxlayer natively speaks the Open Knowledge Format, Google Cloud's open convention for agent-friendly knowledge: a directory of Markdown files with YAML frontmatter. ctxlayer's doc library is, structurally, an OKF bundle — so an org's curated context is portable, not locked in.

• Import — register an OKF git repo as a source (or paste/upload a .md). Frontmatter (type, description, resource, tags, title, …) is parsed onto the doc; unknown / extra producer keys (okf_version, etc.) are kept verbatim.
• Edit — the doc editor's right rail is the frontmatter editor. The OKF fields carry an OKF badge with a tooltip pointing at the matching spec key.
• Export — "Export as OKF (.md)" in the rail, or GET /api/docs/:id/export, emits a spec-compliant file: synthesised frontmatter (rail fields + preserved unknown keys) followed by the body.
• Git write-back — edits to a doc synced from an OKF repo propose a PR that keeps and refreshes the frontmatter block.

OKF tags map to ctxlayer's free-form tags (not the team/product tags, which gate visibility). Full mapping, round-trip contract, and fidelity caveats: docs/plan/M-okf.md.

Quickstart (contributors hacking on ctxlayer)

These steps are for local development of this codebase. End users of a deployed ctxlayer and operators standing it up don't need any of this; see Deploying ctxlayer below.

brew install mkcert nss           # macOS contributors only; see docs/plan/G-conventions.md G11 for Linux/Windows
bun install
cp .dev.vars.example .dev.vars    # then edit it — see "Filling in .dev.vars" below
bun run migrate:local             # apply D1 migrations to the local (miniflare) DB
bun run seed:local                # load fixture teams + products (NOT upstreams/docs — add those in-app)
bun run dev                       # or split-terminals: dev:worker + dev:web (recommended)
bun run verify                    # typecheck + lint (Biome) + unit + integration tests (all offline)

Filling in .dev.vars

To sign in locally you need at least one IdP. The quickest is a GitHub OAuth app (https://github.com/settings/developers → "New OAuth App"):

• Authorization callback URL: https://localhost:8787/idp/github/callback
• Put its client id/secret in GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET.
• Set ALLOWED_GITHUB_USERS=<your-login> — the allowlist gates who may sign in.
• Set ADMIN_EMAILS=<your-email> so the admin pages are reachable.
• Generate ENCRYPTION_KEY and SESSION_COOKIE_SECRET with openssl rand -base64 32 each.
• PUBLIC_BASE_URL is already set to https://localhost:8787 in the example.

Local dev needs no Cloudflare account — miniflare emulates D1, KV, R2, and Queues offline. Workers AI and Vectorize have no local emulator, so search_docs returns nothing locally (the reindex consumer soft-skips Vectorize in dev) — that's expected; exercise RAG end-to-end against a real deploy. bun run verify is fully offline; bun run verify:full additionally runs the smoke suite, which needs a running Worker (bun run dev:worker) or a preview URL.

The first dev run calls scripts/setup-dev-tls.mjs via the predev hook and generates a locally-trusted cert in .dev-tls/. Both Vite and Wrangler then serve HTTPS on localhost — required for the __Host- session cookie to work in dev. The cert never leaves your machine.

Two dev-loop notes: prefer separate terminals (bun run dev:worker + bun run dev:web) when debugging the worker — bun run dev shares one TTY via concurrently, which interleaves streams and worsens wrangler's stack-trace elision. And to wire Claude Desktop to a local ctxlayer, shim it through mcp-remote with the mkcert CA ("command": "npx", "args": ["-y", "mcp-remote", "https://localhost:8787/mcp"] plus NODE_EXTRA_CA_CERTS pointing at mkcert -CAROOT/rootCA.pem in claude_desktop_config.json), and connect upstreams in the browser first at https://localhost:5173/upstreams — proxied tools register only for users with stored credentials at session init.

Deploying ctxlayer to production

If you're standing up an instance of ctxlayer for your org (no source edits), you don't need bun run dev or mkcert. Cloudflare's edge provides real HTTPS for the public hostname automatically.

The install has four phases: (1) provision Cloudflare resources, (2) configure at least one identity provider, (3) set deployment secrets, (4) deploy and pin to a custom domain. The step-by-step sequence is in the four numbered subsections below.

1. Provision Cloudflare resources

wrangler login                  # or set CLOUDFLARE_API_TOKEN + CLOUDFLARE_ACCOUNT_ID
bun install
bun run bootstrap               # provisions D1, KV, R2, both Vectorize indexes,
                                # all four queues, and patches the IDs into wrangler.toml
bun run migrate:remote          # applies migrations 0001..N to remote D1

bootstrap is idempotent — re-runnable any time. The manual fallback (if you're not using the script) is:

wrangler d1 create ctxlayer
wrangler kv namespace create OAUTH_KV
wrangler r2 bucket create ctxlayer-docs
wrangler vectorize create ctxlayer-docs --dimensions 768 --metric cosine
wrangler vectorize create ctxlayer-docs-lexical --dimensions 768 --metric cosine   # hybrid search's lexical leg — without it search silently degrades to dense-only
wrangler queues create ctxlayer-usage
wrangler queues create ctxlayer-reindex
wrangler queues create ctxlayer-git-sync
wrangler queues create ctxlayer-jobs        # async submit→poll for slow upstream tools — without it those calls never run
# then replace the committed resource IDs in wrangler.toml with the printed
# ones (the binding comments there document which IDs to swap)

2. Identity provider configuration

ctxlayer ships GitHub and Google sign-in. Enabling an IdP is a two-sided contract: the OAuth app at the provider, and the matching client-id/secret + allowlist on the worker. At least one IdP must be enabled — the sign-in page hides any IdP whose *_CLIENT_ID secret is unset, and the allowlist is what decides who can sign in under the default open_domain policy (§2d covers the request/invite alternatives).

Plan the hostname first. Every IdP callback URL bakes in PUBLIC_BASE_URL. If you intend to use a custom domain (recommended — see §4 below), decide it now so you register callbacks against the real hostname once, instead of re-registering after a workers.dev bootstrap.

2a. GitHub OAuth App

1. GitHub → Settings → Developer settings → OAuth Apps → New OAuth App. *(Org-owned apps live under the org's settings; personal apps under the user's. Org-owned is preferable for centralise