LLM-Shield-Proxy

LLM-Shield-Proxy is a self-hosted gateway for OpenAI-compatible LLM APIs. It replaces the personal data and secrets it detects (emails, card numbers, SSNs, API keys and more) before a request goes to the model provider, and puts the original values back into the streamed response before your application sees it. Your application changes only its base_url and the key it sends.

Already running LiteLLM? LiteLLM includes it as a built-in guardrail, guardrail: llm_shield_proxy, so LiteLLM keeps its own routing and calls the Shield to redact each request and restore each reply. See LiteLLM's setup page. It is on LiteLLM's main branch and not yet in a tagged LiteLLM release; until then, use one of the wirings in Running behind LiteLLM. On a LiteLLM without the module, that config does not stop the proxy: LiteLLM logs one error line, Skipping guardrail 'llm-shield': invalid configuration, proxy is starting WITHOUT this guardrail: Unsupported guardrail: llm_shield_proxy, starts, and every request reaches the provider unredacted (seen on 1.105.0, the newest image). Check that line is absent from the startup log before sending anything real.

Try it in a minute, with no API key

pip install llm-shield-proxy

UPSTREAM_BASE_URL=http://127.0.0.1:8765 UPSTREAM_API_KEY=unused VALID_VIRTUAL_KEYS=sk-demo llm-shield-proxy --port 4000 &

pii-leak-benchmark selfcheck --target-base-url http://127.0.0.1:4000/v1 --target-api-key sk-demo

The second command starts the proxy. The third sends prompts full of synthetic emails, SSNs, card numbers and API keys through it, and plays the model provider at 127.0.0.1:8765, so it can see exactly what the proxy forwarded. Nothing calls a real model. You should see:

  CLEAN

  No fixture value reached the upstream; required restore checks passed.

  Data types tested
    TYPE               RESULT        WHAT IT MEANS
    AWS_ACCESS_KEY_ID  contained     never reached the upstream in this run
    CREDIT_CARD        contained     never reached the upstream in this run
    EMAIL              contained     never reached the upstream in this run
    GITHUB_TOKEN       contained     never reached the upstream in this run
    SLACK_TOKEN        contained     never reached the upstream in this run
    SSN                contained     never reached the upstream in this run

Run the same check with nothing in the middle (--target-base-url capture://self) and every row reads LEAK. On Windows PowerShell, use the PowerShell version of these commands.

Use it with your application

The proxy needs two keys. VALID_VIRTUAL_KEYS lists the keys your clients send to the proxy; any other key gets a 401. The provider key (OPENAI_API_KEY here) is what the proxy sends upstream, so your clients never hold it.

pip install llm-shield-proxy
export VALID_VIRTUAL_KEYS=sk-my-client-key
export OPENAI_API_KEY=sk-y…[redacted]
llm-shield-proxy --host 127.0.0.1 --port 8000

Then point your existing client at it:

from openai import OpenAI

client = OpenAI(api_key="sk-m…[redacted]", base_url="http://localhost:8000/v1")
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Contact Sarah at sarah@example.com."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

For another provider, set UPSTREAM_BASE_URL and that provider's key; every setting is described in .env.example and the deployment guide.

With Docker, docker-compose.yml sets both keys for you. Put your OpenAI key in OPENAI_API_KEY first; the demo sends the client key demo-key:

docker compose up -d
curl http://localhost:8000/healthz
python examples/demo.py

<img src="website/docs/LLM-Shield-Proxy-paper-v2.gif" width="600" alt="Terminal demonstration of LLM-Shield-Proxy masking and streaming rehydration" />

How it works

Before sending a request to the model provider, the proxy finds configured types of sensitive data and replaces their values. As the provider streams its response, the proxy joins replacement tokens that were split across SSE events and restores values that the client is allowed to receive. For structured JSON, it changes string values without changing the JSON syntax. Test this behavior with the schemas used by your provider and tools.

<a href="website/docs/assets/diagram-dual-pipeline.svg?v=2"> <img src="website/docs/assets/diagram-dual-pipeline.svg?v=2" alt="LLM privacy proxy dual-pipeline redaction architecture" width="900" /> </a>

The maintained component map and deployment diagrams live in the architecture guide, architecture whitepaper, and deployment guide.

| Area | What is implemented | Where the evidence stops | |---|---|---| | Detection | 11 native Tier 1 data types, Tier 2 Shannon entropy, optional Tier 3 ONNX NER, BYOR rules | Supported types · no recall guarantee on unlabeled traffic | | Streaming privacy | Sliding-window SSE rehydration, bounded streaming JSON lexer | Architecture · conformance method | | Masking | Synthetic, structural-tag, scrub, operator-keyed stateless crypto | Masking guide · plaintext still exists in process memory | | Security controls | SSRF/DNS-rebinding egress checks, request policy, rate and blast-radius limits, canary tripwires | Security · not a substitute for network policy | | Evidence plane | Hash-linked audit records, Ed25519 receipts, OSCAL output, compliance packs | Compliance overview · tamper-evident, not WORM without immutable retention | | MCP governance | Scoped JSON-RPC subset with RBAC and egress policy | Research-scoped; MCP guide · not a complete MCP transport |

Deployment choices

In standard mode, detection, masking, policy checks, and value restoration run inside your gateway. Only the masked request is sent to the external model provider:

<a href="website/docs/assets/diagram-standard.svg?v=3"> <img src="website/docs/assets/diagram-standard.svg?v=3" alt="Standard LLM privacy gateway deployment" width="900" /> </a>

In air-gapped mode, the masked request goes to an internal model gateway. Network policy must still block direct provider access, telemetry, and other unintended outbound traffic:

<a href="website/docs/assets/diagram-airgapped.svg?v=3"> <img src="website/docs/assets/diagram-airgapped.svg?v=3" alt="Air-gapped LLM egress gateway deployment" width="900" /> </a>

See deployment topologies, air-gapped egress, and the Kubernetes/Helm deployment guide.

Every feature is labelled Supported, Beta, Experimental, or Research. The label states how the feature was tested and what remains untested: feature catalog · stability policy · limitations.

It supports SOC 2, HIPAA, GDPR, EU AI Act and NIST/ISO evidence programs by supplying technical controls and artifacts. It does not certify a deployment, guarantee complete detection, or make network policy optional.

Also in this repository

The proxy is one of four packages here. The other three are standalone and do not install it:

1. pii-leak-benchmark tests an OpenAI-compatible streaming gateway. It checks whether the gateway sends the test values to its model provider and whether the client gets the original values back.
2. mcp-ssrf-check checks an MCP server you operate for missing Host and Origin validation, unbound session ids, and URL-fetching tools that reach loopback. It talks only to your server and to a listener it opens on your own machine. It also runs as a GitHub Action that writes the result table to the job summary.
3. chunk-invariance turns the split-boundary rule into one test assertion: every way of splitting an input into chunks must stream to the same output as filtering it whole. Python on PyPI, and TypeScript on npm (chunk-invariance-js/).

The benchmark: does a gateway leak in streams?

Measure a gateway

pip install pii-leak-benchmark

# The negative control: no gateway at all, raw pass-through. MUST report outcome=fail.
pii-leak-benchmark \
  --target-base-url capture://self \
  --target-name raw-pass-through-negative-control --target-version 1 \
  --redaction-claimed claimed \
  --redaction-claim-citation https://github.com/ninadphalak/LLM-Shield-Proxy/blob/main/website/docs/conformance/reproducing.md \
  --redaction-enabled \
  --redaction-config-reference "synthetic control: declared redaction intentionally absent"

# Your gateway, already configured to send upstream traffic to http://127.0.0.1:8765/v1
pii-leak-benchmark --target-base-url http://127.0.0.1:4000/v1 --target-name your-gateway

Reproduce a published result instead of measuring a gateway

One bounded experiment, offline, no gateway or account. It re-runs the chunk-local and length-bounded-retention inspectors at the published seed and diffs every field of the result against the published reports:

git clone https://github.com/ninadphalak/LLM-Shield-Proxy.git
cd LLM-Shield-Proxy
python -m pip install ./pii-leak-benchmark
python benchmarks/reproduce_fragmentation.py --out reproduction

About two minutes. Exit status 0 means every field matched except timestamps and wall-clock timings. The same command runs in CI on Ubuntu, macOS and Windows across Python 3.11 and 3.12 (fragmentation-reproduction in .github/workflows/benchmark.yml). Full walkthrough and what the numbers mean: reproduce the fragmentation result.

Check your own gateway

One question, one command: does your deployment send raw personal data to its upstream?

pip install "pii-leak-benchmark>=0.2.0"

# Establish the floor first. No gateway at all: this MUST report LEAK.
pii-leak-benchmark selfcheck --tar