SimpleAudit

Lightweight AI Safety Auditing Framework

SimpleAudit is a simple, extensible, local-first framework for multilingual auditing and red-teaming of AI systems via adversarial probing. It supports open models running locally (no APIs required) and can optionally run evaluations against API-hosted models. SimpleAudit does not collect or transmit user data by default and is designed for minimal setup.

</div>

See the standards and best practices for creating custom test scenarios.

Why SimpleAudit?

| Tool | Complexity | Dependencies | Cost | Approach | |------|------------|--------------|------|----------| | SimpleAudit | ⭐ Simple | 2 packages | $ Low | Adversarial probing | | Petri | ⭐⭐⭐ Complex | Many | $$$ High | Multi-agent framework | | RAGAS | ⭐⭐ Medium | Several | Free | Metrics only | | Custom | ⭐⭐⭐ Complex | Varies | Varies | Build from scratch |

</div>

Installation

Install from PyPI (recommended):

pip install -U simpleaudit

# With plotting support
pip install -U simpleaudit[plot]

Install from GitHub (for latest development features):

pip install -U git+https://github.com/kelkalot/simpleaudit.git

Quick Start

from simpleaudit import ModelAuditor

# Audit HuggingFace model using GPT-4o as judge
auditor = ModelAuditor(
    # Required: Target model configuration
    # First: ollama run hf.co/NbAiLab/borealis-4b-instruct-preview-gguf:BF16
    model="hf.co/NbAiLab/borealis-4b-instruct-preview-gguf:BF16",  # Target model name/identifier
    provider="ollama",  # Target provider (ollama, openai, anthropic, etc.)
    # api_key=None,  # Target API key (uses env var if not provided)
    # base_url=None,  # Custom base URL for target API
    # system_prompt="You are a helpful assistant.",  # System prompt for target model
    
    # Required: Judge model configuration
    judge_model="gpt-4o",  # Judge model name (usually more capable)
    judge_provider="openai",  # Judge provider (can differ from target)
    # judge_api_key=None,  # Judge API key (uses env var if not provided)
    # judge_base_url=None,  # Custom base URL for judge API
    
    # Auditing configuration
    # verbose=False,  # Print detailed logs (default: False)
    # show_progress=True,  # Show progress bars (default: True)
)

# Run built-in safety scenarios
results = await auditor.run_async("safety", max_turns=5, max_workers=10)  # Jupyter / async context
# results = auditor.run("safety", max_turns=5, max_workers=10)  # Script / sync context

# View results
results.summary()
results.plot()
results.save("./my_audit_results/audit_results.json")

💡 View results interactively:

# Option 1: Run directly with uvx (no installation needed, requires uv)
uvx simpleaudit[visualize] serve --results_dir ./my_audit_results

# Option 2: Install and run locally
pip install simpleaudit[visualize]
simpleaudit serve --results_dir ./my_audit_results

This will spin-up a local web server to explore results with scenario details. 👉 Check for live demo. See visualization/README.md for more options and features.

Note: Option 1 requires uv to be installed (install guide).

Running Experiments

Run the same scenario pack across multiple models and compare results.

from simpleaudit import AuditExperiment

experiment = AuditExperiment(
    models=[
        {
            "model": "gpt-4o-mini",
            "provider": "openai",
            "system_prompt": "Be helpful and safe.",
            # "api_key": "sk-...",  # uses env var if not provided
            # "base_url": "https://api.openai.com/v1",  # Optional custom API endpoint
        },
        {
            "model": "claude-sonnet-4-20250514",
            "provider": "anthropic",
            "system_prompt": "Be helpful and safe.",
            # "api_key": "sk-...",  #uses env var if not provided
            # "base_url": "https://api.anthropic.com/v1",  # Optional custom API endpoint
        },
    ],
    judge_model="gpt-4o",
    judge_provider="openai",
    # judge_api_key="",
    # judge_base_url="https://api.openai.com/v1",
    show_progress=True,
    verbose=True,
)

# Script / sync context
results_by_model = experiment.run("safety", max_workers=10)

# Jupyter / async context
# results_by_model = await experiment.run_async("safety", max_workers=10)

for model_name, results in results_by_model.items():
    print(f"\n===== {model_name} =====")
    results.summary()

Using Different Providers

Supported providers include: Anthropic, Azure, Azure OpenAI, Bedrock, Cerebras, Cohere, Databricks, DeepSeek, Fireworks, Gateway, Gemini, Groq, Hugging Face, Inception, Llama, Llama.cpp, Llamafile, LM Studio, Minimax, Mistral, Moonshot, Nebius, Ollama, OpenAI, OpenRouter, Perplexity, Platform, Portkey, SageMaker, SambaNova, Together, Vertex AI, Vertex AI Anthropic, vLLM, Voyage, Watsonx, xAI, Z.ai and many more.

SimpleAudit supports any provider supported by any-llm-sdk. Just specify the provider and any required API key. If the provider isn't installed, you will be prompted to install it.

# Audit GPT-4o-mini using Claude as judge
auditor = ModelAuditor(
    model="gpt-4o-mini",
    provider="openai",           # Uses OPENAI_API_KEY env var
    judge_model="claude-sonnet-4-20250514",
    judge_provider="anthropic",  # Uses ANTHROPIC_API_KEY env var
)

# Audit Claude using GPT-4o as judge
auditor = ModelAuditor(
    model="claude-sonnet-4-20250514",
    provider="anthropic",        # Uses ANTHROPIC_API_KEY env var
    judge_model="gpt-4o",
    judge_provider="openai",     # Uses OPENAI_API_KEY env var
)

# Any other provider - see all at https://mozilla-ai.github.io/any-llm/providers
auditor = ModelAuditor(
    model="model-name",
    provider="your-provider",
    judge_model="more-capable-model",  # Use a different, ideally more capable model
    judge_provider="judge-provider",
)

Local Models (No Target API Key Required)

# Audit your own custom HuggingFace model via Ollama, judged by GPT-4o 
# Audit

Simpleaudit

Install / Use

README