Awesome AI Safety

A curated collection of resources for building safe, aligned, and trustworthy AI systems.

Covers the full stack of AI safety: alignment, interpretability, evaluation, formal verification, governance, and verifiable AI. Focused on working tools, code, and actionable resources, not just papers.

---

Contents

• Alignment & Training
• Interpretability & Mechanistic Analysis
• Red Teaming & Evaluation
• Formal Verification & Robustness
• Verifiable AI & ZKML
• Governance, Policy & Compliance
• Safety Benchmarks & Datasets
• Foundational Papers
• Organizations
• Courses & Educational Resources

---

Alignment & Training

Tools and frameworks for aligning AI systems with human values and intentions.

RLHF & Preference Optimization

• TRL - Hugging Face library for RLHF, DPO, PPO, and SFT training of language models. The standard open-source alignment training library.
• OpenRLHF - High-performance RLHF framework built on Ray, vLLM, and DeepSpeed. Scales to 70B+ models.
• Direct Preference Optimization - Reference implementation of DPO, which simplifies RLHF by eliminating the separate reward model.
• DeepSpeed-Chat - Microsoft's end-to-end RLHF pipeline (SFT, reward modeling, PPO) with DeepSpeed integration.
• RewardBench - Allen AI benchmark for evaluating reward models used in alignment training.

Guardrails & Output Safety

• NeMo Guardrails - NVIDIA toolkit for adding programmable safety rails to LLM applications.
• Guardrails AI - Framework for adding structure, type, and quality guarantees to LLM outputs.
• LLM Guard - Self-hosted toolkit for sanitizing and securing LLM interactions. Covers prompt injection detection, PII filtering, toxicity checks.
• Llama Guard - Meta's safety classifier models for content moderation of LLM inputs and outputs. Part of PurpleLlama.
• Alignment Handbook - Hugging Face recipes for aligning language models with human and AI preferences. Practical guides for SFT, DPO, and RLHF.
• ai-evaluation - LLM evaluation framework with 50+ metrics, LLM-as-Judge, and guardrail scanners (jailbreak, PII, injection).
• Future AGI - Self-hostable end-to-end agent engineering platform with tracing, evals, guardrails, and gateway.

Representation & Activation Engineering

• Representation Engineering - Top-down approach to AI transparency. Read and control model behavior via representation-level interventions.
• repeng - Library for building RepE control vectors with language models. Steer model behavior at inference time.
• Circuit Breakers - Interrupt harmful model behavior by operating on internal representations rather than output filtering.
• Honest LLaMA (ITI) - Inference-Time Intervention: shift model activations to elicit more truthful answers.

---

Interpretability & Mechanistic Analysis

Understanding what neural networks learn and how they compute.

Libraries & Frameworks

• TransformerLens - The primary library for mechanistic interpretability of GPT-style models. Hook into and analyze any transformer internal.
• SAELens - Train, analyze, and use Sparse Autoencoders on language models. Central to much of the recent feature extraction research.
• nnsight - Interpret and manipulate neural network internals. Supports causal interventions and tracing in large models (David Bau's group).
• pyvene - Stanford NLP's unified framework for activation patching, causal tracing, and representation engineering.
• CircuitsVis - Visualization tools for attention patterns and circuit-level interpretability.
• Baukit - Toolkit for editing and understanding neural network representations.
• OpenAI Sparse Autoencoder - OpenAI's implementation for extracting interpretable features via sparse autoencoders.

Platforms & Resources

• Neuronpedia - Interactive platform for exploring individual neuron and feature behaviors in language models.
• Transformer Circuits Thread - Anthropic's ongoing publication series on mechanistic interpretability of transformers.

---

Red Teaming & Evaluation

Testing AI systems for dangerous capabilities, vulnerabilities, and failure modes.

Automated Red Teaming

• garak - LLM vulnerability scanner. Probes for hallucination, toxicity, prompt injection, data leakage, and more.
• HarmBench - Standardized evaluation framework for automated red teaming of LLMs (Center for AI Safety).
• PurpleLlama - Meta's safety suite: CyberSecEval for cybersecurity risk evaluation, Llama Guard for content safety.
• Anthropic Evals - Anthropic's public evaluation suite for dangerous capabilities and safety properties.
• promptfoo - LLM evaluation and red-teaming tool with safety-specific plugins for toxicity, PII, and jailbreak testing.

Evaluation Frameworks

• Inspect AI - UK AI Safety Institute's framework for evaluating AI capabilities and alignment. Task-based and extensible.
• METR Task Standard - Task format for evaluating dangerous autonomous capabilities (from the team that evaluates frontier models pre-deployment).
• EleutherAI LM Evaluation Harness - De facto standard for LLM benchmarking across hundreds of tasks, including safety-relevant ones.
• Vivaria - METR's tool for running AI agents on evaluation tasks. Used internally for frontier model capability assessments.
• METR Public Tasks - Task collections for evaluating dangerous capabilities of autonomous AI agents.
• traceAI - OpenTelemetry-native tracing for LLM and agent apps with 50+ framework integrations.

Adversarial Attacks & Jailbreaking

• LLM Attacks - Universal and transferable adversarial attacks on aligned language models (GCG attack).
• JailbreakBench - Open robustness benchmark for jailbreaking language models. Tracks attack and defense methods.
• StrongREJECT - Benchmark for evaluating how well models refuse harmful jailbreak attempts.

---

Formal Verification & Robustness

Mathematically proving properties about neural network behavior.

Neural Network Verifiers

• α,β-CROWN - GPU-accelerated neural network verifier using bound propagation and branch-and-bound. Multi-year VNN-COMP winner.
• auto_LiRPA - Automatic Linear Relaxation based Perturbation Analysis. General-purpose certified robustness library underlying α,β-CROWN.
• ERAN - ETH Zurich's certification tool using abstract interpretation. Handles ReLU, sigmoid, tanh, and MaxPool.
• NNV - Verification for deep neural networks and neural network control systems, focused on safety-critical applications.

Certified Robustness

• Randomized Smoothing - Reference implementation of Cohen et al. 2019. Scalable certified robustness via randomized smoothing.
• VNN-COMP - Annual competition benchmarking neural network verification tools. Defines standard benchmarks.

---

Verifiable AI & ZKML

Cryptographic techniques for proving properties about AI systems without revealing their internals.

Zero-Knowledge ML Frameworks

• EZKL - Zero-knowledge proofs of ML model inference. Converts ONNX models to ZK circuits. The most mature ZKML framework.
• Giza / Orion - Open-source framework for provable machine learning. ONNX runtime for verifiable inference on-chain. (Archived)
• RISC Zero zkVM - General-purpose zero-knowledge virtual machine based on RISC-V and zk-STARKs. Supports ML workloads.
• zkml - Proof-of-concept ZKML library using the Halo2 proving system. Converts ONNX models to Halo2 format.

Curated ZKML Resources

• [awesome-zkml](http