<div align="center">

🧠 Andrej Karpathy 2026 — AI Systems Engineering

From flashy demos → production-grade autonomous systems

8 production-grade projects • 33,600+ lines of Python • 437 source files • All local, all free

</div>

---

🎯 What Is This?

A hands-on engineering curriculum inspired by Andrej Karpathy's 2026 roadmap focused on the boring things that actually compound in modern AI systems:

| Skill | Where You Learn It | Codebase Link | |---|---|---| | Context Engineering | CodeLensAI — deciding what context matters for a PR review | ./CodeLensAI | | Tool Design | QueryForge — building narrow, safe, composable tools | ./queryforge | | Agent Orchestration | VentureMind — coordinating 5 specialist agents | ./VentureMind | | Eval Discipline | EvalOps — CI/CD for prompts with regression detection | ./EvalOps | | Agent Learning | Agent Learning Pilot — measuring if agents improve with feedback | ./agent-learning-pilot | | Reliability Engineering | FailSafe Agent — circuit breakers, fallback models, audit trails | ./failsafe-agent | | Harness Engineering | AutoForge — sandboxed code execution with verification loops | ./autoforge | | Full-Stack AI Ops | SentinelOps — all of the above in one capstone system | ./SentinelOps |

Every project runs 100% locally using Ollama — zero paid API keys required.

---

⚡ Project Suite Matrix

| # | Project | Core Domain | Port | Dashboard URL | Direct Docs | |---|---|---|---|---|---| | 1 | EvalOps | Prompt Regression & Evals | 8000 | localhost:8000/dashboard | EvalOps/README.md | | 2 | QueryForge | Multi-Tool Agent Routing | 8001 | localhost:8001 | queryforge/README.md | | 3 | VentureMind | Due Diligence Multi-Agent | 8002 | localhost:8002 | VentureMind/README.md | | 4 | CodeLensAI | Context Packing PR Review | 8003 | localhost:8003 | CodeLensAI/README.md | | 5 | Agent Learning Pilot | Feedback & Retention Science | 8004 | localhost:8004 | agent-learning-pilot/README.md | | 6 | FailSafe Agent | Fault-Tolerant Support Agent | 8005 | localhost:8005 | failsafe-agent/README.md | | 7 | AutoForge | Autonomous Coding Sandbox | 8006 | localhost:8006 | autoforge/README.md | | 8 | SentinelOps ★ | SRE DevOps Incident Response | 8007 | localhost:8007 | SentinelOps/README.md |

---

📐 Architecture Overview

graph TB
    subgraph "Project 1: EvalOps"
        E1["Golden Datasets"] --> E2["Eval Runner"]
        E2 --> E3["LLM-as-Judge + Scoring"]
        E3 --> E4["Regression Engine"]
        E4 --> E5["Dashboard :8000"]
    end

    subgraph "Project 2: QueryForge"
        Q1["Natural Language Query"] --> Q2["Tool Router"]
        Q2 --> Q3["SQL / Web / PDF / API"]
        Q3 --> Q4["Result Synthesis"]
        Q4 --> Q5["Dashboard :8001"]
    end

    subgraph "Project 3: VentureMind"
        V1["Startup Name"] --> V2["Orchestrator"]
        V2 --> V3["Market + Competitor + Finance + Legal + Summary Agents"]
        V3 --> V4["Due Diligence Report"]
        V4 --> V5["Dashboard :8002"]
    end

    subgraph "Project 4: CodeLensAI"
        C1["PR / Code Diff"] --> C2["Context Packer"]
        C2 --> C3["Review Agent"]
        C3 --> C4["Feedback + Scores"]
        C4 --> C5["Dashboard :8003"]
    end

    subgraph "Project 5: Agent Learning Pilot"
        A1["Benchmark Tasks"] --> A2["Agent + Feedback Loop"]
        A2 --> A3["Retention + Transfer Analysis"]
        A3 --> A4["Dashboard :8004"]
    end

    subgraph "Project 6: FailSafe Agent"
        F1["Support Request"] --> F2["Circuit Breaker"]
        F2 --> F3["Primary / Fallback LLM"]
        F3 --> F4["SHA-256 Audit Ledger"]
        F4 --> F5["Dashboard :8005"]
    end

    subgraph "Project 7: AutoForge"
        AF1["GitHub Issue"] --> AF2["Code Agent"]
        AF2 --> AF3["Sandbox Execution"]
        AF3 --> AF4["Test + Verify"]
        AF4 --> AF5["Dashboard :8006"]
    end

    subgraph "Project 8: SentinelOps ★"
        S1["Incident Alert"] --> S2["Multi-Agent Orchestrator"]
        S2 --> S3["Log + Metrics + Runbook Tools"]
        S3 --> S4["Resolution + Escalation"]
        S4 --> S5["Dashboard :8007"]
    end

---

🗂️ Project Catalog

1️⃣ EvalOps — LLM Regression Testing Platform

CI/CD for your prompts and AI agents.

| Attribute | Specification | |---|---| | Directory | ./EvalOps | | Port | 8000 | | Core Skill | Eval Discipline | | Key Features | Golden datasets, LLM-as-judge, pairwise comparison, regression detection, human feedback loop | | Tech | FastAPI, Ollama, SQLite, Chart.js, Watchdog | | Code Volume | 3,301 lines across 25 files |

What it does: Run benchmark evaluations against any local LLM model. Score outputs with deterministic scoring (Jaccard, exact match) and LLM-as-judge. Detect regressions automatically when prompt changes degrade quality. Review everything in a live dashboard.

cd EvalOps
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
make seed && make run    # → http://localhost:8000/dashboard

---

2️⃣ QueryForge — Multi-Tool Research Agent

Ask a question, the agent picks the tools.

| Attribute | Specification | |---|---| | Directory | ./queryforge | | Port | 8001 | | Core Skill | Tool Design | | Key Features | SQL query generation, web search, PDF summarization, chart rendering, tool selection reasoning | | Tech | Google ADK, FastAPI, Ollama, aiosqlite, matplotlib | | Code Volume | 3,821 lines across 34 files |

What it does: Type a natural language question like "Find our highest churn customer segment and compare it with industry trends." The agent decides which tools to call (SQL, web search, PDF reader, API), executes them in the right order, and synthesizes a coherent answer.

cd queryforge
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
uvicorn src.api.main:app --port 8001 --reload  # → http://localhost:8001

---

3️⃣ VentureMind — Startup Due Diligence Multi-Agent System

5 specialist agents analyze a startup before investment.

| Attribute | Specification | |---|---| | Directory | ./VentureMind | | Port | 8002 | | Core Skill | Agent Orchestration | | Key Features | Parallel agent execution, market research, competitor analysis, financial modeling, legal review, report synthesis | | Tech | Google ADK, FastAPI, Ollama, aiosqlite | | Code Volume | 6,605 lines across 51 files |

What it does: Provide a startup name and description. Five specialized agents (Market Research, Competitor Analysis, Financial Analysis, Legal Review, Summarization) run in parallel, share findings, and produce a comprehensive due diligence report.

cd VentureMind
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
python scripts/seed_demo_samples.py
uvicorn src.api.app:app --port 8002 --reload  # → http://localhost:8002

---

4️⃣ CodeLensAI — Autonomous PR Review Agent

AI reviewer that knows what context matters.

| Attribute | Specification | |---|---| | Directory | ./CodeLensAI | | Port | 8003 | | Core Skill | Context Engineering | | Key Features | Code diff analysis, context compression, review scoring, security/performance flags, review history | | Tech | FastAPI, Ollama, SQLite | | Code Volume | 4,238 lines across 34 files |

What it does: Paste a code diff or connect a repo. The agent analyzes changed files, retrieves relevant architecture docs and coding standards, compresses context to fit the model window, and produces a detailed code review with scores for correctness, security, and maintainability.

cd CodeLensAI
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
python codelens/scripts/seed_demo_reviews.py
uvicorn codelens.src.api.main:app --port 8003 --reload  # → http://localhost:8003

---

5️⃣ Agent Learning Pilot — Can Agents Learn From Feedback?

A controlled experiment measuring agent improvement over time.

| Attribute | Specification | |---|---| | Directory | ./agent-learning-pilot | | Port | 8004 | | Core Skill | Agent Learning & Evaluation | | Key Features | Controlled A/B experiments, retention curves, transfer learning analysis, cost-vs-capability tracking | | Tech | FastAPI, Ollama, pandas, matplotlib | | Code Volume | 4,759 lines across 189 files |

What it does: Runs controlled experiments to measure whether coding agents genuinely improve when given structured feedback. Tracks retention (do improvements stick?), transfer (do improvements generalize?), and cost-effectiveness across experiment conditions.

cd agent-learning-pilot
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
python scripts/serve_results.py 8004  # → http://localhost:8004

---

6️⃣ FailSafe Agent — Resilient AI Support Agent

When the model fails, the system doesn't.

| Attribute | Specification | |---|---| | Directory | ./failsafe-agent | | Port | 8005 | | Core Skill | Reliability Engineering | | Key Features | Circuit breakers, fallback LLM chains, retry with exponential backoff, SHA-256 audit ledger, human escalation | | Tech | FastAPI, Ollama, SQLite, hashlib | | Code Volume | 5,317 lines across 54 files |

What it does: A customer support agent that handles refund requests, policy lookups, and payment actions — with production-grade resilience. If the primary LLM times out, a circuit breaker trips and falls back to a deterministic engine. Every action is recorded in a cryptographic SHA-256 audit ledger.

cd failsafe-agent
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
uvicorn src.main:app --port 8005 --reload  # → http://localhost:8005

---

7️⃣ AutoForge — Autonomous Software Engineering Sandbox

Agent writes code, sandbox runs it, tests verify it.

| Attribute | Specification | |---|---| | Directory | ./autoforge | | Port | 8006 | | Core Skill | Harness Engineering | | Key Features | Issue-to-PR pipeline, sandboxed code execution, test verification loops, artifact logging, rollback | | Tech | FastAPI, Ollama, SQLite, subprocess isolation | | Code Volume | 4,747 lines across 41 files |

**What it d