evals-context
Provides context about the Roo Code evals system structure in this monorepo
Install / Use
npx skills add foryourhealth111-pixel/Vibe-Skills --skill evals-contextInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
OperationsSupported Platforms
Tags
Our assessment of evals-context
evals-context scores 91/100 on our quality scale, 246th of 745 Operations skills we index (top 34%).
Its SKILL.md is 7.3 KB long, well organised into 22 sections with 6 code examples: a thorough specification that gives an agent plenty to work with.
With 3,433 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 3 months ago, so evals-context is actively maintained.
- Our last check on 2026-10-28 found the source still online.
- It is released under the Apache-2.0 license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-09-30. Automated pattern scan on 2026-09-30. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
evals-context compared with similar skills
All 4 of these similar skills score higher than evals-context; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| evals-context (this skill)by foryourhealth111-pixel | 91 | 3.4k | 3mo ago | SKILL.md |
| x402by browser-use | 100 | 121.0k | 3d ago | SKILL.md |
| algorithmic-artby anthropics | 100 | 177.9k | 55d ago | SKILL.md |
| pptxby anthropics | 100 | 177.9k | 55d ago | SKILL.md |
| designby nextlevelbuilder | 100 | 149.0k | 6d ago | SKILL.md |
Frequently asked questions
- How do I install evals-context?
- Run
npx skills add foryourhealth111-pixel/Vibe-Skills --skill evals-context. The install tabs above show the steps for each supported agent. - Which AI agents does evals-context work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is evals-context safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is Apache-2.0-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is evals-context still maintained?
- The repository was last updated about 3 months ago, so evals-context is actively maintained.
Skill content
View source on GitHubname: evals-context description: Provides context about the Roo Code evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).
Evals Codebase Context
When to Use This Skill
Use this skill when the task involves:
- Modifying or debugging the evals execution infrastructure
- Adding new eval exercises or languages
- Working with the evals web interface (apps/web-evals)
- Modifying the public evals display page on roocode.com
- Understanding where evals code lives in this monorepo
When NOT to Use This Skill
Do NOT use this skill when:
- Working on unrelated parts of the codebase (extension, webview-ui, etc.)
- The task is purely about the VS Code extension's core functionality
- Working on the main website pages that don't involve evals
Key Disambiguation: Two "Evals" Locations
This monorepo has two distinct evals-related locations that can cause confusion:
| Component | Path | Purpose |
| --------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------- |
| Evals Execution System | packages/evals/ | Core eval infrastructure: CLI, DB schema, Docker configs |
| Evals Management UI | apps/web-evals/ | Next.js app for creating/monitoring eval runs (localhost:3446) |
| Website Evals Page | apps/web-roo-code/src/app/evals/ | Public roocode.com page displaying eval results |
| External Exercises Repo | Roo-Code-Evals | Actual coding exercises (NOT in this monorepo) |
Directory Structure Reference
packages/evals/ - Core Evals Package
packages/evals/
├── ARCHITECTURE.md # Detailed architecture documentation
├── ADDING-EVALS.md # Guide for adding new exercises/languages
├── README.md # Setup and running instructions
├── docker-compose.yml # Container orchestration
├── Dockerfile.runner # Runner container definition
├── Dockerfile.web # Web app container
├── drizzle.config.ts # Database ORM config
├── src/
│ ├── index.ts # Package exports
│ ├── cli/ # CLI commands for running evals
│ │ ├── runEvals.ts # Orchestrates complete eval runs
│ │ ├── runTask.ts # Executes individual tasks in containers
│ │ ├── runUnitTest.ts # Validates task completion via tests
│ │ └── redis.ts # Redis pub/sub integration
│ ├── db/
│ │ ├── schema.ts # Database schema (runs, tasks)
│ │ ├── queries/ # Database query functions
│ │ └── migrations/ # SQL migrations
│ └── exercises/
│ └── index.ts # Exercise loading utilities
└── scripts/
└── setup.sh # Local macOS setup script
apps/web-evals/ - Evals Management Web App
apps/web-evals/
├── src/
│ ├── app/
│ │ ├── page.tsx # Home page (runs list)
│ │ ├── runs/
│ │ │ ├── new/ # Create new eval run
│ │ │ └── [id]/ # View specific run status
│ │ └── api/runs/ # SSE streaming endpoint
│ ├── actions/ # Server actions
│ │ ├── runs.ts # Run CRUD operations
│ │ ├── tasks.ts # Task queries
│ │ ├── exercises.ts # Exercise listing
│ │ └── heartbeat.ts # Controller health checks
│ ├── hooks/ # React hooks (SSE, models, etc.)
│ └── lib/ # Utilities and schemas
apps/web-roo-code/src/app/evals/ - Public Website Evals Page
apps/web-roo-code/src/app/evals/
├── page.tsx # Fetches and displays public eval results
├── evals.tsx # Main evals display component
├── plot.tsx # Visualization component
└── types.ts # EvalRun type (extends packages/evals types)
This page displays eval results on the public roocode.com website. It imports types from @roo-code/evals but does NOT run evals.
Architecture Overview
The evals system is a distributed evaluation platform that runs AI coding tasks in isolated VS Code environments:
┌─────────────────────────────────────────────────────────────┐
│ Web App (apps/web-evals) ──────────────────────────────── │
│ │ │
│ ▼ │
│ PostgreSQL ◄────► Controller Container │
│ │ │ │
│ ▼ ▼ │
│ Redis ◄───► Runner Containers (1-25 parallel) │
└─────────────────────────────────────────────────────────────┘
Key components:
- Controller: Orchestrates eval runs, spawns runners, manages task queue (p-queue)
- Runner: Isolated Docker container with VS Code + Roo Code extension + language runtimes
- Redis: Pub/sub for real-time events (NOT task queuing)
- PostgreSQL: Stores runs, tasks, metrics
Common Tasks Quick Reference
Adding a New Eval Exercise
- Add exercise to Roo-Code-Evals repo (external)
- See
packages/evals/ADDING-EVALS.mdfor structure
Modifying Eval CLI Behavior
Edit files in packages/evals/src/cli/:
runEvals.ts- Run orchestrationrunTask.ts- Task executionrunUnitTest.ts- Test validation
Modifying the Evals Web Interface
Edit files in apps/web-evals/src/:
app/runs/new/new-run.tsx- New run formactions/runs.ts- Run server actions
Modifying the Public Evals Display Page
Edit files in apps/web-roo-code/src/app/evals/:
Database Schema Changes
- Edit
packages/evals/src/db/schema.ts - Generate migration:
cd packages/evals && pnpm drizzle-kit generate - Apply migration:
pnpm drizzle-kit migrate
Running Evals Locally
# From repo root
pnpm evals
# Opens web UI at http://localhost:3446
Ports (defaults):
- PostgreSQL: 5433
- Redis: 6380
- Web: 3446
Testing
# packages/evals tests
cd packages/evals && npx vitest run
# apps/web-evals tests
cd apps/web-evals && npx vitest run
Key Types/Exports from @roo-code/evals
The package exports are defined in packages/evals/src/index.ts:
- Database queries:
getRuns,getTasks,getTaskMetrics, etc. - Schema types:
Run,Task,TaskMetrics - Used by both
apps/web-evalsandapps/web-roo-code
Related Skills
x402
121.0kSet up Browser Use Cloud payments with x402 — pay per request from a crypto wallet (USDC on Base mainnet), no signup or API key.
algorithmic-art
177.9kCreating algorithmic art using p5.js with seeded randomness and interactive parameter exploration. Use this when users request creating art using code, generative art, algorithmic art, flow fields, or particle systems.
pptx
177.9kUse this skill any time a .pptx or .potx file is involved in any way — as input, output, or both. This includes: creating slide decks, pitch decks, or presentations; reading, parsing, or extracting text from any .pptx or .potx file (even if the extracted content will be used elsewhere, like in an em…
design
149.0kComprehensive design skill: brand identity, design tokens, UI styling, logo generation (55 styles, Gemini, Atlas Cloud, or MuAPI AI), corporate identity program (50 deliverables, CIP mockups), HTML presentations (Chart.js), banner design (22 styles, social/ads/web/print), icon design (15 styles, SVG…
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.