ai-sre-incident-response
Build AI-focused SRE incident response practices for LLM outages, degraded quality, runaway cost events, and safety regressions.
Install / Use
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill ai-sre-incident-responseInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AI & Machine LearningSupported Platforms
Tags
Our assessment of ai-sre-incident-response
ai-sre-incident-response scores 87/100 on our quality scale, 504th of 950 AI & Machine Learning skills we index.
Its SKILL.md is 11 KB long, well organised into 28 sections with 7 code examples: a thorough specification that gives an agent plenty to work with.
With 1,113 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 6 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
- Our last check on 2026-10-11 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 98/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-07. Automated pattern scan on 2026-10-07. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
ai-sre-incident-response compared with similar skills
All 4 of these similar skills score higher than ai-sre-incident-response; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| ai-sre-incident-response (this skill)by BagelHole | 87 | 1.1k | 6mo ago | SKILL.md |
| claude-memby thedotmack | 100 | 117.6k | today | CLAUDE.md |
| Understand-Anythingby Egonex-AI | 100 | 92.8k | 1d ago | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.8k | today | CLAUDE.md |
| CowAgentby zhayujie | 100 | 48.0k | today | CLAUDE.md |
Frequently asked questions
- How do I install ai-sre-incident-response?
- Run
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill ai-sre-incident-response. The install tabs above show the steps for each supported agent. - Which AI agents does ai-sre-incident-response work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is ai-sre-incident-response safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 98/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is ai-sre-incident-response still maintained?
- The repository was last updated about 6 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
Skill content
View source on GitHubname: ai-sre-incident-response description: Build AI-focused SRE incident response practices for LLM outages, degraded quality, runaway cost events, and safety regressions. license: MIT metadata: author: devops-skills version: "1.0"
AI SRE Incident Response
Apply SRE rigor to AI systems where incidents include quality regressions, unsafe outputs, and budget explosions.
When to Use This Skill
- An LLM endpoint begins returning degraded or hallucinated answers
- Token spend spikes beyond budget thresholds
- A model provider goes down and traffic must fail over
- Safety guardrails fire at abnormal rates
- A new model deployment causes latency or accuracy regression
Prerequisites
- Prometheus and Alertmanager deployed with scrape targets for AI services
- Grafana dashboards for golden signals (latency, error rate, cost, quality)
- On-call rotation configured in PagerDuty, Opsgenie, or equivalent
- Runbook repository accessible to responders
- Rollback mechanism for model and prompt versions (GitOps or feature flags)
AI Incident Classes
- Availability incident: model/provider unavailable, timeout storm.
- Quality incident: answer accuracy or tool success drops below SLO.
- Safety incident: harmful or policy-violating outputs increase.
- Cost incident: unexpected token or provider spend spike.
Severity Framework
| Severity | Criteria | Response Time | Notification | |----------|----------|---------------|--------------| | SEV1 | User-facing outage, compliance risk, data leak | 5 min | Page on-call + incident commander | | SEV2 | Major degradation in key flows | 15 min | Page on-call | | SEV3 | Limited impact or internal-only issue | 1 hour | Slack alert | | SEV4 | Cosmetic or low-priority regression | Next business day | Ticket |
Golden Signals for AI Services
- Request success rate
- Latency (queue + generation + tool execution)
- Hallucination/groundedness proxy metrics
- Cost per minute and per tenant
- Guardrail violation rate
Prometheus Alert Rules
# prometheus-ai-alerts.yaml
groups:
- name: ai-service-alerts
rules:
- alert: ModelEndpointDown
expr: up{job="llm-inference"} == 0
for: 2m
labels:
severity: sev1
annotations:
summary: "LLM inference endpoint {{ $labels.instance }} is down"
runbook_url: "https://runbooks.internal/ai/model-outage"
- alert: HighHallucinationRate
expr: |
rate(llm_hallucination_detected_total[10m])
/ rate(llm_requests_total[10m]) > 0.15
for: 5m
labels:
severity: sev2
annotations:
summary: "Hallucination rate above 15% for {{ $labels.model }}"
runbook_url: "https://runbooks.internal/ai/quality-regression"
- alert: TokenCostExplosion
expr: |
sum(rate(llm_token_cost_dollars[5m])) by (tenant)
> 0.50
for: 3m
labels:
severity: sev2
annotations:
summary: "Token spend exceeds $0.50/min for tenant {{ $labels.tenant }}"
runbook_url: "https://runbooks.internal/ai/cost-spike"
- alert: LatencyP95Exceeded
expr: |
histogram_quantile(0.95,
rate(llm_request_duration_seconds_bucket[5m])
) > 5
for: 5m
labels:
severity: sev2
annotations:
summary: "LLM p95 latency exceeds 5s for {{ $labels.service }}"
- alert: GuardrailViolationSpike
expr: |
rate(llm_guardrail_violations_total[10m])
/ rate(llm_requests_total[10m]) > 0.05
for: 5m
labels:
severity: sev1
annotations:
summary: "Guardrail violations above 5% for {{ $labels.model }}"
runbook_url: "https://runbooks.internal/ai/safety-incident"
- alert: ModelQualityDrop
expr: |
llm_eval_score{metric="groundedness"} < 0.70
for: 10m
labels:
severity: sev2
annotations:
summary: "Groundedness score dropped below 0.70 for {{ $labels.model }}"
- alert: ProviderErrorRateHigh
expr: |
rate(llm_provider_errors_total[5m])
/ rate(llm_provider_requests_total[5m]) > 0.10
for: 3m
labels:
severity: sev2
annotations:
summary: "Provider {{ $labels.provider }} error rate above 10%"
Response Playbooks
Model Outage Runbook
TRIGGER: ModelEndpointDown fires for > 2 minutes
RESPONDER: On-call AI platform engineer
1. Acknowledge alert in PagerDuty.
2. Check provider status page (e.g., status.openai.com).
3. Verify network connectivity:
curl -s -o /dev/null -w "%{http_code}" https://api.provider.com/health
4. If provider is down:
a. Enable fallback model route in gateway config.
b. kubectl set env deployment/llm-gateway FALLBACK_ENABLED=true
c. Verify fallback traffic is flowing via Grafana dashboard.
5. If self-hosted model is down:
a. Check pod status: kubectl get pods -l app=llm-inference -n ai
b. Check GPU health: kubectl logs -l app=llm-inference --tail=50
c. Restart if OOM: kubectl rollout restart deployment/llm-inference -n ai
6. Freeze all deployments:
kubectl annotate deployment --all deploy-freeze=true -n ai
7. Communicate ETA in #incident-channel.
8. When resolved, unfreeze and run smoke tests.
Quality Regression Runbook (Hallucination Spike)
TRIGGER: HighHallucinationRate or ModelQualityDrop fires
RESPONDER: On-call AI engineer + ML lead
1. Acknowledge alert. Open incident ticket.
2. Identify scope:
- Which model version? Check deployment metadata.
- Which routes/tenants affected? Filter by labels in Grafana.
3. Check recent changes:
- Model version promotion in last 24h?
- Prompt template changes in last 24h?
- Retrieval index rebuild in last 24h?
4. If recent model change:
kubectl rollout undo deployment/llm-inference -n ai
5. If recent prompt change:
git revert <commit> && git push # triggers GitOps redeploy
6. Increase trace sampling to 100% for affected route:
kubectl set env deployment/llm-gateway TRACE_SAMPLE_RATE=1.0
7. Run offline eval suite against current production:
python run_evals.py --target prod --suite quality --compare baseline
8. Confirm metrics return to baseline before closing.
Token Cost Explosion Runbook
TRIGGER: TokenCostExplosion fires
RESPONDER: On-call platform engineer
1. Identify top consumers:
Query: topk(10, sum(rate(llm_token_cost_dollars[15m])) by (tenant, model, route))
2. Check for runaway loops:
- Agent retry storms (exponential token growth per request)
- Missing max_tokens caps on new routes
- Cache bypass due to config change
3. Apply immediate caps:
kubectl patch configmap llm-quotas -n ai --patch '
data:
max_tokens_per_request: "4096"
rpm_limit: "60"
'
4. Enable semantic cache if disabled:
kubectl set env deployment/llm-gateway CACHE_ENABLED=true
5. Route traffic to cheaper model tier:
kubectl set env deployment/llm-gateway DEFAULT_MODEL=gpt-4o-mini
6. Notify affected tenants of temporary limits.
7. Open postmortem with cost attribution analysis.
Escalation Procedures
Level 1 (0-15 min): On-call AI platform engineer
Level 2 (15-30 min): AI platform team lead + affected product owner
Level 3 (30-60 min): Engineering director + security (if safety incident)
Level 4 (60+ min): VP Engineering + legal (if compliance/data incident)
Safety incidents always start at Level 2 minimum.
Provider-side incidents: open support ticket immediately at Level 1.
Detection Queries (PromQL)
# Request success rate by model
1 - (
sum(rate(llm_requests_total{status="error"}[5m])) by (model)
/ sum(rate(llm_requests_total[5m])) by (model)
)
# Cost per successful answer
sum(rate(llm_token_cost_dollars[5m])) by (route)
/ sum(rate(llm_requests_total{status="success"}[5m])) by (route)
# Hallucination rate trend (1h window, 5m steps)
rate(llm_hallucination_detected_total[1h])
/ rate(llm_requests_total[1h])
# Latency breakdown by stage
histogram_quantile(0.95, rate(llm_retrieval_duration_seconds_bucket[5m]))
histogram_quantile(0.95, rate(llm_generation_duration_seconds_bucket[5m]))
histogram_quantile(0.95, rate(llm_tool_execution_duration_seconds_bucket[5m]))
# Tenant cost leaderboard
topk(10, sum(rate(llm_token_cost_dollars[1h])) by (tenant))
Postmortem Requirements
- Timeline with detector and responder timestamps
- Blast radius by tenant and feature
- Missed signals and alert tuning actions
- Concrete hardening tasks with owners and due dates
- Cost impact (dollars, tokens, affected requests)
- Customer communication log
Postmortem Template
## Incident Summary
- **Severity**: SEVx
- **Duration**: start_time - end_time (Xh Ym)
- **Detection**: How was it detected? (alert / customer report / manual)
- **Impact**: X tenants, Y requests, $Z cost
## Timeline
| Time (UTC) | Event |
|------------|-------|
| HH:MM | Alert fired |
| HH:MM | Responder acknowledged |
| HH:MM | Root cause identified |
| HH:MM | Mitigation applied |
| HH:MM | Incident resolved |
## Root Cause
[Description]
## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| Tune alert threshold | @engineer | YYYY-MM-DD | Open |
| Add fallback route | @platform | YYYY-MM-DD | Open |
Chaos Engineering for AI Systems
Regularly test incident readiness:
- Provider failover drill: block provider API at network level, verify fallback activates within SLO.
- Model rollback drill: deploy known-bad model version, verify automated quality gate catches it.
- Cost cap drill: simulate runaway token usage, verify quotas trigger before budget threshold.
- Cache failure drill: disable semantic cache, verify system degrades gracefully.
Troubleshooting
| Symptom | Check | Fix | |---------|-------|-----| | All requests timing out | Provider status page, DNS resolution | Enable fallback provider | | Gradual quality decline | Recent model/prompt deployments | Roll back to last known good | | Sudden cost spike | Per-tenant token usage dashboard | Apply emergency token caps | | Guardrail violations spike | Model version, prompt injection logs | Enable stricter input filtering | | Intermittent 503 errors | Pod restarts, GPU OOM events | Increase memory limits or reduce batch size |
Related Skills
- incident-response - Standard incident process and evidence
- alerting-oncall - Paging and escalation policy
- llm-cost-optimization - Spend controls and efficiency patterns
- agent-observability - Instrument requests, traces, and costs
- rag-observability-evals - RAG quality monitoring
Related Skills
claude-mem
117.6kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
Understand-Anything
92.8kGraphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
headroom
79.8kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
CowAgent
48.0kOpen-source personal AI assistant & Agent Harness. Plans tasks, runs tools and skills, self-evolves with memory and knowledge. Multi-agent, multi-model, multi-channel. Lightweight, extensible, one-line install.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
