---

name: ares-debug description: Diagnose a stuck, slow, or broken Ares operation by triangulating across three data sources — SSM (live ares logs + Redis on EC2), Grafana Loki (historical logs via mcp__grafana__query_loki_logs), and OTEL traces in Tempo. Use when an operation is hung/wedged, a worker keeps crashing, the orchestrator stops making progress, or a task fails with no obvious local clue. Default deployment is EC2 (kali-ares); K8s notes included for completeness.

Debugging Ares

You are debugging a running or recent Ares operation. Pick the cheapest source first; only escalate if it doesn't answer the question.

Read this before you do anything

Do not declare an op healthy from process liveness, NATS/Redis ping, or token-rate alone. A wedged Ares op happily presents as status=running, workers active, Redis green, cache hit ≥80%, and tokens climbing — while making zero external progress for hours. This has happened. Don't repeat it.

The only valid "healthy" verdict requires a comparison:

1. Compare this op's objective state now vs. 60s ago — has_domain_admin, domain compromise count, hosts owned, creds, hashes, vulns exploited. If none changed, that's churn, not progress.
2. Compare this op to recent ops' baseline. Pull ares ops list and look at how long prior ops took to hit DA / 2nd domain. If this op is more than ~2× slower to a milestone the last 3 ops hit, treat it as wedged regardless of token rate.

Token churn is the signature of the LLM re-evaluating the same frozen state every tick; high cache-hit rate (>80%) on a slow op is a symptom of the wedge, not evidence of health.

Worker per-role log mtimes are not a signal. In steady state the orchestrator centralizes everything via NATS into /var/log/ares/orchestrator.log; per-role files (recon.log, cracker.log, etc.) stay near-empty. Don't read into stale mtimes.

Before you propose a code fix

Ares timeline events (evt-exploit-fail-* in ares:op:*:timeline) and "Assistance needed" strings the LLM emits ("the tool schema does not accept X", "current toolset lacks Y", "tool requires password but only hash available") are the failing LLM agent's confabulated explanation of its own failure — not a bug report. The agent does not know its own tool schemas or the orchestrator's dispatch layer, and it will invent plausible-sounding gaps that don't exist.

Before recommending a fix from one:

1. Open the tool wrapper in ares-tools/src/**/*.rs — does the tool actually accept the arg the LLM said was missing?
2. Open the LLM-facing schema in ares-llm/src/tool_registry/**/*.rs — does it declare the field?
3. Open the automation dispatcher in ares-cli/src/orchestrator/automation/*.rs — does it inject the credential/state from Redis into the payload?

If all three already do the thing, the LLM was confabulating. The real failure is elsewhere — the tool ran and hit a Kerberos error, dispatch timed out, worker didn't have the credential in state, etc. Grep the orchestrator log for the actual dispatch record + tool stdout/stderr; those are ground truth. Timeline events are not.

Tight-loop / wedge signatures (grep the orchestrator tail for these first)

Run Step 0, then before drawing any conclusion grep the tail of orchestrator.log for each pattern below. If any hit, that's almost certainly your wedge:

| Pattern (regex) | Means | |---|---| | clearing dedup for retry | Wrapper-level retry loop; same task being re-dispatched every tick | | Dispatching <same_tool> ... <same_target> repeated ≥3× | Automation hot loop with no backoff | | KDC_ERR_TGT_REVOKED\|KDC_ERR_S_PRINCIPAL_UNKNOWN\|KDC_ERR_PREAUTH_FAILED\|TGT has been revoked | Kerberos error that will not self-heal; orchestrator may be retrying anyway | | tool exited with code Some\(0\) followed by stderr content | Zero-exit-with-error: wrapper treats stderr-on-zero-exit as transient and re-tries | | Same task_id shape (e.g. trust_raise_child_<hex>) repeated with distinct hex per tick | Dedup key churning instead of blacklisting | | Processing real-time discoveries count=1 ticking every 5s with no other state change | Orchestrator stuck in discovery-replay loop |

If you don't see these but the op is slow vs. baseline, escalate to Loki / Tempo for cross-tick LLM latency or tool-call stalls.

What goes where

| Source | Latency | Coverage | How to query | |-------------------|----------|-------------------------------------------------|----------------------------------------------------------| | task ec2:status (with AWS_PROFILE=personal AWS_REGION=us-east-1) | seconds | Worker process state, Redis ping | Bash | | task ec2:runtime (same prefix) | seconds | Per-op token/cost/domain banner | Bash | | Loki (Grafana) | seconds | Historical /var/log/ares/*.log + syslog/auth | mcp__grafana__query_loki_logs (datasourceUid loki) | | Tempo (Grafana) | seconds | OTEL traces of LLM calls + tool dispatch | mcp__grafana__* Tempo proxy tools | | SSM task ec2:exec (same prefix) | ~5-15s | Anything on the host (redis-cli, journalctl) | Bash, never tail -f | | task ec2:logs | streaming| Live tail of one role's log | DO NOT use in Claude — it's an interactive SSM session |

Rule: never run task ec2:logs from an agent — it opens an interactive SSM session that won't terminate. Always use Loki (preferred) or task ec2:exec AWS_PROFILE=personal AWS_REGION=us-east-1 CMD='tail -n 200 /var/log/ares/<role>.log'.

AWS auth: every task ec2:* command in this skill must run against the personal profile in us-east-1. The lab SSO profile is unreliable and the EC2 box lives in us-east-1 under personal.

Either export once per shell:

export AWS_PROFILE=personal
export AWS_DEFAULT_REGION=us-east-1
export TARGET_PROFILE=personal
export TARGET_REGION=us-east-1

…or prefix every invocation with AWS_PROFILE=personal AWS_REGION=us-east-1. The commands below use the prefix form so they're copy-paste-safe in a fresh shell.

Step 0 — mandatory baseline triage (run all in parallel, on every invocation)

Do not skip any of these. Do not respond to the user with a verdict until you've inspected each output. The point of this step is to make it impossible to declare "healthy" without the evidence.

# 0a. Current op id + status
task ec2:ops AWS_PROFILE=personal AWS_REGION=us-east-1 EC2_NAME=kali-ares LATEST=true

# 0b. Current op objective state + tokens
task ec2:runtime AWS_PROFILE=personal AWS_REGION=us-east-1 EC2_NAME=kali-ares LATEST=true

# 0c. Process / Redis / NATS health
task ec2:status AWS_PROFILE=personal AWS_REGION=us-east-1 EC2_NAME=kali-ares

# 0d. The single most important probe — orchestrator tail. Grep it for the wedge signatures listed above.
task ec2:exec AWS_PROFILE=personal AWS_REGION=us-east-1 EC2_NAME=kali-ares \
  CMD='tail -n 300 /var/log/ares/orchestrator.log'

# 0e. Historical baseline — last several ops, to compare runtime-to-milestone
ares --ec2 kali-ares --ec2-profile personal --ec2-region us-east-1 ops list | head -20

# 0f. Failed tasks for the current op
ares --ec2 kali-ares --ec2-profile personal --ec2-region us-east-1 ops tasks --latest --status failed | head -80

Pull op-YYYYMMDD-HHMMSS from 0a/0b and use that as $OP below. After collecting:

1. Grep the 0d output for each pattern in the "Tight-loop / wedge signatures" table. If any hits ≥3 times, you have your root cause; jump to reporting.
2. Compare 0b's Domains compromised and Vulns exploited against the runtime banner of recent ops in 0e. If the prior 3 ops compromised more domains in less time at this point, the current op is regressed regardless of how healthy 0a/0c look.
3. Read 0f — the failure mode of the first 5-10 failed tasks usually points at the role/tool that's flailing.

Only proceed past Step 0 to deeper probes (Loki, Tempo, SSM journals) if none of the above lands a verdict.

Step 1 — fast triage (Loki, last hour)

Loki has every ares log line shipped from the EC2 box. Datasource UID is loki. Logs are JSON; the actual line is in the message field, with labels app="ares", deployment="alpha-operator-range-kali-ares", job=<role>.log.

Run these in parallel:

mcp__grafana__query_loki_logs
  datasourceUid: "loki"
  logql: '{app="ares", deployment="alpha-operator-range-kali-ares"} |~ "(?i)error|fatal|panic|traceback|RUST_BACKTRACE"'
  limit: 30

mcp__grafana__query_loki_logs
  datasourceUid: "loki"
  logql: '{app="ares", deployment="alpha-operator-range-kali-ares", job="orchestrator.log"} |~ "WARN|ERROR"'
  limit: 30

Narrow by role when you know the suspect: change job="orchestrator.log" to one of recon.log, credential_access.log, cracker.log, acl.log, privesc.log, lateral.log, coercion.log.

Narrow by op id (substring match on the log line):

logql: '{app="ares", deployment="alpha-operator-range-kali-ares"} |= "op-20260630-201500"'

Use query_loki_stats first when you're guessing the selector — it tells you whether the stream has any entries before you waste a query_loki_logs call.

Step 2 — failed tasks (operation-level)

task red:multi:tasks:list LATEST=true STATUS=failed   # K8s
ares --ec2 kali-ares --ec2-profile personal --ec2-region us-east-1 ops tasks --latest --status failed   # EC2

Failed tasks include the worker's error message and the role that failed. Cross-reference against Loki by role + timestamp.

Step 3 — wedge detection (objective state frozen)

The canonical wedge is NOT "tokens flatlined" — tokens almost always keep climbing during a wedge because the LLM re-evaluates the same frozen state every tick. The canonical wedge is "objective state frozen while tokens climb." Probe state, not tokens:

# Snapshot 1
task ec2:exec AWS_PROFILE=personal AWS_REGION=us-east-1 EC2_NAME=kali-ares \
  CMD='redis-cli hmget "ares:op:'"$OP"':meta" has_domain_admin has_golden_ticket target_ips initialized; echo ---; redis-cli scard "ares:op:'"$OP"':creds" 2>/dev/null; redis-cli scard "ares:op:'"$OP"':hashes" 2>/dev/null; redis-cli scard "ares:op:'"$OP"':hosts" 2>/dev/null'
# wait 60s
# Snapshot 2 — same command. Diff the two. Identical = wedge.

Cross-check against tokens: pull ec2:runtime at both snapshots. Tokens climbing + state identical = textbook wedge. Tokens climbing + state changing = healthy. Tokens flatlined + state identical = orchestrator hung (rarer).

If wedged, two further probes pinpoint where:

# Outbound HTTPS from orchestrator — zero connections = LLM API stall
task ec2:exec AWS_PROFILE=personal AWS_REGION=us-east-1 EC2_NAME=kali-ares CMD='ORCH=$(pgrep -f "ares orchestrator" | head -1); echo "orch_pid=$ORCH"; sudo ss -tnp 2>/dev/null | grep "pid=$ORCH" | grep -v 127.0.0.1 | wc -l'

# Loki search for retry/throttle/dedup markers in the last 30 minutes
mcp__grafana__query_loki_logs
  datasourceUid: "loki"
  logql: '{app="ares", deployment="alpha-operator-range-kali-ares", job="orchestrator.log"} |~ "clearing dedup for retry|KDC_ERR_|Task deferred|throttler|stale|wedge"'
  limit: 80

Remedy depends on root cause:

• Hot retry loop on a tool (clearing dedup for retry) → fix the dedup/blacklist logic in the relevant automation/auto_*.rs; in the meantime task ec2:stop-op ... LATEST=true to stop the burn.
• LLM API stall → restart workers, check the model provi