<p align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="assets/logo-wordmark-dark.svg"> <img alt="lastwords" src="assets/logo-wordmark.svg" height="72"> </picture> </p> <p align="center"> <a href="https://artifacthub.io/packages/search?repo=lastwords"><img alt="Artifact Hub" src="https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/lastwords"></a> <a href="https://github.com/Arvo-AI/lastwords/releases/latest"><img alt="Release" src="https://img.shields.io/github/v/release/Arvo-AI/lastwords"></a> <a href="LICENSE"><img alt="License" src="https://img.shields.io/github/license/Arvo-AI/lastwords"></a> </p>

lastwords

Memory snapshots before OOM kills, and logs and core dumps from crashes, for Kubernetes.

lastwords runs on every node. When a container gets close to its memory limit, it records what is using the memory. When a container dies, it saves the logs of that instance, the pod's events and any core dump. Everything stays on the node (or in S3), and you can read it with kubectl lastwords or from an AI agent over MCP.

helm install lastwords oci://ghcr.io/arvo-ai/charts/lastwords -n lastwords --create-namespace

The problem

When you look at a pod that died hours ago, most of the evidence is gone:

• OOMKilled leaves no heap dump. The kernel kills the process with SIGKILL. -XX:+HeapDumpOnOutOfMemoryError never fires, and Go, Node and Python get no warning either.
• kubectl logs --previous only keeps one restart. After a few crash-loop restarts, the logs from the first failure are gone.
• Events expire after one hour (--event-ttl, default 1h).
• Core dumps go nowhere. core_pattern is set per node, so crashing containers either dump nothing or dump to a path inside a container that is about to be deleted.

None of it can be recovered later. It has to be collected while the container is alive or right after it dies, which is what lastwords does.

What it captures

| When | What lastwords saves | |---|---| | A container nears its memory limit (90%, again at 97%, or earlier if it is climbing fast) | Memory breakdown (heap vs page cache vs kernel), a memory timeline, processes. Java: class histogram, heap info, native memory, thread dump. Go: heap profile with the top allocation sites. Optional full heap dump. | | A container is OOMKilled | The memory curve up to the kill, logs of the instance that died, pod events, and links to the snapshots taken before the kill | | A process inside a container is OOM-killed but the container survives | What Kubernetes never reports: which process died and the memory state | | A container crashes (non-zero exit, signal) | Log tail of the dead instance, the crash site parsed from Go/Java/Python stack traces, events, exit code meaning | | A process dumps core (SIGSEGV, SIGABRT, ...) | The gzipped core, memory maps, binary and image, matched to the pod | | A pod is evicted | Eviction reason, every container's logs, events |

Every capture comes with a summary.md that says what happened in plain words:

# checkout-5498f94695-526hm/checkout was OOMKilled at 16:51:38 UTC (limit 384.0 MiB); 1 snapshot taken before the kill

## What lastwords found
- Memory climbed steadily for 1m1s, the pattern of a leak or an unbounded cache. Raising the limit would only delay the next kill.
- Before the kill (67% of limit): Largest application classes on the heap:
  com.example.checkout.CheckoutService$CartSnapshot: 2860 instances ...
- Before the kill (67% of limit): One class dominates the Java heap: byte[] holds 179.2 MiB (97%).

# order-consumer-76d5fcb94-8vm4p/consumer crashed at 03:09:30 UTC: Error, exit code 2

## What lastwords found
- Crashed in main.registerDiscount at /src/main.go:19 (assignment to entry in nil map).

Use it

kubectl lastwords list                      # everything from the last 7 days
kubectl lastwords list -n shop --kind oomkill
kubectl lastwords show 20261102T030712Z-af14b5
kubectl lastwords get 20261102T030712Z-af14b5 jvm-class-histogram.txt
kubectl lastwords top                       # containers closest to their memory limit right now

Install the CLI (it is also a kubectl plugin):

# macOS / Linux
curl -sSL https://github.com/Arvo-AI/lastwords/releases/latest/download/lastwords_$(uname -s)_$(uname -m | sed s/aarch64/arm64/).tar.gz | tar xz
sudo mv lastwords /usr/local/bin/ && sudo ln -sf /usr/local/bin/lastwords /usr/local/bin/kubectl-lastwords

The CLI talks to the agents through the Kubernetes API server, so it needs nothing but your kubeconfig.

Guides

Measured on EKS with the failures in this repo; every command and output is from a real run.

• Get a Java heap histogram before Kubernetes OOMKills the pod
• Why -XX:+HeapDumpOnOutOfMemoryError never writes a file in Kubernetes
• Java pod OOMKilled at 80% of its memory limit
• Get core dumps from crashing Kubernetes pods
• Find what is leaking in a Go pod before it is OOMKilled

Let your AI agent read it

lastwords ships an MCP server. Claude Code, Codex, Cursor and any MCP client can list and read captures:

claude mcp add lastwords -- kubectl-lastwords mcp
codex mcp add lastwords -- kubectl-lastwords mcp

Then ask: "the checkout pods restarted overnight, why?" Tools: lastwords_list_captures, lastwords_get_capture, lastwords_read_file, lastwords_watchlist. All read-only.

Get told

helm upgrade lastwords oci://ghcr.io/arvo-ai/charts/lastwords -n lastwords --reuse-values \
  --set notifications.slackWebhookUrl=https://hooks.slack.com/services/... \
  --set notifications.webhookUrl=https://your-service/lastwords \
  --set notifications.webhookSecret=$(openssl rand -hex 32)

With a secret set, every webhook delivery is signed: X-Lastwords-Signature: sha256=<HMAC-SHA256 of "<X-Lastwords-Timestamp>.<raw body>">. Verify it before acting on the event, and reject old timestamps:

import hmac, hashlib, time

def verify(secret: bytes, headers, body: bytes, tolerance=300) -> bool:
    ts = headers["X-Lastwords-Timestamp"]
    if abs(time.time() - int(ts)) > tolerance:
        return False
    want = "sha256=" + hmac.new(secret, ts.encode() + b"." + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(want, headers["X-Lastwords-Signature"])

Go receivers can use github.com/Arvo-AI/lastwords/webhook.Verify.

Every capture also becomes a Kubernetes Event on the pod (LastWordsOOMKill, LastWordsMemoryPressure, LastWordsCrash, ...), so it shows up in kubectl describe pod and anything that watches events. Prometheus metrics are on :9095/metrics.

Configuration

Common settings (see values.yaml for all):

| Value | Default | | |---|---|---| | capture.threshold | 0.90 | Capture at this share of the memory limit (working set, like the kubelet) | | capture.lastGasp | 0.97 | Second capture right before the kill | | capture.predictive | true | Capture early if the limit will be hit within predictSeconds | | capture.jvmGCHeadroom | 0.25 | JVMs are usually killed by a GC at ~75-80% of the limit, so their tiers aim below it | | capture.heapDumps | false | Full Java heap dumps for every JVM. Pods can opt in with the lastwords.dev/heap-dump: "true" annotation | | capture.goPprofPort | 6060 | Where Go programs serve net/http/pprof | | coreDumps.enabled | false | Capture core dumps (sets the node's kernel.core_pattern, needs a privileged agent) | | storage.maxBytes | 5 GiB | Disk budget per node | | s3.bucket | | Upload captures to S3, GCS (interoperability) or MinIO | | ignoreNamespaces | [kube-system] | Never watch these | | api.createSecret | true | Create the API token secret. Set false with Argo CD or Flux and create lastwords-api yourself (why) |

Pod annotations:

| Annotation | | |---|---| | lastwords.dev/ignore: "true" | Don't watch this pod | | lastwords.dev/threshold: "0.8" | Per-pod capture threshold | | lastwords.dev/heap-dump: "true" | Allow a full heap dump for this pod | | lastwords.dev/pprof-port: "8080" | Go pprof port |

Uninstall

helm uninstall lastwords -n lastwords

On shutdown the agent puts the node's original core_pattern back (checked on EKS: systemd-coredump again after uninstall). Captures and the core helper binary stay in /var/lib/lastwords on each node, so a reinstall keeps your history, and a node whose agent was killed before it could restore core_pattern still has a working helper to hand cores to. Delete that directory on the nodes to remove everything.

How it works

A DaemonSet runs one small agent per node (a static Go binary; requests 20m CPU and 48 MiB). Measured on EKS (t3.large, Amazon Linux 2023): 8m CPU and 12 MiB on a quiet node, 20m CPU and 15 MiB with 35 containers, about 0.5m CPU per container watched.

1. It reads every container's memory accounting from the node's cgroup tree twice a second (memory.current, memory.stat, memory.events, PSI), computes the working set the way the kubelet does, and keeps a 10-minute timeline.
2. When a container crosses the threshold, or is climbing fast enough to hit its limit within 15 seconds, it collects evidence while the process is still alive. For the JVM it uses the HotSpot attach protocol (what jcmd uses) from outside the container; nothing is installed in your images. For Go it pulls /debug/pprof/heap.
3. It watches pods on its node. When a container terminates with an error, it copies the dead instance's log file before the next restart rotates it, and saves the pod's events.
4. With core dumps on, it points kernel.core_pattern at a small helper that writes the gzipped core next to the captures, and the agent matches it to the pod.
5. Captures stay on the node's disk under a size and age budget, optionally upload to S3, and are served to the CLI and MCP server through the API server's pod proxy.

More in docs/how-it-works.md.

Security

lastwords reads other processes' memory accounting, so it needs node-level access. Read docs/security.md before installing it in production. In short:

• Without core dumps the agent is not privileged. It runs with hostPID and the capabilities SYS_PTRACE, DAC_READ_SEARCH, DAC_OVERRIDE, CHOWN, FOWNER, KILL, plus a read-only root filesystem.
• Core dumps need privileged to set kernel.core_pattern. They are off by default.
• Full heap dumps are off by default because they contain whatever was in memory. The default class histogram contains class names and counts, not data.
• Command lines are redacted for arguments that look like secrets.
• Reading captures needs RBAC on pods/proxy in the lastwords namespace plus the API token secret. The chart creates a lastwords-reader Role.

FAQ

My pod was OOMKilled. How do I find out what used the memory? If lastwords was running, kubectl lastwords list --kind oomkill shows the kill, and the linked pressure capture shows what was using memory before it. If it wasn't running, nothing can recover that information. Install it so the next kill is captured.

Why didn't HeapDumpOnOutOfMemoryError produce a heap dump? That flag fires when the JVM itself runs out of heap. A container OOM kill is the kernel sending SIGKILL when the whole process (heap plus metaspace, threads, direct buffers, native memory) hits the cgroup limit. The JVM never gets a chance to run. lastwords takes the histogram before that point.

How do I get core dumps from Kubernetes pods? helm upgrade lastwords ... --set coreDumps.enabled=true. Cores are