---

license: Apache-2.0 name: doca-dpa-hl-tracer description: > Use this skill when the user runs doca_dpa_hl_tracer to capture/decode DPA-side traces at the programming-events layer (kernel entry/exit, sync points, comm primitive calls, RDMA WR submission, completion drain) — picking TRACE vs CRIT, tuning the JSON config (file-size limits

• file_size_limit_policy, thread priorities/cores), decoding against the matching DPA-side ELF, or diagnosing empty/noisy captures. Trigger even when the user does not explicitly mention "DOCA DPA tracer" or "high-level tracer" — typical implicit phrasings include "DPA kernel returns wrong result but host completions look clean", "kernel-entry to first-comm latency is huge", "RDMA WR to drain gap on the DPA", "trace file truncated mid-run", "TRACE doubled my DPA latency", or "tracer wrote a file but parser shows zero events". Refuse and route elsewhere for writing DPA kernels, DPA-Comms/DPA-Verbs programming, raw per-cycle DPA profiling, host-side doca-dpa debugging, or production DPA telemetry — those belong to other skills. metadata: kind: tool compatibility: > Requires DOCA SDK installed at /opt/mellanox/doca on Linux (Ubuntu 22.04/24.04 or RHEL/SLES) with a BlueField device whose DPA processor is exposed to the host, plus the DOCA DPA Tools optional component (binary at /opt/mellanox/doca/tools/doca_dpa_hl_tracer). Requires a DPACC-built DPA-side ELF and a live doca-dpa-launched workload for events to fire.

---

DOCA DPA High-Level Tracer

Where to start: This is a tool skill for invoking doca_dpa_hl_tracer — the documented host-side CLI that captures DPA-side execution traces in higher-level terms (DPA programming events: kernel entry / exit, sync points, comm primitive calls, RDMA WR submission, completions) rather than raw cycle counts. Open TASKS.md and start at ## configure for the mode-vs-overhead decision and the JSON config layout, then ## run for the capture → decode → render pipeline. Open CAPABILITIES.md when the question is what does this tool actually trace, which DPA programming events does it expose, what is the trace-overhead vs fidelity tradeoff, or how does it slot into a DPA debug loop alongside doca-dpa and doca-debug. If DPA is not the right surface for the user's question (e.g. the bug is host-side, the bug is in the DPACC-produced image, the user wants raw cycle counts), the path-selection rule in CAPABILITIES.md ## Capabilities and modes routes the agent before any capture is attempted.

Example questions this skill answers well

The CLASSES of doca_dpa_hl_tracer questions this skill is built to answer, each with one worked example. The class is the load-bearing piece; the worked example is one instance.

• "My DPA kernel is doing the wrong thing — where do I look?" — worked example: "my host-side doca_dpa_kernel_launch_update_* completes, but the kernel's reported result is wrong; no host-side DOCA_ERROR_*". Answered by the when DPA-side high-level tracing is the right surface gate in CAPABILITIES.md ## Capabilities and modes
  • the capture → decode → render flow in TASKS.md ## run + the which DPA programming events to focus on rule in TASKS.md ## debug.
• "My DPA kernel is slow at a granularity that doesn't show up in cycle profiles — how do I see kernel-entry to first-comm-call latency?" — worked example: "my DPA kernel runs but the time between launch and the first RDMA WR submission is bigger than I expected". Answered by the event-taxonomy table in CAPABILITIES.md ## Capabilities and modes
  • the iterative loop in TASKS.md ## test which treats trace overhead, mode (TRACE vs CRIT), and capture window as axes to tune.
• "How do I capture a trace without burying the DPA in observation overhead?" — worked example: "TRACE mode is producing too much data and my measured DPA latency went up by 2x compared to without the tracer". Answered by the mode-vs-overhead tradeoff in CAPABILITIES.md ## Capabilities and modes
  • the CRIT-first guidance in TASKS.md ## configure (start with critical-events-only; widen to TRACE only when the bug demands per-event detail).
• "My trace file got truncated mid-run — how should I configure the file-size limits?" — worked example: "binary trace file hit 5 GB and the capture stopped". Answered by the log_file_max_size_in_bytes / bin_file_max_size_in_bytes / file_size_limit_policy triple in CAPABILITIES.md ## Capabilities and modes
  • the JSON config layout in TASKS.md ## configure.
• "Is the tracer on my install, and is it paired with the matching doca-dpa library and DPACC compiler version?" — worked example: "is the tracer ABI on my install compatible with the DPA image my DPACC just produced?". Answered by the version-overlay in CAPABILITIES.md ## Version compatibility, which redirects to the canonical doca-version chain and adds the tracer ↔ doca-dpa library ↔ DPACC compiler match rule.
• "The capture file looks empty / decode failed — is the install broken, no events fired, or am I tracing the wrong thing?" — worked example: "doca_dpa_hl_tracer ran, wrote a file, but the parser shows zero events". Answered by the layered error taxonomy in CAPABILITIES.md ## Error taxonomy (install / device-binding / DPA-image-instrumented / capture-window / decode-vs-elf / overhead-saturated / version / cross-cutting) + the layered walk in TASKS.md ## debug.

Audience

This skill serves external developers, platform operators, and AI agents who have already brought up a DPA-side workload through doca-dpa and now need higher-level visibility into what the DPA kernel is actually doing on the wire — DPA programming events ordering, sync gaps, comm-call latencies, RDMA-WR / completion timing — without dropping all the way down to raw cycle counters. Concretely:

• A DPA developer who can launch their kernel cleanly from the host side but whose kernel's result is wrong or whose DPA-side performance is below expectation, and who needs a DPA-side ground truth before triaging.
• A platform operator running a DPA-using workload (RDMA offload from accelerator, custom CC algorithm via doca-pcc) and needs to localize a regression to the DPA side without instrumenting the application.
• An AI agent producing a DPA-side trace report as evidence for the host-side doca-dpa TASKS.md ## debug ladder when the host side reports clean completions but the DPA-side behaviour is wrong.

It is not for users debugging the tracer binary itself, not a substitute for the live public DOCA DPA Tools guide, not the right place for users learning how to write a DPA kernel (that audience belongs in doca-dpa plus the public DOCA DPA / DPACC / DPA-Comms / DPA-Verbs guides), and not the right place for raw per-instruction cycle profiling (different surface, different tool — route via doca-public-knowledge-map ## DOCA tools).

The tracer is shipped as a CLI binary under /opt/mellanox/doca/tools/, not a library you link against. The skill uses the same kind: tool three-file shape as the rest of the bundle so the agent's task-verb contract is uniform across libraries, services, and tools.

Language scope

doca_dpa_hl_tracer is a C++ host-side CLI. Its inputs are its JSON config file, the DPA-side ELF (the doca_dpa_app-class image produced by DPACC), and a running DPA-side workload that the host-side doca-dpa lifecycle already started. Its outputs are a binary trace file (bin_file) and a human-readable log file (log_file). The skill keeps the workflow guidance language-neutral — the DPA-side workload it traces can be C compiled by DPACC or any other DPA translation unit DPACC accepts — and routes per-language questions to the public DPA / DPACC guides via doca-public-knowledge-map.

When to load this skill

Load this skill when the user is — or the agent needs to — invoke doca_dpa_hl_tracer on a real host with DOCA installed against a BlueField with a DPA processor visible to the host, and the host-side doca-dpa lifecycle has already brought a DPA workload up at least once. Concretely:

• Capturing a DPA-side trace to localize a DPA kernel's wrong-result or wrong-ordering behaviour when the host-side doca-dpa lifecycle reports clean completions.
• Capturing a DPA-side trace to localize a DPA-side performance gap (kernel-entry to first-comm latency, RDMA-WR-issue to completion gap, sync-point dwell time) at a granularity above raw cycle counts.
• Choosing between TRACE and CRIT capture modes based on the bug-vs-overhead tradeoff and the available capture window.
• Tuning the JSON config (thread priorities, core affinities, file size limits, file-size-limit policy) so the capture itself does not perturb the workload more than the bug it is investigating.
• Decoding a captured bin_file against the matching DPA-side ELF to render the human-readable event stream.
• Capturing a side-effect-bounded trace as prerequisite evidence for a host-side doca-dpa TASKS.md ## debug ladder step.

Do not load this skill for general DOCA orientation, DPA-side programming model questions, raw cycle profiling, or DOCA / DPACC install. For those, route to doca-public-knowledge-map, doca-dpa, or doca-setup.

What this skill provides

This is a thin loader. Substantive material lives in two companion files:

• CAPABILITIES.md — what doca_dpa_hl_tracer captures: the DPA programming event taxonomy (kernel entry / exit, sync points, comm primitive calls, RDMA WR submission and completion drain), the two documented capture modes (TRACE for full per-event, CRIT for critical-events only), the trace-overhead-vs-fidelity tradeoff, the config-file shape (receiver / binary-writer / file-writer / printer threads with priority + core affinity, file size limits, file_size_limit_policy), the capture-window + workload-must-be-running invariant, the ELF-must-match-image rule for decode, the version-availability overlay (tracer ↔ doca-dpa library ↔ DPACC compiler), the layered error taxonomy (install / device-binding / image-instrumented / capture-window / decode / overhead-saturated / version / cross-cutting), the observability surface (binary trace file + log file + tool's own stderr), and the safety policy (capture is bounded; tracing is not a production observability surface).
• TASKS.md — step-by-step workflows for the in-scope task verbs: install (route to host-side DOCA install + DPA prerequisites), configure (mode + JSON config layout + capture window), build (route to install — the binary is shipped, the DPA-side application is user-built by DPACC), modify (refuse — do not patch the binary; modify the JSON config and th