Doca Dpa

DOCA DPA capabilities, version overlay, errors, observability, safety

Where to start: Pick the H2 anchor that matches your question (host-side doca_dpa context / loaded DPA image / DPA execution context / kernel launch + completion / dual capability discovery / env preconditions / errors) and read that section end-to-end. The tables in each section are the load-bearing content; the prose around them is interpretation.

Read this file when the loader sent you here from SKILL.md. For the how of executing each pattern (the verbs configure / build / modify / run / test / debug), jump to TASKS.md. For the canonical DOCA version-handling rules that this skill layers a DPA overlay on top of, see doca-version/SKILL.md.

Pattern overview

Every DPA question this skill teaches resolves into one of SIX patterns. The patterns are CLASSES — they apply across every DOCA DPA release and every host + BlueField + DPACC combination.

Two cross-cutting rules that apply to every pattern above:

• The two-side-program model is non-negotiable. Every DPA application is two programs: the host side (this skill, doca-dpa) and the DPA side (the user's source compiled by dpacc, possibly using the DPA device-side components — the comm archive libdoca_dpa_dev_comm.a and the verbs archive libdoca_dpa_dev_verbs.a (shipped within doca-dpa) — from inside the kernel). An agent that treats the application as one program — for example by proposing that the host code "call the DPA kernel directly as a function" — has the model wrong for every version of DPA. The host launches; the DPA executes; the two sides are coupled by the kernel function signature and the launch arguments.

• Capability is a TWO-axis question. "Is this DPA feature available on this host" requires BOTH a DOCA cap-query against the active doca_devinfo (the doca_dpa_cap_* family) AND the BlueField generation actually carrying a DPA processor exposed to the host. An agent that quotes only one axis will miss the "the doc page says feature X exists but it returns DOCA_ERROR_NOT_SUPPORTED on my hardware" cases.

Capabilities and modes

The two orthogonal selection axes for any DPA design from the host side are which BlueField DPA instance (doca_dpa per doca_dev that maps to a DPA-capable BlueField) and which DPA application image (the doca_dpa_app produced by dpacc from the user's DPA-side source) the host is going to load onto that DPA. Choose both before writing any host-side launch code, then drill into the relevant capability-query.

Two-side-program model — the host side and the DPA side.

The agent's rule: when the user asks "how do I write the DPA kernel", that is the DPA-side question — route to the public DPA / DPACC guide via doca-public-knowledge-map/SKILL.md. When the user asks "how do I load my DPA kernel from the host and launch it", that is this skill's scope. Two distinct questions; two distinct surfaces.

The per-DPA-instance doca_dpa context — one per DPA the host drives.

A multi-BlueField host that wants to drive more than one DPA needs one doca_dpa per BlueField — there is no "global DPA context". The agent must ask which BlueField (which doca_dev) the user intends to drive before recommending any doca_dpa_* call.

The loaded DPA application image (doca_dpa_app) — the output of dpacc made visible to the host.

The agent's rule: the host's view of "which DPA kernel functions exist" is exactly the set of entry points in the loaded doca_dpa_app, which is exactly the set of function names the user marked as DPA kernels in the DPA-side source that dpacc compiled. If the user is "trying to launch a kernel that's not in the image", that is a build-side question — go back to ## build and to the DPACC guide via doca-public-knowledge-map/SKILL.md, not a host-side launch-call fix.

The DPA execution context (doca_dpa_thread) — one DPA processor thread per context.

The agent's anti-pattern alert: "create one DPA thread per kernel invocation" defeats the entire reason to use the DPA (thread-create overhead per launch). A persistent DPA thread processing a stream of launches is the right shape; a fresh-thread-per-launch is the wrong shape.

Host-initiated kernel launch + completion model.

Dual-axis capability discovery — the only rule. Before sizing any thread count, assuming a DPA feature is available, or proposing a kernel-launch shape works on the user's BlueField, run BOTH a DOCA cap-query AND confirm the BlueField generation actually exposes the DPA processor. Either axis missing the support fails the feature.

DOCA 3.5 DPA capability additions. The DPA public header now exposes EU group affinity, validation resources, and dev thread retrigger capabilities. Use the doca_dpa_eu_group_affinity_* family to create, set, get, clear, and destroy EU group affinity objects; use doca_dpa_thread_get_group_affinity and doca_dpa_thread_set_group_affinity to attach an affinity to a doca_dpa_thread; use doca_dpa_get_validation_resources and doca_dpa_set_validation_resources to obtain and apply validation resources; use doca_dpa_resources_get_num_eu_groups and doca_dpa_resources_get_eu_groups to enumerate EU groups from a resources object; and call doca_dpa_dev_thread_retrigger from DPA-side code to retrigger the current dev thread. All of these are declared in the DPA public header. A developer targeting DOCA 3.5 must query and apply these capabilities through the doca_dpa/doca_dpa_thread lifecycle instead of assuming a fixed EU-group layout or a fixed validation-resource set: obtain validation resources with doca_dpa_get_validation_resources, apply them with doca_dpa_set_validation_resources, enumerate EU groups with the doca_dpa_resources_get_* family, and set per-thread EU group affinity with the doca_dpa_thread_*_group_affinity family before relying on any EU-group placement.

Configuration shape. Mandatory preconditions before any doca_dpa_kernel_launch_update_* call: the doca_dpa Core context must be created against a doca_dev that maps to a DPA-capable BlueField; the DPA application image (doca_dpa_app) compiled by dpacc must be loaded into the doca_dpa; at least one doca_dpa_thread must exist on top of it; the doca_dpa Core context must be at the started lifecycle stage; the host launch arguments must match the shape that the DPA-side kernel function expects. Optional configurations (thread count, completion attachment topology) are program-side tunables that ride on top of the same cap-query rule.

comms

DPA Comms is the DPA-SIDE component — the archive libdoca_dpa_dev_comm.a with device header doca_dpa_dev_comch_msgq.h, shipped as part of doca-dpa (there is NO separate doca-dpa-comms pkg-config module) — the DPA kernel itself calls from INSIDE the DPA processor to send / receive small messages between DPA threads or to signal another DPA thread on the same doca_dpa_app. It is NOT host-side; it does NOT replace doca-comch (host ↔ DPU PCIe control-plane messaging) and does NOT replace doca-rdma (host-to-remote-peer RDMA). The host-side parent surface this section sits beneath is everything in ## Capabilities and modes above; this section adds only what DPA Comms changes on top.

Routing rule the agent must surface first. When the user asks about messaging on or from the DPA, walk this disambiguation BEFORE recommending any symbol:

The agent's anti-pattern alert: adding the DPA device-side comm archive (libdoca_dpa_dev_comm.a) to the host link line is a strong signal the user has the side wrong. Its symbols are linked into the DPA-side translation unit by dpacc, not into the host executable.

DPA-side primitive families — pick at least one per kernel. The exact symbol set is install-bound and lives in the DPA-side headers DPACC compiles against (doca_dpa_dev_comch_msgq.h); the agent must NOT quote symbol names from memory and must route the user to the on-disk samples at /opt/mellanox/doca/samples/doca_dpa/ plus the public DOCA DPA Comms guide via doca-public-knowledge-map/SKILL.md. The family-level shape is stable enough to teach:

Capability discovery — no per-primitive cap family. There is NO doca_dpa_comms_cap_* host cap family. The only DPA host-side capability queries are doca_dpa_cap_is_supported (is a DPA exposed to the host on this doca_devinfo at all) and doca_dpa_cap_get_max_kernel_time_alive_supported. Whether a specific DPA-Comms primitive is available is a function of the BlueField generation and the matched DOCA + DPACC install, read from the device header doca_dpa_dev_comch_msgq.h and the shipped sample — NOT from a runtime cap query. An agent that proposes a "DPA-side runtime cap check" or quotes a doca_dpa_comms_cap_* call has the model wrong. Because the comm component is the archive libdoca_dpa_dev_comm.a shipped within doca-dpa (no separate doca-dpa-comms.pc), the version anchor is the single pkg-config --modversion doca-dpa, which must agree with doca_caps --version and the installed dpacc per the DOCA Compatibility Policy. A doca-dpa / dpacc version skew is the canonical "my DPA-side comms call returns DOCA_ERROR_DRIVER on the host completion" root cause — surface BOTH versions in the report.

DPA-Comms error overlay. Every error surfaces back to the host through the parent skill's doca_dpa_completion; the DPA kernel itself does not surface errors to anyone but its host program. Disambiguate the rows below BEFORE falling back to the parent overlay in ## Error taxonomy:

Safety overlay — parent-skill prerequisite matrix. This section inherits the parent's env-precondition matrix in ## Safety policy plus: the parent host-side flow must already be green end-to-end (a trivial DPA kernel with NO DPA-Comms calls launches and completes on this host + image per TASKS.md ## test step 1); the single doca-dpa version agrees with the installed dpacc (there is no separate doca-dpa-comms.pc); the BlueField generation + matched install actually expose the comm primitives the kernel will call; the two-side-program signature on the DPA-Comms endpoint handles is consistent across host launch call and DPA-side kernel signature. Do NOT start writing DPA-side comms code on a broken parent flow — fix that first. The DPA kernel does NOT own DPA-Comms teardown; teardown follows the host-side doca-dpa lifecycle (destroy threads → release the loaded app → destroy the doca_dpa).

For the configure / build / modify / run / test / debug shape specific to DPA-Comms, see TASKS.md ## comms.

verbs

DPA Verbs is the DPA-SIDE raw-verbs RDMA surface — the archive libdoca_dpa_dev_verbs.a with device header doca_dpa_dev_verbs.h, shipped as part of doca-dpa (there is NO separate doca-dpa-verbs pkg-config module) — the DPA kernel itself calls from INSIDE the DPA processor to post work requests (sends, RDMA reads, RDMA writes, atomics) on QPs the host configured. It is the latency-tuning escape hatch beneath the host-side parent surface in ## Capabilities and modes above — not a default. Most RDMA work belongs on host-side doca-rdma/SKILL.md; DPA Verbs exists when the host round-trip is the measured bottleneck AND the DPA kernel needs RDMA semantics from inside the kernel body.

The 4-way RDMA matrix — the load-bearing first decision. DOCA splits RDMA across two axes: execution side (host CPU vs DPA processor) and abstraction level (high-level tasks vs raw verbs):

The agent's rule, before any code-level discussion: (1) confirm DPA-resident compute is already on doca-dpa (the parent); (2) confirm the host round-trip is the measured latency bottleneck (profile / histogram / per-op cost — not an intuition); (3) confirm the user needs RDMA semantics, not local DPA messaging (that is ## comms). If any check fails, the right answer is to climb back up to host-side RDMA — recommending DPA-side verbs for a host-bottleneck workload is a misroute.

Host-configures-QP / DPA-uses-QP — the coupling rule. Single non-negotiable invariant for everything below.

The agent's anti-pattern alert: "the DPA kernel creates its own QP" has the model wrong — QPs are host-side. The DPA-side code consumes handles the host setup made available; it does NOT own QP create / configure / state transitions.

DPA-side primitive surface. WR-post (send, RDMA read, RDMA write, atomic) on a host-configured QP handle, plus optional completion-poll inside the kernel. Exact symbol names are install-bound (they live in doca_dpa_dev_verbs.h); the agent must NOT quote them from memory and must route the user to /opt/mellanox/doca/samples/doca_dpa/ plus the public DOCA DPA Verbs guide via doca-public-knowledge-map/SKILL.md.

Capability discovery — no per-verb cap family. There is NO doca_dpa_verbs_cap_* host cap family. The only DPA host-side capability queries are doca_dpa_cap_is_supported and doca_dpa_cap_get_max_kernel_time_alive_supported. Which specific verb / opcode the DPA hardware exposes is determined by the BlueField generation and the matched DOCA + DPACC install, read from doca_dpa_dev_verbs.h and the shipped sample — the DPA-side translation unit cannot cap-query from inside the kernel, and there is no host-side per-verb query to substitute. Because the verbs component is the archive libdoca_dpa_dev_verbs.a shipped within doca-dpa (no separate doca-dpa-verbs.pc), the version chain is the single pkg-config --modversion doca-dpa agreeing with doca_caps --version and the installed dpacc per the DOCA Compatibility Policy.

DPA-Verbs error overlay. Add to the parent overlay in ## Error taxonomy:

Climb-back rule. DPA-side verbs is a targeted latency-tuning surface. Once the specific RDMA op that drove the drop into DPA- side verbs is wired and the latency win is measured, the agent should explicitly ask whether remaining RDMA work can move back up to host-side doca-rdma/SKILL.md. The two-side-program maintenance cost (DPACC + DOCA version pinning, cap-query density, host / DPA boundary debugging) is real; do not pay it for RDMA the host can handle.

DOCA 3.5 DPA API families. Use the families doca_dpa_eu_group_affinity_*, doca_dpa_thread_*_group_affinity, doca_dpa_get_validation_resources, doca_dpa_set_validation_resources, doca_dpa_resources_get_num_eu_groups, doca_dpa_resources_get_eu_groups, and doca_dpa_dev_thread_retrigger from the DPA public header.

For the configure / build / modify / run / test / debug shape specific to DPA-Verbs, see TASKS.md ## verbs.

Version compatibility

For the canonical DOCA version-detection chain, the four-way match rule, NGC container semantics, and the headers-win-over-docs rule, see doca-version/SKILL.md. The body lives there; this skill does not duplicate it.

The DPA-specific overlay is:

• DOCA must match the DPACC compiler per the DOCA Compatibility Policy. The DPACC compiler is the build-time component that turns DPA-side source into the binary embedded in the host executable as a doca_dpa_app; the host-side runtime that loads it is doca-dpa. Mismatched DOCA + DPACC versions fail at link time (missing symbols on either side) or at launch time (DOCA_ERROR_DRIVER from a doca_dpa_kernel_launch_update_* call) in ways that look like hardware bugs but are version-skew bugs. The agent must surface BOTH pkg-config --modversion doca-dpa AND the installed dpacc version, cross-check them against the DOCA Compatibility Policy at https://docs.nvidia.com/doca/sdk/doca-compatibility-policy/index.html, and route any disagreement to doca-version TASKS.md ## debug (doca-version/TASKS.md#debug) before any DPA-layer diagnosis. Per the cross-cutting cap-query rule in doca-version CAPABILITIES.md ## Observability (doca-version/CAPABILITIES.md#observability), the doca_dpa_cap_* query against the active doca_devinfo is the runtime authority for "is this DPA feature on this hardware + this DOCA install", and the four-way-match check (doca-dpa.pc plus doca-common.pc plus the matching DPACC) per doca-version CAPABILITIES.md ## Version compatibility (doca-version/CAPABILITIES.md#version-compatibility) catches the DOCA upgraded but DPACC didn't partial-install pattern before it surfaces as a launch failure.

• DOCA 3.5 DPA API. The release adds the EU group affinity family, the validation resources family, and the dev thread retrigger family; use those APIs from the DPA public API header.

Error taxonomy

DPA-specific overlays on the cross-library DOCA_ERROR_* taxonomy. The cross-library taxonomy itself lives in doca-programming-guide CAPABILITIES.md ## Error taxonomy (doca-programming-guide/CAPABILITIES.md#error-taxonomy); the rows below are the DPA surface meaning that the agent must disambiguate before falling back to the cross-library response.

The agent's rule: never recommend a retry loop on DOCA_ERROR_* without first identifying which of the rows above is the cause. _AGAIN is the only one that wants a drain-then-retry; the others want investigation (env / version / lifecycle / two-side-program signature mismatch), not retry.

Observability

DPA observability surface is two-sided: there is a host-side observability surface (per-launch completions delivered through doca_dpa_completion, the DOCA logger, cap-query snapshots) AND a DPA-side observability surface (the DPA developer tools documented in the public DPA Tools umbrella reachable via doca-public-knowledge-map/SKILL.md — the DPA debugger, the DPA process-state inspector, and the DPA statistics tool). The agent must reach for both, not just one — a hung DPA kernel that produces no host-side completion is almost always visible on the DPA side via one of the DPA Tools.

Three primary signals the agent should reach for:

1. Host-side per-launch completions via doca_dpa_completion. Every kernel launch the host submits produces a completion (success or failure) the host reads through the doca_dpa_completion attached to that launch's doca_dpa_thread. Absence of a completion for a submitted launch is always a host-side missing-progress bug OR a DPA-side kernel that has not yet exited (commonly because the kernel is in an infinite loop with no termination signal). Confirm by progressing the PE before reaching for DPA-side tooling.

2. Capability snapshot at configure time. The output of doca_dpa_cap_* against the active doca_devinfo together with the installed pkg-config --modversion doca-dpa and the installed dpacc version is the baseline of "what the library + the hardware + the DPACC compiler said was possible" before any kernel was launched. Save it; if a runtime call later returns DOCA_ERROR_NOT_SUPPORTED the diff against this baseline is the bug.

3. DPA-side developer tools (route via doca-public-knowledge-map/SKILL.md). When the host-side completion never arrives and the host side is healthy, the DPA-side kernel is running but stuck. The public DPA Tools umbrella names the DPA debugger (attach-and-step inside the DPA kernel), the DPA process-state inspector, and the DPA statistics tool. The agent must NAME the existence of these tools and route the user there; the per-tool surface is out of scope for this skill.

For cross-cutting observability primitives (--sdk-log-level, the DOCA_LOG_LEVEL env var, the trace build flavor) see doca-debug CAPABILITIES.md ## Observability (doca-debug/CAPABILITIES.md#observability). For the install-tree observability (logger names, package layout, sample tree) defer to doca-public-knowledge-map/SKILL.md.

Safety policy

Overlay on the bundle-wide hardware-safety meta-policy. The rules below are this skill's per-artifact overlay on the cross-cutting rules in doca-hardware-safety CAPABILITIES.md ## Safety policy (doca-hardware-safety/CAPABILITIES.md#safety-policy) (specifically ### Per-artifact overlay pattern (doca-hardware-safety/CAPABILITIES.md#per-artifact-overlay-pattern)). When the two layers disagree, the stricter wins; when either layer says STOP, the agent stops.

DPA's safety surface is env-precondition-driven AND two-side-program-driven. The two most common DPA first-app failures are (1) the host's BlueField is not on a generation that exposes the DPA processor to the host, and (2) the host-side launch call and the DPA-side kernel signature disagree because one side was rebuilt while the other was not. The agent's job is to verify both before any doca_dpa_kernel_launch_update_* call, not after the first DOCA_ERROR_DRIVER.

The env-precondition matrix the agent must walk for any new host-side DPA setup:

Do not partial-rebuild one side. A host-side rebuild against a new DOCA install with the DPA-side image still built by an old dpacc, or a DPA-side rebuild without rebuilding the host that embeds it, fails DPA in non-obvious ways: the host launch call may succeed at submit and the DPA kernel may diverge silently or return DOCA_ERROR_DRIVER at launch. The fix is to rebuild both sides against the matched DOCA + DPACC versions per the DOCA Compatibility Policy, not to silence the error.

Lifecycle ordering is DPA-aware. The doca_dpa_thread objects must be destroyed BEFORE the doca_dpa_app is released; the doca_dpa_app release must precede destroying the doca_dpa; the doca_dpa itself follows the universal DOCA Core teardown order on top of the doca_dev. Out-of-order teardown surfaces as DOCA_ERROR_BAD_STATE on subsequent calls but also leaves the DPA processor in an undefined state for the next process; the agent must surface this ordering explicitly.

Deferred topic boundaries

This skill scopes itself to the host-side DOCA DPA library. Adjacent topics the agent will get asked but should route elsewhere:

• DPA-side kernel programming itself (how to write the function body the DPA processor runs; the DPA-side memory model; DPA-side allocation; intrinsics) — outside this skill. Route to the public DOCA DPA programming guide and the DPACC compiler guide via doca-public-knowledge-map/SKILL.md; this skill assumes the user has the DPA-side kernel and is asking how to load and launch it from the host.

• DPA device-side comms and verbs archives — shipped within doca-dpa and covered in ## comms / ## verbs; public guides per SKILL.md ## Related skills.

• DPA developer tools (the DPA debugger, the DPA process-state inspector, the DPA statistics tool) — named in ## Observability for routing, but the per-tool surface lives in the public DPA Tools umbrella via doca-public-knowledge-map/SKILL.md.

• DPACC compiler internals (flags, target options, how the host + DPA split-build is wired) — out of scope. Route to the public DOCA DPACC Compiler guide via doca-public-knowledge-map/SKILL.md.

• Core context internals, the cross-cutting DOCA_ERROR_* taxonomy and the cross-cutting debug ladder — owned by doca-programming-guide and doca-debug; see SKILL.md ## Related skills. This skill's ## debug carries the DPA overlay for layers 5-7 (DPA-driver route, two-side-program signature mismatch).

• GPU-initiated networking from a CUDA kernel — a separate DOCA-into-a-non-CPU-target path (CUDA on an NVIDIA GPU, not the DPA on a BlueField). Lives in doca-gpunetio/SKILL.md. The two skills share the DOCA-into-target-processor shape (paired-library setup, env-precondition matrix, two-axis capability rule, smoke-before-scale, dedicated completion mechanism) but differ in the target processor, the kernel toolchain (DPACC vs CUDA toolkit), the allocation primitives, and the kernel-side library set; the agent should treat them as siblings, not synonyms.