Doca Gpunetio

DOCA GPUNetIO capabilities, version overlay, errors, observability, safety

Where to start: Pick the H2 anchor that matches your question (per-GPU context / GPU-visible queue handles / persistent-kernel pattern / dual capability discovery / env preconditions / errors) and read that section end-to-end. The tables in each section are the load-bearing content; the prose around them is interpretation.

Read this file when the loader sent you here from SKILL.md. For the how of executing each pattern (the verbs configure / build / modify / run / test / debug), jump to TASKS.md. For the canonical DOCA version-handling rules that this skill layers a GPUNetIO overlay on top of, see doca-version/SKILL.md.

Pattern overview

Every GPUNetIO question this skill teaches resolves into one of SIX patterns. The patterns are CLASSES — they apply across every GPUNetIO release and every host + GPU + NIC combination.

Two cross-cutting rules that apply to every pattern above:

• GPUNetIO sits on top of doca-eth, not below it. The agent recommending that the user "create a GPU queue first and the Ethernet queue follows from it" has the layering backwards for every version of GPUNetIO. Configure doca-eth first, then layer GPUNetIO handles on top.

• Capability is a TWO-axis question. "Is this feature supported on this host" requires BOTH a DOCA cap-query against the active doca_devinfo AND a CUDA device-capability check (cudaGetDeviceProperties). Either axis missing the support fails the feature. An agent that quotes only one axis will miss half the "silently fails on this GPU" cases.

Capabilities and modes

The two orthogonal selection axes for any GPUNetIO design are which CUDA device (doca_gpu per device ordinal) and which doca-eth queue is being exposed to the GPU (RX, TX, or both). Choose both before writing any CUDA kernel code, then drill into the relevant capability-query.

The per-GPU doca_gpu context — one per GPU (created from the GPU's PCIe bus-id string).

A multi-GPU host needs one doca_gpu per device the user wants to drive — there is no "global GPU context". The agent must ask which CUDA device the user intends to drive before recommending any doca_gpu_* call, and must surface cudaGetDeviceProperties(devOrdinal) as the canonical way to confirm the device is GPUNetIO-capable.

The GPU-visible queue handles — layered on doca-eth.

The GPU-visible handles do not own the underlying queue — they expose it to the GPU. Tearing down the underlying doca-eth queue without first tearing down the GPU-visible handle is a use-after-free on the GPU side; the agent must surface this ordering explicitly when discussing lifecycle.

Persistent-kernel pattern — the canonical first-app shape. The default GPUNetIO usage shape is one long-running CUDA kernel that polls the GPU-visible RX queue in a loop, processes each packet in place on the GPU, and optionally pushes results out via the GPU-visible TX queue. This is the pattern the GPU Packet Processing reference application uses; the agent should treat it as the default and treat "launch a kernel per packet" as an anti-pattern that defeats the entire reason to use GPUNetIO (the kernel-launch overhead per packet swamps the network savings).

Dual-axis capability discovery — the only rule. Before sizing any queue, assuming a feature is available, or recommending the persistent-kernel pattern works on the user's device, run BOTH a DOCA cap-query AND a CUDA device-capability check. Either axis missing the support fails the feature.

DOCA 3.5 public API surface. The GPUNetIO public header is the source of truth for enumerator families and function signatures; do not copy constant lists or prototypes from older DOCA releases. In DOCA 3.5 the following constant families were added or changed: DOCA_GPUNETIO_ETH_READY_MODE_*, DOCA_GPUNETIO_VERBS_READY_MODE_*, DOCA_GPUNETIO_MLX5_WQE_CTRL_*, DOCA_GPUNETIO_VERBS_CQ_*, DOCA_GPUNETIO_VERBS_ATOMIC_EXT_BYTES_*, DOCA_GPUNETIO_VERBS_SYNC_SCOPE_*, DOCA_GPUNETIO_VERBS_NIC_HANDLER_*, and DOCA_GPUNETIO_VERBS_GPU_CODE_OPT_*. See the public header for the exact enumerators.

DOCA 3.5 also added the doca_gpu_verbs_get_qp_* getter family (doca_gpu_verbs_get_qp_cq_rq, doca_gpu_verbs_get_qp_cq_sq, doca_gpu_verbs_get_qp_qp) for retrieving the CQ/RQ, CQ/SQ, and underlying QP from a doca_gpu_verbs_qp; use these getters to obtain QP/CQ members rather than relying on removed or changed fields. DOCA 3.5 added doca_gpu_verbs_req_notify_cq for requesting completion notification on a doca_verbs_cq, and added doca_gpu_nic_handler_to_string for converting a NIC-handler enumerator to a string.

Configuration shape. Mandatory preconditions before any doca_gpu_* call: the underlying doca_eth_rxq / doca_eth_txq must be configured and the doca-eth context must be at the right lifecycle stage for the GPUNetIO layer to attach; the CUDA toolkit must be installed and cudaSetDevice(devOrdinal) must succeed against the target GPU; the CUDA buffers that will hold the receive payload pool must be allocated via cudaMalloc (or equivalent CUDA allocator) and registered with DOCA via the doca_buf_arr_create_* family. Optional configurations (queue sizing, persistent-kernel block / thread counts on the CUDA side) are program-side tunables that ride on top of the same cap-query rule.

Version compatibility

For the canonical DOCA version-detection chain, the four-way match rule, NGC container semantics, and the headers-win-over-docs rule, see doca-version/SKILL.md. The body lives there; this skill does not duplicate it.

The GPUNetIO-specific overlay is:

• DOCA must match CUDA per the DOCA Compatibility Policy. The public DOCA Compatibility Policy at https://docs.nvidia.com/doca/sdk/doca-compatibility-policy/index.html defines which CUDA toolkit version a given DOCA release supports. Mismatched CUDA + DOCA combos fail at link time (missing CUDA-side symbols) or at runtime (DOCA_ERROR_DRIVER from doca_gpu_create) in confusing ways. The agent must compare pkg-config --modversion doca-gpunetio with pkg-config --modversion doca-common, require those DOCA versions to agree, then compare nvcc --version (and cat /usr/local/cuda/version.txt when present) against the DOCA Compatibility Policy before any GPUNetIO bring-up.

• DOCA 3.5 changed the GPUNetIO public API. The supported constant families are DOCA_GPUNETIO_ETH_READY_MODE_*, DOCA_GPUNETIO_MLX5_WQE_CTRL_*, DOCA_GPUNETIO_VERBS_ATOMIC_EXT_*, DOCA_GPUNETIO_VERBS_CQ_*, DOCA_GPUNETIO_VERBS_GPU_CODE_OPT_*, DOCA_GPUNETIO_VERBS_NIC_HANDLER_*, DOCA_GPUNETIO_VERBS_READY_MODE_*, and DOCA_GPUNETIO_VERBS_SYNC_SCOPE_*. The agent must treat a constant from outside these families as a DOCA 3.5 API mismatch and route to doca-version TASKS.md ## debug (doca-version/TASKS.md#debug) before GPUNetIO-layer diagnosis. The authoritative list is in the installed GPUNetIO public header that declares doca_gpu_verbs_bridge_export_qp; do not copy constant values or prototypes from memory.

• The DOCA cap-query AND cudaGetDeviceProperties are both runtime authorities. Per the cross-cutting cap-query rule in doca-version CAPABILITIES.md ## Observability (doca-version/CAPABILITIES.md#observability), the DOCA-side cap query is the runtime authority for "is this GPUNetIO feature on this DOCA install". The CUDA-side device-properties query is the runtime authority for "is this GPU capable of GPU-initiated networking at all". Either being false fails the feature; the agent must report both.

• doca-gpunetio.pc, doca-common.pc, and the matching CUDA toolkit must all line up at the four-way-match check (per doca-version CAPABILITIES.md ## Version compatibility (doca-version/CAPABILITIES.md#version-compatibility)). A common partial-install pattern is that DOCA was upgraded but the CUDA toolkit on the host was not, or vice versa; the agent must surface that as a four-way-match failure and route to doca-version TASKS.md ## debug (doca-version/TASKS.md#debug) before any GPUNetIO-layer diagnosis.

Error taxonomy

GPUNetIO-specific overlays on the cross-library DOCA_ERROR_* taxonomy. The cross-library taxonomy itself lives in doca-programming-guide CAPABILITIES.md ## Error taxonomy (doca-programming-guide/CAPABILITIES.md#error-taxonomy); the rows below are the GPUNetIO surface meaning that the agent must disambiguate before falling back to the cross-library response.

The agent's rule: never recommend a retry loop on DOCA_ERROR_* without first identifying which of the rows above is the cause. _AGAIN is the only one that wants a device-side drain-then-retry; the others want investigation (env / version / lifecycle), not retry.

Observability

GPUNetIO observability surface is two-sided: there is a DOCA-side observability surface (per-task completions, the DOCA logger, capability-query snapshots) AND a CUDA-side observability surface (CUDA device counters, the persistent-kernel's own in-kernel logging, CUDA error returns from launch / sync). The agent must reach for both, not just one — a hung persistent kernel that produces no DOCA log line is almost always visible on the CUDA side via cuda-gdb or kernel-internal counters.

Three primary signals the agent should reach for:

1. DOCA-side per-task completions. Every queue submit produces a completion (success or failure) that the persistent kernel can read from the device side. Absence of a completion for submitted packets is always a queue-not-drained bug; the persistent kernel needs an explicit drain step in its loop.

2. Capability snapshot at configure time. The output of doca_eth_rxq_cap_is_type_supported(devinfo, ...) (the doca-eth query in doca_eth_rxq.h) AND cudaGetDeviceProperties(devOrdinal) together is the baseline of "what the library + the GPU said was possible" before any kernel was launched. Save both; if a runtime call later returns DOCA_ERROR_NOT_SUPPORTED the diff against this baseline is the bug.

3. CUDA-side health. nvidia-smi -q -d UTILIZATION while the persistent kernel runs shows whether the GPU is actually busy with work; a persistent kernel that reports 0% util is stuck on a queue drain. cuda-gdb attached to the host process can step into the device-side kernel when DOCA-side completions stop arriving without an explicit error.

DOCA 3.5 added two observability/debugging entry points in the public GPUNetIO header (doca_gpunetio.h). Use doca_gpu_nic_handler_to_string() to convert a doca_gpu_dev_verbs_nic_handler value to a string before logging bridge-export configuration; do not log the raw enum. Use doca_gpu_verbs_req_notify_cq() to request a completion notification on a struct doca_verbs_cq obtained from a doca_gpu_verbs_qp through the new doca_gpu_verbs_get_qp_* getter family, so the persistent kernel can wait for completions instead of polling blindly. See doca_gpunetio.h for the exact signatures and the DOCA_GPUNETIO_VERBS_* and DOCA_GPUNETIO_ETH_* constant families.

For cross-cutting observability primitives (--sdk-log-level, the DOCA_LOG_LEVEL env var, the trace build flavor) see doca-debug CAPABILITIES.md ## Observability (doca-debug/CAPABILITIES.md#observability). For the install-tree observability (logger names, package layout, sample tree) defer to doca-public-knowledge-map/SKILL.md.

Safety policy

Overlay on the bundle-wide hardware-safety meta-policy. The rules below are this skill's per-artifact overlay on the cross-cutting rules in doca-hardware-safety CAPABILITIES.md ## Safety policy (doca-hardware-safety/CAPABILITIES.md#safety-policy) (specifically ### Per-artifact overlay pattern (doca-hardware-safety/CAPABILITIES.md#per-artifact-overlay-pattern)). When the two layers disagree, the stricter wins; when either layer says STOP, the agent stops.

GPUNetIO's safety surface is env-precondition-driven. Unlike RDMA (permission-driven) or Comch (representor-visibility-driven), the single most common GPUNetIO first-app failure is that the host's CUDA + DOCA combination is not what the user thinks it is — and the agent's job is to verify that before any doca_gpu_* call, not after the first DOCA_ERROR_DRIVER.

The env-precondition matrix the agent must walk for any new GPUNetIO setup:

Do not partial-install one side. A CUDA-only install without DOCA, or a DOCA install with no CUDA toolkit, fails GPUNetIO in non-obvious ways: pkg-config may succeed for DOCA while the link line fails on CUDA symbols, or DOCA may succeed at create-time while the first runtime call returns DOCA_ERROR_DRIVER from CUDA below. The fix is to align both sides per the DOCA Compatibility Policy, not to silence the error.

Lifecycle ordering is GPU-aware. The GPU-visible handle must be destroyed BEFORE the underlying doca-eth queue is destroyed; the doca_gpu context must be destroyed BEFORE the CUDA context is unbound from the process. Out-of-order teardown surfaces as DOCA_ERROR_BAD_STATE on subsequent calls but also leaves the GPU in an undefined state for the next process; the agent must surface this ordering explicitly.

Deferred topic boundaries

This skill scopes itself to the DOCA GPUNetIO library. Adjacent topics the agent will get asked but should route elsewhere:

• CUDA programming in general (CUDA kernel design, CUDA memory model, stream / event semantics, multi-stream scheduling) — outside this skill. Route to the upstream CUDA toolkit documentation; this skill assumes the user already understands CUDA C++ and is asking how to wire a DOCA queue into a CUDA kernel.

• DOCA Ethernet queue setup (doca_eth_rxq / doca_eth_txq bring-up, RSS, representor selection on the NIC side) — GPUNetIO depends on it but does not redefine it. Route to doca-eth/SKILL.md.

• DOCA DPA (DPA-side networking primitives, DPA kernel programming) — the sibling DOCA library for offloading device-side compute onto a different non-CPU target. The agent should NAME DPA when asked about "DOCA libraries that integrate with non-CPU compute" (see the worked example in TASKS.md ## modify) but should not invent DPA API surfaces here. Route via doca-public-knowledge-map/SKILL.md to the public DOCA DPA guide.

• DOCA Core context and progress engine internals — owned by doca-programming-guide/SKILL.md. This skill uses the Core lifecycle; it does not redefine it.

• Cross-cutting DOCA_ERROR_* taxonomy — owned by doca-programming-guide CAPABILITIES.md ## Error taxonomy (doca-programming-guide/CAPABILITIES.md#error-taxonomy). This skill adds the GPUNetIO overlay, not the taxonomy itself.

• Cross-cutting debug ladder (install / version / build / link / runtime / program / driver) — owned by doca-debug ## debug (doca-debug/TASKS.md#debug). This skill's ## debug redirects there for layer 1-4; layers 5-7 carry the GPUNetIO-specific overlay (including the CUDA-driver-layer route).