Doca Urom Svc

DOCA UROM Service — Capabilities

Where to start: The pattern overview below names the recurring DOCA-UROM-Service-class operational patterns. Pick the pattern first, then drill into the H2 that owns the substance. For the how of executing each pattern, jump to TASKS.md.

This file enumerates the DOCA UROM Service's documented capabilities, deployment shape, configuration axes, and operational behaviors as described in the public DOCA UROM Service Guide. Treat it as a map of what is documented, not a substitute for reading the live page when configuring a real deployment. For the public URL itself, route through doca-public-knowledge-map ## DOCA services (doca-public-knowledge-map/SKILL.md#doca-services) — this skill does not duplicate the URL routing.

Pattern overview

Every DOCA-UROM-Service-class question this skill teaches resolves into one of SIX patterns. The patterns are CLASSES — they apply across every service deployment, not just one BlueField generation or one MPI / UCX consumer.

Two cross-cutting rules that apply to every pattern above:

• The host-library + DPU-service paired contract is load-bearing. The DOCA UROM Service does nothing on its own — its inputs are operations enqueued by the paired host-side doca-urom library. An agent that walks a user through this service in isolation, without naming the paired host library and the version-coupling rule, has the model wrong for every deployment. The corollary: a perfectly healthy service container that does not pair with any host's doca-urom version is operationally indistinguishable from a misconfigured one.

• Operate the documented path; do not invent one. The service's CLI-flag / env surface, container image source, supported UCX components / collectives, and queue-sizing knobs are all documented in the public DOCA UROM Service Guide. Quoting daemon flags, image tags, UCX component names, or container-runtime flags not in the public guide is the most common hallucination failure mode for this skill.

Capabilities and modes

Service shape

DOCA UROM Service is a long-running container that ships from NGC and runs on the BlueField Arm cores. The container is the daemon: it owns the service-side execution state for UROM offloads, receives enqueued operations from paired host processes through the DOCA contract, and dispatches them onto the underlying RDMA substrate. There is no host-side service binary the user installs — the service is the container; the host's relationship to the service is to enqueue operations through the doca-urom library.

Three architectural properties the operator must hold throughout:

• The service is the EXECUTOR; the library is the PUBLISHER. Per the paired-contract model below, the service's purpose is to execute remote memory operations that host processes have enqueued through doca-urom. The service does NOT execute operations the library didn't enqueue, does NOT discover work on its own, and does NOT drive the host's progress engine. Conflating the two is the single most common UROM-deployment design error.

• The container is the unit of deployment. Operators do not start the service as a host binary; they start the service container per the public Container Deployment Guide pattern (same shape as every other DOCA service container — see the sibling doca-dms/SKILL.md and doca-firefly/SKILL.md for the same shape on different per-service domains).

• Service behavior is configured by CLI flags and env, not by a mounted config file. The daemon binary (doca_urom_daemon) parses its options through DOCA's doca_argp (e.g. --max-msg-size / -m, and the log level -l / --sdk-log-level, passed via the container's SERVICE_ARGS), and reads the plugin search path from the UROM_PLUGIN_PATH environment variable. doca_urom.yaml mounts only the plugins/ directory and the log directory — there is no mounted service config file. The agent should quote the daemon's actual flags from the public guide / --help, NOT invent config-file keys.

Publisher / executor paired-contract model

DOCA UROM Service is one half of a TWO-component contract. The host-side doca-urom library publishes operations; this service executes them. Both halves must be present at versions the DOCA Compatibility Policy supports, before any operation can flow.

The agent's rule: when the user asks "how do I write host-side code that enqueues UROM operations / how do I size the per-context queue on the host / how do I drive doca_pe_progress for completions", that is the host-library question and the right artifact is doca-urom/SKILL.md. When the user asks "how do I deploy / start / stop / scale the service on the BlueField / how do I configure which UCX components it exposes / how does the host pair over DOCA Comch", that is this skill's scope. Two distinct artifacts; two distinct surfaces.

Deployment shape

The public DOCA UROM Service Guide documents the container deployment on BlueField Arm. The shape lines up with every other DOCA service container — pull from NGC, set the daemon's CLI flags / env (SERVICE_ARGS, UROM_PLUGIN_PATH) and mount the plugins/ directory, start under the documented runtime (the BlueField OS's container manager per the public Container Deployment Guide). For the canonical container-deployment recipe shared with the other DOCA service containers, route through doca-public-knowledge-map ## DOCA services (doca-public-knowledge-map/SKILL.md#doca-services).

Two deployment-shape rules:

• BlueField Arm only. The DOCA UROM Service is a BlueField-side service; it does not run on the host. The host's relationship to the service is via the DOCA contract carried over doca-urom and the underlying RDMA fabric.

• One service per BlueField. The service drives the BlueField's UROM execution state for the BlueField as a whole. The daemon reaches the host over DOCA Comch and moves bytes over the underlying doca-rdma substrate; there is no service listen-port the operator chooses — the control transport is the Comch endpoint pair, and the daemon allocates its internal worker ports automatically. Running two service containers competing for the same execution state on the same BlueField is a configuration error, not a redundancy strategy.

Configuration axes

Every DOCA UROM Service deployment must commit to three configuration axes before starting the container. Get any one wrong and the service either rejects host enqueues, stalls them silently, or accepts them but cannot actually execute the operation family the host intended. The axes are jointly documented in the public DOCA UROM Service Guide; quote the exact valid values from there rather than from memory.

The RDMA side follows least privilege: export only the memory regions the intended UROM operation requires, with only the documented access permissions needed for that operation. Do not broaden an export or grant read/write/atomic permissions merely to make bring-up easier.

The agent's rule: the configuration-axes decision precedes container start. A deployment that starts the container before the operator can name which UCX components / collectives the service exposes, what queue depth supports the intended in-flight load, and how the host pairs over DOCA Comch is going to debug the wrong axis first. Force the decision up front.

Pairing surface — host library plus RDMA substrate

The DOCA UROM Service sits in the middle of two paired surfaces; neither is optional and the agent must surface both:

The agent's rule: when the user mentions the host doca-urom library or the RDMA fabric in the same breath as this service, name BOTH paired surfaces in the same response. Naming only one is how the deployment's actual failure layer gets misattributed — host symptoms blamed on the service, service symptoms blamed on the substrate, substrate symptoms blamed on the library.

Version compatibility

For the canonical DOCA version-detection chain, the four-way match rule, NGC container semantics, and the headers-win-over-docs rule, see doca-version/SKILL.md. The body lives there; this skill does not duplicate it.

The DOCA-UROM-Service-specific overlay is the load-bearing version-coupling rule:

• The host-side library version and the DPU-side service version must agree per the DOCA Compatibility Policy. This is the single most consequential version axis the agent must surface for any UROM deployment: a host-side doca-urom upgraded without the DPU-side service being upgraded (or vice versa) does NOT fail loudly — it fails subtly, often as DOCA_ERROR_NOT_SUPPORTED for an operation family that DOES exist on one side but not the other, or as silent stalls on collectives that one side believes the pair supports. Surface BOTH pkg-config --modversion doca-urom on the host AND the service container tag on the BlueField; cross-check them against the DOCA Compatibility Policy. This pairing is what the cross-cutting overlay in doca-urom CAPABILITIES.md ## Version compatibility (doca-urom/CAPABILITIES.md#version-compatibility) documents on the host side; this skill documents the same pairing from the service side.

• The service container tag is the runtime version anchor. Same pattern as doca-dms/SKILL.md and doca-firefly/SKILL.md: the service container ships from NGC with its own tag that may lag the BlueField host's DOCA package version, and the relevant version anchor for an as-deployed service is the container tag pulled, not pkg-config --modversion on the BlueField. Always quote both versions when the user reports a service behavior; if they diverge, route to doca-version TASKS.md ## debug (doca-version/TASKS.md#debug) layer 2 before diagnosing the service behavior itself.

• The exposed UCX-component / collective surface is version-bound. Which collective primitives the service can expose (all-to-all, all-reduce, broadcast, …) is jointly conditional on the service container version, the BlueField generation, and the underlying DOCA install. When the user asks "does this collective work on my deployment?", the authoritative answer is the public DOCA UROM Service Guide page whose version matches the container tag pulled — NOT agent memory and NOT a guess from a different DOCA release.

• Read the public DOCA UROM Service Guide version header. The guide is versioned; the on-page version must match the container tag the operator is using. A mismatch between the docs version and the container tag is the canonical "my config doesn't match what the docs say" failure mode and often masquerades as a paired-version mismatch with the host library.

Error taxonomy

DOCA UROM Service errors fall into four layers, each with its own owner. The agent's rule: walk the layers in order; do NOT skip down without clearing the layer above. Notably, the host-visible DOCA_ERROR_* for a service-side cause is documented on the host side in doca-urom CAPABILITIES.md ## Error taxonomy (doca-urom/CAPABILITIES.md#error-taxonomy); this table maps each host-visible symptom back to the service-side layer the operator must touch.

The agent's rule: never recommend a service config change without first identifying which of the four layers is the cause. The most common debug failure for this skill is misreading a layer-4 symptom (host-library / service version mismatch) as a layer-2 (queue / handler) problem and chasing config knobs that cannot fix it. A second trap: a host-visible DOCA_ERROR_NOT_PERMITTED is a DOCA Comch-pairing / RDMA-permission signal (or a host-side doca_dev access problem), NOT a service-side authorization rejection — there is no such authz layer in the shipped binary.

Observability

Documented observability surfaces the agent should reach for, in order of how cheaply they answer the "is the service actually executing what the host enqueued" question:

1. Container state. First — is the service container actually running on the BlueField? The BlueField container manager reports container status, restart count, and the container's stdout / stderr log stream. A restart loop is a layer-1 (container runtime) symptom per ## Error taxonomy; diagnose it before touching any service config. A non-running container makes every host-side doca-urom call meaningless.

2. Service-side logs. The container's stdout (and any documented log destination the public DOCA UROM Service Guide specifies) is the primary service observability surface. Look for: (a) queue-saturation or handler-stuck lines (layer 2); (b) underlying-substrate-error lines (layer 3); (c) paired-version-mismatch lines if the service detects an incompatible host (layer 4). The agent should NOT invent log line formats; quote what the live container is emitting.

3. Host-side completion surface. The host's doca-urom library reports operation completions through the DOCA progress engine. The host-side observability is documented in doca-urom CAPABILITIES.md ## Observability (doca-urom/CAPABILITIES.md#observability); the service-side reading of "how many operations did we actually execute" is read from the service's own logs and from the substrate counters below. When the host says "submitted N, completed M" and the service says it executed K, the differences between N / M / K identify which layer is dropping work.

4. Underlying RDMA substrate counters. When the service-side surface shows operations being received but the user reports the remote-side memory does not have the expected bytes, the right diagnostic is the substrate counters per doca-rdma CAPABILITIES.md ## Observability (doca-rdma/CAPABILITIES.md#observability) on each BlueField in the path. The agent must NAME the existence of this surface and route the user there; the per-counter details belong in the substrate skill.

5. Version snapshot at deploy time. The service container tag, pkg-config --modversion doca-urom on the paired host(s), and the BlueField's doca_caps --version are the baseline of "which paired-version pair the operator deployed". Save them; if a runtime failure later looks like a layer-4 paired-version mismatch, the diff against this baseline (one side upgraded, the other didn't) is the bug.

For the cross-library debug-time observability (DOCA_LOG_LEVEL, --sdk-log-level, the trace build flavor — relevant when the service calls into a DOCA library that emits structured logs), see doca-debug CAPABILITIES.md ## Observability (doca-debug/CAPABILITIES.md#observability).

Safety policy

Overlay on the bundle-wide hardware-safety meta-policy. The rules below are this skill's per-artifact overlay on the cross-cutting rules in doca-hardware-safety CAPABILITIES.md ## Safety policy (doca-hardware-safety/CAPABILITIES.md#safety-policy) (specifically ### Per-artifact overlay pattern (doca-hardware-safety/CAPABILITIES.md#per-artifact-overlay-pattern)). When the two layers disagree, the stricter wins; when either layer says STOP, the agent stops.

The DOCA UROM Service's safety surface is path-selection first, then the version-contract rule, then the smoke-before-scale rule, then the operational disciplines around the container itself.

• Path-selection rule (load-bearing). Deploy this service ONLY when the HPC environment actually benefits from DPU offload of remote memory operations. Concretely:

  • Deploy when the host fleet uses the doca-urom library AND the workload pattern (MPI / UCX collectives, dense remote-memory traffic) is one where host-CPU cost of posting RDMA work is the bottleneck AND the BlueField generation in use can support the intended UCX components / collectives.

  • Do NOT deploy when the host fleet is not using doca-urom, when the HPC stack is neither MPI nor UCX (this service won't help — it executes UROM-shaped offloads, not arbitrary networking), or when the BlueField generation is too constrained to expose the intended offload surface. In those cases the right answer is to keep the host's existing communication path and explicitly tell the user "this service is the wrong tool here; here's why" — not to deploy it speculatively and end up debugging an offload that was never going to help.

• Version-contract rule (load-bearing). The host-side doca-urom library version and this service's container tag are a paired contract — see ## Version compatibility. A service deployment that does not name the paired-host library version, OR that proceeds against a pair the DOCA Compatibility Policy doesn't support, will fail subtly: not with a loud refusal at deploy time, but with DOCA_ERROR_NOT_SUPPORTED on operations one side believed the pair supports, or with silent stalls on collectives. The agent must always surface BOTH versions and the policy check up front.

• Cap-query at deploy time. Before declaring the service ready, the operator must confirm that the BlueField generation in use, the DOCA install on the BlueField, and this service version actually expose the UCX components / collectives the host workload intends to offload. The device-side cap surface is the authoritative answer for "can this BlueField even host this offload" — see the device-axis row of the cap query the host runs via doca-urom CAPABILITIES.md ## Capabilities and modes (doca-urom/CAPABILITIES.md#capabilities-and-modes); the answer must agree with the service's exposed surface or the deployment is misconfigured before any host enqueue.

• Smoke before scale. Before pointing the full HPC workload at the service, the operator must walk a smoke: service container running on the BlueField; ONE paired host's doca-urom doca_ctx_start succeeds against this BlueField; ONE host enqueue (a simple put or get) is observed at the service (in logs) AND ONE completion fires on the host's progress engine. Only then layer the collective patterns on top. A workload that comes up before the smoke passes does not isolate which of the four layers in ## Error taxonomy is wrong, and bisection across container / queue / substrate / version becomes much harder.

• One service per BlueField. Two service containers on the same BlueField competing for the same UROM execution state is a configuration error; the agent must NOT recommend it as a redundancy strategy. UROM redundancy is a cluster-side concern (multiple BlueFields, multiple service instances each owning their own BlueField) that does not require multiple service containers on one DPU.

• Don't paper over a transport-substrate problem from the service. When the symptom is "operations enqueued, never complete" and the layer is "underlying RDMA substrate down" per ## Error taxonomy, the honest answer is "the RDMA fabric isn't carrying the traffic; the fix is at the substrate, not in this service's config". Silently masking a substrate failure inside the service — by, e.g., shortening timeouts to make the host see the failure faster — is a user-visible regression dressed up as helpfulness.

• There is no service-side authorization model to invent. Access to the service is governed by the DOCA Comch endpoint pairing and the underlying RDMA permissions, not by any service-side authorization list, credential, or per-host allow-list in the shipped binary. The agent must NOT propose authentication / authorization schemes the service does not implement; a host-visible DOCA_ERROR_NOT_PERMITTED is a Comch / RDMA / doca_dev signal, not a UROM-service authz decision.

• Intended-host-only pairing is the access boundary. Before start, identify the exact host endpoint(s) permitted to pair, verify that mapping through the documented read-only doca-comch/SKILL.md surface, and refuse startup when the pairing is broad, ambiguous, or shows an unintended peer.

• RDMA exports are least privilege. Export only the memory regions needed by the intended workload and grant only its required read, write, or atomic permissions. Verify the documented doca-rdma capability, connection, and permission state through doca-rdma TASKS.md ## test (doca-rdma/TASKS.md#test) before service start; never use a wider export as a diagnostic shortcut.

• Pre-start substrate gate. Container start is blocked until both checks are green: the Comch view identifies only the intended host pairing, and the RDMA view confirms the intended peer, narrowly scoped exports, required permissions, and healthy transport. Capture both read-only outputs as the pre-start evidence.

Public-source pointer

The single canonical public source for the DOCA UROM Service is the DOCA UROM Service Guide, reachable through doca-public-knowledge-map ## DOCA services (doca-public-knowledge-map/SKILL.md#doca-services). Verify that the version of the guide matches the service container tag pulled on the BlueField AND that the host-side doca-urom library version pairs with that container tag per the DOCA Compatibility Policy — the service's config surface, supported UCX components / collectives, queue knobs, and observability output are documented to evolve, so config keys and exposed surfaces can change between releases.