Omniverse Realtime Viewer

ovrtx Rendering

Triggers

Use this skill for ovrtx, renderer.step, step_async, LdrColor, AOVs, render vars, HdrColor, NormalSD, RenderProduct resolution, write_attribute, omni:xform, OVRTX_BIN_PATH, magenta materials, or RenderApi errors.

ovrtx is a headless RTX renderer driven from Python. The app owns the render loop, camera updates, frame extraction, and any streaming/display handoff.

For ovrtx renderer behavior, Python/C API behavior, release notes, or behavior not covered here, read references/dependencies for acquisition guidance and supplemental dependency documentation.

Core API

from ovrtx import Renderer, RendererConfig, Device, Semantic, PrimMode

renderer = Renderer(config=RendererConfig(
    sync_mode=True,
    active_cuda_gpus="0",
    keep_system_alive=True,
))
print(renderer.version)
renderer.open_usd("/path/to/composite.usda")
products = renderer.step(render_products={"/Session/Render/Viewport"}, delta_time=1.0 / 60.0)
with products as ctx:
    product = ctx["/Session/Render/Viewport"]

sync_mode=True blocks until the GPU frame is complete. Async pipelines can use False, but then buffer lifetime and frame readiness need explicit care.

renderer.step() returns RenderProductSetOutputs, not a Python dict. It supports [], in, keys(), values(), and items(), but not .get() or .update(). Some installed builds also support context-manager cleanup. Generated code should use context-manager cleanup when __enter__ is available, and otherwise consume the mapping-like result directly while copying required frame data before the next step.

renderer.step_async() enqueues the same frame work and returns an Operation. Poll op.query_status() from the runtime owner when you need the render loop to stay responsive, then call op.wait() before reading RenderProductSetOutputs. Do not mutate the stage while a step operation is in flight.

Stage Composition APIs

ovrtx 0.3 uses explicit stage composition:

• renderer.open_usd(path) replaces the active root layer with a file/URL.

• renderer.open_usd_from_string(usda) replaces the active root layer with generated inline USDA, commonly a wrapper root with subLayers and viewer-owned camera/render prims.

• renderer.add_usd_reference(path, prefix_path="/Runtime/Asset") adds referenced content under an existing root stage and returns a handle.

• renderer.add_usd_reference_from_string(usda, prefix_path="/Runtime/Asset") is the inline-string additive-reference path.

• renderer.remove_usd(handle) removes additive content by handle.

• renderer.reset_stage() clears the stage to empty. It is not needed for normal root replacement because open_usd* replaces the root.

Do not use older implicit stage-addition APIs as the main load path in 0.3 docs or examples.

Frame Extraction

products = renderer.step(render_products={RENDER_PRODUCT_PATH}, delta_time=dt)
with products as ctx:
    if RENDER_PRODUCT_PATH in ctx:
        product = ctx[RENDER_PRODUCT_PATH]
        for frame in product.frames:
            if "LdrColor" in frame.render_vars:
                with frame.render_vars["LdrColor"].map(device=Device.CUDA) as rv:
                    rgba_cuda = wp.from_dlpack(rv)  # H x W x 4, channel-last
                with frame.render_vars["LdrColor"].map(device=Device.CPU) as rv:
                    pixels = np.from_dlpack(rv).copy()  # H x W x 4 RGBA

CUDA mapping exposes linear CUDA memory. CPU mapping transfers data to host. For local UI, copy inside the map context before returning.

A 0.3 render variable output can contain one or more named tensors plus named params. For single-tensor outputs such as LdrColor, use the mapped object itself as the DLPack producer (np.from_dlpack(rv) / wp.from_dlpack(rv)). For multi-tensor outputs, address tensors by name (rv["Coordinates"], rv["Intensity"]) and params through rv.params["hitCount"]. Do not write new code against older single-tensor convenience access.

C maps an ovrtx_render_var_output_t with ovrtx_map_render_var_output(). Iterate its tensors[] and params[] by name; do not assume tensor index 0 is the only payload unless the RenderVar contract says it is.

Frame Result Lifetime

RenderProductSetOutputs, products, frames, mapped render var outputs, and tensor wrappers are per-step views into ovrtx-owned output. Map and copy render vars while the owning RenderProductSetOutputs object is still alive, inside the same render-loop step that produced them. When the step result supports __enter__, use with products as ctx: for deterministic cleanup; when it does not, still copy the data before the next renderer.step().

Do not return a frame, render var output, mapped tensor, DLPack wrapper, or NumPy/Warp view that still depends on the mapped ovrtx output. Do not hold references to frame data across later renderer.step() calls; the next step can invalidate handles from the previous step.

If frame data must move downstream to an encoder, streaming layer, UI bridge, worker queue, or logger, copy it before the step result goes out of scope. For CPU paths, use an owned array such as np.from_dlpack(rv).copy(). For CUDA paths, copy into an application-owned persistent CUDA buffer, such as a Warp array allocated outside the map context, before passing that buffer to the next stage.

Render Product Resolution

Set the viewer-owned RenderProduct resolution in the session or composite layer before loading the stage. The render product path must be the same path passed to renderer.step(render_products={...}).

Browser-streamed Omniverse Realtime Viewer apps should keep this resolution fixed, typically 1920x1080, and let the browser display the video with object-fit: contain. ovrtx does not expose a renderer.resize() API, and ovstream encoders should not be treated as live-resizable.

Render Vars And AOVs

The render product controls which AOVs ovrtx attempts to produce:

def RenderProduct "ViewportTexture0"
{
    rel camera = </OVCamera>
    rel orderedVars = [
        </Render/Vars/LdrColor>,
        </Render/Vars/HdrColor>,
        </Render/Vars/Depth>,
        </Render/Vars/Normal>,
        </Render/Vars/InstanceSeg>,
        </Render/Vars/SemanticSeg>,
        </Render/Vars/Metallic>,
        </Render/Vars/Roughness>,
        </Render/Vars/Emissive>,
        </Render/Vars/Diffuse>,
        </Render/Vars/Specular>,
        </Render/Vars/AO>,
        </Render/Vars/DirectDiffuse>,
        </Render/Vars/DirectSpecular>,
        </Render/Vars/IndirectDiffuse>,
        </Render/Vars/IndirectSpecular>,
        </Render/Vars/MotionVectors>,
    ]
}

def RenderVar "Normal"
{
    uniform string sourceName = "NormalSD"
}

frame.render_vars is keyed by source name, not necessarily by RenderVar prim name. For example, the RenderVar "Normal" prim appears as NormalSD in Python.

Common displayable AOVs for viewer apps:

All image tensors are channel-last [H, W, C]; scalar lanes are [H, W, 1], not [H, W]. The composite stage can request more render vars for future use, but do not expose them until they map to real non-empty data.

AOV Discovery

Discover available display AOVs from real frame output, then filter through a conservative allowlist.

DISPLAY_AOVS = (
    "LdrColor",
    "HdrColor",
    "NormalSD",
    "InstanceSegmentationSD",
    "SemanticSegmentationSD",
)

def _update_available_aovs(self, render_vars: Any, notify: bool = False) -> None:
    names = set(render_vars.keys()) if hasattr(render_vars, "keys") else set(render_vars)
    available = {name for name in DISPLAY_AOVS if name in names}
    if not available:
        available = {"LdrColor"}
    self._available_aovs = available

Log the full render_vars key list once when investigating new ovrtx versions, but use the allowlist for UI state. Presence in render_vars does not guarantee that mapping succeeds or that the tensor is non-empty.

AOV Conversion

ovstream expects one BGRA8 CUDA image. Convert every selected AOV into a persistent wp.uint8 [H,W,4] buffer:

with fout.render_vars[aov_name].map(device=Device.CUDA) as rv:
    # Single-tensor output. For multi-tensor outputs, use rv["TensorName"].
    src = wp.from_dlpack(rv)
    shape = tuple(int(dim) for dim in src.shape)
    height, width = shape[0], shape[1]
    channels = shape[2] if len(shape) >= 3 else 1
    dtype = src.dtype
    dim = (width, height)

    if dtype == wp.uint8 and len(shape) == 3 and channels == 4:
        wp.copy(self._stream_buf, src)
        wp.launch(_swap_rb, dim=dim, inputs=[self._stream_buf], device="cuda:0")
        return True

    if dtype == wp.uint32 and len(shape) == 3 and channels == 1:
        wp.launch(_colorize_seg_3d, dim=dim, inputs=[src, self._stream_buf], device="cuda:0")
        return True

Include kernels for uint8, uint16, uint32, float32, and float16, with channel-last scalar, RGB, and RGBA forms. See aov-switching for the full dispatch rules.

Live Attribute Writes

ovrtx consumes Fabric attributes such as omni:xform; standard authored USD xformOp:* is not the live update path.

xforms = np.array([...], dtype=np.float64).reshape(n, 4, 4)
renderer.write_attribute(
    prim_paths=paths,
    attribute_name="omni:xform",
    tensor=xforms,
    semantic=Semantic.XFORM_MAT4x4,
    prim_mode=PrimMode.CREATE_NEW,
    data_access=0,  # 0=ASYNC for GPU tensors, 1=SYNC for CPU numpy
)

PrimMode.EXISTING_ONLY silently skips attributes not already registered. Use CREATE_NEW for camera, animation, and EffectLayer attributes that may not exist in Fabric yet.

Renderer Config And Process State

Set renderer-wide behavior at construction time:

• sync_mode=True keeps step() blocking until the frame is complete. Use step_async() for responsive loops that can poll an Operation.

• active_cuda_gpus="0" or "0,1" restricts which CUDA-visible devices ovrtx can use. Values are indices into CUDA_VISIBLE_DEVICES.

• keep_system_alive=True keeps the ovrtx system initialized across idle/reset periods, which is useful for long-running viewers that replace stages repeatedly. Choose the value once at renderer creation.

The ovrtx logging callback is process-global state, not renderer-owned state. Install it once during process startup if the app needs to route ovrtx logs into its logger. Avoid registering a different callback per request, per viewer tab, or per renderer instance; in multi-viewer processes the last process-global registration is the one that matters.

First-Run Shader Compilation

The first renderer.step() on a cold machine or fresh cache can take 2-5 minutes while RTX pipelines and shaders compile. Treat this as a normal first-run cost before assuming the process is hung.

Use a longer timeout for first validation and CI runs, at least 300 seconds for the first rendered frame. Check the ovrtx log for shader or pipeline compilation progress before killing the process. GPU utilization can show 0% during parts of shader compilation because the work may be CPU-side, driver-side, or CUDA compute rather than graphics utilization.

Subsequent steps are usually fast once the shader cache is populated. If every run pays the full cold-start cost, check the local cache configuration in dependencies and make sure the cache directory is writable and persistent for the intended environment.

Schema Registration And Path Dictionary

ovrtx runtime population is schema-driven. Built-in stage data such as Cameras, RenderProducts, RenderVars, omni:xform, pickability, and selection-outline attributes use ovrtx-supported schemas. If an app depends on custom authored attributes, register the schema before loading the stage, or opt in with root-layer customLayerData.populateAllAuthoredAttributes = true when broad authored-attribute population is acceptable. The broad flag can significantly increase memory usage.

Path and token IDs returned by C queries or pick-hit buffers are not user-facing strings. Resolve them through the renderer path dictionary (ovrtx_get_path_dictionary() and path-dictionary utilities). Python high-level stage queries return path strings, while Python pick-hit decoding uses Renderer.resolve_prim_path_id().

Geometry Streaming

For USD assets, prefer open_usd* and add_usd_reference* composition. Use geometry streaming only for runtime-generated geometry or high-frequency geometry updates that are owned by the application. Keep schemas registered before streaming attributes, keep stream ownership/lifetime explicit, and continue to render with ovrtx; do not replace this with browser-side geometry rendering.

Multi-GPU

RendererConfig(active_cuda_gpus="0,1") enables multiple CUDA-visible GPUs for rendering, subject to the installed ovrtx build and RenderProduct configuration. Per-RenderProduct uint[] deviceIds = [...] is an allow-list into CUDA_VISIBLE_DEVICES; use it when a product must run on a specific device.

For WebRTC streaming and Warp conversion, keep the selected display RenderProduct on the CUDA device used by the stream buffer or add an explicit copy. Picking currently requires the picking RenderProduct to run on CUDA-visible GPU 0; pin pick products with deviceIds = [0] when multi-GPU rendering is enabled.

Environment

export OVRTX_SKIP_USD_CHECK=1
export OVRTX_BIN_PATH="$(python3 -c 'import ovrtx, os; print(os.path.join(os.path.dirname(ovrtx.__file__), "bin"))')"
export LD_LIBRARY_PATH="$OVRTX_BIN_PATH/plugins${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

If a separate USD build is present, ovrtx's bundled USD/plugins must resolve first or USD debug symbols can be registered twice (SDF_ASSET fatal errors).

Import Order

Use one import discipline per process:

• Streaming/direct ovrtx server: set OVRTX_SKIP_USD_CHECK, construct Renderer, then import pxr if direct queries are needed.

• Lightweight local Omniverse Realtime Viewer and OvGear paths: import pxr first, then let local renderer/adapter code import ovrtx as documented in local-viewer and ovui guidance.

• Windows streaming path: keep pxr in a subprocess; do not import it in the ovrtx process.

The reason is USD plugin registry ownership. Mixing usd-core and ovrtx's bundled USD in the wrong order can cause MDL resolver failures, _tf DLL import failures, or duplicate symbol crashes.

Minimal Render Example

import os
import numpy as np
from PIL import Image
os.environ["OVRTX_SKIP_USD_CHECK"] = "1"
from ovrtx import Renderer, RendererConfig, Device

renderer = Renderer(config=RendererConfig(sync_mode=True))
renderer.open_usd("/path/to/composite.usda")
for _ in range(60):
    products = renderer.step(render_products={"/Render/RenderProduct"}, delta_time=1/60)
    with products as ctx:
        product = ctx["/Render/RenderProduct"]
        for frame in product.frames:
            if "LdrColor" in frame.render_vars:
                with frame.render_vars["LdrColor"].map(device=Device.CPU) as rv:
                    Image.fromarray(np.from_dlpack(rv).copy()).save("output.png")

MDL Material Resolution

ovrtx bundles MDL assets under ovrtx/bin/library/mdl/, including Base/OmniPBR.mdl, Base/OmniGlass.mdl, and mdl/nvidia/core_definitions.mdl. Without OVRTX_BIN_PATH pointing at ovrtx/bin, materials importing ::OmniPBR::OmniPBR or ::nvidia::core_definitions::* can render magenta.

UJITSO "multi-node material unsupported" warnings are informational if the full MDL compiler can find the library. They become a problem when OVRTX_BIN_PATH or LD_LIBRARY_PATH is wrong.

Common Errors

See also: aov-switching, stage-loading, camera-controls, render-settings, selection-feedback, selection-animation, streaming-server.