Omniverse Realtime Viewer

ovui Local Setup, Shell, And Renderer

2. Install Dependencies And Configure The Environment

Read references/dependencies FIRST. Its nvidia-runtime.md file is the source of truth for NVIDIA runtime locations. Do not guess or repeat ovui package URLs, wheel names, artifact locations, or fallback install commands in this recipe.

Do this:

• Resolve the Python environment from the current NVIDIA-Omniverse/ovui README.md, AGENTS.md, relevant skills/omniverse-ui-* references, and package metadata before creating the venv. Do not hard-code a Python version in generated local ovui setup notes.

• Install ovrtx using the package guidance in references/dependencies.

• Install ovstage using the package guidance in references/dependencies when the local runtime uses OVStage as the live stage substrate.

• Install ovui using the current PyPI package guidance in references/dependencies.

• Install any selected local UI companion packages from the same ovui package set as the base UI package.

• Install usd-core only when the app creates a USD query worker. Resolve a release compatible with the selected runtime package set after reviewing the current upstream runtime documentation and release notes. Keep routine pxr imports out of the local process that owns OVRTX and live OVStage unless the exact wheel set and import path are verified.

• Install ovphysx only in a bounded physics worker or isolated attached-stage compatibility probe unless the selected local app has verified PhysX.attach_ovstage(stage, read_ordinal=...) for the exact OVStage/OVPhysX ABI.

• Install numpy for matrices, camera math, and CPU frame handling (pip install numpy).

• Install warp-lang only when the local app needs CUDA-side display or image-processing utilities (pip install warp-lang).

• Do not install ovstream or frontend streaming packages for a local-only viewer.

Set these environment contracts before starting the app:

• OVRTX_SKIP_USD_CHECK=1 must be set before ovrtx is imported or the renderer is constructed.

• OVRTX_BIN_PATH must point at the ovrtx bin directory when materials or renderer plugins fail to resolve.

• The ovrtx plugin library path must be first in the dynamic library path if another USD build is present.

• PYTHONPATH must include any ovui import paths required by the selected package.

• A real display must be available. For headless validation, use a configured X display such as Xvfb.

Decision points:

• Prefer a subprocess for USD queries and keep the main app focused on ovui, OVRTX, and live OVStage/session state.

• If USD registry or DLL conflicts appear, treat that as confirmation that pxr queries belong in a subprocess.

• If OVPhysX attach fails with missing OVStage bridge symbols, run physics in a bounded child process and hand pose samples back to the parent; do not call OVPhysX in the local viewer process.

• If the user only needs rendering and camera navigation, defer usd-core until hierarchy or bounds queries are required.

• If running on Windows, read references/windows-native-setup, especially Local ovui On Windows, before changing import order or subprocess strategy. Skip its streaming-only frontend/WebRTC sections for local-only apps.

• If local UI imports report missing package metadata or missing VIEWPORT_CAMERA_POSE_SOURCE, read references/dependencies and use one compatible local UI package set.

Common failure modes:

• usd-core detected, duplicate USD debug symbols, _tf import failures, or MDL resolver crashes usually mean import order or library path is wrong, or a second USD-populating runtime entered the viewer process.

• Magenta materials usually mean OVRTX_BIN_PATH or plugin library path is missing.

• ovui import failure usually means the selected package was not installed or its required import path is not on PYTHONPATH.

• A blank or crashing local window in CI usually means no real display is available.

Read for depth: see references/dependencies, references/local-viewer, references/ovrtx-rendering, and references/windows-native-setup for the full environment contracts.

3. Build The ovui Window Shell

Do this in local_app/app.py and local_app/viewport.py:

• Initialize ovui once with the requested title, width, height, and target FPS.

• Create one main window that fills the app window.

• Use a large central viewport, a compact header or toolbar, an optional scene tree or info sidebar, and an optional render settings panel.

• Put the rendered image inside a viewport frame that can resize with the window.

• Add a transparent hit surface above the image for mouse input.

• Route toolbar actions to runtime commands such as load scene, reload, fit camera, clear selection, toggle tree, and toggle render settings.

• For lightweight viewers, prefer a path field plus one LOAD button for scene loading. Add native file dialogs only after they are validated in the same display/session environment as the app.

• Wrap ovui callbacks so exceptions are logged and do not tear down the app loop.

Critical contracts:

• Use fill_app_window=True for the main ovui window so the UI frame tracks GLFW window resizes.

• The viewport widget must report its current size or provide a reliable fallback so letterbox math remains valid.

• The transparent mouse surface must be above the image and marked as receiving mouse events.

• Header, tree, info, and settings widgets must not intercept viewport gestures unless the pointer is actually over those controls.

• The app loop must call runtime tick/update work from one predictable place.

• If a selected-prim transform gizmo is visible, enqueue local omni:xform updates through the application-owned transform adapter. The serialized runtime writes OVStage, waits for completion, advances the write floor, and renders the committed ordinal; a visual-only handle is not complete.

Decision points:

• If the user wants a compact visual tool, keep a header plus viewport and place advanced panels in collapsible sidebars.

• If the user wants a dense inspection app, reserve width for a stage tree and prim info panel from the start.

• If the user wants a context menu, show it on right-button release only when the drag threshold was not exceeded.

• If the user wants a camera gizmo, build it as a local ovui scene overlay rather than using streaming overlay patterns.

• If the user wants selected-prim movement in a lightweight shell, either use a proven transform manipulator that emits transform deltas or add an app-owned fallback that treats presses near the selected pivot as transform drags.

Common failure modes:

• Without fill_app_window=True, the OS window resizes but the UI frame stays at the initial dimensions.

• If the hit surface is behind the image or not opaque to mouse events, orbit, pan, zoom, and picking never receive input.

• A separate OPEN button can be worse than no button if the native dialog is not implemented or cannot appear under the target display stack. Keep the path-field LOAD path reliable and make dialogs secondary.

• A selected-prim gizmo that appears but does not move the prim usually means camera/orbit consumed the gesture, or the transform adapter did not write and publish the selected prim's local omni:xform before the render step. See references/prim-transform-safety for parent transforms and animation coordination.

• Uncaught callback exceptions can stop the app loop or leave drag state stuck.

• DOM-style browser assumptions do not apply; input comes from ovui callbacks in the desktop process.

Read for depth: see references/local-viewer, references/viewer-input-routing, references/camera-controls, and references/viewport-overlays for ovui shell, input routing, context menu, and local overlay guidance.

4. Construct The ovrtx Renderer Runtime

Do this in local_app/renderer_runtime.py:

• Create the ovrtx renderer after environment variables are set and import order is settled.

• Use synchronous rendering first.

• Store the active render product path, render width, render height, current frame index, and whether a valid stage is loaded.

• Expose render-loop-only operations for loading a scene, resetting the stage, stepping a frame, mapping render vars, enqueueing and decoding native pick queries, writing native selection outline groups, and writing live attributes.

• Keep renderer mutations serialized with scene loading, scene reset, settings changes, and camera writes.

• Use an application-owned OVStage for scene data and attach the renderer to it. Publish runtime mutations through OVStage/session state at monotonic ordinals and render only committed publications. Keep native picking and selection outline groups on OVRTX.

Critical contracts:

• The application calls renderer.step() explicitly. ovrtx does not run a hidden app loop for the viewer.

• Pass the exact viewer RenderProduct path to every step call.

• Extract LdrColor for local image display. It is RGBA8 from ovrtx.

• For local UI display, copy CPU-mapped pixel data inside the map context before returning it to the widget.

• Write live camera and object transforms through the OVStage runtime adapter, using local omni:xform values and the publication flow in references/prim-transform-safety.

• Direct renderer.write_attribute and renderer bindings are only for an explicitly standalone renderer-owned compatibility path. Follow references/ovrtx-rendering for its semantic and create-new prim mode; do not mix those writes with an attached OVStage.

Decision points:

• If the app only needs basic local viewing, start with LdrColor only.

• Object selection uses native ovrtx pick queries. Do not add segmentation render vars just to make picking work.

• If the app needs high-FPS local display, profile CPU readback first before introducing a CUDA-to-UI path.

• If the renderer reports stale GPU hangs after a crash, inspect running Python GPU processes before changing code.

Common failure modes:

• Unable to find RenderProduct prim means scene setup did not create the path used by renderer.step().

• Black frame usually means camera relation, render product resolution, render var source, or camera transform is invalid.

• Live camera changes doing nothing usually means the app wrote xformOp:transform instead of omni:xform, used existing-only prim mode, or published above the OVStage write floor.

• Crashes during scene switches usually mean renderer.step() overlapped a reset, load, or layer mutation.

Read for depth: see references/ovrtx-rendering for the full renderer construction, frame extraction, and live attribute contract.

5. Implement Scene Loading

Do this in local_app/scene_loader.py and call it only from the render loop or serialized runtime load path:

• Resolve the requested URL/path against the configured asset root, allowed schemes, and security policy.

• Create viewer-owned camera, RenderProduct, RenderVar, and RenderSettings data through one inline root/session USDA string when the user stage lacks viewer render config.

• Load the user stage without modifying it.

• Store the viewer camera path and render product path in runtime state.

• Reset selection, native selection outline groups, stage query caches, pending pick queries, and loading status for the new stage.

• Fit the camera to the stage bounds unless the user requested preserving the current camera or using an authored stage camera.

Critical contracts:

• Every loaded stage needs Camera -> RenderProduct -> RenderVar -> RenderSettings wiring that ovrtx can find.

• The viewer camera path must be the same path used by camera controls when writing omni:xform.

• Do not inject lights unless the user explicitly asks for viewer-controlled lighting.

• Include segmentation render vars only for explicit debug/AOV display modes, not for picking.

• Prefer OVStage population.open_usd_from_string() for inline roots that sublayer the user USD and author viewer render config. Publish the ordinal before the attached renderer consumes it.

• Do not call renderer-owned reference or layer-add APIs in an attached viewer; use OVStage population operations through the serialized runtime path.

Decision points:

• Use a single inline root USDA string with subLayers = [@user_scene@] when the user file needs viewer camera/render-product/render-var data. Copy source upAxis and metersPerUnit onto that root as described in stage-loading.

• If the user stage has an authored camera and the requested policy is stage-camera, copy its focal length, apertures, clipping range, and transform into the viewer camera.

• If the user requests persistent camera across scene switches, keep camera state but sanitize and refit only when the old state is invalid for the new bounds.

• If the user requests viewer lighting controls, add explicit viewer-owned light prims only with a verified live apply path or an explicit reload/profile workflow; otherwise leave lighting untouched and omit live lighting controls.

Common failure modes:

• Inline roots that omit or misquote the user sublayer path fail composition or break relative asset resolution.

• Camera path mismatch makes input appear connected but the view never moves.

• A stage-load operation that reports an error must not be treated as a successful load just because the enqueue call returned.

Read for depth: see references/stage-loading, references/render-settings, references/selection-feedback, and references/stage-hierarchy for the full scene setup contract.

6. Display The Rendered Image

Do this in local_app/viewport.py:

• Create an ImageBridge with the render width and height.

• Display the bridge provider through an ovui image widget using preserve-aspect fit.

• Before connecting ovrtx output, push a synthetic RGBA gradient through the same provider path and capture a window screenshot proving the native widget paints nonblank, non-solid pixels.

• On each rendered frame, update the bridge with copied RGBA pixels from LdrColor.

• Compute the visible image rectangle inside the viewport widget after every resize.

• Store the current widget size, visible image offset, visible image size, render width, and render height for input mapping.

• Show explicit idle, loading, and error states when no frame should be displayed.

Critical contracts:

• The ovrtx LdrColor buffer is RGBA8 and is suitable for local image display through ImageBridge.

• Copy CPU-mapped pixel data while the render var map context is still open.

• Preserve-aspect display creates letterboxing. All pick and camera coordinates must pass through the same letterbox transform.

• If render resolution changes, recreate or resize all dependent state: render product, ImageBridge, letterbox math, pending pick coordinate state, and viewport overlays.

• Do not run scene load/reset work while the image update is reading render output.

• A nonblank direct LdrColor artifact plus a blank ovui window points at presentation, not scene setup. Fix or switch the ovui presentation path before changing camera, render product, or USD composition.

• If ImageBridge or ByteImageProvider updates do not paint in the active ovui runtime, validate with another ovui-native presentation path such as a RasterImageProvider screenshot/frame fallback instead of continuing to debug renderer state.

Decision points:

• If the target is a simple viewer, keep render resolution fixed and let the image widget scale with letterboxing.

• If the target requires pixel-perfect viewport resolution, treat resize as an explicit render product reconfiguration path.

• If the app needs a screenshot command, read from the same copied LdrColor frame used by display unless a higher-quality render path is requested.

• If CPU readback is too slow, investigate a GPU-native UI path only after the basic Omniverse Realtime Viewer is correct.

Common failure modes:

• Reading pixels after the map context closes returns invalid or stale data.

• Ignoring letterboxing makes picks offset and camera drag speed inconsistent.

• Recreating the image bridge on every frame causes flicker, memory churn, or UI stalls.

• Changing render resolution without updating viewport math and pick query coordinates causes selection to drift.

• A wrong background color can be a style-color packing problem. Use ui.color("#76b900") or ui.color(r, g, b, a) rather than hand-packed integers.

Read for depth: see references/local-viewer, references/ovrtx-rendering, and references/object-selection for image bridge, frame extraction, and coordinate mapping details.