Omniverse Realtime Viewer

Streaming Client And Protocol

7. Build The React WebRTC Client

Do this in frontend/src/streaming/ and frontend/src/App.tsx:

• Create a streaming provider that resolves host and signaling port from URL parameters first, environment variables second, a stream.config.json file third, and browser hostname plus default port last.

• Render the video element with the exact id expected by AppStreamer before attempting to connect.

• Connect with the Direct stream mode and the resolved signaling host/port from a stable React effect.

• Style the video with object-fit: contain so the fixed-resolution stream scales to the available page area without distortion.

• Expose status, error message, connection lifecycle events, a guarded send helper, and custom-event subscription to the rest of the app.

• Mount the viewport video first, then overlay DOM controls such as toolbar, scene picker, tree, property panel, status, and settings.

• Clean up custom-event subscriptions on component unmount.

Critical contracts:

• Use server in the Direct config, not signalingServer.

• Do not configure mediaServer or mediaPort for the browser client. SDP negotiates media.

• Do not construct sign-in URLs, append custom signaling paths, or add auth/session fields to the browser WebRTC config. Orchestrators may launch and route the session, but the frontend should receive a standalone ovstream Direct host and signaling port.

• The video element must exist before connect starts.

• Gate initial data-channel sends on connected status.

• Keep the AppStreamer.connect() effect dependent only on immutable connection config. Do not let stateful message routers recreate the effect after openStageResult, getChildrenResult, or status events arrive.

• Catch rejected AppStreamer.sendMessage() Promises during connect/disconnect windows.

• Give DOM overlays explicit z-index above the video layer.

• Use modest reconnect settings to avoid repeated previous session already running churn.

• Let the streaming library forward mouse, keyboard, wheel, and touch input through NVST's native input channel. Do not duplicate these as app JSON.

• Do not send viewport-size messages when CSS layout changes. Keep the stream fixed and rely on NVST letterbox coordinate mapping.

Decision points:

• If the app should auto-load a default scene, send openStageRequest only after connected status is reached.

• If the server loads an initial scene before the browser connects, rely on server initial-state push rather than requiring the frontend to guess.

• If the user wants a dense tool UI, keep it as DOM overlay around the video. Use server-side overlays only when they must be part of the streamed pixels.

• If the client runs from a different machine, expose host and signaling port through URL parameters.

Production stream.config.json

For production builds served as static files (e.g., from a Docker container or CDN), the frontend cannot rely on Vite environment variables or dev-server proxying. Place a stream.config.json in the frontend dist/ directory:

{
  "source": "local",
  "local": {
    "server": "<server-ip-or-hostname>",
    "signalingPort": 49100,
    "mediaPort": null,
    "mediaServer": "<server-ip-or-hostname>"
  }
}

• server: IP or hostname of the ovstream signaling server, reachable from the client browser.

• signalingPort: WebSocket signaling port (default 49100).

• mediaPort: Set to null to let SDP negotiation determine the media port.

• mediaServer: Usually the same as server; set differently only if media routes through a separate IP.

The frontend should fetch stream.config.json at startup and use its values as defaults when URL parameters are not provided. This enables zero-rebuild reconfiguration of the streaming target for containerized or remote deployments.

Common failure modes:

• Connecting before the video element exists produces a connection that cannot display video.

• Setting media port to the signaling port produces connected-with-no-video failures.

• Sending requests before data channel readiness drops initial scene loads.

• One rendered frame then black can be caused by React effect cleanup calling AppStreamer.terminate() after normal app state updates.

• A frontend waiting for getChildrenResponse while the server sends getChildrenResult leaves the tree empty.

Read for depth: see references/streaming-client and references/streaming-lifecycle for the full React/AppStreamer contract.

8. Define The Data-Channel Protocol

Do this before wiring UI features:

• Define the app envelope as event_type plus payload in both directions.

• Make the server unwrap the browser library's outer message envelope when present. The app message may arrive inside a data field rather than as the top-level object.

• Register handlers by exact event name.

• Validate each payload before mutating renderer or USD state.

• Send all responses and pushed events through one guarded send helper.

Use this message set for the complete Omniverse Realtime Viewer:

Critical contracts:

• Exact event names matter. Use openStageResult, getChildrenResult, and stageSelectionChanged.

• paths: [] in selectPrimsRequest clears selection.

• Children semantics must be stable: expandable-but-not-loaded is truthy, loaded children is an array, and leaf is null or absent.

• Properties and variants responses must include the requested prim_path so the frontend can ignore stale responses.

• Message handlers that load scenes, set variants, or change heavy settings must enqueue work for the render thread.

• Message handlers that only query cached state may respond immediately if they do not touch renderer-owned state.

• setRenderSettingRequest must reject keys that are not in the server capability list. Success means the active viewer state changed, or an explicit non-live action was accepted.

• For runtime prim discovery, prefer renderer.query_prims(...) / query_prims_async(...) and return its resolved path strings. Use a pxr worker only for queries not exposed through ovrtx native stage APIs.

• Push openStageResult and root getChildrenResult after a client connects if a stage is already loaded.

Decision points:

• If stage hierarchy queries are slow or risky in the renderer process, move them behind a subprocess owned by stage_queries.py.

• If multiple UI panels can request the same data, make responses idempotent and keyed by prim path instead of relying on request ordering.

• If a feature is optional, still reserve its event names in one central protocol table to avoid drift.

Common failure modes:

• Failing to unwrap the outer browser envelope makes the server see messageType instead of event_type.

• Using older response names such as getChildrenResponse breaks current frontend routing.

• Sending app state before the data channel is ready causes permanent loading indicators unless initial state is pushed on connection.

Read for depth: see references/streaming-messages, references/streaming-lifecycle, and references/stage-hierarchy for the full protocol and query contracts.

14. Wire Frontend Components

Do this after the stream connects and core messages work:

• Viewport renders the video element and any DOM overlay controls.

• Toolbar sends fit camera, reset view, render settings toggle, and debug view commands.

• ScenePicker lists available scenes and sends openStageRequest.

• StageTree requests children lazily and sends selectPrimsRequest on row selection.

• PrimInfoPanel subscribes to selection and requests properties/variants for the active prim.

• RenderSettingsPanel renders backend-advertised capabilities, displays effective settings, and sends validated changes.

• StatusBar displays connection, loading state, current scene, FPS if available, and latest viewer error.

Critical contracts:

• The video element id must match AppStreamer config exactly.

• Keep data-channel sends behind the streaming provider's connected-state guard.

• Treat server events as authoritative for loaded scene, selection, settings, and errors.

• Do not duplicate pointer input handling in DOM unless it is for UI widgets outside the video viewport.

• Clean up event subscriptions when components unmount.

Decision points:

• If a panel is optional, still keep the provider and message types stable so the feature can be added later.

• If a UI widget overlaps the video, ensure it does not intercept pointer events meant for camera/selection unless the widget is actively being used.

• If the app needs keyboard shortcuts, decide which shortcuts are browser UI shortcuts and which should pass through to the streamed app input path.

Common failure modes:

• DOM overlays intercept all pointer events and camera controls stop working.

• Frontend local state diverges when it assumes requests succeeded instead of waiting for server events.

• Components leak subscriptions and process every server event multiple times after navigation.

Read for depth: see references/streaming-client, references/streaming-messages, and references/streaming-lifecycle for the full frontend contracts.