Omniverse Realtime Viewer

Viewer Backend Interface

Triggers

Use this skill for ViewerBackend, local shared UI, backend-agnostic React viewer UI, reusable viewer components, viewer interface adapters, or cross-transport UI.

Use this when a frontend should share the same panels, inspectors, selection controls, asset list, and viewer widgets across WebRTC streaming, Electron SHM, Tauri SHM/native IPC, or a future transport.

Purpose

The local viewer UI module is generated React component code plus a TypeScript interface contract. The contract lets UI components depend on one ViewerBackend object instead of transport-specific APIs such as AppStreamer messages, Electron preload calls, Tauri commands, or future IPC layers.

The shared UI does not render USD or 3D content. All USD rendering still uses ovrtx in the appropriate server or native process. In OVStage-backed apps, the backend runtime also owns stage lifetime, ordinals, write floors, and renderer publication. The UI displays a video, canvas, or pixel surface owned by the transport and overlays reusable React controls that call ViewerBackend.

Read These Skills

Local Module Setup

Generate these files inside the frontend app unless the user requests a different local structure:

frontend/src/viewer-ui/
  ViewerBackend.ts
  types.ts
  StageTree.tsx
  PropertyPanel.tsx
  Inspector.tsx
  PrimIconSprite.tsx
  USDAsset.tsx
  index.ts

Use index.ts as the local barrel export. Do not add an external package dependency for shared viewer UI.

Import components and types from the local module:

import {
  Inspector,
  PrimIconSprite,
  PropertyPanel,
  StageTree,
  USDAsset,
  type FrameData,
  type PrimNode,
  type PrimProperty,
  type USDAssetItem,
  type ViewerBackend,
} from './viewer-ui';

The local index.ts exports Inspector, InspectorProps, PrimIconSprite, PropertyPanel, StageTree, StageTreeProps, USDAsset, and all shared viewer types.

ViewerBackend Interface

Implement this interface at the transport boundary. Components should call this interface, not raw WebRTC, Electron, Tauri, or server-message APIs.

export interface ViewerBackend {
  connect(): Promise<void>;
  disconnect(): void;
  loadStage(path: string): Promise<void>;
  resize?(width: number, height: number): Promise<void>;
  sendRenderScale?(scale: number): Promise<void>;
  setCamera(params: CameraParams): Promise<void>;
  cameraMouseButton(input: PointerInput): Promise<boolean>;
  cameraMouseMove(x: number, y: number): Promise<void>;
  cameraWheel(delta: number): Promise<void>;
  onFrame(callback: (frame: FrameData) => void): () => void;
  onStats?(callback: (stats: FrameBudgetStats) => void): () => void;
  onLoadProgress?(callback: (progress: LoadProgressEvent) => void): () => void;
  onAOVStateChanged?(callback: (active: string, available: string[]) => void): () => void;
  changeAOV?(aov: string): Promise<void>;
  getCapabilities?(): Promise<ViewerCapabilities>;
  invokeFeature?<T = unknown>(feature: string, payload: Record<string, unknown>): Promise<T>;
  onSelectionChanged(callback: (paths: string[]) => void): () => void;
  pick(x: number, y: number): Promise<string | null>;
  getStageTree(rootPath?: string): Promise<PrimNode[]>;
  selectPrims(paths: string[]): Promise<void>;
  getProperties(path: string): Promise<PrimProperty[]>;
}

Method contract:

Critical Coordinate Contract

pick(x, y), cameraMouseButton(input), and cameraMouseMove(x, y) use viewport-element-local CSS pixel coordinates, not window coordinates.

For a local adapter that owns DOM gestures, subtract the viewport element rect. WebRTC/AppStreamer owns native viewport input: subscribe to its resulting selection events instead of forwarding a second DOM camera/pick path. The local release handler below assumes its adapter also received the matching press and motion events:

function toViewportPoint(event: React.PointerEvent, viewport: HTMLElement) {
  const rect = viewport.getBoundingClientRect();
  return {
    x: event.clientX - rect.left,
    y: event.clientY - rect.top,
  };
}

const { x, y } = toViewportPoint(event, viewportElement);
const clicked = await backend.cameraMouseButton({ x, y, button: event.button, pressed: false });
if (clicked) {
  const pickedPath = await backend.pick(x, y);
  await backend.selectPrims(pickedPath ? [pickedPath] : []);
}

If a backend needs render-product pixels, map from viewport-local CSS pixels to the contained image/canvas area inside the backend or viewport adapter. Do not pass raw PointerEvent.clientX/clientY into ViewerBackend.

Frame Delivery Contract

FrameData.pixels is optional. WebRTC/AppStreamer transports usually render into a <video> element and can only report frame timing, decoded dimensions, or stats through onFrame. SHM, Tauri, or future pixel transports may provide RGBA pixels.

Shared UI components must not require pixel data. Viewport display components may choose the transport-specific surface:

Type Catalog

export interface CameraParams {
  azimuth?: number;
  elevation?: number;
  distance?: number;
  target?: [number, number, number];
}

export interface FrameData {
  width: number;
  height: number;
  encoding: 'rgba';
  pixels?: Uint8ClampedArray;
  frameIndex?: number;
  renderScale?: number;
}

export interface FrameBudgetStats {
  timestampMs: number;
  fps: number;
  sourceFps: number;
  frameIntervalMs: number;
  deliveryMs: number;
  queueDepth: number;
  skippedFrames: number;
  droppedFrames: number;
  renderScale: number;
  backpressure: boolean;
  frame_time_ms?: number;
  stream_time_ms?: number;
  frames_rendered?: number;
  gpu_encoder_active?: boolean;
  roundTripTimeMs?: number;
  packetsLost?: number;
  jitter?: number;
  framesDecoded?: number;
  freezeCount?: number;
  framesPerSecond?: number;
}

export type LoadProgressPhase =
  | 'resolving_asset'
  | 'loading_stage'
  | 'compiling_shaders'
  | 'ready';

export interface LoadProgressEvent {
  phase: LoadProgressPhase;
  stage_url?: string;
}

export type PrimType = 'xform' | 'scope' | 'geom' | 'light' | 'camera';

export interface PrimNode {
  name?: string;
  path: string;
  children?: PrimNode[] | null;
  hasChildren?: boolean;
  type?: PrimType;
}

export type USDPrim = PrimNode;

export interface PrimProperty {
  name: string;
  type: string;
  value: string;
}

export interface USDAssetItem {
  name: string;
  url: string;
}

export interface PointerInput {
  x: number;
  y: number;
  button: number;
  pressed: boolean;
}

export interface ViewerCapabilities {
  aovs?: string[];
  renderSettings?: string[];
  runtimeTransforms?: boolean;
  pickEffects?: string[];
  physicsImpulse?: boolean;
  transport?: 'webrtc' | 'electron-shm' | 'tauri' | 'ovui' | 'native' | string;
}

Normalize transport payloads to these types before data reaches shared components. In particular, convert server fields such as has_children or boolean children into hasChildren, and convert property dictionaries into PrimProperty[].

Capability Extensions

Keep optional features out of the required interface so the same UI can run on WebRTC, Electron SHM, Tauri, ovui, or future transports. Use getCapabilities() to decide whether to show controls, and use invokeFeature() or a typed adapter method in the transport adapter for the actual command.

Useful feature names:

• runtimeTransform: move, rotate, or scale selected prims through the runtime owner.

• pickEffect: apply a verified selection-driven material, visibility, or effect change.

• physicsImpulse: run a bounded physics worker and apply returned pose samples through parent runtime state.

Rules:

• Shared components should never receive live OVStage handles, OVRTX handles, DLPack buffers, SHM pointers, or process-local tokens.

• The backend adapter translates feature commands into data-channel messages, Tauri invokes, Electron IPC, or local API calls.

• Physics controls should be disabled unless the backend reports either a verified PhysX.attach_ovstage(stage, read_ordinal=...) path or a bounded OVPhysX worker path.

• A missing capability is a normal unsupported state, not a reason to add a browser-side renderer or mutate user USD files.

Shared Components

When generating components, keep prop names small and stable. Do not bake transport APIs into components; adapt the transport to ViewerBackend instead.

Minimum prop contracts:

export interface StageTreeProps {
  backend: ViewerBackend;
  selectedPaths: string[];
  rootPath?: string;
}

export interface PropertyPanelProps {
  properties: PrimProperty[];
}

export interface InspectorProps {
  backend: ViewerBackend;
  selectedPaths: string[];
}

export interface USDAssetProps {
  asset: USDAssetItem;
  backend?: ViewerBackend;
  onLoad?: (asset: USDAssetItem) => void;
}

Minimum behavior:

• StageTree loads root nodes with backend.getStageTree(rootPath) on mount, loads children when an expandable row opens, highlights selectedPaths, and calls backend.selectPrims([path]) on row activation.

• PropertyPanel renders a compact name/type/value table and treats values as display strings unless another skill adds editing behavior.

• Inspector subscribes to selected paths through props, fetches properties for the primary selected path with backend.getProperties(path), cancels stale responses on selection change, and renders an empty state for no selection.

• PrimIconSprite may be a no-op component when the app uses text labels or CSS icons instead of an SVG sprite.

• USDAsset renders one loadable asset row or button and calls onLoad(asset) when supplied; otherwise it calls backend.loadStage(asset.url) when a backend prop is supplied.

Implementing a New Backend

Use the hook-per-transport pattern:

useWebRTCBackend.ts -> AppStreamer + data-channel messages + video timing
useShmBackend.ts    -> Electron preload API + SHM frame pump + JSON control messages
useTauriBackend.ts  -> Tauri commands/events/channels + native frame delivery

Implementation rules:

• Return one stable ViewerBackend object from the hook with useMemo.

• Keep transport objects, request resolvers, caches, and subscriber sets in refs.

• Convert transport events into shared callbacks: frame, stats, load progress, AOV state, and selection.

• Add timeouts to request/response promises for loadStage, pick, getStageTree, and getProperties.

• Resolve responses by request id when the protocol supports it; otherwise bucket by URL/path/message type.

• Normalize all hierarchy nodes, property values, AOV names, and selected paths at the backend boundary.

• Make disconnect() clear subscribers only when the app is truly tearing down; ordinary event-handler cleanup should only unsubscribe that handler.

• Preserve the single render-thread/server ownership rules from the transport skill. The React backend is an adapter, not a renderer.

Resolver pattern:

type Resolver<T> = {
  resolve: (value: T) => void;
  reject: (error: Error) => void;
  timer: number;
};

const selectionHandlers = useRef(new Set<(paths: string[]) => void>());
const frameHandlers = useRef(new Set<(frame: FrameData) => void>());
const treeResolvers = useRef(new Map<string, Resolver<PrimNode[]>[]>());
const propertyResolvers = useRef(new Map<string, Resolver<PrimProperty[]>[]>());
const treeCache = useRef(new Map<string, PrimNode[]>());

function emitSelection(paths: string[]) {
  for (const handler of selectionHandlers.current) handler(paths);
}

function resolveBucket<T>(buckets: Map<string, Resolver<T>[]>, key: string, value: T) {
  const bucket = buckets.get(key) || [];
  buckets.delete(key);
  for (const resolver of bucket) {
    window.clearTimeout(resolver.timer);
    resolver.resolve(value);
  }
}

Transport notes:

• WebRTC/AppStreamer: connect() owns one AppStreamer connection. Route onCustomEvent messages into resolver buckets and subscriber sets. onFrame usually publishes timing from video playback quality or stats; leave pixels undefined.

• Electron SHM: connect() attaches preload/native SHM APIs and starts one frame pump. Convert SHM frames to the viewport's upload path and publish shared FrameData only for data the shared UI needs.

• Tauri SHM/native IPC: connect() registers Tauri event listeners and frame channels once. Decode binary RGBA frames when the React layer owns a canvas; otherwise publish timing/dimensions and let the viewport adapter own pixels.

Integration Recipe

1. Generate frontend/src/viewer-ui/ with the ViewerBackend types and local React components required by the app.

2. Implement a backend hook for the chosen transport: useWebRTCBackend, useShmBackend, useTauriBackend, or a new hook.

3. Mount the transport viewport surface: <video>, <canvas>, or a native pixel-presenting component.

4. Give the viewport one input owner. WebRTC uses AppStreamer native input; local DOM adapters convert coordinates before calling camera/pick methods.

5. Subscribe to backend selection and frame/stat events in React effects.

6. Render shared UI components against the backend rather than raw transport APIs.

7. Keep transport-specific message names and native handles inside the backend hook.

WebRTC skeleton (the backend hook owns AppStreamer input and connection state):

function ViewerApp() {
  const backend = useWebRTCBackend(config); // stable identity for this endpoint
  const [connectionError, setConnectionError] = useState<string | null>(null);
  const [selectedPaths, setSelectedPaths] = useState<string[]>([]);

  useEffect(() => {
    let active = true;
    void backend.connect().catch(error => {
      if (active) setConnectionError(String(error));
    });
    return () => { active = false; backend.disconnect(); };
  }, [backend]);

  useEffect(() => backend.onSelectionChanged(setSelectedPaths), [backend]);


  return (
    <>
      <PrimIconSprite />
      {connectionError && <div role="alert">{connectionError}</div>}
      <main className="viewer-shell">
        <section className="viewport">
          <video id="remote-video" />
        </section>
        <aside className="sidebar">
          <StageTree backend={backend} selectedPaths={selectedPaths} />
          <Inspector backend={backend} selectedPaths={selectedPaths} />
        </aside>
      </main>
    </>
  );
}

Keep the dependency direction the same: app UI calls local viewer UI components, those components call ViewerBackend, and only backend hooks know the transport.