Omniverse Realtime Viewer

Tauri SHM Transform Gizmo

Triggers

Use this skill for Tauri SHM viewer transform manipulation, client-rendered gizmos, canvas overlay handles, translate/rotate/scale controls, or gizmo-first input dispatch in the shared-memory local viewer path.

Use this whenever a Tauri SHM frontend needs to manipulate selected USD prims with an interactive transform gizmo while the rendered 3D image still comes from ovrtx through shared memory. The browser WebView renders only the overlay UI; it never renders the USD scene.

Architecture Overview

React WebView
  |-- WebGLViewport.tsx       -> displays SHM frame and dispatches pointer input
  |-- GizmoOverlay.tsx        -> pointer-events:none 2D canvas overlay
  |-- TransformGizmo.ts       -> hit testing, drag state, transform commands
  |-- GizmoRenderer.ts        -> 2D handle drawing
  `-- useShmTauriBackend.ts   -> message routing and request tracking

Tauri Rust backend
  |-- shm_reader.rs           -> queues commands to SHM reader thread
  `-- SHM reader thread       -> sends commands to ovrtx process

The gizmo is client-rendered. It draws translate, rotate, and scale handles on a transparent 2D canvas positioned over the viewport. The canvas should use pointer-events: none; the viewport receives pointer events and asks the gizmo to hit-test before forwarding input to the camera or picking path.

The drag lifecycle uses window-level pointermove, pointerup, and pointercancel listeners. Do not depend on pointer capture for this path.

Transform updates are sent over Tauri IPC as fire-and-forget messages. Continuous dragging must not block on Rust, the SHM reader thread, or a response from the renderer process.

WebKit2GTK Constraint

This is the critical Linux Tauri constraint: setPointerCapture on or around pointer-events: none overlay elements can swallow pointerup in WebKit2GTK. The result is a stuck drag, stale gizmo state, or camera input that never receives a release.

The required pattern is:

1. Configure the gizmo with capturePointer: false.

2. Keep the overlay canvas at pointer-events: none.

3. Hit-test the gizmo from WebGLViewport during onPointerDown.

4. If the gizmo starts a drag, register window-level pointermove, pointerup, and pointercancel listeners.

5. Remove those listeners when the drag ends or the component unmounts.

This differs from browser viewers where pointer capture may be acceptable. For Tauri on Linux, use window-level listeners instead.

Gizmo-First Input Dispatch

WebGLViewport should check the transform gizmo before sending pointer input to camera orbit, pan, zoom, or pick logic.

function onPointerDown(event: React.PointerEvent<HTMLCanvasElement>) {
  const viewportPoint = toViewportPoint(event);

  if (gizmo.current?.pointerDown(viewportPoint, event)) {
    beginWindowDragListeners();
    event.preventDefault();
    event.stopPropagation();
    return;
  }

  backend.sendInputEvent({
    type: "pointerDown",
    x: viewportPoint.x,
    y: viewportPoint.y,
    button: event.button,
    modifiers: readModifiers(event),
  });
}

The gizmo owns the drag only after a successful handle hit. Non-gizmo pointer events continue to the viewport's normal camera and selection input path.

Canvas Overlay

GizmoOverlay.tsx should render a full-size canvas over the SHM viewport:

• absolute-position the canvas over the displayed frame;

• set pointer-events: none;

• resize for CSS pixels and device pixel ratio;

• clear and redraw whenever selection, camera matrices, viewport rect, or active drag state changes;

• draw handles in screen space with depth-aware ordering where possible.

Use the same letterboxed image rect used by the SHM frame display. Gizmo handle positions must match the visible rendered image, not the full WebView content area when the viewport is letterboxed.

Camera Matrix Flow

The server can send camera matrices through these message types:

• cameraMatricesUpdate: explicit view/projection matrix update for overlays;

• cameraState: initial camera state after stage load or viewer setup;

• cameraUpdate: camera state after orbit, pan, zoom, fit, or other camera changes.

The frontend should normalize both messages into the gizmo's camera cache and dispatch a DOM event for overlay consumers:

window.dispatchEvent(
  new CustomEvent("ovrtx-camera-update", { detail: cameraState }),
);

The gizmo uses the latest camera view/projection data to project the selected object origin and axes into screen space.

Matrix convention is row-major storage with column-vector multiplication:

clip = projection * view * world * position

Keep this convention explicit in gizmo/math.ts; inconsistent row/column interpretation will make handles drift, invert, or disappear behind the camera.

Scale Gizmo: Local vs World Axes

Scale handles must align with the selected object's local coordinate frame, not the world axes. This is especially visible when a prim has an authored rotation: world-axis scale handles modify the wrong visual direction and make the gizmo feel detached from the object.

Extract normalized basis vectors from the selected transform matrix rows 0, 1, and 2:

const localX = normalize([m[0][0], m[0][1], m[0][2]]);
const localY = normalize([m[1][0], m[1][1], m[1][2]]);
const localZ = normalize([m[2][0], m[2][1], m[2][2]]);

Use those local basis vectors to draw and hit-test scale handles. Translate and rotate handles should use world axes, which is the standard behavior for this viewer path.

IPC Architecture

Both send_message and send_input_event must be fire-and-forget on the Rust side. They should queue a command to the SHM reader thread and return immediately.

Do not use a blocking mpsc::channel round trip for SHM commands. That pattern can deadlock when the Tauri IPC thread waits for a reply while the SHM reader thread is busy inside a frame callback or waiting on the same event flow. The command path should be:

Tauri command -> enqueue command to SHM reader thread -> return Ok(())

Request/response operations still need correlation, but the wait belongs in the frontend:

const pendingByRequest = new Map<string, PendingRequest>();

function sendAndWait<T>(message: ShmMessage, timeoutMs = 1500): Promise<T> {
  const requestId = crypto.randomUUID();
  backend.sendMessage({ ...message, requestId });

  return new Promise((resolve, reject) => {
    const timeout = window.setTimeout(() => {
      pendingByRequest.delete(requestId);
      reject(new Error(`Timed out waiting for ${message.type}`));
    }, timeoutMs);

    pendingByRequest.set(requestId, { resolve, reject, timeout });
  });
}

Use this request map for discrete queries such as getTransform. Continuous drag updates should not use sendAndWait.

Server Message Contract

The Tauri frontend expects the ovrtx process to handle two transform-specific messages:

• getTransformRequest: request/response query for the selected prim. Respond with getTransformResult and payload { matrix, position }, where matrix is the row-major 4x4 world transform and position is its world origin.

• set_transform: fire-and-forget drag update with payload { path, matrix }. Queue this onto the renderer/render-loop owner and apply it there; do not mutate USD/ovrtx state from the SHM reader or IPC thread.

For live ovrtx writes, follow prim-transform-safety: initialize new omni:xform attributes from the real world transform before rendering, and recreate bindings after stage reloads.

Drag Message Coalescing

Rapid pointer movement can generate more transform messages than the backend can consume. Coalesce continuous drag sends so the backend sees the latest transform at a bounded rate. The validated sample uses a 33 ms throttle with a single pending latest message:

const TRANSFORM_SEND_INTERVAL_MS = 33;
let lastSendTime = 0;
let pendingSend: number | null = null;
let lastMessage: ShmSetTransformMessage | null = null;

function sendThrottled(message: ShmSetTransformMessage) {
  lastMessage = message;
  const elapsed = performance.now() - lastSendTime;

  if (lastSendTime === 0 || elapsed >= TRANSFORM_SEND_INTERVAL_MS) {
    clearPendingSend();
    lastMessage = null;
    doSend(message);
    return;
  }

  clearPendingSend();
  pendingSend = window.setTimeout(() => {
    pendingSend = null;
    const next = lastMessage;
    lastMessage = null;
    if (next) doSend(next);
  }, TRANSFORM_SEND_INTERVAL_MS - elapsed);
}

This keeps the latest drag result while avoiding unbounded message growth. Do not use sendAndWait for drag updates. If your transport exposes a promise, catch failures for logging but do not make pointer motion wait for a response.

Message Listener Stability

Register the Tauri listen("shm-message", ...) handler once. Do not re-register it whenever selection, camera, pending requests, or component state changes. Re-registration creates an unsubscribe/subscribe gap where response messages can be missed.

Use a ref for the current handler logic:

const handleMessageRef = useRef<(message: ShmMessage) => void>(() => {});

handleMessageRef.current = (message) => {
  routeMessage(message);
};

useEffect(() => {
  let unlisten: (() => void) | undefined;

  listen<ShmMessage>("shm-message", (event) => {
    handleMessageRef.current(event.payload);
  }).then((dispose) => {
    unlisten = dispose;
  });

  return () => {
    unlisten?.();
  };
}, []);

Keep pendingByRequest in a ref or stable store so request/response routing does not require listener churn.

Generated File Layout

Use this generated app file layout for the Tauri SHM transform gizmo:

• clients/tauri-shm/src/components/GizmoOverlay.tsx

• clients/tauri-shm/src/gizmo/TransformGizmo.ts

• clients/tauri-shm/src/gizmo/math.ts

• clients/tauri-shm/src/gizmo/GizmoRenderer.ts

• clients/tauri-shm/src/components/WebGLViewport.tsx

• clients/tauri-shm/src-tauri/src/shm_reader.rs

• clients/tauri-shm/src/hooks/useShmTauriBackend.ts

Implementation Checklist

1. Add the overlay canvas and make it visually track the SHM frame rect.

2. Cache camera state from cameraMatricesUpdate, cameraState, and cameraUpdate messages.

3. Dispatch CustomEvent("ovrtx-camera-update") after camera messages are normalized.

4. Implement getTransformRequest/getTransformResult and set_transform on the ovrtx process side.

5. Query the selected prim transform with a request id and timeout.

6. Draw translate and rotate handles along world axes.

7. Draw scale handles along the selected prim's local transform basis.

8. In WebGLViewport, route pointer down through gizmo hit testing first.

9. Drive active drags with window-level move/up/cancel listeners.

10. Send drag transforms with fire-and-forget IPC and latest-message throttling.

11. Keep listen("shm-message", ...) registered once with ref-based routing.

Anti-Patterns

// Wrong for Tauri/WebKit2GTK: pointerup can be swallowed.
element.setPointerCapture(event.pointerId);

// Wrong: scale axes should come from the selected object's local transform.
const scaleAxes = [WORLD_X, WORLD_Y, WORLD_Z];

// Wrong: continuous drag updates must not wait for request/response messages.
await sendAndWait({ type: "setTransform", transform });

// Wrong: blocks the IPC thread waiting on the SHM reader thread.
let (reply_tx, reply_rx) = std::sync::mpsc::channel();
reader_tx.send(Command::SendMessage { message, reply_tx })?;
reply_rx.recv()?;

• Do not use setPointerCapture in Tauri/WebKit2GTK.

• Do not use synchronous IPC for SHM commands.

• Do not re-register event listeners on state changes.

• Do not draw scale handles along world axes.

• Do not send unbounded transform messages during drag.

See also: tauri-local-viewer, electron-shm-viewer, webgl-shm-transport, prim-transform-safety, viewer-input-routing, camera-controls, object-selection.