Omniverse Realtime Viewer

Transform Manipulator

Use this when an Omniverse Realtime Viewer needs interactive translate, rotate, or scale controls for a selected USD prim. Also read ovstage-data-plane for runtime transform writes, prim-transform-safety for live local/world transforms and reset-stack inheritance. Use viewport-overlays when choosing server-side ovui composition. For application-parameter manipulators such as section planes, read section views.

For ovui-owned gizmo, widget, or headless overlay behavior beyond this skill, read references/dependencies for acquisition guidance and supplemental dependency documentation.

Implementation Contract

Implement these pieces in the generated app as needed:

• Local ovui frame gizmo: draw translate, rotate, and scale handles directly into the numpy RGBA frame before ByteImageProvider receives it.

• Viewport input router: convert letterboxed viewport coordinates into render coordinates and give the gizmo pointer priority before camera controls.

• App tool state: own active tool, keyboard shortcuts, toolbar state, selected path synchronization, animation pause/resume, and pending transform updates.

• Runtime bridge: write live omni:xform values through OVStage for immediate renderer feedback, then optionally commit final persistent edits through the selected session-layer path.

• Optional WebRTC headless overlay: draw the same manipulator grammar through ovui overlay primitives, then alpha-composite the RGBA overlay over the streamed BGRA frame.

• Message/input handler: route pointer and keyboard input to the gizmo first, then to camera or selection handlers only when the gizmo does not consume it.

• Shared projection helper: expose one world_to_screen() convention for headless overlays, hit testing, labels, and tooltips.

• Server-composited presentation only: blend overlay RGBA over the ovrtx BGRA stream buffer without copying the full video frame through browser-side 3D rendering.

Architecture

Choose a presentation path while sharing interaction state and projection math: target identity, active tool, projected handles, hit testing, and drag state. Prim targets additionally share OVStage transform writes and optional USD/session authoring; parameter targets share their owner-applied parameter model.

WebRTC path:
  ovrtx renderer.step() -> LdrColor CUDA frame
  -> headless ovui TransformGizmoOverlay draws SceneView primitives
  -> copy ovui RGBA output to CUDA
  -> CUDA alpha blend over BGRA stream buffer
  -> ovstream WebRTC video frame

Local ovui path:
  ovrtx renderer.step() -> CPU numpy RGBA frame
  -> TransformGizmo.draw(frame) draws anti-aliased lines/polygons
  -> ImageBridge.update(frame) -> ByteImageProvider

A WebRTC viewer may instead draw 2D SVG/canvas/DOM handles over the video. Project them from authoritative camera/target state and align them with the visible image rectangle. These are UI overlays: scene geometry, materials, and section caps continue to render through OVRTX. Use a server-composited overlay when handles must appear in the encoded frame. For Tauri SHM, also read tauri-shm-transform-gizmo for its platform-specific input lifecycle.

A parameter gizmo emits plane/light/tool intent to the owner; it need not select or mutate a USD prim. Reuse projection, drag snapshots, cancellation, and input priority without forcing parameter edits through the prim-transform protocol. See viewer-input-routing for browser overlay gesture ownership.

Input Priority

The gizmo must receive mouse input before the camera controller. If the camera sees the press first, orbit steals the drag and the manipulator feels broken.

def on_mouse_pressed(x, y, button, modifier):
    local = viewport_to_render_coords(x, y, clamp=False)
    if local and transform_gizmo.mouse_pressed(local[0], local[1], button):
        camera_dragging = False
        return

    mapped = camera_button_from_ovui(button)
    if mapped is not None:
        lx, ly = viewport_to_render_coords(x, y, clamp=True)
        camera.on_mouse_button_down(lx, ly, mapped)

During drag, keep routing all move and release events to the gizmo, even when the pointer leaves the rendered image. Clamp coordinates while dragging so release can finish cleanly.

def on_mouse_moved(x, y):
    dragging = bool(getattr(transform_gizmo, "dragging", False))
    local = viewport_to_render_coords(x, y, clamp=dragging)
    if local and transform_gizmo.mouse_moved(local[0], local[1]):
        return
    if camera_dragging:
        camera.on_mouse_move(*viewport_to_render_coords(x, y, clamp=True))

For server-composited WebRTC/headless overlays, the server input bridge follows the same rule: transform_gizmo.handle_input(event) runs before camera input dispatch, and returns True when the gizmo consumed the event.

In lightweight local ovui shells, do not assume that drawing a SceneView gizmo also wires low-level handle drag events into the application. If the selected handle/pivot can be hit but no transform callback fires, add an app-owned fallback: project the selected prim pivot into the visible image rect, treat an LMB press within a generous radius as transform intent, and route the whole mouse-down to the transform model. The release must not also enqueue a pick.

Hit Testing

Project the gizmo origin and each world-axis tip to screen space every frame:

AXES = {
    "x": np.array([1.0, 0.0, 0.0]),
    "y": np.array([0.0, 1.0, 0.0]),
    "z": np.array([0.0, 0.0, 1.0]),
}
HIT_RADIUS_PX = 35.0
AXIS_LENGTH_PX = 88.0
RING_RADII_PX = {"x": 76.0, "y": 64.0, "z": 52.0}

For translate and scale, choose the closest axis line segment from center to tip.

def distance_to_segment(point, start, end):
    segment = end - start
    denom = float(np.dot(segment, segment))
    if denom <= 1e-9:
        return float(np.linalg.norm(point - start))
    t = np.clip(np.dot(point - start, segment) / denom, 0.0, 1.0)
    return float(np.linalg.norm(point - (start + segment * t)))

hit = min(axis_cache, key=lambda item: distance_to_segment(point, origin, item.end))
axis = hit.name if hit.distance <= HIT_RADIUS_PX else None

For rotate, compare the click radius to the ring radii and choose the nearest ring.

distance = float(np.linalg.norm(point - screen_origin))
delta, axis = min(
    (abs(distance - radius), axis_name)
    for axis_name, radius in RING_RADII_PX.items()
)
return axis if delta <= HIT_RADIUS_PX else None

Choose hit targets in display pixels for the actual viewport scale and input latency; validate them after resizing. For a near-degenerate projected axis, use a valid alternate interaction constraint or disable it with an explanation. A fabricated screen direction alone does not define a world-space drag.

Projection And Drag Math

Project the axis direction, not only the axis endpoint. Try larger world-space scales until the projected direction has a usable length.

FALLBACK_AXIS_SCREEN = {
    "x": np.array([1.0, 0.0]),
    "y": np.array([0.0, -1.0]),
    "z": np.array([-0.72, 0.52]),
}

def project_axis_direction(origin, world_axis, axis_name):
    for scale in (1.0, 10.0, 100.0, 1000.0):
        projected = world_to_top_left_screen(origin + world_axis * scale)
        if projected is None:
            continue
        delta = projected - screen_origin
        length = float(np.linalg.norm(delta))
        if length > 0.5 and math.isfinite(length):
            return delta / length, length / scale

    fallback = FALLBACK_AXIS_SCREEN[axis_name]
    return fallback / np.linalg.norm(fallback), 1.0

Drag math is screen-space first. Store the starting transform and compute each update from that base; do not accumulate small increments frame-to-frame.

delta_px = float(np.dot(mouse - drag.start_mouse, drag.screen_direction))

if drag.tool == "translate":
    amount = delta_px / max(drag.pixels_per_world_unit, 1e-6)
    next_position = drag.start_position + AXES[drag.axis] * amount
    next_transform = drag.start_transform.copy()
    next_transform[3, :3] = next_position
elif drag.tool == "rotate":
    next_transform = rotate_transform(
        drag.start_transform,
        AXES[drag.axis],
        delta_px * 0.01,  # radians
    )
elif drag.tool == "scale":
    factor = max(0.05, 1.0 + delta_px / 140.0)
    next_transform = scale_transform(drag.start_transform, drag.axis, factor)

The viewer matrix convention is row-major with translation in row 3. Keep world_to_screen() coordinate conventions consistent: if the projection helper returns bottom-left origin, convert once to top-left origin before hit testing.

Runtime Transform Handoff And USD Commit

The manipulator contributes local matrices to the runtime owner's existing tick. Snapshot the selected prim's live local and world transforms at drag start, then compute each drag sample from that base. Convert world intent through the current reset-aware effective parent from prim transform safety.

import numpy as np

def resolve_drag_transform(runtime, path: str, world_transform: np.ndarray) -> np.ndarray:
    # App-owned adapter: identity for a reset-stack target, otherwise the
    # current parent world matrix composed from runtime ancestor locals.
    parent_world = runtime.read_effective_parent_transform(path)
    local = np.asarray(world_transform, dtype=np.float64) @ np.linalg.inv(parent_world)
    return local

Pointer callbacks enqueue world intent with the stage generation. The runtime owner calls this resolver using its current transform snapshot. Reject a singular parent and report the edit failure. Adapter methods belong to the application; a pointer callback must not read stage data, allocate an ordinal, or render. The resolved locals join camera, animation, and other interaction updates in the owner's batch. The owner writes the final local matrices with the write_local_transforms helper, using AttributeSemantic.MATRIX, PrimMode.UPSERT, and one 16-lane float64 element per queried prim. It publishes once after the tick's writes complete, then renders that ordinal. Do not use standalone OVRTX attribute APIs to mutate an attached OVStage.

Session-layer xformOp authoring is useful for persistent/editor-style edits, but it should not be the only path for a realtime viewer drag. If both are required, write live omni:xform through OVStage during the drag and commit the final edit through the chosen session-layer/undo path on release.

Author optional persistent manipulator edits into the session layer so user USD files remain non-destructive. Commit the accepted local matrix from the runtime transform model; do not recompute it from a stale authored parent in a USD worker. Preserve the target's reset-stack state when authoring the corresponding ops.

from pxr import Gf, Usd, UsdGeom

def apply_local_transform_to_prim(stage, path, local_transform):
    prim = stage.GetPrimAtPath(path)
    if not prim or not prim.IsValid() or not prim.IsA(UsdGeom.Xformable):
        return False

    local = gf_matrix_from_numpy(local_transform)
    translate, rotate_xyz, scale = decompose_common_matrix(local)
    with Usd.EditContext(stage, stage.GetSessionLayer()):
        return write_xform_ops(UsdGeom.Xformable(prim), translate, rotate_xyz, scale)

Write xform ops directly. Do not use UsdGeom.XformCommonAPI.SetRotate() for mixed assets because existing prims may have Gf.Vec3d rotate/scale attributes while SetRotate() expects Gf.Vec3f.

def set_vec3_op(op, values):
    attr = op.GetAttr()
    type_name = attr.GetTypeName().cppTypeName if attr else ""
    if "Vec3d" in type_name or "double" in type_name:
        op.Set(Gf.Vec3d(*values), Usd.TimeCode.Default())
    else:
        op.Set(Gf.Vec3f(*[float(v) for v in values]), Usd.TimeCode.Default())

def write_xform_ops(xformable, translate, rotate_xyz, scale):
    ops = {op.GetOpName(): op for op in xformable.GetOrderedXformOps()}
    t_op = ops.get("xformOp:translate") or xformable.AddTranslateOp()
    r_op = ops.get("xformOp:rotateXYZ") or xformable.AddRotateXYZOp()
    s_op = ops.get("xformOp:scale") or xformable.AddScaleOp()

    t_op.Set(Gf.Vec3d(*translate), Usd.TimeCode.Default())
    set_vec3_op(r_op, rotate_xyz)
    set_vec3_op(s_op, scale)
    return True

Use the same transform model to refresh the inspector and gizmo after each accepted edit. Discard queued edits and cached transforms on a stage-generation change. See prim-transform-safety for baseline, ancestry, and restoration rules.

Drawing

Use the same visual grammar in both render paths:

• X axis: red (1.0, 0.18, 0.14).

• Y axis: green (0.22, 0.92, 0.32).

• Z axis: blue (0.25, 0.54, 1.0).

• Active axis: lighter/brighter variant while dragging.

• Translate: axis lines with arrowheads.

• Rotate: concentric rings with different radii per axis.

• Scale: axis lines with square handles at endpoints.

• Center: small white square at the projected origin.

Local frame drawing should alpha-blend anti-aliased line and polygon masks into the numpy frame. Headless ovui drawing should use omni.ui_scene.scene.Line and PolygonMesh under the shared overlay SceneView. Implement the CUDA compositor in the app to blend the exported RGBA over the BGRA stream; no compositor module is supplied by this package.

Animation Interaction

Selection animation and manipulator drags share the owner's transform model. The gizmo's transform callback queues world intent for the resolver above. After the owner has composed animation and edits, project the current runtime world transform so handles stay attached to the rendered object, including when an ancestor moves:

world = runtime.read_world_transform(path)
gizmo.set_selection(path, world_transform=world)

On drag start, freeze the animation first, then use the current visible runtime transform as start_transform. On drag end, update the animator base transform from the final OVStage pose before resuming so it does not snap back to the pre-drag position.

Tool Selection

Use conventional editor shortcuts and toolbar buttons:

Q = none/select
W = translate
E = rotate
R = scale

The active tool gates visibility and behavior. No active tool means no manipulator is rendered and no manipulator hit testing should consume camera input.

def set_active_tool(tool):
    normalized = None if tool in (None, "", "none", "select") else tool
    gizmo.set_active_tool(normalized)
    toolbar.set_checked(normalized)

For local ovui, wire keyboard shortcuts on the viewport hit rectangle and toolbar buttons in the header. For WebRTC, send a JSON app message such as {"event_type": "setActiveTool", "payload": {"tool": "translate"}} and update the accepted tool state in the chosen overlay presentation.

Common Pitfalls

• XformCommonAPI.SetRotate() only accepts GfVec3f; directly write existing xform ops and preserve their attribute type.

• Read viewer-input-routing for ovstream mouse button normalization before combining gizmo, camera, and pick handlers.

• Render-pixel hit radii must scale with the displayed image; test the actual display-space target size.

• Hover consumption is intentional: contains() == True can block camera input so the cursor can communicate that the gizmo is interactive. Only start transforms on press hits.

• Convert viewport coordinates through the same letterbox mapping used for picking and camera input.

• Do not call renderer.step() concurrently with scene reset, OVStage transform publication, or stage mutation.

• Keep one shared ovui headless overlay window; multiple windows can break frame export.

• If an axis projects to near-zero length, use an alternate valid constraint or disable that handle; preserve finite world-space intent.

• If the gizmo is visible but the prim stays stationary, verify input ownership first and then verify that the drag callback reaches the OVStage omni:xform publication path for the selected prim.

Checklist

• [ ] Read viewport-overlays for WebRTC/headless ovui composition or local-viewer for inline ByteImageProvider drawing.

• [ ] Read ovstage-data-plane before writing live omni:xform or combining gizmo drags with selection animation.

• [ ] Route gizmo input before camera input for press, move, release, and wheel.

• [ ] Project origin and axes every frame from the current camera view/projection.

• [ ] Use start_transform and start_mouse as the base for each drag update.

• [ ] During drag, publish the selected prim's live omni:xform through OVStage; on release, optionally commit to a session-layer USD edit path.

• [ ] Author USD xform ops in the session layer with parent compensation and Vec3f/Vec3d type preservation.

• [ ] Validate with a measured transform delta and OVStage write/render consume ordinal, not only a screenshot of a visible manipulator.

• [ ] Freeze animation during drag and resume from the new base transform after drag end.

See also: viewport-overlays, ovstage-data-plane, ovstage-ovrtx-integration, viewer-input-routing, camera-controls, object-selection, selection-animation, local-viewer, streaming-messages.