NVIDIA IsaacTeleop Converts XR Tracking into Robot Actions
A tutorial examines the retargeting engine at the center of NVIDIA IsaacTeleop, a framework that converts XR hand tracking and motion-controller input into commands for simulated and physical robots. The walkthrough does not require a headset, OpenXR runtime or simulator: all inputs are created with NumPy and processed on a Colab CPU, allowing each calculation to be displayed.
The engine uses typed TensorGroup structures to organize tracker data. HandInput represents 26 OpenXR hand joints, while ControllerInput provides slots for poses, validity flags, buttons and analog controls. The system checks data types and reports missing values instead of silently using stale information; optional groups indicate when tracking is unavailable.
The tutorial also implements a custom pinch detector with adjustable threshold and joint-selection parameters. It then uses built-in gripper and SE(3) retargeters, combines processing nodes into a graph that produces an action vector at every step, and applies a world-frame transformation. The workflow covers running, pausing and stopping the state machine, as well as mapping controller input to a dexterous hand. Parameter changes can be retained across restarts.
Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.