DexNDM

CLOSING THE REALITY GAP FOR
Dexterous IN-HAND ROTATION VIA Joint-wise NEURAL DYNAMICS Model

¹Tsinghua University ²Peking University ³Shanghai Qi Zhi Institute ⁴Galbot

DexNDM presents a sim-to-real approach that enables unprecedented in-hand rotation in the real world. We master a wide object distribution, including challenging geometries and complex shapes, across rich wrist orientations.

In-Hand Rotation Skills (Highlight)

Cuboid, 16cm x 3cm x 3cm (2x)

Cuboid, 20cm x 3cm x 3cm (2x)

SharpaWave, Cuboid, 15cm x 2cm x 2cm (3min)

Cuboid, 20cm x 4cm x 4cm (2x)

Cuboid, 16cm x 3cm x 3cm (2x)

Book, 16cm x 3 cm x 23cm (2x)

Complex Geometry (2x)

Small Objects (2x)

Skill Galleries

-z Axis. Reorienting diverse objects in the air along the principle axis (0, 0, -1).

-z Axis. Reorienting challenging small objects in the air along the principle axis (0, 0, -1).

z Axis. Reorienting diverse objects in the air along the principle axis (0, 0, 1).

-y Axis. Reorienting diverse objects in the air along the principle axis (0, -1, 0).

x Axis. Reorienting diverse objects in the air along the principle axis (1, 0, 0).

y Axis. Reorienting diverse objects in the air along the principle axis (0, 1, 0).

-x Axis. Reorienting diverse objects in the air along the principle axis (-1, 0, 0).

Various Rotation Axes. Reorienting objects in the air across diverse rotation axes, beyond principle x, -x, y, -y, z, -z.

Diverse Wrist Orientations. Reorienting objects in the air under rich wrist orientations, including palm up, palm down, thumb up, thumb down, base up, base down, and more general orientations. We also demonstrate the gravity-invariant rotation under changing gravity axes.

Rich wrist orientations along with diverse rotation axes. Reorienting objects in the air across diverse rotation axes under various wrist orientations.

Comparisons to AnyRotate. We show superior performance in rotating challenging objects used in AnyRotate. Objects are either 3D printed or bought in the store.

Comparisons to Visual Dexterity. We show on-par or superior performance in rotating challenging objects used in Visual Dexterity. Objects are either 3D printed.

Comparisons to UAN and ASAP. They cannot work in our task, failing in rotating the most basic cylinder shape.

Application: Teleoperating Complex Dexterous Manipulation Tasks

Assembly Tasks. Teleoperating the robot to complete assembly tasks: lightbulb (4x), four-legged table (4x), plug (2x), four-legged short table (4x), round table (style 1, 4x), round table (style 2, 4x), nut (2x), nut v2 (2x), lid (4x), lightbulb horizontal (2x), round table horizontal (2x).

Orientation-Sensative Placement Tasks. Teleoperating the robot to grasp-reorient-place objects: small cup (2x), corn (2x), broccoli (2x), bread (2x), rice dumpling (2x), sweet potato (2x), steamed bun (2x), pen (2x), thin pen (2x).