REFERENCE / REAL ROBOTREF3D Keypoint

Cup Arrangement

Left of plate

TASK PROGRESSp = 0.00
Reference video

3D rollout

Mouse: left drag pan · right drag rotate · wheel zoom

ReadyDrag · Pan   Two fingers · Rotate   Pinch · Zoom
Reference video
Reference videoChoose a reference, then watch its motion.
Preparing the reference library…
METHOD / MAP + IP

From a sparse reference to motion

  1. Specify what matters

    A reference constrains selected entries of the seven-dimensional action: x, y, z, θx, θy, θz and gripper G. Solid blue cells mark supplied reference constraints. Flow constraints form continuous bands. Inpainting fills completed entries in blue; unknown entries remain gray.

  2. Align by task progress

    The method estimates progress from the current observation and robot state, then selects a local reference window. This demonstration uses illustrative progress that follows video time with gentle speed variation; it is not a learned progress estimate.

  3. Complete the action

    The Inpainting Policy conditions on that window, visual observations and robot state to reconstruct the missing action entries.

2D Keypoint / FlowTwo selected axes: XY or YZ3D Keypoint / Flowx · y · z in spaceKeypose / 7D Posex · y · z · θx · θy · θz · GOrientationθx · θy · θzMixMultiple reference types together

This page uses recorded replays, not live inference. Progress alignment is illustrative. Keypoints are sparse; flows are dense. Simulation references use recorded TCP poses from pickup onward. 2D references use their selected plane: YZ for Flow and XY for sparse points, with a cyan TCP projection. Simulation markers remain visible after playback. Full poses include rotation and gripper opening. Real-robot videos use the supplied effects; their 3D replay is a reconstruction. Curated LIBERO examples have manual image-space reference illustrations and synchronized video details. Their robot states and calibrated 3D references are not available; no 7D values are inferred from those videos.

Based on the local paper Robotic Spatiotemporal Editing, §§3.1–3.3 and Fig. 6. Project website repository ↗