Robotic Spatiotemporal Editing: A Unified Interface for Controllable Action Generation via Masked Action Reconstruction

01

Abstract

Generative policies can synthesize robot behaviors, yet controlling and editing them through diverse spatiotemporal references remains challenging. Real-world manipulation requires robots to satisfy user-specified constraints on where, when, and how actions are executed. Our key insight is that spatiotemporal control references, such as keypoints, flows, orientations, and keyposes, can be viewed as partial specifications of the same temporal Cartesian action space. We introduce the MAP (Masked Action Space) interface to unify these action references and formulate controllable manipulation as an action inpainting problem. Our Inpainting Policy (IP) reconstructs missing action dimensions of the end-effector, combining progress-guided alignment with multi-mask training to support unseen action constraint combinations. We further adopt a training-free RePaint variant that iteratively inpaints the action during denoising to reduce control error. On ManiSkill and LIBERO-10, our policy achieves average task success rates of 92.67% and 64.67%, respectively. Real-world experiments demonstrate spatiotemporal control with average positional and rotational errors of 43.3 mm and 7.8°, respectively.

02

Method

RoSE system overview showing the world model, multimodal masked action representation, robot pose generation, and action inpainting pipeline — **Figure 1: Unified spatiotemporal control via action inpainting — Top.** The **Masked Action Space (MAP)** interface unifies heterogeneous references from upstream models and human inputs as partial specifications of temporal Cartesian actions. We use progress values instead of raw **timesteps**.

RoSE action inpainting examples showing multiple control modalities and the policy output — **Figure 1 — Bottom.** Our **Inpainting Policy (IP)** completes these partial specifications through action inpainting, supporting potentially unlimited spatiotemporal reference types and unseen combinations.

PROGRESS-GUIDED SLIDING WINDOW(1) A progress-guided sliding window aligns MAP with the current task progress and provides a temporally anchored MAP condition.

MULTI-MODAL JOINT TRAINING(2) Multi-modal joint training over heterogeneous mask families enables structured control and zero-shot generalization to unseen spatiotemporal compositions.

REPAINT TRAINING-FREE INFERENCE STRATEGY(3) We adopt RePaint (Lugmayr et al., 2022) as a training-free inference strategy, which repeatedly reinserts the specified action tokens during the denoising process of the diffusion-based policy, thereby improving adherence to the provided constraints.

Progress-guided MAP window and action-inpainting base model — **Figure 2(a): Progress-guided action inpainting.** Camera observations and proprioceptive information estimate the current task progress, anchor a local MAP window, and condition the base model to reconstruct the missing action tokens.

Multi-modal joint training over heterogeneous masked action references — **Figure 2(b): Multi-modal joint training.** Joint training over heterogeneous mask families enables structured control and supports generalization to unseen spatiotemporal reference compositions.

RePaint denoising strategy for action inpainting — **Figure 2(c): RePaint inference.** During denoising, the specified action tokens are repeatedly reinserted so the completed action chunk adheres more closely to the provided constraints.

03

Experiments

UNIFIED CONTROL VIA JOINT TRAINING

Method	Multi-task (LIBERO-10)		Single-task (ManiSkill)
Method	Avg. SR (%) ↑	CE(all) ↓	Lift Peg SR (%) ↑	Pick Cube SR (%) ↑	Poke Cube SR (%) ↑	Avg. SR (%) ↑	CE(all) ↓
DP	60.00	—	86.33	69.33	69.67	75.11	—
DiT Policy	59.67	—	94.67	87.33	93.00	91.67	—
STC	50.67	0.060	72.67	65.67	91.67	76.67	0.041
IP	64.67	0.045	100.00	90.33	87.67	92.67	0.023
IP-RePaint	51.33	0.026	95.80	82.13	62.20	80.04	0.014

Table 1: Unified control via joint training. Each policy checkpoint is evaluated across reference types using the average top-three picks of success rate and control error (CE). ManiSkill uses a separate policy for each task, whereas LIBERO-10 uses a shared multitask policy.

Joint-Training Performance. Table 1 shows that STC performs worse than its corresponding base policies, indicating that directly conditioning action generation on control signals can interfere with task SR. In contrast, Inpainting Policy improves task completion while achieving lower control error, performing well in both multi-task and single-task settings and achieving the highest average success rates of 64.67% and 92.67%, respectively. The training-free RePaint variant further reduces CE with relative reductions of 42.2% and 39.1%, respectively, demonstrating a trade-off between control accuracy and task completion.

Bar chart comparing success rates by reference type and line chart showing control error over training

Figure 5: Mask-type breakdown in joint training. Left: We compare success rates across reference types, including the unseen mixed mask Mix0, to evaluate controllability and compositional generalization. Right: Control error decreases as training progresses.

Mask-Type Breakdown and Training Dynamics. Figure 5 shows that Inpainting Policy provides the strongest average control performance across all control types, especially on the unseen Mix0 mask (80.56%). STC performs strongly on dense 2D/3D flow (80.00%), but degrades sharply under sparse constraints. The right plot shows that Inpainting Policy progressively improves its control capability during training, with CE decreasing from approximately 0.049 to 0.022. In contrast, RePaint maintains the lowest CE, around 0.014–0.015, across all checkpoints. Notably, since the Mix0 composition is absent from the training set, its high success rate indicates that the policy can compose learned mask patterns without additional finetuning.

STUDY ON ACTION INPAINTING

Plots studying mask ratio and performance under the same number of spatiotemporal tokens

Research notesHover to read ↓

Figure 4: Effect of spatiotemporal information quantity and structure. (a) We vary the visible ratio to examine how the number of references affects task success. (b) We compare mask types with approximately equal numbers of spatiotemporal tokens, alongside a base policy (a DiT policy), to assess how constraint structure influences the performance of Inpainting Policy. Here, r denotes the visible ratio, i.e., the proportion of action information that is not masked in MAP.

Mask Ratio and Mask-Type Analysis. Figure 4(a) shows that increasing the visible-action ratio generally improves task success, indicating that additional known action components provide stronger guidance for reconstructing executable behaviors. For the structured 3D-point mask, success increases from 57.0% at a ratio of 0.02 to 91.7% at 0.50. Although the random mask also improves with more visible actions, its performance is less stable and reaches only 71.7% at the highest ratio. This gap suggests that structured control signals convey more task-relevant information than randomly selected action components. Figure 4(b) compares different mask types under approximately the same number of visible spatiotemporal tokens. Across all mask types, IP outperforms the unconditioned DiT policy. Positional and full-pose constraints provide the strongest guidance, whereas orientation-only constraints are less informative for task completion. The random-mask result further shows that semantically structured action references are more effective than an equivalent number of arbitrary visible tokens.

CONTROL WITH REAL-WORLD REFERENCES

Panel A: cup arrangement and liquid-pouring rollouts with progress-aligned control references

Research notesHover to read ↓

Figure 6: Real-world spatiotemporal editing. Here, p_r denotes the progress associated with a user-provided reference, whereas p denotes the robot’s current execution progress. The rollouts and corresponding MAP masks illustrate how these references guide execution.

Result. As shown in Figure 6, with three or more reference points, the policy achieves success rates above 93%, while the cup-to-reference-point distance ranges from 18.2 to 63.7 mm, with an average of 43.3 mm. In the liquid-pouring task, the tube-tilt error remains below 8°, with a mean of 7.8°. These results show that providing explicit control constraints substantially improves task completion and reference adherence: denser references produce smoother motion and smaller deviations from the target. In contrast, when only a single reference is provided at the final progress point, the task completion rate falls below 30%, revealing a limitation of sparse late-stage supervision and an important direction for improving the subsequent editing system. Supplementary Section 2.6 provides additional recorded SR and normalized CE results (Tables 3–5), plus deployment tips.

04

Demo Videos

Cup arrangement experiment with control references and success-rate comparison between the base policy and RoSE

Liquid-pouring experiment with angular control references and success-rate comparison between the base policy and RoSE

01 / 05

1Cup placement · left

Grasp the cup from its right side and place it at the left side of the plate.

02 / 05

2Cup placement · right

Grasp the cup from its right side and place it at the right side of the plate.

03 / 05

3Cup placement · middle

Grasp the cup from its right side and place it at the middle of the plate.

04 / 05

4Cup placement · front

Grasp the cup from its right side and place it at the front of the plate.

05 / 05

5Cup placement · gripper left

Grasp the cup from its left side and place it at the middle of the plate.

01 / 02

Liquid pouring · θ₁

02 / 02

Liquid pouring · θ₃

05

Citation

@article{rose2026,
  title   = {Robotic Spatiotemporal Editing: A Unified Interface for Controllable Action Generation via Masked Action Reconstruction},
  author  = {Li, Yanbang and He, Bu and Tian, Yi and Huang, Yu-Chiao and Zhang, Chenyu and Chen, Ruoqu and Liu, Yuanzhe and Gao, Yuxiang and Xu, Mengdi},
  journal = {arXiv preprint},
  year    = {2026}
}

RoSE