WALA | Learning Executable Latent Actions

Abstract

Generalizable robot policies typically rely on robot demonstrations with action annotations, yet such data are expensive to collect and difficult to scale. In contrast, large-scale and readily available human videos record rich physical interactions, but lack action annotations that can be directly used for robot control. We present WALA, a framework that jointly learns executable latent actions from action-labeled demonstrations and action-free videos.

WALA first pretrains a semantic-geometric latent action model on videos without action annotations, enabling it to learn action-relevant representations from scene evolution between the current observation and multiple sparsely sampled future observations. Specifically, WALA forms semantic and geometric future deltas, from which the encoder extracts latent action targets, while the decoder predicts future deltas in the DINOv3 feature space and dense depth space. This avoids raw pixel reconstruction, reducing the influence of appearance details while preserving task-relevant semantic and geometric structure.

During policy training, the pretrained encoder remains frozen to provide stable latent action targets, while the decoder serves as a trainable latent world model. The latent actions generated by the vision-language backbone are jointly supervised by robot action prediction, latent action target matching, and future dynamics prediction. Action-labeled demonstrations provide both executable control and dynamics supervision, whereas action-free videos require no robot action labels and still participate in training through latent action and future dynamics supervision. In this way, WALA connects physical scene evolution in videos with executable robot control.

Experiments show that WALA achieves strong performance on RoboTwin and reaches an average success rate of 75.2% on RoboCasa, setting a new state-of-the-art result. Additional real-robot experiments further evaluate its generalization ability across diverse manipulation tasks.

Method

The framework consists of two stages. In the first stage, we pretrain a semantic-geometric latent action model on videos without action annotations. The model uses the current observation and future semantic-geometric deltas to infer latent action targets, then predicts future DINOv3 feature deltas and dense depth changes. In the second stage, we integrate the pretrained model into policy learning. The frozen encoder provides stable latent action targets, while the decoder is used as a trainable latent world model.

Semantic-geometric latent action model pretraining diagram. — **Semantic-geometric latent action model pretraining.** Given the current observation and multiple sparsely sampled future observations, WALA extracts RGB features with a frozen DINOv3 encoder and obtains dense depth maps from depth observations or a frozen depth estimator. The encoder infers latent action targets from observed semantic and geometric deltas, while the decoder predicts future DINOv3 feature deltas and dense depth changes from the current state and latent actions.

Policy training diagram with latent world model supervision. — **Policy training.** The pretrained latent action encoder is frozen and provides stable latent action targets from observed future deltas. The decoder is integrated as a trainable latent world model. The vision-language backbone generates unified latent actions from multi-view observations, language instructions, robot states, and action queries. These latent actions are supervised by robot action prediction, latent action target matching, and future dynamics prediction, tying them to future scene evolution and executable control.

Experiments

RoboTwin 2.0

Main results on RoboTwin

Group	Method	Clean	Random
VLA-based	π_0.5	82.7	76.8
	X-VLA	70.0	69.0
	StarVLA-α	88.2	88.3
	InternVLA-A1	89.4	89.6
WAM-based	Motus	88.7	87.0
	Fast-WAM	91.9	91.8
	LingBot-VA	92.9	91.5
Ours	WALA	90.6	92.8

Baseline results are collected from publicly released reports.

RoboCasa-GR1-Tabletop

Main results on RoboCasa

Group	Method	Avg.
VLA-based	StarVLA Qwen3-PI	43.9
	GR00T-N1.6	47.6
	StarVLA Qwen3-GR00T	47.8
	StarVLA Qwen3-OFT	48.8
	StarVLA-α	57.3
	ABoT-M0	58.3
	RLDX-1	58.7
	FrameSkip	59.5
WAM-based	DiT4DiT	50.8
	LDA-1B	55.4
	DIAL	70.2
Ours	WALA	75.2

Baseline results are collected from publicly released reports.

Ablation

Ablation study on RoboCasa-GR1-Tabletop

	Supervision
Variant	LAM Pretrain	LAM Target	Semantic Pred.	Geometric Pred.	Avg.	Gain
Base Policy	--	--	--	--	54.2	--
+ World w/o LAM Pretrain	--	--	✓	✓	67.8	+13.6
+ Semantic World	✓	--	✓	--	68.8	+14.6
+ Semantic-Geometric World	✓	--	✓	✓	71.0	+16.8
+ LAM Target	✓	✓	--	--	67.6	+13.4
Full WALA	✓	✓	✓	✓	75.2	+21.0

Data Scaling

Data scaling on RoboCasa

Line chart comparing Base Policy, WALA with the same action-labeled data, and WALA with action-free videos on RoboCasa across total data budgets.

Line chart comparing Base Policy and WALA with the same amount of action-labeled RoboCasa demonstrations.

Line chart showing WALA performance as action-free RoboCasa videos are added on top of 10 percent action-labeled data, with Base Policy 10 percent and 100 percent labeled reference lines.

Real Robot

Real-robot experiments

Group	Method	Robot demos / task	Human videos / task	Basic Pick-Place	Stack Paper Cups	Insert Flowers	Disassemble Blocks	Avg.	Latency
Baselines	π₀	200	0	25/30	15/30	12/30	10/30	51.7	170 ms
Baselines	π_0.5	200	0	26/30	15/30	16/30	13/30	58.3	190 ms
Ours	WALA	200	0	28/30	21/30	21/30	20/30	75.0	70 ms
	WALA	50	400	26/30	21/30	17/30	17/30	67.5	70 ms
	WALA	200	400	30/30	25/30	24/30	21/30	83.3	70 ms

Each policy is trained as a single multi-task model and evaluated over 30 trials per task. Adding action-free similar-scene human videos improves WALA from 75.0% to 83.3%. With only 50 robot demonstrations per task and 400 human videos, WALA still outperforms both baselines trained with 200 robot demonstrations per task. Latency is measured on an RTX 4090 GPU.

Real-World Robot Demo Videos