MoAct: Learning Visual Transition Dynamics
for Robot Action Generation

Anonymous Authors
Under Review
Code Models
Overview of MoAct
Figure 1. Overview of MoAct. MoAct is a visual-transition-aware vision-language-action model that learns where and how scenes evolve before generating actions. It extracts compact transition tokens from large-scale robot and egocentric human videos and couples them with action generation through a dual-stream Mixture-of-Transformers.

Abstract

Modeling future observations allows vision-language-action (VLA) models to anticipate scene changes and improve action generation. However, extracting latent representations that retain the information needed for accurate prediction and control remains a key challenge. To address this challenge, we introduce MoAct, a framework that uses compact visual-transition tokens to model changes in future visual context and guide action generation. MoAct learns visual-transition dynamics from large-scale video data and connects a visual-transition dynamics expert with an action expert through a dual-stream Mixture-of-Transformers (MoT) architecture. This design allows predicted scene changes to guide policy learning. Extensive experiments in simulation and the real world show that MoAct outperforms existing future-guided methods. Egocentric human videos also help MoAct retain much of its performance with fewer robot demonstrations. For scene adaptation, MoAct requires only additional egocentric human videos from the target scenes, with no new robot demonstrations.

Method

MoAct represents scene evolution as transitions between adjacent observations, rather than as independently predicted future states.

MoAct architecture
Figure 2. Model architecture. MoAct models where and how scenes change to guide action generation. The visual-transition dynamics expert predicts compact transition representations from VLM backbone features. A dual-stream MoT passes these representations to the action expert, which generates actions through flow matching.

Architecture. A Qwen3.5-0.8B backbone encodes the current observation and the language instruction. The visual-transition dynamics expert uses learnable queries to produce eight transition tokens, each trained to encode the change from a preceding DINOv3 feature grid to its successor. Through the dual-stream Mixture-of-Transformers, these tokens enter the action expert as layer-wise keys and values, and the action expert generates action chunks by flow matching. The DINOv3 encoder and the feature predictor are used only during training.

Training. In Stage 1, MoAct learns general visual-transition dynamics from a large-scale mixture of robot and egocentric human manipulation videos. In Stage 2, these dynamics are adapted to the target scenes while a newly initialized action expert is trained on robot demonstrations.

Visual-transition dynamics modeling
Figure 3. Visual-transition dynamics modeling in SimplerEnv. Given adjacent visual contexts, MoAct compactly models their visual transition with the visual-transition expert. The current visual feature is transformed into a successor feature through the corresponding visual-transition token.

Simulation Results

Success rates (%) in SimplerEnv. Bold denotes the best result in each column. Baseline results are taken from WoG, with averages recomputed from the listed per-task values.

Table 1. Google Robot evaluation in SimplerEnv. Success rates (%) under Visual Matching and Variant Aggregation.

Visual MatchingVariant AggregationOverall
ModelSizePick Coke CanMove NearDrawerAvg.Pick Coke CanMove NearDrawerAvg.Avg.
π03B72.765.338.358.875.263.725.654.856.8
π0-FAST3B75.367.542.961.977.668.231.359.060.5
OpenVLA7B16.346.235.632.754.547.717.740.036.3
GR00T N12B47.070.018.145.078.862.513.251.548.3
Moto0.5B74.060.443.159.2–––––
VITA2B57.555.858.957.4–––––
DeFI2B54.260.738.651.253.958.224.045.448.3
WoG7B89.082.562.578.087.975.019.360.769.4
MoAct (Ours)1B96.077.159.777.687.765.044.765.871.7

Table 2. WidowX evaluation in SimplerEnv. Grasp rate and task success rate (%) across the four tasks.

Spoon on TowelStack BlocksCarrot on PlateEggplant in BasketOverall
ModelSizeGraspSuccessGraspSuccessGraspSuccessGraspSuccessGraspSuccess
π03B45.829.125.00.050.016.791.662.553.127.1
π0-FAST3B62.529.158.521.954.010.883.366.664.632.1
OpenVLA7B4.10.033.00.012.50.08.34.114.51.0
GR00T N12B83.362.554.245.870.816.741.720.862.536.5
UniVLA7B76.452.866.72.879.255.687.566.777.544.5
ViPRA7B79.266.762.554.254.250.091.779.271.962.5
WoG7B95.879.275.033.070.850.0100.091.785.463.5
MoAct (Ours)1B87.583.379.245.875.070.895.862.584.465.6

The released checkpoints, evaluation code, and episode-level results are available in the anonymous repository.

Real-World Experiments

We evaluate MoAct on an AgileX PiPER X tabletop dual-arm platform. Three Intel RealSense D435 cameras provide one external view and two wrist-mounted views. The evaluation suite comprises package storage (PK), sausage placement (SP), bowl stacking (BS), and towel folding (TF).

Real-world setup and tasks
Figure 4. Real-world setup and tasks. The robotic platform and sensors (left) and the execution of the four tasks (right). Frames in each row follow temporal order from left to right.

Task Execution

MoAct fine-tuned with 100 robot demonstrations per task.

Store packages in the box
Place the sausage on the plate
Stack the bowls on the plate
Fold the towel

Quantitative Results

Real-world results
Figure 5. Real-world task performance. Left: π0.5 and MoAct on the four in-distribution tasks. Right: MoAct before and after adaptation to out-of-distribution package-storage scenes using additional egocentric human videos from those scenes. Both settings use the same 100 robot demonstrations. E denotes expert demonstrations and HV denotes egocentric human videos.

Scene Adaptation

Starting from MoAct trained on 100 robot demonstrations, we add 50 egocentric human videos collected in out-of-distribution package-storage scenes, without collecting additional robot demonstrations.

Store packages in the box
Towels as packages, cardboard box
Store packages in the box
Plastic packages, basket as container
Store packages in the box
Towels as packages, basket as container

Egocentric Human Videos

In-distribution human videos
Figure 6. In-distribution egocentric human videos collected for the four tasks.
Out-of-distribution human videos
Figure 7. Egocentric human videos collected in the out-of-distribution package-storage scenes.

Simulation Rollouts

Episodes generated by the released checkpoints under the SimplerEnv evaluation protocol. Only successful episodes are shown.

Pick coke can
can laid horizontally, original robot appearance
Pick coke can
can laid vertically, original robot appearance
Pick coke can
can standing upright, original robot appearance
Pick coke can
can laid horizontally, original robot appearance
Move 7up can near blue plastic bottle
original robot appearance
Move sponge near apple
robot appearance variant
Move blue plastic bottle near pepsi can
robot recolor 1
Move redbull can near coke can
robot recolor 2
Open bottom drawer
bottom drawer, robot recolor 1
Open middle drawer
middle drawer, robot recolor 1
Open middle drawer
middle drawer, robot recolor 2
Open top drawer
top drawer, robot recolor 1
Close bottom drawer
bottom drawer, robot recolor 1
Close bottom drawer
bottom drawer, robot recolor 2
Close middle drawer
middle drawer, robot recolor 1
Close middle drawer
middle drawer, robot recolor 2
Pick coke can
can laid horizontally, base scene
Pick coke can
can laid horizontally, cabinet scene 1
Pick coke can
can laid horizontally, cabinet scene 2
Pick coke can
can laid vertically, alternative background
Move orange near pepsi can
alternative background
Move blue plastic bottle near pepsi can
alternative background 2
Move sponge near 7up can
base scene
Move 7up can near sponge
cabinet scene 1
Open middle drawer
middle drawer, apartment
Open middle drawer
middle drawer, office
Open top drawer
top drawer, apartment
Open top drawer
top drawer, office
Close bottom drawer
bottom drawer, apartment
Close bottom drawer
bottom drawer, bedroom
Close bottom drawer
bottom drawer, office
Close middle drawer
middle drawer, apartment
Put the spoon on the towel
Put the spoon on the towel
Put the spoon on the towel
Put the spoon on the towel
Put carrot on plate
Put carrot on plate
Put carrot on plate
Put carrot on plate
Stack the green block on the yellow block
Stack the green block on the yellow block
Stack the green block on the yellow block
Stack the green block on the yellow block
Put eggplant into yellow basket
Put eggplant into yellow basket
Put eggplant into yellow basket
Put eggplant into yellow basket

Policy-camera views, motion-interpolated to 30 fps and upsampled 2× for display. Google Robot videos are played at 3× and WidowX videos at 2× speed; each video ends 1.5 s after task success.

BibTeX

@misc{anonymous2026moact,
  title  = {MoAct: Learning Visual Transition Dynamics for Robot Action Generation},
  author = {Anonymous},
  year   = {2026},
  note   = {Under review}
}