Lift, See, Act: Hierarchical Robot Policy Pretraining with 3D Foundation Models

Why LSA?

Learn more than a sparse trajectory

Human videos are abundant, but they do not provide robot actions or explicit 3D structure.
Existing pipelines keep sparse hand trajectories and discard rich intermediate knowledge from 3D foundation models.
LSA transfers dense geometry, interaction focus, and 3D motion into one reusable visual policy encoder.

LSA retains dense 3D representations, structural cues, attention guidance, and motion supervision.

Main pipeline

Hierarchical Pretraining

The data engine lifts diverse human videos into reusable 3D supervision.
Hierarchical pretraining injects knowledge at the layer where it is most useful.
The frozen encoder then supports lightweight closed-loop robot control.

Full LSA architecture: data engine, hierarchical pretraining, and closed-loop control.

Novel contribution highlighted by reviewers

See phase

Attentive guidance: discover emergent hand-object attention

Emergence. Predicting future hand motion encourages attention to emerge around the hand and the object it interacts with.
Mining. A hand-region cue ranks the encoder's internal attention maps; the selected map remains free to include the object.
Broadcasting. One mined target guides the mean attention of each shallow layer while preserving diversity between heads.

Attention maps focusing on hands, robot grippers, and manipulated objects — Human videos (left) and robot scenes (right). Top: with attentive guidance; bottom: without.

1

Let interaction-aware attention emerge

\[\mathcal{L}_{\mathrm{motion}}=\left\|\hat T-T\right\|_1\]

\(T\) is the ground-truth future 3D wrist trajectory; \(\hat T\) is the model's prediction. The \(\ell_1\) norm sums absolute coordinate errors.

Warm up for 10 epochs with this Act-phase objective. Future hand motion depends on hand-object spatial relations, encouraging interaction-aware attention to emerge before we mine a target.

2

Build a soft hand prior

\[h=\operatorname{Softmax}(H/\tau)\]

\(H\) is a hand heatmap built from Gaussian responses around 21 detected hand keypoints and resized to the encoder's patch grid.

\(\tau\) controls softness; Softmax normalizes over all patches. The resulting \(h\) is a vector of hand-prior weights, with \(h_i\) the weight of patch \(i\). It is a cue for selection, not the supervision target.

3

Score the model's emerging attention

\[s_{n,k}=\frac{1}{N}\sum_{i=1}^{N}a_{n,k,i}\,h_i\]

\(n\) indexes a shallow layer, \(k\) an attention head, and \(i\) an image patch; \(N\) is the total number of patches. We search the first \(R=6\) layers.

\(a_{n,k,i}\) is attention from the image-summary [CLS] token to patch \(i\) in layer \(n\), head \(k\). The score \(s_{n,k}\) measures its overlap with the hand prior: a larger score identifies a better candidate.

4

Mine the interaction-aware target

\[\begin{aligned}(n^*,k^*)&=\arg\max_{n,k}s_{n,k}\\ a^*&=a_{n^*,k^*}\end{aligned}\]

\(\arg\max\) returns the winning layer and head, \((n^*,k^*)\). The selected target \(a^*\) is that head's complete attention map over all \(N\) patches.

We retain the full map, not just its hand region. Its emergent attention to the interacting object therefore remains part of the target.

5

Broadcast the pattern through self-distillation

\[\begin{aligned}\tilde a^*&=\phi(\operatorname{sg}(a^*))\\ \tilde a_n&=\phi(\bar a_n)\\ \mathcal{L}_{\mathrm{att}}&=\sum_{n=1}^{R}\left\|\tilde a^*-\tilde a_n\right\|_2^2\end{aligned}\]

\(\bar a_n\) is the mean attention map over all heads in layer \(n\). Min-Max normalization \(\phi\) rescales each map's spatial values to \([0,1]\); stop-gradient \(\operatorname{sg}\) prevents this loss from updating the selected target.

Thus \(\tilde a^*\) is the normalized target and \(\tilde a_n\) the normalized layer mean. The squared \(\ell_2\) distance compares their patchwise values, and \(\mathcal{L}_{\mathrm{att}}\) sums these errors across the first \(R=6\) layers.

Matching each layer's mean attention broadcasts the mined interaction pattern without requiring all individual heads to become identical.

Key evidence

The hierarchy improves accuracy and robustness

92.2%MetaWorld ablation average

+18.9points over no pretraining

68.3%real-world average success

60%data needed to match baseline peak

Human-video pretraining

More pretraining data keeps helping

+9 pts

Success increases from 83.8% to 92.8% as the pretraining set grows.

Robot demonstrations

Less downstream data, stronger policy

LSA Baseline

LSA reaches the baseline's peak performance using only 60% of expert demonstrations.

Real-world evaluation

Four tasks, two unseen appearances

A Franka Research 3 uses RGB observations from a fixed RealSense D435i camera.
Each task uses 30 demonstrations and 10 evaluation trials.
Two unseen towel colors test appearance-level out-of-distribution generalization.

Full overview of the Franka setup, four real-world demonstrations, and two out-of-distribution tasks — Real-world setup, four task demonstrations, and two unseen towel-color variants.

Success rate (%)

Real-world comparison

Best average

Method	Pluck Flower	Pick Trash	Place Toy	Wipe PC	Unseen 1	Unseen 2	Average
DINOv3	20	30	20	30	10	20	21.7
R3M-A	10	30	30	40	0	10	20.0
VIP	50	40	50	40	20	20	36.7
HRP	40	40	60	50	20	40	41.7
LSA (Ours)	50	70	80	80	70	60	68.3

Wipe PC

Pluck Flower

Pick Trash

Place Toy

Wipe PC · Unseen 1

Wipe PC · Unseen 2