Two policies trained on the identical 50-episode dataset, two days apart, on the same 8 GB Jetson. One of them has evidence it generalises. The other has none — and the difference has almost nothing to do with which curve looks lower.
ACT finished at 0.138, SmolVLA at 0.054. That does not make SmolVLA 2.5×
better. ACT's loss is an L1 action error plus 10× a KL divergence term
(kl_weight = 10.0); SmolVLA's is a flow-matching objective on a pretrained
backbone. Different formulas, different scales, different starting points — SmolVLA opens at
2.37 because it starts from smolvla_base, while ACT opens at 11.63 from scratch.
The two curves below are deliberately drawn as separate charts with their own axes. Putting them on one axis would invent a comparison the numbers cannot support.
Log scale on both. Drag across either chart to read values. What is worth comparing is the shape — whether the tail flattens — not the height.
| ACT | SmolVLA | |
|---|---|---|
| Dataset | identical — 50 episodes / 19,453 frames / 5 cup positions, right arm only | |
| Init | from scratch, 51.6 M params | finetuned from smolvla_base |
| Batch size | 2 | 2 |
| Steps | 20,000 — ran to completion | 18,000 of 20,000 — stopped early |
| Epochs | 2.06 | 2.27 |
| Final loss | 0.138 | 0.054 |
| …comparable? | No — different objectives | |
| Tail behaviour | −13.2% over the last 5,000 steps | 0.053 → 0.054 over the last 2,000 steps |
| Validation during training | ✕ eval_freq = None | ✕ eval_steps = 0 |
| Held-out evaluation after | ✕ none | ✓ held-out cup position unseen 0.2300 vs trained 0.2348 = 0.98× |
| Uploaded checkpoint | step 20,000 (and 4,000 at an earlier revision) | step 16,000 |
| Structured training log | ✕ had to be grepped out of a progress bar | ✓ training_log.csv |
| Run on the physical robot | ✕ never | ✕ never |
Three of these are training decisions that could have gone differently. One is a machine that ran out of disk. One is a property of the method, not a mistake.
eval_freq = None, so nothing measured generalisation while it trained and no
held-out check was run afterwards either. This is the single reason ACT cannot be judged.
It was avoidable and it was called out in the project's own guidance: training loss is not
an acceptance criterion.
8 GB of unified memory forced batch 2, and 20,000 steps over 19,453 frames is barely two passes through the data. Gradients are noisy and both runs stopped while still improving. ACT's last 5,000 steps still fell 13.2%; SmolVLA's tail is flatter but rests on two logged points 2,000 steps apart, which is thin evidence for "converged".
KLD fell from 1.122 to 0.004 against kl_weight = 10.0. The latent carries
nothing and the model has degenerated into deterministic chunk regression. Common for ACT
on a small single-task set, but it means the architecture's stated advantage is not in play.
Not divergence, not a manual stop — each SmolVLA checkpoint is 1.26 GB and one was written every 2,000 steps. The step-18,000 write failed partway. Nothing to do with the model, and the same disk pressure is why 8 of ACT's 10 checkpoints were later deleted.
That is why it opens at 2.37 instead of 11.63, and it is the main reason to expect it to do better on 50 episodes: the visual and language priors are already there. This is the property the project wanted to test, not a mistake in either run.
Recording ran with arms: 1.0, so dimensions 0–5 hold a constant across every
frame. Both policies inherit this: neither can drive the left arm, and neither will
transfer to a bimanual task. A bimanual dataset needs its own recording.
Not because its loss is lower — that comparison is meaningless — but because it is the only one of the two with any evidence about generalisation. Trained without one cup position and then evaluated on it, SmolVLA scored 0.2300 against 0.2348 on positions it had seen: a ratio of 0.98×. The unseen position was not harder. That is exactly the position-memorisation question the project needs answered, and ACT has nothing equivalent.
Two caveats that must travel with that recommendation. The evaluation is offline replay of recorded observations — the policy's own mistakes never change the next frame, so it cannot catch compounding error. And neither policy has driven the arm. The acceptance bar is still 8 of 10 physical pickups.
Redoing ACT now would cost hours and answer a question the hardware trial answers better and faster. But the run has three defects worth not repeating, and they are all cheap to avoid.
eval_freq so it actually runs.training_log.csv; ACT's numbers
had to be extracted from 40,000 lines of progress bar. One CSV line per log step.df, is a five-second check.| Signal | What it means | Verdict |
|---|---|---|
| Validation loss turns up while training loss keeps falling | Overfitting. More steps will make it worse, not better. | stop early, don't retrain |
| Held-out-position loss ≫ trained-position loss | The policy memorised coordinates instead of looking. More data or stronger priors, not more steps. | retrain — augment or use a pretrained base |
| Hardware rollout fails, but identically from every cup position | The failure is in calibration, camera, or the inference path — the policy never got a fair chance. | fix the pipeline, don't retrain |
| Hardware rollout works at trained positions, fails 2–3 cm away | Position memorisation confirmed on the real robot. This is the one that justifies a retrain. | retrain |
The cheapest of these to run is the third and fourth: ten physical trials, at least three of them with the cup moved 2–3 cm. Twenty minutes on the robot separates "the model is wrong" from "the plumbing is wrong" better than any amount of retraining does.
Every number above is one of these. Read this before arguing about any of them.
| Term | What it is | On this page |
|---|---|---|
| Loss | One number saying how wrong the model's output was on a batch of examples. Training does nothing but push it down. It says nothing about whether the model is useful — only how well it reproduced the examples it was shown. | ACT 0.138 SmolVLA 0.054 |
| L1 loss | Mean absolute error between the predicted joint angles and the ones the human teleoperator actually commanded. In ACT this is the part that corresponds to "how far off were the motions". | 0.094 |
| KLD (KL divergence) |
How far ACT's internal latent distribution has drifted from a plain unit gaussian. ACT is a CVAE: it is meant to keep a small random latent that captures the operator's style. The training objective adds this term at kl_weight = 10 to stop the latent memorising the answer. |
0.004 collapsed |
| Latent collapse | When KLD falls to near zero the latent has stopped carrying information — the model ignores it and becomes deterministic. Not fatal, but ACT's stated ability to represent multi-modal behaviour is then not in play. | 1.122 → 0.004 |
| Step | One gradient update — one batch in, one adjustment to the weights. | 20,000 / 18,000 |
| Batch size | How many examples go into one step. Small batches make each step's gradient noisy, which is why these curves wobble. Memory caps it here. | 2 |
| Epoch | One full pass over the dataset. steps × batch ÷ frames. Two epochs means the model saw each frame about twice. | 2.06 / 2.27 |
| Gradient norm | The size of the correction being applied. Large early, shrinking as the model settles. A spike means something unstable; a plateau near zero means it has stopped learning. | 378 → 17.3 |
| Convergence | The loss has stopped falling — the curve is flat, not merely low. Judged on the tail, never on the final value. | ACT: no SmolVLA: probably |
| Validation loss | The same loss, measured on data the model was never trained on. The only number that says anything about generalisation. Neither run produced one during training — eval_steps defaults to 0 in LeRobot. |
neither |
| Overfitting | Training loss keeps falling while validation loss turns up: the model is memorising rather than learning. Undetectable without a validation split — which is why ACT's 0.138 proves nothing. | unknown |
| Action chunk | Instead of predicting the next single command, the policy emits a whole run of future actions and executes them open-loop. ACT's is 100 steps, SmolVLA's 50. It is what makes these policies smooth rather than jittery. | 100 / 50 |
| Held-out position | Training with one of the five cup positions removed, then testing on it. Answers "did it learn to look at the cup, or memorise five sets of coordinates". Still offline — the policy's own errors never change the next frame. | 0.98× ✓ |
Both citations are taken from the LeRobot source that actually runs here, not from memory.
papers/2304.13705 ·
code github.com/tonyzhaozh/act ·
cited at modeling_act.py:45
The ALOHA paper. Its argument is that imitation learning fails on fine manipulation because
predicting one action at a time compounds error, so it predicts a chunk of ~100
actions at once and executes them together. The CVAE latent is there to absorb the
inconsistency between human demonstrations. Its KL term follows Kingma & Welling,
papers/1312.6114
App. B — cited directly in the loss code at modeling_act.py:157.
What it buys here: smooth 100-step motions from 50 demonstrations, trained entirely on this Jetson, with no external model and no internet. What it costs: everything is learned from scratch, so 50 episodes is very little, and its multi-modal machinery is idle once the latent collapses.
papers/2506.01844 ·
cited at modeling_smolvla.py:20
A vision-language model with an action expert bolted on. The VLM has already learned
what objects look like and what instructions mean from internet-scale data; finetuning
trains only the small expert that turns those features into joint commands
(freeze_vision_encoder = True, train_expert_only = True are the
shipped defaults). It takes a language instruction — here, "Grab the cup".
What it buys here: priors that 50 episodes could never teach, which is the most likely reason it survived the held-out-position test. What it costs: a much larger checkpoint (1.26 GB per save — the reason the disk filled), and a dependency on a pretrained base.
Taken from LeRobot's own defaults and the SmolVLA finetuning example in its docstring, on this machine.
| Used here | LeRobot default | SmolVLA's own example | |
|---|---|---|---|
| Batch size | 2 | 8 | 64 |
| Steps | 20,000 | 100,000 | 200,000 |
| Samples seen | 40,000 | 800,000 | 12,800,000 |
| Ratio to the example | 1 / 320 | 1 / 16 | 1× |
| Validation | off | off by default | — |
Two things follow. First, "the loss did not flatten" is unsurprising at 1/320 of the reference
throughput — neither run was ever going to converge, and reading either curve as a
verdict on the method is a mistake. Second, the missing validation was not an oversight unique
to this project: eval_steps ships as 0 in LeRobot, so it is off unless
someone turns it on. That is worth knowing before blaming the run.