Evidence archive · 2026-09-09

Generalization trials: 28 runs across 4 objects

This page exists so that every success rate we quote can be traced: to a specific run, to what was on the table at the time, and to a video of it. The organiser warned explicitly against unsupported success-rate claims.

The two figures safe to quote
· Same object (the yellow cup, present in the training data): 9/10 = 90%
· Objects never seen in training (blue cup / blue can / red can) combined: 8/12 = 67%
Always state them as "n of m".

The four objects

Yellow cup start frame — **Yellow cup**
in the training data
**9/10 = 90%** (13 runs, 3 void)

Blue cup start frame — **Blue cup**
never seen in training
**2/3 = 67%** (5 runs, 2 void)

Blue can start frame — **Blue can**
unseen · different material
**3/5 = 60%** (6 runs, 1 void)

Red can start frame — **Red can**
unseen · different material and colour
**3/4 = 75%** (4 runs, 0 void)

Object	Note	Runs	Success	Failure	Void	Rate
Yellow cup	in the training data	13	9	1	3	90%
Blue cup	never seen in training	5	2	1	2	67%
Blue can	unseen · different material	6	3	2	1	60%
Red can	unseen · different material and colour	4	3	1	0	75%
Total		28	17	5	6	77%

How the rate is defined: the denominator excludes void runs — a void is a run discarded for reasons outside the model (an interrupted operation, for instance) and is not counted for or against it. Including voids the overall figure is 17/28 = 61%.

Autonomous footage — not teleop, not replay

Cut to 1:47 for the two-minute technical demo slot, in three parts:
① 11 grasps at 4× speed — for each object, the 2–3 successes with the widest positional separation; each clip is labelled with the object, that object's success rate, and where the cup actually was;
② a 1 → 9 → 16 → 25 grid montage, each tile one real grasp closing, bordered by object colour;
③ the position distribution chart (same image as below).
All source material comes from 05-training/trials/videos/ and keyframes/, with no cosmetic editing.

Where each grasp closed

Every point is the position at which one grasp actually closed, computed by forward kinematics from that frame's joint angles. Colour and shape both encode the object (dual encoding, so it survives colour-blindness); filled = success, hollow = failure, grey = void.

Position distribution of 28 grasps, coloured by object

About 24 cm of lateral spread and 15 cm of reach. This chart is the direct evidence of generalization — one policy, four objects, all at different places.

How to show a juror this is autonomous rather than replayed or teleoperated
· Every run has its own trace file holding two sequences — the policy's raw output and the safety-filtered command. A replay has no policy output at all;
· The trial record carries counts{OK/CLIPPED/HELD}, the safety layer's per-frame intervention tally. The teleop path does not produce it;
· The same model reaches 67% on three objects that never appear in the training data. A replay cannot do that.

Run by run

All 28 in chronological order. Object identification by the operator who was present.

#	Batch	Object	Outcome	Video
1	170405 t1	Yellow cup	success	yes
2	170405 t2	Yellow cup	success	yes
3	171624 t1	Yellow cup	success	yes
4	172016 t1	Yellow cup	success	yes
5	172016 t2	Yellow cup	success	yes
6	172016 t3	Yellow cup	success	yes
7	172016 t4	Yellow cup	void	yes
8	172016 t5	Yellow cup	success	yes
9	172016 t6	Yellow cup	void	yes
10	172016 t7	Yellow cup	void	yes
11	172016 t8	Yellow cup	failure	yes
12	172016 t9	Yellow cup	success	yes
13	172016 t10	Yellow cup	success	yes
14	174614 t1	Blue cup	failure	yes
15	174614 t2	Blue cup	success	yes
16	175304 t1	Blue cup	void	yes
17	175304 t2	Blue cup	success	yes
18	175304 t3	Blue cup	void	yes
19	175304 t4	Blue can	failure	yes
20	180807 t1	Blue can	success	yes
21	180807 t2	Blue can	success	yes
22	180807 t3	Blue can	void	yes
23	180807 t4	Blue can	failure	yes
24	180807 t5	Blue can	success	yes
25	182715 t1	Red can	success	yes
26	182715 t2	Red can	success	yes
27	182715 t3	Red can	failure	yes
28	182715 t4	Red can	success	yes

Where the raw material lives

Content	Path
Trial records (incl. per-frame safety statistics)	`05-training/trials/trials-20260909-*.json`
Keyframes (init / closure / end)	`05-training/trials/keyframes/`
Per-run video	`05-training/trials/videos/`
Policy raw output vs safety-filtered output	`05-training/trials/traces/*.npz`
Model	`xlerobot-smolvla-left-sd-p0.2-b16-u20000` / checkpoint 020000
Training dataset	`xlerobot-team/xlerobot-left-pick-cup-sep3-clean-20260905`

Start frame of all 28 runs

If a juror wants to check what was on the table each time, this one image covers it. Labels read "batch t-number [S success / F failure / V void]".

One thing worth improving: the trial record has no field for which object was used — identity exists only inside the keyframe images, and the table above had to be assembled by having a person identify them batch by batch. Adding a target_object field would turn this kind of check into a query.

Model xlerobot-smolvla-left-sd-p0.2-b16-u20000 · trials dated 2026-09-09 · object labelling by the on-site operator, 2026-09-10 · per-run outcomes taken directly from the trial records with no manual adjustment
This page uses system fonts only and makes no external requests.