XLeRobot Bartender — Complete project status

2026-09-04 · written for anyone joining or returning to the project

Every number here is either measured and reproducible from this repository, or explicitly marked as not measured. Incorporates the 2026-09-04 remote-task results, including one correction that invalidated a figure repeated across several earlier documents — see §7.4. Where a figure comes from someone else's paper or product page, it says so.

1. What we are building, in one paragraph

A bartender robot whose selling point is not throughput. Commercial bartender robots already make drinks quickly and consistently — Makr Shakr Veloce does 240 drinks/hour — and they do it by building the entire installation around the machine: bottles on a fixed rail, glasses in a fixed jig, arms replaying pre-programmed trajectories. Their AI sits in the recommendation layer. The arm does not look at anything.

We are building the one that watches the customer: greets them, makes the drink, rotates the glass so they can see it, and hands it toward where they actually are. That requires learned manipulation rather than recorded trajectories, and it requires perception directed at a person, not just at an object.

The honest state of that ambition: our grasp success rate is currently 8/83 (95 % CI 4.3–18.1 %), and we have not recorded a single pouring demonstration.

2. Hardware

Action and state are 17-dimensional: 6 + 6 arm, 2 head, 3 base velocity.

Why the compute matters strategically: 7.4 GiB cannot hold a 7-billion-parameter VLA. That rules out the OpenVLA family and forces us toward smaller models and cleverer representations rather than bigger ones. It is a constraint, but it is also what makes any result we get transferable to affordable hardware.

3. Data

Largest sets: xlerobot-cup-grasp-20260820-0230 (50 ep — what the current policy was trained on), xlerobot-right-pick-cup-20260826-0042 (40 ep — current gripper), xlerobot-left-pick-cup-sep3-20260903-2015 (29 ep — most recent).

All of it is cup grasping. There are zero demonstrations of pouring, shaking, stirring, presenting or handing over.

3.1 Depth is recorded but unusable by any policy

Depth exists as a side-channel: <dataset>/depth/ep<N>-<timestamp>/frame_NNNNNN.png, 640×480 16-bit PNG in millimetres, ~863 MB total.

It is not in any dataset's feature list, so no training or inference code can read it. Measured quality where it does exist: 76–86 % valid pixels, median depth ≈ 500 mm — the data itself is good. First 1–2 frames per episode are zeros (sensor warm-up); there are outliers up to 32 m that are noise and must be clipped.

Native in-dataset depth recording is implemented and waiting on a service restart.

3.2 One structural fact that shapes everything

head_1    σ = 0.0000    range [-4.308, -4.308]
head_2    σ = 0.0000    range [49.846, 49.846]

Across all 19,453 frames of the training set, the head never moved. The policy has never seen a moving camera. This is why gaze must stay frozen during manipulation and can only follow a person before and after — see §7.

4. Models and what they have actually achieved

Seven checkpoints exist locally. Trial results across 37 result files / 114 trials:

Overall: 8/83 counted trials succeeded — 9.6 %, 95 % CI 4.3–18.1 %.

Two things follow that are easy to get wrong:

1. 0/4 is not a failed model, it is an untested one. Its 95 % upper bound is 60.2 %.
2. Four checkpoints have never touched hardware at all, including b16-3cam-u20k, which was trained on all three cameras and is therefore not comparable to the others in input form.

Zero of the 37 batches were pre-registered (a success criterion fixed before the run). By our own standard, all existing trial data is exploratory.

5. What works today

5.1 A live viewer arrived from the remote side

03-software/conversational_ai/cup_test.py (165 lines) serves the head camera in a browser with face and cup boxes drawn on it: ./run.sh cup_test.py, then open port 8795. It uses YuNet for faces — the same choice we reached independently — and EfficientDet-Lite0-int8 for cups.

Two design details worth keeping: it finds the Orbbec camera by name rather than by a /dev/videoN number, and it falls back to the shared frame in /dev/shm when another program holds the camera. Both directly avoid failure modes this project has hit before.

One caveat that matters (§6.1): it opens the camera at 1280×720, whereas the head camera is recorded — and therefore seen by the policy — at 424×240. Nine times the pixels. A detector that looks convincing in the live viewer may still fail on the frames the policy receives, which is exactly what our offline evaluation found for its default cup model.

A useful corollary: the camera evidently can deliver 1280×720, so raising the recording resolution may be a configuration change rather than new hardware. <span>UNTESTED — bandwidth, disk and retraining cost all unknown</span>

6. What does not work, or does not exist

6.1 Cup detection — measured 2026-09-04, and it failed acceptance

Four pretrained detectors, run offline over recorded frames, every frame reviewed by eye:

Revised twice on 2026-09-04. First downward, after every box was cropped and identified by eye (contact-sheet thumbnails had made a VR controller look like a cup). Then substantially upward, after switching to OWLv2.

The first pass — four detectors, all failing

OWL-ViT's false positives were a VR controller left on the table (5 boxes); Grounding DINO's were cable-grommet holes in the table, the mouse, the gripper and cabling (26 of 27).

The second pass — OWLv2 changes the picture

Same 36 frames, same queries, same selection rule; only the model changed.

The VR controller and the table holes produce zero boxes under OWLv2.

And thresholding went from useless to decisive, which is the structural change:

What this costs, and what is still unknown

One operational prerequisite that will fail silently if forgotten

Background subtraction requires the reference image and the runtime camera pose to match. The head is locked during trials, so within a session this holds. But changing the head preset invalidates the reference, and nothing will warn you. This must go into the operating procedure.

6.2 Two things found while re-checking, both cheap to act on

1. A measurement discipline caught a second mistake of the same kind. The rule adopted was: every filter must report two numbers — false positives removed AND true positives lost. Re-scoring earlier interventions under that rule overturned one verdict:

| Intervention | False positives | Target cups | Net | Previously called | |---|---|---|---|---| | OWL-ViT, drop "a drinking glass" | 6 → 7 (worse) | 5 → 1 | loss | correctly, a loss | | Grounding DINO, drop "a drinking glass" | 26 → 1 | 1 → 1 (no loss) | clear gain | wrongly, "over-filtered" | | OWL-ViT → OWLv2 | 6 → 3 | 5 → 18 | double win | — | | OWLv2 threshold 0.05 → 0.30 | 3 → 0 | 18 → 16 | worthwhile | — |

The Grounding DINO run had been dismissed because only the box count was reported — it dropped to 2/36 and that looked like a collapse. Counting both numbers shows it removed 25 false positives at zero cost. Reporting one number instead of two produced a wrong conclusion twice in one investigation.

1. The cascade's layers need reordering, because layer 1 turned out to do layer 2's job.

| Layer | Was meant to remove | Left to remove after OWLv2 | Its value now | |---|---|---|---| | ① Semantic (open-vocabulary + corrected distractors) | arm, mouse, cables | — | non-cup false positives are zero | | ② Depth plane residual | flat things (the table holes) | 0 — the holes no longer produce boxes | ≈0 as a filter, but still the only source of height/distance for grasping | | ③ Empty-table background difference | permanent scene objects | 3 (the other plastic cup) | the only layer still removing errors |

New order: ① → ③ → ②. This is not a rejection of depth — its role changed from filtering to supplying 3D position, and that still needs intrinsics and depth/RGB alignment, which are on-site work. It should no longer block validating layer ③.

1. Background subtraction at 40 % is free. Removing boxes whose changed-pixel fraction is below 40 % kills 2 of the 3 "other cup" boxes and costs zero target cups. At 50 % it removes all 3 and costs 1 target cup.

7. The design decisions that are settled, and why

7.1 Gaze is phase-based, and this was forced by data

Two independent constraints force the middle row: the policy has never seen a moving head (§3.2), and the depth camera's extrinsics are only valid for one head pose. This was not chosen; it is what survived elimination.

Stage 1 is the cheapest demonstrable win in the project — the arm is still, so it conflicts with nothing, and no commercial product does it.

7.2 First version pushes the glass; it does not hand it over

Handing a glass into someone's hand is a human-robot handover problem whose core difficulty is when to release. Too early and it drops; too late and the person cannot take it. Humans use touch. We have no force sensing (§2). Pushing the glass along the table keeps support under it the whole time and needs no release decision.

7.3 Learned motion, not recorded trajectories

This is the project's one real technical bet, and it is unproven — see §4.

7.4 Every claim carries its evidence level

A worked example from 2026-09-04. Several documents in this repo — including an earlier draft of this one — stated "SmolVLA runs at 146 ms/frame (6.9 Hz)". Re-measuring produced 1329 ms, nine times larger. Tracing it back: the 146 ms figure was a real measurement from 2026-08-21, but of ACT, not SmolVLA; the model name was substituted somewhere in the chain of hand-off documents. Re-measuring both on the same machine confirmed it: ACT 129.5 ms, SmolVLA 1329.2 ms.

A second, framework-level error travelled with it: "6.9 Hz cannot keep up with 30 Hz training" is not a meaningful statement about a chunked policy. SmolVLA plans 50 action steps per inference, which is 1667 ms of motion at 30 Hz. The right question is whether one inference fits in 1667 ms — it does, at 1329 ms, with 337 ms to spare.

Both errors survived multiple documents because nobody re-measured. That is the failure mode this rule exists to catch.

measured / not measured / tried but blocked. Adopted after a review found untested inferences being reported as facts. It applies to other people's papers too: Oat-VLA reports 59 % vs 41 % on real robots; recomputing from its own counts gives 29/49 vs 20/49, overlapping intervals, Fisher p = 0.106 — not significant. We cite its token reduction, not its success rate.

8. Known architectural conflicts

One factual correction found on 2026-09-04: policy_safety.py's own note asserts the training set carries head_motor_2 = 99.649, and concludes from that arithmetic that the head sat 564 counts outside the calibrated range — which is why the absolute clamp skips the head entirely. The data says 49.846, which recomputes to inside the range by 2 counts. The premise is wrong. Harmless while the head is pinned; not harmless once the head moves.

9. Market position

Verified from vendor pages and named sources (full analysis with images: 04-reports/market-analysis/, not published publicly):

The gap, stated narrowly enough to survive checking: conversational bartender robots exist (Cecilia, Yanu) and expressive robot arms exist (Makr Shakr), but no product puts both in the same body — and in all of them the arm motion is pre-programmed. Their sourced description: "The recipe database contains pre-programmed ratios, sequences, and pour volumes", while machine learning "tracks popular combinations, personalizes suggestions". The AI is in the recommendation layer, not the motion layer.

10. People and where they are

Only one person can touch the machine. Everything else is remote, which is why the on-site brief (BRIEF-ONSITE-HARDWARE.md) is written around "anything you don't measure while you are there, nobody can measure later".

11. What happens next

The open decision — this one blocks everything downstream

Since detection failed on disambiguation rather than recognition, four routes exist. Recommendation from the analysis: (a).

On-site (needs the robot)

1. Save the trial head preset — 1 minute, everything geometric depends on it
2. Trial the four never-tested checkpoints, with pre-registration, 10 runs each
3. Camera intrinsics → hand-eye extrinsics → depth/RGB time alignment
4. Restart 8781, record 3–5 episodes of native-format depth for verification
5. Measure joint speed of the rotate-the-glass motion before recording that skill

Remote (parallel, no hardware)

• A — open-vocabulary detection over recorded RGB to settle whether the cup can be located at all. Order decided: OWL-ViT first, then Grounding DINO + SAM. Acceptance is fixed in advance: ≥30 frames checked by eye, ≥90 % correct, holds for at least one different cup
• B — ✅ done 2026-09-04. 1 camera 1210.7 ms, 2 cameras 1278.4 ms, 3 cameras 1329.2 ms — +59.2 ms per camera; all visual cost is 177.7 ms, only 13 % of total. Removing every visual token could save at most that, and a segmentation model is itself a ViT. B0's speed rationale is dead. Recommendation: shelve it rather than delete — it is the only route needing no calibration, so it becomes relevant again if route (a) stalls
• C — ✅ partly done 2026-09-04. Detector decision made: keep YuNet (measured 18.9 ms at 640×480, 10.0 ms at 320×240; 227 KB, already in the repo). The Haar path in face_follow.py was found to be non-functional — it was never "two working options". Head-clamp limits are now derived (head_motor_1 ±149.98°, head_motor_2 ±50.02°) but deliberately not applied yet; recommendation is to land them together with head-following rather than in isolation
• D — ✅ done 2026-09-04, --dry verified, trials_io reads the new fields end to end

The ordering principle

Nothing downstream matters until the cup can be reliably located. Pouring, presenting and handing over all begin with a successful grasp, and grasping currently succeeds 8 times in 83.

12. The three ways this most plausibly fails

All three have a first step that is remote, cheap, and can be done before the next batch of demonstrations is recorded.

Companion documents: BRIEF-FACE-INTERACTION.md · BRIEF-ONSITE-HARDWARE.md · HANDOFF-REMOTE-2026-09-04.md · TOMORROW-2026-09-04.md · three GOAL files · 04-reports/market-analysis/ · STATUS.md (machine-generated, always current).