For: whoever is physically at the robot · From: the remote side · 2026-09-04
Everyone else on this project is remote. You are the only one who can touch the machine, which makes a few things true that are worth stating plainly:
There is a dated, ordered action list in TOMORROW-2026-09-04.md (Chinese). This document is the standing role brief — the things that stay true between sessions.
| Compute | Jetson Orin Nano Super, 7.4 GiB shared CPU/GPU memory, 16 GB swap |
|---|---|
| Arms | Two, 6 DoF each (Feetech serial-bus servos) |
| Head | Pan/tilt, servos 7 and 8, on the left arm bus |
| Cameras | 3 × RGB — head 424×240, left wrist 640×480, right wrist 640×480 |
| Depth | Orbbec, mounted on the moving head |
| Base | 3 velocity dimensions (currently always zero in recordings) |
| Serial buses | /dev/ttyACM0, /dev/ttyACM1 |
| Disk | 32 GB free of 116 GB — this is a hard constraint, see §5.3 |
Action/state vector is 17-dimensional: 6 left arm + 6 right arm + 2 head + 3 base velocity. No force or torque sensing anywhere — servo load is not in the observation.
| Port | Service | What it is |
|---|---|---|
| 8770 | robot-status | The main control panel. Trials, calibration, datasets, jobs |
| 8781 | leader-follower | Interactive recording: move the LEFT arm, the RIGHT follows and records |
| 8790 | xle-upload | Drag-and-drop file drop onto the robot |
Restart any of them with systemctl --user restart <name>. Logs: journalctl --user-unit=<name> -f.
trial preset before anything that will be analysedThe camera sits on a head that moves. Every geometric calibration is only valid for one head pose. And run_policy_trials.py locks the head to wherever it happens to be at startup — it is not a fixed value.
Consequence: if the head was in a different place on Tuesday than on Wednesday, Tuesday's calibration is void and Tuesday's trials are not comparable to Wednesday's. Nothing will warn you. The numbers will just quietly stop meaning the same thing.
Status: the preset named trial does not exist yet. Existing presets in 03-software/configs/head_presets.json: test, cup-grasp-v0 (pan 2003, tilt 2619), sky, justforfun. Creating trial is a one-minute job and everything else depends on it.
cup-grasp-v0 has tilt = 2619,range_max is 2617 — it is sitting 2 counts past the limit.| Check | Why |
|---|---|
| Nothing else is using the serial buses | See §5.1. This has cost us an episode and dropped both arms |
Head at the trial preset | §2 |
| Depth camera on a USB 3 port | On USB 2 the bandwidth is insufficient; the recorder's own message is "多半是 USB 带宽不够(该走 USB3)或相机假死,别开始录" |
| Disk has room | §5.3 |
| Same cup, same lighting, same table layout as the batch you're extending | Otherwise you have created a second dataset, not more of the first |
Press Start exactly once. Pressing it twice used to orphan the recorder process (fixed 2026-09-03, but do not go testing it). The symptom of an orphan was a process sitting at ~30% CPU forever, quietly filling the disk.
The dataset is uploaded once, at the moment recording ends. Nothing uploads it later.
Check meta/upload.json in the dataset directory. If the upload did not happen, it must be pushed manually via /api/dataset/publish — it will not catch up by itself.
Depth frames land in <dataset>/depth/ep<N>-<timestamp>/frame_NNNNNN.png — 640×480 16-bit grayscale PNG, millimetres. They are not in the dataset's feature list, which means no training or inference code can see them.
Native in-dataset depth recording is implemented (orbbec_depth_camera.py) but requires 8781 to be restarted to take effect. When it is, please record 3–5 episodes only first — the goal is to verify the format, not to collect training data. We will check it remotely before you record a full batch.
Measured quality of the existing side-channel depth (so you know what "good" looks like): 76–86 % valid pixels, median depth ≈ 500 mm. The first 1–2 frames of each episode are all zeros — sensor warm-up, expected.
The control panel now asks for a success criterion before the run starts, in the 实机试验 / Physical trials card. Fill in:
Once written, do not change it mid-batch. Change it for the next batch if you must.
You can leave it empty and the run still works — but that batch is then labelled "exploratory" and does not count toward any conclusion. That is not a punishment, it is just what an un-preregistered result is worth.
After each trial the panel asks: success / failure / void.
Setup mistakes, human intervention and hardware faults are void, not failure. A void trial stays out of the denominator. Marking a mis-placed cup as "failure" makes the model look worse than it is, and it is unrecoverable after the fact.
From our own records: a batch of 0/4 is consistent with a true success rate as high as 60.2 % (95 % Clopper-Pearson). A batch of 0/23 bounds it at 14.8 %.
Running 4 trials proves essentially nothing. The panel now shows the interval next to the rate so this is visible rather than something you have to remember.
A hand-typed trace_teleop.py was started while a recording was running. The recording died with [TxRxResult] Port is in use! — the episode was lost and both arms dropped onto the table.
The panel serialises anything it launches. A command you type yourself walks straight past that guard. bus_guard.py exists specifically because documenting the rule was not enough.
**Rule: one program per /dev/ttyACM* bus. No exceptions.**
The post-incident analysis named "lack of operator-visible status feedback" as a contributing factor — the person driving could not see what the safety layer was doing. That is why the panel now shows a large live safety status and an e-stop.
Watch it. It is there because of this.
The root partition is 116 GB with 32 GB free. A full disk does not degrade gracefully — it breaks tooling in confusing ways. Two known ways it fills: model checkpoints pulled back from cloud training, and (historically) an orphaned recorder.
Check df -h / before recording a large batch.
GitHub hard-rejects files over 100 MB, and the history-rewriting tools are blocked in this repo. Never commit model weights. Use HuggingFace, or the upload page at :8790.
Three hand-written cup detectors were built. Detection rates of 79 %, 97.5 %, 97.6 %; jitter down to 0.9 mm; plausible width estimates.
All three were locking onto the robot's own arm. It was only found by looking at the images.
Rule: a perception result is not a result until someone has looked at the pictures. This applies to anything you evaluate on-site too.
These are open questions that cannot be resolved remotely. Even a one-line answer helps.
| Question | Effort | |
|---|---|---|
| 1 | Head tilt: does a larger value look up or down, and how much travel remains from the recording pose? (§2) | 1 min |
| 2 | What cups do we actually have? Plastic / frosted / coloured stemmed glass? Photograph them. Depth cameras handle opaque and frosted fine; transparent glass is the hard case, and it decides whether a depth-based approach is viable at all | 5 min |
| 3 | Where does the customer physically stand relative to the robot? Sitting or standing? This sets the geometry for the whole "hand it to the customer" stage | 2 min |
| 4 | Before any pouring/presenting demonstrations are recorded: teleoperate the rotate-the-glass motion once and record 10 seconds. We will check whether it exceeds the per-frame step cap — which silently slows motion rather than rejecting it, so a whole batch could be recorded with the motion flattened and nobody would see it | 5 min |
| 5 | Is there any visible wear or slack in the gripper or wrist joints? | 2 min |
http://100.107.145.111:8790/.policy_safety.py). Its floor constraints were re-derived on 2026-09-03 — rejected-frame rate went from 96.6 % to 0.49 %. Leave it alone this week.failure. §4.2.Related: TOMORROW-2026-09-04.md — the dated action list. · GOAL-TRIAL-DATA-MANAGEMENT.md — why pre-registration and intervals exist. · GOAL-DEPTH-INTEGRATION.md — the depth plan and the three failed detectors.