← reports index
03-software/brain · 交接说明 · 2026-09-11
交接说明：现在用哪些模型 · 右臂怎么跑起来的 · 倒酒怎么实现的
写于 2026-09-11 16:30，给做 integration 的 Ville。所有数字来自当天的 trials JSON / STATUS.md / 页面，证据等级标在括号里。
1 · 现在用的模型
| 用途 | 模型（HF: Suyang99/…） | 存档 | 数据集（HF: xlerobot-team/…） | 成绩（k/n 按配置分组） |
|---|
| 左臂抓杯 | xlerobot-smolvla-left-sd-p0.2-b16-u20000 | 020000 | xlerobot-left-pick-cup-sep3-clean-20260905 54 集 / 13,693 帧 | 09-09：8/12（60 s）、6/7（40 s）；09-11 早：3/3（25 s，蓝杯） |
| 右臂抓杯（今天新上） | xlerobot-smolvla-right-0826sep8-p0.2-b16-u20000 | 020000 | xlerobot-right-pick-cup-0826sep8-full-20260910 73 集 / 32,216 帧 | 09-11：6/6（单只训练黄杯）；探索轮 3/8、1/5（分布外杯/双杯，不进正式分母） |
| 右臂旧模型（作废） | xlerobot-smolvla-right-sd-p0.2-b16-u20000 | 020000 | xlerobot-right-pick-cup-20260826-0042 40 集 | 0/14 |
| 放杯 | 不是模型：configs/place_pose.json（10 集示范的关节角中位数） | — | xlerobot-left-place-cup-sep7-* | 真机跑通 |
| 倒酒 | 不是模型：scripts/pour_cup.py + configs/pour_pose.json | — | 见第 3 节 | 09-09 真机跑通一次 |
本机路径：05-training/<模型名>/checkpoints/020000/pretrained_model。两个 SmolVLA 的 train_config 除数据集名外逐项相同（145 项）。
开跑配方（右臂，今天 6/6 用的），改任一项就是另一组 k/n：
~/miniconda3/envs/train310/bin/python run_policy_trials.py \
--panel --send --trials 8 --seconds 60 \
--chunk-rate 1 --speed 1.0 --keep-torque --trace \
--model 05-training/xlerobot-smolvla-right-0826sep8-p0.2-b16-u20000/checkpoints/020000/pretrained_model
指令自动从数据集读："Pick up the cup with the right arm."。头必须在 cup-grasp-v0（pan 2003 / tilt 2617），跑前先归位左臂（arm_home.py --arm left --go），再起 job。
2 · 为了让右臂动起来，改了什么
按时间，每条写「改了什么 → 为什么 → 证据」。
数据（决定性的一条）
- 0826 那 40 集杯位覆盖极不均衡（recording-guide-2026-09-08：某档 16 集、某档 1 集，稀疏档夹爪从没闭过），模型学的是节奏不是视觉（闭爪时刻 8.45 s ± 1.42 s，和杯位相关 r 只有 0.16～0.26）。
- sep2 那批头抬着（tilt 2261，看到对面椅子），七个数据集整批不可用（手册 V-07）。
- 09-08 按录制指南补录 sep8（头锁 cup-grasp-v0、杯位 P1～P5 轮换），去掉深度旁路后与 0826 合并成
0826sep8-full 73 集。lerobot 不支持多数据集，合并用 aggregate_datasets.py。
- 09-10 在 4090 上重训（state dropout p=0.2、batch 16、20k 步），09-11 拉回 020000 存档。
训练配方
- state dropout p0.2：之前发现 state↔action 相关 0.99，模型靠状态捷径不看图（记忆 policy-vision-collapse-high-E）。left-sd / right-sd / right-0826sep8 三个模型都是这个配方。
运行时（run_policy_trials.py），右臂能不能动主要靠这几条
- 恒定维钉住：右臂数据集里左臂 6 维 + 头部等 8 维在训练时恒定，lerobot 归一化会把真机读数放大 1e8 倍；喂策略前钉回训练均值（判据 min==max）。没有这条，右臂模型一上机就飞。
- idle_lock：非工作臂输出维钉在当前姿态，否则右臂会甩 29°。
- --chunk-rate 1：推理停顿 46%→22%，横向增益 0.6→0.85，k/n 0/14→3/7（chunk1-trials-2026-09-10）。
- --keep-torque：不加的话 connect 后头掉到 tilt 1901 对着墙，整轮作废。
- 安全层地板按爪俯仰角算（z_floor/y_floor）：固定 z_min 会拒掉 96.6% 的示范帧，改后 0.49%。
- max_reach_m 0.235→0.2459：sep8 伸得更远，0.235 会让 13/73 集被连续 HELD 掐掉整轮。
- 两臂都归位：右臂跑测时左臂若留在上次的姿态，idle_lock 会放弃钉住；今天流程是先
arm_home --arm left --go 再起 job。
- --home-left/right 符号 09-11 更正：基座系 +x 是机器人的右，之前反的。
- 相机竞态（09-11 下午修）：按确认那一秒 8770 可能正在给预览页开相机，试跑 4 次重连全撞上。现在试跑确认后等 10 s 再连；服务端确认时先设让位窗口再关预览流两次。手册 P-13。
- 过热保护 XLeGuard（警告 50 °C / 停机 60 °C）已接进 run_policy_trials；今天右臂峰温 38～48 °C。
补充细节（改进史那路调查核出来的数字）
- 09-08 诊断 0/14 时的真病因：示范在够取行程 99.8% 处闭爪，机器人在 52% 处就闭；横向增益只有 0.59～0.63。补录后 P1 档 5→11 集、P4 档 3→11 集，各档集数的变异系数 0.84→0.37。
- 三路相机都保留（含右腕）；消融实验（12 集）：右腕相机贡献 73% 的闭爪信号，头相机主导横向。
- 恒定维判据从 std==0 改成 min==max（float32 对非零常数会算出假 std，head_motor_2 曾被喂成 z=−240），钉住 8 维，对 32,216 帧训练数据 0 次触发。
- idle_lock 改成每集重新评估（原来只在启动时评估一次；热机时右腕下垂 30° 就放弃钉住，之后每集把空闲臂命令到 0°）。
- 训练：RunPod 4090，torch 2.8.0+cu128，torchcodec 钉 0.7.0（0.11.1 会静默退回 pyav），20k 步一小时出头。
- --grip-gate 保持关闭：时间闸在 7.4 s 放开而真实闭爪在 27.7 s，维度不对。
操作规则（不是代码，但决定成绩能不能算）
- 头对到预设后看图确认（P-03）；k/n 六项一致才合并（V-01）；每集操作者摆好杯、报杯位再 start（今天一集 void 就是没等摆杯自动 start）；一台机器只留一个 Claude 会话驱动（两份模型挤爆 8 GB，今天撞车两次）。
今天右臂的行为观察（探索轮）
- 放着绿杯只抓训练黄杯；蓝杯 1/2、粉杯 0/1、绿易拉罐 0/1：到位后不闭爪。
- 桌上两杯靠近（P2+P3）或左爪里夹着一只杯：往两杯中间空伸，连续出现 3 次。要它选杯，要么补录两杯数据（指令分 yellow/blue，杯位互换，≥50 集，可与单杯集合训），要么用零训练的
cup_mask.py（把非目标杯从头相机画面抹掉，--mask-cup blue，离线验证过、实机未跑）。
3 · 倒酒是怎么实现的
倒酒今天 = 纯脚本关节角序列，不是模型，也没接进 bartender 状态机。
- 入口：
03-software/scripts/pour_cup.py，姿态在 03-software/configs/pour_pose.json。文档串开头写明「确定性脚本，不用策略」：起点固定（归位）、终点固定（一个倒水位）、指令固定，所以轨迹固定；实测端点一致性左 1.33 cm / 右 1.09 cm。
- 分工：左臂端接水杯（start → reach → grasp → carry，把杯子从桌上拿到倒水位端住，wrist_roll 全程只转 8°）；右臂端罐子倾倒（start → arrive → pour → back，pour 就是 wrist_roll 3.2° → 76.7°，行程 85°，夹爪全程不动）。接水杯的位置隐含在左臂 carry 姿态里，不感知。
- 姿态来源：左臂 6 集（
xlerobot-left-pick-cup-pour-20260909-2119），右臂 8 集（xlerobot-right-place-cup-pour-20260909-2133），每段关键姿态取集内中位数，pour_cup.py --recompute-all 重算。两个「真正倒酒」的数据集（bimanual-pour、right-pick-cup-pouring）是 0 集空壳。
- 执行：每段用
arm_home.home() 按墙钟插值（这是唯一在负载下验证过的运动原语），速率 carry 10°/s、pour 6°/s（慢，防溅）。任一段没到位就中止。结束保持力矩（两臂都还端着东西）。左臂若已握杯用 --from-grasp 跳过 reach/grasp，否则会张开把杯子丢了。
- 安全：只经过 arm_home 的到位检查，没有逐点过遥操作安全包络。
- 状态：09-09 真机跑通（录像
05-training/onsite/pour-run-20260909/）。已知问题：左肩 shoulder_lift 在倒水位差 3.9° 到不了（负载堵转，MSG-ORGANISERS-2026-09-09-SERVO-STALL），建议把倒水位往身体收。Demo Day 文档把带液体的倒酒列为 out of scope。
- 接进管线要做的事：state_machine.py 现在只有 8 个状态（IDLE / GREETING / CONFIRMING / PICKING / HANDING / PLACING_DOWN / APOLOGISE / GOING_HOME），没有 POUR；robot_effectors.py 接口是 arm_home / pick / present / place_down / set_context / alarm，没有 pour()。要加一个 POUR 态和一个
pour() effector（先左臂 --from-grasp，再右臂 --go）。搬场地后 pour_pose / place_pose 的关节角都要重教（桌高变了关节角是绝对量）。
4 · 相关文件一览
- 试跑：
03-software/scripts/run_policy_trials.py；面板 8770 /api/run + /api/trials/control。
- 归位：
03-software/scripts/arm_home.py；头：head_aim.py --preset cup-grasp-v0。
- 结果：
05-training/trials/trials-2026*.json、cup-positions.json（杯位/颜色/操作者观察）、STATUS.md（gen_status.py 生成，别手改）。
- 报告页：Space
suyang99-xlerobot-build-reports.static.hf.space（cup-colour-mask、venue-move、train-right-0826sep8、chunk1-trials、handbook）。
← reports index
03-software/brain · handover note · 2026-09-11
Handover note: which models we run · how the right arm was made to work · how pouring is implemented
Written 2026-09-11 16:30 for Ville (integration). Every number comes from that day's trials JSON / STATUS.md / report pages; evidence levels are stated in brackets.
1 · Models in use
| Purpose | Model (HF: Suyang99/…) | Checkpoint | Dataset (HF: xlerobot-team/…) | Result (k/n per configuration) |
|---|
| Left-arm cup grasp | xlerobot-smolvla-left-sd-p0.2-b16-u20000 | 020000 | xlerobot-left-pick-cup-sep3-clean-20260905, 54 ep / 13,693 frames | 09-09: 8/12 (60 s), 6/7 (40 s); 09-11 morning: 3/3 (25 s, blue cup) |
| Right-arm cup grasp (new today) | xlerobot-smolvla-right-0826sep8-p0.2-b16-u20000 | 020000 | xlerobot-right-pick-cup-0826sep8-full-20260910, 73 ep / 32,216 frames | 09-11: 6/6 (single training cup); exploratory rounds 3/8 and 1/5 (out-of-distribution cups / two cups, not in the formal denominator) |
| Old right-arm model (retired) | xlerobot-smolvla-right-sd-p0.2-b16-u20000 | 020000 | xlerobot-right-pick-cup-20260826-0042, 40 ep | 0/14 |
| Place | not a model: configs/place_pose.json (median joint angles of 10 demos) | — | xlerobot-left-place-cup-sep7-* | ran on hardware |
| Pour | not a model: scripts/pour_cup.py + configs/pour_pose.json | — | see section 3 | one hardware run, 09-09 |
Local path: 05-training/<model>/checkpoints/020000/pretrained_model. The two SmolVLA train_configs are identical field by field (145 fields) except the dataset name.
Run recipe (right arm, the one that gave 6/6 today) — change any item and it is another k/n group:
~/miniconda3/envs/train310/bin/python run_policy_trials.py \
--panel --send --trials 8 --seconds 60 \
--chunk-rate 1 --speed 1.0 --keep-torque --trace \
--model 05-training/xlerobot-smolvla-right-0826sep8-p0.2-b16-u20000/checkpoints/020000/pretrained_model
The instruction is read from the dataset automatically: "Pick up the cup with the right arm." The head must be at cup-grasp-v0 (pan 2003 / tilt 2617); home the left arm first (arm_home.py --arm left --go), then launch the job.
2 · What was changed to make the right arm move
Chronological; each item is "what → why → evidence".
Data (the decisive one)
- The 40 episodes of 0826 had badly unbalanced cup-position coverage (recording-guide-2026-09-08: 16 episodes in one band, 1 in another; the gripper never closed in the sparse band). The model had learned tempo, not vision (closure at 8.45 s ± 1.42 s; correlation with cup position only r = 0.16–0.26).
- The sep2 batch was recorded with the head raised (tilt 2261, looking at the chair opposite); all seven datasets unusable (handbook V-07).
- On 09-08 sep8 was re-recorded per the recording guide (head locked at cup-grasp-v0, cup bands P1–P5 rotated); with the depth sidecar stripped it was merged with 0826 into
0826sep8-full, 73 episodes. lerobot does not support multiple datasets, so aggregate_datasets.py merges first.
- Retrained on the 4090 on 09-10 (state dropout p=0.2, batch 16, 20k steps); the 020000 checkpoint was pulled on 09-11.
Training recipe
- State dropout p0.2: state↔action correlation had been 0.99 — the policy was taking a state shortcut and not looking at images (memory policy-vision-collapse-high-E). left-sd, right-sd and right-0826sep8 all use this recipe.
Runtime (run_policy_trials.py) — the items the right arm depends on
- Constant-dimension pin: in the right-arm dataset the 6 left-arm dims plus the head (8 dims) are constant; lerobot normalisation would amplify live readings 1e8×; they are pinned to the training mean before the policy sees them (criterion min==max). Without this the right-arm model flies off the moment it is on hardware.
- idle_lock: output dims of the non-working arm are pinned at the current pose; otherwise the right arm swings ~29°.
- --chunk-rate 1: inference pauses 46%→22%, lateral gain 0.6→0.85, k/n 0/14→3/7 (chunk1-trials-2026-09-10).
- --keep-torque: without it the head drops to tilt 1901 after connect and faces the wall; whole run void.
- Pitch-compensated safety floor (z_floor/y_floor): a fixed z_min rejected 96.6% of demo frames; now 0.49%.
- max_reach_m 0.235→0.2459: sep8 reaches further; at 0.235, 13 of 73 episodes would be killed by 15 consecutive HELD frames.
- Home both arms: if the left arm is left where the last test ended, idle_lock gives up pinning it; today's flow is
arm_home --arm left --go before launching.
- --home-left/right sign corrected 09-11: base-frame +x is the robot's right; it was inverted.
- Camera race (fixed 09-11 afternoon): at the second confirm is pressed, 8770 may be opening a camera for the preview page and all four reconnects collide. Now the trial waits 10 s after confirm before connecting; the server sets the hold window first and stops preview streams twice. Handbook P-13.
- Thermal guard XLeGuard (warn 50 °C / stop 60 °C) is wired into run_policy_trials; right-arm peak today 38–48 °C.
Further details (verified by the improvement-history survey)
- The real cause behind 0/14 on 09-08: demos close the gripper at 99.8% of the reach; the robot closed at 52%. Lateral gain was only 0.59–0.63. After re-recording, band P1 went 5→11 episodes, P4 3→11; the coefficient of variation of band counts fell 0.84→0.37.
- All three cameras kept (including the right wrist); an ablation over 12 episodes showed the right wrist camera carries 73% of the closure signal and the head camera drives lateral motion.
- The constant-dim criterion moved from std==0 to min==max (float32 gives a fake std for non-zero constants; head_motor_2 had been fed as z=−240); 8 dims pinned, 0 triggers over 32,216 training frames.
- idle_lock is now re-evaluated per trial (previously once at start-up; on a warm session the right wrist sagged 30°, the check gave up, and every trial commanded the idle arm to 0°).
- Training: RunPod 4090, torch 2.8.0+cu128, torchcodec pinned to 0.7.0 (0.11.1 silently falls back to pyav), 20k steps in a little over an hour.
- --grip-gate stays off: the time gate released at 7.4 s while real closure was at 27.7 s; wrong dimension.
Operating rules (not code, but they decide whether a result counts)
- Aim the head to the preset and confirm by image (P-03); merge k/n only when all six configuration items match (V-01); the operator places the cup and reports the band before each start (today's one void was an automatic start before the cup was placed); one Claude session drives the machine at a time (two models blow the 8 GB; two collisions today).
Right-arm behaviour observed today (exploratory rounds)
- With a green cup present it only grasps the training yellow cup; blue cup 1/2, pink 0/1, green can 0/1: it reaches the position but does not close.
- Two cups close together (P2+P3), or a cup held in the left gripper: it reaches between them, three times in a row. To make it choose, either record two-cup data (yellow/blue instructions, positions swapped, ≥50 episodes, can be trained together with the single-cup set) or use the zero-training
cup_mask.py (paints the non-target cup out of the head image, --mask-cup blue; verified offline, not yet run on hardware).
3 · How pouring is implemented
Pouring today = a purely scripted joint-angle sequence; not a model, and not wired into the bartender state machine.
- Entry point
03-software/scripts/pour_cup.py, poses in 03-software/configs/pour_pose.json. The docstring says "deterministic script, no policy": fixed start (home), fixed end (one pour position), fixed instruction, therefore a fixed trajectory; measured endpoint consistency left 1.33 cm / right 1.09 cm.
- Roles: the left arm holds the receiving cup (start → reach → grasp → carry: takes the cup from the table to the pour position and holds it; wrist_roll turns only 8° throughout); the right arm holds the can and tips it (start → arrive → pour → back; pour is wrist_roll 3.2° → 76.7°, an 85° travel, gripper never moves). The receiving cup's location is implicit in the left carry pose, not sensed.
- Provenance: left from 6 episodes (
xlerobot-left-pick-cup-pour-20260909-2119), right from 8 (xlerobot-right-place-cup-pour-20260909-2133); each key pose is the per-episode median, recomputed by pour_cup.py --recompute-all. The two "real pour" datasets (bimanual-pour, right-pick-cup-pouring) are empty shells with 0 episodes.
- Execution: each stage interpolates with
arm_home.home() on the wall clock (the only motion primitive verified under load); rates carry 10°/s, pour 6°/s (slow, no splashing). Any stage that does not arrive aborts the sequence. Torque is kept at the end (both arms are still holding things). If the left arm already holds the cup use --from-grasp to skip reach/grasp, otherwise it opens and drops the cup.
- Safety: only arm_home's arrival check; no per-waypoint check against the teleop safety envelope.
- Status: ran on hardware on 09-09 (video in
05-training/onsite/pour-run-20260909/). Known issue: the left shoulder_lift stops 3.9° short at the pour position (stall under load, MSG-ORGANISERS-2026-09-09-SERVO-STALL); suggested fix is moving the pour position closer to the body. The Demo Day document lists pouring with liquid as out of scope.
- To wire it into the pipeline: state_machine.py has 8 states (IDLE / GREETING / CONFIRMING / PICKING / HANDING / PLACING_DOWN / APOLOGISE / GOING_HOME), no POUR; robot_effectors.py exposes arm_home / pick / present / place_down / set_context / alarm, no pour(). Add a POUR state and a
pour() effector (left arm first with --from-grasp, then right with --go). After a venue move both pour_pose and place_pose must be re-taught (joint angles are absolute; the table height changes them).
4 · Files
- Trials:
03-software/scripts/run_policy_trials.py; panel 8770 /api/run + /api/trials/control.
- Homing:
03-software/scripts/arm_home.py; head: head_aim.py --preset cup-grasp-v0.
- Results:
05-training/trials/trials-2026*.json, cup-positions.json (cup position / colour / operator notes), STATUS.md (generated by gen_status.py; do not edit by hand).
- Report pages: Space
suyang99-xlerobot-build-reports.static.hf.space (cup-colour-mask, venue-move, train-right-0826sep8, chunk1-trials, handbook).