# Handover · 2026-09-09 night → 09-10 early morning · Freeze root-caused, pour hardcoded, Demo Day reports shipped > **One line: the "whole machine freezes the moment leader arm starts" is a USB 2.0 bandwidth problem. > Two wrist cameras negotiating YUYV ate the 480 Mbit/s bus and starved the keyboard and mouse on the > same hub. Forcing MJPG gives a measured 30.0 fps — it had been running at 27 and silently dropping > frames the whole time.** > > Also built: `pour_cup.py` (hardcoded pour for both arms, verified on hardware), `snap_record.py` > (three-camera recording), five Demo Day report pages and a 1:47 demo reel. > > ★ I got the question "which cup was used in which trial" **wrong four times tonight**, every time by > inferring a fact from fields I could query instead of looking at the images that actually held it. > It was settled only by sending all 28 init keyframes to the operator who was in the room. > **Lesson: a fact that lives in a picture cannot be recovered from a statistic.** ★ --- ## ⚠️ 1. Read before touching anything: current robot state | | State | |---|---| | **Both arms** | At home pose, **torque DISABLED** (arms are limp) | | **leader-follower service** | **Stopped** (`systemctl --user start leader-follower` to bring it back) | | **8770 dashboard / camera preview** | Working normally | | **`/dev/shm/xlerobot_headcam.jpg`** | **Stale** — that frame is only published while LF runs | **Before commanding any motion, restore torque safely**: read present position → write it as Goal_Position → then enable torque. Calling `enable_torque` directly makes the servo snap toward the goal it remembers from before power-down, which is dangerous with anything in the gripper. --- ## 2. The freeze, root-caused (fixed, NOT committed) **Topology** (`lsusb -t`, Bus 001 shares 480 Mbit/s): the keyboard `1-2.3.4`, the mouse `1-2.3.1` and **both wrist cameras `1-2.3.2/3`** hang off the same Genesys hub; both servo serial buses sit on its parent hub. `leader_follower_server.py` built its `OpenCVCameraConfig` **without passing fourcc**, so it defaulted to `None` and OpenCV negotiated **YUYV at 147 Mbit/s per camera**. Two cameras = 294 Mbit/s, which saturates the 480 Mbit/s bus and starves the 1.5 Mbit/s keyboard sharing that hub. **Measured, both cameras at 640×480, target 30 fps:** | Format | cam0 | cam1 | |---|---|---| | MJPG | **30.6** | **30.0** | | YUYV | 27.2 | 26.4 | **Implication: every dataset recorded before this fix was dropping frames (~27/30)**, not just freezing. The change is `fourcc=None if name == "head" else "MJPG"` — the head camera is the Orbbec on USB3 (Bus 002) and is not on the contended bus. Verified in the 21:19 recording: all 6 episodes at exactly 30.00 fps, zero gaps. **Remote operators are unaffected** — the NIC is PCIe, and both 8770 and 8781 are ThreadingHTTPServer. The symptom only appears to someone sitting at the machine. **Do not pull power when it looks frozen** — the machine is alive and SSH still answers. --- ## 3. Two bugs found and NOT fixed ### 3.1 E-STOP does nothing while leader-follower is running ★SAFETY★ `robot_server.estop()` only SIGTERMs jobs registered in `_jobs`, but **`lf_start()` shells out to `systemctl --user start`, so LF is not in `_jobs`** → LF survives and keeps both serial buses → the follow-up `check_arms.py --release` is refused by `bus_guard.require_free()` (only `robot_server.py` is in `COOPERATIVE`, so LF classifies as `unknown` = exclusive) → `sys.exit(2)` → estop reports `torque release FAILED` and **nothing is released**. LF running means a human is standing at the robot with hands on the arms — precisely the situation the e-stop exists for. `estop()`'s own docstring records that this trap was fixed once ("stop the movers FIRST"), but the fix only covered `_jobs`; LF was added 2026-08-25 as a resident systemd service and was never added to that step. **Fix**: stop `leader-follower.service` inside `estop()` before running the release. **Second, latent issue**: `check_arms.release_all()` does `continue` when a port will not open but still `return 0` — **that path reports success while releasing nothing**. It is currently unreachable because `require_free` refuses first. The same class of hole applies to `place_cup.py` (launched by place_server). ### 3.2 `arm_home.py --up` moves the arm DOWN Both `tip_in_robot` and `tip_in_base` in `arm_fk_full` have **z pointing downward** (larger z = lower): | shoulder_lift | Physically | tip_in_robot z | |---|---|---| | −31 (home, raised) | high | 0.6461 | | +53 (drooped) | low | 1.0978 | `offset_vector()` turns `--up 5` into `dz=+0.05`, which added to `tip_in_base` moves the tip **down**. Following that intuition I called `shift_pose(...,(0,0,+0.03))` intending to raise the left arm and instead pressed it 5 cm toward the table — while my own numbers still claimed "the left arm is 6.1 cm higher than the right". The operator caught it from the camera view, not the maths. **To actually raise, dz must be negative.** See memory `project-fk-z-axis-down`. --- ## 4. Left shoulder-lift servo: a new failure mode (organisers notified) **09-08 was thermal derating at 66 °C. 09-09 it stalled at only 40 °C. These are not the same fault.** Commanding the left arm to the pour pose: `goal 2209 / present 2297`. Re-issuing the goal moved it 43 ticks, then it **stopped 45 ticks (3.9°) short and stayed there**. The same servo reaches the home pose with **0.0° error**. The other five joints all landed within 0.6°. `Status=0`, `Torque_Limit=1000`, and the goal sits well inside the position limits (1440–3866). → **Load-dependent, not thermal, and not a limit clamp.** **This conflicts directly with the demo**: the pour sequence needs the left arm to *hold* an extended pose — exactly the "sustained high-load held pose" the 09-08 mitigation told us to avoid. Message drafted at `00-admin/MSG-ORGANISERS-2026-09-09-SERVO-STALL.md`. ⚠️ `Present_Load` and `Present_Current` read **0 on all six motors**, including joints visibly holding the arm against gravity. **Those two readings are wrong, not informative** — do not cite them as evidence of "no torque". --- ## 5. What was built | File | Purpose | |---|---| | `03-software/scripts/pour_cup.py` | Hardcoded pour for both arms. `--recompute-all` extracts poses from the datasets into `configs/pour_pose.json`; `--arm left/right --go` executes; `--selftest` touches no hardware | | `03-software/scripts/snap_record.py` | Records three camera streams by polling 8770's snapshot endpoint. **Never takes a camera.** Measured ~2 fps — that is the 8770 server's ceiling, concurrency does not help | | `03-software/configs/pour_pose.json` | Key poses extracted from 6 left-arm and 8 right-arm episodes | **Three design decisions in `pour_cup.py`** (all documented in the file header): 1. **Extract key poses, do not replay trajectories.** A DTW alignment check showed the mid-trajectory spread is a **real path difference**, not a timing difference (aligning narrows it by only −1% to 12%), so averaging the episodes is a dead end. But converted to task space those tens of degrees amount to just over 1 cm — the joint errors cancel each other at the fingertip. 2. **All motion goes through `arm_home.home()`** — the only motion primitive verified under load. 3. **The left arm's grasp closes in place.** The first version took `reach` from the frame where the gripper opening peaked, assuming "gripper fully open = at the cup". On hardware the arm stopped **13.4 cm short**: the operator opens the gripper *while still moving toward the cup*, so the peak is at t≈0.25 while closure happens at t≈0.5. The fix is for `reach` and `grasp` to share the arm pose from the **closure** frame and differ only in gripper value. **One more trap at teardown**: `robot.disconnect()` releases torque by default (`place_cup.py` even prints "torque will be released"). In the pour sequence **the left arm is still holding the cup when it finishes**, so copying that would drop it. Changed to `bus.disconnect(disable_torque=False)`. --- ## 6. The 09-09 generalization trials: 28 runs, 4 objects Object labelling comes from **the operator who was in the room** — the trial JSON has **no field for which object was used**, so it can only be read off the keyframes. | Object | Trials | n | Success | Failure | Void | Rate | |---|---|---|---|---|---|---| | **Yellow cup (in training data)** | 1–13 | 13 | 9 | 1 | 3 | **90%** | | Blue cup (unseen) | 14–18 | 5 | 2 | 1 | 2 | 67% | | Blue can (unseen) | 19–24 | 6 | 3 | 2 | 1 | 60% | | Red can (unseen) | 25–28 | 4 | 3 | 1 | 0 | 75% | | Total | | 28 | 17 | 5 | 6 | 77% | **Two numbers safe to quote externally**: same object **9/10 = 90%** (matches what was reported to Erkka); unseen objects combined **8/12 = 67%** (the 80% that was quoted sits somewhat above this). Always state them as "n of m" — the organiser explicitly warned against unsupported success-rate claims. **★ Change worth making: add a `target_object` field to the trial record. ★** See memory `project-trial-objects-0909`. **On the failures**: in failed trials the policy demanded single-frame jumps of **25–41°** on elbow and shoulder-lift which the rate limiter trimmed; successful trials show almost none (3.9°). The clipped landing points are *scattered*, so this is the **step cap** (stage 5), not the absolute clamp. That means the policy output was unstable, not that the safety layer misjudged — **raising the step cap would only let the arm swing at 1230°/s**. Useful by-product: **large clipping is an early failure signal**, so a run can be aborted and retried instead of waiting out the 25 s time cap. --- ## 7. Demo Day reports (published to the Space, also copied to the 8790 download directory) ``` https://suyang99-xlerobot-build-reports.static.hf.space/ demo-day-2026-09-10.html prizes · schedule · who does what odds-2026-09-09.html odds of placing · actions ranked by gain ÷ cost evidence-2026-09-09.html evidence archive (embeds the reel and the position chart) marketing-evidence.html marketing evidence (3 Xiaohongshu posts logged, view counts to fill) demo-reel-2026-09-09.mp4 1:47 reel ``` **Prizes confirmed**: top three (€7,000 pool) plus the Marketing Award (hardware, under €1,000 in value). **There are no other categories** — this closes the open item both source documents carried. Two things score but are not prize categories: potential-customer feedback (bonus points) and the Teleop Knife Fight (experimental side event). **Reel structure** (cut for the 2-minute technical demo slot): 11 grasps at 4× speed (70 s, 2–3 most widely separated positions per object) → 1/9/16/25 grid montage (28 s) → position distribution chart (9 s). **⚠️ The pour sequence is formally out of scope for this Demo Day** — the awards document states `HOLD/POUR with liquid is out of scope` and `empty-cup manipulation is the safer stage plan`. The whole of 09-09 went into something outside that scope, and it is what pushed the left shoulder-lift servo to its limit. --- ## 8. ⚠️ Uncommitted changes (a handover will miss these) ``` M 03-software/scripts/leader_follower_server.py ← the MJPG fix, the most important thing today ?? 03-software/scripts/pour_cup.py ?? 03-software/scripts/snap_record.py ?? 03-software/configs/pour_pose.json ?? 04-reports/space-2026-08-21/{demo-day,odds,evidence,marketing-evidence}*.html + assets ?? 00-admin/MSG-ORGANISERS-2026-09-09-SERVO-STALL.md D 05-training/xlerobot-act-20260820-step20000/* ← removed during model cleanup, git-tracked M 04-reports/space-2026-08-21/*.html ← 8K-boundary mojibake repairs ``` **If `leader_follower_server.py` is not committed, whoever picks this up will not see the MJPG fix.** Also: nine models were deleted (disk 88% → 79%), each verified to exist on HuggingFace with weights before removal; the list is in the scratchpad as `deleted-models-20260909.txt`. The right-arm dataset `xlerobot-right-place-cup-pour-20260909-2133` has been uploaded — it had been blocked by the verify gate because `images/` still held 1740 PNGs from ep8, an unsaved take. --- ## 9. Open items 1. **Commit and push** the changes above to `china-console` 2. **Fix the E-STOP bug** — LF is stopped and the buses are free right now, which is the safe window, and the fix can actually be verified afterwards 3. **Fill in the view counts and screenshots** on the marketing evidence page — the only entry requirement for the Marketing Award 4. **Talk to one plausible customer for 15 minutes** — the cheapest scoring opportunity left 5. Add a `target_object` field to the trial record 6. Move the pour position closer to the body to shorten the moment arm (promised to the organisers). Note the left arm is already near its reach limit in that direction: a requested −5 cm in y produced only −1.5 cm of actual movement, and the two arms can close from 33.7 cm to at best 32.5 cm --- **Related memories**: `project-usb-bus-freeze` · `project-estop-bug` · `project-fk-z-axis-down` · `project-trial-objects-0909`