Brief — Face recognition & audience interaction

For: Ville (remote, currently on exchange in Taiwan) · From: the China side · 2026-09-04

You have face tracking running locally. This brief says what already exists in the repo, where your work fits, and three questions we need answered before we can plan around it.

It is deliberately short on instructions and long on context — you know your code, we don't, and the worst outcome here is two people building the same thing twice.

1. What we saw

You shared a 38-second screen recording (simplescreenrecorder-2026-09-03_19.04.42). Read directly off the frames:

8.4 fps    detect 107.7 ms    faces 1
brightness 136/255

• A green bounding box tracking a face, plus landmark points around the head
• A live latency/FPS readout
• A brightness readout — good call. Lighting is a real variable in a bar, and nobody else on this project is measuring it.

We are not assuming anything beyond that. Everything below is either sourced from the repository or explicitly marked as a question.

2. What is already in the repo (so you don't rebuild it)

Committed 2026-08-14, present on both main and china-console:

face_follow.py's stated design goals are worth reading before you change anything — they were written against exactly the bartender use case:

Measured constants (from the source):

Detector: OpenCV's bundled Haar cascade — no model download needed. The file header explicitly says the detector can be swapped for MediaPipe / YOLO-face without touching the head controller. That layering is good; please keep it.

3. ⚠ Detector convergence — decided 2026-09-04

There were two face detectors in this repo. The decision has now been made: YuNet.

Good news: you already chose YuNet too. Your cup_test.py uses models/yunet.onnx, which is the same file and the same decision we reached independently. That question is closed — we converged without needing to negotiate.

The Haar path in face_follow.py is the one that goes; it was never a working alternative.

4. The three questions we need answered

We cannot plan around your work until we know these. Short answers are fine.

Q1 · What machine was that recording made on?

This is the one that matters most. 107.7 ms means completely different things depending on the answer:

The robot runs on a Jetson Orin Nano Super, 7.4 GiB shared CPU/GPU memory.

The compute budget, measured on that board on 2026-09-04:

So the question for your detector is not "is it fast" in the abstract — it is whether it fits in 337 ms, or whether it can run at a lower rate than the policy. YuNet fits comfortably. A 107.7 ms detector also fits, if that number is from this board.

4.1 ★ Better than answering Q1: measure it on the actual board ★

We can put you on the robot's Tailscale network. Tailscale is a mesh VPN, it works fine from Taiwan, and it would give you a direct route to the Jetson — no port forwarding, no tunnelling, nothing to configure on your side beyond installing the client and accepting an invite.

With that you could:

If that is useful, say so and we will send the invite. It is probably worth it even if you only use it once, because "does it run fast enough on the actual hardware" is a question that will keep coming up, and right now nobody can answer it without you.

Q2 · Which detector / model? — ✅ answered

From your message: models/efficientdet_lite0_int8.tflite for cups and models/yunet.onnx for faces, both already inside 03-software/conversational_ai/models/. Nothing to download. See §9 — we have measured that cup model and it has a problem.

Q3 · Does it do identity, or detection + landmarks? — ✅ answered

YuNet is a detector, so this is detection (+ landmarks), not identity. That means the gap named in §1.2 — "recognise which customer this is" — is still open. Not a criticism: detection is what the greeting stage actually needs. It just means we should not plan around identity yet.

Meaning: can it answer "this is the same person as before" or "this is Alice" — as opposed to "there is a face at (x, y)".

Why this specific question decides everything:

• If detection + landmarks → we already have two of those. Yours may still be better (faster, more robust, better landmarks), but then it is a replacement decision, not a new capability.
• If identity → that is a genuine gap. Our face_perception.py explicitly does not do it. Identity would enable things nothing on the market does: greeting a returning customer, remembering a preference, addressing the right person when several are at the bar.

5. Where this fits in the product

We are building toward a four-stage service. Gaze appears twice — at the start and at the end — and is deliberately frozen in between:

Stage 1 is where your work lands first, and it is the easiest win in the whole project: the arm is still, so it conflicts with nothing, and "the robot looks up at you and talks to you" is already something none of the commercial bartender robots do.

Why the head must be frozen during stage 2 — this is not a preference

Measured across all 50 recorded demonstration episodes (19,453 frames):

head_1    σ = 0.0000    range [-4.308, -4.308]
head_2    σ = 0.0000    range [49.846, 49.846]

The head never moved in any demonstration. The manipulation policy has never seen a moving head camera. Moving it during manipulation puts the input outside the distribution it was trained on. Second, independent reason: the depth camera is mounted on that moving head, so its extrinsic calibration is only valid for one head pose.

6. Two constraints you will hit

6.1 The serial bus is exclusive

face_follow.py opens the left arm bus directly to drive servos 7/8. The manipulation policy owns that same bus while it runs, and overwrites the head every frame. They cannot run simultaneously today.

This is not a theoretical concern. From bus_guard.py's own header:

Our planned resolution: head control stays inside the policy process, and only the target source changes per stage (locked constant vs face-driven). One owner of the bus, no handover. If you see a better way, say so before we build it.

6.2 The safety layer has no concept of a person

grep -in "human|person" policy_safety.py returns three hits, and none of them is "a person was perceived". They are: the per-joint step cap ("the largest single-tick delta any human ever commanded"), a SAFETY.md reference to a human on the stop button, and a comment about an incident.

So today, human safety = a fixed geometric envelope + someone holding the e-stop. "Push the glass toward the customer" means deliberately reaching toward a person, which is exactly what that envelope is shaped to prevent. That needs a separate design, and your perception output is the input it would need: bearing, distance, confidence, and how long a loss counts as "nobody there".

7. What would help us most, in order

1. Answer Q1–Q3 (above). Two lines each is enough — or take the Tailscale route in §4.1 and measure Q1 directly, which is strictly better.
2. Push the code to a branch so we can read it: ``bash git checkout -b feature/face-recognition git add <your files> git commit -m "Face tracking (WIP, not wired to the arm)" git push -u origin feature/face-recognition ` ⚠ Do not commit model weights. A push failed on 2026-09-03 because of a 109 MB file — GitHub hard-rejects anything over 100 MB, and history rewriting tools are blocked in this repo. dlib's shape_predictor_68_face_landmarks.dat alone is ~100 MB. For weights: upload to HuggingFace (directly reachable from where you are), or just tell us the model name so we fetch it ourselves. If you get Tailscale access, the upload page at :8790` takes files up to 512 MB and puts them outside the repo, which is safer anyway.
3. If you have an opinion on §3 (which of the three detectors should survive), tell us. You are the only one who has run one of them on real video.

8. What we are NOT asking you to do

• Don't wire anything into the arm yet. The bus conflict in §6.1 is unresolved.
• Don't modify policy_safety.py. Its floor constraints were re-derived on 2026-09-03 (rejected-frame rate went 96.6% → 0.49%) and we are keeping it stable this week.
• Don't worry about matching our code style yet. We want to see what you have first.

9. ★ What we measured on the cup detector — this is the part to read ★

We evaluated cup detectors offline on 36 frames sampled from three recorded datasets, then cropped and identified every single box by eye. Independently of your work.

EfficientDet-Lite0-int8 produced no boxes at all on our recorded head-camera frames.

And you already knew this — the help text of your own --yolo flag says:

Two independent measurements agreeing is worth more than either alone. So: the default in cup_test.py is the model that does not work on our data. Consider making --yolo the default, or at least printing a warning.

9.1 One thing that may explain why it looks better live than it measures

cup_test.py opens the camera at 1280×720. But the head camera is recorded — and therefore seen by the manipulation policy — at 424×240.

That is three times the linear resolution, nine times the pixels. Small top-down cups are exactly the case where that matters most. So a detector that looks fine in the live viewer may still be useless on the frames the policy actually receives, and our 0/36 result is consistent with that.

This cuts a useful way too: the same camera clearly can deliver 1280×720. Raising the recording resolution may be much cheaper than we assumed — a configuration change rather than new hardware. <span>UNTESTED — bandwidth, disk and retraining cost all unknown</span>

9.2 Two small things in cup_test.py

1. The docstring says /dev/video6, which does not exist on this machine right now. The code is fine — find_head_camera() looks the Orbbec up by name, which is the right approach. Only the comment is stale. For reference, the rest of the project addresses it by a stable path: /dev/v4l/by-path/platform-3610000.usb-usb-0:1.1:1.4-video-index0.
2. The /dev/shm fallback when the camera is busy is a good design. The rest of the project has repeatedly been bitten by two programs grabbing one device — see §6.1. Publishing and reading a shared frame is exactly the right way around it.

9.3 Port 8795

Free on this machine, no conflict with 8770 (panel), 8781 (recording) or 8790 (file drop). If you take the Tailscale route in §4.1, http://100.107.145.111:8795 would reach it directly.

Background documents, if useful: GOAL-AUDIENCE-INTERACTION.md (the audience-interaction plan, including the four-stage sequence and the safety-layer gap) · 03-software/DEMONSTRATION-INTERFACES.md (already discusses PRESENT as a distinct skill, written 2026-08-16).