Real-footage calibrated gate

A gate is only useful if it refuses the right things. This run applies a three-state gate — FLAG, CLEAN, ABSTAIN — to real bimanual robot footage and measures how often clean footage is refused.

LeRobot's ALOHA static towel, candy and ziploc tasks: real robot footage, tracked trajectories, and a split between calibration episodes and test episodes that does not overlap. The three datasets are scored separately, each on its own footage, and the gate's verdict is reported per dataset as well as for the set.

Thresholds are fitted on the calibration episodes and frozen before the test episodes are scored. The clean cohort is presumed violation-free and is not independently labelled, so a flag on one of its clips is not established as wrong; synthetic injected violations measure detection. The two are reported separately, because a gate that confuses them is not a gate.

The headline figures below are the run's own. The video shows the same run as the pipeline produced it.

The overlay's post-hoc audit carries a finding called SUPPORT-PLANE-GAP. On this run it appears on 15 of the 15 flagged tracks (1.000) and on 192 of the 192 (1.000) unflagged ones. It is on every track, so it does not distinguish a flagged track from an unflagged one and it says nothing about why the gate refused what it refused.

That is a property of the rule rather than of the run: the support plane is measured per clip, so the audit re-derives the same quantity the checker used and the two agree by construction. A finding that cannot come out the other way cannot carry a signal.

So the question stays open, and it is not answered here. The clean cohort is presumed violation-free and is not independently labelled, so a flag on one of its clips is not established as wrong. The planned check is an independent review of the flagged tracks by a human, and the result of that review is not yet in.

The table and the videos can disagree, and where they do the page says so rather than picking one. The table carries the verdict the RUN RECORDED; each video carries the verdict RE-DERIVED from that clip's own stored frames, because a video is a rendering of the evidence and not of the record. For 5 of the 209 rows the two differ, and the run's own record is what the table shows.

One more thing the table and the videos do not say on their own: the table has one row per TRACK, not one per clip, and a clip that was tracked more than once has more than one row. For 58 of the 209 rows the per-frame evidence of that specific track is not stored separately -- the artifact kept one track's trajectory for the clip -- so the first flagged frame is shown as a clip-level flag rather than as a frame index that would belong to the other track.

Does the checker catch what it is meant to catch, and what does it cost when it is wrong?

• The object diameter is ASSUMED per dataset and metres-per-pixel is a declared camera calibration, not a measurement. No reference dimension exists in any of the three datasets.
• The track-picker's span band is ONE geometric prior shared by all three datasets. It is not fitted to any dataset's result, and it is not varied per dataset.
• Injected violations are SYNTHETIC and prove only that the detectors can fire on a known violation.
• The clean cohort is PRESUMED violation-free, not independently labelled, so a flag on one of its clips is not established as wrong. The post-hoc audit is where a flag is checked against measured evidence.
• Detector coverage is incomplete: a detector that never became applicable has no measurable rate.
• The three datasets are shorter than SAP v2 assumed: the preregistration assumed 80 episodes each and the released datasets have towel 50, candy 50, ziploc 56. The effective test windows are the preregistered 15-79 intersected with the episodes that exist, and each artifact records the shortfall.
• The towel false-positive rate is 0.045 against a 0.020 target.
• The ziploc false-positive rate is 0.193 against a 0.020 target.
• The gate ABSTAINED on some clips: ziploc abstained on 2 clips. Those clips were not scored, and they are counted as neither flagged nor clean -- an abstention is a refusal to certify, not a pass (reason: too_short_for_detector_window).