Lab
Haga is one gate in a pipeline you already run.
The pipeline below is the order of operations in a release: artifact in, decision out. Haga occupies the middle — after tracking, before the decision is written. It does not replace your evaluation suite, your reviewers, or your sign-off. It adds an adversarial pass that is run the same way every time.
- trackTrajectories are extracted from the artifact. In simulation, from the episode logs; in video, from the tracked points.1
- identifyThe task, the stress tier and the applicable checks are determined, so the same criteria are applied to every run.2
- determineSuccess or violation is measured against a criterion frozen before the run, never after seeing the result.3
- flagThe measured value is compared to the gate threshold, and the verdict is one of three states: flag, clean, abstain.4
- reportThe artifact is written and stored, so the run can be audited later.5
Experiments
Published runs
Every run below states its method, its measurement, its interval, and its limits.
P1-E1 policy stress (robosuite, SAP v2)
A scripted OSC_POSE position controller in robosuite, stressed by tiered mass and friction and scored against a pre-registered relative gate. Simulation only: no learned policy, no real robot.
Lift at the gate tier1.000mild success 1.000 >= 0.80 * nominal 1.000 (ratio 1.000)
P2-E1 real-footage calibrated gate (SAP v2), three ALOHA-family datasets
The three-state gate (FLAG / CLEAN / ABSTAIN) applied to real tracked robot footage from the three ALOHA-family datasets SAP v2 names, with a disjoint calibration and test split per dataset. The support plane is measured per clip and the detectors are gated on the gripper-derived free-frame mask.
VerdictFAILacross towel, candy, ziploc
P2-E2 generative cohort (frozen thresholds)
The frozen-threshold physics-consistency checker scored over three world models' held-out generations, against real control clips. Sensitivity and specificity are reported together; a specificity number is published only when the real-control cohort is large enough to have a usable denominator.
CalibrationThe real-control gate failed, so this cohort's flag rates carry no information
Real-control specificity0.00015 real controls, 15 flagged
Methods
What SAP v2 freezes
SAP v2 is the Scenario Assessment Protocol — Haga's verification protocol, version 2. The frozen document carries only the letters, so this page spells them out: the scenarios a run is scored on and the assessment — endpoints, thresholds and gates — it is scored by.
Version 2 replaced an unverifiable version 1. v1 results are retained, and each is published only with the artifact that reproduces it — the v1 policy-stress ladder has no such artifact, so no page of this site carries it. v2 runs are fresh, pre-registered and independently verifiable. It freezes three things.
- Policy stress (P1). Four tasks, 50 held-out seeds per task, four tiers, the same seeds paired across tiers, and a relative gate: a tier passes when it keeps at least 80% of that task's own nominal success rate. The controller is scripted on purpose, so a difference between tiers is a difference in condition and not a difference in learning.
- Real-footage gate (P2-E1). A three-state gate — FLAG, CLEAN, ABSTAIN — over real tracked robot footage. The support plane is measured per clip rather than declared, the detectors are gated on the gripper-derived free-frame mask, and thresholds are fitted on calibration episodes that are disjoint from the test episodes they are scored on.
- Generative cohort (P2-E2). Frozen thresholds, held-out generations from three models, and at least 15 real control clips so specificity has a usable denominator. A specificity number is published only when that cohort is large enough to have one.
Every number in the lab comes from a Kaggle run under this protocol, and every run ships its limitations with it. Where a run failed, the failure is published with the same prominence as the result.