Haga

Independent verification for physical AI

We check your policy and your world model, and we publish our own research runs, including the failures.

Haga applies two instruments to the two artifacts that decide what a robot does: the policy, and the world it is trained in. Every result ships as a number, an interval, and a limitation.

Request an evaluationRead the lab

Every number on this site is read from the same Kaggle artifact files the Lab runs write — read from the artifact, not retyped. Numbers on Lab pages are verified by a build-time check against the artifact JSON.

Problem

A world that breaks physics teaches a policy to break physics.

World models are trained on synthetic frames and policies are trained inside simulated worlds. Both are graded by the team that built them. A policy that looks robust in a world with wrong physics will not survive contact with a world with right physics. The current workflow does not check this.

  • Policy evaluation is a leaderboard, slow and public, and usually reached only after the release decision.
  • World-model evaluation is often a demo reel; a model that freezes an object mid-air is shown, then shipped.
  • Neither artifact is measured against a criterion that was frozen before the run.
Method

Two instruments, one pipeline.

Haga scores a submitted artifact with a fixed pipeline. The pipeline is the same for every run, so a number from one run is comparable with a number from another.

Policy stress (P1-E1)
A policy is run under tiered mass and friction perturbations, in simulation, with paired seeds. The gate is relative to the policy's own nominal rate, so the comparison is fair across tasks of different difficulty. The controller is scripted — this run measures the harness, not policy robustness.
World-model physics (P2-E1, P2-E2)
Tracked trajectories from a generated world are checked for physical violations against calibrated detectors, with thresholds frozen before the held-out cohort is scored. Real controls are scored alongside to measure specificity.
Featured run

Real-footage calibrated gate

The three-state gate (FLAG / CLEAN / ABSTAIN) applied to real tracked robot footage from the three ALOHA-family datasets SAP v2 names, with a disjoint calibration and test split per dataset. The support plane is measured per clip and the detectors are gated on the gripper-derived free-frame mask.

FAILFAIL on all three datasets under SAP v2 Wilson 95% upper bound ≤ 0.020 rule
DatasetSourceTest episodesTracks scoredFlag rate95% CITargetStatus
towellerobot/aloha_static_towel15-49670.045[0.0153, 0.1236]0.020FAIL
candylerobot/aloha_static_candy15-49830.012[0.0021, 0.0651]0.020FAIL
ziploclerobot/aloha_static_ziploc_slide15-55570.193[0.1113, 0.3134]0.020FAIL

Read the full run → · Simulation baseline: P1-E1 policy stress: control run, not a policy result

  • The contact penetration threshold (0.0080 m) sits below the measured tracking noise floor (0.0160 m); this is the attributed cause of the false positives (see P2-E1).
  • Under the frozen rule a PASS at a 0.020 target needs at least 189 clean tracks with zero flags (280 with one); each dataset here has fewer (67, 83, 57), so none could pass.
Published

Three runs, published as measured.

Each line below is a link to the full run: the method, the table, the interval, and what the run does not show.

Open the lab →Read a sample report

Evaluation

Submit a policy or a world model.

Tell us what should be scored. We acknowledge every submission, agree the criterion and the thresholds before anything runs, and report what we find — including the failures.

Or write directly: haga@mushoodhanif.com

Haga — Independent verification for physical AI