Haga / Lab / P2-E2 generative cohort (frozen thresholds)

Generative world-model cohort

FAILFrozen protocol v1Generated 2026-10-20

The frozen-threshold physics-consistency checker scored over three world models' held-out generations, against real control clips. Sensitivity and specificity are reported together; a specificity number is published only when the real-control cohort is large enough to have a usable denominator.

How much of what the checker sees is a real physical violation, and how much is the checker's own sensitivity?

Proposition

What the run claims

A checker that fires on generated video is only useful if it stays quiet on real video. This run scores one frozen threshold set over held-out generations from three world models, against real controls.

Apparatus

What was measured

Held-out generations from CogVideoX, Cosmos and Wan, plus a real-footage control cohort. Seeds and perspectives are fixed by protocol before anything is generated.

Method

How it was measured

The same frozen checker scores every cohort. Flag rates are reported per model, with intervals. Where the control cohort is too small to support a specificity claim, the site says so instead of publishing a number.

Observations

What was observed

The headline figures below are the run's own. The video shows the same run as the pipeline produced it.

Calibration failed

The real-control gate failed, so this cohort's flag rates carry no information

The real-control specificity is 0.000: 15 of 15 presumed-clean real clips were flagged. The generated clips' flag rates are therefore not a measurement of anything about the generated clips -- a checker that flags presumed-clean real footage as readily as it flags generated footage has not distinguished the two.

The generated clips were flagged at a LOWER rate than the real ones: cogvideox, wan were flagged at 0.889, 0.889 against 1.000 for the real controls.

The cause: Three things, all read off this run's own artifact. (1) The checker was the EARLIER binary FLAG/CLEAN one: this run's source provenance names src/haga/physicsiq/scoring.py and does not name src/haga/physicsiq/gate.py, the SAP v2 three-state gate, and scoring.py has no ABSTAIN state at all. (2) The scale was untrusted on every clip -- scale_trusted is false for all 15, the metres-per-pixel is fitted by gravity with one clip falling back to a guess, and the object diameter is ASSUMED -- and the support plane came out identical (456.0) for all 15 clips rather than measured per clip the way the P2-E1 run measures it. (3) The two numbers are not measurements of the same cohort: P2-E1's 69 tracks are ALOHA towel footage, and none of this cohort's 15 real controls is an ALOHA towel clip.
The fix: Re-score this cohort under the SAP v2 protocol that has already been built and run on the real-footage cohort (P2-E1): the three-state gate with a per-clip measured support plane, a gripper-derived free-frame mask, and a disjoint calibration and test split, with a real-footage real-control cohort and n>=30. The protocol is not still to be designed; the remaining work is to apply the one already run to this generative cohort.

Status: protocol already run on real footage; re-scoring this cohort is the remaining step, and this cohort has no published result yet

cogvideox flagged: 0.889; 8 of 9 held-out clips flagged
cosmos flagged: 1.000; 9 of 9 held-out clips flagged
wan flagged: 0.889; 8 of 9 held-out clips flagged
Real-control specificity: 0.000; 15 real controls, 15 flagged

Observation	Value	95% interval	Note
cogvideox flag rate	0.889	[0.565, 0.9801]	9 clips
cosmos flag rate	1.000	[0.7008, 1]	9 clips
wan flag rate	0.889	[0.565, 0.9801]	9 clips
Real-control specificity	0.000	[0, 0.2039]	15 real controls, 15 flagged

On the actual footage

These are generated clips. A generated clip has no tracked path: the run kept the checks that fired and the frame counts, but not the coordinates, so there is no track to score and the frame count reads as zero. What the video shows is the checker's own flag, drawn on the frames the checker judged. A generated clip has no independent reference, so none is claimed for it — and because this cohort's real controls were all flagged too, the generated flag rates carry no information: the run's verdict is stated by its statistics above, not by these videos. Every label in each video is generated by the harness from its own output.

Demonstration of how Haga detects, flags and reports. Not accuracy evidence: accuracy claims come only from the experiment statistics.

The table above reports a rate for every cohort, including cogvideox (its licence is CogVideoX LICENSE (custom, non-commercial without registration), which requires users who wish to use the models for commercial purposes must register and obtain a basic commercial license). That rate is a real measurement on a real cohort and it is what the experiment found. The FOOTAGE of that cohort is not published here, because the licence that governs it does not permit commercial redistribution. The clips were scored, the numbers are reported, and the videos are withheld — the other cohorts' videos are shown.

NO INDEPENDENT REFERENCE: independent labels do not exist for this clip

A success case

cosmos-heldout-v1 / 0001_left_seed2— SUCCESS

no tracked path: the run kept the checks, not the coordinates

VIDEO: 0001_LEFT_SEED2
CASE TYPE: SUCCESS
WHAT WAS TRACKED: This generated clip has no tracked object — the run kept the checks that fired and the frame counts, but not the coordinates, so there is no track to score and the frame count reads as zero.
WHAT WAS IDENTIFIED: There is no independent reference for this clip, so nothing was identified against an external label — the checker's own flag is what the video shows.
WHAT WAS DETERMINED: The checker's own flag was drawn on the frames the checker judged. There is no independent reference, so no determination against an external label was made — the run's verdict for this cohort is stated by its statistics above.
WHY SUCCESS: This is what the checker did on this generated clip. A generated clip has no independent label, so it cannot be right or wrong — the run's verdict for this cohort is stated by its statistics above. It is the only success case in this run.

Haga reported: ballistic

Post-hoc audit: SUPPORT-PLANE-GAP consistent with; GRASPED-OBJECT-HOVER consistent with; TRACKER-HEALTH consistent with

Flag rate by modelrate

Every bar is one row of the run table below.

Limitations

What this run does not show

How much of what the checker sees is a real physical violation, and how much is the checker's own sensitivity?

Real-control specificity is measured over n=15 real clips, which is the smallest cohort the frozen protocol accepts.
The scoring device is part of the result: CUDA and Apple MPS disagreed on one borderline Cosmos clip. The Kaggle snapshot is canonical.
Held-out protocol v1 was frozen 2026-07-19 with seeds 2-4, n=9 per model.
Position-only checks cannot see a gripper: a grasped, stationary object is indistinguishable from a generative freeze.

Sources

The dataset and the resources

nvidia/Cosmos-Predict2.5-2B — NVIDIA Open Model License
Wan-AI/Wan2.2-TI2V-5B-Diffusers — Apache License 2.0