Haga / Lab / P1-E1 policy stress (robosuite, SAP v2)
Policy stress in simulation
A scripted OSC_POSE position controller in robosuite, stressed by tiered mass and friction and scored against a pre-registered relative gate. Simulation only: no learned policy, no real robot.
Where does the degradation begin, and is the tier where it begins a tier the policy was ever tuned for?
Proposition
What the run claims
A policy that clears the simulator's nominal conditions has not been shown to survive a world that is heavier, lighter or slipperier than the one it was tuned for. This run asks where a scripted controller stops clearing a gate defined relative to its own nominal rate.
Apparatus
What was measured
Robosuite manipulation primitives driven by a scripted OSC_POSE position controller. The controller is scripted on purpose. The comparison is then between stress tiers rather than between policies, and a difference cannot be a difference in learning.
Method
How it was measured
Every task is run at its nominal condition and at a ladder of stress tiers that scale mass and friction. Seeds are paired across tiers, so a tier-to-tier difference is a condition difference and not a sampling difference. The gate is relative: a tier passes when it retains a fixed fraction of that task's own nominal success rate.
Observations
What was observed
The headline figures below are the run's own. The video shows the same run as the pipeline produced it.
Which result is ours
Which result is Haga's policy-stress result?
This page. It is a scripted OSC_POSE position controller in robosuite — not a learned policy, and no real robot — stressed by tiered mass and friction, and scored against a pre-registered gate that is relative to each task's own nominal rate.
An earlier policy-stress ladder circulated with a different set of numbers. It was produced by a scripted controller too, on burned seeds under the first version's harsher perturbation ranges, and it is not reproducible from any stored artifact — so it is not published here, and no page of this site carries it.
Two of the four tasks are insensitive controls: Lift, PickPlaceCan held their nominal rate through every tier, so they measure the harness's reproducibility rather than its sensitivity. The sensitive tasks are Stack, Door.
One task's ladder is not monotonic. The computed comparison: Door at moderate succeeded on 33/50 paired seeds and Door at severe on 38/50; 2 seeds favoured Door at moderate and 7 favoured Door at severe, and McNemar's exact test gives p = 0.1797, which is not below alpha = 0.05, so the difference is not distinguishable from noise
- Lift at the gate tier
- 1.000
- mild success 1.000 >= 0.80 * nominal 1.000 (ratio 1.000)
- Lift nominal
- 1.000
- n=50 held-out seeds
- Stack at the gate tier
- 0.840
- mild success 0.840 >= 0.80 * nominal 0.900 (ratio 0.933)
- Stack nominal
- 0.900
- n=50 held-out seeds
- PickPlaceCan at the gate tier
- 1.000
- mild success 1.000 >= 0.80 * nominal 1.000 (ratio 1.000)
- PickPlaceCan nominal
- 1.000
- n=50 held-out seeds
- Door at the gate tier
- 0.760
- mild success 0.760 >= 0.80 * nominal 0.800 (ratio 0.950)
- Door nominal
- 0.800
- n=50 held-out seeds
| Observation | Value | 95% interval | Note |
|---|---|---|---|
| Lift gate-tier success | 1.000 | [0.9286, 1] | n=50, tier=mild |
| Lift double-run agreement | 1.000 | — | 100/100 seeds matched |
| Stack gate-tier success | 0.840 | [0.7149, 0.9166] | n=50, tier=mild |
| Stack double-run agreement | 1.000 | — | 100/100 seeds matched |
| PickPlaceCan gate-tier success | 1.000 | [0.9286, 1] | n=50, tier=mild |
| PickPlaceCan double-run agreement | 1.000 | — | 100/100 seeds matched |
| Door gate-tier success | 0.760 | [0.6259, 0.857] | n=50, tier=mild |
| Door double-run agreement | 1.000 | — | 100/100 seeds matched |
On the actual footage
These are the simulator's own episodes, rendered with its offscreen camera. A simulator episode has no tracked object and no threshold: the simulator reports whether the episode succeeded, so each clip carries that measured outcome and the comparison against Haga's own report, drawn on the pixels. The measurements come from the canonical Kaggle artifact and only the pixels come from the re-render, so the two are not bit-identical. Every label in each video is generated by the harness from its own output.
Demonstration of how Haga detects, flags and reports. Not accuracy evidence: accuracy claims come only from the experiment statistics.
Haga reported: episode outcome compared with Haga's failure report
Every failure case (6)
Every video the run published for a case the checker got wrong, and none of them is left out. Each says what the checker found and how it reached that conclusion.
episode0051— FAILURE
mild tier, the episode failed
VIDEO: EPISODE0051
CASE TYPE: FAILURE
WHAT WAS TRACKED: The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
WHAT WAS IDENTIFIED: The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
WHAT WAS DETERMINED: Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga reported flag on simulator outcome
WHY FAILURE: The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
SIMULATOR OUTCOME (measured): failure
episode0056— FAILURE
mild tier, the episode failed
VIDEO: EPISODE0056
CASE TYPE: FAILURE
WHAT WAS TRACKED: The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
WHAT WAS IDENTIFIED: The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
WHAT WAS DETERMINED: Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga reported flag on simulator outcome
WHY FAILURE: The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
SIMULATOR OUTCOME (measured): failure
episode0062— FAILURE
mild tier, the episode failed
VIDEO: EPISODE0062
CASE TYPE: FAILURE
WHAT WAS TRACKED: The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
WHAT WAS IDENTIFIED: The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
WHAT WAS DETERMINED: Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga reported flag on simulator outcome
WHY FAILURE: The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
SIMULATOR OUTCOME (measured): failure
episode0063— FAILURE
mild tier, the episode failed
VIDEO: EPISODE0063
CASE TYPE: FAILURE
WHAT WAS TRACKED: The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
WHAT WAS IDENTIFIED: The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
WHAT WAS DETERMINED: Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga reported flag on simulator outcome
WHY FAILURE: The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
SIMULATOR OUTCOME (measured): failure
episode0065— FAILURE
mild tier, the episode failed
VIDEO: EPISODE0065
CASE TYPE: FAILURE
WHAT WAS TRACKED: The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
WHAT WAS IDENTIFIED: The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
WHAT WAS DETERMINED: Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga reported flag on simulator outcome
WHY FAILURE: The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
SIMULATOR OUTCOME (measured): failure
episode0073— FAILURE
mild tier, the episode failed
VIDEO: EPISODE0073
CASE TYPE: FAILURE
WHAT WAS TRACKED: The clip was tracked frame by frame, and the tracked path and the measured value are drawn on the pixels.
WHAT WAS IDENTIFIED: The object was identified in each frame, and the frames the gripper held are the ones the checks run on.
WHAT WAS DETERMINED: Each check was evaluated against its threshold (e.g., contact threshold 0.008, permanence threshold 12 frames), and the frame the first of them fired on is the frame marked on the video. Haga reported flag on simulator outcome
WHY FAILURE: The post-hoc audit checked the finding against the stored per-frame evidence and found nothing that contradicts it.
SIMULATOR OUTCOME (measured): failure
A success case
episode0050— SUCCESS
the episode succeeded
VIDEO: EPISODE0050
CASE TYPE: SUCCESS
WHAT WAS TRACKED: The simulator's own episode was tracked frame by frame with its offscreen camera, and the episode's measured outcome (success or failure) was drawn on the pixels.
WHAT WAS IDENTIFIED: The simulator's internal state was identified in each frame — there is no tracked object because the simulator reports the episode outcome directly, and the frames where the episode was evaluated are the ones Haga's report was compared against.
WHAT WAS DETERMINED: Each frame's simulator outcome was compared against Haga's report. The episode's measured outcome was compared against the expected outcome with no threshold — the match is exact or it is not.
WHY SUCCESS: This episode SUCCEEDED at its stress tier: the policy held, and Haga reported no violation, which is what the simulator measured independently. It is the only success case in this run.
SIMULATOR OUTCOME (measured): success
VIDEO: EPISODE0050
CASE TYPE: SUCCESS
WHAT WAS TRACKED: The simulator's own episode was tracked frame by frame with its offscreen camera, and the episode's measured outcome (success or failure) was drawn on the pixels.
WHAT WAS IDENTIFIED: The simulator's internal state was identified in each frame — there is no tracked object because the simulator reports the episode outcome directly, and the frames where the episode was evaluated are the ones Haga's report was compared against.
WHAT WAS DETERMINED: Each frame's simulator outcome was compared against Haga's report. The episode's measured outcome was compared against the expected outcome with no threshold — the match is exact or it is not.
WHY SUCCESS: This episode SUCCEEDED at its stress tier: the policy held, and Haga reported no violation, which is what the simulator measured independently. This run has 3 success cases; this is one of them, and the other 2 are not shown because they show the same thing. Every failure case is published in full, above.
Every bar is one row of the run table below.
Limitations
What this run does not show
Where does the degradation begin, and is the tier where it begins a tier the policy was ever tuned for?
- Pillar 1 is simulation only: a scripted OSC_POSE position controller in robosuite. No learned policy, no real robot, no real footage.
- The gate is relative to nominal, not absolute: the effective threshold for a task is 0.80 times that task's own nominal rate.
- robosuite is not bit-deterministic, so the reproducibility figure is a measured band rather than a promise of exact replay.
- At n=50 per tier a rate near its threshold is often inconclusive: the Wilson interval straddles the threshold more often than not.
- Only Stack, Door moved under perturbation; Lift, PickPlaceCan held their nominal rate through every tier, so they measure the harness's reproducibility, not its sensitivity.
Sources
The dataset and the resources
- robosuite simulation — MIT License