Robotics · policy evaluation & statistics

policy-eval-harness

A statistics layer for user-provided rollout logs that never reports a success rate without its sample size and confidence interval.

Public · synthetic demo

The problem

Same training loss does not mean the same real success rate.

Nearly matching loss curves can hide materially different measured outcomes. Rollouts need task-level rates, uncertainty, and a formal comparison.

The money shot

Evaluation is split by policy and task, with uncertainty attached

Measured evaluation set

724episodes

2policies

3tasks

8flags raised

policies: policy_a · policy_b
tasks: cable_insert · door_open · pick_place

Statistical contract

Wilson confidence intervals, per-task breakdowns, A/B difference intervals, a two-sample proportion test, and small-sample flags keep the measured rate attached to its evidence.

A rate without n and CI is never emitted as the headline result.

Honest limitations

This is not a robot runtime, RL training framework, or simulator. It only scores rollout logs supplied by the user and does not establish real-world generalization.

This page reports the supplied sample output and its stated limits. Inspect the code and evidence ↗