Read test results

Every CI run against a golden dataset appears on the Test Runs page, graded check by check.

The run list

Each run shows its dataset, its status, and Pass X% / need Y%: the pass rate against the run's pass threshold.

• Completed: the pass rate met the threshold.
• Failed: it didn't, or nothing in the run could be graded.
• N not graded (no checks): test cases without checks. They verify nothing, so they're left out of the pass rate.

Each test case row summarizes its checks by family (trajectory, content, format), then its result and score:

• a check mark: every check in that family passed;
• a red flag: a critical check failed;
• a warning mark: only minor checks failed;
• an empty circle: the test case has no checks in that family.

A test case's detail

Click a row to see exactly what happened in that run:

• Checks: each check with pass/fail, its family and severity, and the reason, e.g. "stripe.refund was never called. The agent called: orders.lookup."
• Tool calls in this run, with any expected tool that was never called highlighted.
• Agent output in this run.
• Rubric, not graded yet: free-text items that need converting into checks.
• The production trace this test case was created from, for comparison. That's the original behavior, not this run.

Pass threshold

A run passes when its pass rate is at least its pass threshold, which you set when creating the run (default 1.0: every graded test case must pass). A run where nothing could be graded never passes. See Run golden tests in CI and the API reference.