Coverage plan
Not "we have 347 evals", but does every behavior your agent handles in production have a test?
Behavioral coverage is one number, on the Coverage plan page, the Dashboard and Dataset Studio:
Behavioral coverage 67%: 4 of 6 behaviors your agent handles in production have a golden test that checks their real traffic.
- A behavior is a capability with traffic in the last 30 days. Capabilities with no traffic aren't counted.
- It's tested once at least one golden test with checks covers a tool path its real requests take (see below). A test linked to the capability whose checks match none of its real traffic doesn't count: it doesn't test what users actually do.
- The goal is every behavior tested.
Coverage says what your tests check, not whether your agent passes them. A fully covered agent can still fail its tests; that's what CI is for.
The plan
Below the number, Test next lists the untested behaviors, busiest and riskiest first: each one's share of traffic × its risk (high 3, medium 2, low 1). Each item shows:
- its share of traffic, requests in the last 30 days and risk;
- why it's untested: no tests yet, or tests that don't check any path its real requests take;
- its busiest path with a real request to start from;
- Add a test, which opens it in Dataset Studio, where Add as test turns that request into a test that covers the path.
Tested lists the behaviors that are covered, each with its busiest path its tests don't check yet: the next test to add once every behavior has one. How much of each capability's traffic is tested, path by path, is in its scorecard on the Semantic Agent Map.
When a test checks real traffic
Coverage is measured on tool paths: the tools a trace called, in order, e.g. orders.lookup → stripe.refund. A golden test case with at least one check covers a path when any of these is true:
- the test was created from a production trace that took exactly this path;
- the test expects tools (
Tool must be called,Tools called in this order), all of them occur on the path, and every other tool on the path only reads data. Lookups needn't be asserted, but anything that changes something must be; - the path calls no tools, and the test expects none (e.g. a refusal test).
So a refund test covers orders.lookup → stripe.refund, but not orders.lookup → stripe.refund → shippo.create_label ("refund and send a replacement"): creating the shipment is untested behavior. Paths are compared as sets of tools; order isn't compared.
A test case without checks covers nothing: it verifies nothing.
Limits
- Coverage is computed from your recent traces (up to the 1,000 most recent the app loads).
- Coverage history over time is not available yet.