The capability scorecard

The Semantic Agent Map gives every capability five numbers and one recommended action, in terms a product manager and an engineer can both act on:

DimensionExample
Usage19,204 requests in the last 30 days
Success87%, down 8 points
RiskHigh: can move money (stripe.refund)
Business impact$133,656 saved, $1.7M at stake
Eval coverage63% of its traffic is tested
PriorityExpand test coverage
The Semantic Agent Map: top priorities and the capability scorecard

Click a capability (here or on the Dashboard map) for its side panel:

  • Name and description.
  • Requests over 30 days, the change against the previous 30 and a daily chart.
  • Failure rate, now and before, and whether it comes from reported outcomes or is inferred from classification flags.
  • Golden tests: how many, how much of the traffic they cover and how many more reach the target.
  • What users ask: representative requests, one from each of the busiest tool paths.
  • How the agent handles it: its most common tool paths, each with its share of traffic and whether a test covers it.
  • Edge cases: open flagged traces and their reasons, linking to the Review Queue.
  • Classification confidence: high, medium or low, from how often Claude agrees with the quick placements in spot checks and from map health (whether its requests hang together), with how its traces were placed.

Risk, business value and goal are folded at the bottom; open them to see the reasons or change them.

A capability's side panel: traffic, failure rate, tests, example requests, tool paths, edge cases and classification confidence

Usage

Requests classified into the capability in the last 30 days, with the change against the 30 days before.

Success

The share of the capability's traffic not flagged as a failure or edge case during classification, over the last 30 days, with the change in percentage points against the previous 30 days.

When a trace comes with an outcome (resolved, escalated or failed, sent with the trace or mapped from an imported column), the outcome decides instead: only resolved counts as a success. See Send traces.

Without outcomes it's a proxy for task success, not a graded one: it can only be as good as the failure signals classification sees. Below 90% the capability is marked as drifting.

Risk

What the capability can do if it gets things wrong. It's suggested from the tools it calls and from what production shows:

  • High: it calls tools that move money or make irreversible changes (refund, charge, payment, transfer, cancel, delete, …).
  • Medium: it calls other tools that change data (create, update, send, …).
  • Low: it only reads (lookup, get, search, track, …), or calls no tools.
  • Adversarial traffic (e.g. prompt-injection attempts) raises the level by one. Escalations to a human are listed as a signal.

Every level comes with its reasons. Your team can override it in its side panel (Risk, business value and goal → Change) with a required reason, e.g. "Regulated: refunds above $500 need a compliance review". The suggestion stays visible next to the override, and Use the suggestion removes it.

Business impact

Value. In the side panel's Risk, business value and goal → Business value, say what one successful outcome is worth:

  • Deflection: each success is a case no human had to handle. Enter the cost of a human-handled case.
  • Revenue: each success is a revenue event, e.g. a recovered cart. Enter the revenue per outcome.
  • No $ outcome: a quality or safety capability, such as a human handoff.

Value over 30 days = usage × success × your $ per outcome. The $ per outcome is your assumption, and it's shown as one.

Money at stake. How much money actually flows through the capability, measured from the amounts in its tool calls:

  1. AgentLasso finds money-looking numeric arguments in the capability's tool calls (e.g. stripe.refund → amount), with how many traces use each and a typical value. IDs, counts and percentages are skipped.
  2. You confirm the field and its unit. Many payment APIs, Stripe included, send cents (8900 = $89.00), so the wrong unit is a 100× error. The preview shows what the typical value means in each unit.
  3. AgentLasso estimates: the average amount per request in recent traces × 30-day usage. It also reports how much of it flows through tool paths no golden test checks.

Eval coverage

The share of the capability's traced traffic whose tool path a golden test checks. The side panel lists the most common paths, with their share of traffic and whether each is tested; the Coverage plan lists every untested one with a real example request. See Coverage plan for exactly how a path counts as tested.

Priority

One recommended action per capability, from transparent rules. The first one that matches wins, and each comes with a one-line reason:

#WhenPriority
1No traffic in the last 30 daysNo traffic yet
2Success fell 3+ points and coverage is at least 70%Investigate regression: the tests may be missing the new failure
3High risk and coverage below 70%Expand test coverage, plus the money flowing through untested paths when money at stake is set up
4Success below 85% and it's among the busiest capabilities (at least a third of the busiest one's usage)Improve the agent
5Success fell 3+ pointsInvestigate regression
6Medium or high risk and coverage below 70%Expand test coverage
7It should have a $ value but none is setSet business value
8OtherwiseMonitor

Top priorities (above the table) ranks the capabilities that need action: roughly traffic × risk weight (high 3, medium 2, low 1) × the untested share, boosted by quality drops. It puts what's busy, risky and untested first.

Next

Measure your whole agent, not one capability at a time: Coverage plan.