Work with your capability map
Your capability map is the list of things your agent does in production, like "process a refund" or "track a shipment". Everything else in AgentLasso is built on it: the scorecard, coverage, risk and the golden tests you prioritize. This guide shows how the four pages under Semantic Agent Map work together to keep that map complete, stable and correct.
Four ideas to know first
- Capability: one thing your agent does for users, with a name (like
process_refund), the tools it uses, and settings such as risk and business value. You decide which capabilities exist. - Assignment: every production trace is placed in exactly one capability, or left unassigned. With semantic grouping on, a trace goes to the capability whose requests it clearly means the same as; otherwise the AI picks from your list. Without it, the free rules place a trace when its tool calls clearly point to one capability, and the AI picks the rest. The AI never creates capabilities.
- Unassigned: a trace that fits none of your capabilities. It keeps the name the AI proposed for it and waits in Discovery for you to decide.
- Alias: when you merge capability A into B, A's name becomes another name for B. Anything that still uses A's name, like an old trace or an AI answer, counts toward B.
Why this matters: if the AI could invent names, the same behavior could be called "refund_request" one day and "process_refund" the next, splitting its history and its numbers in two. Keeping assignment to your list, and naming new capabilities by hand, keeps the map stable.
How the pages fit together
The picture: Dashboard capability map
The Dashboard shows your capabilities as a map, next to a Where to focus list (on phones, the list comes first). Three views:
- Focus matrix (the default): traffic vs. eval coverage. Up = more requests over the last 30 days (log scale), right = more of the capability's traffic on tool paths a golden test covers. The shaded top-left, Test these first, is busy and under the 70% coverage target. The dashed lines are that target and the busy line (a third of the busiest capability's traffic).
- Similarity: capabilities users talk about alike sit together (by meaning with semantic grouping on, else by shared tools).
- Inventory: the same data as a table, busiest first, with every group of new behavior.
In both charts:
- Color: failure rate over the last 30 days: under 5%, 5–15%, over 15%. Size: failed requests, so the biggest bubbles are where most requests fail.
- Border: solid for a registered capability. Behavior no capability covers yet is drawn hollow: dashed when it probably belongs to an existing capability (same proposed name, or shared tools), dotted when it's new. It sits at 0% coverage, since nothing tests it yet; the busiest six are drawn, the rest are counted under the chart and listed in Inventory. Click one to open its side panel: what users asked, how the agent handled it, and Accept as new capability, Add to existing or Ignore, without leaving the map. Open in Discovery shows it among all new behavior.
- Names sit beside their bubble. Hover for the details: tests and how many more are needed, risk, failures to review, outliers, drift and review reasons. Click a capability to open its side panel.
Where to focus ranks the top five things on one scale: busy but under-tested capabilities, failures happening now, capabilities that need review, and sizable unmapped behavior.
- Under-tested and needs-review items, and a capability's name, open its side panel. The panel shows the classification confidence and its reasons, with Assignment one click away.
- New behavior opens its own side panel, as on the map.
- Failures go to the Review Queue, where you approve or dismiss them.
Semantic Agent Map
What it shows: one row per capability, with its scorecard: usage over 30 days, success rate, risk (suggested, or set by your team), business impact, money at stake, eval coverage and a priority with the recommended next step. The top of the page lists your top priorities.
Use it to:
- Decide what to test next. Start with the top priorities. They weigh traffic, quality, risk and missing coverage.
- Set what the numbers can't know:
- override a suggested risk level, giving a reason;
- set the business value per outcome;
- pick the tool field that carries money.
- See how much is waiting: the Discovery button shows how many proposed capabilities and unassigned traces are waiting.
More: The capability scorecard.
Discovery
What it shows: unassigned traffic, grouped by what the user asked for, when semantic grouping is on (Settings → Semantic grouping). A capability is what the user needed, so requests are compared by meaning, not by the tools the agent called, which may be the wrong ones. Each group forms around its most typical request, and a request joins only if it's similar enough to it. The header shows the similarity cutoff in use. It starts at a default and is calibrated on your Validation answers once there are at least 20.
Traces that aren't embedded yet, or every trace when semantic grouping is off, are grouped by the set of tools they called instead. Each card says which: by meaning or by tools. Each group shows:
- its size and 30-day trend;
- Same need, handled N different ways (groups formed by meaning): the ways its traces were handled, with counts. Paths that differ only by a retry or an extra lookup count as one way (shown as "+N variants"). If one way is wrong, for example a refund where the user asked for an exchange, that's a bug in your agent. Review those traces before turning the group into a capability;
- the tools, and example requests;
- the names the AI proposed, with counts. Mixed names can mean the group is really two behaviors;
- how many traces are flagged for review;
- the closest existing capability. If the proposed name already belongs to a capability, the card says so.
For each group, choose one:
- Accept as new capability: when this is a real, distinct thing users need.
- Give it a clear name, such as
exchange_itemorredeem_loyalty_points. Names are saved in lowercase with underscores. Suggest name & goal asks Claude to read the group's most typical requests and propose both (one AI call, only when you click). - Optionally add a business goal.
- Keep the tools that define it checked. Future traces calling them are assigned automatically by the free rules, without an AI call. Only tools at least half the group used start checked: a rarely used tool is marked, because it may be the agent's mistake, and checking it would teach the rules that mistake.
- Give it a clear name, such as
- Add to existing: when it's a variation of something you already have. The closest capability is preselected. You can also add the group's tools to it, so future traces like these are placed automatically.
- Ignore: test traffic, noise or one-offs. You can Undo right away, or find it later under Show ignored.
What happens when you accept or add:
- All of the group's past traces move into the capability.
- Its scorecard history is filled in on the days they happened, so nothing starts from zero and nothing is counted twice.
- The new capability appears on the map.
Tips:
- Busy and growing groups first: the list is sorted by the last 30 days.
- If a group mixes clearly different requests, accept what most of it is, and check later on Validation whether it should be split.
- A group with flagged traces may be a gap where your agent struggles. Look at them in the Review Queue before deciding.
Unexpected paths
When a trace is placed in a capability but the agent called a tool that changes something and belongs to another capability, the trace is flagged. For example, a request in exchange_item where the agent called stripe.refund, a process_refund tool. The reason reads "In exchange_item, but called stripe.refund (a process_refund tool): possibly the wrong action for this request." This happens when the AI or a person assigns a trace, including when you accept or add a Discovery group.
A flagged trace shows up in the Review Queue and under its capability's Failures from production in Dataset Studio, where Add as test makes a test that the other capability's tool is never called, and doesn't count as fulfilled on the scorecard. It relies on each capability's tools, which is why Discovery leaves rarely used tools unchecked when you accept a group. Tools that only read, and escalations to a human (flagged separately), don't trigger it.
Assignment
What it shows (last 30 days):
- How traces were assigned: by meaning (closest capability), by the rules (tools), by AI, by a person (Discovery or a merge), or not at all (waiting in Discovery). Shown overall and per capability. With semantic grouping on, Meaning vs AI shows how often Claude agrees with the matches made by meaning, on the 10% it double-checks.
- Traces from before this was tracked show as "not recorded".
- Rules vs AI: about 10% of the traces the rules place are also sent to the AI as a check. This shows how often they agree, and the latest disagreements.
- This needs an Anthropic API key on the server; without one, the section says so.
- Capabilities: each with its traffic, how its traces were assigned, its rules-vs-AI agreement, its aliases, and Merge into…
Map health (the Health column): each capability is Healthy, Needs review or Not enough data (fewer than 10 traces in the last 30 days), always with the reasons. A capability needs review when any of these is true:
- Spread out: its requests aren't close to each other (below the similarity cutoff). It may be more than one behavior.
- Outliers: at least 20% of its recent traces don't fit it well.
- Too close to another capability (similarity 0.9 or more): they may be the same behavior.
- What users ask for changed from the previous 30 days (by meaning).
- The agent handles it differently than in the previous 30 days: its mix of tool paths shifted.
- People judged pairs inside it as different behaviors on Validation.
Traffic that doubled or halved is shown as a note. The signals by meaning need semantic grouping on; without it, the path, traffic and Validation signals still work. No single "confidence" number is shown: each status is the measured signals and the thresholds above.
Use it to:
- Keep an eye on "Unassigned". A rising share means new behavior is appearing: go to Discovery.
- Read the disagreements. When the rules and the AI disagree, two capabilities usually share tools. Fix it one of two ways:
- give each capability its own distinctive tools;
- if they're really the same behavior, merge them.
- Merge duplicates: Merge into…, pick the capability to keep, confirm. What happens:
- The merged capability's traces, scorecard history, tools and test-case links move to the one you keep.
- Its name becomes an alias.
- Its own settings (risk, business value, money field, goal) are not kept. Note anything you want to keep first.
- A merge can't be undone.
Validation
What it shows: how well the map matches what people think, measured from your answers to one question about two real traces: "Do these two need the same capability from the agent?"
How to label:
- Click Start labeling. Pairs appear side by side: each one's request, reply and tools.
- Answer Same (S), Different (D) or Can't tell (U). K skips. Add an optional note on why.
- You then see what the system thinks and why the pair was picked. Press Enter for the next pair.
What the system thinks stays hidden until you answer, so it can't sway you. Answer based on what the agent has to do, not how the request is worded: "where's my order?" and "has my package shipped?" are the same; "where's my order?" and "change my address" are not.
Which pairs you get are the places the map is most likely wrong:
- Inside one capability: its two most different traces. Checks that the capability is one behavior.
- Across capabilities: capabilities that share tools, or where the rules and AI disagreed. Checks that they're really different.
- Unassigned vs closest: a Discovery trace and its closest capability. Checks whether "Add to existing" would be right.
Reading the results:
- Agreement: how often the assignment matched your answers, overall and per kind of pair, with the number of pairs.
- Under 20 answered pairs, it's marked as a first look.
- "Can't tell" answers are kept but not counted.
- Suggestions need at least 3 answers:
- May be two behaviors: at least half the pairs inside a capability were judged different. Look at its traces and give each behavior its own capability and tools.
- May be one behavior: at least 75% of the pairs across two capabilities were judged the same. Merge them on Assignment.
Answers are never edited. If you answer a pair again, the newest answer counts. Each answer keeps a snapshot, so later merges don't rewrite past results.
Routines
When you connect a new agent
- Send traces: Send traces.
- Open Discovery. Your first traces show up here as proposed capabilities. Accept the real ones, with their tools.
- Check the Semantic Agent Map: set risk and business value for the capabilities that matter.
- Label about 20 pairs on Validation to get a first agreement number.
Every week (about 10 minutes)
- Discovery: handle new groups, busiest first.
- Assignment: check the unassigned share and any new rules-vs-AI disagreements.
- Validation: label 10–20 pairs. Act on any suggestion.
- Semantic Agent Map: work on the top priorities: turn uncovered behavior into golden tests.
After a prompt, model or tool change
- Watch Assignment for a jump in unassigned traffic or disagreements: the agent may now behave differently.
- Label a few pairs on Validation for the capabilities you changed.
When two capabilities seem to overlap
- Check Assignment for disagreements between them.
- Label pairs across them on Validation.
- If people say they're the same, merge; if different, give each its own tools.
When one capability seems to be two behaviors
- Validation says "may be two behaviors", or its traces look mixed on the map.
- Note which requests belong together, and which tools each behavior uses.
- Splitting a capability isn't available yet: its existing traces can't be moved into a new capability from the app. For now, keep the suggestion in mind when you read that capability's numbers.
Questions
- Why is a trace unassigned when one of my capabilities obviously fits? The capability may have no tools set, so the rules can't place it, and the AI didn't judge it a clear match. Add it to that capability from Discovery, and include its tools.
- Can the AI create a capability by itself? No. It can only propose a name. A person decides in Discovery.
- Where did my merged capability go? It's now an alias of the one you kept, listed under that capability on Assignment.
- The Rules vs AI section is empty. It needs
ANTHROPIC_API_KEYon the server and new traffic. - Can I undo a merge? No. Accept and add can't be undone from the page either. Ignore can be.
Next
See how each capability is scored: The capability scorecard.