Quickstart

From your first production trace to a golden test running in CI. You'll need an agent that calls tools (or any LLM app you can describe as request → reply), and about 15 minutes.

1. Create a project

Sign up at agentlasso.dev. You get two projects, switchable at the bottom of the sidebar:

  • your workspace, empty until you send or import traces, so every number in it is your agent's;
  • a Sample project with an example support agent's data (12 capabilities, 8 weeks of traffic dated to when it's created), to see what every screen looks like with traffic: tested and untested behavior, failures to review, a capability that needs review, and new behavior in Discovery. Refresh example data on its banner dates it to today again.

Have more than one agent? + New project… in the same picker creates an empty project with its own API key (up to 20). Rename a project in Settings → Name.

Accounts created before October 2026 started with that example data inside their workspace. Settings → Sample data in your workspace removes it, keeping anything your own traffic or tests use.

In Settings:

  • Copy your project's API key. You'll send traces with it.
  • Write a sentence or two in Agent description about what your agent does. Classification uses it to name capabilities and judge edge cases in your agent's real domain.
  • Turn on Semantic grouping (Embed user requests) before you send or import traces. It's off by default, and it makes the biggest difference to how good your map is:
    • Discovery groups new behavior by what users asked for. Without it, it groups by the tools the agent called, so different requests handled with the same tools land in one group, and one request handled two ways is split in two.
    • New traces are placed in the capability they mean, and the map's health and drift signals by meaning work.
    • Only each trace's user request, after personal data is redacted, is sent to the embeddings provider. Replies, tool calls and arguments aren't.
    • Turned on later, it embeds existing traces in the background, a few hundred per processing round.

2. Send your first trace

Have history already (logs, a database, an observability tool's export)? Production Sync → Import traces brings it in from a file, each request on the day it happened. You can keep using AgentLasso on imports alone, without a live integration; the numbers then update with each import. See Imports instead of a live integration.

Otherwise, POST one request your agent handled. No SDK, no OpenTelemetry envelope needed:

sh
curl -X POST "https://agentlasso.dev/api/v1/telemetry" \  -H "x-api-key: <your-project-api-key>" \  -H "Content-Type: application/json" \  -d '{    "user_input": "Can you cancel my subscription?",    "agent_output": "Done — cancelled effective end of your billing period.",    "tool_calls": [{ "name": "stripe.subscription.cancel", "arguments": { "subscription_id": "sub_123" } }]  }'
  • user_input: what the user asked.
  • agent_output: the agent's final reply.
  • tool_calls: the tools it called, in order, with their arguments. This is what trajectory checks and coverage are measured on, so include it whenever your agent uses tools.

The trace shows up under Production Sync right away. Emails, phone numbers, SSNs and card numbers are redacted before storage (you can turn this off per project in Settings).

Already using OpenTelemetry? Point your exporter at the same endpoint instead:

sh
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://agentlasso.dev/api/v1/telemetry"export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=<your-project-api-key>"

To send real traffic, add the POST to wherever your agent finishes a request, without blocking the response on it. All fields and response codes: Send traces. If you already send traces somewhere else, see Send traces you already have.

3. See what your agent does

Open the Semantic Agent Map. Each trace is classified into a capability:

  • Free, by tool matching: a trace whose tool calls match a registered capability's tools is placed there immediately.
  • By Claude: traces the rules can't place are classified by Claude. This needs an Anthropic API key on the server; until then they wait, and the map says how many.

Traffic that fits none of your capabilities shows up in Semantic Agent Map → Discovery, grouped by the tools it used, where you can accept it as a new capability. How Discovery, Assignment and Validation work together: Work with your capability map.

Click a capability to see its scorecard: usage, success, risk (with the reasons), business impact, eval coverage, and the recommended next step. More in Capabilities and classification and The capability scorecard.

4. Turn real behavior into a golden test

Open Dataset Studio. Capabilities are listed on the left in the Coverage plan's order: untested ones first, busiest and riskiest first. Pick one; on the right you'll see:

  • Untested paths: real requests no test checks yet. Add as test turns one into a test from that trace, with checks on the tools its path calls. It covers that path right away.
  • Failures from production: flagged requests for this capability without a test. Add as test turns one into a test. When the failure says what correct looks like (it called another capability's tool, or it was an attack), the test gets those checks; otherwise the rubric editor opens so you can say what the agent should have done.
  • Tests: its test cases, with their last result and which of its paths each one covers.

Add test also lets you write one, generate a few with AI, or upload a file. New tests get graded checks from whatever their source says (tools a path calls, a generated or uploaded rule, an AI judge criterion). A test with nothing to check yet opens in the rubric editor, or is marked no checks until you add some.

To change or add checks on a test case, click Edit rubric (or Add checks). In the rubric editor:

  1. Add checks, or start from a template:
    • Trajectory: the tools the original trace called, as one-click suggestions ("Calls stripe.refund", "In order: orders.lookup → stripe.refund").
    • Q&A answer: facts the reply must mention.
    • Forbidden content: phrases and personal data the reply must never contain.
    • Refusal / red team: tools that must never be called.
    • AI judge: plain-English criteria graded by Claude (needs your Anthropic key in Settings → AI judge).
  2. Under Try it before saving, grade the draft against the original production trace (or a pasted reply, or your agent's live answer if you've connected it). You'll see Would pass or Would fail and why, check by check.
  3. Close it with Done. Changes save as you go.

A test case without checks isn't graded: it verifies nothing. The test case card says so. Every check type and grading rule: Checks reference.

5. Check your coverage

Open Coverage plan. Behavioral coverage says how many of the behaviors your agent handles in production have a golden test that checks their real traffic, e.g. "4 of 6".

Test next lists the untested ones, busiest and riskiest first, each with a real request to start from. Add a test opens it in Dataset Studio, where Add as test turns that request into a test. Work down the list until every behavior is tested. How it's measured: Coverage plan.

6. Run your tests

With your agent connected (Settings → Your agent, usually just its URL: see Connect your agent), click Run all tests on Test Runs, or Run on one test in Dataset Studio. AgentLasso calls your agent on each test, grades the answers and shows the results on Test Runs. See Run tests from the app.

To gate pull requests, run them in CI. CI can run every golden test, only some capabilities' tests ({"capabilities": ["process_refund"]}), or one suite: a group of tests, shown under Dataset Studio → By suite with its ID and a Run in CI snippet. New tests go into your project's default suite unless their capability's tests live in another one. Your pipeline:

  1. creates an eval run and receives its test cases,
  2. sends each test case's input to your agent,
  3. reports back what the agent said and which tools it called,
  4. gets a graded result, and fails the build if a critical check failed.

Run golden tests in CI has a ready-to-adapt GitHub Actions example; the API reference has every request and response, and Read test results explains the Test Runs page.

Next steps

  • Keep traffic flowing: coverage and the scorecard are only as current as your traces.
  • Add a capability's risk and money field in its side panel (Risk, business value and goal), so coverage weighs what matters most.
  • Prefer to run it yourself? See Self-hosting.