Evaluate agent decisions before go-live.

Find out whether a change to how your agent decides would actually help your users, using the conversations you already have. Add exploration for the strongest evidence.

Get started
$ uv tool install "agentope[llm]"
  1. 1Check what's possible

  2. 2Import the traces

  3. 3Ask the model again

  4. 4Evaluate the change

terminal · support-agent traces
$ agentope check --traces traces.otlp.json.gztool_choice · 12,000 decisions found in traces.otlp.json.gz  Level 1 Logged     ✗  no logged propensities  Level 2 Sampled    ✓  stored inputs: 12,000/12,000  Level 3 Estimated  ✓  context has 3 features: customer_tier, has_order_number, intent  Best available: Level 2$ agentope import --traces traces.otlp.json.gz -o decisions.jsonltraces.otlp.json.gz · 12,000 traces · 45,351 spans  chosen         9,351 from execute_tool spans · 2,649 __none__ (no tool call)wrote decisions.jsonl$ agentope sample decisions.jsonl -o samples.jsonl --n 64 --yessampled 12,000 (decision, policy) pairs → samples.jsonl$ agentope eval decisions.jsonl --samples samples.jsonl --outcomes helpdesk.csv \    --horizon 3d --target policies.py:lookup_when_order_numberEvidence level   L2 Sampled (N=64, α=0.5)Logging policy   0.637  realized mean outcomeTarget policy    0.711  SNIPS   (IPS 0.755)Difference      +0.074  95% CI [+0.058, +0.090]Verdict          ok

Excerpts of a real run on simulated traces. The rule would have resolved +0.074 more tickets (95%: +0.058 to +0.090); the true effect, known because it's simulated, is +0.080.

What you can evaluate

A change to one decision your agent makes, with everything after it held as it was.

  • Which tool the agent calls first

    first_move

    -search_kb

    +lookup_order

    # when there's an order number

  • Which model a request goes to

    router

    -large-model

    +small-model

    # for short questions

  • When to hand over to a person

    handover

    -answer_directly

    +escalate_to_human

    # for pro customers

  • A new prompt or model at that step

    system_prompt

    -prompts/v1.txt

    +prompts/v2.txt

    # same model, new instructions

Not a fit… yet:A whole new agentSeveral changes at onceFree-text answer quality

How much you can trust the answer

It depends on what your traces contain, and you climb by recording more. agentope always uses the strongest evidence you have, and says which.

  1. L3Estimated

    Weakest

    Only what your traces recorded

    Assumes: Nothing important is missing. Can't be checked, so you opt in.

  2. L2Sampled

    Strong

    Traces that include the messages

    Assumes: The model behaves as it did then. agentope checks.

  3. L1Logged

    Strongest

    A little exploration, recorded on the trace

    Assumes: Only that the coin flips were recorded.

  • oktrust it
  • weakuse with care
  • unreliableno answer, and what to record to get one
Evidence levels in detail

Works with your stack

The docs walk through the example step by step, with exploration, and compare what each evidence level gives on the same agent.