Evaluate agent decisions before go-live.
Find out whether a change to how your agent decides would actually help your users, using the conversations you already have. Add exploration for the strongest evidence.
$ uv tool install "agentope[llm]"1Check what's possible
2Import the traces
3Ask the model again
4Evaluate the change
$ agentope check --traces traces.otlp.json.gztool_choice · 12,000 decisions found in traces.otlp.json.gz Level 1 Logged ✗ no logged propensities Level 2 Sampled ✓ stored inputs: 12,000/12,000 Level 3 Estimated ✓ context has 3 features: customer_tier, has_order_number, intent Best available: Level 2$ agentope import --traces traces.otlp.json.gz -o decisions.jsonltraces.otlp.json.gz · 12,000 traces · 45,351 spans chosen 9,351 from execute_tool spans · 2,649 __none__ (no tool call)wrote decisions.jsonl$ agentope sample decisions.jsonl -o samples.jsonl --n 64 --yessampled 12,000 (decision, policy) pairs → samples.jsonl$ agentope eval decisions.jsonl --samples samples.jsonl --outcomes helpdesk.csv \ --horizon 3d --target policies.py:lookup_when_order_numberEvidence level L2 Sampled (N=64, α=0.5)Logging policy 0.637 realized mean outcomeTarget policy 0.711 SNIPS (IPS 0.755)Difference +0.074 95% CI [+0.058, +0.090]Verdict ok
Excerpts of a real run on simulated traces. The rule would have resolved +0.074 more tickets (95%: +0.058 to +0.090); the true effect, known because it's simulated, is +0.080.
What you can evaluate
A change to one decision your agent makes, with everything after it held as it was.
Which tool the agent calls first
first_move
-search_kb
+lookup_order
# when there's an order number
Which model a request goes to
router
-large-model
+small-model
# for short questions
When to hand over to a person
handover
-answer_directly
+escalate_to_human
# for pro customers
A new prompt or model at that step
system_prompt
-prompts/v1.txt
+prompts/v2.txt
# same model, new instructions
Not a fit… yet:A whole new agentSeveral changes at onceFree-text answer quality
How much you can trust the answer
It depends on what your traces contain, and you climb by recording more. agentope always uses the strongest evidence you have, and says which.
L3Estimated
WeakestOnly what your traces recorded
Assumes: Nothing important is missing. Can't be checked, so you opt in.
L2Sampled
StrongTraces that include the messages
Assumes: The model behaves as it did then. agentope checks.
L1Logged
StrongestA little exploration, recorded on the trace
Assumes: Only that the coin flips were recorded.
- oktrust it
- weakuse with care
- unreliableno answer, and what to record to get one
Works with your stack
reads traces from
asks models through
The docs walk through the example step by step, with exploration, and compare what each evidence level gives on the same agent.