Introduction
What agentope does, the question it answers, and when to use it.
Teams change agents constantly: a new system prompt, a different model, a new tool, a different router. Most judge those changes with LLM-as-judge scores or slow A/B tests. agentope estimates a change's effect on real outcomes, such as resolved tickets or purchases, from the logs you already keep.
The question
For one decision point, if a different policy had made the choice instead of the one that ran, and everything else had stayed the same, what would the average outcome have been?
The headline number is the difference between that estimate and what your agent actually achieved, with a 95% interval. Everything else in the report explains how far to trust it.
Decision points
A decision point is one place where the agent picks from a few options:
| Decision point | Options | Who decides today |
|---|---|---|
tool_choice | search, lookup_order, escalate, __none__ | the LLM |
router | small-model, large-model | code |
system_prompt | v1, v2 | a flag |
__none__ is a real option: "the model answered without calling a tool" has to be a candidate, or tool-choice evaluation is biased. agentope evaluates one decision point at a time and holds everything else fixed, because importance weights multiplied over many steps have unusable variance.
Evidence, not just a number
Off-policy evaluation needs to know how likely each logged choice was. Agents rarely record that, and token logprobs are not a reliable substitute, so agentope gets these probabilities in one of three ways and labels the result with its evidence level:
- Logged: a little traffic was randomized, and the probability was recorded.
- Sampled: the logging model is asked again on the stored input, N times.
- Estimated: a model of the logging policy is fit from the logged context.
Each level states the assumptions it needs, and the report ends with a verdict: ok, weak or unreliable. See Evidence levels.
How it works
Your agent has already made thousands of choices, and you know how each one turned out. In some of them it happened to do what the changed version would do. agentope weighs those by how likely the agent was to make that choice, and that gives the changed version's result. Real rows from the worked example:
| Customer asked | Agent did | Rule would | Model | Solved |
|---|---|---|---|---|
| Refund | look up order | search articles | 3 of 64 | |
| Order status, order no. | answer directly | look up order | 14 of 64 | |
| Order status, order no. | look up order | look up order | 34 of 64 | |
| Order status, order no. | search articles | look up order | 20 of 64 | |
| Product question | search articles | search articles | 35 of 64 | |
| Refund, order no. | look up order | look up order | 28 of 64 | |
| Product question | hand to person | search articles | 2 of 64 | |
| Order status | look up order | search articles | 12 of 64 |
Compared with what teams do today
| Measures real outcomes | Time to an answer | Risk to users | Says when it can't answer | |
|---|---|---|---|---|
| LLM-as-judge | No, it scores the text | Minutes | None | No |
| A/B test | Yes | Weeks of live traffic | Some users get the worse version | Not applicable |
| agentope | Yes, the ones you already record | Minutes, from your logs | None | Yes, it refuses to guess |
agentope works from conversations that already happened, so nothing changes for your users while you find out.
What you can evaluate
One decision, changed
- Any level
Which tool the agent calls first
Look up the order before searching the help articles
- Any level
Which model a request goes to
Send short questions to the small model
- Any level
When to hand over to a person
Escalate every pro customer
- Needs the messages (Level 2+)
A new prompt or model at that step
Does the v2 system prompt pick better first moves?
Not a fit (yet)
A whole new agent, or several changes at once
Later steps are assumed to go as they did. Choose the agent per conversation with agentope.randomize instead.
The quality of free-text answers
agentope compares a few distinct options. Map answers to options first, or judge them some other way.
Outcomes you don't record
Every estimate is measured on an outcome you already track, linked by conversation.
Works with
Your traces
- OpenTelemetry GenAIThe gen_ai.* conventions for model calls and tool callstested
- OpenTelemetry CollectorExports from the file exporter, optionally gzippedtested
OpenInferenceArize's conventions for LLM and tool spanstested
Arize PhoenixOTLP exports of Phoenix tracesshould work
LangfuseOTLP exports of Langfuse tracesshould work
Your model (for Level 2)
“Tested” means covered by agentope's own tests. The others speak the same formats and should work; if you try one, an issue saying how it went helps. Outcomes come in as CSV or JSONL, and changes as Python functions, target.jsonl or a target.yaml.
Where to go next
- Quickstart: log, record outcomes, evaluate.
- From traces: start from an OpenTelemetry export.
- Reading a report: what every line means.