Getting started

Introduction

What agentope does, the question it answers, and when to use it.

Teams change agents constantly: a new system prompt, a different model, a new tool, a different router. Most judge those changes with LLM-as-judge scores or slow A/B tests. agentope estimates a change's effect on real outcomes, such as resolved tickets or purchases, from the logs you already keep.

The question

For one decision point, if a different policy had made the choice instead of the one that ran, and everything else had stayed the same, what would the average outcome have been?

The headline number is the difference between that estimate and what your agent actually achieved, with a 95% interval. Everything else in the report explains how far to trust it.

Decision points

A decision point is one place where the agent picks from a few options:

Decision pointOptionsWho decides today
tool_choicesearch, lookup_order, escalate, __none__the LLM
routersmall-model, large-modelcode
system_promptv1, v2a flag

__none__ is a real option: "the model answered without calling a tool" has to be a candidate, or tool-choice evaluation is biased. agentope evaluates one decision point at a time and holds everything else fixed, because importance weights multiplied over many steps have unusable variance.

Evidence, not just a number

Off-policy evaluation needs to know how likely each logged choice was. Agents rarely record that, and token logprobs are not a reliable substitute, so agentope gets these probabilities in one of three ways and labels the result with its evidence level:

  1. Logged: a little traffic was randomized, and the probability was recorded.
  2. Sampled: the logging model is asked again on the stored input, N times.
  3. Estimated: a model of the logging policy is fit from the logged context.

Each level states the assumptions it needs, and the report ends with a verdict: ok, weak or unreliable. See Evidence levels.

How it works

Your agent has already made thousands of choices, and you know how each one turned out. In some of them it happened to do what the changed version would do. agentope weighs those by how likely the agent was to make that choice, and that gives the changed version's result. Real rows from the worked example:

logged conversations, from the traces
Customer askedAgent didRule wouldModelSolved
Refundlook up ordersearch articles3 of 64
Order status, order no.answer directlylook up order14 of 64
Order status, order no.look up orderlook up order34 of 64
Order status, order no.search articleslook up order20 of 64
Product questionsearch articlessearch articles35 of 64
Refund, order no.look up orderlook up order28 of 64
Product questionhand to personsearch articles2 of 64
Order statuslook up ordersearch articles12 of 64
"Model" is how often the model makes the agent's choice when agentope asks it again. In 5,586 of 12,000 conversations the agent happened to do what the new rule would (highlighted). Their outcomes, weighted up where the model rarely makes that choice, estimate the rule: 71.1% resolved, against 63.7% today.

Compared with what teams do today

Measures real outcomesTime to an answerRisk to usersSays when it can't answer
LLM-as-judgeNo, it scores the textMinutesNoneNo
A/B testYesWeeks of live trafficSome users get the worse versionNot applicable
agentopeYes, the ones you already recordMinutes, from your logsNoneYes, it refuses to guess

agentope works from conversations that already happened, so nothing changes for your users while you find out.

What you can evaluate

One decision, changed

  • Which tool the agent calls first

    Look up the order before searching the help articles

    Any level
  • Which model a request goes to

    Send short questions to the small model

    Any level
  • When to hand over to a person

    Escalate every pro customer

    Any level
  • A new prompt or model at that step

    Does the v2 system prompt pick better first moves?

    Needs the messages (Level 2+)

Not a fit (yet)

  • A whole new agent, or several changes at once

    Later steps are assumed to go as they did. Choose the agent per conversation with agentope.randomize instead.

  • The quality of free-text answers

    agentope compares a few distinct options. Map answers to options first, or judge them some other way.

  • Outcomes you don't record

    Every estimate is measured on an outcome you already track, linked by conversation.

Works with

“Tested” means covered by agentope's own tests. The others speak the same formats and should work; if you try one, an issue saying how it went helps. Outcomes come in as CSV or JSONL, and changes as Python functions, target.jsonl or a target.yaml.

Where to go next

Something wrong or missing? Open an issue.