Evidence levels
Where the probabilities of logged choices come from, and what each source assumes.
Concepts
Where the probabilities of logged choices come from, and what each source assumes.
Something wrong or missing? Open an issue.
agentope weights each logged decision by how much more (or less) likely the target policy was to make the same choice. That needs the propensity: the probability the logging policy gave its choice. agentope evaluates at one level at a time and never mixes them. By default it picks the strongest level the data supports; --level chooses explicitly.
The system randomized a little, and recorded the probability it used.
ao.randomize, which records the full distribution.ao.explore. With probability ε the action is drawn uniformly from safe_candidates and the coin flip is logged. Only explored decisions are used; on those the propensity is exactly 1/k.Assumes: the recorded probabilities are the ones used; the target puts no probability on actions the logger could never pick (checked).
Cost: ε of traffic acts randomly within the safe options, and the effective sample is about ε·n. The report estimates how long you'd need to log for a useful interval.
Nothing was randomized, but the stored model input is complete and the logging model can still be called. For each decision agentope asks the model again N times and counts its choices:
p̂(a | x) = (count(a) + α) / (N + α·k) # α = 0.5 by defaultAssumes: the stored input is everything the model saw; the model behaves today as it did when logging; N is large enough. All three are partly checked. See Sampling the model.
No logged or sampled propensities. A multinomial logit of the logging policy is fit from the logged context, cross-fitted by join key.
Assumes: every input that influenced both the choice and the outcome is in context. This can't be checked from data, so Level 3 refuses to run until you pass --accept-unverifiable. Its verdict is at best weak.
No propensities at all; an outcome model only. Planned for v0.4.
The support agent from the worked example, the same rule ("look up the order whenever there's an order number"), evaluated with less and more to work with. Real output on simulated traces; the green line is the true effect.
Traces without the messages
agentope models how the agent chose from what each trace recorded. Here the range just misses the true value: that model is slightly off, and nothing in the data can show it. So Level 3 only runs once you agree, and is never rated better than weak.
Traces with the messages (this example)
A tight range around the true value. It relies on the model behaving today as it did then, which the report checks.
Add exploration: 10% of conversations try a random safe option
Relies on nothing about the model. The range is wider because only the explored conversations count; more traffic or time narrows it. (Exploring changes slightly what the agent did, so its true value differs a little.)
tool = ao.explore(
"tool_choice", llm_choice=tool,
safe_candidates=["search_kb", "lookup_order", "__none__"],
epsilon=0.1, join_key=conversation.id,
log=lambda r: span.set_attributes(ao.span_attributes(r)),
)| Assumption | L1 | L2 | L3 | Checkable? |
|---|---|---|---|---|
| Correct propensities | logged | sampled | modeled | L1 yes · L2 partly · L3 calibration only |
| No direct effect (action map) | ✓ | ✓ | ✓ | no, a design choice |
| Overlap / support | ✓ | ✓ | ✓ | yes (ESS, support checks) |
| Complete stored input | ✓ | partly (hash) | ||
| Replay fidelity | ✓ | partly (model id, reproduction rate) | ||
| No unmeasured confounding | ✓ | no | ||
| Later steps unchanged | ✓ | ✓ | ✓ | no |
| Outcome join correct | ✓ | ✓ | ✓ | partly (match rates) |
The action map turns what the agent did into an action id, and every report states its assumption:
| Map | Action id | Assumes |
|---|---|---|
tool (default) | tool name, or __none__ | which tool is called matters, not its arguments |
toolset | tools called together, e.g. lookup+search | only the set of tools matters |
tool+args:<fields> | tool plus named argument fields | only the tool and those fields matter |
module:function | anything your function returns | whatever it ignores doesn't matter |