Worked example
A support agent's first move, from exploration to a verdict, with real agentope output.
Guides
A support agent's first move, from exploration to a verdict, with real agentope output.
Something wrong or missing? Open an issue.
This walkthrough takes the path with the strongest evidence: the agent lets a small share of conversations try a random option. The home page shows the same agent left unchanged, with agentope asking the model again instead.
A support team runs an agent on qwen2.5-7b-instruct, served with vLLM. Every command and output on this page is real agentope output, produced by examples/support_agent/run.py. The traces are simulated, so the true answer is known and you can see whether agentope gets it right.
At the start of every conversation, the model chooses one of four first moves: search the knowledge base (search_kb), look up the order (lookup_order), hand over to a person (escalate_to_human), or answer directly (__none__).
The team suspects a plain rule would resolve more tickets: look up the order whenever the customer gives an order number, and search the knowledge base otherwise. An A/B test would take weeks of live traffic. They want an answer from conversations they already have.
One change to the agent: wrap the first move in agentope.explore. In 10% of conversations it takes a random option from the three the team is happy to take at random. It never hands over to a person at random, because people are expensive. The coin flip is recorded on the model's span; the other 90% run exactly as before.
import agentope as ao
# The model proposes the first action, as before.
tool = model_choice(conversation)
# New: in 10% of conversations, take a random option the team is happy
# to take at random, and record the coin flip on the model's span.
tool = ao.explore(
"tool_choice",
llm_choice=tool,
safe_candidates=["search_kb", "lookup_order", "__none__"],
candidates=["search_kb", "lookup_order", "escalate_to_human", "__none__"],
epsilon=0.1,
join_key=conversation.id,
log=lambda record: span.set_attributes(ao.span_attributes(record)),
)After 14 days there are 12,000 conversations, exported by the OpenTelemetry Collector. Content capture is off, so there are no prompts or messages, only span attributes. Here is the first conversation that was explored: the model wanted to look up order, and the coin flip chose to search articles instead.
invoke_agent support-agent│ gen_ai.conversation.id=conv-48235│ app.intent=order_status · app.has_order_number=true · app.customer_tier=standard├─ chat qwen2.5-7b-instruct ← the decision│ model wanted lookup_order · agentope.explored=true · agentope.propensity=0.333├─ execute_tool search_kb ← what ran└─ chat qwen2.5-7b-instruct ← final answer
{
"traceId": "30803889fa6197748d118e3781728a07",
"spanId": "ef44c0d53ee4da5a",
"parentSpanId": "7989e9d083a4e629",
"name": "chat qwen2.5-7b-instruct",
"kind": 3,
"startTimeUnixNano": "1792391881010046976",
"attributes": [
{
"key": "gen_ai.operation.name",
"value": {
"stringValue": "chat"
}
},
{
"key": "gen_ai.request.model",
"value": {
"stringValue": "qwen2.5-7b-instruct"
}
},
{
"key": "gen_ai.request.temperature",
"value": {
"doubleValue": 0.7
}
},
{
"key": "gen_ai.response.finish_reasons",
"value": {
"arrayValue": {
"values": [
{
"stringValue": "tool_calls"
}
]
}
}
},
{
"key": "agentope.decision_id",
"value": {
"stringValue": "conv-48235:1"
}
},
{
"key": "agentope.chosen",
"value": {
"stringValue": "search_kb"
}
},
{
"key": "agentope.explored",
"value": {
"boolValue": true
}
},
{
"key": "agentope.propensity",
"value": {
"doubleValue": 0.3333333333333333
}
},
{
"key": "agentope.llm_choice",
"value": {
"stringValue": "lookup_order"
}
},
{
"key": "agentope.safe_candidates",
"value": {
"arrayValue": {
"values": [
{
"stringValue": "search_kb"
},
{
"stringValue": "lookup_order"
},
{
"stringValue": "__none__"
}
]
}
}
}
]
}The helpdesk exports whether each ticket was resolved, with the conversation id as the key:
$ head -4 helpdesk.csvjoin_key,t,value conv-48213,2026-10-21T06:00:32Z,0 conv-48214,2026-10-20T10:36:54Z,1 conv-48215,2026-10-20T21:17:05Z,1
check reports which kinds of evidence the traces support. It needs no outcomes and calls no model.
$ agentope check --traces traces.otlp.json.gz \ --context intent=app.intent,has_order_number=app.has_order_number,customer_tier=app.customer_tiertool_choice · 12,000 decisions found in traces.otlp.json.gz Level 1 Logged ✓ 1,211/12,000 with logged propensities (1,211 explored) Level 2 Sampled ✗ no stored inputs (content capture off) → turn on content capture Level 3 Estimated ✓ context has 3 features: customer_tier, has_order_number, intent (needs --accept-unverifiable) Level 4 Modeled – not available yet (v0.4) Candidates: from agentope.candidates on 100%; __none__ included Best available: Level 1 Warnings: • Without stored inputs, LLM targets (prompt or model changes) can't be evaluated, and only the tool and toolset action maps work
Level 1 is available because of the exploration. Level 2 isn't: without message content, the model can't be asked again on what it saw.
$ agentope import --traces traces.otlp.json.gz \ --context intent=app.intent,has_order_number=app.has_order_number,customer_tier=app.customer_tier \ -o decisions.jsonltraces.otlp.json.gz · 12,000 traces · 45,196 spans tool_choice 12,000 decisions chosen 12,000 from agentope.chosen candidates from agentope.candidates on 12,000/12,000 propensities 1,211 explored (ε=0.1) · 10,789 without context intent, has_order_number, customer_tier stored inputs none (content capture off) → Levels 1 and 3 available wrote decisions.jsonl
Each conversation's first model call becomes a decision. The chosen action comes from the recorded coin flip, so what ran is exactly what was logged.
A policy is a Python function from the situation to probabilities over the options.
"""Policies the support team wants to evaluate before changing the agent."""
def lookup_when_order_number(context, candidates):
"""Look up the order whenever the customer gives an order number."""
if context["has_order_number"]:
return {"lookup_order": 1.0}
return {"search_kb": 1.0}
def escalate_pro(context, candidates):
"""Send every pro customer to a person; otherwise as above."""
if context["customer_tier"] == "pro":
return {"escalate_to_human": 1.0}
return lookup_when_order_number(context, candidates)$ agentope eval decisions.jsonl --outcomes helpdesk.csv --horizon 3d \ --target policies.py:lookup_when_order_numberagentope · tool_choice · target: policies.py:lookup_when_order_number ──────────────────────────────────────────────────────────── Evidence level L1 Logged (ε-exploration) Decisions 1,211 (rewarded 1,211 · expired 0 · 0.0%) 10,789 without logged propensities excluded Logging policy 0.629 realized mean outcome (all 12,000 decisions) Target policy 0.695 SNIPS (IPS 0.694) Difference +0.066 95% CI [+0.022, +0.110] Diagnostics ESS 403 (33% of n) · max weight 3 · min propensity 0.333 support violations 0 Verdict ok Warnings: • 10,789 of 12,000 decisions have no logged propensity and were excluded; the estimate covers only the included decisions This estimate assumes: • Logged propensities are the probabilities actually used when deciding • Whether a decision was explored did not depend on its context (true when agentope.explore made the coin flip) • Which tool is called matters, not its arguments or wording (action map: tool) • Only tool_choice changes; later steps behave as they did under the logging policy • An outcome missing within 3d means outcome = 0
Under the rule, 69.5% of tickets would have been resolved, against 62.9% today: 6.6 points more, with a 95% interval from +0.022 to +0.110.
Because the traces are simulated, the true difference is known: +0.094, inside the interval.
The second policy hands every pro customer to a person. The exploration never handed over at random, so 23.5% of that policy's choices have no evidence behind them.
$ agentope eval decisions.jsonl --outcomes helpdesk.csv --horizon 3d \ --target policies.py:escalate_proagentope · tool_choice · target: policies.py:escalate_pro ──────────────────────────────────────────────────────────── Evidence level L1 Logged (ε-exploration) Decisions 1,211 (rewarded 1,211 · expired 0 · 0.0%) 10,789 without logged propensities excluded Logging policy 0.629 realized mean outcome (all 12,000 decisions) Target policy no trustworthy estimate (SNIPS 0.673 · IPS 0.540 · difference +0.044, shown for diagnosis only) Diagnostics ESS 324 (27% of n) · max weight 3 · min propensity 0.333 support violations 23.5% of target mass (284 decisions) Verdict unreliable • support violations carry 23.5% of target probability (limit 1%) Warnings: • 10,789 of 12,000 decisions have no logged propensity and were excluded; the estimate covers only the included decisions This estimate assumes: • Logged propensities are the probabilities actually used when deciding • Whether a decision was explored did not depend on its context (true when agentope.explore made the coin flip) • Which tool is called matters, not its arguments or wording (action map: tool) • Only tool_choice changes; later steps behave as they did under the logging policy • An outcome missing within 3d means outcome = 0 To strengthen this: • The target puts 23.5% of its probability on actions the logging policy never takes; include them in exploration, or keep the target within the logged actions
agentope shows no estimate instead of a guess, and says what to log to make the question answerable. The raw number, +0.044, happens to land near the true +0.075, but nothing in the data could have told you that.
agentope.randomize, so the outcome reflects everything it does.uv run python examples/support_agent/run.py