agentope
DocsGitHub

Getting started

IntroductionQuickstart

Concepts

Evidence levelsReading a report

Guides

Worked exampleFrom tracesSampling the model

Reference

CLIPython APIData formats

Reference

CLI

Every agentope command and option.

PreviousSampling the modelNext Python API

Something wrong or missing? Open an issue.

agentope 0.3.1·a liminal planes production
DocsGitHub
output
agentope check   what can be evaluated with this data (no LLM calls, no outcomes)
agentope import  turn OTLP JSON traces into decisions.jsonl
agentope sample  Level 2: sample the logging model (and LLM targets) on stored inputs
agentope eval    estimate how a target policy would have done on logged decisions

Exit codes: 0 report produced · 1 refused (a guard fired, or sampling declined) · 2 usage error.

check

shell
agentope check decisions.jsonl
agentope check --traces traces.otlp.json.gz --context intent=app.intent
OptionDefault
decisionsdecisions.jsonl, or use --traces
--tracescheck an OTLP JSON export instead
--decision-pointallonly this decision point
--n32samples per decision, for the cost estimate
--targetan LLM target (target.yaml), to include its sampling cost
--jsonmachine-readable output

Plus the trace import options with --traces.

import

shell
agentope import --traces traces.otlp.json.gz -o decisions.jsonl
OptionDefault
--tracesrequiredOTLP JSON export, optionally gzipped
-o, --outputrequireddecisions.jsonl to write
--decision-pointtool_choicename for the decisions
--inputs-dirinputs/ next to the outputwhere stored model inputs go when content was captured

sample

shell
agentope sample decisions.jsonl -o samples.jsonl --n 64 --base-url http://localhost:8000/v1
OptionDefault
decisionsrequireddecisions.jsonl with stored inputs
-o, --outputrequiredsamples.jsonl, appended to
--n32samples per decision and policy
--targetan LLM target to sample as well; repeatable
--no-loggingonly sample the targets
--base-urlAGENTOPE_BASE_URLOpenAI-compatible endpoint
--modelfrom the stored inputlogging model name on this endpoint, if it differs
--concurrency8parallel requests
--n-per-request1samples per request via the n parameter, if the server supports it
--sampleallonly a random subset of decisions
--seed0seed for --sample
--decision-pointallonly this decision point
--yesdon't ask before calling the LLM

The API key comes from AGENTOPE_API_KEY or OPENAI_API_KEY (optional for local servers).

eval

shell
agentope eval decisions.jsonl --outcomes outcomes.csv --horizon 7d --target policy.py:lookup_first
OptionDefault
decisionsdecisions.jsonl, or use --traces
--tracesevaluate straight from an OTLP JSON export
--outcomesrequiredoutcomes as JSONL or CSV
--horizonrequiredhow long an outcome still counts: 7d, 12h, 30m
--targetrequiredtarget.jsonl, an LLM target (target.yaml), or module:function / file.py:function
--samplessamples.jsonl from sample (Level 2, LLM targets)
--alpha0.5Level 2 smoothing
--decision-pointrequired when decisions cover several points
--levelstrongest available1, 2 or 3
--accept-unverifiableaccept Level 3's untestable assumption
--default-reward0outcome when none arrives within the horizon
--absent-means-defaulta missing outcome really means the default (success-only logs)
--seed0seed for the bootstrap and folds
--bootstrap2000bootstrap resamples
--folds5cross-fitting folds at Level 3
--jsonprint the report as JSON
-o, --outputalso write the JSON report to this file

Trace import options

Shared by import, and by check and eval with --traces.

OptionDefault
--maptoolaction map: tool, toolset, tool+args:<fields>, or module:function
--contextnonecontext features from span attributes: intent=app.intent,tier=app.tier
--join-keygen_ai.conversation.id, session.id, trace idattribute holding the outcome key
--decisionsfirstfirst model call per agent run, or all