AgentOps · trace triage and evaluation

agent-trace-triage

Trace collection does not improve an agent until failures become labeled examples with evidence and ambiguous cases reach a reviewer.

Public · synthetic demo

The problem

Forcing every failed trace into a label creates confident evaluation noise.

Error strings, step budgets, allowed tools, repeated calls, and supplied expected answers can support deterministic labels. Anything outside those rules should remain unresolved.

Input

Seventeen synthetic execution traces with structured failure evidence.

Total traces: 17
Failed traces: 13
Successful traces: 4
Ambiguous traces: 2
Traces with secondary label: 2

The money shot

The largest bucket stays a count, not a vague diagnosis.

Failures

tool_api_error

Share of failures

23.1%

Human review

Malformed arguments, unresolved failures, and non-termination each account for 2 traces and 15.4%. Single-trace buckets remain marked low confidence.

How it's verified

Every label carries the field or step that fired the rule.

Trace	Primary label	Evidence
c001	non_termination_loop	loop count 10 against threshold 6
f003	timeout	9000 ms against budget 5000 ms
f008	wrong_final_answer	expected 42, received 17
r002	needs_review	vague error matched no rule

Honest limitations

Rules cannot understand task semantics they were never given.

No model judges the traces; error vocabularies and tool allow-lists are illustrative.
Wrong-answer detection requires a supplied expected answer.
Ambiguous failures are systematically under-classified by design.
Flat CSV input loses the step detail needed for some rules.

Direct evidence earns a label; everything else earns review. jigonyoo.com · Back to hub