The reliability layer an agent's tool calls need — retry, timeout, circuit breaker, idempotency — with the numbers showing what each one buys. Runs the same workload through naive, retry-only, and reliable so the tradeoffs are visible, not asserted. Offline, deterministic, zero dependencies.
Public · synthetic demoDo nothing → one transient blip fails the task. Just add retries → recovers blips, but a call whose side effect landed but response was lost runs twice (double charge), and during an outage every task hammers the service that's already down. The fix is retry plus idempotency, timeout, and a breaker — and this measures each.
config success calls search-calls dup-charges -------------------------------------------------------------- naive 53/80 66% 80 40 0 retry-only 64/80 80% 139 88 5 reliable 63/80 79% 81 30 0
Retry recovers transient faults — naive 53 → 64/63. Worth it. · But retry without idempotency double-charges: 5 duplicate side effects (real money) when a "ghost" call — charge lands, response lost — gets retried. Reliable adds an idempotency key → 0. · The breaker stops hammering a down service: 88 → 30 calls, 66% less load. · Honest cost: reliable 63 vs 64 — the breaker rejects ~1 request right after recovery (half-open probe), a deliberate availability-vs-load tradeoff.
Bug #1 — retry WITHOUT idempotency double-charges the customer retry-only (no idempotency): server charged the card 2x <-- DOUBLE CHARGE reliable (idempotency key): server charged the card 1x <-- correct Bug #2 — a slow dependency hangs the agent (no timeout) 1.0s budget on a 5.0s call: aborted at 1.0s instead of hanging. Bug #3 — no breaker: every task hammers a service that's already down 12 tasks against a down service: upstream calls sent 3, short-circuited 9.
RetryPolicy exponential backoff + deterministic equal-jitter (no PYTHONHASHSEED dependence) CircuitBreaker CLOSED → OPEN → HALF_OPEN — fail-fast while open, one probe on recovery invoke(timeout=) client-side timeout: a call slower than the budget is aborted, spends only the budget idempotency key a retried side effect is de-duplicated server-side to exactly one execution
Time is a logical clock — no sleep, no threads — so every run is byte-identical and CI can pin the numbers.
Tests pin that retry recovers a transient fault, idempotency turns a retried ghost into one side effect, the timeout spends only its budget, and the breaker opens, short-circuits, and recovers half-open. CI regenerates the workload from a seed and fails if it isn't byte-identical.
· The dependency and its faults are simulated on a logical clock — a demonstrator of the behaviors, not a benchmark of your infra. Point the primitives at real calls for real numbers. · Single-threaded sim, so jitter's thundering-herd benefit is modeled but not measured here. The double-charge, timeout, and breaker effects are real and reproduced. · Idempotency assumes the server honors the key. If your downstream can't dedup, the key must live in your own write path — the kit shows where it goes, it isn't a substitute for it. · Real timeouts race a wall clock and can leave a call half-done — that's the 'ghost' case, which is exactly why idempotency pairs with timeout.