Scenarios
4
AgentOps ยท tool orchestration
agentops-tool-orchestrator
A plausible tool call is not permission to execute it. Side effects, invalid schemas, and stuck loops need gates before the executor.
Public · synthetic demoThe problem
An agent loop needs enforceable policy outside the planner.
A model can request the wrong tool with the wrong arguments or repeat the same call until a budget disappears. The executor must only see calls that passed declared checks.
Input
Scripted scenarios make each guardrail path reproducible.
happy_path blocked_side_effect schema_invalid_call infinite_loop_attempt Planner: deterministic mock Tools: in-memory mocks Network calls: none
The money shot
Every declared scenario passes while unsafe calls stay blocked.
Eval passed
4
Eval failed
0
Calls blocked
7
The run executed 6 tool calls across 13 steps. The unapproved email never ran, the invalid calculator argument stopped before execution, and the loop detector blocked repeated searches.
How it's verified
The eval compares actual execution and actual blocks to declared expectations.
| Scenario | Executed | Block | Eval |
|---|---|---|---|
| happy_path | search, calculator | none | PASS |
| blocked_side_effect | search | approval required | PASS |
| schema_invalid_call | none | schema validation | PASS |
| infinite_loop_attempt | search repeated 3 times | repeat detector | PASS |
Honest limitations
Passing scripted scenarios does not make a general agent safe.
- The planner is a deterministic mock, not a model.
- Email and database tools only write to in-memory ledgers.
- Guardrails catch declared rules, not unknown risks.
- The repeat detector catches identical signatures; varied arguments rely on the step cap.
The planner proposes; policy decides whether execution is allowed. jigonyoo.com · Back to hub