Concepts

The layers

What each layer does, how to switch it on, and what we measured.

#Prompt

Adds a short fixed instruction to requests that offer tools: call the tool instead of answering from memory or working it out yourself, and answer normally if no tool is relevant. It costs a few input tokens and no extra model call, and it is skipped when the caller sets tool_choice: "none".

Flags: -tool-policy (always) or -tool-policy-auto (only for models where it was measured to help: the gpt-5 family today).

Measured: +6.4 points on gpt-5-nano and +16.2 on gpt-5-mini (BFCL multi-turn). No significant effect on gpt-4o-mini, gpt-4.1-mini or gpt-4.1.

#Tools

Guard checks each tool call against the tools the request offered and their JSON schemas. It repairs faults only when the fix is lossless: stripping a markdown fence, dropping prose around the JSON, removing trailing commas, reading empty arguments as an empty object. Anything else goes back to the model for another try, and a call that still fails is refused. -guard-pass-through lets it through instead.

Verify uses a small judge model to ask one question: does any offered tool apply to this request? Two modes:

  • gate: the model answered in plain text although a tool applies, so retry with a call required.
  • veto: the model made a call after the policy told it to, but no tool applies, so retry without the policy.

-verify-scope single (the default) consults the judge only when the conversation is one user request with nothing after it. Deep in an agent session a tool almost always applies, so the judge adds cost and no value there. Verdicts are cached.

Measured: judge plus guard +4.4 points on single-turn BFCL, restoring the irrelevance guardrail the policy alone weakened. On multi-turn runs the full stack was no better than the Prompt layer alone.

#Context

Keeps a long run inside the model's window. Old tool results become one-line placeholders that say how to fetch them again; the latest N stay in full. Optional summaries, a key-facts note, relevance-based keeping, and pruning of the tool list.

Flags: -context-keep-observations, -context-mask-step, -context-mask-within-turn, and others in the reference.

By default the whole latest user turn is protected, which suits chat. A long agent run is often one request followed by dozens of tool calls, and there the protected turn is the entire history. Turn on -context-mask-within-turn for those runs.

Measured: accuracy-neutral on BFCL. On a small overflow test (5 trials), the bare gateway hit the window error in 2 of 5 runs and finished none; with the layer no run hit the error and 3 finished. It did not help the model recall a page it could no longer see.

#Loop guard

Detects an agent going in circles: the same call again, a cycle of calls, the same message, long polling without progress, and streaks of unproductive results. It nudges first, then stops the run with a report. It understands polling and paging, so a healthy agent is not nudged to quit.

Flags: -loop-guard, -loop-warn-only, -loop-max-tool-calls, -loop-allow-repeat.

Measured: 44% fewer tool calls on tasks that cannot be completed, with no false stops on healthy tasks (held-out scenarios). No effect on models that do not loop.

#Model policy

Sends a request to a cheap model first and moves to a stronger one when another layer asks for a retry, the provider errors, or the response is empty.

Flags: -model-cheap, -model-strong, -model-escalate-on-no-call. Reasoning effort can be set with -reasoning-effort.

Measured: not yet shown to work. In a live test, gpt-5-nano first and gpt-4.1-mini on failure scored 75.5% against 87.5% for gpt-4.1-mini alone and cost about twice as much, because the cheap reasoning model emits far more hidden tokens per call. A non-reasoning cheap tier is untested. Lowering reasoning effort to minimal was 69% cheaper and 26 points less accurate.

#Memory

Keeps per-user memory as a temporal graph of facts: a subject, a predicate, an object, the source turn and a date. A newer fact supersedes an older one without deleting it. Retrieval combines keyword search, dense embeddings and a graph walk, adds the surrounding original messages, and lets a small model pick the evidence.

Flags: -memory-dir, -memory-model, -memory-embed-model, -memory-select-model, -memory-top-k, -memory-scope.

Measured: on held-out LoCoMo, +10.5 points over embedding search and +3.9 over Mem0 (given session dates). On LongMemEval it matched plain search at about half the context, which is a tie, not a win.