Concepts

How it works

One pipeline, many layers, and an engine that owns every retry.

#The pipeline

text
your agent -> Rakuvo gateway -> model provider (OpenAI-compatible)
|
|- Before hooks layers run in order and may edit the request
|- provider call
'- After hooks layers run in reverse order and may inspect the
response, ask for a retry, or add notes

A layer implements two methods, Before and After. Because After runs in reverse, the layer closest to the model sees the response first.

On the way in the layers run in this order: model policy, effort, memory, context, tool pruning, loop guard, prompt, guard, verify.

#The engine owns retries

A layer never retries a call itself. It sets a retry flag, and the engine re-runs the whole pipeline on a fresh copy of the original request, up to a limit. Two consequences:

  • Two layers cannot fight each other over a response.
  • A retry never applies a prompt edit twice.

#Everything is traced

Layers write a note for every action and report the tokens they spend on their own model calls (the judge, a summariser, memory extraction). Traces carry per-attempt tokens, cache hits, latency and the layer notes, so the cost of a layer is always visible. It is charged to the layer, not hidden in the bill.

#Fail open

A judge that is down, slow or confused never blocks a response. A layer acts at most once per request unless its page says otherwise.

#Streaming

The gateway accepts streaming requests. The layers run on the complete response, which is then sent to the client as an event stream, so a streamed reply arrives after the model has finished instead of token by token.