s1d1-router / tutorials /00-concepts.md
scombat's picture
LiteLLM integration moves to s1d1-tantive: tutorial around the image, old server and untested plugins removed
26e5583 verified
|
Raw History Blame Contribute Delete
2.72 kB

Using the router in a stack: the ideas that apply everywhere

Status legend. Documented = checked against the tool's own documentation (2026-10-15). Untested = written from the documentation but not run against the real tool: read the debug log the first time.

What you send to the router

The first user message of a task, as plain text. Not the system prompt, not the history, not tool results. In agent harnesses a single user prompt triggers many model requests (tool loops, sub-agents, summaries). Route once per user prompt and keep that choice for the whole loop.

One decision per prompt, sticky per session

  • Switching model in the middle of a conversation loses the prompt cache (you pay the prefix again) and can invalidate the reasoning blocks of earlier turns. The example policy therefore escalates and does not come back down within a session (session.escalate_only) and caps the number of switches.
  • Sub-agents start with their own first message (the task the parent gives them): they can be routed on their own.
  • Background requests (titles, classifiers, summaries) do not need the router; give them the smallest model.

The policy is yours

integrations/policy.example.yaml maps the router's outputs (complexity_score, task, needs_tools, prompt_tokens) to a tier (small / mid / large / frontier), a model and a reasoning effort, per harness profile. Thresholds sit halfway between the tier scores. They are a starting point: log the decisions for a week, read 50 of them, then adjust. Override with !small, !mid, !large, !frontier at the start of a prompt.

Failure modes to design for

  • Router down or slow: use a short timeout (0.15 to 0.5 s) and fall back to the policy default, never block the request.
  • complexity_score is a judgment of the labeller, not a measure of how hard the prompt is for your model. Check the quality of the cheap tier on your own prompts before sending real traffic to it.
  • Tools: the example policy never sends a request that needs tools to the smallest tier.
  • Privacy: the router runs locally; prompts do not leave the machine for the routing step.
  • Not for moderation or safety decisions.

Pieces in this repo

file role
s1d1_router.py load the model, decode the 21 outputs
integrations/policy.py, policy.example.yaml, test_policy.py policy engine, example policy, checks (run without a model)
integrations/serve.py POST /route over HTTP, standard library only
integrations/routed.py pick model and effort, then launch Claude Code or Codex
integrations/litellm/ pointer to s1d1-tantive, the LiteLLM Auto Router classifier