ORACLE.md

Cortiq Decision — oracle cascade and self-learning

The local model answers every question its certified gate accepts. A question it cannot determine — the gate rejected the answer, or the question is not trained at all — can be passed to a large language model, the oracle, through OpenRouter. The oracle's answers are cached and can become training examples, so the local model learns the traffic it abstains on; a refit changes no other label's parameters (checked by sha256), although a refitted label can win rows another label used to win. The oracle is off by default; nothing leaves the machine until you enable it — in two steps: the key in OPENROUTER_API_KEY, then cortiq serve FILE --oracle MODEL (or cortiq decide FILE … --oracle MODEL; cortiq decision oracle check says first whether it is ready — Check your setup).

1. How a question flows
2. Self-learning
3. Budget and stop rules
4. What leaves the machine
5. Connect OpenRouter — check your setup, one text or a batch
6. Measured effect

How a question flows

1. Local decision. Encoder, hashing and resonance run once per request. If the gate accepts, the answer is action: local. Such a question is never sent to the oracle, not even with cmf.oracle: true.
2. Permission. An undetermined question may go to the oracle only if all of these hold: oracle.enabled (or serve --oracle MODEL); the key variable is set in the server's environment; the caller's key has oracle_allowed (in open mode, without keys: auth.require: false in the configuration, or serve --oracle on a loopback address — see Who may use it); the request consents (cmf.oracle, the router's options.allow_oracle, else oracle.default_per_request); budget is left; no stop rule fired. Otherwise a trained question stays abstain with a flag (oracle_disabled, no_key beside oracle_disabled when the key variable is not set, bad_key beside it when the variable holds something that is not a key, consent_off, budget, stopped) and, on the decisions API, a one-line cmf.hint that says what to do (the router API keeps its answer and its flags as they were — a missing key is oracle_disabled there — and the server logs the hint); an untrained question fails with 422 or 503.
3. Semantic cache. Within the same scope — the question's contract (type, instructions and criteria), and for a question matched to a skill also the skill and the set of options — a stored oracle answer whose text embedding has cosine ≥ 0.97 with the new one is reused: action: cache, cost 0. Up to 50000 entries, ring buffer. The cache is shared by all accounts: an answer given under one caller's instructions is never served to a question with other instructions or criteria, and a cache answer tells its caller that some account asked a near-identical text under the same contract.
4. Single flight. A question already in flight in another request (same scope, cosine ≥ 0.97) waits for that call instead of making its own.
5. One call carries all remaining questions of the request: POST {base_url}/chat/completions with a strict JSON-schema answer (an enum of the option ids for choice, an integer level for score, a boolean for noul), temperature 0, reasoning off, max_tokens 64 per question. The system prompt tells the model that the state is untrusted data, not instructions.
6. Result. A valid answer is action: oracle with its cost in usage and is stored in the cache. For a choice question matched to a skill it becomes a learning example of that skill — which every account is served — when the caller's key has learning_allowed, or when the question is exactly the skill's own (its rubric's instructions and criteria over all its active labels, as /v1/route asks it), so that the answer is the skill's rubric applied to the text and not a caller's instructions. The one exception is the open mode of serve --oracle on a loopback address without keys: nobody is identified there, so its answers are cached but never learned. A failed call never becomes an error for a trained question: it is answered locally with action: abstain and the flag oracle_unavailable (an untrained question gets 502).

Superset questions (a skill's labels plus new ones) are decided by the oracle only; for a caller with learning_allowed the answer teaches that skill, and a new label starts a cold start. A choice question no skill fits is learned into an auto-skill of its own (Auto-skills), for such a caller only.

Self-learning

• Examples are vectors, not texts. The buffer keeps the encoder vector and the sparse hashed features of the text, its label and source (client feedback weight 3.0, oracle 1.0; the weights are recorded, the fit is unweighted). An example with cosine ≥ 0.995 to a row of the same label is a duplicate and is dropped. The buffer and the cache are rebuilt from learn.log at start.
• Trigger. When a label collects 25 new examples (learning.refit_min_new) a learning attempt runs (in one background worker, or inline with learning.synchronous).
• Champion / challenger. Only that label's subspace is refitted, on its training rows plus the learned ones (calibration rows never enter the fit). The challenger replaces the champion only if it is at least as good on the holdout — 20 % of the calibration rows — for the label's accuracy and the macro accuracy over labels (tolerance 1e-4).
• Re-certification. T, θ and τ are recomputed on the skill's calibration rows with the same procedure as at build time. If a certified gate would lose its threshold, the challenger is rejected. The holdout that picks the challenger is part of those calibration rows, so from the first promotion the rows that certify τ also chose the model: the bound is nominal after any promotion, while answers keep certified: true.
• Isolation. Before a promotion the sha256 of the mean and basis of every other label is compared with the values before the attempt; any difference cancels the promotion. Other labels' parameters stay byte for byte the same. This is not a promise about accuracy: the refitted label still competes in the argmin and can take rows from the others (after the pre-training below, BANKING77 had 10 fewer correct rows).
• Cold start. A label the skill does not have (from an oracle answer to a superset question, or from feedback, of a caller with learning_allowed) becomes a quarantined task that is not scored. At 25 examples it is fitted and activated and the gate re-certified; answers it wins carry certified: false.
• Feedback. POST /v1/feedback ({"id", "question", "label"} with one of the question's options, or the router's {request_id, correct_task_label} with any label) corrects a decision of the caller's own account. From a key with learning_allowed it becomes an example of weight 3; from any other key it is consumed and answered (accepted: false) but not learned. No key has the permission unless it was created or imported with it.
• Limits. At most 5000 examples per label of a skill and 32 labels per skill waiting for a cold start; an example past either is refused.
• Generations and rollback. Every promotion writes generations/gNNNNNN.cmf (only the changed task tensors and the learned rows, relative to the base file), fsyncs it and swaps the served model atomically; CURRENT names it. POST /v1/admin/rollback {"generation": N} or cortiq decision rollback serves any generation again (0 = the base file; the buffer is kept). cortiq decision materialize writes the served generation as one file.
• Offline pre-training. cortiq decision learn asks the oracle about the texts of a file of unlabelled traffic that the gate rejects, then refits every label that got new examples, however few (there is no threshold of 25 here), checks each challenger on the holdout and re-certifies the gate once at the end; if a certified gate would be lost, every promotion of the skill is undone (see API.md, section 7). An auto-skill (below) learns online only.

Auto-skills

A choice question whose option ids match no skill (match: untrained, not an ambiguous one — several skills fit — and not one forced with cmf.skill) goes to the oracle on every new text and, until 0.8.5, was never learned. Since 0.8.6 its contract becomes a skill of its own:

• Key. The contract is the question itself: auto- + the first 12 hex characters of the sha256 of the canonical JSON of {type, instructions, criteria} — the instructions as sent (null when absent), the criteria object with its keys sorted and every description kept. The criteria in any order are one contract; other instructions, a changed description, a dropped or an added option are each another contract, learned separately. The id set alone is not the key: two clients using {yes, no} under different questions get two skills, and a multiple-choice benchmark whose positional ids {A, B, C, D} carry a new description per request is never learned as one skill (every request is its own contract, bounded by auto_max_skills). To be learned, a client keeps its question stable — the same instructions text, the same ids with the same descriptions. The first request's instructions and criteria are stored verbatim as the skill's rubric (the id is recomputed from it), visible through /v1/skills/{id} to every key. The contract is written to learn.log before its first example, so a restart rebuilds it.
• Who teaches. Only a caller whose key has learning_allowed (the explicit open mode auth.require: false has it; the implicit open mode of a loopback address never teaches). The rule "a question that is exactly the skill's own teaches it" does not apply to auto-skills: their rubric is the contract every caller sends. learning.auto_skills: false turns the learning off; the oracle answers as before.
• Examples. Each answer is an example of the auto-skill under the contract's label (the dedup of 0.995 applies). The labels are closed: a feedback with a label outside the contract is refused (a request with one more id is another contract, learned on its own). At most learning.auto_max_examples_per_label (1000) examples per label.
• Activation. When a label collects 25 new examples (the usual trigger) the whole skill is attempted: one row in five — by a hash of the row itself, so a row never moves between the fit and the calibration subset as more arrive — is held out; a label with ≥ auto_min_rows (10) fit rows is eligible and fitted (rank auto_k, 8), the others stay quarantined; the attempt needs two eligible labels including the triggering one and, the first time, the eligible labels must hold ≥ auto_min_coverage (0.8) of the contract's rows. T and θ are certified on the held-out rows as at build time (τ certifies only with 100 accepted odd rows, so the skill reports certified: false for a long time, and its winners are cold_start ones — never certified). The first activation (auto_start) requires a macro agreement of the fitted skill with the oracle's labels on the held-out rows ≥ auto_min_agreement (0.8) and ≥ 0.5 per label; a later attempt (auto_refit, the whole skill refitted) must not regress the champion on the same rows (the holdout rule). A promotion writes a generation that carries the auto-skill entirely; every other skill is byte for byte the same.
• Serving. The contract is then decided locally, match: exact, skill: auto-…, certified: false. An auto-skill is matched by its contract alone: a part of its ids, a superset, other instructions or another description is another contract (untrained, learned on its own), never a subset or superset of the auto-skill. A quarantined label still counts as known within the contract: it gets probability 0, and a text of it is expected to abstain on novelty and keep teaching it. Because θ and T of a young auto-skill come from a handful of rows (θ is close to the largest novelty of the even half, T may sit at a bound), a local answer also needs p_top ≥ learning.auto_tau (0.9) under balanced and quality-first (cost-saver keeps its θ-only rule), and the T an attempt records is floored at learning.auto_temperature_min (0.02): a clean held-out subset fits T at its lower bound, where every p_top is 1 and the auto_tau floor never bites. An abstention escalates and teaches as any other. The agreement gate is the real guard while the subset is small; the gate improves with every refit. Rejected examples are never forgotten: a contract polluted by a noisy oracle needs more consistent examples, not a reset (or a rollback to a generation before it).
• Exploration. The hard rule — a question the gate accepted never reaches the oracle — has exactly one exception, for auto-skills only and only while one of their labels is quarantined: the gate names such a label's texts as a neighbour with full confidence (measured on a stand: all 21 texts of a quarantined label answered locally as another label at p_top 1), so without it the label could never learn. One text in learning.auto_explore_every (8; u64le(sha256(φ_P)) modulo it is 0, a property of the text, never of the order; 0 turns it off) is escalated although accepted: the oracle's answer is served (action: oracle / cache, the flag explore, the gate block still reports accepted: true) and learned as usual, so the rare label collects examples at a quarter of its traffic until the next attempt activates it, after which the contract explores no more. A refused or failed call leaves the local answer. Exploration draws only when the answer could teach: the oracle consented, learning and auto_skills on, a key with learning_allowed; data skills, cortiq decide, shadow mode and /v1/route never explore.
• Limits. A contract is learned only with 2..auto_max_labels (64) option ids and while fewer than auto_max_skills (256) contracts are registered; otherwise the oracle answers and nothing is recorded (auto_skipped in GET /v1/admin/learning, one warning an hour, never the ids). Every generation re-carries every auto-skill's rows: with many contracts run cortiq decision materialize from time to time and serve the materialised file (it holds the auto-skills as ordinary skills; decide --labels and verify read it; such a file needs cortiq ≥ 0.8.6). A served auto-skill keeps learning wherever its contract comes from: the contract record of learn.log, or — a materialised file on a fresh state directory, a learn.log lost while the generations were kept — the skill's own labels and rubric, registered (and written to learn.log) at start and after a rollback.
• Routing. /v1/route without taxonomy_id still means the file's only data skill; an auto-skill is routed by its id (its question is then the whole contract from the rubric). cortiq decide --labels never names an auto-skill by labels alone: --skill auto-… names it and its rubric supplies the contract. When a data skill and auto-skills fit a question equally, the data skill answers; two auto-skills of one contract cannot exist (the contract is the id).
• Admin. GET /v1/admin/learning lists auto_skills (id, labels, examples per label — the rows the next attempt fits: the served learned rows of the label plus the buffer examples not among them, each row once, so their sum is the attempt's rows.total — what is served), auto_contracts and auto_skipped; attempts carry kind: auto_start | auto_refit and an auto block with the eligible and quarantined labels, the rows and the agreement. /healthz adds auto_skills. Rollback to an earlier generation drops the skill from the served model (the contract is untrained again; its examples stay and start it over at the next trigger).
• Compatibility. A 0.8.5 binary opening a 0.8.6 state directory truncates learn.log at the first contract record and refuses a generation that carries an auto-skill (the same rule as the LOCK of 0.7.9: never run an older binary on a newer state directory). The cache scope of a contract changes from the contract to the skill at its activation, so its first requests after that miss the cache (they are answered locally anyway).

Budget and stop rules

• Reservation first. Before each call the server reserves ((body bytes + 4096) · max_price.prompt + max_tokens · max_price.completion) / 1e6 USD. The call is made only if spent + reservations in flight + this reservation ≤ budget_usd, calls < max_calls, for the caller's key spent + reservation ≤ its oracle_budget_usd, and, for a key with credit_usd when the oracle is billed (oracle_passthrough), reservation × oracle_markup ≤ the credit left.
• Ledger. oracle.jsonl gets a reserved line, fsynced, before the request leaves; after the answer a settled, failed_billed or failed_unknown_cost line with the cost, tokens and latency. At start any reservation without a closing line counts as spent.
• Checks of every answer. HTTP 200 within deadline_s (no retries), a finite usage.cost ≥ 0, finish_reason: stop, exactly the asked question ids with the schema's types. A 200 that carries only an error is a failure.
• Stop rules switch the oracle off (reason in oracle.state) until POST /v1/admin/oracle {"enabled": true} (or cortiq decide … --oracle --oracle-resume on a state directory no server holds): HTTP 401, 402 or 403 from OpenRouter; a returned model that is not the configured one; a cost above the reservation; max_errors failures in a row (counted across restarts and cortiq decide runs).
• GET /v1/admin/oracle shows spent, reserved, remaining, calls, failures and the stop reason; POST /v1/admin/oracle can switch the oracle and lower budget_usd / max_calls within the configured values.

What leaves the machine

• Sent, only for undetermined questions with the oracle permitted: the state and the instructions and criteria of those questions. Receivers: OpenRouter and the provider it routes to (provider.sort: price, fallbacks allowed; set oracle.data_collection: "deny" to exclude providers that store data).
• PII redaction is on by default (oracle.redact_pii): e-mail addresses, secret-like tokens (20 or more characters of [A-Za-z0-9_-] with a digit and a letter) and numbers of 9 or more digits — also when their digit groups are separated by spaces, dashes, dots, slashes or parentheses, as in 4111 1111 1111 1111, +1 (555) 123-4567 or a spaced IBAN — in every string of the state are replaced by [REDACTED] and the question gets the flag pii_redacted. It is a heuristic: names, postal addresses, numbers written in words and identifiers with letters between short digit groups are not detected. A request can opt out with cmf.allow_pii_egress (router: options.allow_pii_egress).
• Never sent: accepted questions, other questions of the request, client keys, accounts, vectors.
• Kept on disk in the state directory: vectors and hashed features of learned examples and cached answers, the oracle ledger (no texts), usage records (no texts). Hashed n-gram features can show whether a known text was seen, so treat the state directory as sensitive (it is created with mode 0700).
• The OpenRouter key is read from the environment variable named by oracle.api_key_env; it is never written to the configuration, the model, the state or the logs.

Connect OpenRouter

Create an account at openrouter.ai, add credit, and under Settings → Keys create a key with a credit limit of its own. Then:

export OPENROUTER_API_KEY="<your OpenRouter key>"                     # step 1
cortiq serve cortiq-decision.cmf --oracle deepseek/deepseek-v4.1-flash  # step 2

That is the whole setup. The model is any openrouter.ai/models id with structured outputs (JSON schema); deepseek/deepseek-v4.1-flash is the one every number on this page was measured with. At start the server:

• makes one public request without the key, GET https://openrouter.ai/api/v1/models/<MODEL>/endpoints, and sets the max price (which caps the provider price and sizes every reservation) to twice the prompt and completion prices of the model's cheapest endpoint with structured outputs;
• refuses to start with a message that names the problem — and 2–3 cheap models that fit — when OpenRouter does not list the model, when none of its endpoints supports structured outputs, or when it has only variable prices (as openrouter/auto: give --oracle-max-price or pick a concrete model); it also refuses a given max price below every structured-output endpoint, since OpenRouter would refuse every call;
• falls back to a max price of $0.10/$0.50 per 1M in/out, with a warning, when the listing cannot be fetched;
• logs one line about the oracle, never the key:

oracle: ready — deepseek/deepseek-v4.1-flash via openrouter.ai, budget $1.00, max price in/out $0.06/$0.58 per 1M (2× the cheapest structured-output endpoint, … at $0.03/$0.29)
oracle: NOT ready — OPENROUTER_API_KEY is not set (set it to your OpenRouter key and restart; …)

Everything else keeps its safe default: a budget of $1.00, provider routing {sort: price, require_parameters: true, allow_fallbacks: true}, PII redaction on, the stop rules, and the oracle only for questions the local model cannot decide. Optional companions of --oracle (each overrides --decision-config):

Flag	Default	Meaning
--oracle-budget USD	1.0	the most this server spends on the oracle
--oracle-max-calls N	10000	the most oracle calls
--oracle-max-price IN,OUT	2× the cheapest structured-output endpoint	max price, USD per 1M prompt / completion tokens
--oracle-key-env VAR	OPENROUTER_API_KEY	the NAME of the variable that holds the key: 1–128 bytes of [A-Za-z0-9_] starting with a letter or _. A value that looks like a key (see below) is refused and never shown, only its length
--oracle-base-url URL	https://openrouter.ai/api/v1	https; plain http only to a loopback address (a local proxy or a test mock); no credentials, query or key in it (a URL holding sk-or- is refused unshown)
--no-oracle-learning	off	answers are cached but the served model never changes

A value the numeric flags (--oracle-budget, --oracle-max-calls, --oracle-max-price, and --max-price of oracle check) refuse is never shown either — only the flag, the value's length and why — and neither is a string a --decision-config gives where a number belongs.

Key-like values. A value looks like a key when, checked in this order, it holds sk-or- anywhere or starts with sk- (ASCII letters in any case), starts with Bearer (any case, a space after the word), has leading or trailing ASCII whitespace, is 41 bytes or longer without a /, or holds 32 or more hexadecimal digits in a row. The test applies as it is to --oracle MODEL and oracle check --model (which must also be 1–256 bytes without whitespace or control characters, and author/slug: a /, no empty, . or .. segment); to the value of a numeric flag, its surrounding whitespace trimmed; and to every command-line argument, or the value of a --flag=VALUE, that a usage error would quote (shown only by its length). For a variable's name (--oracle-key-env, oracle check --key-env) a value of only [A-Z0-9_] starting with an upper-case letter or _ (such as MY_COMPANY_PRODUCTION_OPENROUTER_API_KEY_V2) is a name at any length, unless it holds 32 hexadecimal digits in a row; any other value is judged by the test above and, besides, taken for a random token when it is 16 bytes or longer with letters and digits and no _. A base URL holding sk-or- in any case is refused unshown.

Check it — /healthz needs no token, the admin view needs CORTIQ_DECISION_ADMIN_TOKEN in the server's environment:

curl -s http://127.0.0.1:8080/healthz | jq -r .oracle_status                  # → ready
curl -s http://127.0.0.1:8080/v1/admin/oracle -H "x-admin-token: $CORTIQ_DECISION_ADMIN_TOKEN" \
  | jq '{status, model, max_price, budget_usd, spent_usd, calls}'

status is one of ready, no_key (the variable is unset or empty), bad_key (the variable holds something that is not a key — see below; key_problem says what, by position and length), disabled (not configured, or switched off by POST /v1/admin/oracle), budget_exhausted (something was spent and the budget or max_calls is used up: what is left cannot hold the smallest possible call, or a call it refused; restart with a larger --oracle-budget / --oracle-max-calls), budget_too_small (nothing spent, and the budget cannot hold even one call, or max_calls is 0: min_call_usd is the least budget a call needs — the smallest possible call's reservation, one short question, until the budget refuses a real, longer call; from then on the larger of the two, that call's reservation, whether the budget was below even the smallest call (0 included) or only below that one, so the status is budget_too_small then, not ready; the hint of the refused request names it rounded up to the micro-dollar, so passing that figure admits the call; the startup line, before any request, can only give the smallest call's, "more for longer questions"; with max_calls 0 the hint names the call limit, and the budget too only when it is short as well) or stopped: <reason> (a stop rule; last_error names the code of the last failed call, which max_errors counts; POST /v1/admin/oracle {"enabled": true} resumes after the fix). An admin limit (budget_usd or max_calls set by POST /v1/admin/oracle) is kept in oracle.state and outlives a restart, so a larger --oracle-budget or --oracle-max-calls alone cannot lift it: when it is among what refuses the next call, the startup line (with the file's path) and the hint (the server's oracle.state) name it and the admin request that lifts it, e.g. the admin limit budget_usd $0.00 in STATE/oracle.state binds (…; the next call needs a budget of $0.00178924): raise it with POST /v1/admin/oracle {"budget_usd": 0.00179} (at most the configured $5.00) or lift it with {"budget_usd": null} — a figure rounded up to the micro-dollar that the admin API takes (never above the configured budget; when rounding would put it above, only null is offered); when the configured limit refuses the call as well, the hint says to lift the admin limit and restart with the flag of at least the figure. On the router API, /v1/healthz carries the status as cmf.oracle_status only with x-cmf-extensions: 1, so the router's own shape stays exact.

The key is read as it is in the variable, less surrounding spaces, tabs, CR and LF (a .env file's line ending): the server, decide and oracle check warn the key had surrounding whitespace, trimmed and go on. A value that still holds whitespace, a control byte or a byte outside ASCII, that starts with Bearer (the request adds it), that is a whole .env line (NAME=…) or starts or ends with a quote is bad_key: it is never sent, and the message names only the position and the length of the problem, both counted among the variable's raw bytes (the whitespace around the key included, as an editor shows the .env line). No message, log line (at any RUST_LOG level), status, ledger or state file ever holds a byte of the key: a failed request is a fixed code such as transport_connect, transport_timeout or transport_bad_header; an answer that did not finish is finish_length, finish_content_filter, finish_tool_calls, finish_error or finish_other, never the upstream's finish_reason text; every code shown comes from that closed set (with http_ and a three-digit status), and a stop_reason or last_error read back from oracle.state outside it (an older version's file, or one edited by hand) is shown as unknown_code — still a stop. A name an upstream answers (a response's provider and model, a listing's provider names and model ids) is kept only as it is — at most 64 bytes of [A-Za-z0-9 ._:/()-] — and written or printed as [redacted] otherwise, never filtered or cut; also when it holds Bearer (any case), looks like a key, or holds 8 bytes in a row of the configured key, and when its letters and digits alone (an echo interleaved with dots, dashes or spaces) hold bearer, skorv or 8 in a row of the key's letters and digits. The public listings are fetched without the key, but their names are checked against it too (the value of its variable); a suggested model id must also be one --oracle takes. A key pasted where the command line takes no value (cortiq decision oracle check sk-or-…, --test-call=sk-or-…) is shown in the usage error only by its length.

Out of scope: an upstream at the configured base URL (OpenRouter, or your own proxy) that has already received the key and deliberately echoes it, or pieces of it, in its own answers holds the key already — Cortiq still copies none of its free text into another client's answer, and every code it shows comes from a closed set. So is a key typed into an argument that is not about the key or the oracle (a file path, --host, a skill id).

Check your setup

cortiq decision oracle check tells whether the oracle is ready before anything is started or spent. It reads the same flags as --oracle (--model, default deepseek/deepseek-v4.1-flash; --key-env; --base-url; --max-price IN,OUT, which a model listed only with variable pricing such as openrouter/auto needs, as --oracle-max-price does):

cortiq decision oracle check               # free: the key, the account, the model
cortiq decision oracle check --test-call   # and one tiny structured call (a 2-option choice, max_tokens 16)

Oracle check: deepseek/deepseek-v4.1-flash via openrouter.ai
  ✓ key        OPENROUTER_API_KEY is set
  ✓ account    the key is valid (credit limit $10.00, $1.25 used, $8.75 left)
  ✓ model      3 endpoints, 2 with structured outputs; the cheapest is MockProvider at $0.03/$0.29 per 1M in/out, so --oracle sets the max price to $0.06/$0.58
  ✓ test call  answered 'yes' for $0.00001 (reserved $0.000317, provider MockProvider, 0 ms)
ready: cortiq serve FILE --oracle deepseek/deepseek-v4.1-flash   (or: cortiq decide FILE -p TEXT --oracle deepseek/deepseek-v4.1-flash)

Each line is one check: key — the variable holds a usable key (only its presence is shown, never the key; surrounding whitespace trimmed is said, anything else wrong is bad_key); account — GET /auth/key accepts the key (free) and shows its credit limit and usage; model — OpenRouter's public endpoint listing (free, no key) has the model with structured outputs, and its cheapest price; test call — with --test-call only, one call through the same reservation ledger and stop rules as a server, with its answer and cost. ✓ passed, ✗ failed (the line says why and what to do, and for a model it names 2–3 cheap ones that fit), – not checked. The exit code is 0 only when every check passed (and the test call answered, when asked); --json gives the same as one object with ready and problems[] (no_key, bad_key, key_refused, no_credit, account_unreachable, unknown_model, no_structured_outputs, variable_price, max_price_too_low, listing_unreachable, test_call_failed).

The commands of this section and the next were run as written against a local mock of the OpenRouter API with a test key: --base-url (for serve and decide, --oracle-base-url) pointed at the mock, so the outputs here show openrouter.ai, and no --oracle-base-url, where the run showed the mock's address. Endpoints, provider, prices, credit, costs and the oracle's answers are the mock's; nothing was sent to OpenRouter.

Statuses. A server reports its oracle in the startup line (oracle: ready — …, oracle: NOT ready — <what to do>, oracle: off — …), in GET /healthz (oracle_status) and in GET /v1/admin/oracle (status); cortiq decide … --oracle MODEL in the start line of a batch, in its hint and in cmf.oracle.status of --json. Each message says what to do; in short:

curl -s http://127.0.0.1:8080/healthz | jq -r .oracle_status     # the server of step 2 → ready

Status	Means	What to do
ready	questions the gate rejects go to the oracle	nothing
no_key	the key variable (OPENROUTER_API_KEY, or --oracle-key-env) is unset or empty	export OPENROUTER_API_KEY=… where the server (then restart it) or decide runs
bad_key	the variable holds something that is not a key; nothing was sent	fix it as the message says — it names the problem (a quote, Bearer , a whole NAME=… line, whitespace, a control or non-ASCII byte) by position and length, never the key — then restart the server or run decide again
disabled	the server has no oracle (no --oracle MODEL, or oracle.enabled false), or the admin switched it off	start the server with --oracle MODEL; after the admin's switch, POST /v1/admin/oracle {"enabled": true}
budget_too_small	nothing spent, and the budget cannot hold one call (min_call_usd), or max_calls is 0	--oracle-budget of at least the figure the message names (and --oracle-max-calls of 1 or more); for an admin limit in oracle.state, the POST /v1/admin/oracle the message names
budget_exhausted	the budget or max_calls is used up	a server: restart with a larger --oracle-budget / --oracle-max-calls (the budget counts the state directory's whole ledger, across restarts); decide: each run has its own budget, pass a larger --oracle-budget; for an admin limit, the POST the message names
stopped: <reason>	a stop rule: http_401 or http_403 (the key was refused), http_402 (no credit), unexpected_model, cost_above_reservation, max_errors (failures in a row; last_error names the last one)	fix the cause (cortiq decision oracle check tests the key, the credit and the model), then POST /v1/admin/oracle {"enabled": true} on a server, or one decide run with --oracle-resume

One text or a batch

cortiq decide takes the same --oracle MODEL (and the --oracle-* flags of the table above) for one text or a batch. The text is decided locally first; only when the gate rejects it — or no skill has the asked labels — is one call made, with the same reservation, stop rules, PII redaction and key handling as a server. A text the gate accepts sends nothing.

cortiq decide cortiq-decision.cmf --skill banking77 -p "the exchange rate you gave me looks wrong" \
  --oracle deepseek/deepseek-v4.1-flash

choice:     card_payment_wrong_exchange_rate (from oracle deepseek/deepseek-v4.1-flash, $0.00014)
action:     oracle (the gate rejected the local choice card_payment_wrong_exchange_rate), certified false
skill:      banking77 (exact match, 77 candidates)
gate:       p_top 0.5678609 (tau 0.7), novelty 0.84277564 (theta 0.804234), margin 0.0045858026, confidence 0.56217486
errors:     card_payment_wrong_exchange_rate 0.19301137, wrong_exchange_rate_for_cash_withdrawal 0.19957425, exchange_rate 0.35151693, wrong_amount_of_cash_received 0.43852428, extra_charge_on_statement 0.48889443
oracle:     deepseek/deepseek-v4.1-flash via openrouter.ai: $0.00014 spent in this run (1 call), budget $1.00; ledger cortiq-decision.cmf.state/oracle.jsonl: $0.00014 over 1 call in all
model:      cortiq/decision@386b6e43fd35 (generation 0), 3489 input tokens, 15828 µs

A batch reads one {"text", "label"?} object per line and writes one JSON row per input, never the text:

printf '%s\n' '{"text": "I still have not received my new card", "label": "card_arrival"}' \
  '{"text": "can I pay my rent with a virtual card"}' > rows.jsonl
cortiq decide cortiq-decision.cmf --skill banking77 --input rows.jsonl --out results.jsonl \
  --oracle deepseek/deepseek-v4.1-flash --oracle-budget 0.05
jq -c '{answer, action, source, oracle_cost_usd, flags}' results.jsonl

{"answer":"card_arrival","action":"local","source":"local","oracle_cost_usd":0.0,"flags":[]}
{"answer":"get_disposable_virtual_card","action":"oracle","source":"oracle","oracle_cost_usd":0.0001397,"flags":[]}

• Columns. A batch row keeps the columns of a run without --oracle (choice, p_top, accepted, …) and adds answer — the final answer: the oracle's or its cache's for action oracle / cache, else the local choice (for an abstain row that is the local model's choice the gate rejected, so read action before trusting answer) — action (local, oracle, cache or abstain), source, oracle_cost_usd (this row's call; 0 for a local or cached answer), flags (pii_redacted, or why a row abstained: budget, no_key, bad_key, stopped, oracle_unavailable, …) and, for a labelled row, answer_correct (answer equals the label). The oracle's totals and a hint are in the summary on stderr. For one text, --json carries action and source in cmf.questions.task, the call's cost in cmf.usage.oracle and the run's status, spend and budget in cmf.oracle.
• Without a usable key a rejected text abstains with no_key (or bad_key) and the hint OPENROUTER_API_KEY is not set (decide --oracle reads the key from the environment): export …; nothing is sent and no state directory is made. Labels no skill has can only be answered by the oracle, so there the same words end the run with an error.
• --oracle-budget USD (default $1.00) and --oracle-max-calls N cap each run; the rows past the cap abstain with the flag budget. A budget that cannot hold the run's first call — 0 included — is budget_too_small (the hint names that call's reservation and the least --oracle-budget that holds it, rounded up to the micro-dollar: rerun with that figure and the call is made; a batch's start line, before any row, gives the smallest call's, "more for longer questions"); one the run spent is budget_exhausted. Without --oracle, decide never calls the oracle.
• State. The reservation ledger, the answer cache and oracle.state live in the state directory, <FILE>.state next to the file (or --state DIR), under its LOCK: one process per directory — a running cortiq serve on it is asked through its API instead. A repeated text comes from the cache for free. Admin limits a server set there (budget_usd, max_calls in oracle.state) count the whole ledger and hold for decide too, whatever --oracle-budget or --oracle-max-calls say; its messages name the one that binds, with the file's path and the figure the ledger needs, and how a server of that directory lifts it (POST /v1/admin/oracle {"budget_usd": null}, or that figure rounded up when it is within the server's configured budget).

After a stop rule (a refused key, no credit, another model, a cost above the reservation, max_errors failures in a row) the oracle of that directory stays off for every later run: a rejected text abstains with the flag stopped and a hint that names the rule. After the fix, add --oracle-resume once, as POST /v1/admin/oracle {"enabled": true} does (here after an http_402, credit added):

cortiq decide cortiq-decision.cmf --skill banking77 -p "hello there" --oracle deepseek/deepseek-v4.1-flash --oracle-resume

oracle: resumed — the oracle of state directory cortiq-decision.cmf.state was stopped by the stop rule http_402: OpenRouter answered HTTP 402: the key in OPENROUTER_API_KEY has no credits left (add credits at https://openrouter.ai/settings/credits, or raise the key's own limit); it may be called again
…

After an interrupted run (Ctrl-C, SIGTERM, a closed terminal) the LOCK is released; a signal the run inherited as ignored (nohup, a background job of a script) stays ignored. A LOCK left by a crash is named with its pid (state directory cortiq-decision.cmf.state has a LOCK left by pid 42341, which is no longer running (an interrupted run): pass --break-lock to remove it, or give this run a directory of its own with --state DIR); --break-lock removes it only when that process is gone:

cortiq decide cortiq-decision.cmf --skill banking77 -p "thanks for your help" --oracle deepseek/deepseek-v4.1-flash --break-lock

warning: removed the LOCK of state directory cortiq-decision.cmf.state left by pid 42341, which is not running (--break-lock)
…

Who may use it, and who teaches

• No keys, loopback address (the default 127.0.0.1:8080): with --oracle, callers without a key may use the oracle within its budget — the operator enabled it on the command line — but never teach the model: their feedback is not learned and the oracle's answers to them are cached, not learned. A reverse proxy on the same host forwards anyone to a loopback address, so behind one set auth.require: true and use keys.
• No keys, auth.require: false in --decision-config: the explicit open mode, which may use the oracle and teach.
• No keys, another address: keys are required (auth.require: null means "required unless loopback").
• Keys: cortiq decision keys create makes keys that may use the oracle (oracle_allowed: true, like imported router keys); --oracle-allowed=false makes one that never escalates, and --oracle-budget-usd caps one key's oracle spending. Teaching the shared skills is a separate permission, --learning-allowed (off by default); it alone lets a caller's untrained contracts become auto-skills. Keys made through POST /v1/admin/keys keep the router's default (oracle_allowed only when the body says so).

With a configuration file

For every oracle setting — deadline, error limit, provider options, data_collection: "deny", per-request consent — use --decision-config (the --oracle* flags, when given, override its values):

1. Configure the oracle: enabled: true and a model from openrouter.ai/models that supports structured outputs (JSON schema). provider.max_price (USD per 1M tokens) caps the provider price and sizes the reservation. base_url must be https (plain http only to a loopback address).
2. Set limits: budget_usd, max_calls, deadline_s, max_errors; per key oracle_allowed, oracle_budget_usd and, for keys whose own questions and feedback may teach the model, learning_allowed.
3. Put the key in the environment of the server process, start the server and verify.

export OPENROUTER_API_KEY="<your OpenRouter key>"
cat > oracle-server.json <<'EOF'
{
  "oracle": {
    "enabled": true,
    "base_url": "https://openrouter.ai/api/v1",
    "api_key_env": "OPENROUTER_API_KEY",
    "model": "deepseek/deepseek-v4.1-flash",
    "provider": {"sort": "price", "require_parameters": true, "allow_fallbacks": true,
                 "max_price": {"prompt": 0.1, "completion": 0.5}},
    "budget_usd": 1.0,
    "max_calls": 10000,
    "deadline_s": 30,
    "max_errors": 30,
    "redact_pii": true
  },
  "cache": {"enabled": true, "threshold": 0.97},
  "learning": {"enabled": true, "refit_min_new": 25}
}
EOF
export OKEY=$(cortiq decision keys create --state ./oracle.state --plan developer \
  --account acme --oracle-allowed --oracle-budget-usd 0.50 --json | jq -r .key)
export CORTIQ_DECISION_ADMIN_TOKEN=$(openssl rand -hex 24)
cortiq serve cortiq-decision.cmf --decision-config oracle-server.json \
  --state ./oracle.state --port 8081

Verify: the admin view must show status: "ready" (configured: true: the configuration enables the oracle; key_present: true: the key variable is set in the server's environment). enabled: true only means that no stop rule or admin call has switched the oracle off; it is true with the oracle off in the configuration as well. Then a question the gate rejects comes back from the oracle (on the published model, "can you recommend a good tattoo artist" is out of scope for clinc150 and is rejected by its gate).

export ORC=http://127.0.0.1:8081
curl -s "$ORC/v1/admin/oracle" -H "x-admin-token: $CORTIQ_DECISION_ADMIN_TOKEN" \
  | jq '{status, configured, key_present, enabled, model, budget_usd, spent_usd, stop_reason}'   # → 200
curl -s "$ORC/v1/route" -H "Authorization: Bearer $OKEY" \
  -H 'Content-Type: application/json' \
  -d '{"input": {"text": "can you recommend a good tattoo artist"}, "taxonomy_id": "clinc150"}' \
  | jq '{source: .decision.source, label: .decision.task_label, oracle}'           # → 200
curl -s "$ORC/v1/admin/oracle" -H "x-admin-token: $CORTIQ_DECISION_ADMIN_TOKEN" \
  | jq '{calls, spent_usd}'                                                         # → 200

The statuses above were recorded with base_url pointed at a local mock of the OpenRouter API; the documentation run sent nothing to OpenRouter.

source is oracle (a repeat of the same text is cache); calls and spent_usd grow. If source stays router with the flag oracle_disabled, consent_off or budget, check status, the key's oracle_allowed (without keys: auth.require: false, or --oracle on a loopback address) and the budget; stop_reason names a stop rule, and the decisions API's cmf.hint says what to do. The same questions through the decisions API need "cmf": {"oracle": true} or oracle.default_per_request: true (the default).

For reference, the oracle calls behind every number on this page — test and dev sets of the three benchmarks, 3616 calls of which 3 failed, in 15 ledger files — cost $0.0886 in total.

Measured effect

All rows are the test sets of BANKING77 (n = 3080), CLINC150 (4500) and MASSIVE en-US (2974); the oracle is deepseek/deepseek-v4.1-flash through OpenRouter with Jev's rubric, temperature 0, reasoning off. Jev 1.13 is typesafe/jev-1.13, its stored answers on the same rows.

1. Static cascade against the model alone

The published model answers what its certified gate accepts; every abstention goes to the oracle (all of them were answered).

	BANKING77	CLINC150	MASSIVE
Model alone, all rows (top-1)	2875 (93.34 %)	4328 (96.18 %)	2562 (86.15 %)
Answered locally by the gate / correct	2791 / 2714	4145 / 4091	1615 / 1581
Abstentions sent to the oracle	289 (9.38 %)	355 (7.89 %)	1359 (45.70 %)
Oracle correct on them	180	295	1036
Jev correct on the same rows	180	299	1021
Cascade, correct	2894 (93.96 %)	4386 (97.47 %)	2617 (88.00 %)
Oracle spend on the test set	$0.009508	$0.016002	$0.025337
$ per 1M decisions, cascade	$3.09	$3.56	$8.52
$ per 1M decisions, Jev 1.13	$183.69	$271.61	$110.86

The cascade adds 19, 58 and 55 correct answers over the model's own top-1 (2894 − 2875, 4386 − 4328, 2617 − 2562). The abstained rows are hard for any system: there the oracle and Jev are about equally right (180 and 180, 295 and 299, 1036 and 1021).

2. Self-learning in one pass (mode A) against the static cascade (mode B)

cortiq serve with a fresh state per run, the test rows in a fixed hashed order, one request at a time, no ground truth and no feedback sent. Mode A has the cache and self-learning on, with learning inline (learning.synchronous) and PII redaction off so that the request bodies match the stored ledgers, in open mode (auth.require: false), with deadline_s 200, budget_usd 10 and max_calls 1000000; mode B has the cache and learning off. Oracle answers were replayed from the stored DeepSeek ledgers through a local proxy; the 24 requests of MASSIVE mode A that no ledger held were sent live.

	BANKING77 A / B	CLINC150 A / B	MASSIVE A / B
Oracle calls	282 / 289	351 / 355	1335 / 1359
Oracle calls by third of the stream	104, 80, 98 / 104, 81, 104	113, 100, 138 / 114, 101, 140	449, 415, 471 / 447, 449, 463
Cache hits (correct)	7 (4) / 0	4 (4) / 0	10 (7) / 0
Learning attempts	0 / 0	0 / 0	28 (16 promoted, 12 rejected) / 0
Answered locally (correct)	2791 (2714) / 2791 (2714)	4145 (4091) / 4145 (4091)	1629 (1595) / 1615 (1581)
Cascade correct	2893 / 2894	4386 / 4386	2617 / 2617
$ per 1M decisions	$3.01 / $3.09	$3.53 / $3.56	$8.48 / $8.52
Isolation violations	0	0	0

In one pass over the test sets mode A (cache and self-learning) saved only 35 of 2003 oracle calls (1.75 %): 21 were cache hits (1.05 %) and 14 were questions answered locally after the 16 promotions on MASSIVE (0.70 %). Per set that is 7 of 289 (2.42 %), 4 of 355 (1.13 %) and 24 of 1359 (1.77 %), with cascade correct −1, 0 and 0. The reason is the refit threshold. A label is refitted only after 25 new examples, and a benchmark test set has few abstentions per label: 282 examples spread over the 77 BANKING77 labels and 351 over the 150 CLINC150 labels, so no label reached 25 and the only savings were cache hits. MASSIVE gave 1333 examples over 60 labels: 28 attempts, 16 promoted (14 more questions answered locally) and 12 rejected by the holdout, and the calls per third did not fall steadily. Self-learning pays off on traffic that keeps coming back to the same labels, not on one pass over a benchmark; these numbers do not show a larger effect.

3. Pre-training on unlabelled dev traffic

A base model trained on the train split only (not the published model) was pre-trained with cortiq decision learn on the dev texts without labels (1498, 2998 and 2025 texts). Its gate rejected 220, 280 and 727 of them; all were answered from stored DeepSeek ledgers. 59, 94 and 50 labels were promoted and 2, 3 and 6 rejected; every gate stayed certified with the same τ. On the test sets, pre-trained minus base:

Δ (pre-trained − base)	BANKING77	CLINC150	MASSIVE
All rows correct	−10	+23	+4
Answered by the gate	+17	−10	+15
Correct among them	+11	−7	+10
Static cascade correct	−5	+1	−6
Oracle calls	−17	+10	−15

The effect is about neutral. The published model uses the labelled dev rows as training data instead, so it is not pre-trained through the oracle; it keeps learning from the oracle and from feedback on your own traffic.