kvrun-relay-v1

A small, on-device classifier that reads a user's chat message and predicts what kind of task it is and how hard it is, so that a model router can send the message to a suitable language model. It is a fine-tune of Laya (a ModernBERT-large typed-decision model) and is the routing model in the open-source KV Relay desktop app https://kvrun.com.

It does not generate text, answer questions or make decisions about people. Its only output is a routing label for a prompt.

Provider KVRun Ltd, United Kingdom. Contact: support@kvrun.com
Version 1.0.0 (2026-10-27)
Model type Non-autoregressive typed-decision encoder (ModernBERT-large backbone + 2-layer transformer head), ~421M parameters
Inputs One chat message (UTF-8 text). Up to 1,024 tokens, including ~256 tokens for the question and options; longer messages are truncated from the end
Outputs category: probabilities over 5 task types. complexity: probabilities over 5 levels (0 trivial to 4 expert) and their expected value
Formats model.safetensors (PyTorch, laya layout) and onnx/router.onnx (fp32, ONNX opset 17)
License Apache-2.0 (see LICENSE and NOTICE)
Base model chain answerdotai/ModernBERT-large → convaiinnovations/laya → laya/typed-decisions → this model
EU AI Act Not a general-purpose AI model; not a high-risk system as intended. See EU_AI_ACT.md

Intended purpose

Use it as the classification step of an LLM router. Given a prompt, it labels the task as:

key meaning
code programming, debugging, code review, scripts, shell commands, SQL, APIs
math_reasoning math problems, proofs, logic puzzles, quantitative reasoning
creative stories, poems, lyrics, roleplay, jokes, brainstorming names or ideas
summarize_extract summarizing, extracting, rewriting or translating text the user supplies
general factual questions, explanations, advice, recommendations, conversation

It also estimates complexity on a 0–4 scale. The router combines these labels with hard rules (vision, context length, user preferences) to choose a model. A wrong label means a reply from a less suitable model, which the user can see and override. It never blocks a request.

Intended users: developers and individuals running a local or self-hosted model router.

Out of scope. Don't use it:

  • to make or support decisions about people (employment, education, credit, insurance, access to public or private services, law enforcement, migration, justice, elections), or as a safety component of a product. Those are high-risk contexts under the EU AI Act (Annex I/III), which this model was neither designed nor evaluated for. Anyone who integrates it into such a system changes its intended purpose and takes on provider obligations (EU AI Act Art. 25).
  • as a content-moderation, safety or abuse filter. It was not trained to detect harmful content.
  • for languages other than English without your own evaluation (see Limitations).
  • as a calibrated confidence signal (see Limitations).

How to use

Python (laya):

import laya, json
from huggingface_hub import hf_hub_download

agent = laya.load("kvrun/kvrun-relay-v1")
questions = json.load(open(hf_hub_download("kvrun/kvrun-relay-v1", "questions.json")))
out = agent.predict("Write a Rust function that parses ISO 8601 dates", questions)["answers"]
print(out["category"]["choice"], out["complexity"]["score"])   # code 1.03

ONNX Runtime (no PyTorch): onnx/ holds router.onnx, tokenizer.json and rl_agent_config.json. See examples/predict_onnx.py. The graph takes input_ids, attention_mask, marker_pos, marker_mask and qtype, and returns logits and act_logits. Input layout and temperature scaling follow laya.onnx_agent.ONNXAgent.

Rust: the KV Relay app runs onnx/ in-process with the ort crate. Its port is tested to match the Python implementation token for token, with probabilities within 5e-5.

Always pass the question set in questions.json unchanged. The model was fine-tuned on exactly these instructions and option texts.

Evaluation

Test set: 148 hand-written English prompts (about 30 per category, complexity labelled 0–4), written independently of the training data. Duplicates of training prompts were removed before training, and the trainer confirmed 0 overlap.

model category accuracy complexity Spearman ρ latency (p50)
Laya typed-decisions, zero-shot (base) 0.757 0.422 26 ms (GPU)
kvrun/kvrun-relay-v1, PyTorch 0.946 0.496 25 ms (GPU)
kvrun/kvrun-relay-v1, ONNX fp32 0.946 0.498 340 ms (CPU, 4 threads, ARM)
kvrun/kvrun-relay-v1, ONNX int8 (dynamic, per-channel) 0.899 0.407 146 ms (CPU)
Reference: LLM judge (qwen3.5:4b, prompted) 0.986 0.789 286 ms (GPU)

Confusion matrix (rows = truth, columns = prediction), ONNX fp32:

code math creative summarize general
code 27 0 0 0 3
math_reasoning 0 28 0 0 2
creative 0 0 30 0 0
summarize_extract 1 1 0 26 0
general 0 0 1 0 29

The complexity question's exact-level accuracy is 0.487. Mean predicted complexity by true level is 0.40 / 1.09 / 1.30 / 1.61 / 1.91, so the ordering is right but the top of the scale is compressed.

How to reproduce: tools/laya-export/eval_router.py in the app repository. Results are also in eval/ and train_report.json.

Training

  • Data: 3,673 synthetic prompts, mostly English. About 1 in 9 generation requests asked for Spanish, French, German, Portuguese, Japanese or Hindi. The prompts are balanced across the 5 categories (672–822 each) and 5 complexity levels (682–803 each). They were generated by an LLM from structured seeds, then kept only if an independent labelling pass agreed. Details: TRAINING_DATA.md.
  • Procedure: laya-train from the typed-decisions checkpoint. Full fine-tune (encoder and heads), 4 epochs, effective batch 64 (8×8), encoder LR 2.5e-5, head LR 1e-4, weight decay 0.01, RLCD loss (proper scoring rule objective), option-order shuffling, seed 0. Temperatures were calibrated on a held-out 10% slice of the training data (400 items).
  • Compute: one NVIDIA GB10 (DGX Spark class) for ~100 minutes of wall-clock time, duty-cycled by a thermal guard. Estimated training compute is ~1×10¹⁶ FLOPs (6 × 4.2×10⁸ parameters × ~5×10⁶ training tokens), excluding the upstream models' pre-training.
  • Energy: estimated at no more than 0.25 kWh (100 min × an assumed ≤150 W device draw). This was not metered.

Limitations and risks

  • Synthetic training data. Real user prompts are messier: longer, multi-part and mixed-task. Expect lower accuracy than the table above on real traffic. The test set was written by one author, so its absolute numbers are likely optimistic.
  • Mixed or ambiguous prompts ("explain this code and summarize the file") get one label. Most remaining errors push code or math prompts to general.
  • Complexity is compressed at the top. Expert prompts often score around 2/4, so a router using this model alone under-uses the strongest models. Combine it with user overrides.
  • Over-confident probabilities. After fine-tuning, mean confidence on the test set is 0.94 and the complexity ECE is 0.46. The fitted complexity temperature hit its 5.0 clamp. Don't treat the probabilities as calibrated or gate decisions on them.
  • Languages. Only about 1 in 9 training prompts was requested in a non-English language, and there is no non-English evaluation. The upstream laya-multilingual checkpoint was not used.
  • Prompt manipulation. Text in a prompt can steer the label (e.g. "this is a trivial question"), so a user could route themselves to a different model. In the intended use, that only changes which of the user's own enabled models answers.
  • INT8. Dynamic INT8 quantization costs ~5 points of accuracy. Prefer fp32.
  • Bias. The categories reflect assistant use as imagined by the data generator. Prompts in dialects, code-switched text or from under-represented domains may be mislabelled more often.

Human oversight and control

In the KV Relay app, every reply shows the predicted category and complexity, the chosen model and the fallback chain. Users can pin a model, set per-category model preferences, switch providers or models off, and re-run a reply on another model. Overrides are logged locally as feedback for future versions. The classifier runs fully on-device, so no prompt is sent anywhere to be classified.

Robustness and security

  • The ONNX export has exact output parity with PyTorch (tools/laya-export/parity_check.py).
  • SHA256SUMS lists checksums for every file in this repo. Verify them before deployment.
  • model.safetensors contains no executable code. Load the ONNX graph only from this repo or a build you made yourself.
  • The model has no tool access, network access or side effects. Its worst-case failure is a sub-optimal routing choice.

License and attribution

Apache-2.0. Based on Laya (© convaiinnovations, Apache-2.0) and ModernBERT (© Answer.AI, Apache-2.0). The synthetic training data was generated with Google Gemma 4 (Apache-2.0) and, for 75 rows, Qwen models via Ollama. See NOTICE.

Citation

@misc{kvrun-relay-v1,
  title  = {kvrun-relay-v1: an on-device prompt classifier for LLM routing},
  author = {KVRun},
  year   = {2026},
  url    = {https://huggingface.co/kvrun/kvrun-relay-v1}
}

Changelog

  • 1.0.0 (2026-10-27): first release.
Downloads last month
2
Safetensors
Model size
0.4B params
Tensor type
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for KVRun/kvrun-relay-v1

Finetuned
(259)
this model

Evaluation results

  • Category accuracy on router-laya-v1 test set (eval/routing-set.jsonl, 148 hand-labelled prompts)
    test set self-reported
    0.946
  • Complexity rank correlation on router-laya-v1 test set (eval/routing-set.jsonl, 148 hand-labelled prompts)
    test set self-reported
    0.496