julia-1-router

julia-1-router is a prompt router. Given a user prompt, it picks one model tier: small, medium, large or reasoning. It is a full fine-tune of niko0xdev/julia-1-zorch-triage, built on SupersonicLabs/Julia-1. It has a single decision head, route.

Status: target met on synthetic held-outs. On the fixed synthetic held-out sets, accuracy is above 95% and under-routing is below 2% on both heldout_combo and heldout_style. All evaluation here is synthetic; there are no real user prompts in the metrics below.

Base model	niko0xdev/julia-1-zorch-triage (runtime SupersonicLabs/Julia-1)
Head	route: small, medium, large, reasoning
Languages	Vietnamese (with and without diacritics), Korean, English
Training data	niko0xdev/julia-1-router-data (synthetic, version rt008)
Code	https://github.com/niko0xdev/julia-1-zorch (router training/eval scripts)
License	Apache 2.0

Tiers

Tier	Meaning
small	One step, closed answer, no reasoning: short translation, spelling fix, a simple fact, a unit conversion
medium	A few steps, short writing or code: an email, a summary, a short function, explaining a concept
large	Breadth and volume: system design, multi-file code, long documents, long-context analysis
reasoning	Depth: a chain of steps that must all be right, such as proofs, math, logic puzzles, subtle debugging

Tie-break policy used for labeling: when a task sits between two tiers, it gets the higher one.

Training

• Format: trainer rows {state, question, options, target}. The question is Which model tier should handle this prompt? and the options are ["small", "medium", "large", "reasoning"]. The prompt goes in state.title, with any attachment or context in state.description.
• Data: rt008 train split, 60,000 prompts: 15,000 per tier; vi 18,000, vi_nodiac 15,000, ko 15,000, en 12,000. They come from 174 task families. Held-out sets are unchanged copies of rt001 held-outs.
• Init: continued from the rt007 best checkpoint (itself continued from rt005 best over the earlier public rt002 weights).
• Selection: checkpoint chosen on the in-distribution dev split (1,200 prompts) by task_exact_match. The selected checkpoint is at optimizer step 500 (dev task exact match 1.0). That score is in-distribution and not a held-out result.
• Hyperparameters (training_config.yaml): 1 epoch, learning rate 1e-5, micro batch 1, gradient accumulation 16, warmup 0.08, weight decay 0.01, max length 256, evaluation every 500 steps, seed 20261208.
• Hardware: Google Colab NVIDIA L4.
• Inference policy: raw argmax over the four route logits. A confidence-based escalate-one-tier policy was tried on train-side OOD/dev only (never on held-out) and was not adopted, because the remaining misses were often confidently wrong or skipped more than one tier. See the dataset card and CONFIDENCE_ESCALATE.md in the training repo for the sweep notes.

Iteration history

Held-out sets were never edited. Train data was widened from held-out error analysis only (new train-only families; no held-out prompts or templates copied into train). Leak checks (family / title / style / fulltext overlap) were run each round.

Round	Notes	heldout_combo acc / under	heldout_style acc / under	Gate
rt001	First full synthetic set	(smoke / early)	(smoke / early)	No
rt002	Widened train families; first public push	87.58% / 3.75%	84.25% / 5.00%	No (published as first public checkpoint)
rt003	Continued data/training loop	91.92% / 3.83%	88.92% / 5.50%	No
rt004	More train-only families (VI-heavy)	95.00% / 4.33%	91.00% / 5.58%	No
rt005	More boundary families	95.83% / 2.17%	93.50% / 2.83%	No
rt006	Terse reasoning families; regressed	94.75% / 2.75%	92.33% / 3.08%	No
rt007	Dropped rt006 terse families; VI boundary families; init from rt005 best; forbidden-motif filter	95.75% / 2.25%	96.08% / 2.17%	No (under >= 2%)
rt008	Boundary families for remaining under/over; init from rt007 best	98.42% / 0.17%	96.00% / 0.17%	Yes

Evaluation (synthetic held-out sets)

There are no real user prompts in any evaluation. Both held-out sets are synthetic and frozen since rt001:

• heldout_combo: 20 task families (5 per tier) and the held-out 20% of every slot pool. None of these appear in training. Train-time wrapper styles are used.
• heldout_style: the same held-out families and pool items, wrapped in 5 styles that never appear in training.

Cost weights small 1, medium 3, large 10, reasoning 15 are assumed relative costs, not measured prices.

• Cost ratio vs gold: the total assumed cost of the predicted tiers divided by the total cost of the gold tiers.
• Overspend: the extra cost from over-routing, as a share of the gold cost.
• Under-routing: the predicted tier is cheaper than the gold tier.
• Over-routing: the predicted tier is more expensive than the gold tier.

Set	Prompts	Accuracy	Under-routing	Over-routing	Cost ratio vs gold	Overspend	Target met
heldout_combo	1,200	98.42%	0.17%	1.42%	1.007	0.97%	Yes
heldout_style	1,200	96.00%	0.17%	3.83%	1.024	2.52%	Yes

The target (accuracy above 95% and under-routing below 2% on each held-out set) was met on rt008.

Per-tier accuracy, heldout_combo: small 97.7% / medium 96.7% / large 99.7% / reasoning 99.7%. heldout_style: small 87.0% / medium 97.7% / large 99.3% / reasoning 100.0%.

Per-language accuracy, heldout_combo: en 99.2% / ko 99.7% / vi 98.3% / vi_nodiac 96.7%. heldout_style: en 97.9% / ko 95.0% / vi 96.1% / vi_nodiac 95.3%.

Weakest held-out families, heldout_combo: study_plan 92%, dockerfile 93%, yes_no_fact 93%. heldout_style: yes_no_fact 78%, word_count 83%, synonym 86%, timezone 93%.

Confusion matrix, heldout_combo (rows gold, columns predicted: small, medium, large, reasoning):

gold \ predicted	small	medium	large	reasoning
small	293	7	0	0
medium	0	290	10	0
large	0	1	299	0
reasoning	1	0	0	299

Confusion matrix, heldout_style:

gold \ predicted	small	medium	large	reasoning
small	261	32	0	7
medium	1	293	4	2
large	0	1	298	1
reasoning	0	0	0	300

Recommended use

• Default inference: argmax over the four route probabilities. No confidence escalate is applied in the published checkpoint.
• For high-stakes prompts where over-routing is cheaper than under-routing, you may still escalate one tier when the top probability is low. That policy was not tuned on held-out data and is optional.
• Remaining misses on heldout_style are mostly small-tier prompts (yes_no_fact, word_count, synonym) over-routed to medium or reasoning.

Usage

Load with the julia-1-zorch / Julia-1 routing stack (same layout as other julia-1 classification fine-tunes: model.safetensors, encoder/, tokenizer/, julia_config.json). Example sketch:

# Pseudocode: use your project's router load + predict helpers.
# pred = route_prompt(model, tokenizer, title=prompt, description=optional_context)
# pred in {"small", "medium", "large", "reasoning"}

Limitations

• Metrics are on synthetic held-outs only. Real traffic may differ.
• heldout_style small-tier accuracy is 87.0%; style shift still hurts some short factual prompts.
• Assumed tier costs are placeholders, not vendor prices.
• The public GitHub tree may lag the local router training scripts used for this run.

Files

• model.safetensors: fine-tuned weights (selected rt008 best)
• encoder/, tokenizer/, julia_config.json: runtime config matching Julia-1 classification layout
• training_config.yaml, training_history.json, validation_metrics.json: training record
• heldout_eval.json: full held-out metrics for rt008

Citation

If you use this model, please cite the Hugging Face repo niko0xdev/julia-1-router and the dataset niko0xdev/julia-1-router-data.