card: learning-curve findings; train-your-own first; shared data moves to keppy/evalroute-tasks

1594762 verified 29 days ago

5.39 kB

---
license: mit
language: en
library_name: transformers
pipeline_tag: text-classification
base_model: johnnyboycurtis/ModernBERT-small-v2
tags:
  - evalroute
  - task-routing
  - gonogo
datasets:
  - keppy/evalroute-flywheel
  - keppy/evalroute-tasks
---
# evalroute lane encoder v1

A small text classifier that maps a task request (the sentence you type to a coding agent)
to one of [evalroute](https://github.com/keppy/evalroute)'s nine lanes. It is the optional
first layer of evalroute's classifier, between the keyword rules and the LLM fallback.

**Status: opt-in.** On 19 real held-out tasks it ties the rules+LLM classifier and beats
keyword rules alone, but neither result is distinguishable at 0.95. The decision below was
made by [gonogo](https://github.com/keppy/gonogo), not by us.

```
evalroute install-encoder keppy/evalroute-lane-encoder     # or a local dir
evalroute route "write the launch blog post"                # card says: classification: encoder …
evalroute install-encoder --remove                          # back to rules (+ LLM if you have one)
```

Needs `pip install "evalroute[encoder]"` (torch CPU + transformers). Cold start without the
artifact installed does not import torch.

## Numbers

Eval set: 19 **real** tasks held out from the maintainer's own routing ledger and the
tier-a tasksets — 12 routine-coding, 4 dl-ml-research-engineering, 3 alignment-reasoning.
Never a paraphrase, never seed text. Lanes with fewer than 8 real rows have no eval rows.

| classifier | accuracy | note |
| --- | --- | --- |
| keyword rules only | 31.6% | what a standalone install has today; 10 of 19 hit no keyword |
| rules + LLM fallback | 68.4% | `qwen/qwen3-coder-flash`, one call per task |
| **encoder v1** | **63.2%** [41.0%, 80.9%] | $0 per route, no network |

gonogo `compare` (McNemar, paired by task id):
encoder vs rules: +31.6 pts, p = 0.11 — not distinguishable at 0.95.
encoder vs rules+LLM: −5.3 pts, p = 1.0 — tie.
gonogo `decide` (target 0.9): **ASSIST ONLY** — no confident subset reaches 90%.

Per lane (eval): alignment-reasoning 3/3, dl-ml 3/4, routine-coding 6/12. Every
routine-coding miss went to `hard-agentic-coding` or `prose` — two lanes the model has only
ever seen as paraphrases. Calibration: temperature 1.01, calibration error 0.18 (above
0.15 — treat `defer_below` 0.71 as a cut point, not a probability; it keeps 58% of rows at
≥ 90% accuracy).

## Training data (what is and is not here)

| source | rows | in this repo |
| --- | --- | --- |
| LLM paraphrases of each lane's seed text (training only) | 191 | `train_public.jsonl` |
| tier-a taskset prompts | 23 | `train_public.jsonl` |
| lane seed text (`match_hint`, keywords) | 18 | `train_public.jsonl` |
| maintainer's own ledger routes, pinned with `--lane` | 29 | **withheld** — task text never leaves a machine |

`split.json` lists train/eval ids (ledger ids are opaque route ids). `eval_predictions.jsonl`
has id, label, predicted, confidence — no text. `gonogo.txt` is the verdict as printed.

Five of nine lanes (hard-agentic-coding, long-doc-reading, web-research, math-first-principles,
orchestration) had **zero** real rows. v1 knows them only from paraphrases.

## What we learned after shipping v1 (read this before using it)

A learning curve on the same 19 eval rows, run the morning after:

| training set | eval | routine-coding |
| --- | --- | --- |
| paraphrases + seed + taskset, 0 real rows | 26% | 0/12 |
| … + 14 real rows | 68% | 8/12 |
| … + 29 real rows (= v1) | 47–63% | 4–6/12 |
| seed + taskset + 29 real rows, **no paraphrases** | 74% | 11/12 |

**Paraphrases hurt the lanes they outnumber** (the model learns what a lane sounds like when a
model describes it) and are the only thing holding up the lanes with no real rows. Real rows are
worth ~10× a paraphrase. And the maintainer's own ledger does not fill the empty lanes — 67% of
its labels are coding dispatches.

So the direction changed: **train your own encoder on your own ledger first** (below — free,
local, 30 s) and treat this shared model as a day-one floor. The shared encoder's data is now a
public, human-reviewed corpus — https://huggingface.co/datasets/keppy/evalroute-tasks — real tasks
with asserted lanes, published on purpose, no paraphrases. v2 will be trained from it, with
paraphrases capped to the real-row count and only for lanes that still have none. If you want
to improve the shared model, contribute reviewed rows there, especially for the five empty lanes.

## Train your own

```
evalroute export-cases --out cases.jsonl --tasksets <dir> --seed-text
python scripts/split_cases.py --in cases.jsonl --train train.jsonl --eval eval.jsonl --split split.json
python examples/evalroute_lane_encoder.py --train train.jsonl --eval eval.jsonl --out my-encoder   # thomas
evalroute install-encoder my-encoder
```

v1 trained in 30 s on a laptop CPU. The thomas recipe runs anywhere torch runs; `--backend
modal` exists for larger sets. Compare yours against the shipped one on your own held-out
routes with gonogo before trusting it.

## Files

`model.safetensors`, `config.json`, `tokenizer*.json` — ModernBERT-small-v2 fine-tune, 9 labels.
`label2id.json`, `temperature.json`, `metrics.json` — thomas CONTRACT §4 artifact.
`train_public.jsonl`, `split.json`, `eval_predictions.jsonl`, `gonogo.txt` — the evidence.

	---
	license: mit
	language: en
	library_name: transformers
	pipeline_tag: text-classification
	base_model: johnnyboycurtis/ModernBERT-small-v2
	tags:
	- evalroute
	- task-routing
	- gonogo
	datasets:
	- keppy/evalroute-flywheel
	- keppy/evalroute-tasks
	---

	# evalroute lane encoder v1

	A small text classifier that maps a task request (the sentence you type to a coding agent)
	to one of [evalroute](https://github.com/keppy/evalroute)'s nine lanes. It is the optional
	first layer of evalroute's classifier, between the keyword rules and the LLM fallback.

	Status: opt-in. On 19 real held-out tasks it ties the rules+LLM classifier and beats
	keyword rules alone, but neither result is distinguishable at 0.95. The decision below was
	made by [gonogo](https://github.com/keppy/gonogo), not by us.

	```
	evalroute install-encoder keppy/evalroute-lane-encoder # or a local dir
	evalroute route "write the launch blog post" # card says: classification: encoder …
	evalroute install-encoder --remove # back to rules (+ LLM if you have one)
	```

	Needs `pip install "evalroute[encoder]"` (torch CPU + transformers). Cold start without the
	artifact installed does not import torch.

	## Numbers

	Eval set: 19 real tasks held out from the maintainer's own routing ledger and the
	tier-a tasksets — 12 routine-coding, 4 dl-ml-research-engineering, 3 alignment-reasoning.
	Never a paraphrase, never seed text. Lanes with fewer than 8 real rows have no eval rows.

	\| classifier \| accuracy \| note \|
	\| --- \| --- \| --- \|
	\| keyword rules only \| 31.6% \| what a standalone install has today; 10 of 19 hit no keyword \|
	\| rules + LLM fallback \| 68.4% \| `qwen/qwen3-coder-flash`, one call per task \|
	\| encoder v1 \| 63.2% [41.0%, 80.9%] \| $0 per route, no network \|

	gonogo `compare` (McNemar, paired by task id):
	encoder vs rules: +31.6 pts, p = 0.11 — not distinguishable at 0.95.
	encoder vs rules+LLM: −5.3 pts, p = 1.0 — tie.
	gonogo `decide` (target 0.9): ASSIST ONLY — no confident subset reaches 90%.

	Per lane (eval): alignment-reasoning 3/3, dl-ml 3/4, routine-coding 6/12. Every
	routine-coding miss went to `hard-agentic-coding` or `prose` — two lanes the model has only
	ever seen as paraphrases. Calibration: temperature 1.01, calibration error 0.18 (above
	0.15 — treat `defer_below` 0.71 as a cut point, not a probability; it keeps 58% of rows at
	≥ 90% accuracy).

	## Training data (what is and is not here)

	\| source \| rows \| in this repo \|
	\| --- \| --- \| --- \|
	\| LLM paraphrases of each lane's seed text (training only) \| 191 \| `train_public.jsonl` \|
	\| tier-a taskset prompts \| 23 \| `train_public.jsonl` \|
	\| lane seed text (`match_hint`, keywords) \| 18 \| `train_public.jsonl` \|
	\| maintainer's own ledger routes, pinned with `--lane` \| 29 \| withheld — task text never leaves a machine \|

	`split.json` lists train/eval ids (ledger ids are opaque route ids). `eval_predictions.jsonl`
	has id, label, predicted, confidence — no text. `gonogo.txt` is the verdict as printed.

	Five of nine lanes (hard-agentic-coding, long-doc-reading, web-research, math-first-principles,
	orchestration) had zero real rows. v1 knows them only from paraphrases.

	## What we learned after shipping v1 (read this before using it)

	A learning curve on the same 19 eval rows, run the morning after:

	\| training set \| eval \| routine-coding \|
	\| --- \| --- \| --- \|
	\| paraphrases + seed + taskset, 0 real rows \| 26% \| 0/12 \|
	\| … + 14 real rows \| 68% \| 8/12 \|
	\| … + 29 real rows (= v1) \| 47–63% \| 4–6/12 \|
	\| seed + taskset + 29 real rows, no paraphrases \| 74% \| 11/12 \|

	Paraphrases hurt the lanes they outnumber (the model learns what a lane sounds like when a
	model describes it) and are the only thing holding up the lanes with no real rows. Real rows are
	worth ~10× a paraphrase. And the maintainer's own ledger does not fill the empty lanes — 67% of
	its labels are coding dispatches.

	So the direction changed: train your own encoder on your own ledger first (below — free,
	local, 30 s) and treat this shared model as a day-one floor. The shared encoder's data is now a
	public, human-reviewed corpus — https://huggingface.co/datasets/keppy/evalroute-tasks — real tasks
	with asserted lanes, published on purpose, no paraphrases. v2 will be trained from it, with
	paraphrases capped to the real-row count and only for lanes that still have none. If you want
	to improve the shared model, contribute reviewed rows there, especially for the five empty lanes.

	## Train your own

	```
	evalroute export-cases --out cases.jsonl --tasksets <dir> --seed-text
	python scripts/split_cases.py --in cases.jsonl --train train.jsonl --eval eval.jsonl --split split.json
	python examples/evalroute_lane_encoder.py --train train.jsonl --eval eval.jsonl --out my-encoder # thomas
	evalroute install-encoder my-encoder
	```

	v1 trained in 30 s on a laptop CPU. The thomas recipe runs anywhere torch runs; `--backend
	modal` exists for larger sets. Compare yours against the shipped one on your own held-out
	routes with gonogo before trusting it.

	## Files

	`model.safetensors`, `config.json`, `tokenizer*.json` — ModernBERT-small-v2 fine-tune, 9 labels.
	`label2id.json`, `temperature.json`, `metrics.json` — thomas CONTRACT §4 artifact.
	`train_public.jsonl`, `split.json`, `eval_predictions.jsonl`, `gonogo.txt` — the evidence.