roadius/peewee-mix-v1

Peewee mix-v1

This is the first published checkpoint of Peewee, a fast reflex layer for agent workloads. It answers typed questions about a piece of state in a single encoder forward pass, and returns calibrated probabilities rather than generated text. There are three question types:

• choice: one of a set of options.
• score: a position on an ordered rubric.
• noul: whether a condition holds.

Use it for tool routing, gating, triage and similar decisions where an LLM call is too slow or too expensive.

• Architecture: ModernBERT-large encoder with a two-layer decision head, fine-tuned from Laya's English checkpoint (convaiinnovations/laya), about 400M parameters.
• Training data: typed-decisions train (upsampled 4×) and Open-Jev release-v2-redistributable train, 92,827 items over 4 epochs. The full record is in train_meta.json.
• Context: 1,024 tokens per question, covering the question, its options and the state. Longer states are truncated and reported in usage["truncated"].

Use

Install Peewee from GitHub, then load the checkpoint:

pip install "peewee-decide[server] @ git+https://github.com/roadius2/peewee"

import peewee_decide

agent = peewee_decide.load("roadius/peewee-mix-v1")
res = agent.predict(
    {"ticket": "I was charged twice for my subscription this month"},
    {
        "team": {"type": "choice", "instructions": "Which team should handle this?",
                 "criteria": {"billing": "payments and refunds", "tech": "bugs and outages", "other": None}},
        "urgent": {"type": "noul", "instructions": "The customer needs a reply today."},
    },
)
print(res["answers"]["team"]["choice"], res["answers"]["team"]["confidence"])

As a service, with dynamic batching:

peewee serve --models roadius/peewee-mix-v1

A GPU with 8 GB is enough for one loaded model: about 5 GB at the default 32 questions per batch with 1,024-token inputs. On an RTX 5090, a case with five questions takes about 10–17 ms. On an Apple M5 Max (MPS), it takes about 130–300 ms.

Calibration: fit it to your workload

confidence is the probability of the answer the model reports. That makes a rule like act above 0.9, escalate below meaningful, but only once the temperatures fit your data.

• Built-in temperatures: rl_agent_config.json carries temperatures fitted on a balanced mix of typed-decisions and Open-Jev held-out data.
• Per-dataset files: calibration/ has temperatures for each dataset.
• Your own data: fit temperatures on a few hundred labelled cases from your workload. This only rescales confidence and never changes an answer:

peewee calibrate roadius/peewee-mix-v1 --data my_labelled_cases.jsonl --out my_calibration.json
peewee eval roadius/peewee-mix-v1 --data my_test_cases.jsonl --calibration my_calibration.json

agent = peewee_decide.load("roadius/peewee-mix-v1", calibration="my_calibration.json")

The case format is documented in peewee_decide/data.py.

Evaluation

Accuracy is measured against each dataset's reference answers. ECE is the expected calibration error of confidence against the same answers. Temperatures never change accuracy.

	typed-decisions test	Open-Jev test	Open-Jev OOD
accuracy	0.8005	0.9403	0.8310
ECE, built-in (balanced) temperatures	0.055	0.023	0.147
ECE, that dataset's own calibration file	0.021	0.011	—
TypeSafe Jev 1.13.0, accuracy (measured on the same cases)	0.7385	0.8104	0.8031
TypeSafe Jev 1.13.0, ECE	0.045	0.022	0.049

typed-decisions and Open-Jev test come from the same distributions this model was trained on. Open-Jev OOD is that dataset's out-of-distribution split, which makes it the fair comparison. Methods and raw reports are in the repository: BENCHMARKS.md and reports/.

Limits

• Overconfident on unfamiliar tasks. On Open-Jev OOD, mean confidence is 0.98 with the built-in temperatures, against 0.83 accuracy. Temperatures fitted on familiar data do not fix that. Calibrate on your own data before thresholding confidence on a new kind of task.
• Built-in temperatures follow the reference answers. They are fitted to the reference answers, not the typed-decisions teacher distributions. That sharpens probabilities: on typed-decisions, Brier against the teacher distribution rises from 0.057 to 0.083.
• English only. The model is trained and evaluated in English. Laya's multilingual checkpoint is the starting point for other languages.
• Short context. It sees at most 1,024 tokens per question and truncates longer states.
• Evaluated on benchmark data only. It has been measured on these datasets, not on real traffic. Validate it on your own data.

Attribution

Peewee began as a fork of Laya by Convai Innovations (Apache 2.0), and this checkpoint is fine-tuned from their English model. The encoder is ModernBERT-large (Apache 2.0).

The training data is typed-decisions (Apache 2.0) and Open-Jev (CC0-1.0). No outputs of TypeSafe's Jev were used in training or calibration; Jev appears here only as a measured comparison.

Peewee is not affiliated with, endorsed by or sponsored by Convai Innovations, TypeSafe or the Open-Jev project.

License: Apache 2.0.