assay-compiled-base

A small-tier decision model from the Assay project: typed questions (bool, choice, score) over a state produce calibrated probability distributions with an evidence signal, and no text is generated. It is the small, CPU-friendly tier, with no language model at all. The decoder models (Berk/assay-4b, Berk/assay-27b) are more accurate and larger.

Architecture compiled with late interaction on Alibaba-NLP/gte-modernbert-base (8 query slots). The state is encoded once into token embeddings. Each question is compiled once: the instruction becomes a set of query vectors (learned slots plus a projection of its pooled encoding), each option becomes a vector (its pooled encoding) and its token encodings. A decision is cross-attention from the queries over the state tokens, a small reader MLP, a bilinear score against each option vector plus a per-option bias, and a late-interaction term: the mean over option tokens of the best cosine match among state tokens, times a learned scale. Compiled questions can be cached and reused across states; after the state encode, a decision is a few small matrix products.

Evaluation

split	n	accuracy	Brier	NLL	ECE	confident errors
seen tasks (dev), scaled	6113	0.668	0.442	0.806	0.037	0.019
unseen tasks (holdout), raw	2020	0.606	0.522	0.895	0.133	0.035
unseen tasks (holdout), scaled	2020	0.606	0.494	0.818	0.061	0.011
kev transfer-v4 dev, raw	764	0.542	0.613	1.048	0.174	0.064
kev transfer-v4 dev, scaled	764	0.542	0.572	0.952	0.095	0.012

"Scaled" applies the temperature 1.485 fitted on the seen-task calibration split. Unseen tasks are eleven datasets never trained on; the transfer suite is jaredpalmer/kev-suites transfer-v4 dev, whose sources are excluded from training. Single-text classification (topic, sentiment, spam) is strong; questions that need knowledge (MMLU) or multi-step reasoning are near chance. See the repository's docs/roadmap.md for the full comparison against the other tiers.

CPU latency (milliseconds; the state is encoded once, questions are compiled once and cached, decide runs per state x question set):

runs/compiled-late-gte-base on cpu, 8 threads
questions  encode_state_ms  compile_ms  decide_ms  end_to_end_ms
        1            28.95       29.63      1.381          62.25
        3            30.67       61.52      1.753          94.48
        6            30.64       88.46      2.844         122.80
       12            30.51       88.45      4.867         124.75
       24            30.64       88.42     10.694         131.05

The family

model	size	unseen tasks	transfer-v4
assay-0.6b	0.6B	0.704 / 0.397	0.636 / 0.499
assay-1.7b	1.7B	0.752 / 0.334	0.670 / 0.436
assay-4b	4B	0.803 / 0.271	0.784 / 0.302
assay-8b	8B	0.808 / 0.256	0.818 / 0.267
assay-27b	27B	0.842 / 0.221	0.842 / 0.229
assay-compiled-base	149M	0.606 / 0.494	0.542 / 0.572

Accuracy / Brier after temperature scaling. Same recipe, same splits, different backbones; per-tier abstention and latency are in docs/models.md.

Serving

python -m assay.server --model Berk/assay-compiled-base --port 8000

POST /v1/decide is the native shape; POST /v1/systemone and /v1/systemone/batch accept the shape other open decision models use (criteria options, noul booleans); POST /v1/decide_graph walks a decision tree in one forward pass; POST /v1/agents registers an agent -- a graph plus the actions its outcomes stand for -- and /v1/agents/<name>/run decides a case. Requests arriving together share a pass, and /health and /metrics are for operations. assay.backends.sglang runs the same model on an SGLang deployment.

Guides: deployment and the full API, agents, runnable examples.

Train one on your own data

python -m assay.pipeline --config <your>.json runs training, temperature calibration, evaluation and the conformal thresholds over your own records, and writes a directory this same server and publisher accept. The repository's examples/ has a configuration per tier and a dataset in the record format; records written for other decision models (criteria options, noul booleans) load unchanged.

Usage

from assay import load_model
from assay.schema import Question

model = load_model("Berk/assay-compiled-base", device="cpu")
answers = model.answer(
    "My card was charged twice for order A-104.",
    {
        "refund": Question(type="bool", instructions="Does the customer ask for money back?"),
        "team": Question(type="choice", instructions="Which team should handle this?",
                         options={"billing": "Charges and refunds", "technical": "Bugs"}),
    },
)
print(answers["team"].probabilities, answers["refund"].p_true)

Trained with assay.train_compiled (3.0 epochs, lr 5e-05, head lr 0.0005, batch 32) on the Assay data (55 public datasets rendered as typed questions, synthetic policy and date cases, and 40k generic questions labelled by assay-27b). Each dataset keeps its own licence; the list is in docs/datasets.md.

Limitations

English only. No knowledge beyond what the encoder carries, no arithmetic, no multi-hop reasoning. Calibrated in aggregate on the evaluated distributions, not per answer; check on your own labels before acting on thresholds.

Relationship to other work

Assay is an independent project. Jev and System One are names of TypeSafe AI's products and are mentioned only to describe and compare; kev-suites is Jared Palmer's evaluation data. Assay is not affiliated with or endorsed by either.