--- license: apache-2.0 base_model: Alibaba-NLP/gte-modernbert-base library_name: transformers pipeline_tag: zero-shot-classification tags: - assay - decision-model - calibrated - conformal-prediction - text-classification - cpu language: - en --- # assay-compiled-base A small-tier decision model from the [Assay](https://github.com/bgokden/assay) project: typed questions (`bool`, `choice`, `score`) over a state produce calibrated probability distributions with an `evidence` signal, and no text is generated. It is the small, CPU-friendly tier, with no language model at all. The decoder models ([Berk/assay-4b](https://huggingface.co/Berk/assay-4b), [Berk/assay-27b](https://huggingface.co/Berk/assay-27b)) are more accurate and larger. ## In one line **It runs on a CPU, and it abstains.** 149M parameters, ~29 ms per decision without a GPU. On tasks it has never seen it acts on 8% of bool questions at a 5.0% error rate and hands the rest over rather than guessing -- a conformal prediction set against a fitted error rate, not a threshold on a softmax someone eyeballed. It abstains far more than the decoder tiers because it is far smaller; that is the trade, and it is visible instead of hidden. Accuracy on those unseen tasks is **0.606**. If you have a GPU, start with [Berk/assay-4b](https://huggingface.co/Berk/assay-4b) at 0.803 instead. The rest of the family, and the dataset, are in the [Assay collection](https://huggingface.co/collections/Berk/assay-calibrated-typed-decisions-6ab2fcb2b7eea0b7aaf785ab). Architecture `compiled` with late interaction on `Alibaba-NLP/gte-modernbert-base` (8 query slots). The state is encoded once into token embeddings. Each question is compiled once: the instruction becomes a set of query vectors (learned slots plus a projection of its pooled encoding), each option becomes a vector (its pooled encoding) and its token encodings. A decision is cross-attention from the queries over the state tokens, a small reader MLP, a bilinear score against each option vector plus a per-option bias, and a late-interaction term: the mean over option tokens of the best cosine match among state tokens, times a learned scale. Compiled questions can be cached and reused across states; after the state encode, a decision is a few small matrix products. ## Evaluation | split | n | accuracy | Brier | NLL | ECE | confident errors | |---|---|---|---|---|---|---| | seen tasks (dev), scaled | 6113 | 0.668 | 0.442 | 0.806 | 0.037 | 0.019 | | unseen tasks (holdout), raw | 2020 | 0.606 | 0.522 | 0.895 | 0.133 | 0.035 | | unseen tasks (holdout), scaled | 2020 | 0.606 | 0.494 | 0.818 | 0.061 | 0.011 | | kev transfer-v4 dev, raw | 764 | 0.542 | 0.613 | 1.048 | 0.174 | 0.064 | | kev transfer-v4 dev, scaled | 764 | 0.542 | 0.572 | 0.952 | 0.095 | 0.012 | "Scaled" applies the temperature 1.485 fitted on the seen-task calibration split. Unseen tasks are eleven datasets never trained on; the transfer suite is `jaredpalmer/kev-suites` transfer-v4 dev, whose sources are excluded from training. Single-text classification (topic, sentiment, spam) is strong; questions that need knowledge (MMLU) or multi-step reasoning are near chance. See the repository's `docs/roadmap.md` for the full comparison against the other tiers. CPU latency (milliseconds; the state is encoded once, questions are compiled once and cached, `decide` runs per state x question set): ``` runs/compiled-late-gte-base on cpu, 8 threads questions encode_state_ms compile_ms decide_ms end_to_end_ms 1 28.95 29.63 1.381 62.25 3 30.67 61.52 1.753 94.48 6 30.64 88.46 2.844 122.80 12 30.51 88.45 4.867 124.75 24 30.64 88.42 10.694 131.05 ``` ## The family | model | size | unseen tasks | transfer-v4 | |---|---|---|---| | [assay-0.6b](https://huggingface.co/Berk/assay-0.6b) | 0.6B | 0.704 / 0.397 | 0.636 / 0.499 | | [assay-1.7b](https://huggingface.co/Berk/assay-1.7b) | 1.7B | 0.752 / 0.334 | 0.670 / 0.436 | | [assay-4b](https://huggingface.co/Berk/assay-4b) | 4B | 0.803 / 0.271 | 0.784 / 0.302 | | [assay-8b](https://huggingface.co/Berk/assay-8b) | 8B | 0.808 / 0.256 | 0.818 / 0.267 | | [assay-27b](https://huggingface.co/Berk/assay-27b) | 27B | 0.842 / 0.221 | 0.842 / 0.229 | | [assay-compiled-base](https://huggingface.co/Berk/assay-compiled-base) | 149M | 0.606 / 0.494 | 0.542 / 0.572 | Accuracy / Brier after temperature scaling. Same recipe, same splits, different backbones; per-tier abstention and latency are in [docs/models.md](https://github.com/bgokden/assay/blob/main/docs/models.md). ## Serving ```bash python -m assay.server --model Berk/assay-compiled-base --port 8000 ``` `POST /v1/decide` is the native shape; `POST /v1/systemone` and `/v1/systemone/batch` accept the shape other open decision models use (`criteria` options, `noul` booleans); `POST /v1/decide_graph` walks a decision tree in one forward pass; `POST /v1/agents` registers an agent -- a graph plus the actions its outcomes stand for -- and `/v1/agents//run` decides a case. Requests arriving together share a pass, and `/health` and `/metrics` are for operations. `assay.backends.sglang` runs the same model on an SGLang deployment. Guides: [deployment and the full API](https://github.com/bgokden/assay/blob/main/docs/serving.md), [agents](https://github.com/bgokden/assay/blob/main/docs/agents.md), [runnable examples](https://github.com/bgokden/assay/tree/main/examples). ## Train one on your own data `python -m assay.pipeline --config .json` runs training, temperature calibration, evaluation and the conformal thresholds over your own records, and writes a directory this same server and publisher accept. The repository's [examples/](https://github.com/bgokden/assay/tree/main/examples) has a configuration per tier and a dataset in the record format; records written for other decision models (`criteria` options, `noul` booleans) load unchanged. ## Usage ```python from assay import load_model from assay.schema import Question model = load_model("Berk/assay-compiled-base", device="cpu") answers = model.answer( "My card was charged twice for order A-104.", { "refund": Question(type="bool", instructions="Does the customer ask for money back?"), "team": Question(type="choice", instructions="Which team should handle this?", options={"billing": "Charges and refunds", "technical": "Bugs"}), }, ) print(answers["team"].probabilities, answers["refund"].p_true) ``` Trained with `assay.train_compiled` (3.0 epochs, lr 5e-05, head lr 0.0005, batch 32) on the Assay data (55 public datasets rendered as typed questions, synthetic policy and date cases, and 40k generic questions labelled by assay-27b). Each dataset keeps its own licence; the list is in [docs/datasets.md](https://github.com/bgokden/assay/blob/main/docs/datasets.md). ## Limitations English only. No knowledge beyond what the encoder carries, no arithmetic, no multi-hop reasoning. Calibrated in aggregate on the evaluated distributions, not per answer; check on your own labels before acting on thresholds. ## Relationship to other work Assay is an independent project. Jev and System One are names of TypeSafe AI's products and are mentioned only to describe and compare; kev-suites is Jared Palmer's evaluation data. Assay is not affiliated with or endorsed by either.