| --- |
| license: apache-2.0 |
| base_model: Alibaba-NLP/gte-modernbert-base |
| library_name: transformers |
| pipeline_tag: zero-shot-classification |
| tags: |
| - assay |
| - decision-model |
| - calibrated |
| - conformal-prediction |
| - text-classification |
| - cpu |
| language: |
| - en |
| --- |
| |
| # assay-compiled-base |
|
|
| A small-tier decision model from the [Assay](https://github.com/bgokden/assay) project: |
| typed questions (`bool`, `choice`, `score`) over a state produce calibrated probability |
| distributions with an `evidence` signal, and no text is generated. It is the small, CPU-friendly tier, with no language model at all. The decoder |
| models ([Berk/assay-4b](https://huggingface.co/Berk/assay-4b), |
| [Berk/assay-27b](https://huggingface.co/Berk/assay-27b)) are more accurate and larger. |
|
|
| ## In one line |
|
|
| **It runs on a CPU, and it abstains.** 149M parameters, ~29 ms per decision without a GPU. |
| On tasks it has never seen it acts on 8% of bool questions at a 5.0% error rate and hands the |
| rest over rather than guessing -- a conformal prediction set against a fitted error rate, not |
| a threshold on a softmax someone eyeballed. It abstains far more than the decoder tiers |
| because it is far smaller; that is the trade, and it is visible instead of hidden. |
|
|
| Accuracy on those unseen tasks is **0.606**. If you have a GPU, start with |
| [Berk/assay-4b](https://huggingface.co/Berk/assay-4b) at 0.803 instead. |
|
|
| The rest of the family, and the dataset, are in the [Assay collection](https://huggingface.co/collections/Berk/assay-calibrated-typed-decisions-6ab2fcb2b7eea0b7aaf785ab). |
|
|
| Architecture `compiled` with late interaction on `Alibaba-NLP/gte-modernbert-base` (8 query slots). |
| The state is encoded once into token embeddings. Each question is compiled once: the instruction becomes a set of query vectors (learned slots plus a projection of its pooled encoding), each option becomes a vector (its pooled encoding) and its token encodings. A decision is cross-attention from the queries over the state tokens, a small reader MLP, a bilinear score against each option vector plus a per-option bias, and a late-interaction term: the mean over option tokens of the best cosine match among state tokens, times a learned scale. Compiled questions can be cached and reused across states; after the state encode, a decision is a few small matrix products. |
|
|
| ## Evaluation |
|
|
| | split | n | accuracy | Brier | NLL | ECE | confident errors | |
| |---|---|---|---|---|---|---| |
| | seen tasks (dev), scaled | 6113 | 0.668 | 0.442 | 0.806 | 0.037 | 0.019 | |
| | unseen tasks (holdout), raw | 2020 | 0.606 | 0.522 | 0.895 | 0.133 | 0.035 | |
| | unseen tasks (holdout), scaled | 2020 | 0.606 | 0.494 | 0.818 | 0.061 | 0.011 | |
| | kev transfer-v4 dev, raw | 764 | 0.542 | 0.613 | 1.048 | 0.174 | 0.064 | |
| | kev transfer-v4 dev, scaled | 764 | 0.542 | 0.572 | 0.952 | 0.095 | 0.012 | |
|
|
| "Scaled" applies the temperature 1.485 fitted on the seen-task |
| calibration split. Unseen tasks are eleven datasets never trained on; the transfer suite is |
| `jaredpalmer/kev-suites` transfer-v4 dev, whose sources are excluded from training. Single-text |
| classification (topic, sentiment, spam) is strong; questions that need knowledge (MMLU) or |
| multi-step reasoning are near chance. See the repository's `docs/roadmap.md` for the full |
| comparison against the other tiers. |
|
|
| CPU latency (milliseconds; the state is encoded once, questions are compiled once and cached, `decide` runs per state x question set): |
|
|
| ``` |
| runs/compiled-late-gte-base on cpu, 8 threads |
| questions encode_state_ms compile_ms decide_ms end_to_end_ms |
| 1 28.95 29.63 1.381 62.25 |
| 3 30.67 61.52 1.753 94.48 |
| 6 30.64 88.46 2.844 122.80 |
| 12 30.51 88.45 4.867 124.75 |
| 24 30.64 88.42 10.694 131.05 |
| ``` |
|
|
| ## The family |
|
|
| | model | size | unseen tasks | transfer-v4 | |
| |---|---|---|---| |
| | [assay-0.6b](https://huggingface.co/Berk/assay-0.6b) | 0.6B | 0.704 / 0.397 | 0.636 / 0.499 | |
| | [assay-1.7b](https://huggingface.co/Berk/assay-1.7b) | 1.7B | 0.752 / 0.334 | 0.670 / 0.436 | |
| | [assay-4b](https://huggingface.co/Berk/assay-4b) | 4B | 0.803 / 0.271 | 0.784 / 0.302 | |
| | [assay-8b](https://huggingface.co/Berk/assay-8b) | 8B | 0.808 / 0.256 | 0.818 / 0.267 | |
| | [assay-27b](https://huggingface.co/Berk/assay-27b) | 27B | 0.842 / 0.221 | 0.842 / 0.229 | |
| | [assay-compiled-base](https://huggingface.co/Berk/assay-compiled-base) | 149M | 0.606 / 0.494 | 0.542 / 0.572 | |
|
|
| Accuracy / Brier after temperature scaling. Same recipe, same splits, different backbones; |
| per-tier abstention and latency are in |
| [docs/models.md](https://github.com/bgokden/assay/blob/main/docs/models.md). |
|
|
| ## Serving |
|
|
| ```bash |
| python -m assay.server --model Berk/assay-compiled-base --port 8000 |
| ``` |
|
|
| `POST /v1/decide` is the native shape; `POST /v1/systemone` and `/v1/systemone/batch` accept |
| the shape other open decision models use (`criteria` options, `noul` booleans); `POST |
| /v1/decide_graph` walks a decision tree in one forward pass; `POST /v1/agents` registers an |
| agent -- a graph plus the actions its outcomes stand for -- and `/v1/agents/<name>/run` |
| decides a case. Requests arriving together share a pass, and `/health` and `/metrics` are for |
| operations. `assay.backends.sglang` runs the same model on an SGLang deployment. |
| |
| Guides: [deployment and the full API](https://github.com/bgokden/assay/blob/main/docs/serving.md), |
| [agents](https://github.com/bgokden/assay/blob/main/docs/agents.md), |
| [runnable examples](https://github.com/bgokden/assay/tree/main/examples). |
| |
| ## Train one on your own data |
| |
| `python -m assay.pipeline --config <your>.json` runs training, temperature calibration, |
| evaluation and the conformal thresholds over your own records, and writes a directory this |
| same server and publisher accept. The repository's |
| [examples/](https://github.com/bgokden/assay/tree/main/examples) has a configuration per tier |
| and a dataset in the record format; records written for other decision models (`criteria` |
| options, `noul` booleans) load unchanged. |
| |
| |
| ## Usage |
| |
| ```python |
| from assay import load_model |
| from assay.schema import Question |
|
|
| model = load_model("Berk/assay-compiled-base", device="cpu") |
| answers = model.answer( |
| "My card was charged twice for order A-104.", |
| { |
| "refund": Question(type="bool", instructions="Does the customer ask for money back?"), |
| "team": Question(type="choice", instructions="Which team should handle this?", |
| options={"billing": "Charges and refunds", "technical": "Bugs"}), |
| }, |
| ) |
| print(answers["team"].probabilities, answers["refund"].p_true) |
| ``` |
| |
| Trained with `assay.train_compiled` (3.0 epochs, lr 5e-05, head lr |
| 0.0005, batch 32) on the Assay data (55 public datasets |
| rendered as typed questions, synthetic policy and date cases, and 40k generic questions labelled |
| by assay-27b). Each dataset keeps its own licence; the list is in |
| [docs/datasets.md](https://github.com/bgokden/assay/blob/main/docs/datasets.md). |
| |
| ## Limitations |
| |
| English only. No knowledge beyond what the encoder carries, no arithmetic, no multi-hop |
| reasoning. Calibrated in aggregate on the evaluated distributions, not per answer; check on |
| your own labels before acting on thresholds. |
| |
| ## Relationship to other work |
| |
| Assay is an independent project. Jev and System One are names of TypeSafe AI's products and |
| are mentioned only to describe and compare; kev-suites is Jared Palmer's evaluation data. |
| Assay is not affiliated with or endorsed by either. |
| |
