README.md

---
license: apache-2.0
base_model: Alibaba-NLP/gte-modernbert-base
library_name: transformers
pipeline_tag: zero-shot-classification
tags:
  - assay
  - decision-model
  - calibrated
  - conformal-prediction
  - text-classification
  - cpu
language:
  - en
---

# assay-compiled-base

A small-tier decision model from the [Assay](https://github.com/bgokden/assay) project:
typed questions (`bool`, `choice`, `score`) over a state produce calibrated probability
distributions with an `evidence` signal, and no text is generated. It is the small, CPU-friendly tier, with no language model at all. The decoder
models ([Berk/assay-4b](https://huggingface.co/Berk/assay-4b),
[Berk/assay-27b](https://huggingface.co/Berk/assay-27b)) are more accurate and larger.

## In one line

**It runs on a CPU, and it abstains.** 149M parameters, ~29 ms per decision without a GPU.
On tasks it has never seen it acts on 8% of bool questions at a 5.0% error rate and hands the
rest over rather than guessing -- a conformal prediction set against a fitted error rate, not
a threshold on a softmax someone eyeballed. It abstains far more than the decoder tiers
because it is far smaller; that is the trade, and it is visible instead of hidden.

Accuracy on those unseen tasks is **0.606**. If you have a GPU, start with
[Berk/assay-4b](https://huggingface.co/Berk/assay-4b) at 0.803 instead.

The rest of the family, and the dataset, are in the [Assay collection](https://huggingface.co/collections/Berk/assay-calibrated-typed-decisions-6ab2fcb2b7eea0b7aaf785ab).

Architecture `compiled` with late interaction on `Alibaba-NLP/gte-modernbert-base` (8 query slots).
The state is encoded once into token embeddings. Each question is compiled once: the instruction becomes a set of query vectors (learned slots plus a projection of its pooled encoding), each option becomes a vector (its pooled encoding) and its token encodings. A decision is cross-attention from the queries over the state tokens, a small reader MLP, a bilinear score against each option vector plus a per-option bias, and a late-interaction term: the mean over option tokens of the best cosine match among state tokens, times a learned scale. Compiled questions can be cached and reused across states; after the state encode, a decision is a few small matrix products.

## Evaluation

| split | n | accuracy | Brier | NLL | ECE | confident errors |
|---|---|---|---|---|---|---|
| seen tasks (dev), scaled | 6113 | 0.668 | 0.442 | 0.806 | 0.037 | 0.019 |
| unseen tasks (holdout), raw | 2020 | 0.606 | 0.522 | 0.895 | 0.133 | 0.035 |
| unseen tasks (holdout), scaled | 2020 | 0.606 | 0.494 | 0.818 | 0.061 | 0.011 |
| kev transfer-v4 dev, raw | 764 | 0.542 | 0.613 | 1.048 | 0.174 | 0.064 |
| kev transfer-v4 dev, scaled | 764 | 0.542 | 0.572 | 0.952 | 0.095 | 0.012 |

"Scaled" applies the temperature 1.485 fitted on the seen-task
calibration split. Unseen tasks are eleven datasets never trained on; the transfer suite is
`jaredpalmer/kev-suites` transfer-v4 dev, whose sources are excluded from training. Single-text
classification (topic, sentiment, spam) is strong; questions that need knowledge (MMLU) or
multi-step reasoning are near chance. See the repository's `docs/roadmap.md` for the full
comparison against the other tiers.

CPU latency (milliseconds; the state is encoded once, questions are compiled once and cached, `decide` runs per state x question set):

```
runs/compiled-late-gte-base on cpu, 8 threads
questions  encode_state_ms  compile_ms  decide_ms  end_to_end_ms
        1            28.95       29.63      1.381          62.25
        3            30.67       61.52      1.753          94.48
        6            30.64       88.46      2.844         122.80
       12            30.51       88.45      4.867         124.75
       24            30.64       88.42     10.694         131.05
```

## The family

| model | size | unseen tasks | transfer-v4 |
|---|---|---|---|
| [assay-0.6b](https://huggingface.co/Berk/assay-0.6b) | 0.6B | 0.704 / 0.397 | 0.636 / 0.499 |
| [assay-1.7b](https://huggingface.co/Berk/assay-1.7b) | 1.7B | 0.752 / 0.334 | 0.670 / 0.436 |
| [assay-4b](https://huggingface.co/Berk/assay-4b) | 4B | 0.803 / 0.271 | 0.784 / 0.302 |
| [assay-8b](https://huggingface.co/Berk/assay-8b) | 8B | 0.808 / 0.256 | 0.818 / 0.267 |
| [assay-27b](https://huggingface.co/Berk/assay-27b) | 27B | 0.842 / 0.221 | 0.842 / 0.229 |
| [assay-compiled-base](https://huggingface.co/Berk/assay-compiled-base) | 149M | 0.606 / 0.494 | 0.542 / 0.572 |

Accuracy / Brier after temperature scaling. Same recipe, same splits, different backbones;
per-tier abstention and latency are in
[docs/models.md](https://github.com/bgokden/assay/blob/main/docs/models.md).

## Serving

```bash
python -m assay.server --model Berk/assay-compiled-base --port 8000
```

`POST /v1/decide` is the native shape; `POST /v1/systemone` and `/v1/systemone/batch` accept
the shape other open decision models use (`criteria` options, `noul` booleans); `POST
/v1/decide_graph` walks a decision tree in one forward pass; `POST /v1/agents` registers an
agent -- a graph plus the actions its outcomes stand for -- and `/v1/agents/<name>/run`
decides a case. Requests arriving together share a pass, and `/health` and `/metrics` are for
operations. `assay.backends.sglang` runs the same model on an SGLang deployment.

Guides: [deployment and the full API](https://github.com/bgokden/assay/blob/main/docs/serving.md),
[agents](https://github.com/bgokden/assay/blob/main/docs/agents.md),
[runnable examples](https://github.com/bgokden/assay/tree/main/examples).

## Train one on your own data

`python -m assay.pipeline --config <your>.json` runs training, temperature calibration,
evaluation and the conformal thresholds over your own records, and writes a directory this
same server and publisher accept. The repository's
[examples/](https://github.com/bgokden/assay/tree/main/examples) has a configuration per tier
and a dataset in the record format; records written for other decision models (`criteria`
options, `noul` booleans) load unchanged.


## Usage

```python
from assay import load_model
from assay.schema import Question

model = load_model("Berk/assay-compiled-base", device="cpu")
answers = model.answer(
    "My card was charged twice for order A-104.",
    {
        "refund": Question(type="bool", instructions="Does the customer ask for money back?"),
        "team": Question(type="choice", instructions="Which team should handle this?",
                         options={"billing": "Charges and refunds", "technical": "Bugs"}),
    },
)
print(answers["team"].probabilities, answers["refund"].p_true)
```

Trained with `assay.train_compiled` (3.0 epochs, lr 5e-05, head lr
0.0005, batch 32) on the Assay data (55 public datasets
rendered as typed questions, synthetic policy and date cases, and 40k generic questions labelled
by assay-27b). Each dataset keeps its own licence; the list is in
[docs/datasets.md](https://github.com/bgokden/assay/blob/main/docs/datasets.md).

## Limitations

English only. No knowledge beyond what the encoder carries, no arithmetic, no multi-hop
reasoning. Calibrated in aggregate on the evaluated distributions, not per answer; check on
your own labels before acting on thresholds.

## Relationship to other work

Assay is an independent project. Jev and System One are names of TypeSafe AI's products and
are mentioned only to describe and compare; kev-suites is Jared Palmer's evaluation data.
Assay is not affiliated with or endorsed by either.