Amber 150M
Amber 150M is a 150.8M-parameter typed decision engine. Given a state and typed questions
(multiple-choice choice or yes/no noul, each option described by its criteria), it returns a probability for
every supplied option. It is cross-encoder/ettin-reranker-150m-v1 (Apache-2.0) with every weight
fine-tuned and a small option head. Each question is one encoder pass over
[CLS] instructions [SEP] state [SEP] [MASK] option 1 โฆ [MASK] option K [SEP], so every option sees the
question, the state and the other options. An option's logit is an MLP over its [MASK] token, the mean of its
tokens and [CLS]; probabilities are the softmax over the question's options, and a noul answer is P(true).
๐ฑ๐น Built in Lithuania. Amber is named after Baltic amber, gintaras in Lithuanian, which the sea washes up on the Lithuanian coast.
Decision Index 0.3
Public index 42.75 (raw 56.85). Complete run: 140,178 requests, 0 unsupported, nothing truncated. Median 15.6 ms, mean 36.0 ms, p95 118.5 ms per request on one NVIDIA GeForce RTX 3080 Ti (bf16, torch 2.14.0+cu126).
The board (edition 0.4) ranks on the Full score. That score is 60% private tests of the same skills and 40% private decision tasks from new domains, which the maintainers run on submitted models, so it is not out yet. Board models up to 1.25 times this size (board snapshot 2026-11-04):
| Model | Parameters | Public index | Full score |
|---|---|---|---|
| Amber 150M | 150.8M | 42.75 | pending |
| JEV Ettin 150M | 149.6M | 37.31 | 18.52 |
| Crivo-150M v2 | 149.6M | 34.38 | 18.26 |
| Dinah-1 | 149.6M | 34.32 | 16.36 |
| tasksource Decider nano v5 (150M) | 149.0M | 33.51 | 19.85 |
| Crivo-68M | 68.4M | 32.82 | 19.51 |
| Area | Skill | Raw |
|---|---|---|
| Knowledge & Reasoning | 13.8 | 35.3 |
| Language Understanding | 43.5 | 61.6 |
| Retrieval & Classification | 65.0 | 72.6 |
| Tools & Automation | 61.3 | 63.9 |
| Arts & Human Taste | 37.5 | 56.0 |
Every benchmark in the index
| Area | Benchmark | Metric | Raw | Skill |
|---|---|---|---|---|
| Knowledge & Reasoning | GPQA Diamond | accuracy | 0.286 | 0.048 |
| Knowledge & Reasoning | GSM8K | accuracy | 0.193 | 0.022 |
| Knowledge & Reasoning | ChessBench | accuracy | 0.072 | 0.000 |
| Knowledge & Reasoning | MuSR | accuracy | 0.549 | 0.283 |
| Knowledge & Reasoning | SATA-Bench | case exact accuracy | 0.207 | 0.196 |
| Knowledge & Reasoning | CRUXEval | accuracy | 0.421 | 0.081 |
| Knowledge & Reasoning | CLadder | accuracy | 0.537 | 0.075 |
| Knowledge & Reasoning | HLE | accuracy | 0.180 | 0.018 |
| Knowledge & Reasoning | MMLU-Pro | accuracy | 0.194 | 0.094 |
| Knowledge & Reasoning | BBH | accuracy | 0.579 | 0.390 |
| Knowledge & Reasoning | WinoGrande | accuracy | 0.638 | 0.277 |
| Language Understanding | ContractNLI | macro-F1 | 0.769 | 0.667 |
| Language Understanding | ANLI | macro-F1 | 0.487 | 0.231 |
| Language Understanding | HellaSwag | accuracy | 0.670 | 0.560 |
| Language Understanding | ACOS | per-review F1 | 0.451 | 0.434 |
| Language Understanding | FinEntity | macro-F1 | 0.743 | 0.622 |
| Language Understanding | iSarcasmEval | Sarcasm F1 ยท track A, English | 0.411 | 0.242 |
| Language Understanding | VAST | macro-F1 | 0.754 | 0.631 |
| Language Understanding | NLI4CT | macro-F1 | 0.601 | 0.225 |
| Language Understanding | RAGTruth | F1 on hallucinated class | 0.671 | 0.317 |
| Retrieval & Classification | BANKING77 | macro-F1 | 0.925 | 0.924 |
| Retrieval & Classification | CLINC150 | macro-F1 | 0.904 | 0.903 |
| Retrieval & Classification | BRIGHT | nDCG@10 | 0.290 | 0.197 |
| Retrieval & Classification | Amazon ESCI | macro-F1 | 0.511 | 0.387 |
| Retrieval & Classification | PhishNChips | accuracy | 0.883 | 0.767 |
| Retrieval & Classification | HoVer | accuracy | 0.852 | 0.704 |
| Tools & Automation | BFCL | case exact accuracy | 0.926 | 0.900 |
| Tools & Automation | ToolRet | nDCG@10 | 0.621 | 0.562 |
| Tools & Automation | API-Bank | accuracy | 0.713 | 0.707 |
| Tools & Automation | Home appliances | case exact accuracy | 0.000 | 0.000 |
| Tools & Automation | When2Call | accuracy | 0.865 | 0.819 |
| Arts & Human Taste | BPoMP | accuracy | 0.936 | 0.871 |
| Arts & Human Taste | Humicroedit | accuracy | 0.623 | 0.245 |
| Arts & Human Taste | POP909 | accuracy | 0.003 | 0.000 |
| Arts & Human Taste | cfcolor | accuracy | 0.624 | 0.247 |
| Arts & Human Taste | Habermas | accuracy | 0.575 | 0.383 |
| Arts & Human Taste | New Yorker | accuracy | 0.604 | 0.505 |
Policy, the same for every benchmark: one rendering for every benchmark. A string state is used as is and an
object state becomes key: value lines; an option is its criteria text, or its key. Each question is read with its options in two fixed orders (as given and reversed) and the probabilities are averaged. No
abstention, no truncation, no option filtering. A question longer than 32,768 tokens is unsupported.
Use
With the decision-index kit installed and this repository as the working directory:
pip install -r requirements.txt
PYTHONPATH=. python -m decision_index run --engine amber_uni_engine:UniEngine \
--suite-dir <suite-0.3> --out runs/amber-150m --compact
The scored run used no options. The engine reads the weights, uni_config.json and encoder/ from its own
directory, and nothing is downloaded. Options:
dtype:bf16(default) orfp32.token_budget(default 65536): the most padded tokens per forward pass. A request's questions are sorted by length and packed.max_len(default 32768): a question needing more tokens is unsupported.orders(default 2, fromuni_config.json): averages the probabilities over up to 4 fixed option orders (as given, reversed, rotated by half, rotated by half and reversed).- Long questions: a question over 2,048 tokens runs alone and unpadded. The encoder's sliding-window layers then attend in blocks to the keys within their window, not through a dense L ร L mask, which would take about 14 GiB at 30k tokens. The attention is the same: on a 15,200-token question, Amber 17M's probabilities were bit-identical to the dense path.
Files:
| File | sha256 (first 16) |
|---|---|
amber_banner.jpg |
60b2ad4724e78aa0 |
amber_uni_core.py |
17a4270d6e02cc97 |
amber_uni_engine.py |
541314792b915357 |
encoder/config.json |
5faea8a52633cfef |
encoder/tokenizer.json |
6c8aaa9a542084f2 |
encoder/tokenizer_config.json |
09baa14905255c98 |
model.safetensors |
87776ff6c411d4bc |
requirements.txt |
13efb642d4e4bd72 |
uni_config.json |
ffde5fa04f4fd6b0 |
model.safetensors holds the encoder and the option head (138 tensors, bf16).
It is the exponential moving average (decay 0.999) of the training weights. encoder/ is the base model's config and tokenizer. Requirements:
safetensors==0.8.0
torch==2.14.0
transformers==5.17.0
Training
Two training cycles from cross-encoder/ettin-reranker-150m-v1 at revision 025501c4e0f9bbeb4c5b198318e0089ff061cc14, 13.8 h in all on one RTX 3080 Ti; the last one gives the released weights:
g150-kd18k: 18,000 steps from the base model, seed 1, learning rate 5e-5 for the encoder and 5e-4 for the head, 4 micro-batches of up to 8,192 padded tokens per step, teacherf400-v2(5.9 h).g150-c36k: 18,000 more steps fromg150-kd18k's EMA weights, with a new warm-up and cosine decay, seed 2, learning rate 5e-5 for the encoder and 5e-4 for the head, 8 micro-batches of up to 4,096 padded tokens per step, teacherg400-kd18k(7.9 h).
- Batches. Each step has up to 32,768 padded tokens of whole questions (about 83 to 85 questions). Training sequences are up to 4,096 tokens. A longer one is cut to the largest common cap on the instructions, the state and each option that fits; a cut state keeps its head and tail. Inference never cuts.
- Optimizer. AdamW, 5% warm-up, then cosine decay to 0 in each cycle, weight decay 0.01, dropout 0.1. The weights kept are the exponential moving average (decay 0.999) of the training weights.
- Augmentation. With probability 0.5 a question's options are shuffled. With probability 0.5 its state moves into the instructions, as in many Decision Index rows.
- Loss. Cross-entropy against soft targets: 0.5 gold and 0.5 the option probabilities of a larger or
earlier development model, the teacher.
f400-v2is an Ettin reranker 400M trained for 6,000 steps on the same mix with gold targets only;g400-kd18kis the first 0.4B cycle: 18,000 steps from ettin-reranker-400m-v1, teacherf400-v2. A task family where the teacher's top option matches the gold on less than 60% of its rows keeps the gold alone.
Data
Questions are drawn from four stores, with weights 0.45 / 0.10 / 0.30 / 0.15. Every store drops a row whose text (state, question or options) shares a 13-word span or a short sentence with a Decision Index 0.3 suite row, and rows from sources derived from a suite benchmark.
| Store | Rows | What it is | Licence |
|---|---|---|---|
tsj-c3k-clean |
390,285 | tasksource/tasksource-jev-typed-decisions @6f92aad, train split: at most 3,000 rows per source (358 sources, 215 families). Sources that are a Decision Index benchmark's own dataset are left out, as are 145 families held out for validation and the multilingual set. 1,129 rows were dropped for suite overlap and 6,331 for derived sources. | per row (the dataset's license column) |
tsj-bench |
37,887 | The same dataset's rows from the 21 sources that are Decision Index suite datasets, train splits only: ARC, ANLI, BANKING77, CLINC150, CommonsenseQA, ContractNLI, ESCI, HellaSwag, HoVer, Humicroedit, iSarcasm, NLI4CT, OpenBookQA, RAGTruth, WinoGrande. At most 3,000 per source; 5,887 rows dropped for suite overlap. | per row; ANLI CC BY-NC 4.0 |
amber-di |
323,836 | The Decision Index-format records of Amber 0.4B's training mix, whose card lists sources, splits and filters, plus the MRPC / QQP paraphrase records of its later rounds. Generators built to imitate one benchmark (home appliances, POP909, CLadder, SATA-Bench raw pool) are left out, as are multi-answer and abstaining records. At most 15,000 per source; 61,328 rows dropped for suite overlap. | as listed on the Amber 0.4B card (RAGTruth contexts: MS MARCO non-commercial) |
fc |
134,125 | Tool-use decisions parsed from glaiveai/glaive-function-calling-v2, Team-ACE/ToolACE and NousResearch/hermes-function-calling-v1: which tool to call, whether to call one now, ask or decline, which tool description fits, and which reply comes next. 865 rows dropped for suite overlap. | Apache-2.0 |
The weights are released under CC BY-NC 4.0 because the training data includes non-commercial sources (ANLI under CC BY-NC 4.0, RAGTruth's MS MARCO contexts, QQP under Quora's terms). The base model is Apache-2.0.
Limitations
- Knowledge & Reasoning is its weakest area: a small encoder that does not generate stays near chance on questions that need world knowledge or multi-step arithmetic.
- Home appliances and POP909 are near zero: the generators built for them were left out of training, because their gains on the public benchmarks did not carry over to the private tests.
- Mostly English.
- The probabilities are calibrated on the training mix, not per deployment; re-check thresholds on your own data.
Model tree for MLukauskas/amber-150m
Base model
jhu-clsp/ettin-encoder-150m