Amber 150M

Amber 150M is Amber Ace, one of the seven Amber family characters

Amber 150M is a 150.8M-parameter typed decision engine. Given a state and typed questions (multiple-choice choice or yes/no noul, each option described by its criteria), it returns a probability for every supplied option. It is cross-encoder/ettin-reranker-150m-v1 (Apache-2.0) with every weight fine-tuned and a small option head. Each question is one encoder pass over [CLS] instructions [SEP] state [SEP] [MASK] option 1 โ€ฆ [MASK] option K [SEP], so every option sees the question, the state and the other options. An option's logit is an MLP over its [MASK] token, the mean of its tokens and [CLS]; probabilities are the softmax over the question's options, and a noul answer is P(true).

๐Ÿ‡ฑ๐Ÿ‡น Built in Lithuania. Amber is named after Baltic amber, gintaras in Lithuanian, which the sea washes up on the Lithuanian coast.

Decision Index 0.3

Public index 42.75 (raw 56.85). Complete run: 140,178 requests, 0 unsupported, nothing truncated. Median 15.6 ms, mean 36.0 ms, p95 118.5 ms per request on one NVIDIA GeForce RTX 3080 Ti (bf16, torch 2.14.0+cu126).

The board (edition 0.4) ranks on the Full score. That score is 60% private tests of the same skills and 40% private decision tasks from new domains, which the maintainers run on submitted models, so it is not out yet. Board models up to 1.25 times this size (board snapshot 2026-11-04):

Model Parameters Public index Full score
Amber 150M 150.8M 42.75 pending
JEV Ettin 150M 149.6M 37.31 18.52
Crivo-150M v2 149.6M 34.38 18.26
Dinah-1 149.6M 34.32 16.36
tasksource Decider nano v5 (150M) 149.0M 33.51 19.85
Crivo-68M 68.4M 32.82 19.51
Area Skill Raw
Knowledge & Reasoning 13.8 35.3
Language Understanding 43.5 61.6
Retrieval & Classification 65.0 72.6
Tools & Automation 61.3 63.9
Arts & Human Taste 37.5 56.0
Every benchmark in the index
Area Benchmark Metric Raw Skill
Knowledge & Reasoning GPQA Diamond accuracy 0.286 0.048
Knowledge & Reasoning GSM8K accuracy 0.193 0.022
Knowledge & Reasoning ChessBench accuracy 0.072 0.000
Knowledge & Reasoning MuSR accuracy 0.549 0.283
Knowledge & Reasoning SATA-Bench case exact accuracy 0.207 0.196
Knowledge & Reasoning CRUXEval accuracy 0.421 0.081
Knowledge & Reasoning CLadder accuracy 0.537 0.075
Knowledge & Reasoning HLE accuracy 0.180 0.018
Knowledge & Reasoning MMLU-Pro accuracy 0.194 0.094
Knowledge & Reasoning BBH accuracy 0.579 0.390
Knowledge & Reasoning WinoGrande accuracy 0.638 0.277
Language Understanding ContractNLI macro-F1 0.769 0.667
Language Understanding ANLI macro-F1 0.487 0.231
Language Understanding HellaSwag accuracy 0.670 0.560
Language Understanding ACOS per-review F1 0.451 0.434
Language Understanding FinEntity macro-F1 0.743 0.622
Language Understanding iSarcasmEval Sarcasm F1 ยท track A, English 0.411 0.242
Language Understanding VAST macro-F1 0.754 0.631
Language Understanding NLI4CT macro-F1 0.601 0.225
Language Understanding RAGTruth F1 on hallucinated class 0.671 0.317
Retrieval & Classification BANKING77 macro-F1 0.925 0.924
Retrieval & Classification CLINC150 macro-F1 0.904 0.903
Retrieval & Classification BRIGHT nDCG@10 0.290 0.197
Retrieval & Classification Amazon ESCI macro-F1 0.511 0.387
Retrieval & Classification PhishNChips accuracy 0.883 0.767
Retrieval & Classification HoVer accuracy 0.852 0.704
Tools & Automation BFCL case exact accuracy 0.926 0.900
Tools & Automation ToolRet nDCG@10 0.621 0.562
Tools & Automation API-Bank accuracy 0.713 0.707
Tools & Automation Home appliances case exact accuracy 0.000 0.000
Tools & Automation When2Call accuracy 0.865 0.819
Arts & Human Taste BPoMP accuracy 0.936 0.871
Arts & Human Taste Humicroedit accuracy 0.623 0.245
Arts & Human Taste POP909 accuracy 0.003 0.000
Arts & Human Taste cfcolor accuracy 0.624 0.247
Arts & Human Taste Habermas accuracy 0.575 0.383
Arts & Human Taste New Yorker accuracy 0.604 0.505

Policy, the same for every benchmark: one rendering for every benchmark. A string state is used as is and an object state becomes key: value lines; an option is its criteria text, or its key. Each question is read with its options in two fixed orders (as given and reversed) and the probabilities are averaged. No abstention, no truncation, no option filtering. A question longer than 32,768 tokens is unsupported.

Use

With the decision-index kit installed and this repository as the working directory:

pip install -r requirements.txt
PYTHONPATH=. python -m decision_index run --engine amber_uni_engine:UniEngine \
    --suite-dir <suite-0.3> --out runs/amber-150m --compact

The scored run used no options. The engine reads the weights, uni_config.json and encoder/ from its own directory, and nothing is downloaded. Options:

  • dtype: bf16 (default) or fp32.
  • token_budget (default 65536): the most padded tokens per forward pass. A request's questions are sorted by length and packed.
  • max_len (default 32768): a question needing more tokens is unsupported.
  • orders (default 2, from uni_config.json): averages the probabilities over up to 4 fixed option orders (as given, reversed, rotated by half, rotated by half and reversed).
  • Long questions: a question over 2,048 tokens runs alone and unpadded. The encoder's sliding-window layers then attend in blocks to the keys within their window, not through a dense L ร— L mask, which would take about 14 GiB at 30k tokens. The attention is the same: on a 15,200-token question, Amber 17M's probabilities were bit-identical to the dense path.

Files:

File sha256 (first 16)
amber_banner.jpg 60b2ad4724e78aa0
amber_uni_core.py 17a4270d6e02cc97
amber_uni_engine.py 541314792b915357
encoder/config.json 5faea8a52633cfef
encoder/tokenizer.json 6c8aaa9a542084f2
encoder/tokenizer_config.json 09baa14905255c98
model.safetensors 87776ff6c411d4bc
requirements.txt 13efb642d4e4bd72
uni_config.json ffde5fa04f4fd6b0

model.safetensors holds the encoder and the option head (138 tensors, bf16). It is the exponential moving average (decay 0.999) of the training weights. encoder/ is the base model's config and tokenizer. Requirements:

safetensors==0.8.0
torch==2.14.0
transformers==5.17.0

Training

Two training cycles from cross-encoder/ettin-reranker-150m-v1 at revision 025501c4e0f9bbeb4c5b198318e0089ff061cc14, 13.8 h in all on one RTX 3080 Ti; the last one gives the released weights:

  1. g150-kd18k: 18,000 steps from the base model, seed 1, learning rate 5e-5 for the encoder and 5e-4 for the head, 4 micro-batches of up to 8,192 padded tokens per step, teacher f400-v2 (5.9 h).
  2. g150-c36k: 18,000 more steps from g150-kd18k's EMA weights, with a new warm-up and cosine decay, seed 2, learning rate 5e-5 for the encoder and 5e-4 for the head, 8 micro-batches of up to 4,096 padded tokens per step, teacher g400-kd18k (7.9 h).
  • Batches. Each step has up to 32,768 padded tokens of whole questions (about 83 to 85 questions). Training sequences are up to 4,096 tokens. A longer one is cut to the largest common cap on the instructions, the state and each option that fits; a cut state keeps its head and tail. Inference never cuts.
  • Optimizer. AdamW, 5% warm-up, then cosine decay to 0 in each cycle, weight decay 0.01, dropout 0.1. The weights kept are the exponential moving average (decay 0.999) of the training weights.
  • Augmentation. With probability 0.5 a question's options are shuffled. With probability 0.5 its state moves into the instructions, as in many Decision Index rows.
  • Loss. Cross-entropy against soft targets: 0.5 gold and 0.5 the option probabilities of a larger or earlier development model, the teacher. f400-v2 is an Ettin reranker 400M trained for 6,000 steps on the same mix with gold targets only; g400-kd18k is the first 0.4B cycle: 18,000 steps from ettin-reranker-400m-v1, teacher f400-v2. A task family where the teacher's top option matches the gold on less than 60% of its rows keeps the gold alone.

Data

Questions are drawn from four stores, with weights 0.45 / 0.10 / 0.30 / 0.15. Every store drops a row whose text (state, question or options) shares a 13-word span or a short sentence with a Decision Index 0.3 suite row, and rows from sources derived from a suite benchmark.

Store Rows What it is Licence
tsj-c3k-clean 390,285 tasksource/tasksource-jev-typed-decisions @6f92aad, train split: at most 3,000 rows per source (358 sources, 215 families). Sources that are a Decision Index benchmark's own dataset are left out, as are 145 families held out for validation and the multilingual set. 1,129 rows were dropped for suite overlap and 6,331 for derived sources. per row (the dataset's license column)
tsj-bench 37,887 The same dataset's rows from the 21 sources that are Decision Index suite datasets, train splits only: ARC, ANLI, BANKING77, CLINC150, CommonsenseQA, ContractNLI, ESCI, HellaSwag, HoVer, Humicroedit, iSarcasm, NLI4CT, OpenBookQA, RAGTruth, WinoGrande. At most 3,000 per source; 5,887 rows dropped for suite overlap. per row; ANLI CC BY-NC 4.0
amber-di 323,836 The Decision Index-format records of Amber 0.4B's training mix, whose card lists sources, splits and filters, plus the MRPC / QQP paraphrase records of its later rounds. Generators built to imitate one benchmark (home appliances, POP909, CLadder, SATA-Bench raw pool) are left out, as are multi-answer and abstaining records. At most 15,000 per source; 61,328 rows dropped for suite overlap. as listed on the Amber 0.4B card (RAGTruth contexts: MS MARCO non-commercial)
fc 134,125 Tool-use decisions parsed from glaiveai/glaive-function-calling-v2, Team-ACE/ToolACE and NousResearch/hermes-function-calling-v1: which tool to call, whether to call one now, ask or decline, which tool description fits, and which reply comes next. 865 rows dropped for suite overlap. Apache-2.0

The weights are released under CC BY-NC 4.0 because the training data includes non-commercial sources (ANLI under CC BY-NC 4.0, RAGTruth's MS MARCO contexts, QQP under Quora's terms). The base model is Apache-2.0.

Limitations

  • Knowledge & Reasoning is its weakest area: a small encoder that does not generate stays near chance on questions that need world knowledge or multi-step arithmetic.
  • Home appliances and POP909 are near zero: the generators built for them were left out of training, because their gains on the public benchmarks did not carry over to the private tests.
  • Mostly English.
  • The probabilities are calibrated on the training mix, not per deployment; re-check thresholds on your own data.
Downloads last month

-

Downloads are not tracked for this model. How to track
Safetensors
Model size
0.2B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for MLukauskas/amber-150m

Finetuned
(11)
this model