Amber 68M

Amber 68M is Amberbug, one of the seven Amber family characters

Amber 68M is a 68.9M-parameter typed decision engine. Given a state and typed questions (multiple-choice choice or yes/no noul, each option described by its criteria), it returns a probability for every supplied option. It is cross-encoder/ettin-reranker-68m-v1 (Apache-2.0) with every weight fine-tuned and a small option head. Each question is one encoder pass over [CLS] instructions [SEP] state [SEP] [MASK] option 1 โ€ฆ [MASK] option K [SEP], so every option sees the question, the state and the other options. An option's logit is an MLP over its [MASK] token, the mean of its tokens and [CLS]; probabilities are the softmax over the question's options, and a noul answer is P(true).

๐Ÿ‡ฑ๐Ÿ‡น Built in Lithuania. Amber is named after Baltic amber, gintaras in Lithuanian, which the sea washes up on the Lithuanian coast.

Decision Index 0.3

Public index 37.89 (raw 53.31). Complete run: 140,178 requests, 0 unsupported, nothing truncated. Median 34.0 ms, mean 36.2 ms, p95 62.4 ms per request on one NVIDIA GeForce RTX 3080 Ti (bf16, torch 2.14.0+cu126).

The board (edition 0.4) ranks on the Full score. That score is 60% private tests of the same skills and 40% private decision tasks from new domains, which the maintainers run on submitted models, so it is not out yet. Board models up to 1.25 times this size (board snapshot 2026-11-04):

Model Parameters Public index Full score
Amber 68M 68.9M 37.89 pending
Crivo-68M 68.4M 32.82 19.51
Crivo-32M v2 32.0M 27.34 14.23
Mola-68M 68.3M 27.27 17.60
Crivo-17M 16.9M 22.15 11.65
Bekko System One v0 68M 68.0M 13.01 12.40
Area Skill Raw
Knowledge & Reasoning 9.9 32.6
Language Understanding 32.7 53.7
Retrieval & Classification 63.4 71.2
Tools & Automation 59.6 62.6
Arts & Human Taste 32.9 53.2
Every benchmark in the index
Area Benchmark Metric Raw Skill
Knowledge & Reasoning GPQA Diamond accuracy 0.255 0.007
Knowledge & Reasoning GSM8K accuracy 0.191 0.019
Knowledge & Reasoning ChessBench accuracy 0.086 0.005
Knowledge & Reasoning MuSR accuracy 0.491 0.190
Knowledge & Reasoning SATA-Bench case exact accuracy 0.175 0.164
Knowledge & Reasoning CRUXEval accuracy 0.435 0.104
Knowledge & Reasoning CLadder accuracy 0.522 0.045
Knowledge & Reasoning HLE accuracy 0.134 0.000
Knowledge & Reasoning MMLU-Pro accuracy 0.162 0.058
Knowledge & Reasoning BBH accuracy 0.554 0.353
Knowledge & Reasoning WinoGrande accuracy 0.567 0.133
Language Understanding ContractNLI macro-F1 0.746 0.632
Language Understanding ANLI macro-F1 0.433 0.151
Language Understanding HellaSwag accuracy 0.447 0.263
Language Understanding ACOS per-review F1 0.374 0.354
Language Understanding FinEntity macro-F1 0.682 0.532
Language Understanding iSarcasmEval Sarcasm F1 ยท track A, English 0.261 0.050
Language Understanding VAST macro-F1 0.693 0.540
Language Understanding NLI4CT macro-F1 0.574 0.172
Language Understanding RAGTruth F1 on hallucinated class 0.660 0.296
Retrieval & Classification BANKING77 macro-F1 0.917 0.916
Retrieval & Classification CLINC150 macro-F1 0.894 0.893
Retrieval & Classification BRIGHT nDCG@10 0.246 0.147
Retrieval & Classification Amazon ESCI macro-F1 0.500 0.373
Retrieval & Classification PhishNChips accuracy 0.898 0.797
Retrieval & Classification HoVer accuracy 0.834 0.669
Tools & Automation BFCL case exact accuracy 0.914 0.884
Tools & Automation ToolRet nDCG@10 0.615 0.556
Tools & Automation API-Bank accuracy 0.697 0.691
Tools & Automation Home appliances case exact accuracy 0.000 0.000
Tools & Automation When2Call accuracy 0.829 0.772
Arts & Human Taste BPoMP accuracy 0.907 0.814
Arts & Human Taste Humicroedit accuracy 0.606 0.212
Arts & Human Taste POP909 accuracy 0.011 0.004
Arts & Human Taste cfcolor accuracy 0.600 0.200
Arts & Human Taste Habermas accuracy 0.542 0.335
Arts & Human Taste New Yorker accuracy 0.526 0.408

Policy, the same for every benchmark: one rendering for every benchmark. A string state is used as is and an object state becomes key: value lines; an option is its criteria text, or its key. No abstention, no truncation, no option filtering. A question longer than 32,768 tokens is unsupported.

Use

With the decision-index kit installed and this repository as the working directory:

pip install -r requirements.txt
PYTHONPATH=. python -m decision_index run --engine amber_uni_engine:UniEngine \
    --suite-dir <suite-0.3> --out runs/amber-68m --compact

The scored run used no options. The engine reads the weights, uni_config.json and encoder/ from its own directory, and nothing is downloaded. Options:

  • dtype: bf16 (default) or fp32.
  • token_budget (default 65536): the most padded tokens per forward pass. A request's questions are sorted by length and packed.
  • max_len (default 32768): a question needing more tokens is unsupported.
  • orders (default 1): averages the probabilities over up to 4 fixed option orders (as given, reversed, rotated by half, rotated by half and reversed).
  • Long questions: a question over 2,048 tokens runs alone and unpadded. The encoder's sliding-window layers then attend in blocks to the keys within their window, not through a dense L ร— L mask, which would take about 14 GiB at 30k tokens. The attention is the same: on a 15,200-token question, Amber 17M's probabilities were bit-identical to the dense path.

Files:

File sha256 (first 16)
amber_banner.jpg 78524de2db156af5
amber_uni_core.py 17a4270d6e02cc97
amber_uni_engine.py cf5f5c4f6fe86856
encoder/config.json e93ba6c2e461245b
encoder/tokenizer.json 6c8aaa9a542084f2
encoder/tokenizer_config.json 09baa14905255c98
model.safetensors 735a8667bf1e8406
requirements.txt 13efb642d4e4bd72
uni_config.json aa8833413b98566a

model.safetensors holds the encoder and the option head (120 tensors, bf16). It is the exponential moving average (decay 0.999) of the training weights. encoder/ is the base model's config and tokenizer. Requirements:

safetensors==0.8.0
torch==2.14.0
transformers==5.17.0

Training

One run of 54,000 optimizer steps from cross-encoder/ettin-reranker-68m-v1 at revision d166fa88ddde3c42bc3ee92f7df476d941c8204a (16.7 h on one RTX 3080 Ti, seed 1):

  • Batches. Each step has 4 micro-batches of whole questions, up to 8,192 padded tokens each (about 85 questions per step). Training sequences are up to 4,096 tokens. A longer one is cut to the largest common cap on the instructions, the state and each option that fits; a cut state keeps its head and tail. Inference never cuts.
  • Optimizer. AdamW, learning rate 8e-5 for the encoder and 5e-4 for the head, 5% warm-up, then cosine decay to 0, weight decay 0.01, dropout 0.1.
  • Augmentation. With probability 0.5 a question's options are shuffled. With probability 0.5 its state moves into the instructions, as in many Decision Index rows.
  • Loss. Cross-entropy against soft targets: 0.5 gold and 0.5 the option probabilities of a larger development model. That teacher is an Ettin reranker 150M trained for 6,000 steps on the same mix with gold targets only. A task family where the teacher's top option matches the gold on less than 60% of its rows keeps the gold alone.

Data

Questions are drawn from four stores, with weights 0.45 / 0.10 / 0.30 / 0.15. Every store drops a row whose text (state, question or options) shares a 13-word span or a short sentence with a Decision Index 0.3 suite row, and rows from sources derived from a suite benchmark.

Store Rows What it is Licence
tsj-c3k-clean 390,285 tasksource/tasksource-jev-typed-decisions @6f92aad, train split: at most 3,000 rows per source (358 sources, 215 families). Sources that are a Decision Index benchmark's own dataset are left out, as are 145 families held out for validation and the multilingual set. 1,129 rows were dropped for suite overlap and 6,331 for derived sources. per row (the dataset's license column)
tsj-bench 37,887 The same dataset's rows from the 21 sources that are Decision Index suite datasets, train splits only: ARC, ANLI, BANKING77, CLINC150, CommonsenseQA, ContractNLI, ESCI, HellaSwag, HoVer, Humicroedit, iSarcasm, NLI4CT, OpenBookQA, RAGTruth, WinoGrande. At most 3,000 per source; 5,887 rows dropped for suite overlap. per row; ANLI CC BY-NC 4.0
amber-di 323,836 The Decision Index-format records of Amber 0.4B's training mix, whose card lists sources, splits and filters, plus the MRPC / QQP paraphrase records of its later rounds. Generators built to imitate one benchmark (home appliances, POP909, CLadder, SATA-Bench raw pool) are left out, as are multi-answer and abstaining records. At most 15,000 per source; 61,328 rows dropped for suite overlap. as listed on the Amber 0.4B card (RAGTruth contexts: MS MARCO non-commercial)
fc 134,125 Tool-use decisions parsed from glaiveai/glaive-function-calling-v2, Team-ACE/ToolACE and NousResearch/hermes-function-calling-v1: which tool to call, whether to call one now, ask or decline, which tool description fits, and which reply comes next. 865 rows dropped for suite overlap. Apache-2.0

The weights are released under CC BY-NC 4.0 because the training data includes non-commercial sources (ANLI under CC BY-NC 4.0, RAGTruth's MS MARCO contexts, QQP under Quora's terms). The base model is Apache-2.0.

Limitations

  • Knowledge & Reasoning is its weakest area: a small encoder that does not generate stays near chance on questions that need world knowledge or multi-step arithmetic.
  • Home appliances and POP909 are near zero: the generators built for them were left out of training, because their gains on the public benchmarks did not carry over to the private tests.
  • Mostly English.
  • The probabilities are calibrated on the training mix, not per deployment; re-check thresholds on your own data.
Downloads last month

-

Downloads are not tracked for this model. How to track
Safetensors
Model size
68.9M params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for MLukauskas/amber-68m

Finetuned
(6)
this model