Instructions to use ProjectGaiaUK/gaia-3b-chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ProjectGaiaUK/gaia-3b-chat with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="ProjectGaiaUK/gaia-3b-chat") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("ProjectGaiaUK/gaia-3b-chat") model = AutoModelForCausalLM.from_pretrained("ProjectGaiaUK/gaia-3b-chat", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use ProjectGaiaUK/gaia-3b-chat with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ProjectGaiaUK/gaia-3b-chat" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ProjectGaiaUK/gaia-3b-chat", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/ProjectGaiaUK/gaia-3b-chat
- SGLang
How to use ProjectGaiaUK/gaia-3b-chat with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ProjectGaiaUK/gaia-3b-chat" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ProjectGaiaUK/gaia-3b-chat", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ProjectGaiaUK/gaia-3b-chat" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ProjectGaiaUK/gaia-3b-chat", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use ProjectGaiaUK/gaia-3b-chat with Docker Model Runner:
docker model run hf.co/ProjectGaiaUK/gaia-3b-chat
- Model card —
ProjectGaiaUK/gaia-3b-chat, Round 1, version 1.0- What this model is
- Who made it and why
- Architecture
- How it was trained
- Why this model — the adoption rule, met (12 Nov 2026)
- How to run it
- Evaluation
- Benchmark pass 2 — 641 held-out questions, 1,904 items, English (machine counts, string-level)
- The judged pass — an AI judge's verdicts (Mistral-Small-3.2-24B, prompt v2)
- Intervals on this model's headline figures (12 Nov 2026; no published figure changed)
- Benchmark pass 2, Romanian — the clean held-out set v2ro.1: 107 questions, 313 items (machine counts, string-level)
- Benchmark pass 1 — 31 held-out questions, 90 items, English (machine counts; the small first set)
- Conversation — the chat harness (627 conversations, the fixed quarter; machine counts)
- "Sounds most like a real person" — blind three-way readings by AI readers (Claude, an AI model), never people
- Maths and code — greedy, one answer each, through the chat template (machine counts, Wilson 95 %)
- Long context (machine counts;
reports/eval-chosen/long-v5m-mix70-decline3*-passkey128k.json) - The export, proved (machine counts;
reports/export/gaia-3b-chat-r4-reason-v5m-mix70-decline3/verify.json) - Self-hosted against a metered model — cost per 1,000 answers and the quality trade, on the same items
- Known limits and risks — read these first
- Licence
- Provenance
- How to go back / versions
- What this model is
Model card — ProjectGaiaUK/gaia-3b-chat, Round 1, version 1.0
Released 12 Nov 2026. This model was chosen by the owner (the CEO) on 12 Nov 2026, about 07:35 GMT ("adopt decline3"). Every figure is from this repository's records, or says [not measured] or [pending, job N]; every figure says who read it: machine count (a scorer, a loss, a passkey), an AI judge (Mistral-Small- 3.2-24B, an AI model) or AI readers (Claude, an AI model), never people.
What this model is
A 3-billion-parameter English-and-Romanian conversational language model, trained from scratch by Project Gaia and fine-tuned to hold a conversation, to answer pet-care questions from numbered records placed in its prompt (quoting and citing them), and to say so — in English and in Romanian — when those records do not answer. It is a research model: on these results it is not safe to answer pet-care questions unguarded (see "Known limits and risks", first).
- Published as:
ProjectGaiaUK/gaia-3b-chaton Hugging Face — Project Gaia Round 1, version 1.0. Its base:ProjectGaiaUK/gaia-3b-base-128k. - Name in this research:
gaia-3b-chat-r4-reason-v5m-mix70-decline3("decline3"), Gaia's chosen chat model since 12 Nov 2026. It replacesgaia-3b-chat-r4-reason-v5m-mix70-decline("the decline blend", chosen 11 Nov). - Export:
checkpoints/gaia-3b-chat-r4-reason-v5m-mix70-decline3-hf/on Isambard-AI, Hugging Face Llama format, onemodel.safetensorsin bfloat16 (5,952,097,048 bytes; the folder 5,954,452,604 bytes). Source checkpoint sha2560586f99a…. Export job 8332006, finished 2026-11-12 01:41 UTC. - Languages: English and Romanian. Context: 131,072 tokens. Licence: Apache-2.0. Not a general assistant, not a veterinary adviser; chosen on measurements by machines, an AI judge and AI readers, never confirmed by people.
Who made it and why
Project Gaia is the research of MyPetSitter Ltd (UK-registered; founder and engineer Marian V. Petrisor, the copyright holder of this repository), carried out under UKRI's AI Research Resource (AIRR) award 0261-1548-0489-1 and trained on Isambard-AI.
Acknowledgement. The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR). Isambard-AI is operated by the University of Bristol and is funded by the UK Government’s Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023]. UKRI AI Research Resource (AIRR) Gateway · project 0261-1548-0489-1 · 4,000 GPU hours of computing time. The award supports the research. It is not an endorsement of any product. Cite: McIntosh-Smith, S., Alam S. R. and Woods, C. (2024). “Isambard-AI: a leadership class supercomputer optimised specifically for Artificial Intelligence”. https://doi.org/10.48550/arXiv.2410.11199
How to cite these weights (the same entry as on https://huggingface.co/ProjectGaiaUK; the benchmark's own is its
CITATION.cff, https://huggingface.co/datasets/ProjectGaiaUK/gaia-records-bench):
@misc{petrisor2026gaia3b,
author = {Petrisor, Marian V.},
title = {Project Gaia Round 1: gaia-3b-chat and gaia-3b-base-128k},
year = {2026},
note = {Round 1, version 1.0, released 2026-11-12. Apache-2.0},
url = {https://huggingface.co/ProjectGaiaUK}
}
The research asks (docs/AIRR_GATEWAY_APPLICATION.md) whether a vocabulary and data mix trained jointly for English
and Romanian gives Romanian a fair share (an English-first tokenizer measured here charged Romanian 3.32× the tokens of
English; this project's charges 1.08×), and where a small self-hosted model can and cannot replace a metered API call
for UK pet-care businesses. Round 1 is English and Romanian only (28 Sept 2026); the promised outputs are open weights
under Apache-2.0 with their cards, an open benchmark and the comparison. The research's promise about data, "No model
is trained on pet-care answers written by another AI model", is spelled out under "How it was trained".
Architecture
The same shape as the base model gaia-3b-128k (gaia/model.py, gaia/config.py, configs/gaia-3b-128k.yaml);
the export's config.json is LlamaForCausalLM.
| parameters | 2,976,033,792 (2,774,707,200 without the two embedding matrices) |
| layers | 28 |
width (d_model) |
3,072 |
| attention | 24 query heads sharing 6 key-value heads (grouped-query attention), head size 128, no biases |
| feed-forward | SwiGLU, hidden 8,192 |
| normalisation | RMSNorm (pre-norm), eps 1e-5, computed in float32 |
| positions | RoPE, theta 64,000,000, no position scaling |
| context | 131,072 tokens |
| vocabulary | 32,768 byte-level BPE tokens, 16 of them reserved (ids 0–15); frozen, sha256 da8375e6082226fc64e8222a08504c88bb75d617a6ed38b00d24d7dd169df9a5 |
| embeddings | input and output untied |
| weights as exported | bfloat16 (the training checkpoint is float32) |
Pre-norm RMSNorm → GQA attention with RoPE → residual → RMSNorm → SwiGLU → residual, then a final RMSNorm and a
linear head; Llama's format needed no rearranging of the q/k rows (scripts/export_llama.py, proved by logits below).
How it was trained
The model is the end of a chain. Every stage below is recorded in docs/RUNLOG_3B.md, docs/CHAT_SFT.md,
docs/REASON_GEN.md, docs/SFT_DECLINE.md, docs/DECLINE_V3_PLAN.md and docs/FINISH_PLAN.md; the data in
docs/DATA_CARD.md.
| # | stage | checkpoint | from | what changed |
|---|---|---|---|---|
| 0 | the 3B base, pre-trained from scratch (150B tokens at 2,048; then 8k, 32k and 128k long-context phases, about 3B tokens each) | gaia-3b-128k, step 2,850 |
random weights | see MODEL_CARD_gaia-3b-base.md |
| 1 | round 4 chat fine-tune (SFT) on chat set v5-r4 | gaia-3b-chat-sft-r4, step 1,854 |
0 | learned to converse, to read and cite records |
| 2 | v5m: maths worked examples with replay | gaia-3b-chat-r4-reason-v5m, step 2,114 |
1 | GSM8K more than doubled |
| 3 | ro2: the balanced Romanian replay round | gaia-3b-chat-r4-reason-v5m-ro2, step 1,985 |
2 | Romanian preference up, English and multi-turn down (not adopted on its own) |
| 4 | mix70: a weight average, 0.7 × v5m + 0.3 × ro2 (no training) | gaia-3b-chat-r4-reason-v5m-mix70 |
2, 3 | kept v5m's conversation, took part of ro2's Romanian |
| 5 | decline-v3: the decline fine-tune, English and Romanian | gaia-3b-chat-r4-decline-v3, step 400 |
1 | learned to say the records do not answer, in the question's language |
| 6 | the chosen model = mix70 + (decline-v3 − round 4), a weight change added, no training | gaia-3b-chat-r4-reason-v5m-mix70-decline3 |
4, 5, 1 | declines when its records do not answer, in English and Romanian |
Stage 1 — round 4 chat SFT (configs/gaia-3b-chat-sft-r4.yaml, job 8205960, 8 Nov 2026). From the 128k base's
final weights. Chat set v5-r4: 107,739 training conversations, 43,996,012 tokens, 19,916,323 learned (the
assistant's words and their <|end|>; everything else is context). 3 epochs = 1,854 steps, about 133.10M tokens
processed (the plan's figure), 65,536 tokens a step on 8 GPUs; rate 2e-5, 10 warm-up steps, cosine to a tenth, no weight
decay, clip 1.0; conversations packed into 4,096-token sequences, the long-document ones in sequences of 16k–131,072
tokens. Best validation loss 1.633. GPU hours [not measured] in these records (the plan's estimate: about 5.4).
The data, by who wrote it (per-source counts in the config's header; licences on the data card):
- Written by people: OpenAssistant oasst2 (Apache-2.0; 4,812 English + 11 Romanian), Aya English original annotations (Apache-2.0; 1,951), Topical-Chat (CDLA-Sharing-1.0; 8,313), Persuasion for Good (Apache-2.0; 814), CaSiNo (CC BY 4.0; 866), doc2dial (CC BY 3.0; 3,191), Taskmaster 1–4 (CC BY 4.0; 8,950), MultiWOZ 2.2 (MIT; 588), ConvAI2 (CC BY 4.0; 17,194), Blended Skill Talk with no model suggestion chosen (CC BY 4.0; 2,045), PhotoChat up to the photo (CC BY 4.0; 10,980), the Self-dialogue Corpus (BSD-3-Clause; 23,250).
- The pet-care knowledge — written by people or quoted from open-licence pages: SFT set v1 (frozen 29 Oct 2026): 923 examples whose answers are, word for word, sections of GOV.UK, Defra, nidirect, DAERA and mygov.scot pages (OGL v3.0) and EUR-Lex Romanian texts (CC BY 4.0 / Commission Decision 2011/833/EU), read from numbered records in the prompt and cited; 730 English, 52 Romanian, plus 64 long-document conversations built from the same examples.
- Written by an AI model — conversation style only, pet care screened out. Every conversation touching pets,
animals, animal health, veterinary matters, feeding, training, care, welfare or boarding was dropped before training
(the pre-training's pet-care vocabulary, English and Romanian keyword lists, and for English a topic classifier):
Synthetic-Persona-Chat part 2 (CC BY 4.0; a Google 24B instruction-tuned model; 4,561 kept of 11,001 candidates),
EuroMultiSynth-Instruct Romanian (Apache-2.0;
google/gemma-4-31B-it; 14,301 kept), and 5,066 of this project's own English conversations written by GLM-4.7-FP8 (MIT) in four waves that passed a gate (two fresh AI readers reading at least 85 % of an 80-conversation sample as good: 70, 72, 68 and 70 of 80). - Held out, never trained on: each source's test split (1,772 conversations), 101 pet-care questions from 13 whole SFT v1 pages, 160 long-context items; no training conversation near-duplicates a test or benchmark question.
Stage 2 — v5m, the maths worked examples (configs/gaia-3b-chat-r4-reason-v5m.yaml, job 8260934, 10 Nov 2026).
From round 4. data/sft/reason-v5m: 16,912 rows = 6,343 maths worked examples + 10,569 replayed rows of chat set
v5-r4's own train split (231 replay rows near a benchmark question dropped). 2 epochs = 2,114 steps, rate 1e-5, 10
warm-up steps, cosine to a tenth, 4 GPUs. The 6,343 maths examples ARE AI-written: problem, worked steps and final
answer were written by GLM-4.7-FP8 (zai-org/GLM-4.7-FP8, MIT), each kept only if a separately written checker program
agreed with it when run in a no-network sandbox, with any animal or pet-care word excluded and anything near a HumanEval,
MBPP or GSM8K item dropped. They carry no pet-care knowledge. The code half of the set (4,457) was left out because it
cost code ability. Read by two fresh AI readers (never people): 36–39 of 40 maths examples fit to teach from, by batch.
Stage 3 — ro2, the balanced Romanian replay (configs/gaia-3b-chat-r4-reason-v5m-ro2.yaml, job 8299496, 11 Nov
2026). From v5m. 31,747 rows, nothing new and nothing generated: every Romanian row of round 4's chat set (14,363:
EuroMultiSynth 14,301, SFT v1 51, oasst2 11), an English replay of equal size (14,384; 90 % from conversations of 5+
user turns) and 3,000 of v5m's maths rows. 1 epoch = 1,985 steps at 5e-6; 21 min 43 s of training on 4 GPUs.
Stage 4 — mix70 (scripts/merge_ckpt.py, job 8303289, about 20 s, 11 Nov 2026). The weights of v5m and ro2
averaged in float32, 0.7 × v5m + 0.3 × ro2. No data, no training. The largest difference between any v5m and ro2 weight
was 0.0047.
Stage 5 — decline-v3 (configs/gaia-3b-chat-r4-decline-v3.yaml, job 8329594, 7 min 11 s, 12 Nov 2026; built by
data/build_sft_decline_v3.py, docs/DECLINE_V3_PLAN.md). From round 4 (not from mix70); the same configuration as
decline-v2 with only the data changed. 3,187 rows (sha256 6d12ef36…):
- decline set v2 unchanged, byte for byte (2,861): decline set v1 (wrong-species 257, same-species-other-page 273, owner-wrong-species 150, owner-same-species 100, answerable 485, contrast-answerable 265) and 1,331 ordinary people-written conversations with no records, so the decline does not leak into ordinary chat;
- 326 Romanian rows: 136 answerable (a section's own question with that section among sections of other pages; the answer is the section's own words and its citation), 181 Romanian declines (145 from sections of other pages, 36 from SFT v1's Romanian questions with records of another regulation or unrelated English records) and 9 contrast rows. Their sources: 43 Romanian EUR-Lex summaries registered on the data card before they were fetched (CC BY 4.0; 263 sections, 154 usable) and SFT v1's Romanian examples.
The declining targets are two fixed sentences written by the company, carrying no pet-care knowledge: "My records don't cover that, so I don't want to guess. Please ask your vet." and, confirmed word for word by the owner on 12 Nov 2026, "Înregistrările mele nu acoperă acest subiect, așa că nu vreau să ghicesc. Vă rog să vă adresați medicului veterinar." Kept out of training, checked on what was written (7 tests, each guard shown failing on an injected fault): every record of every benchmark item (gold or neighbour), the 18 held-out Romanian pages, and every question near a benchmark question. 2 epochs = 400 steps at 1e-5. No AI-written answer was added.
Stage 6 — the chosen model (scripts/merge_ckpt.py --add/--sub, job 8329595, 12 Nov 2026): mix70 plus the weight
change the decline-v3 fine-tune made to round 4, mix70 + (decline-v3 − round 4); no data, no training. Adopted by the
owner against the rule of docs/DECLINE_V3_PLAN.md step 4 (below, "Why this model").
What the promise covers, plainly. Every pet-care fact the model was trained to say comes from people or open-licence pages (SFT v1 and the registered EUR-Lex summaries), and the decline sentences are the company's. AI-written text is in the training, of two kinds: the conversations of the three model-written sources (style only, pet care screened out) and the maths worked examples (no pet care). The decline set's English question text came from the project lead's banks, whose files do not say how they were written; its Romanian questions are the EUR-Lex pages' own headings.
Why this model — the adoption rule, met (12 Nov 2026)
Against the decline blend it replaces, on the same items; paired differences resample the items 4,000 times.
| condition | decline3 | the decline blend | who read it |
|---|---|---|---|
| says the records do not answer, Romanian absent items (clean set v2ro.1, 99) | 0.879, every one the Romanian sentence | 0.242, every one in English | machine count |
| … at least half as often as in English | 0.879 against 0.838 | — | machine count |
| says the records do not answer, English absent items (622) | 0.838; paired +0.011 [−0.011, +0.034] | 0.826 | machine count |
| the role slip ("Bună ziua. Am nevoie de ajutorul dumneavoastră…", Romanian absent items, 104) | 0 | 29 | machine count |
| everything else on items v2 (paired) | nothing worse; cites the gold +0.019 [+0.007, +0.030] | — | machine count |
| answerable items judged correct / wrong, English (pass 2) | 0.875 / 0.032 | 0.814 / 0.051 | AI judge |
| answerable items judged correct / wrong, Romanian (v2ro.1) | 0.935 / 0.028 | 0.850 / 0.065 | AI judge |
| maths and code | within their intervals (below) | machine count | |
| the chat harness | level (below) | machine count | |
| blind Romanian openings against mix70 | +2 [−14, +18] | −10 [−26, +6] | AI readers |
| 128k passkey, the same 200 keys | 179 (+9 [+4, +15] on the blend, exact McNemar p 0.004) | 170 | machine count |
The middle point decline3-85 (0.85 × the v3 change) was measured and did not meet the rule (English declining −0.072 [−0.096, −0.050] against the blend) for two more passkey keys (181).
How to run it
The export is a standard Hugging Face Llama folder: config.json, model.safetensors (bfloat16), tokenizer.json,
tokenizer_config.json (the special tokens and the chat template), generation_config.json, README.md,
SHA256SUMS. Verified with transformers 5.14.1, tokenizers 0.22.2, safetensors 0.8.0 and vLLM 0.27.1 (NGC 26.08
image, torch 2.14.0a0) on an NVIDIA GH200.
The chat template (gaia/sft.py ChatTemplate, written into tokenizer_config.json as Jinja): <|bos|>, then
for each message its role token, a newline, its text and <|end|>; for generation, <|assistant|> and a newline:
<|bos|><|system|>\n{system text}<|end|><|user|>\n{question}<|end|><|assistant|>\n
Roles are system (optional, first), user and assistant. The template refuses (raises) any other role, a message
containing a reserved token, a generation prompt that is empty or ends on an assistant turn, and, without a generation
prompt, anything but [system,] user, assistant pairs; vLLM therefore logs a "chat template warm-up failed" warning at
start-up, and serving is not affected. Stop tokens: <|end|> (id 5) and <|endoftext|> (id 0), both set as
eos_token_id in generation_config.json. <|bos|> is id 1 and the template adds it (add_bos_token false); padding
is <|endoftext|>; ids 0–15 are <|endoftext|>, <|bos|>, <|system|>, <|user|>, <|assistant|>, <|end|>,
<|tool_call|>, <|tool_result|> and eight reserved slots (the tool tokens untrained).
transformers: AutoTokenizer.from_pretrained(dir), AutoModelForCausalLM.from_pretrained(dir, torch_dtype=torch.bfloat16), tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt"),
model.generate(..., eos_token_id=[5, 0]). The template's ids equal Gaia's own encoder's for every text (90/90 benchmark
prompts and 90/90 examples equal). vLLM: vllm serve <dir> with the folder's own template and stop ids (the records:
bfloat16, greedy, enforce_eager). Sampling in the records: greedy for every reference figure; temperature 0.7,
top-p 0.9 for the benchmark's seeded repeats; the harness generated up to 768 new tokens; benchmark contexts are cut to
1,800 tokens.
Published without a GGUF file: a GGUF build follows once the tool reads this vocabulary. llama.cpp's
convert_hf_to_gguf.py --vocab-only refuses the folder (BPE pre-tokenizer was not recognized), as it refused the
earlier exports; llama.cpp was not patched. The safetensors in bfloat16 are the published weights.
Evaluation
Who read what. Benchmark, harness, loss, passkey and code/maths figures are machine counts. The judged pass is an AI judge's verdict (Mistral-Small-3.2-24B-Instruct-2506, prompt v2; majority of three runs). The blind readings and the reading of repeat disagreements are by AI readers (Claude, an AI model), never people. The arms compared: a commercial model served by its provider (arm 1), an open 4B model as released (arm 2) and the same fine-tuned on SFT v1 (arm 3). Intervals are 95 % where the records give them.
Benchmark pass 2 — 641 held-out questions, 1,904 items, English (machine counts, string-level)
Items v2 (data/bench/p2/items-v2.jsonl, sha256 2c3b223d…): sections of 66 newly registered OGL pages (GOV.UK,
nidirect, mygov.scot), held out by construction; each question in three contexts: placed (the gold record among two
neighbours, 641), absent (the gold removed, 622; the right reply is that the records do not answer) and retrieved
(the retriever's own, 641; gold present for 558). Greedy on every item, three seeded repeats on a fixed quarter (464
items). Each cell: greedy · the repeats' mean. Every arm: reports/bench-p2/summary.md.
| measure (n) | this model | the decline blend (replaced) | mix70 | commercial model (arm 1) | open 4B model as released (arm 2) |
|---|---|---|---|---|---|
| cites the gold record, retrieved (558) | 0.953 · 0.934 | 0.946 · 0.937 | 0.964 · 0.954 | 0.998 · 0.998 | 0.975 · 0.966 |
| cites only the gold record, placed (641) | 0.928 · 0.923 | 0.908 · 0.917 | 0.894 · 0.912 | 0.788 · 0.773 | 0.886 · 0.833 |
| retrieval end to end (641) | 0.830 · 0.821 | 0.824 · 0.823 | 0.839 · 0.838 | 0.869 · 0.876 | 0.849 · 0.848 |
| hedge carried (85) | 0.965 · 0.900 | 0.941 · 0.900 | 0.941 · 0.900 | 0.718 · 0.933 | 0.471 · 0.433 |
| vet referral carried (63) | 0.968 · 0.889 | 0.936 · 0.815 | 0.905 · 0.759 | 0.841 · 0.926 | 0.381 · 0.389 |
| vet referral where none is due (1,136) | 0.004 · 0.012 | 0.003 · 0.006 | 0.002 · 0.004 | 0.034 · 0.021 | 0.013 · 0.012 |
| says the records do not answer, absent (622) | 0.838 · 0.785 | 0.826 · 0.761 | 0.022 · 0.011 | 0.584 · 0.564 | 0.424 · 0.410 |
| stopped cleanly (1,904) | 0.998 · 0.997 | 0.997 · 0.998 | 0.993 · 0.991 | 0.996 · 1.000 | 1.000 · 1.000 |
The other arms' greedy decline rates: 3B SFT v1 0.029, 1B SFT v1 0.032, the open 4B model fine-tuned on SFT v1 0.030. Against mix70 the decline costs about one point of retrieval end to end (0.830 against 0.839). A decline sentence where a record does answer: 69 of 1,282 answerable items (the blend 67). Arm 1 is incomplete on 3 items: the provider's content filter refused 3 of arm 1's greedy prompts; each refusal is recorded as its answer (empty, stopped "filtered").
The string-level scorer on borderline answers (AI readers, never people). Every item × measure row where the three repeats disagreed was read blind, each answer in full: on pass 2's 448 rows (1,344 answers, 16 readers) the scorer agreed with the readers on 82.5 % — citations 99–100 %, clean stops 97 %, vet referral 73 %, declines 65 %, cautions 64 %; on this model's later rows 95.2 %; on pass 1's 311 rows (933 answers, 16 readers) 89.3 %; on the Romanian set's 100 rows 87.3 %. Declines, cautions and vet referrals are where the string scorer is weak, and where the judged pass earns its place. (A first reading on 11 Nov gave 64 %: it was rendered from a file that keeps only 600 characters of each answer; corrected the same night.)
The judged pass — an AI judge's verdicts (Mistral-Small-3.2-24B, prompt v2)
Majority of three judge runs; greedy answers; "correct" etc. are shares of the answerable items. Prompt v2 (12 Nov 2026) counts a decline only when the answer says the records do not answer; under v1 the judge also called answers that missed the question (such as the Romanian role slip) declines. Checked against AI readers: on 100 English answers v2 agrees with them on 89.0 % (v1 89.0 %); on 90 Romanian absent answers 96.7 % (v1 80.0 %). Every arm's "correct" share is the same under both prompts.
| arm | pass 2 English: correct · wrong · absent declined · species errors | Romanian v2ro.1: correct · wrong · absent declined | pass 1: correct · wrong · absent declined |
|---|---|---|---|
| this model | 0.875 · 0.032 · 0.833 · 23 | 0.935 · 0.028 · 0.879 | 0.742 · 0.032 · 0.821 |
| the decline blend | 0.814 · 0.051 · 0.828 · 22 | 0.850 · 0.065 · 0.333 | 0.742 · 0.065 · 0.750 |
| mix70 | 0.838 · 0.083 · 0.002 · 68 | 0.893 · 0.042 · 0.000 | 0.774 · 0.081 · 0.000 |
| 3B SFT v1 | 0.930 · 0.044 · 0.005 · 57 | 0.986 · 0.014 · 0.000 | 0.887 · 0.048 · 0.000 |
| commercial model (arm 1) | 0.931 · 0.005 · 0.572 · 8 | 0.991 · 0.000 · 0.576 | 0.984 · 0.000 · 0.857 |
| open 4B model as released (arm 2) | 0.912 · 0.014 · 0.453 · 10 | 0.958 · 0.000 · 0.444 | 0.919 · 0.000 · 0.714 |
| open 4B model fine-tuned on SFT v1 (arm 3) | 0.935 · 0.044 · 0.006 · 51 | 0.977 · 0.000 · 0.020 | 0.919 · 0.032 · 0.000 |
The judge agreed with itself on 99.3 % of this model's pass-2 answers; run a second time on the same answers it gave the same majority verdict on 99.96 % (machine count). A second open judge of another family, EuroLLM-22B (Apache-2.0), gives the same majority verdict as Mistral on 86.9 % of pass 2's 26,368 answers; under prompt v2 it misses real declines (an AI reader check: 67.0 % English, 78.9 % Romanian), so it is a cross-check for answerable verdicts only and its decline counts are not quoted.
Intervals on this model's headline figures (12 Nov 2026; no published figure changed)
Each greedy figure (the published one) has a 95 % interval from a bootstrap over questions: the items of one question
are resampled together, 2,000 times. The sampled repeats (temperature 0.7, top-p 0.9) are the three published (s1–s3) and
three more run on 12 Nov for this (s4–s6, jobs 8344940–8344941; judged 8344944–8344945), on the same fixed quarter. The
six-run mean has a 95 % t-interval. reports/intervals/.
| figure (n) | greedy [95 %] | repeats: s1–s3 mean · all six [95 % of the mean] | read by |
|---|---|---|---|
| says the records do not answer, English absent (622; 152 a run) | 0.838 [0.808–0.867] | 0.785 · 0.779 [0.761–0.796] | machine count |
| … Romanian clean set v2ro.1 (99; 27 a run) | 0.879 [0.812–0.941] | 0.815 · 0.821 [0.805–0.837] | machine count |
| cites only the gold record, placed (641) | 0.928 [0.908–0.949] | 0.923 · 0.923 [0.911–0.935] | machine count |
| cites the gold record, retrieved (558) | 0.953 [0.935–0.971] | 0.934 · 0.937 [0.929–0.945] | machine count |
| retrieval end to end (641) | 0.830 [0.802–0.858] | 0.821 · 0.823 [0.816–0.830] | machine count |
| hedge carried (85; 20 a run) · vet referral carried (63; 18 a run) | 0.965 [0.905–1.000] · 0.968 [0.895–1.000] | 0.900 and 0.889 in every one of the six runs | machine count |
| vet referral where none is due (1,136) | 0.004 [0.001–0.010] | 0.012 · 0.012 [0.010–0.013] | machine count |
| stopped cleanly (1,904) | 0.998 [0.995–1.000] | 0.997 · 0.997 [0.995–0.999] | machine count |
| answerable judged correct · wrong, English | 0.875 [0.854–0.895] · 0.032 [0.021–0.043] | 0.865 · 0.867 [0.864–0.871]; wrong 0.051 · 0.052 [0.050–0.055] | AI judge |
| answerable judged correct · wrong, Romanian v2ro.1 | 0.935 [0.897–0.972] · 0.028 [0.009–0.051] | 0.907 · 0.914 [0.898–0.930]; wrong 0.031 · 0.025 [0.015–0.035] | AI judge |
| species errors (a count) | 23 of 1,904 [13–34] | 9 in each of the six runs on the 464-item quarter | AI judge |
| the 128k passkey, 200 fresh keys a seed | 179 of 200 [0.845–0.930] (seed 20261111) | two new seeds: 174 and 180 of 200; pooled 533 of 600 = 0.888 [0.861–0.911] | machine count |
| the chat harness, all 2,660 items (Wilson) | stops 2,608 = 98.1 % [97.5–98.5]; cites the gold 117 of 261 = 44.8 % [38.9–50.9]; preferences 379 of 572 = 66.3 % [62.3–70.0]; hedge 25/28 [73–96]; referral 26/29 [74–96]; vet where none is due 7/232 = 3.0 % [1.5–6.1] | greedy only | machine count |
What the intervals say. Every published greedy figure stands within its interval. The sampled answers decline less often than the greedy ones: in English 0.779 across six runs, against 0.838 greedy. That difference is the decoding (sampling at 0.7), not the seed: the six runs range from 0.757 to 0.803, all well below 0.838. The passkey on new keys (174, 180) brackets the published 179. The comparison with mix70 (189 of 200) was made on the published keys only and is not repeated here. The maths and code figures are greedy, one answer a problem, with their Wilson intervals already in their table. Sampled repeats there would measure a different figure, so none were run. The long-context losses are one deterministic pass over fixed held-out sets and carry no interval.
Benchmark pass 2, Romanian — the clean held-out set v2ro.1: 107 questions, 313 items (machine counts, string-level)
Items v2ro.1 (data/bench/p2ro1/items-v2ro1.jsonl, sha256 ace06e77…; data/build_bench_p2ro1.py): sections of 18
Romanian pages registered on the data card before they were fetched (16 EUR-Lex summaries on animal health, welfare,
transport, veterinary medicines and feed, 2 Your Europe pet-travel pages; CC BY 4.0), with the 5 questions (15 items)
removed that are near a question any model was trained on. Greedy on every item, three seeded repeats on a fixed quarter.
Each cell: greedy · the repeats' mean. Every arm: reports/bench-p2ro1/summary.md.
| measure (n) | this model | the decline blend | mix70 | 3B SFT v1 | commercial model (arm 1) | open 4B model as released |
|---|---|---|---|---|---|---|
| cites the gold record, placed (107) | 0.953 · 0.988 | 0.925 · 0.963 | 0.953 · 0.975 | 0.981 · 1.000 | 1.000 · 1.000 | 1.000 · 1.000 |
| cites only the gold record, placed (107) | 0.953 · 0.963 | 0.925 · 0.938 | 0.944 · 0.963 | 0.981 · 1.000 | 0.851 · 0.914 | 0.963 · 1.000 |
| retrieval end to end (107) | 0.935 · 0.914 | 0.906 · 0.926 | 0.935 · 0.938 | 0.991 · 0.963 | 0.991 · 0.963 | 0.981 · 0.963 |
| hedge carried (24) | 1.000 · 1.000 | 0.875 · 0.500 | 1.000 · 1.000 | 1.000 · 1.000 | 0.625 · 0.500 | 0.458 · 1.000 |
| says the records do not answer, absent (99) | 0.879 · 0.815 | 0.242 · 0.284 | 0.000 · 0.000 | 0.000 · 0.000 | 0.646 · 0.704 | 0.475 · 0.481 |
| stopped cleanly (313) | 0.997 · 0.992 | 0.994 · 1.000 | 0.997 · 1.000 | 0.981 · 0.984 | 1.000 · 1.000 | 0.997 · 1.000 |
It declines in Romanian. Every one of its declines on the Romanian absent items is the confirmed Romanian sentence (none in English); it never begins "Bună ziua. Am nevoie de ajutorul dumneavoastră…" (the decline blend did on 29 of 104, writing as the person asking). A decline sentence where a record does answer: 4 of the 214 answerable items of v2ro.1 (the blend 1); 6 of 224 on all 112 questions (the blend 3). The string scorer's Romanian decline forms were widened on 11 Nov 2026 and checked against an AI reader (96.7 % agreement on 90 answers). On all 112 questions (items v2ro, before the 15 were removed) the figures move by at most 0.025.
Benchmark pass 1 — 31 held-out questions, 90 items, English (machine counts; the small first set)
Items v1 (sha256 f0aa0274…): 8 web-page questions and 23 from Defra's welfare codes (PDF text, columns can run
together); placed 31, absent 28, retrieved 31 (gold present for 26). Greedy · repeats' mean.
| measure (n) | this model | the decline blend | mix70 | round 4 SFT | commercial model | open 4B model as released |
|---|---|---|---|---|---|---|
| cites the gold record, placed (31) | 1.000 · 0.968 | 0.935 · 0.946 | 0.935 · 0.935 | 1.000 · 0.925 | 1.000 · 1.000 | 1.000 · 1.000 |
| retrieval end to end (31) | 0.839 · 0.828 | 0.806 · 0.806 | 0.839 · 0.828 | 0.839 · 0.839 | 0.839 · 0.839 | 0.839 · 0.839 |
| hedge carried (14) | 0.929 · 0.929 | 0.929 · 0.952 | 0.929 · 0.952 | 0.929 · 0.952 | 0.786 · 0.643 | 0.143 · 0.167 |
| vet referral carried (12) | 1.000 · 1.000 | 1.000 · 1.000 | 1.000 · 1.000 | 1.000 · 1.000 | 0.917 · 0.944 | 0.667 · 0.639 |
| vet referral where none is due (45) | 0.000 | 0.000 | 0.000 | 0.000 | 0.156 · 0.104 | 0.044 · 0.022 |
| says the records do not answer, absent (28) | 0.821 · 0.821 | 0.750 · 0.762 | 0.000 | 0.000 | 0.893 · 0.619 | 0.500 · 0.476 |
| stopped cleanly (90) | 0.989 · 0.982 | 0.978 · 0.993 | 0.978 · 0.982 | 1.000 · 0.982 | 1.000 | 1.000 |
Conversation — the chat harness (627 conversations, the fixed quarter; machine counts)
scripts/chat_harness.py, greedy, up to 768 new tokens, on chat set v3.2's held-out items (390 single replies, 171
multi-turn, 21 pet-care, 45 long-document): stopped cleanly 98.4 %, repeated 4-grams 0.0123, right language 627/627, the
people's preferred reply ranked higher on 66.3 % of held-out pairs; on the 66 record-bearing items it cited the gold record
39.4 %, carried the hedge 6/8 and the referral 6/8 (the decline blend on the same quarter: stops 97.6 %, repeats 0.0148,
preferences 66.3 %, cited 36.4 %, hedge 8/8, referral 7/8; round 4 on all 2,660 items: stops 95.5 %, preferences 68.4 %,
cites the gold 44.1 %). A decline sentence in ordinary chat: none in the single and multi-turn conversations; 1 of the
45 long-document items.
On all 2,660 items (job 8337120; machine counts; round 4 SFT on the same items in brackets): stopped cleanly 98.1 % (95.5 %), repeated 4-grams 0.0153 (0.0504), right language 2,658 of 2,660 (2,660), the people's preferred reply ranked higher 66.3 % (68.4 %); on the 261 record-bearing items it cited the gold record 44.8 % (44.1 %), carried the hedge 25 of 28 (27) and the referral 26 of 29 (23), and referred to a vet where none was due on 7 of 232 (2). A decline sentence: none in the 2,399 single and multi-turn conversations; 3 of the 101 pet-care items and 4 of the 160 long-document items.
"Sounds most like a real person" — blind three-way readings by AI readers (Claude, an AI model), never people
15 fresh Claude instances each read one part of 20 items with the brief and no key; the replies of the three systems
were shuffled behind letters; scored on Isambard against the key (reports/eval-3b-blind/blinded-decline3-mix70-r4/,
multi2/blinded-decline3-mix70-r4/). Share of items picked, with 95 % interval; the paired column resamples the same items
4,000 times.
| items | round 4 SFT | mix70 | this model | paired, this model − mix70 |
|---|---|---|---|---|
| Romanian openings (n 100, 4 ties) | 30 % [22–40] | 32 % [24–42] | 34 % [25–44] | +2 [−14, +18] |
| English openings (n 100, 15 ties) | 27 % [19–36] | 22 % [15–31] | 36 % [27–46] | +14 [0, +29] |
| multi-turn, English (n 100, 6 ties) | 25 % [18–34] | 31 % [23–41] | 38 % [29–48] | +7 [−9, +23] |
| multi-turn coherence, 1–5 | 2.94 [2.75–3.13] | 3.22 [3.04–3.40] | 3.02 [2.83–3.21] |
The decline blend, read the same way on 11 Nov: Romanian −10 [−26, +6], English +4, multi-turn +7 against mix70. The earlier readings in the chain are under "Known limits", 3. No person has read any of these conversations.
Maths and code — greedy, one answer each, through the chat template (machine counts, Wilson 95 %)
| set | this model | the decline blend | mix70 | round 4 SFT | the 3B base (plain prompts) |
|---|---|---|---|---|---|
| GSM8K (1,319) | 7.7 % (102) [6.4–9.3] | 7.6 % (100) [6.3–9.1] | 9.0 % [7.6–10.7] | 3.9 % [3.0–5.1] | 3.6 % |
| HumanEval (164) | 7.9 % (13) [4.7–13.1] | 10.4 % (17) [6.6–16.0] | 12.2 % [8.0–18.1] | 10.4 % [6.6–16.0] | 11.0 % |
| MBPP, test 11–510 (500) | 11.4 % (57) [8.9–14.5] | 12.4 % (62) [9.8–15.6] | 13.6 % [10.9–16.9] | 11.6 % [9.1–14.7] | 13.8 % |
Paired differences on the same problems [not measured] for this model. Model-written code ran only in a no-network sandbox. Contamination of these three sets in the pre-training text was not checked (below).
Long context (machine counts; reports/eval-chosen/long-v5m-mix70-decline3*-passkey128k.json)
The 128k passkey on fresh keys (40 keys at each of 5 depths, the same 200 keys for every model, each key's outcome recorded; reruns on the same seed reproduce every total exactly):
| this model | the decline blend | mix70 | the 128k base | |
|---|---|---|---|---|
| keys found of 200 | 179 (0.895) [0.845–0.928] | 170 (0.85) [0.795–0.893] | 189 (0.945) [0.904–0.969] | 168 (0.84), the same keys (job 8290399) |
| by depth 0 / 25 / 50 / 75 / 100 % | 0.75 / 1.00 / 1.00 / 0.975 / 0.75 | 0.75 / 1.00 / 0.875 / 0.975 / 0.65 | 0.925 / 1.00 / 1.00 / 1.00 / 0.80 | 0.95 / 1.00 / 1.00 / 0.975 / 0.275 |
Key by key: this model finds every key the decline blend finds and 9 more (+9 [+4, +15], exact McNemar p = 0.004); against mix70 it finds 10 fewer (178 both, 11 only mix70, 1 only this model; −10 [−17, −4], p = 0.006). Teaching the model to say when its records do not answer costs long-context recall, and this model halves that cost; the rest is real.
Every length, and held-out loss (job 8336701, 20 keys a depth, seed 20261018 — the long evaluation every chosen model had; machine counts):
| this model | the decline blend | mix70 | the 128k base | |
|---|---|---|---|---|
| held-out loss at 2k, English / Romanian / pet care / Python / maths | 2.419 / 2.702 / 1.774 / 0.610 / 1.443 | 2.414 / 2.701 / 1.765 / 0.608 / 1.440 | 2.307 / 2.609 / 1.615 / 0.573 / 1.348 | 2.178 / 2.352 / 1.434 / 0.528 / 1.236 |
| passkey, mean of 5 depths: 2k / 4k / 8k / 16k / 32k / 64k / 128k | 0.95 / 0.96 / 0.98 / 0.99 / 0.99 / 0.99 / 0.90 | 0.95 / 0.96 / 0.98 / 0.99 / 0.99 / 0.99 / 0.84 | 0.97 / 0.96 / 0.98 / 0.99 / 1.00 / 0.99 / 0.90 | 0.84 / 0.73 / 0.80 / 0.67 / 0.86 / 0.81 / 0.79 |
| long-window loss, English 96–128k / pet care 0–32k | 1.922 / 0.839 | 1.917 / 0.837 | 1.820 / 0.617 | 1.685 / 0.306 |
Long context is kept at every length up to 64k; the cost is at 128k (above). Plain-text loss is the decline blend's to within 0.01 on every source, and above mix70's: the decline additions cost held-out loss (English 2.307 → 2.419, pet care 1.615 → 1.774); chat fine-tunes are not judged on it, but it is written here.
The export, proved (machine counts; reports/export/gaia-3b-chat-r4-reason-v5m-mix70-decline3/verify.json)
SHA256SUMS ok (6 files). Chat template against ChatTemplate: 90/90 benchmark prompts, 90/90 examples, the same
refusals. Largest logit difference, transformers float32 on the export against Gaia float32 on 8 prompts: 0.68;
against Gaia with its weights rounded to bf16 (the conversion alone): 2.5e-4; greedy next token 8/8 the same. vLLM
bf16: an English prompt 36/36 tokens identical; two stored pass-1 absent answers 20/20 and 109/109 identical; a second
prompt 16/70 (bf16 against float32 arithmetic, as with earlier exports). TF32 off. The published weights give the
published numbers: the benchmark served from the exported bf16 weights by vLLM through the same runner (the "export arm",
jobs 8332007–8332008) gives greedy answers identical in text to the float32 run's on 1,863 of 1,904 pass-2 items and 320
of 328 Romanian items, with the same decline rates (0.838 and 0.865 both ways). On pass 1 (jobs 8336702–8336703) the export's
greedy answers are identical in text on 87 of 90 items with the same decline and retrieval scores (0.821, 0.839). The
seeded repeats (temperature 0.7) part where bf16 and float32 sample differently — identical text on 74–80 of 90 pass-1
items, 429 of 464 pass-2 items and 69–71 of 84 Romanian items a run (jobs 8336702–8336705) — and their means agree:
says the records do not answer 0.785 against 0.781 (pass 2), 0.821 against 0.810 (Romanian), 0.821 on every pass-1 run.
Self-hosted against a metered model — cost per 1,000 answers and the quality trade, on the same items
The research's third output: where a small self-hosted model can and cannot replace a metered call. Both sides answered the same 1,904 English items of benchmark pass 2, with the same retrieved records in the prompt.
Cost (machine counts and published prices; read 12 Nov 2026). This model is served from its published bf16 weights by vLLM on one GH200 GPU. The benchmark's greedy run was the only run of its engine: 1,904 answers in 25.6 s, after 42 s to start, 74 answers a second. The slowest run of the four was 43 a second (it is the one used for the conservative column). The metered model's cost is its provider's published per-token list prices applied to the tokens it counted on the same items: 3,293 answers (the greedy run and three repeats on the fixed quarter), a mean of 683 prompt tokens and 161 answer tokens, no reasoning tokens (reasoning effort "minimal"), at £0.79 to the dollar.
| per 1,000 answers | this model, 74 a second | this model, conservative (43 a second) | open 4B model fine-tuned (arm 3), 44 a second | commercial model (arm 1) |
|---|---|---|---|---|
| GPU time | 0.0037 GPU h | 0.0065 GPU h | 0.0063 GPU h | — |
| at the award's own valuation of a GPU hour, £2.95 | £0.011 | £0.019 | £0.019 | — |
| at a public cloud's on-demand price for one GH200, $2.59 an hour (£2.05) | £0.008 | £0.013 | £0.013 | — |
| at list price per token | — | — | — | £1.94 |
- At full load, self-hosting this model costs about 1/180 of the metered call per answer (at the award's valuation; about 1/100 on the conservative throughput). The hours used here were awarded compute, so the cash cost to this project was nil. The two GPU-hour prices are what the same work would cost elsewhere.
- Full load is the condition. The figures assume the GPU is busy. A rented GH200 costs the same per hour, idle or not. At the cloud price it breaks even with the metered model at about 1,050 answers an hour (one answer every 3.4 s, all hour). Below that, the metered call is cheaper, unless the model runs on hardware already owned. Smaller hardware than a GH200 was not measured.
- Not counted on either side: the retriever (the same for both), storage, start-up time, and the work of running a server.
The quality trade on the same items (pass 2, English unless marked; greedy answers; this model · the commercial model, each with a 95 % interval from a bootstrap over the questions; a paired difference where the two intervals overlap):
| this model is ahead | read by | |
|---|---|---|
| says the records do not answer, absent items | 0.838 [0.808–0.867] · 0.584 [0.543–0.623]; Romanian clean set 0.879 [0.812–0.941] · 0.646 [0.555–0.745] | machine count |
| cites only the gold record, placed | 0.928 [0.908–0.949] · 0.788 [0.754–0.819] | machine count |
| hedge carried | 0.965 · 0.718; paired difference +0.247 [+0.109, +0.383]. On the sampled repeats it is not ahead (0.900 · 0.933, only 20 items a run) | machine count |
| vet referral where none is due (lower is better) | 0.004 [0.001–0.010] · 0.034 [0.021–0.047] | machine count |
| the commercial model is ahead | read by | |
|---|---|---|
| answerable items judged correct | 0.875 [0.854–0.895] · 0.931 [0.917–0.945] | AI judge |
| judged wrong (lower is better) | 0.032 [0.021–0.043] · 0.005 [0.002–0.009] | AI judge |
| species errors in 1,904 answers (lower is better) | 23 [13–34] · 8 [3–14] | AI judge |
| retrieval end to end | 0.830 · 0.869; paired difference −0.039 [−0.055, −0.023] | machine count |
| no clear difference | read by | |
|---|---|---|
| vet referral carried | 0.968 · 0.841; paired difference +0.127 [0.000, +0.250], two-sided p ≈ 0.06 (63 items); sampled repeats 0.889 · 0.926 (18 items a run) | machine count |
In short: per answer, the self-hosted model is about two orders of magnitude cheaper at full load. It is clearly more
willing to say its records do not answer, in both languages, cites the gold record alone more often, and sends people
to a vet where none is due less often. The metered model is more often judged right on answerable questions, more
rarely wrong, makes fewer species errors and retrieves better end to end. Which matters more depends on the use: this
model drafts for a person to check (see the limits below). Intervals: reports/intervals/ (2,000 resamples of
questions, seed 20261112).
Known limits and risks — read these first
- It was chosen on measurements never confirmed by people. Every "sounds like a person" figure, every reading of the training conversations' quality, every reading of borderline answers and the gate its own generated conversations passed are the judgements of one AI model family (Claude), which may partly be that reader's taste; the judged benchmark pass is another AI model's (Mistral-Small-3.2-24B). No person has read any of this model's answers or conversations.
- Not safe to answer pet-care questions unguarded. The models before the decline work never said the records do not answer (0/28 on pass 1) and answered from neighbouring records instead, rewriting one species' guidance as another's: asked about a cat's natural behaviour, round 4 wrote "The cat evolved as a herd animal … 16–20 hours a day grazing" from the horse record in its context. This model declines on 82–88 % of such items, in English and Romanian, not all (the judge still counted 23 species errors on pass 2's 1,904 answers; mix70 68, the commercial model 8). It also declines on about 5 % of answerable English items. Correctness of advice is measured only by the judge against the gold record; the benchmark's questions are section titles. Run it behind a retrieval guard and a veterinary gate, never as a veterinary adviser.
- Romanian. In the final three-way reading this model is level with mix70 and round 4 on Romanian openings (+2 [−14, +18] against mix70; AI readers). Earlier in the chain, v5m (the maths stage) cost about 10 points of Romanian preference against round 4, mix70 took part of it back (one reading +18 [+3, +33], another +9 [−7, +25]), and the decline blend read −10. Romanian conversation in training is almost entirely model-written (EuroMultiSynth); the people-written Romanian is SFT v1's 52 EUR-Lex examples, oasst2's 11 conversations, and decline-v3's 136 answerable rows from 43 EUR-Lex summaries. Romanian pet-care answering is measured on 107 held-out questions about EU legislation, not on everyday pet-care questions.
- The long-context cost. 179 of 200 hidden numbers at 128k against mix70's 189 (−10 [−17, −4], p 0.006). The decline blend's plain-text loss rose against mix70 (held-out loss at 2k English 2.307 → 2.414, pet care 1.615 → 1.765); this model's is the same to within 0.01 (English 2.419, pet care 1.774).
- Its Romanian held-out set is small and legal in register. 107 questions from 18 EUR-Lex and Your Europe pages; 5
of the original 112 were removed as near a training question (
data/bench/p2ro1/BENCH-P2RO1.json). - Contamination of HumanEval, MBPP and GSM8K was not checked. The pre-training text includes Stack-Edu code and FineMath web maths; these sets have been public since 2021 and copies circulate on the web. Read the scores as an upper bound; the maths examples' near-duplicate check covered their own prompts and outputs only.
- Arm 1 is incomplete on 3 items. The provider's content filter refused 3 of arm 1's greedy prompts on pass 2; the refusals are recorded as its answers, so comparisons on those items are incomplete.
- The string-level scorer is exact on citations and clean stops and weak on declines, cautions and vet referrals (82.5 % agreement with AI readers on pass 2's borderline answers, 89.3 % on pass 1's). The judged pass covers those measures; its decline verdicts use prompt v2 (above), and the second judge's decline counts are not used.
- Small samples, one seed, labels by construction. Pass 1 is 31 questions, the harness's record-bearing items 66, the blind readings 100 items a row; one training run of each stage. "No record answers it" in the decline set means another species, another page or little word overlap, read by nobody.
- No tool use, no safety fine-tune, no preference step (DPO was measured on earlier rounds and not adopted); the tool tokens are untrained. The export is bfloat16, and bf16 serving can part from the float32 weights after tens of tokens; RMSNorm in bf16 differs from Gaia's by at most one rounding.
Licence
Apache-2.0 (the repository's LICENSE, copyright 2026 Marian V. Petrisor). The training data's own terms travel
with the card: ODC-By 1.0 for FineWeb-Edu, FineWeb-2 and FineMath (and Common Crawl's Terms of Use); each code file's
own permissive licence, the attribution index being the SWHID lists code-32k[-x]/<language>/swhids.txt.gz, published
with the base model this one was made from, in its attribution/ folder
(https://huggingface.co/ProjectGaiaUK/gaia-3b-base-128k/tree/main/attribution); "Contains public sector information licensed under the Open Government Licence v3.0."
for the GOV.UK, Defra, nidirect, DAERA and mygov.scot records; "© European Union, https://eur-lex.europa.eu" for
EUR-Lex (SFT v1's Romanian examples and decline-v3's 43 summaries); the conversation sets' licences as listed above; the
maths examples and generated conversations are outputs of MIT- and Apache-2.0-licensed models, which place no condition
on them. docs/DATA_CARD.md is the register; a frozen copy ships with the weights.
Provenance
| source checkpoint | checkpoints/gaia-3b-chat-r4-reason-v5m-mix70-decline3/ckpt.pt on Isambard-AI, sha256 0586f99a… (the full hash is in the export's README.md and SHA256SUMS there) |
| export | checkpoints/gaia-3b-chat-r4-reason-v5m-mix70-decline3-hf/, job 8332006, finished 2026-11-12 01:41 UTC; scripts/export_llama.py, checked by scripts/verify_export.py. verify.json records SHA256SUMS ok (6 files; model.safetensors 5,952,097,048 bytes, tokenizer.json 2,347,671, tokenizer_config.json 4,912, config.json 768, generation_config.json 83, README.md 1,626). The hash values themselves are in SHA256SUMS on Isambard and are not copied into this repository's records. |
| vocabulary | da8375e6082226fc64e8222a08504c88bb75d617a6ed38b00d24d7dd169df9a5 |
| data sets | chat-v5-r4 train.ids.bin 35fe240f5b4e2e38…; reason-v5m 83cb9acf…; decline-v3 6d12ef36… (decline-v2 456294ed… inside it, byte for byte); SFT v1 frozen d6c28c5c… (923 examples) |
| the chain's other hashes | v5m's checkpoint e16fc97b… (its prepared export); round 4, ro2, mix70 and decline-v3: [not measured] in these records |
| jobs | round 4 8205960 · v5m 8260934 · ro2 8299496 · mix70 8303289 · decline-v3 8329594 · this model 8329595 · export 8332006 |
git commits (this repository, branch main) |
the 3B prepared a3da417 (18 Oct 2026) → the 3B's run log e07033e (29 Oct) → round 4 71a39d6 (8 Nov) → v5m d20b0c3 (10 Nov) → mix70 adopted b535e71 → D3 decided f3b7a8e (11 Nov) → decline-v3 built 4f60290 → measured 3613cf6 → the morning summary b0d04be (12 Nov) |
How to go back / versions
driver/chosen-chat-model.json on Isambard names this model and keeps every earlier choice whole under previous
(the file before each change kept beside it as …bak-20261112-decline3, …bak-20261111-mixdec, …-mix70, …), so
going back is replacing the file with that block: decline3 (12 Nov, the owner's "adopt decline3") ← previous: the
decline blend (11 Nov, D3) ← mix70 (11 Nov, "Blend v5m + round 2") ← v5m (10 Nov, "keep for maths") ← round 4 SFT (8 Nov,
the first chosen chat model); every checkpoint in the chain (round 4, v5m, ro1, ro2, mix50, mix70, decline, decline-v2,
the decline blend, dec50/75/85, decline-v3, decline3, decline3-85) is kept on Isambard.
- Downloads last month
- 67
Model tree for ProjectGaiaUK/gaia-3b-chat
Base model
ProjectGaiaUK/gaia-3b-base-128k