Model card β€” ProjectGaiaUK/gaia-3b-base-128k, the 3B base, Round 1, version 1.0

Released 12 Nov 2026. Every figure is from this repository's records; where a figure is not in them it says [not measured]. Every figure here is a machine count (a loss, a passkey, a scorer) except the one blind reading, which is by AI readers (Claude, an AI model), never people. Nothing here says how well this model answers anyone: it is a base model, never chat-tuned, and answer quality, correctness, safety, hedging and referral were measured only on the chat models made from it.

What this model is

A 3-billion-parameter base language model, pre-trained from scratch on 150 billion tokens of openly licensed English, Romanian, pet-care, code and maths text, then taught to read 8k, 32k and finally 131,072 tokens of context in three continued-training phases. It continues text; it was never fine-tuned to converse, follow instructions or stop. It is the starting point of every Gaia chat model (MODEL_CARD_gaia-3b-chat.md), published so the fine-tunes can be reproduced and other fine-tunes made.

  • Published as: ProjectGaiaUK/gaia-3b-base-128k on Hugging Face β€” Project Gaia Round 1, version 1.0. The chat model made from it: ProjectGaiaUK/gaia-3b-chat.
  • Name in this research: gaia-3b-128k, the 128k phase's final weights, step 2,850.
  • Export: checkpoints/gaia-3b-128k-hf/ on Isambard-AI, Hugging Face Llama format, one model.safetensors in bfloat16 (5,952,097,048 bytes; the folder 5,954,452,564 bytes, "5.95 GB"). Source checkpoint sha256 73e421e1…. Export job 8315443, 2026-11-11 16:31 UTC.
  • Languages: English (about 60 % of tokens), Romanian (8.5 %); also Python, JavaScript, TypeScript and web maths.
  • Licence: Apache-2.0.

Who made it and why

Project Gaia is the research of MyPetSitter Ltd (UK-registered; founder and engineer Marian V. Petrisor, the copyright holder of this repository), under UKRI's AI Research Resource (AIRR) award 0261-1548-0489-1, trained on Isambard-AI.

Acknowledgement. The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR). Isambard-AI is operated by the University of Bristol and is funded by the UK Government’s Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023]. UKRI AI Research Resource (AIRR) Gateway Β· project 0261-1548-0489-1 Β· 4,000 GPU hours of computing time. The award supports the research. It is not an endorsement of any product. Cite: McIntosh-Smith, S., Alam S. R. and Woods, C. (2024). β€œIsambard-AI: a leadership class supercomputer optimised specifically for Artificial Intelligence”. https://doi.org/10.48550/arXiv.2410.11199

How to cite these weights (the same entry as on https://huggingface.co/ProjectGaiaUK; the benchmark's own is its CITATION.cff, https://huggingface.co/datasets/ProjectGaiaUK/gaia-records-bench):

@misc{petrisor2026gaia3b,
  author = {Petrisor, Marian V.},
  title  = {Project Gaia Round 1: gaia-3b-chat and gaia-3b-base-128k},
  year   = {2026},
  note   = {Round 1, version 1.0, released 2026-11-12. Apache-2.0},
  url    = {https://huggingface.co/ProjectGaiaUK}
}

The research asks (docs/AIRR_GATEWAY_APPLICATION.md) whether a vocabulary and data mix trained jointly for English and Romanian gives Romanian a fair share of a model's capacity β€” an English-first tokenizer measured here charged Romanian 3.32Γ— the tokens of English; this project's 32,768-token vocabulary charges 1.08Γ— on held-out text, where English-only vocabularies of the same size charge 1.86Γ— β€” and where a small self-hosted model can replace a metered API for UK pet-care businesses. Round 1 covers English and Romanian only (28 Sept 2026). The promised outputs are open weights under Apache-2.0 with their cards, an open benchmark and the comparison. The 3B is the award's stretch model; a 1B (gaia-1b-round1) was trained first.

Architecture

gaia/model.py and gaia/config.py; configs/gaia-3b-round1.yaml for the shape, configs/gaia-3b-128k.yaml for the final context and RoPE. The export's config.json is LlamaForCausalLM.

parameters 2,976,033,792 (2,774,707,200 without the two embedding matrices)
layers 28
width (d_model) 3,072
attention 24 query heads sharing 6 key-value heads (grouped-query attention), head size 128, no biases
feed-forward SwiGLU, hidden 8,192
normalisation RMSNorm (pre-norm), eps 1e-5, computed in float32
positions RoPE, theta 64,000,000 at 131,072 (500,000 at 2,048; 4,000,000 at 8k; 16,000,000 at 32k), no position scaling
context 131,072 tokens
vocabulary 32,768 byte-level BPE tokens, 16 reserved (ids 0–15); frozen, sha256 da8375e6082226fc64e8222a08504c88bb75d617a6ed38b00d24d7dd169df9a5
embeddings input and output untied
weights as exported bfloat16 (the checkpoint is float32)

A decoder-only transformer: pre-norm RMSNorm β†’ GQA attention with RoPE β†’ residual β†’ RMSNorm β†’ SwiGLU β†’ residual, 28 times; a final RMSNorm and a linear head. Llama's format needed no rearranging (scripts/export_llama.py).

How it was trained

Four phases, each from the weights of the one before with a fresh optimiser (docs/RUNLOG_3B.md, docs/LONG_CONTEXT_8K.md, docs/LONG_CONTEXT_128K.md). Two nodes Γ— 4 NVIDIA GH200 on Isambard-AI, AdamW, bfloat16 autocast, ZeRO-1, torch.compile.

phase context Β· RoPE theta steps Γ— tokens a step tokens learning rate GPU hours (Slurm's record)
main run gaia-3b-round1, 18–29 Oct 2026 2,048 Β· 500,000 152,600 Γ— 983,040 150,011,904,000 (50.4 a parameter) 2,000 warm-up steps to 6.0e-4, held to step 122,080, down to 6.0e-5 over the last 20 % 1,706.83, plus β‰ˆ 185 outside Slurm's record (below)
8k phase gaia-3b-8k, 29 Oct 8,192 Β· 4,000,000 2,850 Γ— 1,048,576 2,988,441,600 100 warm-up to 6.0e-5, cosine to 6.0e-6 48.72
32k phase gaia-3b-32k 32,768 Β· 16,000,000 2,850 Γ— 1,048,576 about 2.99B the same [not measured] in these records (sized at about 101)
128k phase gaia-3b-128k 131,072 Β· 64,000,000 2,850 Γ— 1,048,576 about 2.99B the same [not measured] in these records (sized at about 262)

Main run: weight decay 0.1, gradient clip 1.0, an evaluation and checkpoint every 500 steps; median 183,289 tokens a second (MFU 43.5 %), 95.26 GB peak a GPU; 258.5 h on the calendar over ten 24-hour segments with two node failures; every restart resumed from the last checkpoint, so 960 steps (0.6 % of the run) were trained twice. One of the node failures left the trainer running for about 23 h outside Slurm's record (steps 91,000–107,080), about 185 GPU h that Slurm does not hold. The long phases add activation checkpointing and the loss in 8,192-position pieces; the 128k phase kept its residual stream in bfloat16 because float32 did not fit (speed tests: 36,209 tokens a second a node at 32k; 13,965 at 128k, peak 69.4 GB). RoPE theta grew in proportion to the context at the ratio the 8k phase measured on a 160M model; it was not re-measured on a small model for 32k or 128k. The two phases were queued on 3 Nov 2026 on the project lead's word, with Slurm's first start estimated for 5 Nov (docs/LONG_CONTEXT_128K.md, "Queued"); each phase refuses to start until the one before has written its done event at its final step.

Data (docs/DATA_CARD.md; every file checked by size and sha256 against its publisher). Nothing from the private library, the product or any contributor is in it, and this project generated none of it. The records say nothing about model-written text inside these web corpora.

source share tokens trained available times read licence
English β€” FineWeb-Edu sample/100BT 60.5 % 90.76B 105.38B 0.86 ODC-By 1.0
Romanian β€” FineWeb-2 ron_Latn 8.5 % 12.75B 44.57B 0.29 ODC-By 1.0; Common Crawl's Terms of Use
pet care β€” the pet-care pages of FineWeb-Edu (domain-v2-32k, picked by a pet-care vocabulary) 11.0 % 16.50B 4.19B 3.94 ODC-By 1.0
Python β€” Common Pile v0.1 stackv2_edu_filtered 4.0 % 6.00B 17.46B 0.34 each file's own licence, from an allowlist (MIT, Apache-2.0, BSD-2/3, 0BSD, ISC, Unlicense, CC0-1.0, Zlib, BSL-1.0, MIT-0)
JavaScript β€” the same 3.5 % 5.25B 20.55B 0.26 the same
TypeScript β€” the same 2.5 % 3.75B 4.24B 0.89 the same
maths β€” FineMath 4+ and 3+ 10.0 % 15.00B 39.74B 0.38 ODC-By 1.0; Common Crawl's Terms of Use

The long-context phases added no new source and no new text: each phase read, for half of every source's share, that source's documents of 6,000+ tokens (8k), 24,000–262,144 tokens (32k) or 96,000–262,144 tokens (128k), the other half its ordinary stream; where a source had too few very long documents the shorter tier filled in (the config headers list the fallbacks). Documents over 262,144 tokens (mostly generated code and formal-logic dumps) were left out.

Tokenizer. Byte-level BPE, 32,768 tokens, trained on 2,000 MB of UTF-8 per language taken from the last files of the English and Romanian corpora, so the held-out text (the first files) was never seen; chosen from three candidates by the tokenizer ablation (docs/ABLATION_TOKENIZER.md). Every Round 1 model uses it unchanged.

How to run it

The export is a standard Hugging Face Llama folder (config.json, model.safetensors, tokenizer.json, tokenizer_config.json, generation_config.json, README.md, SHA256SUMS), verified with transformers 5.14.1, tokenizers 0.22.2, safetensors 0.8.0 and vLLM 0.27.1 on a GH200.

  • It is a text-continuation model. Give it plain text and read the continuation. The records' evaluations put the prompt after <|endoftext|> (id 0), a document's start in training; the few-shot prompts were the standard ones (HumanEval's own prompt; MBPP's 3-shot [BEGIN]…[DONE]; GSM8K 8-shot ending "The answer is N.").
  • Stop tokens: generation_config.json stops at <|end|> (id 5) or <|endoftext|> (id 0); the base's own end of document is <|endoftext|>, which tokenizer_config.json names as eos_token. It was never taught to stop a reply: in the blind reading it ran to the 384-token cap on 248 of 350 replies.
  • Do not use the chat template. The folder carries the same Jinja template as the chat export (so the two folders differ only in weights and the eos_token name), but the reserved token rows (<|bos|>, <|system|>, <|user|>, <|assistant|>, <|end|>) were never trained in pre-training. Given a chat prompt it echoes the question: vLLM on the export returned "How often should a long-haired rabbit be groomed?" repeated to the 96-token cap, identical to Gaia's own decode (96/96 tokens). add_bos_token is false.
  • transformers: AutoModelForCausalLM.from_pretrained(dir, torch_dtype=torch.bfloat16), plain generate. vLLM: vllm serve <dir> and the completions endpoint (the chat endpoint would apply the template above).
  • Precision. In bfloat16 the base's plain-text continuation parts from its float32 continuation within a few tokens (vLLM bf16 matched 4 of 48 tokens against Gaia's float32 decode, 48/48 against Gaia running the same bf16 weights): the conversion is exact, the rounding is not nothing. Use float32 where the exact training behaviour matters.
  • Published without a GGUF file: a GGUF build follows once the tool reads this vocabulary. llama.cpp (956591f, 11 Nov 2026) convert_hf_to_gguf.py --vocab-only refuses the vocabulary with NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre(), after falling back from the SentencePiece and Llama-HF vocabularies; llama.cpp was not patched. The safetensors in bfloat16 are the published weights.

Evaluation

All machine counts unless marked. The trainer's own held-out estimates are noisy (neighbouring evaluations move by 0.01–0.03); the scripted evaluations are the measurements.

The trainer's held-out loss per token during the main run (50 batches a source)

step all English Romanian pet care Python JavaScript TypeScript maths
1,000 3.258 3.620 4.296 3.142 1.264 1.137 1.154 2.554
50,000 2.124 2.425 2.650 1.960 0.709 0.625 0.622 1.550
122,000 (end of the held rate) 2.106 2.378 2.595 1.881 0.691 0.613 0.599 1.516
152,600 (final) 1.896 2.225 2.391 1.520 0.637 0.547 0.545 1.380

Most of the gain after step 50,000 came in the decay. The pet-care held-out rows flatter the model: 12.2M of the 42.6M held-out pet-care tokens are pages that also sit in the general English stream, so about 58 % of those tokens are expected to have fallen in a training window; English and Romanian are clean.

Short context kept, long context used β€” each phase's scripted evaluation (scripts/eval_long.py)

main run's final (at 2,048) after 8k after 32k, measured at 131,072 after 128k (this model)
held-out loss at 2,048: English / Romanian / pet care / Python / maths 2.185 / 2.365 / 1.452 / 0.527 / 1.235 2.174 / 2.347 / 1.430 / 0.523 / 1.227 2.175 / 2.347 / 1.436 / 0.525 / 1.232 2.178 / 2.352 / 1.434 / 0.528 / 1.236
passkey, mean of 5 depths, 20 keys a depth: 2k / 4k / 8k / 16k / 32k / 64k / 128k 2k 0.90; 4k 0; 8k 0 1.00 / 0.99 / 0.89 (8k: 1/1/1/1/0.45 by depth) 0.90 / 0.85 / 0.87 / 0.97 / 0.81 / 0.46 / 0.16 0.84 / 0.73 / 0.80 / 0.67 / 0.86 / 0.81 / 0.79
passkey at 128k by depth 0 / 25 / 50 / 75 / 100 % β€” β€” 0 / 0 / 0 / 0 / 0.80 0.90 / 1.00 / 1.00 / 0.85 / 0.20
long-window loss, English, 64–128k of a 128k window / 96–128k β€” β€” 2.008 / 1.845 1.882 / 1.685
long-window loss, pet care, 0–32k / 96–128k β€” β€” 0.459 / 1.350 0.306 / 0.656
loss by position on 8k English documents, 0–2k / 2–4k / 4–6k / 6–8k 2.26 / 5.17 / 6.36 / 6.87 2.24 / 2.15 / 2.15 / 2.19 β€” β€”

No phase cost short-context loss (the 2,048-token losses are within 0.01 of each other after the 8k phase), and each phase uses the context it was trained for: the 32k model finds almost no key beyond 32k, the 128k model finds them at every length. The 128k phase's own evaluation (reports/eval-3b-128k.json, in the repository's reports folder on this Mac but not yet committed) measured the 128k model with its bfloat16 residual stream; re-measured in float32 (job 8282820) every loss moved by less than 0.001 and every passkey mean by at most three keys in a hundred (0.81 / 0.74 / 0.82 / 0.68 / 0.85 / 0.79 / 0.79).

The 128k passkey on fresh keys (40 keys a depth, seed 20261111, job 8290399): 0.95 / 1.00 / 1.00 / 0.975 / 0.275 at depths 0–100 %, mean 0.84. Almost every miss is at 100 % depth, where the key sits right before the question: there the base continues the text (" the pass key. The pass…", " the key that is used to…") instead of answering, so the failure is in answering, not in memory. The chat models made from it score 0.70–0.80 at that depth on the same keys.

Intervals (Wilson 95 %, machine counts; added 12 Nov 2026, no figure changed). The fresh-key total, 168 of 200 = 0.84 [0.78–0.88]; by depth 38/40 [0.84–0.99], 40/40 [0.91–1.00], 40/40 [0.91–1.00], 39/40 [0.87–1.00] and 11/40 [0.16–0.43] at 100 %. The first measurement's 20 keys a depth are wide: 0.20 at 100 % is 4 of 20 [0.08–0.42]. The held-out losses are one deterministic pass over fixed held-out sets and carry no interval here.

Beside the 1B, on identical held-out sets (reports/eval-3b-base.json; the main run's final weights at 2,048, not the 128k model)

gaia-1b-round1 (1.2B, 70B tokens) gaia-3b-round1 (150B tokens)
English held-out text (4.51M tokens, in no training stream): loss Β· perplexity Β· bits per byte 2.3319 Β· 10.30 Β· 0.7583 2.1779 Β· 8.83 Β· 0.7082 (βˆ’6.6 % bpb)
Romanian held-out text (4.79M tokens, in no training stream) 2.5450 Β· 12.74 Β· 0.8783 2.3452 Β· 10.44 Β· 0.8094 (βˆ’7.8 %)
pet-care held-out text (8.0M tokens; partly in the general English stream, so not clean) 1.8020 Β· 6.06 Β· 0.6005 1.4286 Β· 4.17 Β· 0.4760 (βˆ’20.7 %)
329 written pet-care answers, bits per answer byte (lower = less surprising) 1.2432 1.1412
reasoning check: 700 three-shot arithmetic and pet-care word problems, four options (chance 25 %) 28.7 % 71.6 %
retrieval dry run (PMI), 231 questions: right record at rank 1 / top 5 / top 10 Β· MRR 62.3 / 87.4 / 92.2 % Β· 0.739 60.2 / 83.5 / 88.3 % Β· 0.709 (not better)
the same by word overlap, no model 69.3 / 91.8 / 94.4 % Β· 0.797 the same

The two runs differ in size and in data (the 3B read code and maths, the 1B none), so this pair cannot separate the two. Intervals (Wilson 95 %, machine counts; added 12 Nov 2026): the reasoning check 501 of 700 = 71.6 % [68.1–74.8] against the 1B's 28.7 % [25.5–32.2]; retrieval at rank 1, 139 of 231 = 60.2 % [53.7–66.3] against the 1B's 62.3 % [55.9–68.3], so "not better" is the honest reading: the two overlap.

Code and maths, plain prompts, greedy (scripts/eval_3b_code.py; Wilson 95 %; the 128k model)

set score 95 % n stops
HumanEval 11.0 % (18) 7.1–16.7 164 131 at a stop text, 33 at 384 tokens
MBPP, test 11–510 13.8 % (69) 11.1–17.1 500 463 at [DONE], 37 at 384
GSM8K, 8-shot 3.6 % (47) 2.7–4.7 1,319 1,061 at the next "Question:", 258 at the cap; 7 gave no number

Model-written code ran only in a no-network sandbox; the reference solutions pass 164/164 and 498/500 there. The low GSM8K figure was read through before it was believed: the model writes "The answer is N." with the wrong N.

The one blind reading β€” AI readers (Claude, an AI model), never people

In the first three-way reading (8 Nov 2026; 4 fresh readers, a seeded 80 of 250 held-out items; the base with the chat tokens started as the fine-tune starts them), the base was picked as sounding most like a person on 0 % of 60 openings and 0 % of 20 multi-turn items, with multi-turn coherence 1.05 of 5 (round 1 chat 2.90; an open 4B model as released 3.60). It is "the before": it does not converse.

The export, proved (reports/export/gaia-3b-128k/verify.json)

SHA256SUMS ok (6 files). Chat template against ChatTemplate 90/90 prompts, 90/90 examples, 14/14 edge cases, the same four refusals. Largest logit difference, transformers float32 on the export against Gaia float32 on 8 prompts: 1.74 (logits up to 36); against Gaia with its weights rounded to bfloat16: 1.9e-4; greedy next token 8/8 the same (worst prompt, every position 0.982). vLLM: prompt ids are Gaia's 3/3; a chat prompt echoed 96/96 identically; Romanian 0/96 and plain text 4/48 against the float32 decode (precision), 10/96 and 48/48 against the same bf16 weights. GGUF refused. TF32 off throughout.

Export verification: the published numbers from the exported weights (reports/export/gaia-3b-128k/verify-loss.json; job 8324217, 11 Nov 2026; scripts/verify_export_loss.py, run by slurm/95_export_verify_loss.sbatch). transformers 5.14.1 loaded the export in bfloat16 with sdpa attention and no Gaia code. It ran on the same windows and keys as the float32 figures above (job 8282820, reports/eval-chosen/long-base128k-fp32.json). SHA256SUMS was checked first: ok, 6 files.

  • Held-out loss at 2,048 tokens, export bf16 against published fp32 (machine count; 1,999,999 tokens scored a source): English 2.1784 / 2.1783 Β· Romanian 2.3520 / 2.3519 Β· pet care 1.4340 / 1.4337 Β· Python 0.5278 / 0.5278 Β· maths 1.2364 / 1.2363. The largest difference is 0.0003 (machine count).

  • Passkey, 20 keys a cell, the published run's keys (seed 20261018), depths 0 / 25 / 50 / 75 / 100 % (machine count):

    • at 2,048 tokens: 1.00 / 1.00 / 0.70 / 0.75 / 0.70 against 1.00 / 1.00 / 0.70 / 0.70 / 0.65 (mean 0.83 against 0.81);
    • at 131,072 tokens: 0.90 / 1.00 / 1.00 / 0.85 / 0.25 against 0.90 / 1.00 / 1.00 / 0.85 / 0.20 (mean 0.80 against 0.79).

    No cell differs by more than one key in twenty, and each difference is one more key found. The misses at 128k and 100 % depth are the same kind as before (" the pass key. The pass…", " the key that is used to…").

  • Memory. A 131,072-token prompt fitted on one GH200 in bf16, with a peak of 27.0 GB (machine count). Each prompt is one prefill; the key/value cache is reused for the at most 6 greedy tokens.

Known limits and risks β€” read these first

  1. Never chat-tuned: it continues text. It does not answer, follow instructions or stop; given a chat prompt it echoes the question. It carries no hedging, no referral, no refusal and no safety behaviour. Nothing here measures answer quality, correctness or safety; that was measured only on the chat models made from it, and those are not safe to answer pet-care questions unguarded either (see their card).
  2. The passkey is weak at 100 % depth: 0.275 on fresh keys at 128k (0.20 in the first measurement), 0.15 at 64k, and already 9 of 20 at the end of an 8k context after the 8k phase. The misses are the model continuing the text rather than answering. The passkey is synthetic retrieval, not reading comprehension.
  3. Contamination of HumanEval, MBPP and GSM8K was not checked. The pre-training text includes Stack-Edu code and FineMath web maths; these sets have been public since 2021 and copies circulate on the web. Read the scores as an upper bound. A full n-gram scan of about 100B tokens is not planned before publication.
  4. The pet-care held-out figures are flattered (above): the clean part of that file (its sample/350BT pages) has not been scored. Near-duplicates across the pet-care split, read 3.94 times, were not checked.
  5. One run, one seed, for the main run and each phase. The RoPE theta for 32k and 128k follows a rule and was not measured on a small model; the 128k phase's long windows are partly packed documents, and only code and maths have held-out documents a whole 128k window long (English and pet care: none).
  6. The 128k phase trained with a bfloat16 residual stream (float32 did not fit); the chat fine-tunes inherit that choice. Re-measuring in float32 moved no read figure by more than 0.003.
  7. Web text as its publishers released it. FineWeb-Edu, FineWeb-2 and FineMath are filtered Common Crawl; the records say nothing about model-written text, bias or personal data inside them beyond their publishers' own declarations. Common Pile's code is a 2025 snapshot of The Stack v2's selection, so later author opt-outs are not reflected.
  8. Romanian is 8.5 % of the tokens and 0.29 of its available stream; the pet-care pages were read 3.94 times.
  9. Precision: the export is bfloat16; bf16 serving changes the plain-text continuation within a few tokens.
  10. GPU hours of the 32k and 128k phases are not in this repository's records (the hours log lives in the project schedule outside it); about 185 GPU h of the main run are outside Slurm's record (a node failure that did not stop the trainer).

Licence

Apache-2.0 (the repository's LICENSE, copyright 2026 Marian V. Petrisor). Attribution the training data requires, carried by this card: ODC-By 1.0 for FineWeb-Edu (HuggingFaceFW/fineweb-edu), FineWeb-2 (HuggingFaceFW/fineweb-2) and FineMath (HuggingFaceTB/finemath), with Common Crawl's Terms of Use where their cards say so; for the code, each file's own MIT / Apache-2.0 / BSD (and the rest of the allowlist) attribution, the index being the SWHID lists code-32k/<language>/swhids.txt.gz and code-32k-x/…, published with these weights in their attribution/ folder (https://huggingface.co/ProjectGaiaUK/gaia-3b-base-128k/tree/main/attribution; Software Heritage's principles for training on its archive: an open licence on the weights, every file identified, the opt-outs as The Stack v2 applied them). docs/DATA_CARD.md is the register; a frozen copy ships with the weights.

Provenance

source checkpoint checkpoints/gaia-3b-128k/ckpt.pt on Isambard-AI, step 2,850, sha256 73e421e1… (the full hash is in the export's README.md and SHA256SUMS there)
earlier phases' checkpoints checkpoints/gaia-3b-32k/ckpt.pt (2,850) ← gaia-3b-8k/ckpt.pt (2,850) ← gaia-3b-round1/ckpt.pt (152,600; 11.9 GB of float32 weights) and its pre-decay ckpt-step122000.pt
export checkpoints/gaia-3b-128k-hf/, job 8315443, 2026-11-11 16:31 UTC; scripts/export_llama.py, checked by scripts/verify_export.py
export checksums verify.json records SHA256SUMS ok (6 files; model.safetensors 5,952,097,048 bytes, tokenizer.json 2,347,671, tokenizer_config.json 4,918, config.json 768, generation_config.json 83, README.md 1,580). The hash values themselves are in SHA256SUMS on Isambard and are not copied into this repository's records.
vocabulary data/shards/general-32k/tokenizer.json on Isambard-AI (and inside each export), da8375e6082226fc64e8222a08504c88bb75d617a6ed38b00d24d7dd169df9a5
jobs main run 7523556–58, 7624580–84 (two runs each of 7624580 and 7624582); 8k phase 7624589, its evaluation 7624591; 32k and 128k phases 8044755–57 and 8044764–67 as queued; fresh-key passkey 8290399; base beside the 1B 7829754; export 8315443; export verification (loss and passkey from the bf16 export) 8324217
git commits (this repository, main) the 3B's data on the card 3437403 and the 3B proven ready a3da417 (18 Oct 2026) β†’ the 3B's run log e07033e (29 Oct) β†’ the 32k and 128k phases prepared 6d534a4 (29 Oct) and sized 2cabdb5 (3 Nov) β†’ F9 export 7560406 (11 Nov) β†’ f26fa89 (HEAD when this card was written, 11 Nov 2026)

How to go back / versions

The base has no previous chain of its own β€” it is the only version of the 3B base; going back is a checkpoint, not a file: gaia-3b-128k (step 2,850) ← gaia-3b-32k (2,850) ← gaia-3b-8k (2,850) ← gaia-3b-round1 (152,600, with its pre-decay step 122,000 also kept), all on Isambard.

Downloads last month
82
Safetensors
Model size
3B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for ProjectGaiaUK/gaia-3b-base-128k

Finetunes
1 model