--- library_name: transformers tags: - openeurollm - experimental - hybrid-architecture - mixture-of-experts --- # OpenEuroLLM nucleus experiments Intermediate Hugging Face checkpoints from OpenEuroLLM's hybrid-architecture ("nucleus") pre-training experiments on MareNostrum 5. These are research checkpoints, not a release: architectures, training schedules and export formats change between runs. ## Branches Every checkpoint is its own branch: `_iter_<7-digit step>[_]`, for example `ambitious_v4_2ph_hd256_fp8_iter_0024000` or `q35c_gqa_iter_0008000`. `main` holds only this card. Variants: `_probe` and `_hfNNN` are additional exports of the same training checkpoint, made with a different version of the export code. Tokens trained = step x tokens per step (below). The `ambitious_v4_2ph*` and `q35c_gqa` runs train with `` at the start of each document, and their tokenizers add it. ## Runs | Run (branch prefix) | Architecture | Parameters (total / active per token) | Tokens per step | |---|---|---|---| | `ambitious_v4_2ph_fp8` | `oellm_ambitious`: attention, Kimi delta attention and MoE blocks (360 experts, 5 active), engram n-gram tables, mHC residual; head dim 128 | 163.5B / 3.3B | 67.1M | | `ambitious_v4_2ph_hd256_fp8` | the same run continued from step 9815 with attention head dim 256 | 163.6B / 3.4B | 67.1M | | `ambitious_v3`, `ambitious_v4_ldm_fp8`, `ambitious_e216_eng40`, `ambitious_ldm_e216`, `ckdab_*` | earlier or branched variants of the same family | varies (91B to 164B) | 67.1M | | `q35c_gqa`, `q35_conservative_run2` | `oellm_q35`: Qwen3.5-style hybrid of full and linear attention, MoE (256 experts, 8 active) | about 35B / 3.5B | 16.8M | | `t5_s3_*` | small `oellm_ambitious` models for sparse-attention ablations, 1,500 steps | about 12.9B / 1.2B | 1.05M | About half of the `oellm_ambitious` parameters are engram n-gram tables (about 80 GB in bf16), which are looked up, not multiplied, so they add little compute per token. ### FP8 experts In the `*_fp8` runs, training gathers the routed-expert weights in FP8 (E4M3, one fp32 scale per 1 x 128 tile) for every forward pass, while the master weights stay in bf16. The exports apply the same quantization to the master weights, so the routed experts are stored as FP8 bytes plus fp32 scale tensors, exactly the values the training forward pass computed with. All other weights are bf16. The `q35*` runs are bf16 throughout. ## Loading Each branch carries its own modeling code: ```python from transformers import AutoModelForCausalLM, AutoTokenizer repo, revision = "openeurollm/nucleus-experiments", "q35c_gqa_iter_0008000" model = AutoModelForCausalLM.from_pretrained(repo, revision=revision, trust_remote_code=True, dtype="auto") tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision) ``` The linear-attention layers need [flash-linear-attention](https://github.com/fla-org/flash-linear-attention). Branches may include `export_validation/`: the loss of the exported model on 64 held-out documents (`score.json`) and on a subset where the training framework's own loss is known (`paired.json`).