nucleus-experiments / README.md
maxidl's picture
Model card
00222bc verified
|
Raw History Blame Contribute Delete
3.14 kB
---
library_name: transformers
tags:
- openeurollm
- experimental
- hybrid-architecture
- mixture-of-experts
---
# OpenEuroLLM nucleus experiments
Intermediate Hugging Face checkpoints from OpenEuroLLM's hybrid-architecture ("nucleus") pre-training
experiments on MareNostrum 5. These are research checkpoints, not a release: architectures, training
schedules and export formats change between runs.
## Branches
Every checkpoint is its own branch: `<run>_iter_<7-digit step>[_<variant>]`, for example
`ambitious_v4_2ph_hd256_fp8_iter_0024000` or `q35c_gqa_iter_0008000`. `main` holds only this card.
Variants: `_probe` and `_hfNNN` are additional exports of the same training checkpoint, made with a
different version of the export code.
Tokens trained = step x tokens per step (below). The `ambitious_v4_2ph*` and `q35c_gqa` runs train with
`<bos>` at the start of each document, and their tokenizers add it.
## Runs
| Run (branch prefix) | Architecture | Parameters (total / active per token) | Tokens per step |
|---|---|---|---|
| `ambitious_v4_2ph_fp8` | `oellm_ambitious`: attention, Kimi delta attention and MoE blocks (360 experts, 5 active), engram n-gram tables, mHC residual; head dim 128 | 163.5B / 3.3B | 67.1M |
| `ambitious_v4_2ph_hd256_fp8` | the same run continued from step 9815 with attention head dim 256 | 163.6B / 3.4B | 67.1M |
| `ambitious_v3`, `ambitious_v4_ldm_fp8`, `ambitious_e216_eng40`, `ambitious_ldm_e216`, `ckdab_*` | earlier or branched variants of the same family | varies (91B to 164B) | 67.1M |
| `q35c_gqa`, `q35_conservative_run2` | `oellm_q35`: Qwen3.5-style hybrid of full and linear attention, MoE (256 experts, 8 active) | about 35B / 3.5B | 16.8M |
| `t5_s3_*` | small `oellm_ambitious` models for sparse-attention ablations, 1,500 steps | about 12.9B / 1.2B | 1.05M |
About half of the `oellm_ambitious` parameters are engram n-gram tables (about 80 GB in bf16), which
are looked up, not multiplied, so they add little compute per token.
### FP8 experts
In the `*_fp8` runs, training gathers the routed-expert weights in FP8 (E4M3, one fp32 scale per
1 x 128 tile) for every forward pass, while the master weights stay in bf16. The exports apply the same
quantization to the master weights, so the routed experts are stored as FP8 bytes plus fp32 scale
tensors, exactly the values the training forward pass computed with. All other weights are bf16. The
`q35*` runs are bf16 throughout.
## Loading
Each branch carries its own modeling code:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
repo, revision = "openeurollm/nucleus-experiments", "q35c_gqa_iter_0008000"
model = AutoModelForCausalLM.from_pretrained(repo, revision=revision, trust_remote_code=True, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)
```
The linear-attention layers need [flash-linear-attention](https://github.com/fla-org/flash-linear-attention).
Branches may include `export_validation/`: the loss of the exported model on 64 held-out documents
(`score.json`) and on a subset where the training framework's own loss is known (`paired.json`).