Instructions to use openeurollm/nucleus-experiments with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use openeurollm/nucleus-experiments with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("openeurollm/nucleus-experiments", device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from openeurollm/nucleus-experiments: direct link, hf CLI and curl.
- Browser
- Download file 3.14 kB
-
https://huggingface.co/openeurollm/nucleus-experiments/resolve/main/README.md
- Command line
-
hf download hf://openeurollm/nucleus-experiments/README.md
-
curl -L -o README.md https://huggingface.co/openeurollm/nucleus-experiments/resolve/main/README.md
3.14 kB
| library_name: transformers | |
| tags: | |
| - openeurollm | |
| - experimental | |
| - hybrid-architecture | |
| - mixture-of-experts | |
| # OpenEuroLLM nucleus experiments | |
| Intermediate Hugging Face checkpoints from OpenEuroLLM's hybrid-architecture ("nucleus") pre-training | |
| experiments on MareNostrum 5. These are research checkpoints, not a release: architectures, training | |
| schedules and export formats change between runs. | |
| ## Branches | |
| Every checkpoint is its own branch: `<run>_iter_<7-digit step>[_<variant>]`, for example | |
| `ambitious_v4_2ph_hd256_fp8_iter_0024000` or `q35c_gqa_iter_0008000`. `main` holds only this card. | |
| Variants: `_probe` and `_hfNNN` are additional exports of the same training checkpoint, made with a | |
| different version of the export code. | |
| Tokens trained = step x tokens per step (below). The `ambitious_v4_2ph*` and `q35c_gqa` runs train with | |
| `<bos>` at the start of each document, and their tokenizers add it. | |
| ## Runs | |
| | Run (branch prefix) | Architecture | Parameters (total / active per token) | Tokens per step | | |
| |---|---|---|---| | |
| | `ambitious_v4_2ph_fp8` | `oellm_ambitious`: attention, Kimi delta attention and MoE blocks (360 experts, 5 active), engram n-gram tables, mHC residual; head dim 128 | 163.5B / 3.3B | 67.1M | | |
| | `ambitious_v4_2ph_hd256_fp8` | the same run continued from step 9815 with attention head dim 256 | 163.6B / 3.4B | 67.1M | | |
| | `ambitious_v3`, `ambitious_v4_ldm_fp8`, `ambitious_e216_eng40`, `ambitious_ldm_e216`, `ckdab_*` | earlier or branched variants of the same family | varies (91B to 164B) | 67.1M | | |
| | `q35c_gqa`, `q35_conservative_run2` | `oellm_q35`: Qwen3.5-style hybrid of full and linear attention, MoE (256 experts, 8 active) | about 35B / 3.5B | 16.8M | | |
| | `t5_s3_*` | small `oellm_ambitious` models for sparse-attention ablations, 1,500 steps | about 12.9B / 1.2B | 1.05M | | |
| About half of the `oellm_ambitious` parameters are engram n-gram tables (about 80 GB in bf16), which | |
| are looked up, not multiplied, so they add little compute per token. | |
| ### FP8 experts | |
| In the `*_fp8` runs, training gathers the routed-expert weights in FP8 (E4M3, one fp32 scale per | |
| 1 x 128 tile) for every forward pass, while the master weights stay in bf16. The exports apply the same | |
| quantization to the master weights, so the routed experts are stored as FP8 bytes plus fp32 scale | |
| tensors, exactly the values the training forward pass computed with. All other weights are bf16. The | |
| `q35*` runs are bf16 throughout. | |
| ## Loading | |
| Each branch carries its own modeling code: | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| repo, revision = "openeurollm/nucleus-experiments", "q35c_gqa_iter_0008000" | |
| model = AutoModelForCausalLM.from_pretrained(repo, revision=revision, trust_remote_code=True, dtype="auto") | |
| tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision) | |
| ``` | |
| The linear-attention layers need [flash-linear-attention](https://github.com/fla-org/flash-linear-attention). | |
| Branches may include `export_validation/`: the loss of the exported model on 64 held-out documents | |
| (`score.json`) and on a subset where the training framework's own loss is known (`paired.json`). | |