File size: 2,046 Bytes
7fcca91 7775c2d 7fcca91 7775c2d 7fcca91 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | ---
library_name: transformers
language:
- en
tags:
- spark-gpt
- qwen3-moe
- from-scratch
- stories
- checkpoints
---
# Lil Bard historical checkpoints
This repository contains 25 loadable Transformers checkpoints from the
pretraining run for [`N8Programs/lil-bard`](https://huggingface.co/N8Programs/lil-bard).
The final step-25,485 model is kept in that separate repository so ordinary
users do not download the full checkpoint history.
Checkpoints span step 1,000 through step 25,000 in increments of 1,000. Each
subfolder contains a complete `Qwen3MoeForCausalLM` export and tokenizer.
## Loading a checkpoint
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "N8Programs/lil-bard-checkpts"
subfolder = "step-0001000"
tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder=subfolder)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
subfolder=subfolder,
dtype=torch.bfloat16,
device_map="auto",
)
```
Available subfolders are `step-0001000`, `step-0002000`, ..., through
`step-0025000`.
These historical exports retain the fused expert-tensor layout used during the
run. They are tested with Transformers 5.11.0. Transformers 4.57.1 users should
use the cross-version final checkpoint in `N8Programs/lil-bard`.
## Run summary
Lil Bard is a 172,052,992-parameter English story MoE with 58,806,784 active
parameters per token, 8 experts with top-2 routing, and an 8,192-entry
whole-document byte-level BPE tokenizer. It was pretrained from scratch for
2,492,032,616 real loss tokens on two NVIDIA GB10 systems.
The final model, architecture and data details, evaluations, usage example,
and limitations are documented in the
[`N8Programs/lil-bard` model card](https://huggingface.co/N8Programs/lil-bard).
The complete training trace is available in the
[`lil_bard_moe_8x2_full` W&B run](https://wandb.ai/n8programs/sparkgpt/runs/mxk8gln1).
Only model exports are included here. The larger optimizer/scheduler resume
checkpoints are not uploaded.
|