| --- |
| library_name: transformers |
| language: |
| - en |
| tags: |
| - spark-gpt |
| - qwen3-moe |
| - from-scratch |
| - stories |
| - checkpoints |
| --- |
| |
| # Lil Bard historical checkpoints |
|
|
| This repository contains 25 loadable Transformers checkpoints from the |
| pretraining run for [`N8Programs/lil-bard`](https://huggingface.co/N8Programs/lil-bard). |
| The final step-25,485 model is kept in that separate repository so ordinary |
| users do not download the full checkpoint history. |
|
|
| Checkpoints span step 1,000 through step 25,000 in increments of 1,000. Each |
| subfolder contains a complete `Qwen3MoeForCausalLM` export and tokenizer. |
|
|
| ## Loading a checkpoint |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| repo_id = "N8Programs/lil-bard-checkpts" |
| subfolder = "step-0001000" |
| |
| tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder=subfolder) |
| model = AutoModelForCausalLM.from_pretrained( |
| repo_id, |
| subfolder=subfolder, |
| dtype=torch.bfloat16, |
| device_map="auto", |
| ) |
| ``` |
|
|
| Available subfolders are `step-0001000`, `step-0002000`, ..., through |
| `step-0025000`. |
|
|
| These historical exports retain the fused expert-tensor layout used during the |
| run. They are tested with Transformers 5.11.0. Transformers 4.57.1 users should |
| use the cross-version final checkpoint in `N8Programs/lil-bard`. |
|
|
| ## Run summary |
|
|
| Lil Bard is a 172,052,992-parameter English story MoE with 58,806,784 active |
| parameters per token, 8 experts with top-2 routing, and an 8,192-entry |
| whole-document byte-level BPE tokenizer. It was pretrained from scratch for |
| 2,492,032,616 real loss tokens on two NVIDIA GB10 systems. |
|
|
| The final model, architecture and data details, evaluations, usage example, |
| and limitations are documented in the |
| [`N8Programs/lil-bard` model card](https://huggingface.co/N8Programs/lil-bard). |
| The complete training trace is available in the |
| [`lil_bard_moe_8x2_full` W&B run](https://wandb.ai/n8programs/sparkgpt/runs/mxk8gln1). |
|
|
| Only model exports are included here. The larger optimizer/scheduler resume |
| checkpoints are not uploaded. |
|
|
