File size: 4,053 Bytes
ae597ef 7c0f24d ae597ef | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 | ---
library_name: transformers
pipeline_tag: text-generation
language:
- en
tags:
- spark-gpt
- qwen3-moe
- from-scratch
- stories
---
# Lil Bard 172M MoE
Lil Bard is a small English story language model pretrained from scratch. It is
a base model, not an instruction-tuned or chat model.
The model has 172,052,992 total parameters and 58,806,784 active parameters per
token. It uses 16 transformer layers, width 512, 8 feed-forward experts with
top-2 routing, and a maximum exported context length of 32,768 tokens.
## Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "N8Programs/lil-bard"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
device_map="auto",
)
inputs = tokenizer("Once upon a time", return_tensors="pt").to(model.device)
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=200,
do_sample=True,
temperature=0.8,
top_p=0.95,
pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```
The tokenizer automatically prepends BOS. Its special-token IDs are EOS 0,
BOS 8190, and PAD 8191.
## Architecture
| Property | Value |
|---|---:|
| Total parameters | 172,052,992 |
| Active parameters/token | 58,806,784 |
| Layers | 16 |
| Hidden size | 512 |
| Attention heads / KV heads | 4 / 2 |
| Head dimension | 128 |
| Experts / selected experts | 8 / 2 |
| Dense MLP size | 1,536 |
| Expert MLP size | 768 |
| Vocabulary | 8,192 |
| Maximum exported context | 32,768 |
| Published weight dtype | BF16 |
The checkpoint uses the stock Transformers `Qwen3MoeForCausalLM` layout. MoE
expert weights are stored as per-expert `gate_proj`, `up_proj`, and `down_proj`
tensors for compatibility across Transformers releases; loading has been tested
with Transformers 4.57.1 and 5.11.0.
## Tokenizer
The 8,192-entry tokenizer is a byte-level BPE tokenizer trained on a balanced
1.5-million-document sample of the corpus. It does not use regex, whitespace,
or word pretokenization. BOS, EOS, PAD, and UNK are distinct tokens.
## Training data
The corpus contained 8,732,634 documents drawn from:
- [`klusai/ds-tf1-en-3m`](https://huggingface.co/datasets/klusai/ds-tf1-en-3m)
- [`karpathy/tinystories-gpt4-clean`](https://huggingface.co/datasets/karpathy/tinystories-gpt4-clean)
- A deterministic 3-million-row sample of
[`littlelearner/LittleCurriculum`](https://huggingface.co/datasets/littlelearner/LittleCurriculum)
A canonical validation set excluded 1,000 DS-TF1 test rows and 1,000
TinyStories test rows from training.
The model trained for exactly 2,492,032,616 real loss tokens over 25,485
distributed steps on two NVIDIA GB10 systems. Whole-document packing achieved
99.4713% utilization. Training used a local adaptation of
[`N8python/spark-gpt`](https://github.com/N8python/spark-gpt).
The complete training trace is available in the
[`lil_bard_moe_8x2_full` W&B run](https://wandb.ai/n8programs/sparkgpt/runs/mxk8gln1).
## Evaluation
| Evaluation | Result |
|---|---:|
| Canonical validation loss | 1.42228 nats/token |
| ARC-Easy zero-shot accuracy | 32.15% |
| ARC-Easy zero-shot normalized accuracy | 32.79% |
ARC-Easy was evaluated on all 2,376 test questions with lm-eval 0.4.12 in
BF16, using the base-model prompt format and an explicit BOS token.
## Historical checkpoints
To keep ordinary downloads of this repository small, the 25 periodic
checkpoints are published separately in
[`N8Programs/lil-bard-checkpts`](https://huggingface.co/N8Programs/lil-bard-checkpts).
They span step 1,000 through step 25,000 in increments of 1,000.
## Limitations
This model was trained primarily on simple synthetic stories. It has limited
world knowledge and reasoning ability, may produce repetitive or incoherent
text, and has not been safety-tuned. Do not use it for factual, medical, legal,
financial, or other high-stakes decisions.
|