BubblSpace MoE-0.9B-A0.2B (base, mid-training checkpoint)
Fully open source β model weights, every checkpoint, the full training pipeline, and the FineWeb-Edu data-prep code are all public, start to finish. Nothing here is held back: github.com/thefirehacker/bubblspace-MoE-0.9B-A0.2B
Live training graphs/logs (public W&B run): forge.coreweave.com/wandb/firehacker/moe0-9b-a0-2b/runs/goa8jk6e
Disclaimer
This model, its weights/checkpoints, and the training/data-prep code are published for learning and research purposes only. It is an experimental, independently-trained small language model, not a finished or production-grade product.
AIEDX Private Limited bears no responsibility or liability for any output produced by this model or any software built using it. Use is entirely at your own risk, provided "AS IS" without warranty of any kind, consistent with the Apache License 2.0.
On the nature of AI output: this is a small, base-pretrained (non-instruction-tuned) model. Its outputs are statistically plausible continuations of text, not verified facts, advice, or reasoning β it can be confidently wrong, biased, inconsistent, or nonsensical, especially at this parameter scale and training stage. Do not rely on its output for factual, medical, legal, financial, or safety-critical purposes.
This is an intermediate checkpoint from an in-progress pretraining run, not a finished model.
- Architecture: GQA (16 heads / 4 kv-heads), QK-norm, RoPE, dropless top-8 SwiGLU MoE (64 experts), ZeroCenteredRMSNorm, tied embeddings.
- Total params: 915,573,760 (~915M total / ~211M active per token)
- Checkpoint step: 10,000 / 19,073 total planned steps (10B token run, FineWeb-Edu only)
- Trained with Muon + AdamW, bf16, on a single H100.
- Tokenizer: SmolLM2-360M base + BubblSpace special tokens (chat/FIM/tool/reasoning), vocab 49,280.
This checkpoint is base-pretrained only β no instruction tuning, no code/agentic data yet. Expect plausible-sounding but not reliably factual completions, and no question-answering behavior (it continues text, it does not follow instructions).
Weights are provided as model.safetensors (raw state_dict of the custom Transformer class β
this is NOT a transformers-library-compatible checkpoint; you need the model code from
https://github.com/TheFireHacker/bubblspace-MoE0.9B-A2b to load it).
Training config (base.yaml)
# ============================================================
# MoE-0.9B-A0.2B v2 β ~915M total / ~211M active params
# Qwen3-Next-inspired, single H100 80GB, Muon + AdamW
# ============================================================
run_name: moe0-9b-a0-2b-base
model:
n_layers: 16
d_model: 1024
vocab_size: 49280 # bubblspace-tokenizer real length 49164, padded to a multiple of 128
tie_embeddings: true
max_seq_len: 2048
norm_eps: 1.0e-6
attention:
n_heads: 16
head_dim: 64
n_kv_heads: 4
qk_norm: true
rope_theta: 10000
output_gate: true
bias: false
moe:
n_experts: 64
top_k: 8
expert_hidden: 256
activation: silu
router_init_std: 0.02
backend: auto # auto | grouped_mm | fallback β resolved at startup by probing torch._grouped_mm
load_balance_coef: 0.01
router_z_loss_coef: 0.001
init:
std: 0.02
# residual_proj_std = std / sqrt(2 * n_layers), applied to attn.o_proj and experts.W_down
optimizer:
muon:
lr: 6.0e-4
momentum: 0.95
nesterov: true
ns_steps: 5
weight_decay: 0.1
adamw:
lr: 6.0e-4
betas: [0.9, 0.95]
eps: 1.0e-8
weight_decay: 0.1
grad_clip: 1.0
schedule:
warmup_steps: 2000
decay: cosine # cosine | wsd
min_lr_ratio: 0.1
wsd_decay_frac: 0.2 # only used when decay: wsd
training:
dataset: HuggingFaceFW/fineweb-edu
dataset_config: sample-10BT
total_tokens: 10_000_000_000
seq_len: 2048
global_batch_tokens: 524288
micro_batch: 16
precision: bf16
compile: true
checkpoint_every: 1000
log_every: 10
expert_hist_every: 100
attn_logit_every: 100
data:
data_dir: /lambda/nfs/h100-f1/data
checkpoint_dir: /lambda/nfs/h100-f1/checkpoints
tokenizer:
path: ./bubblspace-tokenizer
doc_separator: "<|endoftext|>"
wandb:
project: moe0-9b-a0-2b
entity: null
Benchmarks
Standard base-LM benchmark suite via lm-evaluation-harness (loglikelihood-scored, multiple-choice tasks only β appropriate for a base-pretrained, non-instruction-tuned model; chat/instruction-following benchmarks would not be meaningful here).
step_7000 (step 7,000 / 19,073 planned steps, ~37% through the single
10B-token FineWeb-Edu pretraining run)
| Task | Metric | Score | Random baseline |
|---|---|---|---|
| PIQA | acc_norm | 64.7% | 50% (2-choice) |
| ARC-Easy | acc_norm | 51.1% | ~25% (multi-choice) |
| HellaSwag | acc_norm | 37.0% | 25% (4-choice) |
| LAMBADA (openai) | acc (exact last-word match) | 29.0% | ~0% (strict exact-match) |
| LAMBADA (openai) | perplexity | 58.1 | β |
| WinoGrande | acc | 52.4% | 50% (2-choice) |
| ARC-Challenge | acc_norm | 28.0% | 25% (4-choice) |
Tasks rewarding broad surface-pattern learning (PIQA, ARC-Easy, HellaSwag) are clearly above chance already. The two hardest, most reasoning-heavy benchmarks (ARC-Challenge, WinoGrande) sitting near chance is expected at this model size and training stage β not a sign of a problem.
step_8000 (step 8,000 / 19,073 planned steps, ~42% through the run)
| Task | Metric | step_7000 | step_8000 | Ξ |
|---|---|---|---|---|
| PIQA | acc_norm | 64.7% | 64.5% | -0.2 (noise) |
| ARC-Easy | acc_norm | 51.1% | 50.7% | -0.4 (noise) |
| HellaSwag | acc_norm | 37.0% | 38.0% | +1.0 |
| LAMBADA (openai) | acc | 29.0% | 31.0% | +2.0 |
| LAMBADA (openai) | perplexity | 58.1 | 50.3 | -7.8 (better) |
| WinoGrande | acc | 52.4% | 51.7% | -0.7 (noise) |
| ARC-Challenge | acc_norm | 28.0% | 27.6% | -0.4 (noise) |
Most metrics are flat within statistical noise over just 1,000 steps. LAMBADA is the exception β a real, consistent improvement (perplexity down 13%, accuracy up 2 points), the expected signal for the metric most dependent on total tokens seen. More checkpoints will be benchmarked the same way as training progresses.
step_9000 (step 9,000 / 19,073 planned steps, ~47% through the run)
| Task | Metric | step_8000 | step_9000 | Ξ |
|---|---|---|---|---|
| PIQA | acc_norm | 64.5% | 65.9% | +1.4 |
| ARC-Easy | acc_norm | 50.7% | 50.8% | +0.1 (noise) |
| HellaSwag | acc_norm | 38.0% | 38.5% | +0.5 |
| LAMBADA (openai) | acc | 31.0% | 30.6% | -0.4 (noise) |
| LAMBADA (openai) | perplexity | 50.3 | 49.3 | -1.0 (better) |
| WinoGrande | acc | 51.7% | 51.5% | -0.2 (noise) |
| ARC-Challenge | acc_norm | 27.6% | 27.2% | -0.4 (noise) |
PIQA shows a real move this step (+1.4, clear of its ~1.1% stderr), and HellaSwag keeps climbing steadily. LAMBADA perplexity continues its monotonic improvement (58.1 β 50.3 β 49.3), now 15% better than the step_7000 baseline. The reasoning-heavy benchmarks (ARC-Challenge, WinoGrande) remain flat within noise, as expected at this model size and training stage.
Try it yourself
This repo is self-contained β model code, weights, tokenizer, and a generation script are all included, so you don't need the (private) training repo to run it.
Or run it directly in Colab, no setup required: https://github.com/thefirehacker/FirstBreakAI-Code/blob/main/notebooks/bubblspace_moe0_9b_a0_2b.ipynb
git lfs install
git clone https://huggingface.co/TheFireHacker/bubblspace-moe0.9b-a0.2b
cd bubblspace-moe0.9b-a0.2b
pip install -r requirements.txt
python generate.py --prompt "Your prompt here" --max_new_tokens 100
Needs a CUDA GPU for reasonable speed (falls back to CPU automatically, but will
be slow). moe.backend auto-resolves to torch._grouped_mm if available on
your GPU/torch build, otherwise a portable gather-loop fallback β both produce
identical outputs.