BubblSpace MoE-0.9B-A0.2B (base, mid-training checkpoint)

Fully open source β€” model weights, every checkpoint, the full training pipeline, and the FineWeb-Edu data-prep code are all public, start to finish. Nothing here is held back: github.com/thefirehacker/bubblspace-MoE-0.9B-A0.2B

Live training graphs/logs (public W&B run): forge.coreweave.com/wandb/firehacker/moe0-9b-a0-2b/runs/goa8jk6e

Disclaimer

This model, its weights/checkpoints, and the training/data-prep code are published for learning and research purposes only. It is an experimental, independently-trained small language model, not a finished or production-grade product.

AIEDX Private Limited bears no responsibility or liability for any output produced by this model or any software built using it. Use is entirely at your own risk, provided "AS IS" without warranty of any kind, consistent with the Apache License 2.0.

On the nature of AI output: this is a small, base-pretrained (non-instruction-tuned) model. Its outputs are statistically plausible continuations of text, not verified facts, advice, or reasoning β€” it can be confidently wrong, biased, inconsistent, or nonsensical, especially at this parameter scale and training stage. Do not rely on its output for factual, medical, legal, financial, or safety-critical purposes.

This is an intermediate checkpoint from an in-progress pretraining run, not a finished model.

  • Architecture: GQA (16 heads / 4 kv-heads), QK-norm, RoPE, dropless top-8 SwiGLU MoE (64 experts), ZeroCenteredRMSNorm, tied embeddings.
  • Total params: 915,573,760 (~915M total / ~211M active per token)
  • Checkpoint step: 10,000 / 19,073 total planned steps (10B token run, FineWeb-Edu only)
  • Trained with Muon + AdamW, bf16, on a single H100.
  • Tokenizer: SmolLM2-360M base + BubblSpace special tokens (chat/FIM/tool/reasoning), vocab 49,280.

This checkpoint is base-pretrained only β€” no instruction tuning, no code/agentic data yet. Expect plausible-sounding but not reliably factual completions, and no question-answering behavior (it continues text, it does not follow instructions).

Weights are provided as model.safetensors (raw state_dict of the custom Transformer class β€” this is NOT a transformers-library-compatible checkpoint; you need the model code from https://github.com/TheFireHacker/bubblspace-MoE0.9B-A2b to load it).

Training config (base.yaml)

# ============================================================
# MoE-0.9B-A0.2B v2  β€”  ~915M total / ~211M active params
# Qwen3-Next-inspired, single H100 80GB, Muon + AdamW
# ============================================================

run_name: moe0-9b-a0-2b-base

model:
  n_layers: 16
  d_model: 1024
  vocab_size: 49280          # bubblspace-tokenizer real length 49164, padded to a multiple of 128
  tie_embeddings: true
  max_seq_len: 2048
  norm_eps: 1.0e-6

  attention:
    n_heads: 16
    head_dim: 64
    n_kv_heads: 4
    qk_norm: true
    rope_theta: 10000
    output_gate: true
    bias: false

  moe:
    n_experts: 64
    top_k: 8
    expert_hidden: 256
    activation: silu
    router_init_std: 0.02
    backend: auto              # auto | grouped_mm | fallback β€” resolved at startup by probing torch._grouped_mm
    load_balance_coef: 0.01
    router_z_loss_coef: 0.001

  init:
    std: 0.02
    # residual_proj_std = std / sqrt(2 * n_layers), applied to attn.o_proj and experts.W_down

optimizer:
  muon:
    lr: 6.0e-4
    momentum: 0.95
    nesterov: true
    ns_steps: 5
    weight_decay: 0.1
  adamw:
    lr: 6.0e-4
    betas: [0.9, 0.95]
    eps: 1.0e-8
    weight_decay: 0.1
  grad_clip: 1.0

schedule:
  warmup_steps: 2000
  decay: cosine               # cosine | wsd
  min_lr_ratio: 0.1
  wsd_decay_frac: 0.2         # only used when decay: wsd

training:
  dataset: HuggingFaceFW/fineweb-edu
  dataset_config: sample-10BT
  total_tokens: 10_000_000_000
  seq_len: 2048
  global_batch_tokens: 524288
  micro_batch: 16
  precision: bf16
  compile: true
  checkpoint_every: 1000
  log_every: 10
  expert_hist_every: 100
  attn_logit_every: 100

data:
  data_dir: /lambda/nfs/h100-f1/data
  checkpoint_dir: /lambda/nfs/h100-f1/checkpoints

tokenizer:
  path: ./bubblspace-tokenizer
  doc_separator: "<|endoftext|>"

wandb:
  project: moe0-9b-a0-2b
  entity: null

Benchmarks

Standard base-LM benchmark suite via lm-evaluation-harness (loglikelihood-scored, multiple-choice tasks only β€” appropriate for a base-pretrained, non-instruction-tuned model; chat/instruction-following benchmarks would not be meaningful here).

step_7000 (step 7,000 / 19,073 planned steps, ~37% through the single

10B-token FineWeb-Edu pretraining run)

Task Metric Score Random baseline
PIQA acc_norm 64.7% 50% (2-choice)
ARC-Easy acc_norm 51.1% ~25% (multi-choice)
HellaSwag acc_norm 37.0% 25% (4-choice)
LAMBADA (openai) acc (exact last-word match) 29.0% ~0% (strict exact-match)
LAMBADA (openai) perplexity 58.1 β€”
WinoGrande acc 52.4% 50% (2-choice)
ARC-Challenge acc_norm 28.0% 25% (4-choice)

Tasks rewarding broad surface-pattern learning (PIQA, ARC-Easy, HellaSwag) are clearly above chance already. The two hardest, most reasoning-heavy benchmarks (ARC-Challenge, WinoGrande) sitting near chance is expected at this model size and training stage β€” not a sign of a problem.

step_8000 (step 8,000 / 19,073 planned steps, ~42% through the run)

Task Metric step_7000 step_8000 Ξ”
PIQA acc_norm 64.7% 64.5% -0.2 (noise)
ARC-Easy acc_norm 51.1% 50.7% -0.4 (noise)
HellaSwag acc_norm 37.0% 38.0% +1.0
LAMBADA (openai) acc 29.0% 31.0% +2.0
LAMBADA (openai) perplexity 58.1 50.3 -7.8 (better)
WinoGrande acc 52.4% 51.7% -0.7 (noise)
ARC-Challenge acc_norm 28.0% 27.6% -0.4 (noise)

Most metrics are flat within statistical noise over just 1,000 steps. LAMBADA is the exception β€” a real, consistent improvement (perplexity down 13%, accuracy up 2 points), the expected signal for the metric most dependent on total tokens seen. More checkpoints will be benchmarked the same way as training progresses.

step_9000 (step 9,000 / 19,073 planned steps, ~47% through the run)

Task Metric step_8000 step_9000 Ξ”
PIQA acc_norm 64.5% 65.9% +1.4
ARC-Easy acc_norm 50.7% 50.8% +0.1 (noise)
HellaSwag acc_norm 38.0% 38.5% +0.5
LAMBADA (openai) acc 31.0% 30.6% -0.4 (noise)
LAMBADA (openai) perplexity 50.3 49.3 -1.0 (better)
WinoGrande acc 51.7% 51.5% -0.2 (noise)
ARC-Challenge acc_norm 27.6% 27.2% -0.4 (noise)

PIQA shows a real move this step (+1.4, clear of its ~1.1% stderr), and HellaSwag keeps climbing steadily. LAMBADA perplexity continues its monotonic improvement (58.1 β†’ 50.3 β†’ 49.3), now 15% better than the step_7000 baseline. The reasoning-heavy benchmarks (ARC-Challenge, WinoGrande) remain flat within noise, as expected at this model size and training stage.

Try it yourself

This repo is self-contained β€” model code, weights, tokenizer, and a generation script are all included, so you don't need the (private) training repo to run it.

Or run it directly in Colab, no setup required: https://github.com/thefirehacker/FirstBreakAI-Code/blob/main/notebooks/bubblspace_moe0_9b_a0_2b.ipynb

git lfs install
git clone https://huggingface.co/TheFireHacker/bubblspace-moe0.9b-a0.2b
cd bubblspace-moe0.9b-a0.2b
pip install -r requirements.txt
python generate.py --prompt "Your prompt here" --max_new_tokens 100

Needs a CUDA GPU for reasonable speed (falls back to CPU automatically, but will be slow). moe.backend auto-resolves to torch._grouped_mm if available on your GPU/torch build, otherwise a portable gather-loop fallback β€” both produce identical outputs.

Downloads last month

-

Downloads are not tracked for this model. How to track
Safetensors
Model size
0.9B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support