Nebulixlabs/Horizon-10M-Base

🌌 Horizon-10M

A Tiny, Modern, From-Scratch Causal Language Model

Built with 🖤 by Nebulixlabs

---

🧠 Model Overview

Horizon-10M-Base is a compact, modern-architecture causal language model trained completely from scratch on a 400M-token slice of HuggingFaceFW/fineweb-edu. It uses a GPT-style decoder stack with RoPE positional embeddings, RMSNorm, and a SwiGLU MLP — the same core design patterns used in modern LLMs like LLaMA, just scaled way down (~10M params) for fast experimentation, education, and low-resource environments.

---

🏗️ Architecture Details

Component	Value
🔢 Parameters	~10.06M
📚 Vocabulary size	8,192
📐 Hidden size (d_model)	320
🧱 Layers	6
👁️ Attention heads	5 (head_dim = 64)
🔁 MLP hidden size	864
🪟 Context length	128 tokens
🎯 Positional encoding	Rotary (RoPE)
⚖️ Normalization	RMSNorm (pre-norm)
🚀 MLP type	SwiGLU (gated, SiLU activation)
🔗 Weight tying	Input embedding ↔ LM head
🎲 Dropout	0.0

Attention: Multi-head self-attention with RoPE, using PyTorch's fused scaled_dot_product_attention (causal masking).

MLP block: SwiGLU gated feed-forward (w2(silu(w1(x)) * w3(x))), same family used in LLaMA/Mistral.

---

📊 Training Details

Setting	Value
📦 Dataset	HuggingFaceFW/fineweb-edu (sample-10BT config)
🔤 Tokenizer	Custom Byte-Level BPE, vocab size 8,192 (trained from scratch on fineweb-edu)
🪙 Total training tokens	~400,000,000
📏 Sequence length	128
🧮 Batch size	512 sequences (65,536 tokens/step)
📈 Peak LR	6e-4 (cosine decay → 6e-5)
🔥 Warmup steps	200
⚙️ Optimizer	AdamW (fused), betas=(0.9, 0.95), weight decay 0.1
✂️ Grad clipping	1.0
🖥️ Hardware	2× NVIDIA T4 (Kaggle)
⏱️ Precision	Mixed precision (fp16 autocast + GradScaler)
🧵 Data pipeline	Pre-tokenized to a flat uint16 memmap binary for near-zero dataloading overhead

The tokenizer is a custom-trained Byte-Level BPE (not reused from any existing model), trained on ~40k documents sampled from fineweb-edu, with special tokens <pad>, <bos>, <eos>, <unk>.

---

💻 How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "Nebulixlabs/Horizon-10M-Base"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)

output = model.generate_text(
    tokenizer,
    prompt="The history of artificial intelligence",
    max_new_tokens=60,
    temperature=0.75,
    top_k=40,
    top_p=0.9,
    repetition_penalty=1.18
)
print(output)

Generation tips:

• Keep prompts short — the model only has a 128-token context window.
• Lower temperature (0.6–0.8) for more stable, less erratic output.
• repetition_penalty around 1.1–1.2 helps reduce looping on such a small model.

---

🎯 Intended Use

• ✅ Educational purposes — understanding how modern decoder-only LLMs (RoPE, RMSNorm, SwiGLU) are built and trained end-to-end.
• ✅ Fast local experimentation on limited hardware (CPU-friendly inference).
• ✅ Baseline/reference model for architecture or training-pipeline experiments.

🚫 Out-of-Scope Use

• ❌ Any production, factual, medical, legal, or safety-critical use.
• ❌ Long-form or knowledge-intensive generation — 10M params / 128-token context can't support this.
• ❌ Use as a source of factual information — the model will hallucinate.

---

⚠️ Limitations & Biases

• Trained on a relatively small slice of fineweb-edu; inherits whatever biases and gaps exist in that web-scraped educational text corpus.
• Small vocabulary (8,192 tokens) and short context (128 tokens) significantly limit fluency, coherence, and factual grounding.
• No RLHF / instruction-tuning / safety-alignment has been applied — this is a raw base model, not a chat/assistant model.
• Outputs can be repetitive, ungrammatical, or nonsensical, especially beyond a few sentences.

---

📜 Citation

@misc{horizon10m2026,
  title  = {Horizon-10M-Base: A Small From-Scratch Causal Language Model},
  author = {Nebulixlabs},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Nebulixlabs/Horizon-10M-Base}}
}

---

🔗 Connect with Nebulixlabs

Made with 🖤 by Nebulixlabs