🌌 Horizon-10M-Instruct

An SFT Fine-Tune of Horizon-10M-Base — Short, Direct Answers Instead of Drift

Built with 🖤 by Nebulixlabs

---

🧠 Model Overview

Horizon-10M-Instruct is a supervised fine-tune of Nebulixlabs/Horizon-10M-Base, a ~10M-parameter, from-scratch decoder-only LM (RoPE + RMSNorm + SwiGLU). It was SFT'd on Nebulixlabs/FineTuning, a small, focused instruction dataset of 1,000 rows.

The motivation: in testing, the base model would predict a handful of correct words after a prompt and then drift into unrelated, garbled continuation. Rather than scaling up data or parameters, this fine-tune targets that specific failure mode with a short, high-signal SFT set — and in our own testing it now reliably gives a direct, on-topic answer instead of trailing off. Don't expect more than a small instruct model can give, but for its size this is, to our knowledge, unusually coherent.

---

🏗️ Architecture Details

Same backbone as the base model — fine-tuning changed only the weights, not the architecture.

Component	Value
🔢 Parameters	~10.06M
📚 Vocabulary size	8,192
📐 Hidden size (d_model)	320
🧱 Layers	6
👁️ Attention heads	5 (head_dim = 64)
🔁 MLP hidden size	864
🪟 Context length	128 tokens
🎯 Positional encoding	Rotary (RoPE)
⚖️ Normalization	RMSNorm (pre-norm)
🚀 MLP type	SwiGLU (gated, SiLU activation)
🔗 Weight tying	Input embedding ↔ LM head

---

📊 Fine-Tuning Details

Setting	Value
🧬 Base model	Nebulixlabs/Horizon-10M-Base
📦 SFT dataset	Nebulixlabs/FineTuning
🔢 SFT rows	1,000
🎯 Method	Supervised fine-tuning (SFT) on top of the pretrained base checkpoint
🔤 Tokenizer	Same custom Byte-Level BPE as the base model (vocab size 8,192)
🪟 Sequence length	128 tokens

The goal of this SFT pass was narrow and deliberate: a small, curated dataset rather than a large generic instruction set, aimed squarely at fixing the base model's tendency to go off-topic after the first few tokens.

---

🧪 Sample Raw Output

Unedited generation from Horizon-10M-Instruct:

Prompt: Tell me the full form of ISRO.

Horizon: The brilliant Indian Space Research Organisation is officially
declared as the national space agency.

Compared to the base model's tendency to drift into unrelated text after a few correct words, this fine-tune stays on-topic and answers directly — the result we were aiming for with the 1,000-row SFT set. We haven't seen another model at this parameter scale produce answers this coherent, though we'd genuinely like to be proven wrong — let us know if you find one.

---

💻 How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM
repo_id = "Nebulixlabs/Horizon-10M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
output = model.generate_text(
    tokenizer,
    prompt="Tell me the full form of ISRO.",
    max_new_tokens=60,
    temperature=0.75,
    top_k=40,
    top_p=0.9,
    repetition_penalty=1.18
)
print(output)

Generation tips:

• Keep prompts short and direct — the model only has a 128-token context window.
• Lower temperature (0.6–0.8) for more stable, less erratic output.
• repetition_penalty around 1.1–1.2 helps reduce looping on such a small model.

---

🎯 Intended Use

• ✅ Educational purposes — seeing how a small amount of targeted SFT reshapes a tiny from-scratch base model's behavior.
• ✅ Short, direct Q&A-style prompts similar in style to the fine-tuning set.
• ✅ Fast local experimentation on limited hardware (CPU-friendly inference).
• ✅ Baseline/reference model for small-scale SFT experiments.

🚫 Out-of-Scope Use

• ❌ Any production, factual, medical, legal, or safety-critical use.
• ❌ Long-form, multi-turn, or knowledge-intensive conversation — 10M params / 128-token context and a 1,000-row SFT set can't support this.
• ❌ Use as a source of factual information — the model will still hallucinate outside the narrow style it was tuned on.

---

⚠️ Limitations & Biases

• Set realistic expectations: this is a ~10M-parameter model fine-tuned on only 1,000 rows — it is good at short, direct answers in the style it was tuned on, not general-purpose instruction following.
• Inherits all the base model's limitations (small 8,192 vocab, 128-token context, biases from its pretraining mixture).
• The small, targeted SFT set means coverage is narrow; prompts far outside that style can still produce incoherent or off-topic output.
• No RLHF or broader safety alignment has been applied.

---

📜 Citation

@misc{horizon10minstruct2026,
  title  = {Horizon-10M-Instruct: A Small SFT-Tuned Causal Language Model},
  author = {Nebulixlabs},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Nebulixlabs/Horizon-10M-Instruct}}
}

---

🔗 Connect with Nebulixlabs

Made with 🖤 by Nebulixlabs