🌌 Horizon-10M-Semantic

An Ultra-Fast, Lightweight Semantic Embedding Model

Built with 🖤 by Nebulixlabs

---

🧠 Model Overview

Horizon-10M-Semantic is an ultra-fast, lightweight semantic embedding model adapted directly from Nebulixlabs/Horizon-10M-Base. While the base model was pre-trained from scratch on a 500M-token mixture, Horizon-10M-Semantic has been further trained on an additional 10M external tokens specifically to convert the causal language model into an effective 320-dimensional semantic representation engine.

The entire Horizon series is built from the ground up with one primary philosophy: blistering speed and near-zero latency. With an estimated semantic accuracy of around ~75%, this model knowingly trades off heavy, state-of-the-art precision in favor of extreme inference speed and minimal memory footprint—making it ideal for real-time applications, CPU-only environments, and edge devices where every millisecond counts.

---

⚡ Why Choose Horizon-10M-Semantic? (Speed vs. Accuracy)

Metric	Details
🎯 Estimated Accuracy	~75% (Moderate — suited for lightweight filtering and fast matching)
⚡ Inference Speed	Ultra-Fast (Sub-millisecond to few-millisecond latency even on standard CPUs)
📐 Embedding Dimension	320-d dense vectors (compact storage & rapid cosine similarity search)
🪙 Additional Training	+10,000,000 external tokens on top of the 500M-token base checkpoint

💡 When should you use this model?
If you need state-of-the-art 90%+ retrieval accuracy for complex production RAG, look at larger embedding models (100M–1B+ params). However, if you need instantaneous responses, high-throughput clustering, rapid initial candidate filtering, or local on-device search without a GPU, Horizon-10M-Semantic is engineered specifically for that workflow.

---

🏗️ Architecture Details

Component	Value
🔢 Parameters	~10.06M
🧱 Base Model	Nebulixlabs/Horizon-10M-Base
📚 Vocabulary size	8,192 (Custom Byte-Level BPE)
📐 Output Embedding Dimension (d_model)	320
🧱 Layers	6
👁️ Attention heads	5 (head_dim = 64)
🔁 MLP hidden size	864
🪟 Context length	128 tokens
🎯 Positional encoding	Rotary (RoPE)
⚖️ Normalization	RMSNorm (pre-norm)
🚀 MLP type	SwiGLU (gated, SiLU activation)
🎲 Dropout	0.0

---

📦 Training Pipeline (Base + Semantic Adaptation)

1. Base Pre-training (500M Tokens)

Inherited from Horizon-10M-Base, pre-trained on a 500M-token mixture:

• 🌐 HuggingFaceFW/fineweb-edu (440M tokens — 88.0%)
• ➗ HuggingFaceTB/finemath (30M tokens — 6.0%)
• 📘 HuggingFaceTB/smollm-corpus (25M tokens — 5.0%)
• ⭐ epfml/FineWeb-HQ (5M tokens — 1.0%)

2. Semantic Adaptation (+10M External Tokens)

To transform the causal base model into a semantic embedding extractor, the model underwent continued training on an additional 10,000,000 external tokens, aligning its 320-dimensional latent space to capture sentence-level semantic similarity and representations.

---

💻 How to Use

You can easily generate 320-dimensional semantic embeddings and compute cosine similarity using transformers and torch:

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

repo_id = "Nebulixlabs/Horizon-10M-Semantic"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

def get_embeddings(texts):
    inputs = tokenizer(
        texts,
        padding=True,
        truncation=True,
        max_length=128,
        return_tensors="pt"
    )
    with torch.no_grad():
        outputs = model(**inputs)
        # Mean pooling over token embeddings (accounting for attention mask)
        hidden_states = outputs.last_hidden_state if hasattr(outputs, "last_hidden_state") else outputs[0]
        attention_mask = inputs["attention_mask"].unsqueeze(-1).expand(hidden_states.size()).float()
        sum_embeddings = torch.sum(hidden_states * attention_mask, 1)
        sum_mask = torch.clamp(attention_mask.sum(1), min=1e-9)
        embeddings = sum_embeddings / sum_mask
        # L2 Normalize embeddings for cosine similarity
        return F.normalize(embeddings, p=2, dim=1)

sentences = [
    "Artificial intelligence is transforming modern software.",
    "Machine learning is changing how we build computer programs.",
    "I love eating hot pizza on a rainy evening."
]

embeddings = get_embeddings(sentences)

# Compute cosine similarity
sim_1_2 = torch.matmul(embeddings[0], embeddings[1]).item()
sim_1_3 = torch.matmul(embeddings[0], embeddings[2]).item()

print(f"Similarity (AI vs ML):    {sim_1_2:.4f}")
print(f"Similarity (AI vs Pizza): {sim_1_3:.4f}")

Note: If your custom model class includes a built-in helper method (e.g., model.encode() or model.get_embeddings()), you can call that directly as well!

---

🎯 Intended Use

• ✅ Ultra-fast semantic search & filtering where low latency is prioritized over maximum precision.
• ✅ Edge AI & CPU-only deployments (IoT, mobile, local desktop tools, browser/WASM ports).
• ✅ First-stage retrieval / rapid deduplication across large text collections using compact 320-d vectors.
• ✅ Educational & research benchmarking for small-scale representation learning (~10M parameter regime).

🚫 Out-of-Scope Use

• ❌ High-stakes or mission-critical retrieval (medical, legal, financial compliance) — the model has an estimated ~75% accuracy and will miss nuanced semantic relationships.
• ❌ Long-document embedding — restricted to a 128-token context window (best for short sentences, titles, queries, and snippets).

---

⚠️ Limitations & Trade-offs

• Accuracy vs. Speed Trade-off: At ~10M parameters and ~75% accuracy, this model is designed for raw speed. It may struggle with complex negation, deep domain-specific jargon, or subtle semantic nuances.
• Short Context Window: Inputs longer than 128 tokens are truncated. Keep inputs concise (search queries, sentences, short paragraphs).
• Small Vocabulary: Uses a compact 8,192-token Byte-Level BPE tokenizer; rare words or non-English text will be broken down into more subword tokens.

---

📜 Citation

@misc{horizon10m_semantic2026,
  title  = {Horizon-10M-Semantic: An Ultra-Fast Lightweight Semantic Embedding Model},
  author = {Nebulixlabs},
  year   = {2026},
  howpublished = {\url{[https://huggingface.co/Nebulixlabs/Horizon-10M-Semantic](https://huggingface.co/Nebulixlabs/Horizon-10M-Semantic)}}
}

---

🔗 Connect with Nebulixlabs

Made with 🖤 by Nebulixlabs