🪄 Horizon-10M-Corrector

An Ultra-Compact, LM-Based Spell Correction Model (~10.06M Params)

Built with 🖤 by Nebulixlabs

---

🧠 Model Overview

Horizon-10M-Corrector is an ultra-compact, language-model-based spelling and typo correction model trained via Supervised Fine-Tuning (SFT).

• Base Architecture: Built upon Nebulixlabs/Horizon-10M-Base, inheriting its modern LLaMA-style decoder-only design (Rotary Position Embeddings [RoPE], RMSNorm pre-normalization, and SwiGLU gated MLPs).
• Context-Aware Error Resolution: Unlike traditional rule-based lookups or edit-distance metrics (e.g., Levenshtein/Hunspell), Horizon-10M-Corrector leverages full bidirectional-in-context causal modeling to correct phonetic mistakes, keystroke slips, and character transpositions.
• High Efficiency-to-Size Ratio: At just 10.06M parameters, the model achieves **70% correction accuracy** on held-out evaluation sets, making it exceptionally well-suited for edge deployments, browser extensions, and bare-metal CPU runtimes.
• Supervised Loss Masking: During training, the prompt prefix (Input: ...\nCorrected text:) is masked using -100 labels. Gradient updates are calculated strictly on the corrected target tokens and the concluding <eos> token.
• Weight Duplication Preservation: Maintains tied input/output embeddings (tok_emb and lm_head). Both weight tensors are explicitly serialized in the final model.safetensors file to retain checkpoint structure and ensure plug-and-play loader compatibility.

---

⚙️ Fine-Tuning Recipe & Hyperparameters

The model was fine-tuned on a Google Colab instance equipped with an NVIDIA Tesla T4 GPU using the following specifications:

Hyperparameter / Config	Value
Base Model	Nebulixlabs/Horizon-10M-Base
Fine-Tuning Dataset	Nebulixlabs/Kuch-Bhi
Supervised Columns	user (source input) $\rightarrow$ correct (clean target)
Training Set Size	2,000 paired rows
Held-out Evaluation	5 validation instances
Epochs	5
Batch Size	16
Optimizer	AdamW (betas=(0.9, 0.95), weight_decay=0.01)
Peak Learning Rate	3e-5
Gradient Clipping	1.0
Training Precision	Mixed Precision (FP16 autocast + GradScaler)
Export Format	FP32 model.safetensors
Context Window	128 tokens
Loss Function	Cross-Entropy (ignore_index=-100)

---

🏗️ Architecture Specifications

Attribute	Specification
🔢 Total Unique Parameters	~10.06M
📚 Vocabulary Size	8,192 (Byte-Level BPE, unchanged)
📐 Hidden Dimension (d_model)	320
🧱 Transformer Layers	6
👁️ Attention Heads	5 (head_dim = 64)
🔁 MLP Intermediate Dimension	864 (SwiGLU)
🪟 Context Length	128 tokens
🎯 Positional Encoding	Rotary Positional Embeddings (RoPE)
⚖️ Normalization	RMSNorm (Pre-norm)
🔗 Embedding Scheme	Tied (tok_emb.weight $\leftrightarrow$ lm_head.weight)

---

💻 How to Use

The fine-tuned model expects the structured prompt format used during training:

Input: <misspelled_text>
Corrected text:

PyTorch / Hugging Face Transformers Inference

# ============================================================
# HORIZON-10M / NUTRAL SPELL CORRECTOR - INFERENCE
# Loads your fine-tuned local model and corrects entered text.
# ============================================================

import os
import zipfile
import torch

from pathlib import Path
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast
from safetensors.torch import load_file

# -------------------- MODEL PATH --------------------

MODEL_DIR = Path("/content/Nutral-Kuch-Bhi-Finetuned")
ZIP_PATH = Path("/content/Nutral-Kuch-Bhi-Finetuned.zip")

DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
USE_CUDA = torch.cuda.is_available()

print("=" * 55)
print("NUTRAL SPELL CORRECTOR")
print("=" * 55)
print("Device:", DEVICE)

# If the model folder is missing, extract the saved ZIP.
if not MODEL_DIR.is_dir():
    if ZIP_PATH.is_file():
        MODEL_DIR.mkdir(parents=True, exist_ok=True)

        with zipfile.ZipFile(ZIP_PATH, "r") as zf:
            zf.extractall(MODEL_DIR)

    else:
        from google.colab import files

        print("Upload your Nutral-Kuch-Bhi-Finetuned.zip file.")
        uploaded = files.upload()

        zip_file = next(
            (
                name for name in uploaded
                if name.lower().endswith(".zip")
            ),
            None,
        )

        if zip_file is None:
            raise FileNotFoundError(
                "Fine-tuned model ZIP upload nahi hui."
            )

        MODEL_DIR.mkdir(parents=True, exist_ok=True)

        with zipfile.ZipFile(zip_file, "r") as zf:
            zf.extractall(MODEL_DIR)

# -------------------- LOAD ORIGINAL TOKENIZER --------------------

tokenizer_path = MODEL_DIR / "tokenizer.json"

if not tokenizer_path.is_file():
    raise FileNotFoundError(
        f"Tokenizer nahi mila: {tokenizer_path}"
    )

tokenizer = PreTrainedTokenizerFast(
    tokenizer_file=str(tokenizer_path),
    bos_token="<bos>",
    eos_token="<eos>",
    pad_token="<pad>",
    unk_token="<unk>",
)

BOS_ID = tokenizer.bos_token_id
EOS_ID = tokenizer.eos_token_id

print("Tokenizer loaded:", len(tokenizer), "tokens")

# -------------------- LOAD FINETUNED MODEL --------------------

weights_path = MODEL_DIR / "model.safetensors"

if not weights_path.is_file():
    raise FileNotFoundError(
        f"Fine-tuned weights nahi mile: {weights_path}"
    )

print("Loading fine-tuned model...")

try:
    model = AutoModelForCausalLM.from_pretrained(
        str(MODEL_DIR),
        trust_remote_code=True,
        use_safetensors=True,
        local_files_only=True,
    )

except Exception as first_error:
    # Fallback: instantiate the local architecture and load safetensors.
    print("Standard loading failed; trying direct safetensors loading.")
    print("Reason:", str(first_error)[:250])

    from transformers import AutoConfig

    config = AutoConfig.from_pretrained(
        str(MODEL_DIR),
        trust_remote_code=True,
        local_files_only=True,
    )

    model = AutoModelForCausalLM.from_config(
        config,
        trust_remote_code=True,
    )

    state_dict = load_file(
        str(weights_path),
        device="cpu",
    )

    incompatible = model.load_state_dict(
        state_dict,
        strict=False,
    )

    print("Missing keys:", incompatible.missing_keys)
    print("Unexpected keys:", incompatible.unexpected_keys)

    del state_dict

model.to(DEVICE)
model.eval()

# Read context length from the original model configuration.
context_values = []

for attr in (
    "context_len",
    "max_position_embeddings",
    "n_positions",
    "max_seq_len",
    "seq_length",
):
    value = getattr(model.config, attr, None)

    if isinstance(value, int) and value >= 16:
        context_values.append(value)

MAX_LENGTH = min([128] + context_values)

print("Model loaded successfully!")
print("Parameters:", f"{sum(p.numel() for p in model.parameters()):,}")
print("Context length:", MAX_LENGTH)
print("Ready for spelling correction.\n")

# -------------------- MODEL FORWARD --------------------

def get_logits(input_ids):
    try:
        output = model(input_ids=input_ids)

    except TypeError:
        try:
            output = model(idx=input_ids)
        except TypeError:
            output = model(input_ids)

    if hasattr(output, "logits"):
        logits = output.logits

    elif isinstance(output, (tuple, list)):
        logits = output[0]

    else:
        logits = output

    if not torch.is_tensor(logits) or logits.ndim != 3:
        raise RuntimeError(
            "Model se expected [batch, sequence, vocab] logits nahi mile."
        )

    return logits

# -------------------- SPELL CORRECTION --------------------

@torch.inference_mode()
def correct_spelling(text, max_new_tokens=64):
    text = text.strip()

    if not text:
        return ""

    # Use exactly the same prompt format used during fine-tuning.
    prompt = (
        f"Input: {text}\n"
        f"Corrected text:"
    )

    input_ids = tokenizer.encode(
        prompt,
        add_special_tokens=False,
    )

    # Add BOS manually, matching the fine-tuning code.
    if BOS_ID is not None:
        if not input_ids or input_ids[0] != BOS_ID:
            input_ids = [BOS_ID] + input_ids

    generated_ids = []

    for _ in range(max_new_tokens):
        context_ids = input_ids[-MAX_LENGTH:]

        ids = torch.tensor(
            [context_ids],
            dtype=torch.long,
            device=DEVICE,
        )

        with torch.autocast(
            device_type="cuda",
            dtype=torch.float16,
            enabled=USE_CUDA,
        ):
            logits = get_logits(ids)

        # Greedy decoding: choose the most likely next token.
        next_id = int(
            logits[0, -1, :].float().argmax().item()
        )

        if EOS_ID is not None and next_id == EOS_ID:
            break

        generated_ids.append(next_id)
        input_ids.append(next_id)

    corrected = tokenizer.decode(
        generated_ids,
        skip_special_tokens=True,
    ).strip()

    # Keep only the first output line.
    if corrected:
        corrected = corrected.splitlines()[0].strip()

    return corrected

# -------------------- INTERACTIVE LOOP --------------------

print("=" * 55)
print("Enter text to correct.")
print("Type 'exit' to stop.")
print("=" * 55)

while True:
    try:
        user_input = input("\nInput text: ").strip()
    except (EOFError, KeyboardInterrupt):
        print("\nInference stopped.")
        break

    if user_input.lower() in {"exit", "quit", "q"}:
        print("Inference stopped.")
        break

    if not user_input:
        print("Please enter some text.")
        continue

    try:
        corrected_text = correct_spelling(user_input)

        print("\nCorrected text:")
        print(corrected_text if corrected_text else "(No output generated)")

    except Exception as e:
        print("Inference error:", repr(e))

---

🎯 Intended Use

• ✅ Edge & On-Device Applications: Sub-15ms local inference on standard CPU, mobile architectures, and embedded devices without cloud API dependencies.
• ✅ Pipeline Normalization: Preprocessing automated speech recognition (ASR) transcriptions, OCR results, and user queries prior to downstream search or intent indexing.
• ✅ Research & Educational Baseline: Demonstrating how small base models (~10M parameters) adapt to specific closed-domain downstream tasks under instruction fine-tuning.

🚫 Out-of-Scope Use

• ❌ Multi-paragraph document editing or long-form grammatical structural rewriting.
• ❌ Factual generation, reasoning, or general conversational assistant tasks (limited parameter scale invites hallucinations outside correction bounds).
• ❌ Inputs significantly exceeding 100 tokens.

---

⚠️ Limitations & Technical Considerations

• Strict Context Boundary: Context length is capped at 128 tokens. Sentences must be segmented into clauses or short sentences for optimal performance.
• Vocabulary Constraints: The custom Byte-Level BPE tokenizer contains 8,192 tokens. Extremely rare named entities, technical jargon, or domain-specific identifiers may be over-regularized into common alternatives.
• Single-Line Targets: Trained predominantly on single-sentence input-output pairs; multi-line formatted strings should be passed sequentially.

---

📜 Citation

@misc{horizon10mcorrector2026,
  title  = {Horizon-10M-Corrector: An Ultra-Compact LM-Based Spell Correction Model},
  author = {Nebulixlabs},
  year   = {2026},
  howpublished = {\url{[https://huggingface.co/Nebulixlabs/Horizon-10M-Corrector](https://huggingface.co/Nebulixlabs/Horizon-10M-Corrector)}}
}

---

🔗 Connect with Nebulixlabs

Made with 🖤 by Nebulixlabs