anuj0456's picture
Update README.md
feea2d5 verified
|
Raw History Blame Contribute Delete
7.98 kB
---
license: apache-2.0
language:
- en
- hi
- bn
- mr
- ta
- te
- gu
- kn
- ml
- pa
library_name: pytorch
pipeline_tag: text-generation
tags:
- small-language-model
- multi-head-latent-attention
- indic
- kitefish
base_model: KiteFishAI/NanoI1-155M
---
# NanoI1-155M-Instruct
**NanoI1** is a 155M-parameter decoder-only language model trained from scratch by
[KiteFish AI](https://kitefishai.com) on ~1.0T tokens. It is built for low-cost, on-premise deployment in
Indian regulated industries (banking, insurance, financial services). Its main uses are structured extraction,
classification and grounded Q&A, where data residency and inference cost matter more than open-ended
general knowledge.
This repository holds the **instruction-tuned** model. The base model is
[KiteFishAI/NanoI1-155M](https://huggingface.co/KiteFishAI/NanoI1-155M).
| | |
|---|---|
| Parameters | 155.3M total: 78.0M transformer layers, 38.6M input embedding, 38.7M output projection |
| Architecture | Decoder-only Transformer, Multi-Head Latent Attention (MLA), SwiGLU, RMSNorm, RoPE |
| Training tokens | ~998B (pretraining 629B + extension 319B + anneal 50B), then SFT |
| Context length | 2,048 tokens (trained) |
| Tokenizer | Byte-level BPE, 50,304 tokens, trained on our own mixture |
| Languages | English; Hindi and 8 other Indian languages (limited) |
| Chat format | ChatML (`<\|im_start\|>role ... <\|im_end\|>`) |
| Licence | Apache-2.0 |
## Quick start
The model uses a custom architecture, so loading needs the Python files in this repository (no `transformers`
class yet).
```bash
pip install torch safetensors tokenizers huggingface_hub
```
```python
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("KiteFishAI/NanoI1-155M-Instruct")
sys.path.insert(0, path)
from modeling_nanoi1 import from_pretrained, chat
model, tok = from_pretrained(path) # GPU if available, bf16 on GPU
print(chat(model, tok, [
{"role": "system", "content": "You are a concise assistant for a bank's operations team."},
{"role": "user", "content": "Explain what a bank's KYC process is in two sentences."},
], temperature=0))
```
## Architecture
| Hyper-parameter | Value |
|---|---|
| Layers | 12 |
| Hidden size | 768 |
| Attention heads | 12 (head dim 64) |
| KV latent dimension (MLA) | 256 |
| FFN | SwiGLU, 2,048 hidden |
| Normalisation | Pre-norm RMSNorm (ε = 1e-6), QK-norm |
| Position encoding | RoPE, θ = 5,000,000 |
| Embeddings | Untied input / output |
| Vocabulary | 50,304 (byte-level BPE, 32 special tokens) |
Multi-Head Latent Attention caches one 256-d latent vector per token per layer instead of full keys and values
(2 × 768), so the KV cache is **6× smaller** than standard multi-head attention at the same width.
## Training
| Stage | Tokens | Data | LR schedule |
|---|---|---|---|
| 1. Pretraining | 629B (steps 0–600k) | Our 798B-token corpus, each token at most once: English web, code, peS2o papers, multilingual web, Project Gutenberg, Proof-Pile math, English Wikipedia | linear warmup, then constant 6e-4 |
| 2. Extension | 319B (600k–904k) | Unseen remainder of the stage-1 corpus (169B) + FineWeb-Edu (60B), The Stack v2 code (50B), Institutional Books 1.0 English (25B), FineMath (15B) | constant 6e-4 |
| 3. Anneal | 50B (904k–952k) | High-quality mix: FineWeb-Edu, Cosmopedia, FineMath, Dolmino math & FLAN, OpenCoder annealing corpus, StackExchange, FineWeb-2 Indic, plus ~35% replay of stage-1 sources | linear decay 6e-4 → 0 |
| 4. SFT | 2 epochs | smol-smoltalk + Aya (English and 9 Indian languages) | 1e-4, cosine to 1e-5 |
- **Benchmark decontamination:** every document added in stages 2–3 that shares a 13-word sequence with the
test set of any benchmark below was removed.
- **Optimiser:** AdamW (β = 0.9/0.95, weight decay 0.1, gradient clip 1.0), bf16 mixed precision.
- **Batch:** global batch of 1,048,576 tokens (512 × 2,048).
- **Hardware:** 8× NVIDIA H100 NVL, ~1.45M tokens/s.
- **Data loading:** a deterministic exactly-once sampler, so every training window is used once per pass and
restarts resume to the exact sample.
## Evaluation (base model)
Base checkpoint (stage 3), evaluated with lm-evaluation-harness 0.4.13 at 2,048 context. All models used the
same prompts, shots and greedy decoding. Scores in %, Pile in bits per byte (lower is better).
| Benchmark | Shots | **NanoI1-155M** | Gemma-3-270M | Qwen3-0.6B-Base |
|---|---|---|---|---|
| WinoGrande | 5 | 52.6 | 54.0 | 59.9 |
| ARC-Easy | 25 | 58.9 | 59.0 | 74.9 |
| ARC-Challenge | 25 | 28.1 | 30.0 | 45.1 |
| OpenBookQA | 0 | **32.2** | 30.6 | 34.6 |
| RACE-High | 5 | **33.1** | 31.4 | 37.3 |
| NaturalQuestions | 5 | 3.2 | 4.1 | 7.6 |
| MMLU-Redux | 5 | 27.8 | 28.2 | 32.1 |
| MMLU-Pro | 5 | 7.8 | 8.5 | 25.5 |
| GPQA | 5 | **26.6** | 25.9 | 29.9 |
| BBH | 3 | 18.4 | 22.7 | 39.4 |
| GSM8K | 8 | **4.0** | 1.8 | 48.9 |
| MATH | 4 | 2.2 | 2.4 | 29.9 |
| HumanEval | 0 | **9.1** | 2.4 | 29.3 |
| EvalPlus | 0 | **4.3** | 1.2 | 13.0 |
| INCLUDE (44 languages) | – | **26.0** | 25.5 | 36.3 |
| MMMLU | 5 | 23.6 | 25.6 | 38.7 |
| Pile test (BPB ↓) | – | 0.870 | 0.868 | 0.716 |
**Reading the table:**
- **Against Gemma-3-270M:** NanoI1 is ahead on code (HumanEval ~4×), grade-school maths, reading
comprehension and OpenBookQA. It is behind on commonsense benchmarks (HellaSwag, PIQA) and factual recall
(TriviaQA). It has fewer parameters in its transformer layers (78M vs ~100M) and saw roughly 1/6 of
Gemma's training tokens.
- **Against Qwen3-0.6B-Base:** Qwen3-0.6B, with about 4× the parameters and ~36T tokens, is stronger
everywhere.
- **Near chance:** many knowledge benchmarks (MMLU-Pro, GPQA, SuperGPQA, MMMLU) sit close to random guessing
for all models under 300M parameters, so their differences are small.
### Instruct model
| Benchmark | NanoI1-155M-Instruct | SmolLM2-135M-Instruct | SmolLM2-360M-Instruct | Qwen2.5-0.5B-Instruct |
|---|---|---|---|---|
| IFEval (prompt-level loose) | *[fill after `run_sft.sh eval`]* | | | |
## Intended use
- **Good for:** structured extraction, classification, routing and short grounded Q&A over provided documents,
especially after fine-tuning on your own task data. Also a fast draft model, on-device assistant or research
baseline.
- **Not for:** open-ended factual questions without context, maths and multi-step reasoning, legal, medical or
financial advice, or any decision about a person without human review.
## Limitations
- **It invents content.** At 155M parameters the model makes up facts and can add fields that are not in the
source text when extracting JSON. For production extraction, use schema-constrained decoding (e.g. llguidance,
Outlines or XGrammar) and validate outputs.
- **Short context.** It was trained at 2,048 tokens; the RoPE base supports longer contexts but quality beyond
2k is untested.
- **Limited Indian-language ability.** Indian languages are a small share of training, so quality there is
well below English.
- **Bias.** The model reflects biases present in web, book and code data.
- **No safety training.** No preference or safety tuning has been applied yet. Add your own guardrails.
## Training data and licences
All third-party datasets were used under their published terms:
- **ODC-By:** FineWeb-Edu, FineMath, FineWeb-2, smollm-corpus (Cosmopedia v2), OpenCoder annealing corpus,
Dolmino mix.
- **Apache-2.0:** smol-smoltalk, Aya.
- **The Stack v2:** per-file licences.
- **Institutional Books 1.0:** Institutional Data Initiative terms.
See each dataset card for details. Some anneal data (Cosmopedia, parts of FLAN and Dolmino math) was generated
by other language models.
## Citation
```bibtex
@techreport{kitefish2026nanoi1,
title = {NanoI1: A 155M-Parameter Language Model Trained on One Trillion Tokens for Indian Regulated Industries},
author = {Anuj and {KiteFish AI}},
institution = {KiteFish AI},
year = {2026}
}
```