--- license: apache-2.0 language: - en - hi - bn - mr - ta - te - gu - kn - ml - pa library_name: pytorch pipeline_tag: text-generation tags: - small-language-model - multi-head-latent-attention - indic - kitefish base_model: KiteFishAI/NanoI1-155M --- # NanoI1-155M-Instruct **NanoI1** is a 155M-parameter decoder-only language model trained from scratch by [KiteFish AI](https://kitefishai.com) on ~1.0T tokens. It is built for low-cost, on-premise deployment in Indian regulated industries (banking, insurance, financial services). Its main uses are structured extraction, classification and grounded Q&A, where data residency and inference cost matter more than open-ended general knowledge. This repository holds the **instruction-tuned** model. The base model is [KiteFishAI/NanoI1-155M](https://huggingface.co/KiteFishAI/NanoI1-155M). | | | |---|---| | Parameters | 155.3M total: 78.0M transformer layers, 38.6M input embedding, 38.7M output projection | | Architecture | Decoder-only Transformer, Multi-Head Latent Attention (MLA), SwiGLU, RMSNorm, RoPE | | Training tokens | ~998B (pretraining 629B + extension 319B + anneal 50B), then SFT | | Context length | 2,048 tokens (trained) | | Tokenizer | Byte-level BPE, 50,304 tokens, trained on our own mixture | | Languages | English; Hindi and 8 other Indian languages (limited) | | Chat format | ChatML (`<\|im_start\|>role ... <\|im_end\|>`) | | Licence | Apache-2.0 | ## Quick start The model uses a custom architecture, so loading needs the Python files in this repository (no `transformers` class yet). ```bash pip install torch safetensors tokenizers huggingface_hub ``` ```python import sys from huggingface_hub import snapshot_download path = snapshot_download("KiteFishAI/NanoI1-155M-Instruct") sys.path.insert(0, path) from modeling_nanoi1 import from_pretrained, chat model, tok = from_pretrained(path) # GPU if available, bf16 on GPU print(chat(model, tok, [ {"role": "system", "content": "You are a concise assistant for a bank's operations team."}, {"role": "user", "content": "Explain what a bank's KYC process is in two sentences."}, ], temperature=0)) ``` ## Architecture | Hyper-parameter | Value | |---|---| | Layers | 12 | | Hidden size | 768 | | Attention heads | 12 (head dim 64) | | KV latent dimension (MLA) | 256 | | FFN | SwiGLU, 2,048 hidden | | Normalisation | Pre-norm RMSNorm (ε = 1e-6), QK-norm | | Position encoding | RoPE, θ = 5,000,000 | | Embeddings | Untied input / output | | Vocabulary | 50,304 (byte-level BPE, 32 special tokens) | Multi-Head Latent Attention caches one 256-d latent vector per token per layer instead of full keys and values (2 × 768), so the KV cache is **6× smaller** than standard multi-head attention at the same width. ## Training | Stage | Tokens | Data | LR schedule | |---|---|---|---| | 1. Pretraining | 629B (steps 0–600k) | Our 798B-token corpus, each token at most once: English web, code, peS2o papers, multilingual web, Project Gutenberg, Proof-Pile math, English Wikipedia | linear warmup, then constant 6e-4 | | 2. Extension | 319B (600k–904k) | Unseen remainder of the stage-1 corpus (169B) + FineWeb-Edu (60B), The Stack v2 code (50B), Institutional Books 1.0 English (25B), FineMath (15B) | constant 6e-4 | | 3. Anneal | 50B (904k–952k) | High-quality mix: FineWeb-Edu, Cosmopedia, FineMath, Dolmino math & FLAN, OpenCoder annealing corpus, StackExchange, FineWeb-2 Indic, plus ~35% replay of stage-1 sources | linear decay 6e-4 → 0 | | 4. SFT | 2 epochs | smol-smoltalk + Aya (English and 9 Indian languages) | 1e-4, cosine to 1e-5 | - **Benchmark decontamination:** every document added in stages 2–3 that shares a 13-word sequence with the test set of any benchmark below was removed. - **Optimiser:** AdamW (β = 0.9/0.95, weight decay 0.1, gradient clip 1.0), bf16 mixed precision. - **Batch:** global batch of 1,048,576 tokens (512 × 2,048). - **Hardware:** 8× NVIDIA H100 NVL, ~1.45M tokens/s. - **Data loading:** a deterministic exactly-once sampler, so every training window is used once per pass and restarts resume to the exact sample. ## Evaluation (base model) Base checkpoint (stage 3), evaluated with lm-evaluation-harness 0.4.13 at 2,048 context. All models used the same prompts, shots and greedy decoding. Scores in %, Pile in bits per byte (lower is better). | Benchmark | Shots | **NanoI1-155M** | Gemma-3-270M | Qwen3-0.6B-Base | |---|---|---|---|---| | WinoGrande | 5 | 52.6 | 54.0 | 59.9 | | ARC-Easy | 25 | 58.9 | 59.0 | 74.9 | | ARC-Challenge | 25 | 28.1 | 30.0 | 45.1 | | OpenBookQA | 0 | **32.2** | 30.6 | 34.6 | | RACE-High | 5 | **33.1** | 31.4 | 37.3 | | NaturalQuestions | 5 | 3.2 | 4.1 | 7.6 | | MMLU-Redux | 5 | 27.8 | 28.2 | 32.1 | | MMLU-Pro | 5 | 7.8 | 8.5 | 25.5 | | GPQA | 5 | **26.6** | 25.9 | 29.9 | | BBH | 3 | 18.4 | 22.7 | 39.4 | | GSM8K | 8 | **4.0** | 1.8 | 48.9 | | MATH | 4 | 2.2 | 2.4 | 29.9 | | HumanEval | 0 | **9.1** | 2.4 | 29.3 | | EvalPlus | 0 | **4.3** | 1.2 | 13.0 | | INCLUDE (44 languages) | – | **26.0** | 25.5 | 36.3 | | MMMLU | 5 | 23.6 | 25.6 | 38.7 | | Pile test (BPB ↓) | – | 0.870 | 0.868 | 0.716 | **Reading the table:** - **Against Gemma-3-270M:** NanoI1 is ahead on code (HumanEval ~4×), grade-school maths, reading comprehension and OpenBookQA. It is behind on commonsense benchmarks (HellaSwag, PIQA) and factual recall (TriviaQA). It has fewer parameters in its transformer layers (78M vs ~100M) and saw roughly 1/6 of Gemma's training tokens. - **Against Qwen3-0.6B-Base:** Qwen3-0.6B, with about 4× the parameters and ~36T tokens, is stronger everywhere. - **Near chance:** many knowledge benchmarks (MMLU-Pro, GPQA, SuperGPQA, MMMLU) sit close to random guessing for all models under 300M parameters, so their differences are small. ### Instruct model | Benchmark | NanoI1-155M-Instruct | SmolLM2-135M-Instruct | SmolLM2-360M-Instruct | Qwen2.5-0.5B-Instruct | |---|---|---|---|---| | IFEval (prompt-level loose) | *[fill after `run_sft.sh eval`]* | | | | ## Intended use - **Good for:** structured extraction, classification, routing and short grounded Q&A over provided documents, especially after fine-tuning on your own task data. Also a fast draft model, on-device assistant or research baseline. - **Not for:** open-ended factual questions without context, maths and multi-step reasoning, legal, medical or financial advice, or any decision about a person without human review. ## Limitations - **It invents content.** At 155M parameters the model makes up facts and can add fields that are not in the source text when extracting JSON. For production extraction, use schema-constrained decoding (e.g. llguidance, Outlines or XGrammar) and validate outputs. - **Short context.** It was trained at 2,048 tokens; the RoPE base supports longer contexts but quality beyond 2k is untested. - **Limited Indian-language ability.** Indian languages are a small share of training, so quality there is well below English. - **Bias.** The model reflects biases present in web, book and code data. - **No safety training.** No preference or safety tuning has been applied yet. Add your own guardrails. ## Training data and licences All third-party datasets were used under their published terms: - **ODC-By:** FineWeb-Edu, FineMath, FineWeb-2, smollm-corpus (Cosmopedia v2), OpenCoder annealing corpus, Dolmino mix. - **Apache-2.0:** smol-smoltalk, Aya. - **The Stack v2:** per-file licences. - **Institutional Books 1.0:** Institutional Data Initiative terms. See each dataset card for details. Some anneal data (Cosmopedia, parts of FLAN and Dolmino math) was generated by other language models. ## Citation ```bibtex @techreport{kitefish2026nanoi1, title = {NanoI1: A 155M-Parameter Language Model Trained on One Trillion Tokens for Indian Regulated Industries}, author = {Anuj and {KiteFish AI}}, institution = {KiteFish AI}, year = {2026} } ```