|
Download README.md from KiteFishAI/NanoI1-155M-Instruct: direct link, hf CLI and curl.
- Browser
- Download file 7.98 kB
-
https://huggingface.co/KiteFishAI/NanoI1-155M-Instruct/resolve/main/README.md
- Command line
-
hf download hf://KiteFishAI/NanoI1-155M-Instruct/README.md
-
curl -L -o README.md https://huggingface.co/KiteFishAI/NanoI1-155M-Instruct/resolve/main/README.md
7.98 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| - hi | |
| - bn | |
| - mr | |
| - ta | |
| - te | |
| - gu | |
| - kn | |
| - ml | |
| - pa | |
| library_name: pytorch | |
| pipeline_tag: text-generation | |
| tags: | |
| - small-language-model | |
| - multi-head-latent-attention | |
| - indic | |
| - kitefish | |
| base_model: KiteFishAI/NanoI1-155M | |
| # NanoI1-155M-Instruct | |
| **NanoI1** is a 155M-parameter decoder-only language model trained from scratch by | |
| [KiteFish AI](https://kitefishai.com) on ~1.0T tokens. It is built for low-cost, on-premise deployment in | |
| Indian regulated industries (banking, insurance, financial services). Its main uses are structured extraction, | |
| classification and grounded Q&A, where data residency and inference cost matter more than open-ended | |
| general knowledge. | |
| This repository holds the **instruction-tuned** model. The base model is | |
| [KiteFishAI/NanoI1-155M](https://huggingface.co/KiteFishAI/NanoI1-155M). | |
| | | | | |
| |---|---| | |
| | Parameters | 155.3M total: 78.0M transformer layers, 38.6M input embedding, 38.7M output projection | | |
| | Architecture | Decoder-only Transformer, Multi-Head Latent Attention (MLA), SwiGLU, RMSNorm, RoPE | | |
| | Training tokens | ~998B (pretraining 629B + extension 319B + anneal 50B), then SFT | | |
| | Context length | 2,048 tokens (trained) | | |
| | Tokenizer | Byte-level BPE, 50,304 tokens, trained on our own mixture | | |
| | Languages | English; Hindi and 8 other Indian languages (limited) | | |
| | Chat format | ChatML (`<\|im_start\|>role ... <\|im_end\|>`) | | |
| | Licence | Apache-2.0 | | |
| ## Quick start | |
| The model uses a custom architecture, so loading needs the Python files in this repository (no `transformers` | |
| class yet). | |
| ```bash | |
| pip install torch safetensors tokenizers huggingface_hub | |
| ``` | |
| ```python | |
| import sys | |
| from huggingface_hub import snapshot_download | |
| path = snapshot_download("KiteFishAI/NanoI1-155M-Instruct") | |
| sys.path.insert(0, path) | |
| from modeling_nanoi1 import from_pretrained, chat | |
| model, tok = from_pretrained(path) # GPU if available, bf16 on GPU | |
| print(chat(model, tok, [ | |
| {"role": "system", "content": "You are a concise assistant for a bank's operations team."}, | |
| {"role": "user", "content": "Explain what a bank's KYC process is in two sentences."}, | |
| ], temperature=0)) | |
| ``` | |
| ## Architecture | |
| | Hyper-parameter | Value | | |
| |---|---| | |
| | Layers | 12 | | |
| | Hidden size | 768 | | |
| | Attention heads | 12 (head dim 64) | | |
| | KV latent dimension (MLA) | 256 | | |
| | FFN | SwiGLU, 2,048 hidden | | |
| | Normalisation | Pre-norm RMSNorm (ε = 1e-6), QK-norm | | |
| | Position encoding | RoPE, θ = 5,000,000 | | |
| | Embeddings | Untied input / output | | |
| | Vocabulary | 50,304 (byte-level BPE, 32 special tokens) | | |
| Multi-Head Latent Attention caches one 256-d latent vector per token per layer instead of full keys and values | |
| (2 × 768), so the KV cache is **6× smaller** than standard multi-head attention at the same width. | |
| ## Training | |
| | Stage | Tokens | Data | LR schedule | | |
| |---|---|---|---| | |
| | 1. Pretraining | 629B (steps 0–600k) | Our 798B-token corpus, each token at most once: English web, code, peS2o papers, multilingual web, Project Gutenberg, Proof-Pile math, English Wikipedia | linear warmup, then constant 6e-4 | | |
| | 2. Extension | 319B (600k–904k) | Unseen remainder of the stage-1 corpus (169B) + FineWeb-Edu (60B), The Stack v2 code (50B), Institutional Books 1.0 English (25B), FineMath (15B) | constant 6e-4 | | |
| | 3. Anneal | 50B (904k–952k) | High-quality mix: FineWeb-Edu, Cosmopedia, FineMath, Dolmino math & FLAN, OpenCoder annealing corpus, StackExchange, FineWeb-2 Indic, plus ~35% replay of stage-1 sources | linear decay 6e-4 → 0 | | |
| | 4. SFT | 2 epochs | smol-smoltalk + Aya (English and 9 Indian languages) | 1e-4, cosine to 1e-5 | | |
| - **Benchmark decontamination:** every document added in stages 2–3 that shares a 13-word sequence with the | |
| test set of any benchmark below was removed. | |
| - **Optimiser:** AdamW (β = 0.9/0.95, weight decay 0.1, gradient clip 1.0), bf16 mixed precision. | |
| - **Batch:** global batch of 1,048,576 tokens (512 × 2,048). | |
| - **Hardware:** 8× NVIDIA H100 NVL, ~1.45M tokens/s. | |
| - **Data loading:** a deterministic exactly-once sampler, so every training window is used once per pass and | |
| restarts resume to the exact sample. | |
| ## Evaluation (base model) | |
| Base checkpoint (stage 3), evaluated with lm-evaluation-harness 0.4.13 at 2,048 context. All models used the | |
| same prompts, shots and greedy decoding. Scores in %, Pile in bits per byte (lower is better). | |
| | Benchmark | Shots | **NanoI1-155M** | Gemma-3-270M | Qwen3-0.6B-Base | | |
| |---|---|---|---|---| | |
| | WinoGrande | 5 | 52.6 | 54.0 | 59.9 | | |
| | ARC-Easy | 25 | 58.9 | 59.0 | 74.9 | | |
| | ARC-Challenge | 25 | 28.1 | 30.0 | 45.1 | | |
| | OpenBookQA | 0 | **32.2** | 30.6 | 34.6 | | |
| | RACE-High | 5 | **33.1** | 31.4 | 37.3 | | |
| | NaturalQuestions | 5 | 3.2 | 4.1 | 7.6 | | |
| | MMLU-Redux | 5 | 27.8 | 28.2 | 32.1 | | |
| | MMLU-Pro | 5 | 7.8 | 8.5 | 25.5 | | |
| | GPQA | 5 | **26.6** | 25.9 | 29.9 | | |
| | BBH | 3 | 18.4 | 22.7 | 39.4 | | |
| | GSM8K | 8 | **4.0** | 1.8 | 48.9 | | |
| | MATH | 4 | 2.2 | 2.4 | 29.9 | | |
| | HumanEval | 0 | **9.1** | 2.4 | 29.3 | | |
| | EvalPlus | 0 | **4.3** | 1.2 | 13.0 | | |
| | INCLUDE (44 languages) | – | **26.0** | 25.5 | 36.3 | | |
| | MMMLU | 5 | 23.6 | 25.6 | 38.7 | | |
| | Pile test (BPB ↓) | – | 0.870 | 0.868 | 0.716 | | |
| **Reading the table:** | |
| - **Against Gemma-3-270M:** NanoI1 is ahead on code (HumanEval ~4×), grade-school maths, reading | |
| comprehension and OpenBookQA. It is behind on commonsense benchmarks (HellaSwag, PIQA) and factual recall | |
| (TriviaQA). It has fewer parameters in its transformer layers (78M vs ~100M) and saw roughly 1/6 of | |
| Gemma's training tokens. | |
| - **Against Qwen3-0.6B-Base:** Qwen3-0.6B, with about 4× the parameters and ~36T tokens, is stronger | |
| everywhere. | |
| - **Near chance:** many knowledge benchmarks (MMLU-Pro, GPQA, SuperGPQA, MMMLU) sit close to random guessing | |
| for all models under 300M parameters, so their differences are small. | |
| ### Instruct model | |
| | Benchmark | NanoI1-155M-Instruct | SmolLM2-135M-Instruct | SmolLM2-360M-Instruct | Qwen2.5-0.5B-Instruct | | |
| |---|---|---|---|---| | |
| | IFEval (prompt-level loose) | *[fill after `run_sft.sh eval`]* | | | | | |
| ## Intended use | |
| - **Good for:** structured extraction, classification, routing and short grounded Q&A over provided documents, | |
| especially after fine-tuning on your own task data. Also a fast draft model, on-device assistant or research | |
| baseline. | |
| - **Not for:** open-ended factual questions without context, maths and multi-step reasoning, legal, medical or | |
| financial advice, or any decision about a person without human review. | |
| ## Limitations | |
| - **It invents content.** At 155M parameters the model makes up facts and can add fields that are not in the | |
| source text when extracting JSON. For production extraction, use schema-constrained decoding (e.g. llguidance, | |
| Outlines or XGrammar) and validate outputs. | |
| - **Short context.** It was trained at 2,048 tokens; the RoPE base supports longer contexts but quality beyond | |
| 2k is untested. | |
| - **Limited Indian-language ability.** Indian languages are a small share of training, so quality there is | |
| well below English. | |
| - **Bias.** The model reflects biases present in web, book and code data. | |
| - **No safety training.** No preference or safety tuning has been applied yet. Add your own guardrails. | |
| ## Training data and licences | |
| All third-party datasets were used under their published terms: | |
| - **ODC-By:** FineWeb-Edu, FineMath, FineWeb-2, smollm-corpus (Cosmopedia v2), OpenCoder annealing corpus, | |
| Dolmino mix. | |
| - **Apache-2.0:** smol-smoltalk, Aya. | |
| - **The Stack v2:** per-file licences. | |
| - **Institutional Books 1.0:** Institutional Data Initiative terms. | |
| See each dataset card for details. Some anneal data (Cosmopedia, parts of FLAN and Dolmino math) was generated | |
| by other language models. | |
| ## Citation | |
| ```bibtex | |
| @techreport{kitefish2026nanoi1, | |
| title = {NanoI1: A 155M-Parameter Language Model Trained on One Trillion Tokens for Indian Regulated Industries}, | |
| author = {Anuj and {KiteFish AI}}, | |
| institution = {KiteFish AI}, | |
| year = {2026} | |
| } | |
| ``` | |