KiteFishAI/NanoI1-155M-Base

NanoI1-155M (base)

NanoI1 is a 155M-parameter decoder-only language model trained from scratch by KiteFish AI on ~1.0T tokens. It is built for low-cost, on-premise deployment in Indian regulated industries (banking, insurance, financial services). Its main uses are structured extraction, classification and grounded Q&A, where data residency and inference cost matter more than open-ended general knowledge.

This repository holds the base (pretrained) model, best used as a starting point for fine-tuning. For chat and instruction following use KiteFishAI/NanoI1-155M-Instruct.

Parameters	155.3M total: 78.0M transformer layers, 38.6M input embedding, 38.7M output projection
Architecture	Decoder-only Transformer, Multi-Head Latent Attention (MLA), SwiGLU, RMSNorm, RoPE
Training tokens	~998B (pretraining 629B + extension 319B + anneal 50B)
Context length	2,048 tokens (trained)
Tokenizer	Byte-level BPE, 50,304 tokens, trained on our own mixture
Languages	English; Hindi and 8 other Indian languages (limited)
Licence	Apache-2.0

Quick start

The model uses a custom architecture, so loading needs the Python files in this repository (no transformers class yet).

pip install torch safetensors tokenizers huggingface_hub

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("KiteFishAI/NanoI1-155M")
sys.path.insert(0, path)
import torch
from modeling_nanoi1 import from_pretrained
from chat import generate

model, tok = from_pretrained(path)
ids = [tok.token_to_id("<|endoftext|>")] + tok.encode("The Reserve Bank of India regulates", add_special_tokens=False).ids
out = generate(model, ids, max_new_tokens=40, temperature=0)       # greedy continuation
print(tok.decode(out))

To fine-tune, load the weights the same way and train with your own loop, or use sft.py from the training repository with the original checkpoint.

Architecture

Hyper-parameter	Value
Layers	12
Hidden size	768
Attention heads	12 (head dim 64)
KV latent dimension (MLA)	256
FFN	SwiGLU, 2,048 hidden
Normalisation	Pre-norm RMSNorm (ε = 1e-6), QK-norm
Position encoding	RoPE, θ = 5,000,000
Embeddings	Untied input / output
Vocabulary	50,304 (byte-level BPE, 32 special tokens)

Multi-Head Latent Attention caches one 256-d latent vector per token per layer instead of full keys and values (2 × 768), so the KV cache is 6× smaller than standard multi-head attention at the same width.

Training

Stage	Tokens	Data	LR schedule
1. Pretraining	629B (steps 0–600k)	Our 798B-token corpus, each token at most once: English web, code, peS2o papers, multilingual web, Project Gutenberg, Proof-Pile math, English Wikipedia	linear warmup, then constant 6e-4
2. Extension	319B (600k–904k)	Unseen remainder of the stage-1 corpus (169B) + FineWeb-Edu (60B), The Stack v2 code (50B), Institutional Books 1.0 English (25B), FineMath (15B)	constant 6e-4
3. Anneal	50B (904k–952k)	High-quality mix: FineWeb-Edu, Cosmopedia, FineMath, Dolmino math & FLAN, OpenCoder annealing corpus, StackExchange, FineWeb-2 Indic, plus ~35% replay of stage-1 sources	linear decay 6e-4 → 0

• Benchmark decontamination: every document added in stages 2–3 that shares a 13-word sequence with the test set of any benchmark below was removed.
• Optimiser: AdamW (β = 0.9/0.95, weight decay 0.1, gradient clip 1.0), bf16 mixed precision.
• Batch: global batch of 1,048,576 tokens (512 × 2,048).
• Hardware: 8× NVIDIA H100 NVL, ~1.45M tokens/s.
• Data loading: a deterministic exactly-once sampler, so every training window is used once per pass and restarts resume to the exact sample.

Evaluation (base model)

Base checkpoint (stage 3), evaluated with lm-evaluation-harness 0.4.13 at 2,048 context. All models used the same prompts, shots and greedy decoding. Scores in %, Pile in bits per byte (lower is better).

Benchmark	Shots	NanoI1-155M	Gemma-3-270M	Qwen3-0.6B-Base
WinoGrande	5	52.6	54.0	59.9
ARC-Easy	25	58.9	59.0	74.9
ARC-Challenge	25	28.1	30.0	45.1
OpenBookQA	0	32.2	30.6	34.6
RACE-High	5	33.1	31.4	37.3
TriviaQA	5	10.2	14.3	20.1
NaturalQuestions	5	3.2	4.1	7.6
MMLU-Redux	5	27.8	28.2	32.1
MMLU-Pro	5	7.8	8.5	25.5
GPQA	5	26.6	25.9	29.9
BBH	3	18.4	22.7	39.4
GSM8K	8	4.0	1.8	48.9
MATH	4	2.2	2.4	29.9
HumanEval	0	9.1	2.4	29.3
EvalPlus	0	4.3	1.2	13.0
INCLUDE (44 languages)	–	26.0	25.5	36.3
MMMLU	5	23.6	25.6	38.7
Pile test (BPB ↓)	–	0.870	0.868	0.716

Reading the table:

• Against Gemma-3-270M: NanoI1 is ahead on code (HumanEval ~4×), grade-school maths, reading comprehension and OpenBookQA. It is behind on commonsense benchmarks (HellaSwag, PIQA) and factual recall (TriviaQA). It has fewer parameters in its transformer layers (78M vs ~100M) and saw roughly 1/6 of Gemma's training tokens.
• Against Qwen3-0.6B-Base: Qwen3-0.6B, with about 4× the parameters and ~36T tokens, is stronger everywhere.
• Near chance: many knowledge benchmarks (MMLU-Pro, GPQA, SuperGPQA, MMMLU) sit close to random guessing for all models under 300M parameters, so their differences are small.

Intended use

• Good for (after fine-tuning): structured extraction, classification, routing and short grounded Q&A over provided documents, especially after fine-tuning on your own task data. Also a fast draft model, on-device assistant or research baseline.
• Not for: open-ended factual questions without context, maths and multi-step reasoning, legal, medical or financial advice, or any decision about a person without human review.

Limitations

• It invents content. At 155M parameters the model makes up facts and can add fields that are not in the source text when extracting JSON. For production extraction, use schema-constrained decoding (e.g. llguidance, Outlines or XGrammar) and validate outputs.
• Short context. It was trained at 2,048 tokens; the RoPE base supports longer contexts but quality beyond 2k is untested.
• Limited Indian-language ability. Indian languages are a small share of training, so quality there is well below English.
• Bias. The model reflects biases present in web, book and code data.
• No safety training. This base model has no instruction, preference or safety tuning. Add your own guardrails.

Training data and licences

All third-party datasets were used under their published terms:

• ODC-By: FineWeb-Edu, FineMath, FineWeb-2, smollm-corpus (Cosmopedia v2), OpenCoder annealing corpus, Dolmino mix.
• The Stack v2: per-file licences.
• Institutional Books 1.0: Institutional Data Initiative terms.

See each dataset card for details. Some anneal data (Cosmopedia, parts of FLAN and Dolmino math) was generated by other language models.

Citation

@techreport{kitefish2026nanoi1,
  title       = {NanoI1: A 155M-Parameter Language Model Trained on One Trillion Tokens for Indian Regulated Industries},
  author      = {Anuj and {KiteFish AI}},
  institution = {KiteFish AI},
  year        = {2026}
}