| --- |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - novi |
| - novi-nano |
| - causal-lm |
| - gpt2 |
| - from-scratch |
| --- |
| |
| # Novi-Nano-Base |
|
|
|  |
|
|
| **Novi-Nano-Base** is a tiny causal language model trained from scratch by **Novi-AI**. |
|
|
| With just **1,258,560 parameters**, Novi-Nano explores language modeling at an extremely small scale while remaining compatible with the Hugging Face Transformers ecosystem. |
|
|
| ⚡ **1.26M parameters · 300M training tokens · 256-token context** |
|
|
| ## Model Details |
|
|
| ### Architecture |
|
|
| | Property | Value | |
| | --------------- | --------------------: | |
| | Model type | Causal Language Model | |
| | Parameters | **1,258,560** | |
| | Vocabulary size | **8,192** | |
| | Context length | **256** | |
| | Embedding size | **96** | |
| | Layers | **4** | |
| | Attention heads | **4** | |
| | FFN size | **384** | |
| | Tensor type | **F32** | |
|
|
| ## Training |
|
|
| Novi-Nano-Base was trained from scratch using approximately **300 million training tokens**. |
|
|
| ### Training Statistics |
|
|
| | Metric | Result | |
| | --------------------------- | --------------: | |
| | Training tokens | **300,023,808** | |
| | Best validation loss | **5.418699** | |
| | Final validation loss | **5.418699** | |
| | Final validation perplexity | **225.5853** | |
|
|
| ## Tokenizer |
|
|
| Novi-Nano uses a custom tokenizer with a vocabulary size of **8,192 tokens**. |
|
|
| The tokenizer was trained using data from: |
|
|
| * FineWeb-Edu |
| * FineWeb-HQ |
| * SmolLM-Cosmopedia |
|
|
| ## Intended Use |
|
|
| Novi-Nano-Base is primarily intended for: |
|
|
| * 🔬 Research and experimentation |
| * 🧪 Small-model language-model experiments |
| * 🎓 Educational purposes |
| * 🛠️ Fine-tuning experiments |
| * 💻 Lightweight local inference |
|
|
| As a **base model**, it is not specifically instruction-tuned for following user commands or acting as a conversational assistant. |
|
|
| ## Limitations |
|
|
| Novi-Nano-Base is an extremely small experimental language model. |
|
|
| Because of its size and short context window, it will have significant limitations compared with modern billion-parameter language models. |
|
|
| It may: |
|
|
| * Generate incoherent text |
| * Repeat phrases |
| * Produce factual errors |
| * Struggle with complex instructions |
| * Have limited world knowledge |
| * Perform poorly on reasoning tasks |
| * Lose context beyond its 256-token window |
|
|
| This model should be considered a **research and experimentation model**, rather than a production-ready general-purpose LLM. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| model_id = "Novi-AI/Novi-Nano-Base" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id) |
| |
| prompt = "Hello, my name is" |
| |
| inputs = tokenizer(prompt, return_tensors="pt") |
| |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=50, |
| ) |
| |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Project History |
|
|
| Novi AI follows the earlier **AppleMind** experiments, with Novi becoming the primary project for developing small language models. |
|
|
| **AppleMind → Novi AI → Novi-Nano** 🚀 |
|
|
| ## Acknowledgements |
|
|
| Novi-Nano was built using the open-source machine-learning ecosystem and datasets made available by the community. |
|
|
| Special thanks to: |
|
|
| * Hugging Face 🤗 |
| * FineWeb |
| * SmolLM |
| * Cosmopedia |
|
|
| ## License |
|
|
| This model is released under the **Apache 2.0** license. |
|
|
| --- |
|
|
| ## 🧠 Novi AI |
|
|
| **Small models. Big experiments.** |
|
|
| Novi-Nano is intentionally tiny — exploring how far a language model can go with just a fraction of the parameters used by modern LLMs. |
|
|
| *Novi AI 2026 — Project Kairo* |
|
|
