README.md

---
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- novi
- novi-nano
- novi-nano-instruct
- causal-lm
- gpt2
- from-scratch
- instruction-tuning
- chatml
datasets:
- Novi-AI/Novi-510x
---

# Novi-Nano-Instruct

![Novi-Nano Banner](banner.jpg)

**Novi-Nano-Instruct** is a tiny instruction-tuned causal language model developed by **Novi-AI**.

It is based on **Novi-Nano-Base** and fine-tuned on a small instruction dataset to experiment with instruction following and conversational behavior at an extremely small scale.

⚡ **1.26M parameters · 500 training examples · 256-token context**

## Model Details

### Architecture

| Property        |                    Value |
| --------------- | -----------------------: |
| Model type      |    Causal Language Model |
| Base model      | `Novi-AI/Novi-Nano-Base` |
| Parameters      |            **1,258,848** |
| Vocabulary size |                **8,195** |
| Context length  |                  **256** |
| Embedding size  |                   **96** |
| Layers          |                    **4** |
| Attention heads |                    **4** |
| FFN size        |                  **384** |
| Tensor type     |                  **F32** |

## Instruction Tuning

Novi-Nano-Instruct was trained from **Novi-Nano-Base** using a small instruction dataset containing **510 examples**.

### Dataset

| Split      | Examples |
| ---------- | -------: |
| Training   |  **500** |
| Validation |   **10** |

The model uses a ChatML-style format with:

```text
<|im_start|>
<|im_end|>
```

Training loss was applied specifically to the assistant responses, allowing the model to focus on learning how to respond to user instructions.

### Training Configuration

| Property                |    Value |
| ----------------------- | -------: |
| Epochs                  |    **5** |
| Batch size              |   **16** |
| Gradient accumulation   |    **2** |
| Effective batch size    |   **32** |
| Maximum sequence length |  **256** |
| Learning rate           | **2e-5** |
| Precision               | **FP32** |
| Device                  |  **CPU** |

## Training Statistics

The final training run produced:

| Metric                      |          Result |
| --------------------------- | --------------: |
| Final validation loss       |    **5.153667** |
| Final validation perplexity |    **173.0650** |
| Training examples           |         **500** |
| Validation examples         |          **10** |
| Training time               | **~32 seconds** |

Because the validation set contains only **10 examples**, these metrics should be considered experimental rather than a comprehensive benchmark.

## Tokenizer

Novi-Nano-Instruct uses the custom tokenizer developed for Novi-Nano.

The original tokenizer vocabulary was **8,192 tokens**, with additional tokens already present in the tokenizer.

Two ChatML tokens were added for instruction tuning:

* `<|im_start|>` — **8193**
* `<|im_end|>` — **8194**

The final tokenizer size is **8,195 tokens**.

The tokenizer was originally trained using data from:

* FineWeb-Edu
* FineWeb-HQ
* SmolLM-Cosmopedia

## Intended Use

Novi-Nano-Instruct is primarily intended for:

* 🔬 Research and experimentation
* 🧪 Small-model instruction-tuning experiments
* 🎓 Educational purposes
* 💬 Tiny conversational-model experiments
* 💻 Lightweight local inference
* 🛠️ Experimenting with extremely small instruction-tuned models

As an **experimental 1.26M-parameter model**, it is not intended to compete with modern billion-parameter language models.

## Limitations

Novi-Nano-Instruct is an extremely small experimental language model trained on only **500 instruction examples**.

Because of its size and limited training data, it may:

* Generate incoherent text
* Repeat phrases
* Produce unrelated responses
* Fail to follow instructions
* Produce factual errors
* Have very limited world knowledge
* Perform poorly on reasoning tasks
* Struggle with longer conversations
* Lose context beyond its 256-token window
* Produce malformed or unexpected responses

Generation quality is currently **highly experimental**. The model can generate text, but it does not yet consistently produce reliable assistant-style responses.

This model should be considered a **research and experimentation model**, rather than a production-ready conversational AI.

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Novi-AI/Novi-Nano-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [
    {
        "role": "system",
        "content": "You are Novi-Nano, a helpful AI assistant."
    },
    {
        "role": "user",
        "content": "Give a synonym for 'quiet'."
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Chat Template

Novi-Nano-Instruct uses a ChatML-style conversation format:

```text
<|im_start|>system
You are Novi-Nano, a helpful AI assistant.<|im_end|>
<|im_start|>user
Give a synonym for 'quiet'.<|im_end|>
<|im_start|>assistant
A synonym is 'silent'.<|im_end|>
```

For generation, the assistant message is opened automatically by the chat template.

## Project History

Novi AI follows the earlier **AppleMind** experiments, with Novi becoming the primary project for developing small language models.

**AppleMind → Novi AI → Novi-Nano → Novi-Nano-Instruct** 🚀

## Acknowledgements

Novi-Nano was built using the open-source machine-learning ecosystem and datasets made available by the community.

Special thanks to:

* Hugging Face 🤗
* FineWeb
* SmolLM
* Cosmopedia

## License

This model is released under the **Apache 2.0** license.

---

## 🧠 Novi AI

**Small models. Big experiments.**

Novi-Nano-Instruct explores instruction tuning at an extremely small scale, with just **1.26 million parameters** and **500 training examples**.

It is intentionally tiny — exploring how far instruction following can go with a fraction of the parameters used by modern LLMs.

*Novi AI 2026 — Project Kairo*