README.md

---

language: 
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- novi
- novi-micro
- causal-lm
- from-scratch
- bananaall

---

# Novi-Micro-Base

![Novi-Micro Banner](banner.jpg)

**Novi-Micro-Base** is a tiny causal language model trained from scratch by **Novi-AI**.

With approximately **5.04 million parameters**, Novi-Micro explores language modeling at a small scale while using a substantially larger context window and training corpus than earlier Novi models.

⚡ **5.04M parameters · 1B training tokens · 2,048-token context**

## Model Details

### Architecture

Novi-Micro-Base uses a custom **BananaMind 2-style decoder architecture** with RMSNorm, Rotary Position Embeddings (RoPE), grouped-query attention, QK normalization, SwiGLU feed-forward layers, and tied input/output embeddings.

| Property           |                             Value |
| ------------------ | --------------------------------: |
| Model type         |             Causal Language Model |
| Architecture       |            **BananaMind 2-style** |
| Parameters         |                         **5.04M** |
| Vocabulary size    |                        **16,384** |
| Context length     |                  **2,048 tokens** |
| Attention          | **Grouped-Query Attention (GQA)** |
| Position encoding  |                          **RoPE** |
| Normalization      |                       **RMSNorm** |
| Feed-forward       |                        **SwiGLU** |
| QK normalization   |                       **Enabled** |
| Embeddings         |                          **Tied** |
| Training precision |                          **FP32** |

## Training

Novi-Micro-Base was trained from scratch for approximately **1 billion tokens**.

The training configuration used a **5M parameter target**, a sequence length of **2,048 tokens**, and a single dataset source.

### Training Configuration

| Setting               |             Value |
| --------------------- | ----------------: |
| Training mode         |   **Pretraining** |
| Target size           | **5M parameters** |
| Actual parameters     |         **5.04M** |
| Training tokens       | **1,000,000,000** |
| Sequence length       |         **2,048** |
| Batch size            |             **2** |
| Gradient accumulation |             **8** |
| Effective batch size  |  **16 sequences** |
| Learning rate         |      **2 × 10⁻⁴** |
| Scheduler             |        **Cosine** |
| Warmup ratio          |            **3%** |
| Weight decay          |          **0.01** |
| Maximum gradient norm |           **1.0** |
| Optimizer steps       |        **30,518** |
| Precision             |          **FP32** |
| Torch compile         |       **Enabled** |
| Random seed           |          **1337** |
| Checkpoint interval   |     **100 steps** |
| Logging interval      |      **10 steps** |

The model was trained until the selected **1,000,000,000-token target** was consumed.

## Dataset

Novi-Micro-Base was trained using:

* **FineWeb-HQ**

The training configuration allocated a token target of exactly:

**1,000,000,000 tokens**

Documents were streamed from the dataset and packed into contiguous **2,048-token sequences** before being passed to the model.

## Tokenizer

Novi-Micro uses a custom **byte-level BPE tokenizer** with a vocabulary size of **16,384 tokens**.

The tokenizer was trained from samples drawn from the training dataset before model pretraining.

Special tokens include:

* `[PAD]`
* `[BOS]`
* `[EOS]`
* `[UNK]`

The tokenizer uses ByteLevel pre-tokenization and decoding.

## Architecture Details

Novi-Micro uses a decoder-only Transformer architecture.

### Attention

The model uses **grouped-query attention (GQA)**, where multiple query heads share key and value heads.

Rotary Position Embeddings (**RoPE**) are applied to the query and key representations.

The BananaMind 2-style architecture also applies **RMSNorm to the query and key head dimensions**.

### Feed-Forward Network

Each Transformer block uses a **SwiGLU** feed-forward network:

```text
SwiGLU(x) = SiLU(gate(x)) × up(x)
```

The result is projected back to the model's hidden dimension.

### Normalization

The model uses **RMSNorm** before both the attention and feed-forward sublayers.

### Embeddings

The input token embeddings and language-model output embeddings are **tied**, reducing the number of independent parameters.

## Intended Use

Novi-Micro-Base is primarily intended for:

* 🔬 Research and experimentation
* 🧪 Small-model language-model experiments
* 🎓 Educational purposes
* 🛠️ Fine-tuning experiments
* 💻 Lightweight local inference
* 🤖 Exploring language modeling at the million-parameter scale

As a **base model**, Novi-Micro-Base is not instruction-tuned and is not specifically trained to follow user commands or behave as a conversational assistant.

## Limitations

Novi-Micro-Base is a very small experimental language model.

With approximately **5 million parameters**, it is dramatically smaller than modern general-purpose language models and should not be expected to match their capabilities.

It may:

* Generate incoherent text
* Repeat phrases
* Produce factual errors
* Struggle with complex instructions
* Have limited world knowledge
* Perform poorly on difficult reasoning tasks
* Produce grammatically unusual text
* Fail to maintain coherent long-form generations

A 2,048-token context window does not eliminate the limitations caused by the model's small parameter count.

This model should be considered a **research and experimentation model**, rather than a production-ready general-purpose LLM.

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Novi-AI/Novi-Micro-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

prompt = "Hello, my name is"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

Because Novi-Micro uses a custom architecture, `trust_remote_code=True` is required when loading the model through Transformers.

## Project History

Novi-Micro-Base is part of **Project Kairo**, the development codename for the Novi model project.

The Novi series follows the earlier **AppleMind** experiments and represents the primary model-development line of Novi-AI.

**AppleMind → Novi-Nano → Novi-Micro → future Novi models** 🚀

Novi-Micro substantially expands upon Novi-Nano by increasing the model to approximately **5.04M parameters**, expanding the context window from 256 to **2,048 tokens**, and training on approximately **1 billion tokens**.

## Training Infrastructure

The model was trained using GPU compute through the **BananaAll** training environment.

The training worker supports GPU acceleration through CUDA and Intel XPU, with Torch compilation enabled for supported environments.

This model was trained from scratch rather than fine-tuned from an existing language model checkpoint.

## Acknowledgements

Novi-Micro was built using the open-source machine-learning ecosystem and datasets made available by the community.

Special thanks to:

* Hugging Face 🤗
* FineWeb
* FineWeb-HQ
* The open-source Transformers ecosystem

## License

This model is released under the **Apache 2.0** license.

---

## 🧠 Novi AI

**Small models. Big experiments.**

Novi-Micro is intentionally tiny — exploring how far a language model can go with only a few million parameters and approximately one billion training tokens.

*Novi AI 2026 — Project Kairo*