Blame for README.md

---
language:
  - en
library_name: transformers
pipeline_tag: text-generation
license: apache-2.0
tags:
  - novi-ai
  - language-model
  - pretrained-model
  - preview
  - small-language-model
  - bananamind-2-style
  - causal-lm
  - base-model
  - pretrained-from-scratch
  - project-kairo
  - triton
datasets:
  - HuggingFaceTB/smollm-corpus
  - mlfoundations/dclm-baseline-1.0
  - HuggingFaceTB/finemath
  - nvidia/OpenMathInstruct-2
  - HuggingFaceTB/smoltalk
---

# Novi Mini Base Preview 1

**An early preview of Novi Mini Base by Novi AI.**

Novi Mini Base is a compact language model pretraining project featuring a **BananaMind 2-style architecture**, using a downscaled Cagliostro v3 design as its architectural basis.

This repository contains an intermediate checkpoint from the ongoing pretraining run.

## Model Information

| Property | Details |
|---|---|
| Model name | Novi Mini Base (Preview 1) |
| Organization | Novi AI |
| Model type | Causal Language Model |
| Architecture style | BananaMind 2-style |
| Architectural basis | Downscaled Cagliostro v3 |
| Parameters | **9,968,128 (~9.97M)** |
| Hidden size | 256 |
| Transformer layers | 10 |
| Attention heads | 4 |
| Key/value heads | 2 |
| Head dimension | 64 |
| Intermediate size | 768 |
| Tokenizer | Byte-level BPE |
| Vocabulary size | 8,192 |
| Context length | 2,048 tokens during training |
| Training objective | Causal next-token prediction |
| Training token target | 15 billion |
| Training schedule | Warmup + stable phase + cosine cooldown |
| Model class | `ProjectKairoMiniForCausalLM` |

## Training Progress

This preview corresponds to checkpoint **168000**.

| Metric | Value |
|---|---:|
| Current checkpoint | **168,000** |
| Total planned steps | **457,764** |
| Overall progress | **36.70%** |
| Stable-phase progress | **43.18%** |
| Target training tokens | **15,000,000,000** |
| Stable phase | 85% |
| Cosine cooldown | 15% |

Training is ongoing. The final model may differ from this preview.

## Architecture

Novi Mini Base follows a **BananaMind 2-style decoder-only Transformer architecture**.

Its architectural basis is a scaled-down Cagliostro v3 design, adapted for a compact configuration.

### Architecture Components

| Component | Implementation |
|---|---|
| Attention | Grouped-Query Attention (GQA) |
| Position encoding | Rotary Position Embeddings (RoPE) |
| Query normalization | RMSNorm |
| Key normalization | RMSNorm |
| Main normalization | RMSNorm |
| Feed-forward network | SwiGLU |
| Attention implementation | PyTorch Scaled Dot Product Attention |
| Input embeddings | Learned token embeddings |
| Output projection | Language-model head |
| Embedding weights | Tied input/output embeddings |

### Model Dimensions

| Configuration | Value |
|---|---:|
| Vocabulary size | 8,192 |
| Hidden size | 256 |
| Transformer layers | 10 |
| Query heads | 4 |
| KV heads | 2 |
| Head dimension | 64 |
| Intermediate dimension | 768 |
| RoPE theta | 100,000 |
| RMSNorm epsilon | 0.000001 |

## Triton Acceleration

The training script includes optional custom Triton kernels for selected operations.

The accelerated operations include:

- **RMSNorm:** Fused normalization forward and backward operations.
- **SwiGLU:** Fused activation and multiplication forward and backward operations.
- **RoPE:** Rotary position embedding forward and backward operations.

Attention continues to use PyTorch SDPA, while linear projections use the model's existing PyTorch layers.

Triton is optional. If it is unavailable or disabled, the model falls back to its PyTorch implementations.

Set `NOVI_DISABLE_TRITON=1` to explicitly disable the Triton kernels.

## Training

Novi Mini Base is pretrained from scratch using **BananaAll**.

The training script configures a total target of **15 billion tokens**, divided into two training phases.

### Training Configuration

| Setting | Value |
|---|---:|
| Training framework | BananaAll |
| Training mode | Pretraining |
| Training objective | Next-token prediction |
| Target tokens | **15 billion** |
| Sequence length | 2,048 |
| Per-device batch size | 2 |
| Gradient accumulation | 8 |
| Effective batch size | 16 sequences on one GPU |
| Learning rate | **0.003** |
| Warmup steps | 2,000 |
| Stable phase | 85% |
| Cooldown phase | 15% |
| Scheduler | Custom warmup-stable-cosine cooldown |
| Weight decay | 0.01 |
| Adam beta 1 | 0.9 |
| Adam beta 2 | 0.95 |
| Maximum gradient norm | 1.0 |
| Precision | Automatic |
| Torch compilation | Enabled, with fallback |
| Random seed | 1337 |
| Logging interval | 100 steps |
| Checkpoint interval | 500 steps |

The configured 457,764 optimizer steps assume the single-GPU batch configuration used for this run. The script calculates the planned step count from the token quotas, sequence length, batch size, and gradient accumulation.

### Learning-Rate Schedule

The training schedule has three stages:

1. **Warmup:** The learning rate increases over the first 2,000 optimizer steps.
2. **Stable:** The learning rate remains at its configured maximum through the stable training phase.
3. **Cosine cooldown:** The learning rate gradually decreases during the final 15% of training.

The stable phase and cooldown phase each have their own dataset mixture.

## Training Data

The training configuration uses five Hugging Face dataset repositories, with multiple configurations from some repositories.

### Dataset Sources

| Dataset | Configuration |
|---|---|
| `HuggingFaceTB/smollm-corpus` | `fineweb-edu-dedup` |
| `HuggingFaceTB/smollm-corpus` | `cosmopedia-v2` |
| `mlfoundations/dclm-baseline-1.0` | Default |
| `HuggingFaceTB/finemath` | `finemath-3plus` |
| `HuggingFaceTB/finemath` | `infiwebmath-3plus` |
| `nvidia/OpenMathInstruct-2` | Default |
| `HuggingFaceTB/smoltalk` | `all` |

All configured sources use their `train` split.

### Stable Phase Dataset Weights

| Dataset configuration | Weight |
|---|---:|
| SmolLM Corpus — FineWeb-Edu Dedup | 43.7% |
| DCLM Baseline 1.0 | 28.3% |
| SmolLM Corpus — Cosmopedia v2 | 16.0% |
| FineMath — FineMath 3+ | 5.0% |
| OpenMathInstruct 2 | 3.0% |
| FineMath — InfiWebMath 3+ | 2.0% |
| SmolTalk | 2.0% |
| **Total** | **100%** |

### Cosine Cooldown Dataset Weights

| Dataset configuration | Weight |
|---|---:|
| SmolLM Corpus — FineWeb-Edu Dedup | 37.0% |
| DCLM Baseline 1.0 | 5.0% |
| SmolLM Corpus — Cosmopedia v2 | 25.0% |
| FineMath — FineMath 3+ | 15.0% |
| OpenMathInstruct 2 | 13.0% |
| SmolTalk | 5.0% |
| **Total** | **100%** |

The training worker streams dataset rows and samples from the configured mixture. Each phase receives its own token allocation, with the dataset weights determining the relative token quotas.

The training script also records the resolved dataset token allocations in `dataset_tokens.json` when the run completes.

## Tokenizer

Novi Mini Base uses a byte-level BPE tokenizer trained from the configured pretraining dataset mixture.

| Property | Value |
|---|---|
| Tokenizer algorithm | BPE |
| Pre-tokenizer | ByteLevel |
| Decoder | ByteLevel |
| Vocabulary size | 8,192 |
| Minimum token frequency | 2 |
| Special tokens | `[PAD]`, `[BOS]`, `[EOS]`, `[UNK]` |
| Tokenizer training samples | 10,000 |

The tokenizer is trained separately before model initialization. The model vocabulary is then configured to match the tokenizer.

## Checkpoints

The training script automatically uploads checkpoints to Hugging Face every 500 optimizer steps.

It also supports automatic recovery by locating the newest complete remote checkpoint and resuming training from its saved state.

A complete recovery checkpoint includes model weights, optimizer state, scheduler state, trainer state, and random-number-generator state.

**Checkpoint repository:**

https://huggingface.co/Novi-AI/Novi-Mini-Base-Checkpoints

## Loading the Model

Novi Mini Base uses a custom Transformers architecture.

The model repository includes custom configuration and modeling files. Loading may require `trust_remote_code=True`.

Only enable remote model code after reviewing and trusting the repository's implementation.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Novi-AI/Novi-Mini-Base-Preview-1"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model.eval()

prompt = "The future of small language models is"

inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

This is a basic text-continuation example. Novi Mini Base is a pretrained base model, not an instruction-tuned chatbot.

## Intended Use

This is an experimental pretrained model intended for:

- Language-model experimentation.
- Fine-tuning experiments.
- Small-model research.
- Architecture evaluation.
- Testing lightweight language models.
- Studying compact autoregressive pretraining.

This preview is not intended to be treated as a finished production model.

## Limitations

Novi Mini Base has approximately 9.97 million parameters, making it considerably smaller than most general-purpose language models.

It may:

- Produce repetitive or incoherent text.
- Generate inaccurate or fabricated information.
- Struggle with complex reasoning.
- Have limited factual knowledge.
- Fail to maintain consistency over longer sequences.
- Perform poorly on instruction-following tasks.

The model is a base pretrained checkpoint, so it is not specifically optimized for conversational instruction following.

## Evaluation

No independent benchmark results are included in this preview.

Training progress and loss measurements should not be interpreted as benchmark scores or proof of model quality.

## Project

Developed by **Novi AI** as part of **Project Kairo**.

- Preview model: `Novi-AI/Novi-Mini-Base-Preview-1`
- Main model: `Novi-AI/Novi-Mini-Base`
- Checkpoints: `Novi-AI/Novi-Mini-Base-Checkpoints`

## Disclaimer

This repository contains an intermediate training preview. Performance, capabilities, and output quality are not representative of the final model.

The model is experimental and may produce inaccurate, incomplete, or nonsensical outputs.

---

*Novi AI 2026 — Project Kairo*