Novi-AI/Novi-Mini-Base-Preview-1

Novi Mini Base Preview 1

An early preview of Novi Mini Base by Novi AI.

Novi Mini Base is a compact language model pretraining project featuring a BananaMind 2-style architecture, using a downscaled Cagliostro v3 design as its architectural basis.

This repository contains an intermediate checkpoint from the ongoing pretraining run.

Model Information

Property	Details
Model name	Novi Mini Base (Preview 1)
Organization	Novi AI
Model type	Causal Language Model
Architecture style	BananaMind 2-style
Architectural basis	Downscaled Cagliostro v3
Parameters	9,968,128 (~9.97M)
Hidden size	256
Transformer layers	10
Attention heads	4
Key/value heads	2
Head dimension	64
Intermediate size	768
Tokenizer	Byte-level BPE
Vocabulary size	8,192
Context length	2,048 tokens during training
Training objective	Causal next-token prediction
Training token target	15 billion
Training schedule	Warmup + stable phase + cosine cooldown
Model class	ProjectKairoMiniForCausalLM

Training Progress

This preview corresponds to checkpoint 168000.

Metric	Value
Current checkpoint	168,000
Total planned steps	457,764
Overall progress	36.70%
Stable-phase progress	43.18%
Target training tokens	15,000,000,000
Stable phase	85%
Cosine cooldown	15%

Training is ongoing. The final model may differ from this preview.

Architecture

Novi Mini Base follows a BananaMind 2-style decoder-only Transformer architecture.

Its architectural basis is a scaled-down Cagliostro v3 design, adapted for a compact configuration.

Architecture Components

Component	Implementation
Attention	Grouped-Query Attention (GQA)
Position encoding	Rotary Position Embeddings (RoPE)
Query normalization	RMSNorm
Key normalization	RMSNorm
Main normalization	RMSNorm
Feed-forward network	SwiGLU
Attention implementation	PyTorch Scaled Dot Product Attention
Input embeddings	Learned token embeddings
Output projection	Language-model head
Embedding weights	Tied input/output embeddings

Model Dimensions

Configuration	Value
Vocabulary size	8,192
Hidden size	256
Transformer layers	10
Query heads	4
KV heads	2
Head dimension	64
Intermediate dimension	768
RoPE theta	100,000
RMSNorm epsilon	0.000001

Triton Acceleration

The training script includes optional custom Triton kernels for selected operations.

The accelerated operations include:

• RMSNorm: Fused normalization forward and backward operations.
• SwiGLU: Fused activation and multiplication forward and backward operations.
• RoPE: Rotary position embedding forward and backward operations.

Attention continues to use PyTorch SDPA, while linear projections use the model's existing PyTorch layers.

Triton is optional. If it is unavailable or disabled, the model falls back to its PyTorch implementations.

Set NOVI_DISABLE_TRITON=1 to explicitly disable the Triton kernels.

Training

Novi Mini Base is pretrained from scratch using BananaAll.

The training script configures a total target of 15 billion tokens, divided into two training phases.

Training Configuration

Setting	Value
Training framework	BananaAll
Training mode	Pretraining
Training objective	Next-token prediction
Target tokens	15 billion
Sequence length	2,048
Per-device batch size	2
Gradient accumulation	8
Effective batch size	16 sequences on one GPU
Learning rate	0.003
Warmup steps	2,000
Stable phase	85%
Cooldown phase	15%
Scheduler	Custom warmup-stable-cosine cooldown
Weight decay	0.01
Adam beta 1	0.9
Adam beta 2	0.95
Maximum gradient norm	1.0
Precision	Automatic
Torch compilation	Enabled, with fallback
Random seed	1337
Logging interval	100 steps
Checkpoint interval	500 steps

The configured 457,764 optimizer steps assume the single-GPU batch configuration used for this run. The script calculates the planned step count from the token quotas, sequence length, batch size, and gradient accumulation.

Learning-Rate Schedule

The training schedule has three stages:

1. Warmup: The learning rate increases over the first 2,000 optimizer steps.
2. Stable: The learning rate remains at its configured maximum through the stable training phase.
3. Cosine cooldown: The learning rate gradually decreases during the final 15% of training.

The stable phase and cooldown phase each have their own dataset mixture.

Training Data

The training configuration uses five Hugging Face dataset repositories, with multiple configurations from some repositories.

Dataset Sources

Dataset	Configuration
HuggingFaceTB/smollm-corpus	fineweb-edu-dedup
HuggingFaceTB/smollm-corpus	cosmopedia-v2
mlfoundations/dclm-baseline-1.0	Default
HuggingFaceTB/finemath	finemath-3plus
HuggingFaceTB/finemath	infiwebmath-3plus
nvidia/OpenMathInstruct-2	Default
HuggingFaceTB/smoltalk	all

All configured sources use their train split.

Stable Phase Dataset Weights

Dataset configuration	Weight
SmolLM Corpus — FineWeb-Edu Dedup	43.7%
DCLM Baseline 1.0	28.3%
SmolLM Corpus — Cosmopedia v2	16.0%
FineMath — FineMath 3+	5.0%
OpenMathInstruct 2	3.0%
FineMath — InfiWebMath 3+	2.0%
SmolTalk	2.0%
Total	100%

Cosine Cooldown Dataset Weights

Dataset configuration	Weight
SmolLM Corpus — FineWeb-Edu Dedup	37.0%
DCLM Baseline 1.0	5.0%
SmolLM Corpus — Cosmopedia v2	25.0%
FineMath — FineMath 3+	15.0%
OpenMathInstruct 2	13.0%
SmolTalk	5.0%
Total	100%

The training worker streams dataset rows and samples from the configured mixture. Each phase receives its own token allocation, with the dataset weights determining the relative token quotas.

The training script also records the resolved dataset token allocations in dataset_tokens.json when the run completes.

Tokenizer

Novi Mini Base uses a byte-level BPE tokenizer trained from the configured pretraining dataset mixture.

Property	Value
Tokenizer algorithm	BPE
Pre-tokenizer	ByteLevel
Decoder	ByteLevel
Vocabulary size	8,192
Minimum token frequency	2
Special tokens	[PAD], [BOS], [EOS], [UNK]
Tokenizer training samples	10,000

The tokenizer is trained separately before model initialization. The model vocabulary is then configured to match the tokenizer.

Checkpoints

The training script automatically uploads checkpoints to Hugging Face every 500 optimizer steps.

It also supports automatic recovery by locating the newest complete remote checkpoint and resuming training from its saved state.

A complete recovery checkpoint includes model weights, optimizer state, scheduler state, trainer state, and random-number-generator state.

Checkpoint repository:

https://huggingface.co/Novi-AI/Novi-Mini-Base-Checkpoints

Loading the Model

Novi Mini Base uses a custom Transformers architecture.

The model repository includes custom configuration and modeling files. Loading may require trust_remote_code=True.

Only enable remote model code after reviewing and trusting the repository's implementation.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Novi-AI/Novi-Mini-Base-Preview-1"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model.eval()

prompt = "The future of small language models is"

inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

This is a basic text-continuation example. Novi Mini Base is a pretrained base model, not an instruction-tuned chatbot.

Intended Use

This is an experimental pretrained model intended for:

• Language-model experimentation.
• Fine-tuning experiments.
• Small-model research.
• Architecture evaluation.
• Testing lightweight language models.
• Studying compact autoregressive pretraining.

This preview is not intended to be treated as a finished production model.

Limitations

Novi Mini Base has approximately 9.97 million parameters, making it considerably smaller than most general-purpose language models.

It may:

• Produce repetitive or incoherent text.
• Generate inaccurate or fabricated information.
• Struggle with complex reasoning.
• Have limited factual knowledge.
• Fail to maintain consistency over longer sequences.
• Perform poorly on instruction-following tasks.

The model is a base pretrained checkpoint, so it is not specifically optimized for conversational instruction following.

Evaluation

No independent benchmark results are included in this preview.

Training progress and loss measurements should not be interpreted as benchmark scores or proof of model quality.

Project

Developed by Novi AI as part of Project Kairo.

• Preview model: Novi-AI/Novi-Mini-Base-Preview-1
• Main model: Novi-AI/Novi-Mini-Base
• Checkpoints: Novi-AI/Novi-Mini-Base-Checkpoints

Disclaimer

This repository contains an intermediate training preview. Performance, capabilities, and output quality are not representative of the final model.

The model is experimental and may produce inaccurate, incomplete, or nonsensical outputs.

---

Novi AI 2026 — Project Kairo