Novi-Micro-Base

Novi-Micro-Base is a tiny causal language model trained from scratch by Novi-AI.

With approximately 5.04 million parameters, Novi-Micro explores language modeling at a small scale while using a substantially larger context window and training corpus than earlier Novi models.

⚡ 5.04M parameters · 1B training tokens · 2,048-token context

Model Details

Architecture

Novi-Micro-Base uses a custom BananaMind 2-style decoder architecture with RMSNorm, Rotary Position Embeddings (RoPE), grouped-query attention, QK normalization, SwiGLU feed-forward layers, and tied input/output embeddings.

Property	Value
Model type	Causal Language Model
Architecture	BananaMind 2-style
Parameters	5.04M
Vocabulary size	16,384
Context length	2,048 tokens
Attention	Grouped-Query Attention (GQA)
Position encoding	RoPE
Normalization	RMSNorm
Feed-forward	SwiGLU
QK normalization	Enabled
Embeddings	Tied
Training precision	FP32

Training

Novi-Micro-Base was trained from scratch for approximately 1 billion tokens.

The training configuration used a 5M parameter target, a sequence length of 2,048 tokens, and a single dataset source.

Training Configuration

Setting	Value
Training mode	Pretraining
Target size	5M parameters
Actual parameters	5.04M
Training tokens	1,000,000,000
Sequence length	2,048
Batch size	2
Gradient accumulation	8
Effective batch size	16 sequences
Learning rate	2 × 10⁻⁴
Scheduler	Cosine
Warmup ratio	3%
Weight decay	0.01
Maximum gradient norm	1.0
Optimizer steps	30,518
Precision	FP32
Torch compile	Enabled
Random seed	1337
Checkpoint interval	100 steps
Logging interval	10 steps

The model was trained until the selected 1,000,000,000-token target was consumed.

Dataset

Novi-Micro-Base was trained using:

• FineWeb-HQ

The training configuration allocated a token target of exactly:

1,000,000,000 tokens

Documents were streamed from the dataset and packed into contiguous 2,048-token sequences before being passed to the model.

Tokenizer

Novi-Micro uses a custom byte-level BPE tokenizer with a vocabulary size of 16,384 tokens.

The tokenizer was trained from samples drawn from the training dataset before model pretraining.

Special tokens include:

• [PAD]
• [BOS]
• [EOS]
• [UNK]

The tokenizer uses ByteLevel pre-tokenization and decoding.

Architecture Details

Novi-Micro uses a decoder-only Transformer architecture.

Attention

The model uses grouped-query attention (GQA), where multiple query heads share key and value heads.

Rotary Position Embeddings (RoPE) are applied to the query and key representations.

The BananaMind 2-style architecture also applies RMSNorm to the query and key head dimensions.

Feed-Forward Network

Each Transformer block uses a SwiGLU feed-forward network:

SwiGLU(x) = SiLU(gate(x)) × up(x)

The result is projected back to the model's hidden dimension.

Normalization

The model uses RMSNorm before both the attention and feed-forward sublayers.

Embeddings

The input token embeddings and language-model output embeddings are tied, reducing the number of independent parameters.

Intended Use

Novi-Micro-Base is primarily intended for:

• 🔬 Research and experimentation
• 🧪 Small-model language-model experiments
• 🎓 Educational purposes
• 🛠️ Fine-tuning experiments
• 💻 Lightweight local inference
• 🤖 Exploring language modeling at the million-parameter scale

As a base model, Novi-Micro-Base is not instruction-tuned and is not specifically trained to follow user commands or behave as a conversational assistant.

Limitations

Novi-Micro-Base is a very small experimental language model.

With approximately 5 million parameters, it is dramatically smaller than modern general-purpose language models and should not be expected to match their capabilities.

It may:

• Generate incoherent text
• Repeat phrases
• Produce factual errors
• Struggle with complex instructions
• Have limited world knowledge
• Perform poorly on difficult reasoning tasks
• Produce grammatically unusual text
• Fail to maintain coherent long-form generations

A 2,048-token context window does not eliminate the limitations caused by the model's small parameter count.

This model should be considered a research and experimentation model, rather than a production-ready general-purpose LLM.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Novi-AI/Novi-Micro-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

prompt = "Hello, my name is"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Because Novi-Micro uses a custom architecture, trust_remote_code=True is required when loading the model through Transformers.

Project History

Novi-Micro-Base is part of Project Kairo, the development codename for the Novi model project.

The Novi series follows the earlier AppleMind experiments and represents the primary model-development line of Novi-AI.

AppleMind → Novi-Nano → Novi-Micro → future Novi models 🚀

Novi-Micro substantially expands upon Novi-Nano by increasing the model to approximately 5.04M parameters, expanding the context window from 256 to 2,048 tokens, and training on approximately 1 billion tokens.

Training Infrastructure

The model was trained using GPU compute through the BananaAll training environment.

The training worker supports GPU acceleration through CUDA and Intel XPU, with Torch compilation enabled for supported environments.

This model was trained from scratch rather than fine-tuned from an existing language model checkpoint.

Acknowledgements

Novi-Micro was built using the open-source machine-learning ecosystem and datasets made available by the community.

Special thanks to:

• Hugging Face 🤗
• FineWeb
• FineWeb-HQ
• The open-source Transformers ecosystem

License

This model is released under the Apache 2.0 license.

---

🧠 Novi AI

Small models. Big experiments.

Novi-Micro is intentionally tiny — exploring how far a language model can go with only a few million parameters and approximately one billion training tokens.

Novi AI 2026 — Project Kairo