| --- |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| license: apache-2.0 |
| tags: |
| - novi-ai |
| - language-model |
| - pretrained-model |
| - preview |
| - small-language-model |
| - bananamind-2-style |
| - causal-lm |
| - base-model |
| - pretrained-from-scratch |
| - project-kairo |
| - triton |
| datasets: |
| - HuggingFaceTB/smollm-corpus |
| - mlfoundations/dclm-baseline-1.0 |
| - HuggingFaceTB/finemath |
| - nvidia/OpenMathInstruct-2 |
| - HuggingFaceTB/smoltalk |
| --- |
| |
| # Novi Mini Base Preview 1 |
|
|
| **An early preview of Novi Mini Base by Novi AI.** |
|
|
| Novi Mini Base is a compact language model pretraining project featuring a **BananaMind 2-style architecture**, using a downscaled Cagliostro v3 design as its architectural basis. |
|
|
| This repository contains an intermediate checkpoint from the ongoing pretraining run. |
|
|
| ## Model Information |
|
|
| | Property | Details | |
| |---|---| |
| | Model name | Novi Mini Base (Preview 1) | |
| | Organization | Novi AI | |
| | Model type | Causal Language Model | |
| | Architecture style | BananaMind 2-style | |
| | Architectural basis | Downscaled Cagliostro v3 | |
| | Parameters | **9,968,128 (~9.97M)** | |
| | Hidden size | 256 | |
| | Transformer layers | 10 | |
| | Attention heads | 4 | |
| | Key/value heads | 2 | |
| | Head dimension | 64 | |
| | Intermediate size | 768 | |
| | Tokenizer | Byte-level BPE | |
| | Vocabulary size | 8,192 | |
| | Context length | 2,048 tokens during training | |
| | Training objective | Causal next-token prediction | |
| | Training token target | 15 billion | |
| | Training schedule | Warmup + stable phase + cosine cooldown | |
| | Model class | `ProjectKairoMiniForCausalLM` | |
|
|
| ## Training Progress |
|
|
| This preview corresponds to checkpoint **168000**. |
|
|
| | Metric | Value | |
| |---|---:| |
| | Current checkpoint | **168,000** | |
| | Total planned steps | **457,764** | |
| | Overall progress | **36.70%** | |
| | Stable-phase progress | **43.18%** | |
| | Target training tokens | **15,000,000,000** | |
| | Stable phase | 85% | |
| | Cosine cooldown | 15% | |
|
|
| Training is ongoing. The final model may differ from this preview. |
|
|
| ## Architecture |
|
|
| Novi Mini Base follows a **BananaMind 2-style decoder-only Transformer architecture**. |
|
|
| Its architectural basis is a scaled-down Cagliostro v3 design, adapted for a compact configuration. |
|
|
| ### Architecture Components |
|
|
| | Component | Implementation | |
| |---|---| |
| | Attention | Grouped-Query Attention (GQA) | |
| | Position encoding | Rotary Position Embeddings (RoPE) | |
| | Query normalization | RMSNorm | |
| | Key normalization | RMSNorm | |
| | Main normalization | RMSNorm | |
| | Feed-forward network | SwiGLU | |
| | Attention implementation | PyTorch Scaled Dot Product Attention | |
| | Input embeddings | Learned token embeddings | |
| | Output projection | Language-model head | |
| | Embedding weights | Tied input/output embeddings | |
|
|
| ### Model Dimensions |
|
|
| | Configuration | Value | |
| |---|---:| |
| | Vocabulary size | 8,192 | |
| | Hidden size | 256 | |
| | Transformer layers | 10 | |
| | Query heads | 4 | |
| | KV heads | 2 | |
| | Head dimension | 64 | |
| | Intermediate dimension | 768 | |
| | RoPE theta | 100,000 | |
| | RMSNorm epsilon | 0.000001 | |
|
|
| ## Triton Acceleration |
|
|
| The training script includes optional custom Triton kernels for selected operations. |
|
|
| The accelerated operations include: |
|
|
| - **RMSNorm:** Fused normalization forward and backward operations. |
| - **SwiGLU:** Fused activation and multiplication forward and backward operations. |
| - **RoPE:** Rotary position embedding forward and backward operations. |
|
|
| Attention continues to use PyTorch SDPA, while linear projections use the model's existing PyTorch layers. |
|
|
| Triton is optional. If it is unavailable or disabled, the model falls back to its PyTorch implementations. |
|
|
| Set `NOVI_DISABLE_TRITON=1` to explicitly disable the Triton kernels. |
|
|
| ## Training |
|
|
| Novi Mini Base is pretrained from scratch using **BananaAll**. |
|
|
| The training script configures a total target of **15 billion tokens**, divided into two training phases. |
|
|
| ### Training Configuration |
|
|
| | Setting | Value | |
| |---|---:| |
| | Training framework | BananaAll | |
| | Training mode | Pretraining | |
| | Training objective | Next-token prediction | |
| | Target tokens | **15 billion** | |
| | Sequence length | 2,048 | |
| | Per-device batch size | 2 | |
| | Gradient accumulation | 8 | |
| | Effective batch size | 16 sequences on one GPU | |
| | Learning rate | **0.003** | |
| | Warmup steps | 2,000 | |
| | Stable phase | 85% | |
| | Cooldown phase | 15% | |
| | Scheduler | Custom warmup-stable-cosine cooldown | |
| | Weight decay | 0.01 | |
| | Adam beta 1 | 0.9 | |
| | Adam beta 2 | 0.95 | |
| | Maximum gradient norm | 1.0 | |
| | Precision | Automatic | |
| | Torch compilation | Enabled, with fallback | |
| | Random seed | 1337 | |
| | Logging interval | 100 steps | |
| | Checkpoint interval | 500 steps | |
|
|
| The configured 457,764 optimizer steps assume the single-GPU batch configuration used for this run. The script calculates the planned step count from the token quotas, sequence length, batch size, and gradient accumulation. |
|
|
| ### Learning-Rate Schedule |
|
|
| The training schedule has three stages: |
|
|
| 1. **Warmup:** The learning rate increases over the first 2,000 optimizer steps. |
| 2. **Stable:** The learning rate remains at its configured maximum through the stable training phase. |
| 3. **Cosine cooldown:** The learning rate gradually decreases during the final 15% of training. |
|
|
| The stable phase and cooldown phase each have their own dataset mixture. |
|
|
| ## Training Data |
|
|
| The training configuration uses five Hugging Face dataset repositories, with multiple configurations from some repositories. |
|
|
| ### Dataset Sources |
|
|
| | Dataset | Configuration | |
| |---|---| |
| | `HuggingFaceTB/smollm-corpus` | `fineweb-edu-dedup` | |
| | `HuggingFaceTB/smollm-corpus` | `cosmopedia-v2` | |
| | `mlfoundations/dclm-baseline-1.0` | Default | |
| | `HuggingFaceTB/finemath` | `finemath-3plus` | |
| | `HuggingFaceTB/finemath` | `infiwebmath-3plus` | |
| | `nvidia/OpenMathInstruct-2` | Default | |
| | `HuggingFaceTB/smoltalk` | `all` | |
|
|
| All configured sources use their `train` split. |
|
|
| ### Stable Phase Dataset Weights |
|
|
| | Dataset configuration | Weight | |
| |---|---:| |
| | SmolLM Corpus — FineWeb-Edu Dedup | 43.7% | |
| | DCLM Baseline 1.0 | 28.3% | |
| | SmolLM Corpus — Cosmopedia v2 | 16.0% | |
| | FineMath — FineMath 3+ | 5.0% | |
| | OpenMathInstruct 2 | 3.0% | |
| | FineMath — InfiWebMath 3+ | 2.0% | |
| | SmolTalk | 2.0% | |
| | **Total** | **100%** | |
|
|
| ### Cosine Cooldown Dataset Weights |
|
|
| | Dataset configuration | Weight | |
| |---|---:| |
| | SmolLM Corpus — FineWeb-Edu Dedup | 37.0% | |
| | DCLM Baseline 1.0 | 5.0% | |
| | SmolLM Corpus — Cosmopedia v2 | 25.0% | |
| | FineMath — FineMath 3+ | 15.0% | |
| | OpenMathInstruct 2 | 13.0% | |
| | SmolTalk | 5.0% | |
| | **Total** | **100%** | |
|
|
| The training worker streams dataset rows and samples from the configured mixture. Each phase receives its own token allocation, with the dataset weights determining the relative token quotas. |
|
|
| The training script also records the resolved dataset token allocations in `dataset_tokens.json` when the run completes. |
|
|
| ## Tokenizer |
|
|
| Novi Mini Base uses a byte-level BPE tokenizer trained from the configured pretraining dataset mixture. |
|
|
| | Property | Value | |
| |---|---| |
| | Tokenizer algorithm | BPE | |
| | Pre-tokenizer | ByteLevel | |
| | Decoder | ByteLevel | |
| | Vocabulary size | 8,192 | |
| | Minimum token frequency | 2 | |
| | Special tokens | `[PAD]`, `[BOS]`, `[EOS]`, `[UNK]` | |
| | Tokenizer training samples | 10,000 | |
|
|
| The tokenizer is trained separately before model initialization. The model vocabulary is then configured to match the tokenizer. |
|
|
| ## Checkpoints |
|
|
| The training script automatically uploads checkpoints to Hugging Face every 500 optimizer steps. |
|
|
| It also supports automatic recovery by locating the newest complete remote checkpoint and resuming training from its saved state. |
|
|
| A complete recovery checkpoint includes model weights, optimizer state, scheduler state, trainer state, and random-number-generator state. |
|
|
| **Checkpoint repository:** |
|
|
| https://huggingface.co/Novi-AI/Novi-Mini-Base-Checkpoints |
|
|
| ## Loading the Model |
|
|
| Novi Mini Base uses a custom Transformers architecture. |
|
|
| The model repository includes custom configuration and modeling files. Loading may require `trust_remote_code=True`. |
|
|
| Only enable remote model code after reviewing and trusting the repository's implementation. |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "Novi-AI/Novi-Mini-Base-Preview-1" |
| |
| tokenizer = AutoTokenizer.from_pretrained( |
| model_id, |
| trust_remote_code=True, |
| ) |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| trust_remote_code=True, |
| ) |
| |
| model.eval() |
| |
| prompt = "The future of small language models is" |
| |
| inputs = tokenizer(prompt, return_tensors="pt") |
| |
| with torch.no_grad(): |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=100, |
| do_sample=True, |
| temperature=0.8, |
| top_p=0.9, |
| pad_token_id=tokenizer.pad_token_id, |
| eos_token_id=tokenizer.eos_token_id, |
| ) |
| |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| This is a basic text-continuation example. Novi Mini Base is a pretrained base model, not an instruction-tuned chatbot. |
|
|
| ## Intended Use |
|
|
| This is an experimental pretrained model intended for: |
|
|
| - Language-model experimentation. |
| - Fine-tuning experiments. |
| - Small-model research. |
| - Architecture evaluation. |
| - Testing lightweight language models. |
| - Studying compact autoregressive pretraining. |
|
|
| This preview is not intended to be treated as a finished production model. |
|
|
| ## Limitations |
|
|
| Novi Mini Base has approximately 9.97 million parameters, making it considerably smaller than most general-purpose language models. |
|
|
| It may: |
|
|
| - Produce repetitive or incoherent text. |
| - Generate inaccurate or fabricated information. |
| - Struggle with complex reasoning. |
| - Have limited factual knowledge. |
| - Fail to maintain consistency over longer sequences. |
| - Perform poorly on instruction-following tasks. |
|
|
| The model is a base pretrained checkpoint, so it is not specifically optimized for conversational instruction following. |
|
|
| ## Evaluation |
|
|
| No independent benchmark results are included in this preview. |
|
|
| Training progress and loss measurements should not be interpreted as benchmark scores or proof of model quality. |
|
|
| ## Project |
|
|
| Developed by **Novi AI** as part of **Project Kairo**. |
|
|
| - Preview model: `Novi-AI/Novi-Mini-Base-Preview-1` |
| - Main model: `Novi-AI/Novi-Mini-Base` |
| - Checkpoints: `Novi-AI/Novi-Mini-Base-Checkpoints` |
|
|
| ## Disclaimer |
|
|
| This repository contains an intermediate training preview. Performance, capabilities, and output quality are not representative of the final model. |
|
|
| The model is experimental and may produce inaccurate, incomplete, or nonsensical outputs. |
|
|
| --- |
|
|
| *Novi AI 2026 — Project Kairo* |
