Update README.md

README.md

@@ -1,23 +1,27 @@
 ---
 language:
-- en
+  - en
+library_name: transformers
+pipeline_tag: text-generation
 license: apache-2.0
 tags:
-- novi-ai
-- language-model
-- pretrained-model
-- preview
-- small-language-model
-- bananamind-2-style
-- causal-lm
-- base-model
-- pretrained-from-scratch
+  - novi-ai
+  - language-model
+  - pretrained-model
+  - preview
+  - small-language-model
+  - bananamind-2-style
+  - causal-lm
+  - base-model
+  - pretrained-from-scratch
+  - project-kairo
+  - triton
 datasets:
-- HuggingFaceTB/smollm-corpus
-- mlfoundations/dclm-baseline-1.0
-- HuggingFaceTB/finemath
-- nvidia/OpenMathInstruct-2
-- HuggingFaceTB/smoltalk
+  - HuggingFaceTB/smollm-corpus
+  - mlfoundations/dclm-baseline-1.0
+  - HuggingFaceTB/finemath
+  - nvidia/OpenMathInstruct-2
+  - HuggingFaceTB/smoltalk
 ---
 
 # Novi Mini Base Preview 1
@@ -26,24 +30,31 @@ datasets:
 
 Novi Mini Base is a compact language model pretraining project featuring a **BananaMind 2-style architecture**, using a downscaled Cagliostro v3 design as its architectural basis.
 
-This repository contains a preview build from the ongoing pretraining run.
+This repository contains an intermediate checkpoint from the ongoing pretraining run.
 
 ## Model Information
 
 | Property | Details |
 |---|---|
-| Model name | Novi Mini Base |
+| Model name | Novi Mini Base (Preview 1) |
 | Organization | Novi AI |
 | Model type | Causal Language Model |
 | Architecture style | BananaMind 2-style |
 | Architectural basis | Downscaled Cagliostro v3 |
-| Parameters | Approximately 10M |
+| Parameters | **9,968,128 (~9.97M)** |
+| Hidden size | 256 |
+| Transformer layers | 10 |
+| Attention heads | 4 |
+| Key/value heads | 2 |
+| Head dimension | 64 |
+| Intermediate size | 768 |
 | Tokenizer | Byte-level BPE |
 | Vocabulary size | 8,192 |
-| Context length | 2,048 tokens |
-| Training objective | Next-token prediction |
-| Training tokens target | 15 billion |
-| Training schedule | Stable phase + cosine cooldown |
+| Context length | 2,048 tokens during training |
+| Training objective | Causal next-token prediction |
+| Training token target | 15 billion |
+| Training schedule | Warmup + stable phase + cosine cooldown |
+| Model class | `ProjectKairoMiniForCausalLM` |
 
 ## Training Progress
 
@@ -51,18 +62,232 @@ This preview corresponds to checkpoint **168000**.
 
 | Metric | Value |
 |---|---:|
-| Current checkpoint | 168,000 |
-| Total planned steps | 457,764 |
-| Overall progress | 36.70% |
-| Stable-phase progress | 43.18% |
+| Current checkpoint | **168,000** |
+| Total planned steps | **457,764** |
+| Overall progress | **36.70%** |
+| Stable-phase progress | **43.18%** |
+| Target training tokens | **15,000,000,000** |
+| Stable phase | 85% |
+| Cosine cooldown | 15% |
 
 Training is ongoing. The final model may differ from this preview.
 
 ## Architecture
 
-Novi Mini Base follows a **BananaMind 2-style architecture**.
+Novi Mini Base follows a **BananaMind 2-style decoder-only Transformer architecture**.
+
+Its architectural basis is a scaled-down Cagliostro v3 design, adapted for a compact configuration.
+
+### Architecture Components
+
+| Component | Implementation |
+|---|---|
+| Attention | Grouped-Query Attention (GQA) |
+| Position encoding | Rotary Position Embeddings (RoPE) |
+| Query normalization | RMSNorm |
+| Key normalization | RMSNorm |
+| Main normalization | RMSNorm |
+| Feed-forward network | SwiGLU |
+| Attention implementation | PyTorch Scaled Dot Product Attention |
+| Input embeddings | Learned token embeddings |
+| Output projection | Language-model head |
+| Embedding weights | Tied input/output embeddings |
+
+### Model Dimensions
+
+| Configuration | Value |
+|---|---:|
+| Vocabulary size | 8,192 |
+| Hidden size | 256 |
+| Transformer layers | 10 |
+| Query heads | 4 |
+| KV heads | 2 |
+| Head dimension | 64 |
+| Intermediate dimension | 768 |
+| RoPE theta | 100,000 |
+| RMSNorm epsilon | 0.000001 |
+
+## Triton Acceleration
+
+The training script includes optional custom Triton kernels for selected operations.
+
+The accelerated operations include:
+
+- **RMSNorm:** Fused normalization forward and backward operations.
+- **SwiGLU:** Fused activation and multiplication forward and backward operations.
+- **RoPE:** Rotary position embedding forward and backward operations.
+
+Attention continues to use PyTorch SDPA, while linear projections use the model's existing PyTorch layers.
+
+Triton is optional. If it is unavailable or disabled, the model falls back to its PyTorch implementations.
+
+Set `NOVI_DISABLE_TRITON=1` to explicitly disable the Triton kernels.
+
+## Training
+
+Novi Mini Base is pretrained from scratch using **BananaAll**.
+
+The training script configures a total target of **15 billion tokens**, divided into two training phases.
 
-Its architectural basis is a scaled-down Cagliostro v3 design, retaining the intended architectural approach while reducing model size for a compact training configuration.
+### Training Configuration
+
+| Setting | Value |
+|---|---:|
+| Training framework | BananaAll |
+| Training mode | Pretraining |
+| Training objective | Next-token prediction |
+| Target tokens | **15 billion** |
+| Sequence length | 2,048 |
+| Per-device batch size | 2 |
+| Gradient accumulation | 8 |
+| Effective batch size | 16 sequences on one GPU |
+| Learning rate | **0.003** |
+| Warmup steps | 2,000 |
+| Stable phase | 85% |
+| Cooldown phase | 15% |
+| Scheduler | Custom warmup-stable-cosine cooldown |
+| Weight decay | 0.01 |
+| Adam beta 1 | 0.9 |
+| Adam beta 2 | 0.95 |
+| Maximum gradient norm | 1.0 |
+| Precision | Automatic |
+| Torch compilation | Enabled, with fallback |
+| Random seed | 1337 |
+| Logging interval | 100 steps |
+| Checkpoint interval | 500 steps |
+
+The configured 457,764 optimizer steps assume the single-GPU batch configuration used for this run. The script calculates the planned step count from the token quotas, sequence length, batch size, and gradient accumulation.
+
+### Learning-Rate Schedule
+
+The training schedule has three stages:
+
+1. **Warmup:** The learning rate increases over the first 2,000 optimizer steps.
+2. **Stable:** The learning rate remains at its configured maximum through the stable training phase.
+3. **Cosine cooldown:** The learning rate gradually decreases during the final 15% of training.
+
+The stable phase and cooldown phase each have their own dataset mixture.
+
+## Training Data
+
+The training configuration uses five Hugging Face dataset repositories, with multiple configurations from some repositories.
+
+### Dataset Sources
+
+| Dataset | Configuration |
+|---|---|
+| `HuggingFaceTB/smollm-corpus` | `fineweb-edu-dedup` |
+| `HuggingFaceTB/smollm-corpus` | `cosmopedia-v2` |
+| `mlfoundations/dclm-baseline-1.0` | Default |
+| `HuggingFaceTB/finemath` | `finemath-3plus` |
+| `HuggingFaceTB/finemath` | `infiwebmath-3plus` |
+| `nvidia/OpenMathInstruct-2` | Default |
+| `HuggingFaceTB/smoltalk` | `all` |
+
+All configured sources use their `train` split.
+
+### Stable Phase Dataset Weights
+
+| Dataset configuration | Weight |
+|---|---:|
+| SmolLM Corpus — FineWeb-Edu Dedup | 43.7% |
+| DCLM Baseline 1.0 | 28.3% |
+| SmolLM Corpus — Cosmopedia v2 | 16.0% |
+| FineMath — FineMath 3+ | 5.0% |
+| OpenMathInstruct 2 | 3.0% |
+| FineMath — InfiWebMath 3+ | 2.0% |
+| SmolTalk | 2.0% |
+| **Total** | **100%** |
+
+### Cosine Cooldown Dataset Weights
+
+| Dataset configuration | Weight |
+|---|---:|
+| SmolLM Corpus — FineWeb-Edu Dedup | 37.0% |
+| DCLM Baseline 1.0 | 5.0% |
+| SmolLM Corpus — Cosmopedia v2 | 25.0% |
+| FineMath — FineMath 3+ | 15.0% |
+| OpenMathInstruct 2 | 13.0% |
+| SmolTalk | 5.0% |
+| **Total** | **100%** |
+
+The training worker streams dataset rows and samples from the configured mixture. Each phase receives its own token allocation, with the dataset weights determining the relative token quotas.
+
+The training script also records the resolved dataset token allocations in `dataset_tokens.json` when the run completes.
+
+## Tokenizer
+
+Novi Mini Base uses a byte-level BPE tokenizer trained from the configured pretraining dataset mixture.
+
+| Property | Value |
+|---|---|
+| Tokenizer algorithm | BPE |
+| Pre-tokenizer | ByteLevel |
+| Decoder | ByteLevel |
+| Vocabulary size | 8,192 |
+| Minimum token frequency | 2 |
+| Special tokens | `[PAD]`, `[BOS]`, `[EOS]`, `[UNK]` |
+| Tokenizer training samples | 10,000 |
+
+The tokenizer is trained separately before model initialization. The model vocabulary is then configured to match the tokenizer.
+
+## Checkpoints
+
+The training script automatically uploads checkpoints to Hugging Face every 500 optimizer steps.
+
+It also supports automatic recovery by locating the newest complete remote checkpoint and resuming training from its saved state.
+
+A complete recovery checkpoint includes model weights, optimizer state, scheduler state, trainer state, and random-number-generator state.
+
+**Checkpoint repository:**
+
+https://huggingface.co/Novi-AI/Novi-Mini-Base-Checkpoints
+
+## Loading the Model
+
+Novi Mini Base uses a custom Transformers architecture.
+
+The model repository includes custom configuration and modeling files. Loading may require `trust_remote_code=True`.
+
+Only enable remote model code after reviewing and trusting the repository's implementation.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "Novi-AI/Novi-Mini-Base-Preview-1"
+
+tokenizer = AutoTokenizer.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+
+model.eval()
+
+prompt = "The future of small language models is"
+
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=100,
+        do_sample=True,
+        temperature=0.8,
+        top_p=0.9,
+        pad_token_id=tokenizer.pad_token_id,
+        eos_token_id=tokenizer.eos_token_id,
+    )
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+This is a basic text-continuation example. Novi Mini Base is a pretrained base model, not an instruction-tuned chatbot.
 
 ## Intended Use
 
@@ -73,32 +298,38 @@ This is an experimental pretrained model intended for:
 - Small-model research.
 - Architecture evaluation.
 - Testing lightweight language models.
+- Studying compact autoregressive pretraining.
 
 This preview is not intended to be treated as a finished production model.
 
-## Loading the Model
+## Limitations
 
-The model uses a custom architecture. Loading it may require the corresponding Novi Mini Base modeling and configuration files.
+Novi Mini Base has approximately 9.97 million parameters, making it considerably smaller than most general-purpose language models.
 
-Use trusted model code only.
+It may:
 
-## Training
+- Produce repetitive or incoherent text.
+- Generate inaccurate or fabricated information.
+- Struggle with complex reasoning.
+- Have limited factual knowledge.
+- Fail to maintain consistency over longer sequences.
+- Perform poorly on instruction-following tasks.
 
-Novi Mini Base is being pretrained on a planned dataset mixture totaling approximately **15 billion training tokens**.
+The model is a base pretrained checkpoint, so it is not specifically optimized for conversational instruction following.
 
-The training schedule consists of:
+## Evaluation
 
-1. Stable training phase.
-2. Cosine learning-rate cooldown phase.
+No independent benchmark results are included in this preview.
 
-The final training target is 457,764 steps.
+Training progress and loss measurements should not be interpreted as benchmark scores or proof of model quality.
 
 ## Project
 
-Developed by **Novi AI**.
+Developed by **Novi AI** as part of **Project Kairo**.
 
-- Main model: Soon...
-- Checkpoints: https://huggingface.co/Novi-AI/Novi-Mini-Base-Checkpoints
+- Preview model: `Novi-AI/Novi-Mini-Base-Preview-1`
+- Main model: `Novi-AI/Novi-Mini-Base`
+- Checkpoints: `Novi-AI/Novi-Mini-Base-Checkpoints`
 
 ## Disclaimer
 
@@ -106,4 +337,6 @@ This repository contains an intermediate training preview. Performance, capabili
 
 The model is experimental and may produce inaccurate, incomplete, or nonsensical outputs.
 
-*Novi AI 2026 — Project Kairo*
+---
+
+*Novi AI 2026 — Project Kairo*