Create README.md

README.md

@@ -0,0 +1,342 @@
+---
+language:
+  - en
+library_name: transformers
+pipeline_tag: text-generation
+license: apache-2.0
+tags:
+  - novi-ai
+  - language-model
+  - pretrained-model
+  - preview
+  - small-language-model
+  - bananamind-2-style
+  - causal-lm
+  - base-model
+  - pretrained-from-scratch
+  - project-kairo
+  - triton
+datasets:
+  - HuggingFaceTB/smollm-corpus
+  - mlfoundations/dclm-baseline-1.0
+  - HuggingFaceTB/finemath
+  - nvidia/OpenMathInstruct-2
+  - HuggingFaceTB/smoltalk
+---
+
+# Novi Mini Base Preview 2
+
+**A second preview of Novi Mini Base by Novi AI.**
+
+Novi Mini Base is a compact language model pretraining project featuring a **BananaMind 2-style architecture**, using a downscaled Cagliostro v3 design as its architectural basis.
+
+This repository contains an intermediate checkpoint from the ongoing pretraining run.
+
+## Model Information
+
+| Property | Details |
+|---|---|
+| Model name | Novi Mini Base (Preview 2) |
+| Organization | Novi AI |
+| Model type | Causal Language Model |
+| Architecture style | BananaMind 2-style |
+| Architectural basis | Downscaled Cagliostro v3 |
+| Parameters | **9,968,128 (~9.97M)** |
+| Hidden size | 256 |
+| Transformer layers | 10 |
+| Attention heads | 4 |
+| Key/value heads | 2 |
+| Head dimension | 64 |
+| Intermediate size | 768 |
+| Tokenizer | Byte-level BPE |
+| Vocabulary size | 8,192 |
+| Context length | 2,048 tokens during training |
+| Training objective | Causal next-token prediction |
+| Training token target | 15 billion |
+| Training schedule | Warmup + stable phase + cosine cooldown |
+| Model class | `ProjectKairoMiniForCausalLM` |
+
+## Training Progress
+
+This preview corresponds to checkpoint **321000**.
+
+| Metric | Value |
+|---|---:|
+| Current checkpoint | **321,000** |
+| Total planned steps | **457,764** |
+| Overall progress | **70.12%** |
+| Stable-phase progress | **82.50%** |
+| Target training tokens | **15,000,000,000** |
+| Stable phase | 85% |
+| Cosine cooldown | 15% |
+
+Training is ongoing. The final model may differ from this preview.
+
+## Architecture
+
+Novi Mini Base follows a **BananaMind 2-style decoder-only Transformer architecture**.
+
+Its architectural basis is a scaled-down Cagliostro v3 design, adapted for a compact configuration.
+
+### Architecture Components
+
+| Component | Implementation |
+|---|---|
+| Attention | Grouped-Query Attention (GQA) |
+| Position encoding | Rotary Position Embeddings (RoPE) |
+| Query normalization | RMSNorm |
+| Key normalization | RMSNorm |
+| Main normalization | RMSNorm |
+| Feed-forward network | SwiGLU |
+| Attention implementation | PyTorch Scaled Dot Product Attention |
+| Input embeddings | Learned token embeddings |
+| Output projection | Language-model head |
+| Embedding weights | Tied input/output embeddings |
+
+### Model Dimensions
+
+| Configuration | Value |
+|---|---:|
+| Vocabulary size | 8,192 |
+| Hidden size | 256 |
+| Transformer layers | 10 |
+| Query heads | 4 |
+| KV heads | 2 |
+| Head dimension | 64 |
+| Intermediate dimension | 768 |
+| RoPE theta | 100,000 |
+| RMSNorm epsilon | 0.000001 |
+
+## Triton Acceleration
+
+The training script includes optional custom Triton kernels for selected operations.
+
+The accelerated operations include:
+
+- **RMSNorm:** Fused normalization forward and backward operations.
+- **SwiGLU:** Fused activation and multiplication forward and backward operations.
+- **RoPE:** Rotary position embedding forward and backward operations.
+
+Attention continues to use PyTorch SDPA, while linear projections use the model's existing PyTorch layers.
+
+Triton is optional. If it is unavailable or disabled, the model falls back to its PyTorch implementations.
+
+Set `NOVI_DISABLE_TRITON=1` to explicitly disable the Triton kernels.
+
+## Training
+
+Novi Mini Base is pretrained from scratch using **BananaAll**.
+
+The training script configures a total target of **15 billion tokens**, divided into two training phases.
+
+### Training Configuration
+
+| Setting | Value |
+|---|---:|
+| Training framework | BananaAll |
+| Training mode | Pretraining |
+| Training objective | Next-token prediction |
+| Target tokens | **15 billion** |
+| Sequence length | 2,048 |
+| Per-device batch size | 2 |
+| Gradient accumulation | 8 |
+| Effective batch size | 16 sequences on one GPU |
+| Learning rate | **0.003** |
+| Warmup steps | 2,000 |
+| Stable phase | 85% |
+| Cooldown phase | 15% |
+| Scheduler | Custom warmup-stable-cosine cooldown |
+| Weight decay | 0.01 |
+| Adam beta 1 | 0.9 |
+| Adam beta 2 | 0.95 |
+| Maximum gradient norm | 1.0 |
+| Precision | Automatic |
+| Torch compilation | Enabled, with fallback |
+| Random seed | 1337 |
+| Logging interval | 100 steps |
+| Checkpoint interval | 500 steps |
+
+The configured 457,764 optimizer steps assume the single-GPU batch configuration used for this run. The script calculates the planned step count from the token quotas, sequence length, batch size, and gradient accumulation.
+
+### Learning-Rate Schedule
+
+The training schedule has three stages:
+
+1. **Warmup:** The learning rate increases over the first 2,000 optimizer steps.
+2. **Stable:** The learning rate remains at its configured maximum through the stable training phase.
+3. **Cosine cooldown:** The learning rate gradually decreases during the final 15% of training.
+
+The stable phase and cooldown phase each have their own dataset mixture.
+
+## Training Data
+
+The training configuration uses five Hugging Face dataset repositories, with multiple configurations from some repositories.
+
+### Dataset Sources
+
+| Dataset | Configuration |
+|---|---|
+| `HuggingFaceTB/smollm-corpus` | `fineweb-edu-dedup` |
+| `HuggingFaceTB/smollm-corpus` | `cosmopedia-v2` |
+| `mlfoundations/dclm-baseline-1.0` | Default |
+| `HuggingFaceTB/finemath` | `finemath-3plus` |
+| `HuggingFaceTB/finemath` | `infiwebmath-3plus` |
+| `nvidia/OpenMathInstruct-2` | Default |
+| `HuggingFaceTB/smoltalk` | `all` |
+
+All configured sources use their `train` split.
+
+### Stable Phase Dataset Weights
+
+| Dataset configuration | Weight |
+|---|---:|
+| SmolLM Corpus — FineWeb-Edu Dedup | 43.7% |
+| DCLM Baseline 1.0 | 28.3% |
+| SmolLM Corpus — Cosmopedia v2 | 16.0% |
+| FineMath — FineMath 3+ | 5.0% |
+| OpenMathInstruct 2 | 3.0% |
+| FineMath — InfiWebMath 3+ | 2.0% |
+| SmolTalk | 2.0% |
+| **Total** | **100%** |
+
+### Cosine Cooldown Dataset Weights
+
+| Dataset configuration | Weight |
+|---|---:|
+| SmolLM Corpus — FineWeb-Edu Dedup | 37.0% |
+| DCLM Baseline 1.0 | 5.0% |
+| SmolLM Corpus — Cosmopedia v2 | 25.0% |
+| FineMath — FineMath 3+ | 15.0% |
+| OpenMathInstruct 2 | 13.0% |
+| SmolTalk | 5.0% |
+| **Total** | **100%** |
+
+The training worker streams dataset rows and samples from the configured mixture. Each phase receives its own token allocation, with the dataset weights determining the relative token quotas.
+
+The training script also records the resolved dataset token allocations in `dataset_tokens.json` when the run completes.
+
+## Tokenizer
+
+Novi Mini Base uses a byte-level BPE tokenizer trained from the configured pretraining dataset mixture.
+
+| Property | Value |
+|---|---|
+| Tokenizer algorithm | BPE |
+| Pre-tokenizer | ByteLevel |
+| Decoder | ByteLevel |
+| Vocabulary size | 8,192 |
+| Minimum token frequency | 2 |
+| Special tokens | `[PAD]`, `[BOS]`, `[EOS]`, `[UNK]` |
+| Tokenizer training samples | 10,000 |
+
+The tokenizer is trained separately before model initialization. The model vocabulary is then configured to match the tokenizer.
+
+## Checkpoints
+
+The training script automatically uploads checkpoints to Hugging Face every 500 optimizer steps.
+
+It also supports automatic recovery by locating the newest complete remote checkpoint and resuming training from its saved state.
+
+A complete recovery checkpoint includes model weights, optimizer state, scheduler state, trainer state, and random-number-generator state.
+
+**Checkpoint repository:**
+
+https://huggingface.co/Novi-AI/Novi-Mini-Base-Checkpoints
+
+## Loading the Model
+
+Novi Mini Base uses a custom Transformers architecture.
+
+The model repository includes custom configuration and modeling files. Loading may require `trust_remote_code=True`.
+
+Only enable remote model code after reviewing and trusting the repository's implementation.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "Novi-AI/Novi-Mini-Base-Preview-2"
+
+tokenizer = AutoTokenizer.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+)
+
+model.eval()
+
+prompt = "The future of small language models is"
+
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=100,
+        do_sample=True,
+        temperature=0.8,
+        top_p=0.9,
+        pad_token_id=tokenizer.pad_token_id,
+        eos_token_id=tokenizer.eos_token_id,
+    )
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+This is a basic text-continuation example. Novi Mini Base is a pretrained base model, not an instruction-tuned chatbot.
+
+## Intended Use
+
+This is an experimental pretrained model intended for:
+
+- Language-model experimentation.
+- Fine-tuning experiments.
+- Small-model research.
+- Architecture evaluation.
+- Testing lightweight language models.
+- Studying compact autoregressive pretraining.
+
+This preview is not intended to be treated as a finished production model.
+
+## Limitations
+
+Novi Mini Base has approximately 9.97 million parameters, making it considerably smaller than most general-purpose language models.
+
+It may:
+
+- Produce repetitive or incoherent text.
+- Generate inaccurate or fabricated information.
+- Struggle with complex reasoning.
+- Have limited factual knowledge.
+- Fail to maintain consistency over longer sequences.
+- Perform poorly on instruction-following tasks.
+
+The model is a base pretrained checkpoint, so it is not specifically optimized for conversational instruction following.
+
+## Evaluation
+
+No independent benchmark results are included in this preview.
+
+Training progress and loss measurements should not be interpreted as benchmark scores or proof of model quality.
+
+## Project
+
+Developed by **Novi AI** as part of **Project Kairo**.
+
+- Preview model: `Novi-AI/Novi-Mini-Base-Preview-2`
+- Main model: `Novi-AI/Novi-Mini-Base`
+- Checkpoints: `Novi-AI/Novi-Mini-Base-Checkpoints`
+
+## Disclaimer
+
+This repository contains an intermediate training preview. Performance, capabilities, and output quality are not representative of the final model.
+
+The model is experimental and may produce inaccurate, incomplete, or nonsensical outputs.
+
+---
+
+*Novi AI 2026 — Project Kairo*