Nirca Mini, checkpoint 1217308k (step 62,895), as latest

README.md

@@ -0,0 +1,57 @@
+---
+language:
+- en
+tags:
+- ternary
+- mixture-of-experts
+- byte-level
+- chatml
+- webgpu
+datasets:
+- HuggingFaceTB/smoltalk
+- open-thoughts/OpenThoughts-114k
+---
+
+# Nirca Mini
+
+Nirca Mini is a small chat model trained from scratch in a natively ternary format. Its
+linear layers' weights are -1, 0 or +1, with a scale and a bias for each block of 128
+weights. The norms and a few small layers stay in floating point. It was trained
+with quantization-aware training from its first step, so it was never quantized after
+training.
+
+## Files
+
+- `latest/`: the newest published checkpoint. It is replaced as new ones are published.
+- `checkpoints/<tokens>k/`: each published checkpoint, kept. The folder is named by the
+  number of training tokens it has seen, in thousands.
+
+Each folder holds two files:
+- `model.safetensors`: the packed ternary weights in MLX's 2-bit affine layout, group 128.
+- `config.json`: the model's shape, and `step`, which changes with each checkpoint.
+
+A WebGPU page loads a folder by its URL:
+`https://huggingface.co/nirca/nirca-mini/resolve/main/latest/`.
+
+## Architecture
+
+- Byte-level: 256 byte values plus special tokens, 265 in all. It needs no tokenizer file.
+- Width 512, with 8 attention heads and rotary position embeddings.
+- A mixture-of-experts layer with 32 experts, 8 active per token.
+- A recurrent block that repeats with learned halting, up to 24 times, after 2 plain layers.
+- About 109 million parameters.
+- A context of 2,048 bytes.
+- The ChatML chat format (`<|im_start|>` / `<|im_end|>`), with loss on assistant turns only.
+
+## Training data
+
+- [mini-AGI's corpus](https://github.com/volotat/mini-AGI): pretraining text.
+- [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk): chat conversations,
+  including a subset of those that fit within 2,048 bytes.
+- [OpenThoughts-114k](https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k),
+  reformatted as ChatML.
+- Generated chats: synthetic conversations written to give the model its character.
+
+## Limits
+
+It is a small research model and is often wrong or incoherent. Don't rely on what it says.

checkpoints/1217308k/config.json

@@ -0,0 +1,41 @@
+{
+ "name": "nirca-mini",
+ "step": 62895,
+ "quant": "ternary",
+ "quant_scope": "all",
+ "n_experts": 32,
+ "chat_format": "chatml",
+ "cfg": {
+  "vocab_size": 265,
+  "d_model": 512,
+  "n_head": 8,
+  "d_ff": 1408,
+  "block": 2048,
+  "rope_theta": 10000.0,
+  "tie_embeddings": true,
+  "n_prelude": 2,
+  "n_recur": 1,
+  "n_coda": 0,
+  "max_steps": 24,
+  "pool_experts": 32,
+  "pool_d_ff": 2048,
+  "pool_top_k": 8,
+  "pool_resident": 32,
+  "pool_capacity_factor": 2.25,
+  "pool_aux": 0.01,
+  "pool_balance": 0.001,
+  "min_steps": 1,
+  "train_steps_mean": 8.0,
+  "train_depth": "halt:8/2/1",
+  "bptt_window": 24,
+  "ponder_beta": 0.01,
+  "halt_prior": 0.072,
+  "halt_thresh": 0.9,
+  "halt_freeze": true,
+  "adam_moments": "bf16",
+  "adam_beta2": 0.95,
+  "use_pool": true,
+  "pool_depth": 1,
+  "pool_max": 32
+ }
+}

checkpoints/1217308k/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1e10118a19500e4574589b8b534a05333d0fcdbc573445ee50e3cbc36f36b88
+size 32638048

latest.json

@@ -0,0 +1,18 @@
+{
+ "name": "nirca-mini",
+ "step": 62895,
+ "tokens": 1217308397,
+ "checkpoint": "checkpoints/1217308k/",
+ "files": {
+  "config.json": {
+   "url": "https://huggingface.co/nirca/nirca-mini/resolve/main/checkpoints/1217308k/config.json",
+   "sha256": "def7980ea0549de5ec9b85239e5238de44821cda935ad738e1fb61ed7af2c003",
+   "bytes": 804
+  },
+  "model.safetensors": {
+   "url": "https://huggingface.co/nirca/nirca-mini/resolve/main/checkpoints/1217308k/model.safetensors",
+   "sha256": "c1e10118a19500e4574589b8b534a05333d0fcdbc573445ee50e3cbc36f36b88",
+   "bytes": 32638048
+  }
+ }
+}

latest/config.json

@@ -0,0 +1,41 @@
+{
+ "name": "nirca-mini",
+ "step": 62895,
+ "quant": "ternary",
+ "quant_scope": "all",
+ "n_experts": 32,
+ "chat_format": "chatml",
+ "cfg": {
+  "vocab_size": 265,
+  "d_model": 512,
+  "n_head": 8,
+  "d_ff": 1408,
+  "block": 2048,
+  "rope_theta": 10000.0,
+  "tie_embeddings": true,
+  "n_prelude": 2,
+  "n_recur": 1,
+  "n_coda": 0,
+  "max_steps": 24,
+  "pool_experts": 32,
+  "pool_d_ff": 2048,
+  "pool_top_k": 8,
+  "pool_resident": 32,
+  "pool_capacity_factor": 2.25,
+  "pool_aux": 0.01,
+  "pool_balance": 0.001,
+  "min_steps": 1,
+  "train_steps_mean": 8.0,
+  "train_depth": "halt:8/2/1",
+  "bptt_window": 24,
+  "ponder_beta": 0.01,
+  "halt_prior": 0.072,
+  "halt_thresh": 0.9,
+  "halt_freeze": true,
+  "adam_moments": "bf16",
+  "adam_beta2": 0.95,
+  "use_pool": true,
+  "pool_depth": 1,
+  "pool_max": 32
+ }
+}

latest/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1e10118a19500e4574589b8b534a05333d0fcdbc573445ee50e3cbc36f36b88
+size 32638048