| ---
|
| language:
|
| - en
|
| license: apache-2.0
|
| tags:
|
| - bitnet
|
| - 1-bit
|
| - ternary
|
| - slm
|
| - cronus
|
| - onebit-ai
|
| pipeline_tag: text-generation
|
| ---
|
|
|
| # Cronus-150M (Native Ternary SLM)
|
|
|
| **Cronus-150M** is a ~151M parameter general-purpose native ternary small language model developed by **OneBit AI**.
|
| Forward body linear layers use **native ternary weights {-1, 0, +1}** with int8 activation quantization (BitNet b1.58 style) with fp32 master weights.
|
|
|
| ## Model Details
|
| * **Organization:** [OneBit AI](https://huggingface.co/OneBitAI)
|
| * **Architecture:** BitLinear Transformer SLM
|
| * **Parameters:** 151,100,160
|
| * **Hidden Dimension:** 768
|
| * **Layers:** 24
|
| * **Attention Heads:** 12 (Query) / 4 (KV, Grouped-Query Attention)
|
| * **Intermediate Dimension:** 2048
|
| * **Context Length:** 2048
|
| * **Normalization:** RMSNorm with Sub-Layer Normalization (SubLN)
|
| * **Vocabulary Size:** 49,152 (SmolLM2-135M tokenizer)
|
| * **Precision / Training Mode:** `int8_tf32`
|
|
|
| ## Training Status (Burn / Smoke Run)
|
| * **Tokens Trained:** ~149.95M tokens (286 updates, real 524,288 tokens/update)
|
| * **Training Hardware:** NVIDIA GeForce RTX 5070
|
| * **Final Training Loss:** 5.39 (from 10.99 initial)
|
| * **Held-out Probe Loss:** 5.53 (from 6.68 initial)
|
|
|
| ## Files in this Repository
|
| * `model.pt`: Clean model weights (`state_dict`).
|
| * `step_000000286_final.pt`: Full resume training checkpoint (model weights, fp32 master weights, optimizer state).
|
| * `config.json`: Architecture configuration parameters.
|
| * `tokenizer.json`: Tokenizer vocabulary and BPE merges.
|
|
|
