File size: 1,562 Bytes
e984fbe | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | ---
language:
- en
license: apache-2.0
tags:
- bitnet
- 1-bit
- ternary
- slm
- cronus
- onebit-ai
pipeline_tag: text-generation
---
# Cronus-150M (Native Ternary SLM)
**Cronus-150M** is a ~151M parameter general-purpose native ternary small language model developed by **OneBit AI**.
Forward body linear layers use **native ternary weights {-1, 0, +1}** with int8 activation quantization (BitNet b1.58 style) with fp32 master weights.
## Model Details
* **Organization:** [OneBit AI](https://huggingface.co/OneBitAI)
* **Architecture:** BitLinear Transformer SLM
* **Parameters:** 151,100,160
* **Hidden Dimension:** 768
* **Layers:** 24
* **Attention Heads:** 12 (Query) / 4 (KV, Grouped-Query Attention)
* **Intermediate Dimension:** 2048
* **Context Length:** 2048
* **Normalization:** RMSNorm with Sub-Layer Normalization (SubLN)
* **Vocabulary Size:** 49,152 (SmolLM2-135M tokenizer)
* **Precision / Training Mode:** `int8_tf32`
## Training Status (Burn / Smoke Run)
* **Tokens Trained:** ~149.95M tokens (286 updates, real 524,288 tokens/update)
* **Training Hardware:** NVIDIA GeForce RTX 5070
* **Final Training Loss:** 5.39 (from 10.99 initial)
* **Held-out Probe Loss:** 5.53 (from 6.68 initial)
## Files in this Repository
* `model.pt`: Clean model weights (`state_dict`).
* `step_000000286_final.pt`: Full resume training checkpoint (model weights, fp32 master weights, optimizer state).
* `config.json`: Architecture configuration parameters.
* `tokenizer.json`: Tokenizer vocabulary and BPE merges.
|