metadata
language:
- en
license: apache-2.0
tags:
- bitnet
- 1-bit
- ternary
- slm
- cronus
- onebit-ai
pipeline_tag: text-generation
Cronus-150M (Native Ternary SLM)
Cronus-150M is a ~151M parameter general-purpose native ternary small language model developed by OneBit AI. Forward body linear layers use native ternary weights {-1, 0, +1} with int8 activation quantization (BitNet b1.58 style) with fp32 master weights.
Model Details
- Organization: OneBit AI
- Architecture: BitLinear Transformer SLM
- Parameters: 151,100,160
- Hidden Dimension: 768
- Layers: 24
- Attention Heads: 12 (Query) / 4 (KV, Grouped-Query Attention)
- Intermediate Dimension: 2048
- Context Length: 2048
- Normalization: RMSNorm with Sub-Layer Normalization (SubLN)
- Vocabulary Size: 49,152 (SmolLM2-135M tokenizer)
- Precision / Training Mode:
int8_tf32
Training Status (Burn / Smoke Run)
- Tokens Trained: ~149.95M tokens (286 updates, real 524,288 tokens/update)
- Training Hardware: NVIDIA GeForce RTX 5070
- Final Training Loss: 5.39 (from 10.99 initial)
- Held-out Probe Loss: 5.53 (from 6.68 initial)
Files in this Repository
model.pt: Clean model weights (state_dict).step_000000286_final.pt: Full resume training checkpoint (model weights, fp32 master weights, optimizer state).config.json: Architecture configuration parameters.tokenizer.json: Tokenizer vocabulary and BPE merges.