Upload folder using huggingface_hub

README.md

@@ -0,0 +1,43 @@
+---
+language:
+- en
+license: apache-2.0
+tags:
+- bitnet
+- 1-bit
+- ternary
+- slm
+- cronus
+- onebit-ai
+pipeline_tag: text-generation
+---
+
+# Cronus-150M (Native Ternary SLM)
+
+**Cronus-150M** is a ~151M parameter general-purpose native ternary small language model developed by **OneBit AI**.
+Forward body linear layers use **native ternary weights {-1, 0, +1}** with int8 activation quantization (BitNet b1.58 style) with fp32 master weights.
+
+## Model Details
+* **Organization:** [OneBit AI](https://huggingface.co/OneBitAI)
+* **Architecture:** BitLinear Transformer SLM
+* **Parameters:** 151,100,160
+* **Hidden Dimension:** 768
+* **Layers:** 24
+* **Attention Heads:** 12 (Query) / 4 (KV, Grouped-Query Attention)
+* **Intermediate Dimension:** 2048
+* **Context Length:** 2048
+* **Normalization:** RMSNorm with Sub-Layer Normalization (SubLN)
+* **Vocabulary Size:** 49,152 (SmolLM2-135M tokenizer)
+* **Precision / Training Mode:** `int8_tf32`
+
+## Training Status (Burn / Smoke Run)
+* **Tokens Trained:** ~149.95M tokens (286 updates, real 524,288 tokens/update)
+* **Training Hardware:** NVIDIA GeForce RTX 5070
+* **Final Training Loss:** 5.39 (from 10.99 initial)
+* **Held-out Probe Loss:** 5.53 (from 6.68 initial)
+
+## Files in this Repository
+* `model.pt`: Clean model weights (`state_dict`).
+* `step_000000286_final.pt`: Full resume training checkpoint (model weights, fp32 master weights, optimizer state).
+* `config.json`: Architecture configuration parameters.
+* `tokenizer.json`: Tokenizer vocabulary and BPE merges.

config.json

@@ -0,0 +1,19 @@
+{
+  "model_type": "cronus-150m",
+  "architectures": [
+    "CronusForCausalLM"
+  ],
+  "vocab_size": 49152,
+  "hidden_size": 768,
+  "num_hidden_layers": 24,
+  "num_attention_heads": 12,
+  "num_key_value_heads": 4,
+  "intermediate_size": 2048,
+  "max_position_embeddings": 2048,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "sub_layer_norm": true,
+  "tie_word_embeddings": true,
+  "activation_qb": 128,
+  "num_parameters": 151100160
+}

model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c917e19e649e521e945e7034f1880f8b3544008ed852748ffbb5346873f0de14
+size 604461163