File size: 2,393 Bytes
9022ca1 e2d3665 e0436c4 9022ca1 e2d3665 9022ca1 e2d3665 9022ca1 9c6190d 9022ca1 e2d3665 9022ca1 e0436c4 e2d3665 9022ca1 e0436c4 9022ca1 e0436c4 9022ca1 e0436c4 9022ca1 e0436c4 e2d3665 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | ---
license: apache-2.0
pipeline_tag: text-generation
tags:
- base-model
language:
- en
---
# Prochlo-190M-Base
The smallest and first model from CyanoAI — a 190.5M-parameter dense English base LM.
Named after *Prochlorococcus*, the tiniest and most oxygen-producing photosynthetic organism on Earth: small, dense, and outproducing everything its size.
## Model family
| Repo | Stage | Status |
|---|---|---|
| [`CyanoAI/Prochlo-190M-Base`](https://huggingface.co/CyanoAI/Prochlo-190M-Base) | Pre-trained base | this repo |
| [`CyanoAI/Prochlo-190M-SFT`](https://huggingface.co/CyanoAI/Prochlo-190M-SFT) | Supervised fine-tune | available |
| `CyanoAI/Prochlo-190M` | Preference-aligned (DPO + RLAIF) | in training |
All variants share the same architecture, tokenizer, and vocabulary, so they are drop-in interchangeable.
## Model at a glance
| | |
|---|---|
| Parameters | 190.5M dense (untied embeddings) |
| Architecture | LLaMA-style decoder: RMSNorm, RoPE (θ=1e5), SwiGLU, no biases, per-head QK RMSNorm |
| Shape | 12 layers × dim 768, 12 heads, hidden 3072 |
| Vocab | 50,261 (GPT-2 BPE + 4 control tokens) |
| Context | 1024 |
| Precision | FP16 |
| Format | Safetensors, loads with `Qwen3ForCausalLM` (architecture-compatible; trained from scratch by CyanoAI, not derived from Qwen) |
## Pretraining
| | |
|---|---|
| Tokens | ~7.7B |
| Steps | 300,000 |
| Optimizer | Muon (2D matrices) + AdamW |
| Schedule | lr 1e-3, warmup 64, linear decay |
| Final train loss | 1.7947 (ppl ≈ 6.0) |
| Final eval loss | 1.6205 (ppl ≈ 5.06, still descending) |
Training corpus: a curated ~826GB English mixture (web / books / code / reference), stream-packed with the tokenizer jointly shipped in this repo.
## Intended use
This is a **base model** — it speaks fluent Wikipedia-style prose but has not been aligned.
For chat or assistant behaviour, start from the [SFT](https://huggingface.co/CyanoAI/Prochlo-190M-SFT) or aligned variants.
## Load
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("CyanoAI/Prochlo-190M-Base")
model = AutoModelForCausalLM.from_pretrained("CyanoAI/Prochlo-190M-Base", dtype="float16")
```
## Evaluation
Zero-shot on `wikitext` (JSON subset, ~500k tokens): ppl ≈ 109.7.
Not directly comparable to GPT-2-series numbers — different training distribution.
## License
Apache 2.0 |