ckpt latest.json -> {'step': 3814, 'path_in_repo': 'final'}
tensors 200 params 106,194,240 dtypes ['F32']
card: 106,194,240 params from the safetensors header; 6 files hashed
---
license: cc-by-sa-4.0
base_model: Cion-lab/ounce100m-v1
language: en
library_name: transformers
pipeline_tag: text-generation
tags: [ounce100m, from-scratch, base, decoder-only, huggingfacetb]
---

# ounce100m-v1

A **106,194,240-parameter decoder-only transformer trained from scratch on
999,817,216 tokens** of `Cion-lab/ounce100m-mix-v1`, in 3814 optimiser steps on
**2×Nvidia Tesla T4** inside Kaggle notebook sessions. It is the deliverable of a project whose whole point
was to find out what a ~100 M-parameter model reaches from ~1 B tokens of a curated public mix, and to
report it honestly, including where that is near chance.

## Measured facts

| | |
|---|---|
| Parameters | **106,194,240** (recomputed from `model.safetensors`'s own header, not asserted) |
| Architecture | 22 layers, hidden 576, GQA 9 Q / 3 KV heads, SwiGLU 1536, RMSNorm 1e-05, RoPE θ=10000.0, **tied** embeddings, vocab 49,152 |
| Sequence length | 1024 — which is why ARC's 25-shot prompts are truncated from the left by the eval harness |
| Tokens consumed | **999,817,216** (99.98 % of the 1.0 B target) |
| Batch | micro 4 × accum 32 × 2 ranks = 262,144 tokens/step |
| Optimiser | `adamw_torch`, LR 0.0006, warmup fraction 0.02, trapezoid decay over the final 20.0 % of the schedule |
| Precision | fp16 autocast + fp32 master weights + GradScaler — asserted at the start of every session, and each session's log line `precision: fp16=True bf16=False scaler=GradScaler` is the evidence |
| Throughput | 12,150 tok/s in the session that produced this export |
| Attention | `eager` |
| Gradient checkpointing | `False` |
| Randomness | model seed 20260919, data seed 20260919 |
| Wall clock | 4.59 h in the session that produced this export. The run is split across sessions at checkpoint boundaries, so this is **not** the total training time |
| Loss | 4.43187255859375 -> 4.264701843261719 |
| Validation PPL | **75.98** on Cion-lab/ounce100m-mix-v1's held-out `val/` shards (22,934,043 tokens, never trained on) |

## Data

`Cion-lab/ounce100m-mix-v1`: 1,109,714,831 training tokens in 139 shards + 22,934,043
held-out validation tokens, cut with `HuggingFaceTB/SmolLM2-135M:tokenizer.json` (49,152 vocab,
Apache-2.0), from 15 public sources at ≥15
per shard. Format and loading code are in that dataset's card.

**Contamination.** Overlap was measured mechanically against benchmark **train/validation/dev** material of
the eight evaluation tasks (13-token windows, counts only — no benchmark item text was ever read, and test
splits stayed untouched until evaluation). `audit.json` in the dataset repo records
`overlap_total = 0` over 1,073,047 documents and
21,898 held-out documents, `tasks_covered` = 8/8.
Building it required dropping 5878 documents
(17,366,967 tokens)
that matched reference material; the exact exclusion masks ship in `filter/` so the mix is reproducible.

## Evaluate it

```bash
pip install "lm-eval==0.4.13"
lm_eval --model hf --model_args pretrained=Cion-lab/ounce100m-v1,dtype=float16,trust_remote_code=False \
  --tasks arc_challenge,arc_easy,hellaswag,mmlu,piqa,truthfulqa_mc1,truthfulqa_mc2,winogrande,gsm8k \
  --batch_size 8 --seed 42 --log_samples
```

The run's own results, with shot counts and chance levels beside every number, are in `eval/` here. Four of
the eight tasks (Winogrande, HellaSwag, PIQA, TruthfulQA) are scored by the harness on *validation* — the
splits this project was allowed to read — and the other four on test.

## Files

| file | sha256 | bytes |
|---|---|---|
| `config.json` | be63e8d2dc708f8b… | 723 |
| `cursor.json` | c5ea3ba9293f3278… | 205 |
| `generation_config.json` | 1dac20576af54bdd… | 219 |
| `model.safetensors` | 669ac130a8853cc2… | 424,799,272 |
| `run_summary.json` | d7789a4e70e296c3… | 968 |
| `tokenizer.json` | 9ca9acddb6525a19… | 2,104,556 |

`cursor.json` is the data position this checkpoint corresponds to: samples consumed, step, seed, the
corpus fingerprint and a hash of the visit-order permutation, so the resume can be checked rather than
trusted.

## Not in this model

No instruction tuning, no RLHF, no chat template — it is a base model, deliberately. It has ~1 B tokens of
training against the ~300 B of the comparable published reference models, so expect scores at or near
chance on several tasks; that is the experiment's result, not a defect to be engineered around.