ckpt latest.json -> {'step': 3814, 'path_in_repo': 'final'} tensors 200 params 106,194,240 dtypes ['F32'] card: 106,194,240 params from the safetensors header; 6 files hashed
license: cc-by-sa-4.0 base_model: Cion-lab/ounce100m-v1 language: en library_name: transformers pipeline_tag: text-generation tags: [ounce100m, from-scratch, base, decoder-only, huggingfacetb]
ounce100m-v1
A 106,194,240-parameter decoder-only transformer trained from scratch on
999,817,216 tokens of Cion-lab/ounce100m-mix-v1, in 3814 optimiser steps on
2×Nvidia Tesla T4 inside Kaggle notebook sessions. It is the deliverable of a project whose whole point
was to find out what a ~100 M-parameter model reaches from ~1 B tokens of a curated public mix, and to
report it honestly, including where that is near chance.
Measured facts
| Parameters | 106,194,240 (recomputed from model.safetensors's own header, not asserted) |
| Architecture | 22 layers, hidden 576, GQA 9 Q / 3 KV heads, SwiGLU 1536, RMSNorm 1e-05, RoPE θ=10000.0, tied embeddings, vocab 49,152 |
| Sequence length | 1024 — which is why ARC's 25-shot prompts are truncated from the left by the eval harness |
| Tokens consumed | 999,817,216 (99.98 % of the 1.0 B target) |
| Batch | micro 4 × accum 32 × 2 ranks = 262,144 tokens/step |
| Optimiser | adamw_torch, LR 0.0006, warmup fraction 0.02, trapezoid decay over the final 20.0 % of the schedule |
| Precision | fp16 autocast + fp32 master weights + GradScaler — asserted at the start of every session, and each session's log line precision: fp16=True bf16=False scaler=GradScaler is the evidence |
| Throughput | 12,150 tok/s in the session that produced this export |
| Attention | eager |
| Gradient checkpointing | False |
| Randomness | model seed 20260919, data seed 20260919 |
| Wall clock | 4.59 h in the session that produced this export. The run is split across sessions at checkpoint boundaries, so this is not the total training time |
| Loss | 4.43187255859375 -> 4.264701843261719 |
| Validation PPL | 75.98 on Cion-lab/ounce100m-mix-v1's held-out val/ shards (22,934,043 tokens, never trained on) |
Data
Cion-lab/ounce100m-mix-v1: 1,109,714,831 training tokens in 139 shards + 22,934,043
held-out validation tokens, cut with HuggingFaceTB/SmolLM2-135M:tokenizer.json (49,152 vocab,
Apache-2.0), from 15 public sources at ≥15
per shard. Format and loading code are in that dataset's card.
Contamination. Overlap was measured mechanically against benchmark train/validation/dev material of
the eight evaluation tasks (13-token windows, counts only — no benchmark item text was ever read, and test
splits stayed untouched until evaluation). audit.json in the dataset repo records
overlap_total = 0 over 1,073,047 documents and
21,898 held-out documents, tasks_covered = 8/8.
Building it required dropping 5878 documents
(17,366,967 tokens)
that matched reference material; the exact exclusion masks ship in filter/ so the mix is reproducible.
Evaluate it
pip install "lm-eval==0.4.13"
lm_eval --model hf --model_args pretrained=Cion-lab/ounce100m-v1,dtype=float16,trust_remote_code=False \
--tasks arc_challenge,arc_easy,hellaswag,mmlu,piqa,truthfulqa_mc1,truthfulqa_mc2,winogrande,gsm8k \
--batch_size 8 --seed 42 --log_samples
The run's own results, with shot counts and chance levels beside every number, are in eval/ here. Four of
the eight tasks (Winogrande, HellaSwag, PIQA, TruthfulQA) are scored by the harness on validation — the
splits this project was allowed to read — and the other four on test.
Files
| file | sha256 | bytes |
|---|---|---|
config.json |
be63e8d2dc708f8b… | 723 |
cursor.json |
c5ea3ba9293f3278… | 205 |
generation_config.json |
1dac20576af54bdd… | 219 |
model.safetensors |
669ac130a8853cc2… | 424,799,272 |
run_summary.json |
d7789a4e70e296c3… | 968 |
tokenizer.json |
9ca9acddb6525a19… | 2,104,556 |
cursor.json is the data position this checkpoint corresponds to: samples consumed, step, seed, the
corpus fingerprint and a hash of the visit-order permutation, so the resume can be checked rather than
trusted.
Not in this model
No instruction tuning, no RLHF, no chat template — it is a base model, deliberately. It has ~1 B tokens of training against the ~300 B of the comparable published reference models, so expect scores at or near chance on several tasks; that is the experiment's result, not a defect to be engineered around.