| ckpt latest.json -> {'step': 3814, 'path_in_repo': 'final'} |
| tensors 200 params 106,194,240 dtypes ['F32'] |
| card: 106,194,240 params from the safetensors header; 6 files hashed |
| --- |
| license: cc-by-sa-4.0 |
| base_model: Cion-lab/ounce100m-v1 |
| language: en |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: [ounce100m, from-scratch, base, decoder-only, huggingfacetb] |
| --- |
| |
| # ounce100m-v1 |
| |
| A **106,194,240-parameter decoder-only transformer trained from scratch on |
| 999,817,216 tokens** of `Cion-lab/ounce100m-mix-v1`, in 3814 optimiser steps on |
| **2×Nvidia Tesla T4** inside Kaggle notebook sessions. It is the deliverable of a project whose whole point |
| was to find out what a ~100 M-parameter model reaches from ~1 B tokens of a curated public mix, and to |
| report it honestly, including where that is near chance. |
| |
| ## Measured facts |
| |
| | | | |
| |---|---| |
| | Parameters | **106,194,240** (recomputed from `model.safetensors`'s own header, not asserted) | |
| | Architecture | 22 layers, hidden 576, GQA 9 Q / 3 KV heads, SwiGLU 1536, RMSNorm 1e-05, RoPE θ=10000.0, **tied** embeddings, vocab 49,152 | |
| | Sequence length | 1024 — which is why ARC's 25-shot prompts are truncated from the left by the eval harness | |
| | Tokens consumed | **999,817,216** (99.98 % of the 1.0 B target) | |
| | Batch | micro 4 × accum 32 × 2 ranks = 262,144 tokens/step | |
| | Optimiser | `adamw_torch`, LR 0.0006, warmup fraction 0.02, trapezoid decay over the final 20.0 % of the schedule | |
| | Precision | fp16 autocast + fp32 master weights + GradScaler — asserted at the start of every session, and each session's log line `precision: fp16=True bf16=False scaler=GradScaler` is the evidence | |
| | Throughput | 12,150 tok/s in the session that produced this export | |
| | Attention | `eager` | |
| | Gradient checkpointing | `False` | |
| | Randomness | model seed 20260919, data seed 20260919 | |
| | Wall clock | 4.59 h in the session that produced this export. The run is split across sessions at checkpoint boundaries, so this is **not** the total training time | |
| | Loss | 4.43187255859375 -> 4.264701843261719 | |
| | Validation PPL | **75.98** on Cion-lab/ounce100m-mix-v1's held-out `val/` shards (22,934,043 tokens, never trained on) | |
|
|
| ## Data |
|
|
| `Cion-lab/ounce100m-mix-v1`: 1,109,714,831 training tokens in 139 shards + 22,934,043 |
| held-out validation tokens, cut with `HuggingFaceTB/SmolLM2-135M:tokenizer.json` (49,152 vocab, |
| Apache-2.0), from 15 public sources at ≥15 |
| per shard. Format and loading code are in that dataset's card. |
|
|
| **Contamination.** Overlap was measured mechanically against benchmark **train/validation/dev** material of |
| the eight evaluation tasks (13-token windows, counts only — no benchmark item text was ever read, and test |
| splits stayed untouched until evaluation). `audit.json` in the dataset repo records |
| `overlap_total = 0` over 1,073,047 documents and |
| 21,898 held-out documents, `tasks_covered` = 8/8. |
| Building it required dropping 5878 documents |
| (17,366,967 tokens) |
| that matched reference material; the exact exclusion masks ship in `filter/` so the mix is reproducible. |
|
|
| ## Evaluate it |
|
|
| ```bash |
| pip install "lm-eval==0.4.13" |
| lm_eval --model hf --model_args pretrained=Cion-lab/ounce100m-v1,dtype=float16,trust_remote_code=False \ |
| --tasks arc_challenge,arc_easy,hellaswag,mmlu,piqa,truthfulqa_mc1,truthfulqa_mc2,winogrande,gsm8k \ |
| --batch_size 8 --seed 42 --log_samples |
| ``` |
|
|
| The run's own results, with shot counts and chance levels beside every number, are in `eval/` here. Four of |
| the eight tasks (Winogrande, HellaSwag, PIQA, TruthfulQA) are scored by the harness on *validation* — the |
| splits this project was allowed to read — and the other four on test. |
|
|
| ## Files |
|
|
| | file | sha256 | bytes | |
| |---|---|---| |
| | `config.json` | be63e8d2dc708f8b… | 723 | |
| | `cursor.json` | c5ea3ba9293f3278… | 205 | |
| | `generation_config.json` | 1dac20576af54bdd… | 219 | |
| | `model.safetensors` | 669ac130a8853cc2… | 424,799,272 | |
| | `run_summary.json` | d7789a4e70e296c3… | 968 | |
| | `tokenizer.json` | 9ca9acddb6525a19… | 2,104,556 | |
|
|
| `cursor.json` is the data position this checkpoint corresponds to: samples consumed, step, seed, the |
| corpus fingerprint and a hash of the visit-order permutation, so the resume can be checked rather than |
| trusted. |
|
|
| ## Not in this model |
|
|
| No instruction tuning, no RLHF, no chat template — it is a base model, deliberately. It has ~1 B tokens of |
| training against the ~300 B of the comparable published reference models, so expect scores at or near |
| chance on several tasks; that is the experiment's result, not a defect to be engineered around. |
|
|
|
|
