Update model card
Browse files
README.md
CHANGED
|
@@ -57,9 +57,7 @@ Complete settings are in [`config.json`](config.json). `special_tokens_map.json`
|
|
| 57 |
|
| 58 |
## Training and released-model lineage
|
| 59 |
|
| 60 |
-
Pretraining used approximately 13B token positions on a single RTX 4090. The exact retained packed count was 13,000,005,634 tokens; optimizer updates traversed 12,999,720,960 model-input positions.
|
| 61 |
-
|
| 62 |
-
Stage A used selected serialized tokens from FineWeb-Edu dedup (71.11%), DCLM-Edu (20.32%), FineWiki EN (5.08%), and Python-Edu (3.50%). Stage B added structured tutorials, PES2O, and StackExchange while retaining the other source families. The stage budgets, source shares, pinned identities, and accounting definitions are in [`PRETRAIN_SOURCE_MIXTURE.csv`](tables/PRETRAIN_SOURCE_MIXTURE.csv) and the technical report.
|
| 63 |
|
| 64 |
The selected weight lineage is:
|
| 65 |
|
|
@@ -70,13 +68,13 @@ alpha075 = P2 + 0.75 × (P3_step320 − P2)
|
|
| 70 |
|
| 71 |
P2 used 12,000 selected conversations from `HuggingFaceTB/smol-smoltalk` at revision `f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc`, default configuration, train split. The seven retained labels are values in the collection's `source` column, not seven independently pinned component repositories. P2 supervised every assistant turn's content and trailing EOS.
|
| 72 |
|
| 73 |
-
Later DPO, DeepSeek-response-distillation, LoRA, and unified Base-SFT experiments are **not ancestors of these weights**. The soft-logit lab used separate external models. Consult [`MODEL_PROVENANCE.json`](MODEL_PROVENANCE.json) and the report rather than combining exposures from independent branches.
|
| 74 |
|
| 75 |
## Evaluation
|
| 76 |
|
| 77 |
### Public multiple-choice likelihood
|
| 78 |
|
| 79 |
-
All three models were evaluated on the same full splits and frozen FP32 protocol.
|
| 80 |
|
| 81 |
| Model | ARC-Easy acc | ARC-Easy acc_norm | PIQA acc | PIQA acc_norm |
|
| 82 |
|---|---:|---:|---:|---:|
|
|
|
|
| 57 |
|
| 58 |
## Training and released-model lineage
|
| 59 |
|
| 60 |
+
Pretraining used approximately 13B token positions on a single RTX 4090. The exact retained packed count was 13,000,005,634 tokens; optimizer updates traversed 12,999,720,960 model-input positions. More precisely, Stage A used selected serialized tokens from FineWeb-Edu dedup (71.11%), DCLM-Edu (20.32%), FineWiki EN (5.08%), and Python-Edu (3.50%). Stage B added structured tutorials, PES2O, and StackExchange while retaining the other source families. The stage budgets, source shares, pinned identities, and accounting definitions are in [`PRETRAIN_SOURCE_MIXTURE.csv`](tables/PRETRAIN_SOURCE_MIXTURE.csv) and the technical report.
|
|
|
|
|
|
|
| 61 |
|
| 62 |
The selected weight lineage is:
|
| 63 |
|
|
|
|
| 68 |
|
| 69 |
P2 used 12,000 selected conversations from `HuggingFaceTB/smol-smoltalk` at revision `f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc`, default configuration, train split. The seven retained labels are values in the collection's `source` column, not seven independently pinned component repositories. P2 supervised every assistant turn's content and trailing EOS.
|
| 70 |
|
| 71 |
+
Later DPO, DeepSeek-response-distillation, LoRA, and unified Base-SFT experiments did not show significant help so are **not ancestors of these weights**. The soft-logit lab used separate external models. Consult [`MODEL_PROVENANCE.json`](MODEL_PROVENANCE.json) and the report rather than combining exposures from independent branches.
|
| 72 |
|
| 73 |
## Evaluation
|
| 74 |
|
| 75 |
### Public multiple-choice likelihood
|
| 76 |
|
| 77 |
+
Here we list the some benchmarks compared with SmolLM-135M-Instruct and SmolLM2-135M-Instruct. All three models were evaluated on the same full splits and frozen FP32 protocol.
|
| 78 |
|
| 79 |
| Model | ARC-Easy acc | ARC-Easy acc_norm | PIQA acc | PIQA acc_norm |
|
| 80 |
|---|---:|---:|---:|---:|
|