yqi0 commited on
Commit
5b577b9
·
verified ·
1 Parent(s): 0f1a91a

Update model card

Browse files
Files changed (1) hide show
  1. README.md +3 -5
README.md CHANGED
@@ -57,9 +57,7 @@ Complete settings are in [`config.json`](config.json). `special_tokens_map.json`
57
 
58
  ## Training and released-model lineage
59
 
60
- Pretraining used approximately 13B token positions on a single RTX 4090. The exact retained packed count was 13,000,005,634 tokens; optimizer updates traversed 12,999,720,960 model-input positions. These are different accounting quantities, not interchangeable measures of processed training targets.
61
-
62
- Stage A used selected serialized tokens from FineWeb-Edu dedup (71.11%), DCLM-Edu (20.32%), FineWiki EN (5.08%), and Python-Edu (3.50%). Stage B added structured tutorials, PES2O, and StackExchange while retaining the other source families. The stage budgets, source shares, pinned identities, and accounting definitions are in [`PRETRAIN_SOURCE_MIXTURE.csv`](tables/PRETRAIN_SOURCE_MIXTURE.csv) and the technical report.
63
 
64
  The selected weight lineage is:
65
 
@@ -70,13 +68,13 @@ alpha075 = P2 + 0.75 × (P3_step320 − P2)
70
 
71
  P2 used 12,000 selected conversations from `HuggingFaceTB/smol-smoltalk` at revision `f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc`, default configuration, train split. The seven retained labels are values in the collection's `source` column, not seven independently pinned component repositories. P2 supervised every assistant turn's content and trailing EOS.
72
 
73
- Later DPO, DeepSeek-response-distillation, LoRA, and unified Base-SFT experiments are **not ancestors of these weights**. The soft-logit lab used separate external models. Consult [`MODEL_PROVENANCE.json`](MODEL_PROVENANCE.json) and the report rather than combining exposures from independent branches.
74
 
75
  ## Evaluation
76
 
77
  ### Public multiple-choice likelihood
78
 
79
- All three models were evaluated on the same full splits and frozen FP32 protocol. Values are the existing measurements, not new runs.
80
 
81
  | Model | ARC-Easy acc | ARC-Easy acc_norm | PIQA acc | PIQA acc_norm |
82
  |---|---:|---:|---:|---:|
 
57
 
58
  ## Training and released-model lineage
59
 
60
+ Pretraining used approximately 13B token positions on a single RTX 4090. The exact retained packed count was 13,000,005,634 tokens; optimizer updates traversed 12,999,720,960 model-input positions. More precisely, Stage A used selected serialized tokens from FineWeb-Edu dedup (71.11%), DCLM-Edu (20.32%), FineWiki EN (5.08%), and Python-Edu (3.50%). Stage B added structured tutorials, PES2O, and StackExchange while retaining the other source families. The stage budgets, source shares, pinned identities, and accounting definitions are in [`PRETRAIN_SOURCE_MIXTURE.csv`](tables/PRETRAIN_SOURCE_MIXTURE.csv) and the technical report.
 
 
61
 
62
  The selected weight lineage is:
63
 
 
68
 
69
  P2 used 12,000 selected conversations from `HuggingFaceTB/smol-smoltalk` at revision `f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc`, default configuration, train split. The seven retained labels are values in the collection's `source` column, not seven independently pinned component repositories. P2 supervised every assistant turn's content and trailing EOS.
70
 
71
+ Later DPO, DeepSeek-response-distillation, LoRA, and unified Base-SFT experiments did not show significant help so are **not ancestors of these weights**. The soft-logit lab used separate external models. Consult [`MODEL_PROVENANCE.json`](MODEL_PROVENANCE.json) and the report rather than combining exposures from independent branches.
72
 
73
  ## Evaluation
74
 
75
  ### Public multiple-choice likelihood
76
 
77
+ Here we list the some benchmarks compared with SmolLM-135M-Instruct and SmolLM2-135M-Instruct. All three models were evaluated on the same full splits and frozen FP32 protocol.
78
 
79
  | Model | ARC-Easy acc | ARC-Easy acc_norm | PIQA acc | PIQA acc_norm |
80
  |---|---:|---:|---:|---:|