anandselvadurai-techus commited on
Commit
1c35d59
·
verified ·
1 Parent(s): 429b898

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -5
README.md CHANGED
@@ -156,8 +156,8 @@ Ethoric-10M was trained in two stages.
156
 
157
  | Stage | Tokens | Focus |
158
  |---|---|---|
159
- | 1. Stable | 15B | Broad mix of educational web text, synthetic textbooks, question-answer data and math |
160
- | 2. Cooldown | 4B | Learning rate decayed; mix shifted toward higher-quality math, QA and tutorial-style text |
161
 
162
  Training packs multiple documents into each 512-token sequence. Document masking keeps them separate: attention and Canon convolutions never reach across an EOS token.
163
 
@@ -181,9 +181,6 @@ The cooldown added new math sources such as FineMath, more question-answer and t
181
 
182
  † ELI5, GooAQ, FactBase-DCLM, Cosmopedia-v2, Natural Questions, Pretrain-Behaviors, UltraData-Math
183
 
184
- - **Tokenizer:** all data was tokenized with the model's 5,048-token BPE tokenizer, and 1% was held out for validation.
185
- - **Blending:** sources were blended per dataset using the weights above.
186
-
187
  ## Limitations
188
 
189
  - **Tiny model:** with 9.97M parameters and 256-dim hidden states, generations are often repetitive, incoherent or factually wrong. The model is a research artifact for studying small-scale LMs, not an assistant.
 
156
 
157
  | Stage | Tokens | Focus |
158
  |---|---|---|
159
+ | Stable | 15B | Broad mix of educational web text, synthetic textbooks, question-answer data and math |
160
+ | Cooldown | 4B | Learning rate decayed; mix shifted toward higher-quality math, QA and tutorial-style text |
161
 
162
  Training packs multiple documents into each 512-token sequence. Document masking keeps them separate: attention and Canon convolutions never reach across an EOS token.
163
 
 
181
 
182
  † ELI5, GooAQ, FactBase-DCLM, Cosmopedia-v2, Natural Questions, Pretrain-Behaviors, UltraData-Math
183
 
 
 
 
184
  ## Limitations
185
 
186
  - **Tiny model:** with 9.97M parameters and 256-dim hidden states, generations are often repetitive, incoherent or factually wrong. The model is a research artifact for studying small-scale LMs, not an assistant.