Update README.md
Browse files
README.md
CHANGED
|
@@ -37,7 +37,9 @@ The model is trained with autoregressive next-token prediction.
|
|
| 37 |
|
| 38 |
Training summary:
|
| 39 |
|
| 40 |
-
* pretraining data: approximately **1T tokens**
|
|
|
|
|
|
|
| 41 |
* optimizer: **ROCA Optimizer**
|
| 42 |
* dtype: bfloat16
|
| 43 |
|
|
|
|
| 37 |
|
| 38 |
Training summary:
|
| 39 |
|
| 40 |
+
* pretraining data: approximately **1T tokens** with 8k context.
|
| 41 |
+
* midtraining data: approximately **50B tokens** with 32k context.
|
| 42 |
+
* posttraining data (SFT & modified-SLIME): approximately **5B tokens**.
|
| 43 |
* optimizer: **ROCA Optimizer**
|
| 44 |
* dtype: bfloat16
|
| 45 |
|