Harley-ml commited on
Commit
d3d8b7f
·
verified ·
1 Parent(s): 37e98a2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -41,8 +41,8 @@ We tested two different architecture configurations on 500M tokens of FineWeb to
41
  - RMS Norm Eps: `1e-06`
42
  - Pretraining Tp: `1`
43
  - Use Cache: `false`
44
- - **Total Parameters**: `3,889,920`
45
  - **Hidden Size/Layers**: `6.10`
 
46
 
47
  ### Config B: Shallow-Wide (code name: `width_311`)
48
 
@@ -65,8 +65,8 @@ We tested two different architecture configurations on 500M tokens of FineWeb to
65
  - RMS Norm Eps: `1e-06`
66
  - Pretraining Tp: `1`
67
  - Use Cache: `false`
68
- - **Total Parameters**: `3,816,576`
69
  - **Hidden Size/Layers**: `21.33`
 
70
 
71
  ## Training Setup
72
 
 
41
  - RMS Norm Eps: `1e-06`
42
  - Pretraining Tp: `1`
43
  - Use Cache: `false`
 
44
  - **Hidden Size/Layers**: `6.10`
45
+ - **Total Parameters**: `3,889,920`
46
 
47
  ### Config B: Shallow-Wide (code name: `width_311`)
48
 
 
65
  - RMS Norm Eps: `1e-06`
66
  - Pretraining Tp: `1`
67
  - Use Cache: `false`
 
68
  - **Hidden Size/Layers**: `21.33`
69
+ - **Total Parameters**: `3,816,576`
70
 
71
  ## Training Setup
72