cfahlgren1 HF Staff commited on
Commit
a739c7e
·
verified ·
1 Parent(s): dc714b2

make the 3b-token run v2 and add the size comparison

Browse files
README.md CHANGED
@@ -27,10 +27,10 @@ model-index:
27
  metrics:
28
  - type: typing_saved
29
  name: Typing saved (%)
30
- value: 4.0
31
  - type: suggestion_precision
32
  name: Suggestions that were right (%)
33
- value: 76
34
  - task:
35
  type: text-generation
36
  name: Autocomplete (typing saved at a 50% cutoff)
@@ -40,10 +40,10 @@ model-index:
40
  metrics:
41
  - type: typing_saved
42
  name: Typing saved (%)
43
- value: 7.3
44
  - type: suggestion_precision
45
  name: Suggestions that were right (%)
46
- value: 78
47
  - task:
48
  type: text-generation
49
  name: Autocomplete (typing saved at a 50% cutoff)
@@ -53,10 +53,10 @@ model-index:
53
  metrics:
54
  - type: typing_saved
55
  name: Typing saved (%)
56
- value: 4.3
57
  - type: suggestion_precision
58
  name: Suggestions that were right (%)
59
- value: 79
60
  ---
61
 
62
  <p align="center">
@@ -89,31 +89,59 @@ v1 guesses one letter at a time, so its suggestions are short word endings. v2 i
89
 
90
  | Writing | v1 | v2 | SmolLM2-135M (ceiling, ~100 MB) |
91
  |---|---|---|---|
92
- | Articles (Wikipedia) | 1.5% | **4.0%** | 8.1% |
93
- | Messages (OpenAssistant) | 2.0% | **7.3%** | 16.8% |
94
- | Email (Enron, unseen senders) | 3.0% | **4.3%** | 5.7% |
95
 
96
  | Writing (v2) | Suggests at | Right | 2+ words suggested at | 2+ words right |
97
  |---|---|---|---|---|
98
- | Articles (Wikipedia) | 11% | 76% | 3% | 74% |
99
- | Messages (OpenAssistant) | 16% | 78% | 8% | 72% |
100
- | Email (Enron, unseen senders) | 11% | 79% | 3% | 60% |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
101
 
102
  **In the browser.**
103
 
104
  | | |
105
  |---|---|
106
  | Download | **13.3 MB**: int8 weights with one scale per 64 values, expanded to float32 when the page loads |
107
- | Matches full precision | same next word 99% of the time, same 5-word path 92%, typing saved within 0.2 points |
108
- | Speed | **4 ms** per suggestion (14 ms p95) in a Web Worker, Chrome on an M3 Max; typing stays under one frame |
109
 
110
  The browser computes in float32 on purpose: onnxruntime-web's int8 kernels were 8–11× slower than its float kernels for this model, and int8 activations changed the top word too often. Only the download is compressed.
111
 
112
- **Training.** 1.5B tokens (60% [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), 30% [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk), 10% Enron email from training senders), half of it as fill-in-the-middle examples. 28 minutes on one A100 (about $1.50) plus 20 minutes of data prep on a CPU job (about $0.34). Validation loss 2.99. [Training curves on Trackio](https://huggingface.co/spaces/cfahlgren1/tinynext-trackio).
113
 
114
  **Limits.**
115
 
116
- - Text after the cursor doesn't help yet: when there is some, v2 does slightly worse than with the text before alone. Experiments are under way.
117
  - It still leans toward office email, and it only sees keyboard characters.
118
  - It's English only.
119
 
@@ -128,7 +156,8 @@ The browser computes in float32 on purpose: onnxruntime-web's int8 kernels were
128
  | `v2/train.py` | Data prep, training, eval, export and upload in one script |
129
  | `v2/eval.py`, `v2/eval_set.json` | The eval harness and the fixed text it scores |
130
  | `v2/export.py`, `v2/compress.py` | Re-export to ONNX, and the int8 download format |
131
- | `v2/metrics/*.json` | Eval results for v1, v2 and SmolLM2-135M |
 
132
  | `v2/report.json` | Training settings, loss history and export checks |
133
 
134
  </details>
@@ -138,8 +167,8 @@ The browser computes in float32 on purpose: onnxruntime-web's int8 kernels were
138
 
139
  ```bash
140
  # Prepare data once on a CPU job, then train on a GPU from the prepared files.
141
- hf jobs uv run --flavor cpu-xl --secrets HF_TOKEN train.py --stage prepare
142
- hf jobs uv run --flavor a100-large --secrets HF_TOKEN train.py --stage train
143
  ```
144
 
145
  </details>
 
27
  metrics:
28
  - type: typing_saved
29
  name: Typing saved (%)
30
+ value: 3.9
31
  - type: suggestion_precision
32
  name: Suggestions that were right (%)
33
+ value: 78
34
  - task:
35
  type: text-generation
36
  name: Autocomplete (typing saved at a 50% cutoff)
 
40
  metrics:
41
  - type: typing_saved
42
  name: Typing saved (%)
43
+ value: 8.8
44
  - type: suggestion_precision
45
  name: Suggestions that were right (%)
46
+ value: 75
47
  - task:
48
  type: text-generation
49
  name: Autocomplete (typing saved at a 50% cutoff)
 
53
  metrics:
54
  - type: typing_saved
55
  name: Typing saved (%)
56
+ value: 4.0
57
  - type: suggestion_precision
58
  name: Suggestions that were right (%)
59
+ value: 78
60
  ---
61
 
62
  <p align="center">
 
89
 
90
  | Writing | v1 | v2 | SmolLM2-135M (ceiling, ~100 MB) |
91
  |---|---|---|---|
92
+ | Articles (Wikipedia) | 1.5% | **3.9%** | 8.1% |
93
+ | Messages (OpenAssistant) | 2.0% | **8.8%** | 16.8% |
94
+ | Email (Enron, unseen senders) | 3.0% | **4.0%** | 5.7% |
95
 
96
  | Writing (v2) | Suggests at | Right | 2+ words suggested at | 2+ words right |
97
  |---|---|---|---|---|
98
+ | Articles (Wikipedia) | 10% | 78% | 4% | 71% |
99
+ | Messages (OpenAssistant) | 19% | 75% | 9% | 70% |
100
+ | Email (Enron, unseen senders) | 11% | 78% | 4% | 51% |
101
+
102
+ **Which v2.** Three runs of the same 12.5M-parameter model, typing saved on articles / messages / email. *Editing* is the same test with text after the cursor, where fill-in-the-middle can use it.
103
+
104
+ | Run | Training tokens | Fill-in-the-middle | Typing | Editing | Val loss |
105
+ |---|---|---|---|---|---|
106
+ | **v2 (this one)** | **3B** | half the examples | **3.9 / 8.8 / 4.0** | **3.1 / 7.6 / 4.2** | **2.89** |
107
+ | First v2 | 1.5B | half the examples | 4.0 / 7.3 / 4.3 | 2.8 / 6.3 / 3.6 | 2.99 |
108
+ | [No FIM](v2/experiments/tinynext-v2-nofim) | 1.5B | none | 3.9 / 8.8 / 4.2 | 3.0 / 6.9 / 4.1 | 2.94 |
109
+
110
+ At 1.5B tokens, fill-in-the-middle cost the model more than it gave: half its training went to a task it rarely uses. With twice the tokens it stops costing anything on typing and helps when editing. The first v2 is in this repo's history.
111
+
112
+ <details>
113
+ <summary><b>Would a bigger model help?</b></summary>
114
+
115
+ [Pythia](https://huggingface.co/EleutherAI/pythia-160m) models are trained the same way at every size, so they show what size alone buys. Typing saved, 300 pauses per kind of writing (tinynext: 1,000):
116
+
117
+ | Model | Parameters | Articles | Messages | Email |
118
+ |---|---|---|---|---|
119
+ | Pythia | 14M | 3.6% | 6.1% | 2.4% |
120
+ | Pythia | 31M | 4.0% | 6.6% | 4.0% |
121
+ | Pythia | 70M | 7.0% | 8.0% | 4.9% |
122
+ | Pythia | 160M | 7.3% | 10.5% | 7.9% |
123
+ | SmolLM2 | 135M | 8.1% | 16.8% | 5.7% |
124
+ | **tinynext v2** | **12.5M** | **3.9%** | **8.8%** | **4.0%** |
125
+
126
+ tinynext matches Pythia models 2–3× its size because it trained on writing like what people type. Size starts to pay off between 30M and 70M parameters, and better data (SmolLM2) matters as much as size.
127
+
128
+ </details>
129
 
130
  **In the browser.**
131
 
132
  | | |
133
  |---|---|
134
  | Download | **13.3 MB**: int8 weights with one scale per 64 values, expanded to float32 when the page loads |
135
+ | Matches full precision | same next word 96% of the time, same 5-word path 90% |
136
+ | Speed | **5 ms** per suggestion (13 ms p95) in a Web Worker, Chrome on an M3 Max; typing stays under one frame |
137
 
138
  The browser computes in float32 on purpose: onnxruntime-web's int8 kernels were 8–11× slower than its float kernels for this model, and int8 activations changed the top word too often. Only the download is compressed.
139
 
140
+ **Training.** 3B tokens (60% [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), 30% [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk), 10% Enron email from training senders), half of it as fill-in-the-middle examples. 56 minutes on one A100 at 920K tokens a second, from data tokenized once on a CPU job. Validation loss 2.89. [Training curves on Trackio](https://huggingface.co/spaces/cfahlgren1/tinynext-trackio).
141
 
142
  **Limits.**
143
 
144
+ - offline.page only suggests at the end of a paragraph, so the text after the cursor is the next paragraph. Mid-line suggestions, where fill-in-the-middle should help most, aren't built yet.
145
  - It still leans toward office email, and it only sees keyboard characters.
146
  - It's English only.
147
 
 
156
  | `v2/train.py` | Data prep, training, eval, export and upload in one script |
157
  | `v2/eval.py`, `v2/eval_set.json` | The eval harness and the fixed text it scores |
158
  | `v2/export.py`, `v2/compress.py` | Re-export to ONNX, and the int8 download format |
159
+ | `v2/metrics/*.json` | Eval results for v1, v2, the browser format, SmolLM2-135M and Pythia 14M–160M |
160
+ | `v2/experiments/` | The no-FIM run, and the 3B-token run as trained (the same weights as `v2/`) |
161
  | `v2/report.json` | Training settings, loss history and export checks |
162
 
163
  </details>
 
167
 
168
  ```bash
169
  # Prepare data once on a CPU job, then train on a GPU from the prepared files.
170
+ hf jobs uv run --flavor cpu-xl --secrets HF_TOKEN train.py --stage prepare --tokens 3e9 --fineweb-files 4
171
+ hf jobs uv run --flavor a100-large --secrets HF_TOKEN train.py --stage train --tokens 3e9 --fineweb-files 4
172
  ```
173
 
174
  </details>
v2/eval.py CHANGED
@@ -348,6 +348,10 @@ def hf_causal_lm(name: str, repo: str) -> TokenLM:
348
 
349
  texts = [tokenizer.decode([i]) for i in range(len(tokenizer))]
350
  stops = {tokenizer.eos_token_id} if tokenizer.eos_token_id is not None else set()
 
 
 
 
351
  encode = lambda text: tokenizer.encode(text, add_special_tokens=False) or [tokenizer.eos_token_id]
352
  return TokenLM(name, next_logits, encode, texts, stops)
353
 
 
348
 
349
  texts = [tokenizer.decode([i]) for i in range(len(tokenizer))]
350
  stops = {tokenizer.eos_token_id} if tokenizer.eos_token_id is not None else set()
351
+ # Some models pad the output layer past the tokenizer (Pythia: 50,304 vs 50,277). Those ids are never real text.
352
+ padded = model.get_output_embeddings().weight.shape[0]
353
+ stops |= set(range(len(texts), padded))
354
+ texts += [""] * (padded - len(texts))
355
  encode = lambda text: tokenizer.encode(text, add_special_tokens=False) or [tokenizer.eos_token_id]
356
  return TokenLM(name, next_logits, encode, texts, stops)
357
 
v2/metrics/pythia-14m.json ADDED
@@ -0,0 +1,411 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "pythia-14m",
3
+ "sees_after": false,
4
+ "cutoffs": {
5
+ "0.3": {
6
+ "articles/typing": {
7
+ "pauses": 300,
8
+ "shown": 0.1767,
9
+ "right": 0.4717,
10
+ "avg_right_chars": 5.72,
11
+ "multiword_shown": 0.0767,
12
+ "multiword_right": 0.3913,
13
+ "saved_with_tab": 0.0397,
14
+ "saved_word_by_word": 0.0478
15
+ },
16
+ "articles/editing": {
17
+ "pauses": 300,
18
+ "shown": 0.1667,
19
+ "right": 0.56,
20
+ "avg_right_chars": 5.57,
21
+ "multiword_shown": 0.0767,
22
+ "multiword_right": 0.5652,
23
+ "saved_with_tab": 0.0433,
24
+ "saved_word_by_word": 0.05
25
+ },
26
+ "messages/typing": {
27
+ "pauses": 300,
28
+ "shown": 0.2067,
29
+ "right": 0.6613,
30
+ "avg_right_chars": 6.83,
31
+ "multiword_shown": 0.1133,
32
+ "multiword_right": 0.7059,
33
+ "saved_with_tab": 0.0778,
34
+ "saved_word_by_word": 0.0864
35
+ },
36
+ "messages/editing": {
37
+ "pauses": 300,
38
+ "shown": 0.19,
39
+ "right": 0.5439,
40
+ "avg_right_chars": 6.81,
41
+ "multiword_shown": 0.1133,
42
+ "multiword_right": 0.5294,
43
+ "saved_with_tab": 0.0586,
44
+ "saved_word_by_word": 0.0728
45
+ },
46
+ "email/typing": {
47
+ "pauses": 300,
48
+ "shown": 0.1467,
49
+ "right": 0.6364,
50
+ "avg_right_chars": 5.18,
51
+ "multiword_shown": 0.0433,
52
+ "multiword_right": 0.5385,
53
+ "saved_with_tab": 0.0403,
54
+ "saved_word_by_word": 0.0414
55
+ },
56
+ "email/editing": {
57
+ "pauses": 300,
58
+ "shown": 0.1567,
59
+ "right": 0.6383,
60
+ "avg_right_chars": 5.1,
61
+ "multiword_shown": 0.0533,
62
+ "multiword_right": 0.4375,
63
+ "saved_with_tab": 0.0425,
64
+ "saved_word_by_word": 0.045
65
+ }
66
+ },
67
+ "0.4": {
68
+ "articles/typing": {
69
+ "pauses": 300,
70
+ "shown": 0.14,
71
+ "right": 0.5238,
72
+ "avg_right_chars": 5.86,
73
+ "multiword_shown": 0.0533,
74
+ "multiword_right": 0.4375,
75
+ "saved_with_tab": 0.0358,
76
+ "saved_word_by_word": 0.0414
77
+ },
78
+ "articles/editing": {
79
+ "pauses": 300,
80
+ "shown": 0.12,
81
+ "right": 0.6667,
82
+ "avg_right_chars": 5.58,
83
+ "multiword_shown": 0.0433,
84
+ "multiword_right": 0.7692,
85
+ "saved_with_tab": 0.0372,
86
+ "saved_word_by_word": 0.0392
87
+ },
88
+ "messages/typing": {
89
+ "pauses": 300,
90
+ "shown": 0.1633,
91
+ "right": 0.7755,
92
+ "avg_right_chars": 6.45,
93
+ "multiword_shown": 0.07,
94
+ "multiword_right": 0.8571,
95
+ "saved_with_tab": 0.0681,
96
+ "saved_word_by_word": 0.0708
97
+ },
98
+ "messages/editing": {
99
+ "pauses": 300,
100
+ "shown": 0.13,
101
+ "right": 0.6667,
102
+ "avg_right_chars": 6.54,
103
+ "multiword_shown": 0.0533,
104
+ "multiword_right": 0.5625,
105
+ "saved_with_tab": 0.0472,
106
+ "saved_word_by_word": 0.0522
107
+ },
108
+ "email/typing": {
109
+ "pauses": 300,
110
+ "shown": 0.0933,
111
+ "right": 0.8214,
112
+ "avg_right_chars": 5.09,
113
+ "multiword_shown": 0.0167,
114
+ "multiword_right": 0.8,
115
+ "saved_with_tab": 0.0325,
116
+ "saved_word_by_word": 0.0328
117
+ },
118
+ "email/editing": {
119
+ "pauses": 300,
120
+ "shown": 0.11,
121
+ "right": 0.7576,
122
+ "avg_right_chars": 4.92,
123
+ "multiword_shown": 0.0267,
124
+ "multiword_right": 0.625,
125
+ "saved_with_tab": 0.0342,
126
+ "saved_word_by_word": 0.035
127
+ }
128
+ },
129
+ "0.5": {
130
+ "articles/typing": {
131
+ "pauses": 300,
132
+ "shown": 0.1133,
133
+ "right": 0.6765,
134
+ "avg_right_chars": 5.57,
135
+ "multiword_shown": 0.0233,
136
+ "multiword_right": 0.7143,
137
+ "saved_with_tab": 0.0356,
138
+ "saved_word_by_word": 0.0364
139
+ },
140
+ "articles/editing": {
141
+ "pauses": 300,
142
+ "shown": 0.09,
143
+ "right": 0.8148,
144
+ "avg_right_chars": 5.36,
145
+ "multiword_shown": 0.0367,
146
+ "multiword_right": 0.8182,
147
+ "saved_with_tab": 0.0328,
148
+ "saved_word_by_word": 0.0347
149
+ },
150
+ "messages/typing": {
151
+ "pauses": 300,
152
+ "shown": 0.13,
153
+ "right": 0.8974,
154
+ "avg_right_chars": 6.23,
155
+ "multiword_shown": 0.0467,
156
+ "multiword_right": 0.9286,
157
+ "saved_with_tab": 0.0606,
158
+ "saved_word_by_word": 0.0606
159
+ },
160
+ "messages/editing": {
161
+ "pauses": 300,
162
+ "shown": 0.1,
163
+ "right": 0.7333,
164
+ "avg_right_chars": 5.86,
165
+ "multiword_shown": 0.0267,
166
+ "multiword_right": 0.625,
167
+ "saved_with_tab": 0.0358,
168
+ "saved_word_by_word": 0.0389
169
+ },
170
+ "email/typing": {
171
+ "pauses": 300,
172
+ "shown": 0.0667,
173
+ "right": 0.85,
174
+ "avg_right_chars": 5.06,
175
+ "multiword_shown": 0.0067,
176
+ "multiword_right": 1.0,
177
+ "saved_with_tab": 0.0239,
178
+ "saved_word_by_word": 0.0239
179
+ },
180
+ "email/editing": {
181
+ "pauses": 300,
182
+ "shown": 0.08,
183
+ "right": 0.9167,
184
+ "avg_right_chars": 4.68,
185
+ "multiword_shown": 0.01,
186
+ "multiword_right": 1.0,
187
+ "saved_with_tab": 0.0286,
188
+ "saved_word_by_word": 0.0286
189
+ }
190
+ },
191
+ "0.6": {
192
+ "articles/typing": {
193
+ "pauses": 300,
194
+ "shown": 0.0933,
195
+ "right": 0.7143,
196
+ "avg_right_chars": 5.65,
197
+ "multiword_shown": 0.02,
198
+ "multiword_right": 0.6667,
199
+ "saved_with_tab": 0.0314,
200
+ "saved_word_by_word": 0.0322
201
+ },
202
+ "articles/editing": {
203
+ "pauses": 300,
204
+ "shown": 0.0767,
205
+ "right": 0.913,
206
+ "avg_right_chars": 5.48,
207
+ "multiword_shown": 0.03,
208
+ "multiword_right": 0.8889,
209
+ "saved_with_tab": 0.0319,
210
+ "saved_word_by_word": 0.0325
211
+ },
212
+ "messages/typing": {
213
+ "pauses": 300,
214
+ "shown": 0.1067,
215
+ "right": 0.9062,
216
+ "avg_right_chars": 6.0,
217
+ "multiword_shown": 0.0333,
218
+ "multiword_right": 1.0,
219
+ "saved_with_tab": 0.0483,
220
+ "saved_word_by_word": 0.0483
221
+ },
222
+ "messages/editing": {
223
+ "pauses": 300,
224
+ "shown": 0.0667,
225
+ "right": 0.9,
226
+ "avg_right_chars": 6.06,
227
+ "multiword_shown": 0.0167,
228
+ "multiword_right": 0.6,
229
+ "saved_with_tab": 0.0303,
230
+ "saved_word_by_word": 0.0328
231
+ },
232
+ "email/typing": {
233
+ "pauses": 300,
234
+ "shown": 0.0433,
235
+ "right": 0.8462,
236
+ "avg_right_chars": 4.73,
237
+ "multiword_shown": 0.0033,
238
+ "multiword_right": 1.0,
239
+ "saved_with_tab": 0.0144,
240
+ "saved_word_by_word": 0.0144
241
+ },
242
+ "email/editing": {
243
+ "pauses": 300,
244
+ "shown": 0.0533,
245
+ "right": 0.9375,
246
+ "avg_right_chars": 4.73,
247
+ "multiword_shown": 0.0067,
248
+ "multiword_right": 1.0,
249
+ "saved_with_tab": 0.0197,
250
+ "saved_word_by_word": 0.0197
251
+ }
252
+ },
253
+ "0.7": {
254
+ "articles/typing": {
255
+ "pauses": 300,
256
+ "shown": 0.0667,
257
+ "right": 0.85,
258
+ "avg_right_chars": 5.65,
259
+ "multiword_shown": 0.0167,
260
+ "multiword_right": 0.6,
261
+ "saved_with_tab": 0.0267,
262
+ "saved_word_by_word": 0.0275
263
+ },
264
+ "articles/editing": {
265
+ "pauses": 300,
266
+ "shown": 0.0567,
267
+ "right": 0.8824,
268
+ "avg_right_chars": 5.47,
269
+ "multiword_shown": 0.0267,
270
+ "multiword_right": 0.875,
271
+ "saved_with_tab": 0.0228,
272
+ "saved_word_by_word": 0.0233
273
+ },
274
+ "messages/typing": {
275
+ "pauses": 300,
276
+ "shown": 0.08,
277
+ "right": 0.9583,
278
+ "avg_right_chars": 5.61,
279
+ "multiword_shown": 0.0167,
280
+ "multiword_right": 1.0,
281
+ "saved_with_tab": 0.0358,
282
+ "saved_word_by_word": 0.0358
283
+ },
284
+ "messages/editing": {
285
+ "pauses": 300,
286
+ "shown": 0.04,
287
+ "right": 1.0,
288
+ "avg_right_chars": 5.83,
289
+ "multiword_shown": 0.0133,
290
+ "multiword_right": 1.0,
291
+ "saved_with_tab": 0.0194,
292
+ "saved_word_by_word": 0.0194
293
+ },
294
+ "email/typing": {
295
+ "pauses": 300,
296
+ "shown": 0.0267,
297
+ "right": 0.875,
298
+ "avg_right_chars": 4.43,
299
+ "multiword_shown": 0.0,
300
+ "multiword_right": 0.0,
301
+ "saved_with_tab": 0.0086,
302
+ "saved_word_by_word": 0.0086
303
+ },
304
+ "email/editing": {
305
+ "pauses": 300,
306
+ "shown": 0.03,
307
+ "right": 0.8889,
308
+ "avg_right_chars": 4.5,
309
+ "multiword_shown": 0.0,
310
+ "multiword_right": 0.0,
311
+ "saved_with_tab": 0.01,
312
+ "saved_word_by_word": 0.01
313
+ }
314
+ }
315
+ },
316
+ "examples": [
317
+ {
318
+ "domain": "articles",
319
+ "scenario": "typing",
320
+ "before": "epted an athletic scholarship to attend the University of Fl",
321
+ "suggestion": "orida",
322
+ "truth": "orida in Gainesvi",
323
+ "right": true
324
+ },
325
+ {
326
+ "domain": "articles",
327
+ "scenario": "typing",
328
+ "before": "lade, however, due to the club's involvement in the 2006 Ita",
329
+ "suggestion": "lian",
330
+ "truth": "lian football sc",
331
+ "right": true
332
+ },
333
+ {
334
+ "domain": "messages",
335
+ "scenario": "typing",
336
+ "before": "T's ability to generate coherent and contextually relevant r",
337
+ "suggestion": "esponses",
338
+ "truth": "esponses is based on",
339
+ "right": true
340
+ },
341
+ {
342
+ "domain": "messages",
343
+ "scenario": "typing",
344
+ "before": " prioritize the development of tools and models that are des",
345
+ "suggestion": "igned to",
346
+ "truth": "igned to be inclusiv",
347
+ "right": true
348
+ },
349
+ {
350
+ "domain": "messages",
351
+ "scenario": "typing",
352
+ "before": "n the corner of the CPU and the socket that you align to ens",
353
+ "suggestion": "ures",
354
+ "truth": "ure it's install",
355
+ "right": false
356
+ },
357
+ {
358
+ "domain": "messages",
359
+ "scenario": "editing",
360
+ "before": "e at which a population is growing.\nL'Hopital's rule can als",
361
+ "suggestion": "o be used",
362
+ "truth": "o be used to solve pr",
363
+ "right": true
364
+ },
365
+ {
366
+ "domain": "messages",
367
+ "scenario": "editing",
368
+ "before": " to choose exactly which components you want to use, and cus",
369
+ "suggestion": "tomize",
370
+ "truth": "tomize your machin",
371
+ "right": true
372
+ },
373
+ {
374
+ "domain": "messages",
375
+ "scenario": "editing",
376
+ "before": " will then want to apply a small pea sized blob to the cente",
377
+ "suggestion": "r of",
378
+ "truth": "r of the CPU. In",
379
+ "right": true
380
+ },
381
+ {
382
+ "domain": "email",
383
+ "scenario": "typing",
384
+ "before": "lt time but am confident that I would make an excellent addi",
385
+ "suggestion": "tion to",
386
+ "truth": "tion to your team f",
387
+ "right": true
388
+ },
389
+ {
390
+ "domain": "email",
391
+ "scenario": "typing",
392
+ "before": ". I recommend we agree. Although, we are in the content deve",
393
+ "suggestion": "lopment",
394
+ "truth": "lopment phase the t",
395
+ "right": true
396
+ },
397
+ {
398
+ "domain": "email",
399
+ "scenario": "editing",
400
+ "before": "ured products and marketing of electricity for the United St",
401
+ "suggestion": "ates",
402
+ "truth": "ates ? Actively ",
403
+ "right": true
404
+ }
405
+ ],
406
+ "latency_ms": {
407
+ "p50": 35.11,
408
+ "p95": 176.56,
409
+ "note": "Measured in this eval process, not in a browser."
410
+ }
411
+ }
v2/metrics/pythia-160m.json ADDED
@@ -0,0 +1,435 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "pythia-160m",
3
+ "sees_after": false,
4
+ "cutoffs": {
5
+ "0.3": {
6
+ "articles/typing": {
7
+ "pauses": 300,
8
+ "shown": 0.3433,
9
+ "right": 0.6019,
10
+ "avg_right_chars": 6.74,
11
+ "multiword_shown": 0.1867,
12
+ "multiword_right": 0.6071,
13
+ "saved_with_tab": 0.1161,
14
+ "saved_word_by_word": 0.1294
15
+ },
16
+ "articles/editing": {
17
+ "pauses": 300,
18
+ "shown": 0.2967,
19
+ "right": 0.6854,
20
+ "avg_right_chars": 6.85,
21
+ "multiword_shown": 0.16,
22
+ "multiword_right": 0.6042,
23
+ "saved_with_tab": 0.1161,
24
+ "saved_word_by_word": 0.1314
25
+ },
26
+ "messages/typing": {
27
+ "pauses": 300,
28
+ "shown": 0.36,
29
+ "right": 0.6111,
30
+ "avg_right_chars": 7.82,
31
+ "multiword_shown": 0.2233,
32
+ "multiword_right": 0.6119,
33
+ "saved_with_tab": 0.1433,
34
+ "saved_word_by_word": 0.1611
35
+ },
36
+ "messages/editing": {
37
+ "pauses": 300,
38
+ "shown": 0.3567,
39
+ "right": 0.5981,
40
+ "avg_right_chars": 7.56,
41
+ "multiword_shown": 0.2,
42
+ "multiword_right": 0.6,
43
+ "saved_with_tab": 0.1344,
44
+ "saved_word_by_word": 0.1558
45
+ },
46
+ "email/typing": {
47
+ "pauses": 300,
48
+ "shown": 0.2733,
49
+ "right": 0.622,
50
+ "avg_right_chars": 6.02,
51
+ "multiword_shown": 0.0867,
52
+ "multiword_right": 0.5,
53
+ "saved_with_tab": 0.0853,
54
+ "saved_word_by_word": 0.0956
55
+ },
56
+ "email/editing": {
57
+ "pauses": 300,
58
+ "shown": 0.25,
59
+ "right": 0.6533,
60
+ "avg_right_chars": 6.16,
61
+ "multiword_shown": 0.1,
62
+ "multiword_right": 0.6333,
63
+ "saved_with_tab": 0.0839,
64
+ "saved_word_by_word": 0.0889
65
+ }
66
+ },
67
+ "0.4": {
68
+ "articles/typing": {
69
+ "pauses": 300,
70
+ "shown": 0.26,
71
+ "right": 0.7051,
72
+ "avg_right_chars": 6.31,
73
+ "multiword_shown": 0.1033,
74
+ "multiword_right": 0.7097,
75
+ "saved_with_tab": 0.0964,
76
+ "saved_word_by_word": 0.1019
77
+ },
78
+ "articles/editing": {
79
+ "pauses": 300,
80
+ "shown": 0.2267,
81
+ "right": 0.75,
82
+ "avg_right_chars": 6.69,
83
+ "multiword_shown": 0.11,
84
+ "multiword_right": 0.7273,
85
+ "saved_with_tab": 0.0947,
86
+ "saved_word_by_word": 0.1031
87
+ },
88
+ "messages/typing": {
89
+ "pauses": 300,
90
+ "shown": 0.29,
91
+ "right": 0.7356,
92
+ "avg_right_chars": 7.34,
93
+ "multiword_shown": 0.1733,
94
+ "multiword_right": 0.6731,
95
+ "saved_with_tab": 0.1306,
96
+ "saved_word_by_word": 0.1425
97
+ },
98
+ "messages/editing": {
99
+ "pauses": 300,
100
+ "shown": 0.2367,
101
+ "right": 0.7324,
102
+ "avg_right_chars": 6.85,
103
+ "multiword_shown": 0.1167,
104
+ "multiword_right": 0.6857,
105
+ "saved_with_tab": 0.0989,
106
+ "saved_word_by_word": 0.1097
107
+ },
108
+ "email/typing": {
109
+ "pauses": 300,
110
+ "shown": 0.22,
111
+ "right": 0.697,
112
+ "avg_right_chars": 6.02,
113
+ "multiword_shown": 0.0567,
114
+ "multiword_right": 0.6471,
115
+ "saved_with_tab": 0.0769,
116
+ "saved_word_by_word": 0.0828
117
+ },
118
+ "email/editing": {
119
+ "pauses": 300,
120
+ "shown": 0.1933,
121
+ "right": 0.7241,
122
+ "avg_right_chars": 6.14,
123
+ "multiword_shown": 0.0767,
124
+ "multiword_right": 0.6522,
125
+ "saved_with_tab": 0.0717,
126
+ "saved_word_by_word": 0.0756
127
+ }
128
+ },
129
+ "0.5": {
130
+ "articles/typing": {
131
+ "pauses": 300,
132
+ "shown": 0.2033,
133
+ "right": 0.7541,
134
+ "avg_right_chars": 5.74,
135
+ "multiword_shown": 0.0533,
136
+ "multiword_right": 0.8125,
137
+ "saved_with_tab": 0.0733,
138
+ "saved_word_by_word": 0.075
139
+ },
140
+ "articles/editing": {
141
+ "pauses": 300,
142
+ "shown": 0.18,
143
+ "right": 0.8333,
144
+ "avg_right_chars": 6.29,
145
+ "multiword_shown": 0.0833,
146
+ "multiword_right": 0.8,
147
+ "saved_with_tab": 0.0786,
148
+ "saved_word_by_word": 0.0839
149
+ },
150
+ "messages/typing": {
151
+ "pauses": 300,
152
+ "shown": 0.2133,
153
+ "right": 0.8125,
154
+ "avg_right_chars": 7.27,
155
+ "multiword_shown": 0.11,
156
+ "multiword_right": 0.7273,
157
+ "saved_with_tab": 0.105,
158
+ "saved_word_by_word": 0.1097
159
+ },
160
+ "messages/editing": {
161
+ "pauses": 300,
162
+ "shown": 0.1733,
163
+ "right": 0.7885,
164
+ "avg_right_chars": 6.59,
165
+ "multiword_shown": 0.08,
166
+ "multiword_right": 0.7083,
167
+ "saved_with_tab": 0.075,
168
+ "saved_word_by_word": 0.0839
169
+ },
170
+ "email/typing": {
171
+ "pauses": 300,
172
+ "shown": 0.19,
173
+ "right": 0.8596,
174
+ "avg_right_chars": 5.78,
175
+ "multiword_shown": 0.04,
176
+ "multiword_right": 0.9167,
177
+ "saved_with_tab": 0.0786,
178
+ "saved_word_by_word": 0.0789
179
+ },
180
+ "email/editing": {
181
+ "pauses": 300,
182
+ "shown": 0.1467,
183
+ "right": 0.8636,
184
+ "avg_right_chars": 5.58,
185
+ "multiword_shown": 0.05,
186
+ "multiword_right": 0.8,
187
+ "saved_with_tab": 0.0589,
188
+ "saved_word_by_word": 0.0603
189
+ }
190
+ },
191
+ "0.6": {
192
+ "articles/typing": {
193
+ "pauses": 300,
194
+ "shown": 0.15,
195
+ "right": 0.8,
196
+ "avg_right_chars": 5.72,
197
+ "multiword_shown": 0.0267,
198
+ "multiword_right": 0.875,
199
+ "saved_with_tab": 0.0572,
200
+ "saved_word_by_word": 0.0575
201
+ },
202
+ "articles/editing": {
203
+ "pauses": 300,
204
+ "shown": 0.1367,
205
+ "right": 0.878,
206
+ "avg_right_chars": 6.31,
207
+ "multiword_shown": 0.0667,
208
+ "multiword_right": 0.8,
209
+ "saved_with_tab": 0.0631,
210
+ "saved_word_by_word": 0.0672
211
+ },
212
+ "messages/typing": {
213
+ "pauses": 300,
214
+ "shown": 0.1667,
215
+ "right": 0.86,
216
+ "avg_right_chars": 6.47,
217
+ "multiword_shown": 0.0833,
218
+ "multiword_right": 0.76,
219
+ "saved_with_tab": 0.0772,
220
+ "saved_word_by_word": 0.0814
221
+ },
222
+ "messages/editing": {
223
+ "pauses": 300,
224
+ "shown": 0.1367,
225
+ "right": 0.878,
226
+ "avg_right_chars": 6.36,
227
+ "multiword_shown": 0.06,
228
+ "multiword_right": 0.8333,
229
+ "saved_with_tab": 0.0636,
230
+ "saved_word_by_word": 0.0686
231
+ },
232
+ "email/typing": {
233
+ "pauses": 300,
234
+ "shown": 0.1633,
235
+ "right": 0.8571,
236
+ "avg_right_chars": 5.43,
237
+ "multiword_shown": 0.0233,
238
+ "multiword_right": 0.8571,
239
+ "saved_with_tab": 0.0633,
240
+ "saved_word_by_word": 0.0636
241
+ },
242
+ "email/editing": {
243
+ "pauses": 300,
244
+ "shown": 0.1167,
245
+ "right": 0.8857,
246
+ "avg_right_chars": 5.39,
247
+ "multiword_shown": 0.0333,
248
+ "multiword_right": 0.8,
249
+ "saved_with_tab": 0.0464,
250
+ "saved_word_by_word": 0.0475
251
+ }
252
+ },
253
+ "0.7": {
254
+ "articles/typing": {
255
+ "pauses": 300,
256
+ "shown": 0.12,
257
+ "right": 0.8611,
258
+ "avg_right_chars": 5.94,
259
+ "multiword_shown": 0.0267,
260
+ "multiword_right": 0.875,
261
+ "saved_with_tab": 0.0511,
262
+ "saved_word_by_word": 0.0514
263
+ },
264
+ "articles/editing": {
265
+ "pauses": 300,
266
+ "shown": 0.1067,
267
+ "right": 1.0,
268
+ "avg_right_chars": 5.41,
269
+ "multiword_shown": 0.0367,
270
+ "multiword_right": 1.0,
271
+ "saved_with_tab": 0.0481,
272
+ "saved_word_by_word": 0.0481
273
+ },
274
+ "messages/typing": {
275
+ "pauses": 300,
276
+ "shown": 0.1333,
277
+ "right": 0.9,
278
+ "avg_right_chars": 5.86,
279
+ "multiword_shown": 0.0467,
280
+ "multiword_right": 0.7143,
281
+ "saved_with_tab": 0.0586,
282
+ "saved_word_by_word": 0.0611
283
+ },
284
+ "messages/editing": {
285
+ "pauses": 300,
286
+ "shown": 0.0967,
287
+ "right": 0.9655,
288
+ "avg_right_chars": 5.96,
289
+ "multiword_shown": 0.0367,
290
+ "multiword_right": 0.9091,
291
+ "saved_with_tab": 0.0464,
292
+ "saved_word_by_word": 0.0481
293
+ },
294
+ "email/typing": {
295
+ "pauses": 300,
296
+ "shown": 0.11,
297
+ "right": 0.9697,
298
+ "avg_right_chars": 5.53,
299
+ "multiword_shown": 0.0133,
300
+ "multiword_right": 1.0,
301
+ "saved_with_tab": 0.0492,
302
+ "saved_word_by_word": 0.0492
303
+ },
304
+ "email/editing": {
305
+ "pauses": 300,
306
+ "shown": 0.08,
307
+ "right": 0.9583,
308
+ "avg_right_chars": 5.43,
309
+ "multiword_shown": 0.0133,
310
+ "multiword_right": 0.75,
311
+ "saved_with_tab": 0.0347,
312
+ "saved_word_by_word": 0.0353
313
+ }
314
+ }
315
+ },
316
+ "examples": [
317
+ {
318
+ "domain": "articles",
319
+ "scenario": "typing",
320
+ "before": "epted an athletic scholarship to attend the University of Fl",
321
+ "suggestion": "orida",
322
+ "truth": "orida in Gainesvi",
323
+ "right": true
324
+ },
325
+ {
326
+ "domain": "articles",
327
+ "scenario": "typing",
328
+ "before": "lade, however, due to the club's involvement in the 2006 Ita",
329
+ "suggestion": "lian",
330
+ "truth": "lian football sc",
331
+ "right": true
332
+ },
333
+ {
334
+ "domain": "articles",
335
+ "scenario": "typing",
336
+ "before": "Street/Perimeter Road, just south of the Miami International",
337
+ "suggestion": " Airport.",
338
+ "truth": " Airport runway.",
339
+ "right": false
340
+ },
341
+ {
342
+ "domain": "articles",
343
+ "scenario": "typing",
344
+ "before": "Travel, a sports-related travel agency, and coaches figure s",
345
+ "suggestion": "kating",
346
+ "truth": "katers at the Extr",
347
+ "right": false
348
+ },
349
+ {
350
+ "domain": "articles",
351
+ "scenario": "editing",
352
+ "before": "le tree, with a number of ecotypes, and in forms closely res",
353
+ "suggestion": "embling",
354
+ "truth": "embling other speci",
355
+ "right": true
356
+ },
357
+ {
358
+ "domain": "messages",
359
+ "scenario": "typing",
360
+ "before": "T's ability to generate coherent and contextually relevant r",
361
+ "suggestion": "esponses",
362
+ "truth": "esponses is based on",
363
+ "right": true
364
+ },
365
+ {
366
+ "domain": "messages",
367
+ "scenario": "typing",
368
+ "before": " prioritize the development of tools and models that are des",
369
+ "suggestion": "igned to",
370
+ "truth": "igned to be inclusiv",
371
+ "right": true
372
+ },
373
+ {
374
+ "domain": "messages",
375
+ "scenario": "editing",
376
+ "before": "e at which a population is growing.\nL'Hopital's rule can als",
377
+ "suggestion": "o be used to",
378
+ "truth": "o be used to solve probl",
379
+ "right": true
380
+ },
381
+ {
382
+ "domain": "messages",
383
+ "scenario": "editing",
384
+ "before": " will then want to apply a small pea sized blob to the cente",
385
+ "suggestion": "r of the",
386
+ "truth": "r of the CPU. Instal",
387
+ "right": true
388
+ },
389
+ {
390
+ "domain": "email",
391
+ "scenario": "typing",
392
+ "before": "lt time but am confident that I would make an excellent addi",
393
+ "suggestion": "tion to",
394
+ "truth": "tion to your team f",
395
+ "right": true
396
+ },
397
+ {
398
+ "domain": "email",
399
+ "scenario": "typing",
400
+ "before": ". I recommend we agree. Although, we are in the content deve",
401
+ "suggestion": "lopment",
402
+ "truth": "lopment phase the t",
403
+ "right": true
404
+ },
405
+ {
406
+ "domain": "email",
407
+ "scenario": "typing",
408
+ "before": "f-time supplement to timely comments than to file the whole ",
409
+ "suggestion": "thing",
410
+ "truth": "thing late, parti",
411
+ "right": true
412
+ },
413
+ {
414
+ "domain": "email",
415
+ "scenario": "editing",
416
+ "before": "ured products and marketing of electricity for the United St",
417
+ "suggestion": "ates",
418
+ "truth": "ates ? Actively ",
419
+ "right": true
420
+ },
421
+ {
422
+ "domain": "email",
423
+ "scenario": "editing",
424
+ "before": " action is necessary as contractually we are coming to the e",
425
+ "suggestion": "nd of",
426
+ "truth": "nd of the accepta",
427
+ "right": true
428
+ }
429
+ ],
430
+ "latency_ms": {
431
+ "p50": 65.63,
432
+ "p95": 231.75,
433
+ "note": "Measured in this eval process, not in a browser."
434
+ }
435
+ }
v2/metrics/pythia-31m.json ADDED
@@ -0,0 +1,427 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "pythia-31m",
3
+ "sees_after": false,
4
+ "cutoffs": {
5
+ "0.3": {
6
+ "articles/typing": {
7
+ "pauses": 300,
8
+ "shown": 0.23,
9
+ "right": 0.5507,
10
+ "avg_right_chars": 5.92,
11
+ "multiword_shown": 0.0867,
12
+ "multiword_right": 0.5769,
13
+ "saved_with_tab": 0.0625,
14
+ "saved_word_by_word": 0.0697
15
+ },
16
+ "articles/editing": {
17
+ "pauses": 300,
18
+ "shown": 0.2,
19
+ "right": 0.75,
20
+ "avg_right_chars": 6.04,
21
+ "multiword_shown": 0.1,
22
+ "multiword_right": 0.7,
23
+ "saved_with_tab": 0.0756,
24
+ "saved_word_by_word": 0.0811
25
+ },
26
+ "messages/typing": {
27
+ "pauses": 300,
28
+ "shown": 0.25,
29
+ "right": 0.6133,
30
+ "avg_right_chars": 7.46,
31
+ "multiword_shown": 0.1667,
32
+ "multiword_right": 0.58,
33
+ "saved_with_tab": 0.0953,
34
+ "saved_word_by_word": 0.1139
35
+ },
36
+ "messages/editing": {
37
+ "pauses": 300,
38
+ "shown": 0.2667,
39
+ "right": 0.6125,
40
+ "avg_right_chars": 7.76,
41
+ "multiword_shown": 0.1533,
42
+ "multiword_right": 0.6522,
43
+ "saved_with_tab": 0.1056,
44
+ "saved_word_by_word": 0.1178
45
+ },
46
+ "email/typing": {
47
+ "pauses": 300,
48
+ "shown": 0.1967,
49
+ "right": 0.5932,
50
+ "avg_right_chars": 5.29,
51
+ "multiword_shown": 0.05,
52
+ "multiword_right": 0.4667,
53
+ "saved_with_tab": 0.0514,
54
+ "saved_word_by_word": 0.0564
55
+ },
56
+ "email/editing": {
57
+ "pauses": 300,
58
+ "shown": 0.2033,
59
+ "right": 0.6066,
60
+ "avg_right_chars": 5.32,
61
+ "multiword_shown": 0.06,
62
+ "multiword_right": 0.5556,
63
+ "saved_with_tab": 0.0547,
64
+ "saved_word_by_word": 0.0581
65
+ }
66
+ },
67
+ "0.4": {
68
+ "articles/typing": {
69
+ "pauses": 300,
70
+ "shown": 0.18,
71
+ "right": 0.6111,
72
+ "avg_right_chars": 5.94,
73
+ "multiword_shown": 0.0667,
74
+ "multiword_right": 0.65,
75
+ "saved_with_tab": 0.0544,
76
+ "saved_word_by_word": 0.0583
77
+ },
78
+ "articles/editing": {
79
+ "pauses": 300,
80
+ "shown": 0.1567,
81
+ "right": 0.766,
82
+ "avg_right_chars": 5.61,
83
+ "multiword_shown": 0.0667,
84
+ "multiword_right": 0.75,
85
+ "saved_with_tab": 0.0561,
86
+ "saved_word_by_word": 0.0603
87
+ },
88
+ "messages/typing": {
89
+ "pauses": 300,
90
+ "shown": 0.2033,
91
+ "right": 0.7541,
92
+ "avg_right_chars": 6.8,
93
+ "multiword_shown": 0.1133,
94
+ "multiword_right": 0.7059,
95
+ "saved_with_tab": 0.0869,
96
+ "saved_word_by_word": 0.0956
97
+ },
98
+ "messages/editing": {
99
+ "pauses": 300,
100
+ "shown": 0.1867,
101
+ "right": 0.6607,
102
+ "avg_right_chars": 6.57,
103
+ "multiword_shown": 0.09,
104
+ "multiword_right": 0.6296,
105
+ "saved_with_tab": 0.0675,
106
+ "saved_word_by_word": 0.0778
107
+ },
108
+ "email/typing": {
109
+ "pauses": 300,
110
+ "shown": 0.14,
111
+ "right": 0.6905,
112
+ "avg_right_chars": 5.21,
113
+ "multiword_shown": 0.0233,
114
+ "multiword_right": 0.5714,
115
+ "saved_with_tab": 0.0419,
116
+ "saved_word_by_word": 0.0444
117
+ },
118
+ "email/editing": {
119
+ "pauses": 300,
120
+ "shown": 0.1267,
121
+ "right": 0.7368,
122
+ "avg_right_chars": 5.21,
123
+ "multiword_shown": 0.0367,
124
+ "multiword_right": 0.6364,
125
+ "saved_with_tab": 0.0406,
126
+ "saved_word_by_word": 0.0425
127
+ }
128
+ },
129
+ "0.5": {
130
+ "articles/typing": {
131
+ "pauses": 300,
132
+ "shown": 0.1167,
133
+ "right": 0.7143,
134
+ "avg_right_chars": 5.72,
135
+ "multiword_shown": 0.0333,
136
+ "multiword_right": 0.7,
137
+ "saved_with_tab": 0.0397,
138
+ "saved_word_by_word": 0.0428
139
+ },
140
+ "articles/editing": {
141
+ "pauses": 300,
142
+ "shown": 0.11,
143
+ "right": 0.9091,
144
+ "avg_right_chars": 5.63,
145
+ "multiword_shown": 0.0433,
146
+ "multiword_right": 0.8462,
147
+ "saved_with_tab": 0.0469,
148
+ "saved_word_by_word": 0.0486
149
+ },
150
+ "messages/typing": {
151
+ "pauses": 300,
152
+ "shown": 0.16,
153
+ "right": 0.7917,
154
+ "avg_right_chars": 6.26,
155
+ "multiword_shown": 0.0833,
156
+ "multiword_right": 0.76,
157
+ "saved_with_tab": 0.0661,
158
+ "saved_word_by_word": 0.0731
159
+ },
160
+ "messages/editing": {
161
+ "pauses": 300,
162
+ "shown": 0.12,
163
+ "right": 0.7778,
164
+ "avg_right_chars": 6.04,
165
+ "multiword_shown": 0.0467,
166
+ "multiword_right": 0.7143,
167
+ "saved_with_tab": 0.0469,
168
+ "saved_word_by_word": 0.0544
169
+ },
170
+ "email/typing": {
171
+ "pauses": 300,
172
+ "shown": 0.11,
173
+ "right": 0.8485,
174
+ "avg_right_chars": 5.18,
175
+ "multiword_shown": 0.01,
176
+ "multiword_right": 0.6667,
177
+ "saved_with_tab": 0.0403,
178
+ "saved_word_by_word": 0.0406
179
+ },
180
+ "email/editing": {
181
+ "pauses": 300,
182
+ "shown": 0.0833,
183
+ "right": 0.84,
184
+ "avg_right_chars": 5.05,
185
+ "multiword_shown": 0.01,
186
+ "multiword_right": 0.3333,
187
+ "saved_with_tab": 0.0294,
188
+ "saved_word_by_word": 0.03
189
+ }
190
+ },
191
+ "0.6": {
192
+ "articles/typing": {
193
+ "pauses": 300,
194
+ "shown": 0.0967,
195
+ "right": 0.8276,
196
+ "avg_right_chars": 5.71,
197
+ "multiword_shown": 0.0233,
198
+ "multiword_right": 0.7143,
199
+ "saved_with_tab": 0.0381,
200
+ "saved_word_by_word": 0.0389
201
+ },
202
+ "articles/editing": {
203
+ "pauses": 300,
204
+ "shown": 0.0833,
205
+ "right": 1.0,
206
+ "avg_right_chars": 5.48,
207
+ "multiword_shown": 0.0233,
208
+ "multiword_right": 1.0,
209
+ "saved_with_tab": 0.0381,
210
+ "saved_word_by_word": 0.0381
211
+ },
212
+ "messages/typing": {
213
+ "pauses": 300,
214
+ "shown": 0.1333,
215
+ "right": 0.9,
216
+ "avg_right_chars": 5.94,
217
+ "multiword_shown": 0.0467,
218
+ "multiword_right": 0.8571,
219
+ "saved_with_tab": 0.0594,
220
+ "saved_word_by_word": 0.0619
221
+ },
222
+ "messages/editing": {
223
+ "pauses": 300,
224
+ "shown": 0.1067,
225
+ "right": 0.9062,
226
+ "avg_right_chars": 5.86,
227
+ "multiword_shown": 0.03,
228
+ "multiword_right": 0.8889,
229
+ "saved_with_tab": 0.0472,
230
+ "saved_word_by_word": 0.0489
231
+ },
232
+ "email/typing": {
233
+ "pauses": 300,
234
+ "shown": 0.0533,
235
+ "right": 0.9375,
236
+ "avg_right_chars": 5.27,
237
+ "multiword_shown": 0.0067,
238
+ "multiword_right": 0.5,
239
+ "saved_with_tab": 0.0219,
240
+ "saved_word_by_word": 0.0222
241
+ },
242
+ "email/editing": {
243
+ "pauses": 300,
244
+ "shown": 0.0433,
245
+ "right": 0.9231,
246
+ "avg_right_chars": 4.92,
247
+ "multiword_shown": 0.0067,
248
+ "multiword_right": 0.5,
249
+ "saved_with_tab": 0.0164,
250
+ "saved_word_by_word": 0.0164
251
+ }
252
+ },
253
+ "0.7": {
254
+ "articles/typing": {
255
+ "pauses": 300,
256
+ "shown": 0.0667,
257
+ "right": 0.85,
258
+ "avg_right_chars": 5.94,
259
+ "multiword_shown": 0.01,
260
+ "multiword_right": 1.0,
261
+ "saved_with_tab": 0.0281,
262
+ "saved_word_by_word": 0.0281
263
+ },
264
+ "articles/editing": {
265
+ "pauses": 300,
266
+ "shown": 0.06,
267
+ "right": 1.0,
268
+ "avg_right_chars": 5.78,
269
+ "multiword_shown": 0.02,
270
+ "multiword_right": 1.0,
271
+ "saved_with_tab": 0.0289,
272
+ "saved_word_by_word": 0.0289
273
+ },
274
+ "messages/typing": {
275
+ "pauses": 300,
276
+ "shown": 0.12,
277
+ "right": 0.9167,
278
+ "avg_right_chars": 5.82,
279
+ "multiword_shown": 0.0333,
280
+ "multiword_right": 0.9,
281
+ "saved_with_tab": 0.0533,
282
+ "saved_word_by_word": 0.0542
283
+ },
284
+ "messages/editing": {
285
+ "pauses": 300,
286
+ "shown": 0.08,
287
+ "right": 1.0,
288
+ "avg_right_chars": 5.67,
289
+ "multiword_shown": 0.01,
290
+ "multiword_right": 1.0,
291
+ "saved_with_tab": 0.0378,
292
+ "saved_word_by_word": 0.0378
293
+ },
294
+ "email/typing": {
295
+ "pauses": 300,
296
+ "shown": 0.0267,
297
+ "right": 1.0,
298
+ "avg_right_chars": 5.62,
299
+ "multiword_shown": 0.0033,
300
+ "multiword_right": 1.0,
301
+ "saved_with_tab": 0.0125,
302
+ "saved_word_by_word": 0.0125
303
+ },
304
+ "email/editing": {
305
+ "pauses": 300,
306
+ "shown": 0.0267,
307
+ "right": 1.0,
308
+ "avg_right_chars": 4.62,
309
+ "multiword_shown": 0.0,
310
+ "multiword_right": 0.0,
311
+ "saved_with_tab": 0.0103,
312
+ "saved_word_by_word": 0.0103
313
+ }
314
+ }
315
+ },
316
+ "examples": [
317
+ {
318
+ "domain": "articles",
319
+ "scenario": "typing",
320
+ "before": "epted an athletic scholarship to attend the University of Fl",
321
+ "suggestion": "orida",
322
+ "truth": "orida in Gainesvi",
323
+ "right": true
324
+ },
325
+ {
326
+ "domain": "articles",
327
+ "scenario": "typing",
328
+ "before": "lade, however, due to the club's involvement in the 2006 Ita",
329
+ "suggestion": "lian",
330
+ "truth": "lian football sc",
331
+ "right": true
332
+ },
333
+ {
334
+ "domain": "articles",
335
+ "scenario": "typing",
336
+ "before": "Street/Perimeter Road, just south of the Miami International",
337
+ "suggestion": " Airport",
338
+ "truth": " Airport runway.",
339
+ "right": true
340
+ },
341
+ {
342
+ "domain": "articles",
343
+ "scenario": "editing",
344
+ "before": "le tree, with a number of ecotypes, and in forms closely res",
345
+ "suggestion": "embling",
346
+ "truth": "embling other speci",
347
+ "right": true
348
+ },
349
+ {
350
+ "domain": "articles",
351
+ "scenario": "editing",
352
+ "before": "ing any kind of direct discrimination that was prominent els",
353
+ "suggestion": "ewhere in",
354
+ "truth": "ewhere in the country",
355
+ "right": true
356
+ },
357
+ {
358
+ "domain": "messages",
359
+ "scenario": "typing",
360
+ "before": "T's ability to generate coherent and contextually relevant r",
361
+ "suggestion": "esponses",
362
+ "truth": "esponses is based on",
363
+ "right": true
364
+ },
365
+ {
366
+ "domain": "messages",
367
+ "scenario": "typing",
368
+ "before": " prioritize the development of tools and models that are des",
369
+ "suggestion": "igned to",
370
+ "truth": "igned to be inclusiv",
371
+ "right": true
372
+ },
373
+ {
374
+ "domain": "messages",
375
+ "scenario": "editing",
376
+ "before": "e at which a population is growing.\nL'Hopital's rule can als",
377
+ "suggestion": "o be used to",
378
+ "truth": "o be used to solve probl",
379
+ "right": true
380
+ },
381
+ {
382
+ "domain": "messages",
383
+ "scenario": "editing",
384
+ "before": " to choose exactly which components you want to use, and cus",
385
+ "suggestion": "tomize",
386
+ "truth": "tomize your machin",
387
+ "right": true
388
+ },
389
+ {
390
+ "domain": "messages",
391
+ "scenario": "editing",
392
+ "before": " will then want to apply a small pea sized blob to the cente",
393
+ "suggestion": "r of",
394
+ "truth": "r of the CPU. In",
395
+ "right": true
396
+ },
397
+ {
398
+ "domain": "email",
399
+ "scenario": "typing",
400
+ "before": "lt time but am confident that I would make an excellent addi",
401
+ "suggestion": "tion to",
402
+ "truth": "tion to your team f",
403
+ "right": true
404
+ },
405
+ {
406
+ "domain": "email",
407
+ "scenario": "typing",
408
+ "before": ". I recommend we agree. Although, we are in the content deve",
409
+ "suggestion": "lopment",
410
+ "truth": "lopment phase the t",
411
+ "right": true
412
+ },
413
+ {
414
+ "domain": "email",
415
+ "scenario": "editing",
416
+ "before": "ured products and marketing of electricity for the United St",
417
+ "suggestion": "ates",
418
+ "truth": "ates ? Actively ",
419
+ "right": true
420
+ }
421
+ ],
422
+ "latency_ms": {
423
+ "p50": 39.33,
424
+ "p95": 184.65,
425
+ "note": "Measured in this eval process, not in a browser."
426
+ }
427
+ }
v2/metrics/pythia-70m.json ADDED
@@ -0,0 +1,427 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "pythia-70m",
3
+ "sees_after": false,
4
+ "cutoffs": {
5
+ "0.3": {
6
+ "articles/typing": {
7
+ "pauses": 300,
8
+ "shown": 0.2933,
9
+ "right": 0.5114,
10
+ "avg_right_chars": 6.56,
11
+ "multiword_shown": 0.1433,
12
+ "multiword_right": 0.4419,
13
+ "saved_with_tab": 0.0819,
14
+ "saved_word_by_word": 0.1
15
+ },
16
+ "articles/editing": {
17
+ "pauses": 300,
18
+ "shown": 0.2633,
19
+ "right": 0.6582,
20
+ "avg_right_chars": 6.52,
21
+ "multiword_shown": 0.1333,
22
+ "multiword_right": 0.575,
23
+ "saved_with_tab": 0.0942,
24
+ "saved_word_by_word": 0.1036
25
+ },
26
+ "messages/typing": {
27
+ "pauses": 300,
28
+ "shown": 0.3033,
29
+ "right": 0.6484,
30
+ "avg_right_chars": 7.44,
31
+ "multiword_shown": 0.1833,
32
+ "multiword_right": 0.6545,
33
+ "saved_with_tab": 0.1219,
34
+ "saved_word_by_word": 0.1369
35
+ },
36
+ "messages/editing": {
37
+ "pauses": 300,
38
+ "shown": 0.2733,
39
+ "right": 0.5976,
40
+ "avg_right_chars": 7.24,
41
+ "multiword_shown": 0.1667,
42
+ "multiword_right": 0.56,
43
+ "saved_with_tab": 0.0986,
44
+ "saved_word_by_word": 0.1136
45
+ },
46
+ "email/typing": {
47
+ "pauses": 300,
48
+ "shown": 0.24,
49
+ "right": 0.625,
50
+ "avg_right_chars": 5.56,
51
+ "multiword_shown": 0.0633,
52
+ "multiword_right": 0.5263,
53
+ "saved_with_tab": 0.0694,
54
+ "saved_word_by_word": 0.0756
55
+ },
56
+ "email/editing": {
57
+ "pauses": 300,
58
+ "shown": 0.2167,
59
+ "right": 0.6,
60
+ "avg_right_chars": 5.82,
61
+ "multiword_shown": 0.07,
62
+ "multiword_right": 0.619,
63
+ "saved_with_tab": 0.0631,
64
+ "saved_word_by_word": 0.0664
65
+ }
66
+ },
67
+ "0.4": {
68
+ "articles/typing": {
69
+ "pauses": 300,
70
+ "shown": 0.2167,
71
+ "right": 0.6923,
72
+ "avg_right_chars": 5.98,
73
+ "multiword_shown": 0.08,
74
+ "multiword_right": 0.625,
75
+ "saved_with_tab": 0.0747,
76
+ "saved_word_by_word": 0.0817
77
+ },
78
+ "articles/editing": {
79
+ "pauses": 300,
80
+ "shown": 0.2033,
81
+ "right": 0.7541,
82
+ "avg_right_chars": 6.28,
83
+ "multiword_shown": 0.0867,
84
+ "multiword_right": 0.7308,
85
+ "saved_with_tab": 0.0803,
86
+ "saved_word_by_word": 0.0853
87
+ },
88
+ "messages/typing": {
89
+ "pauses": 300,
90
+ "shown": 0.22,
91
+ "right": 0.7273,
92
+ "avg_right_chars": 6.9,
93
+ "multiword_shown": 0.12,
94
+ "multiword_right": 0.6944,
95
+ "saved_with_tab": 0.0919,
96
+ "saved_word_by_word": 0.1025
97
+ },
98
+ "messages/editing": {
99
+ "pauses": 300,
100
+ "shown": 0.1933,
101
+ "right": 0.7759,
102
+ "avg_right_chars": 6.64,
103
+ "multiword_shown": 0.1033,
104
+ "multiword_right": 0.7419,
105
+ "saved_with_tab": 0.0831,
106
+ "saved_word_by_word": 0.0917
107
+ },
108
+ "email/typing": {
109
+ "pauses": 300,
110
+ "shown": 0.1867,
111
+ "right": 0.6786,
112
+ "avg_right_chars": 5.37,
113
+ "multiword_shown": 0.04,
114
+ "multiword_right": 0.5,
115
+ "saved_with_tab": 0.0567,
116
+ "saved_word_by_word": 0.0614
117
+ },
118
+ "email/editing": {
119
+ "pauses": 300,
120
+ "shown": 0.14,
121
+ "right": 0.7143,
122
+ "avg_right_chars": 5.37,
123
+ "multiword_shown": 0.03,
124
+ "multiword_right": 0.7778,
125
+ "saved_with_tab": 0.0447,
126
+ "saved_word_by_word": 0.045
127
+ }
128
+ },
129
+ "0.5": {
130
+ "articles/typing": {
131
+ "pauses": 300,
132
+ "shown": 0.1867,
133
+ "right": 0.75,
134
+ "avg_right_chars": 5.98,
135
+ "multiword_shown": 0.06,
136
+ "multiword_right": 0.7222,
137
+ "saved_with_tab": 0.0697,
138
+ "saved_word_by_word": 0.0717
139
+ },
140
+ "articles/editing": {
141
+ "pauses": 300,
142
+ "shown": 0.1533,
143
+ "right": 0.8696,
144
+ "avg_right_chars": 6.42,
145
+ "multiword_shown": 0.0667,
146
+ "multiword_right": 0.85,
147
+ "saved_with_tab": 0.0714,
148
+ "saved_word_by_word": 0.0742
149
+ },
150
+ "messages/typing": {
151
+ "pauses": 300,
152
+ "shown": 0.1733,
153
+ "right": 0.8462,
154
+ "avg_right_chars": 6.55,
155
+ "multiword_shown": 0.0833,
156
+ "multiword_right": 0.76,
157
+ "saved_with_tab": 0.08,
158
+ "saved_word_by_word": 0.0867
159
+ },
160
+ "messages/editing": {
161
+ "pauses": 300,
162
+ "shown": 0.14,
163
+ "right": 0.8333,
164
+ "avg_right_chars": 6.37,
165
+ "multiword_shown": 0.0567,
166
+ "multiword_right": 0.7647,
167
+ "saved_with_tab": 0.0619,
168
+ "saved_word_by_word": 0.0669
169
+ },
170
+ "email/typing": {
171
+ "pauses": 300,
172
+ "shown": 0.1467,
173
+ "right": 0.7273,
174
+ "avg_right_chars": 5.56,
175
+ "multiword_shown": 0.02,
176
+ "multiword_right": 0.6667,
177
+ "saved_with_tab": 0.0494,
178
+ "saved_word_by_word": 0.05
179
+ },
180
+ "email/editing": {
181
+ "pauses": 300,
182
+ "shown": 0.0967,
183
+ "right": 0.8276,
184
+ "avg_right_chars": 5.33,
185
+ "multiword_shown": 0.0167,
186
+ "multiword_right": 0.8,
187
+ "saved_with_tab": 0.0356,
188
+ "saved_word_by_word": 0.0356
189
+ }
190
+ },
191
+ "0.6": {
192
+ "articles/typing": {
193
+ "pauses": 300,
194
+ "shown": 0.13,
195
+ "right": 0.8205,
196
+ "avg_right_chars": 5.97,
197
+ "multiword_shown": 0.0367,
198
+ "multiword_right": 0.8182,
199
+ "saved_with_tab": 0.0531,
200
+ "saved_word_by_word": 0.0536
201
+ },
202
+ "articles/editing": {
203
+ "pauses": 300,
204
+ "shown": 0.11,
205
+ "right": 0.9091,
206
+ "avg_right_chars": 5.87,
207
+ "multiword_shown": 0.0467,
208
+ "multiword_right": 0.8571,
209
+ "saved_with_tab": 0.0489,
210
+ "saved_word_by_word": 0.0508
211
+ },
212
+ "messages/typing": {
213
+ "pauses": 300,
214
+ "shown": 0.1467,
215
+ "right": 0.9091,
216
+ "avg_right_chars": 6.35,
217
+ "multiword_shown": 0.0633,
218
+ "multiword_right": 0.8421,
219
+ "saved_with_tab": 0.0706,
220
+ "saved_word_by_word": 0.0733
221
+ },
222
+ "messages/editing": {
223
+ "pauses": 300,
224
+ "shown": 0.11,
225
+ "right": 0.9091,
226
+ "avg_right_chars": 6.33,
227
+ "multiword_shown": 0.0367,
228
+ "multiword_right": 0.8182,
229
+ "saved_with_tab": 0.0528,
230
+ "saved_word_by_word": 0.0558
231
+ },
232
+ "email/typing": {
233
+ "pauses": 300,
234
+ "shown": 0.1067,
235
+ "right": 0.8125,
236
+ "avg_right_chars": 5.42,
237
+ "multiword_shown": 0.01,
238
+ "multiword_right": 0.6667,
239
+ "saved_with_tab": 0.0392,
240
+ "saved_word_by_word": 0.0394
241
+ },
242
+ "email/editing": {
243
+ "pauses": 300,
244
+ "shown": 0.0733,
245
+ "right": 0.8636,
246
+ "avg_right_chars": 5.11,
247
+ "multiword_shown": 0.0033,
248
+ "multiword_right": 1.0,
249
+ "saved_with_tab": 0.0269,
250
+ "saved_word_by_word": 0.0269
251
+ }
252
+ },
253
+ "0.7": {
254
+ "articles/typing": {
255
+ "pauses": 300,
256
+ "shown": 0.1033,
257
+ "right": 0.8387,
258
+ "avg_right_chars": 6.04,
259
+ "multiword_shown": 0.0267,
260
+ "multiword_right": 0.875,
261
+ "saved_with_tab": 0.0436,
262
+ "saved_word_by_word": 0.0439
263
+ },
264
+ "articles/editing": {
265
+ "pauses": 300,
266
+ "shown": 0.0867,
267
+ "right": 0.9615,
268
+ "avg_right_chars": 5.52,
269
+ "multiword_shown": 0.0367,
270
+ "multiword_right": 0.9091,
271
+ "saved_with_tab": 0.0383,
272
+ "saved_word_by_word": 0.0389
273
+ },
274
+ "messages/typing": {
275
+ "pauses": 300,
276
+ "shown": 0.1267,
277
+ "right": 0.9474,
278
+ "avg_right_chars": 6.11,
279
+ "multiword_shown": 0.0433,
280
+ "multiword_right": 0.9231,
281
+ "saved_with_tab": 0.0611,
282
+ "saved_word_by_word": 0.0619
283
+ },
284
+ "messages/editing": {
285
+ "pauses": 300,
286
+ "shown": 0.0867,
287
+ "right": 0.9615,
288
+ "avg_right_chars": 6.36,
289
+ "multiword_shown": 0.0267,
290
+ "multiword_right": 0.875,
291
+ "saved_with_tab": 0.0442,
292
+ "saved_word_by_word": 0.0461
293
+ },
294
+ "email/typing": {
295
+ "pauses": 300,
296
+ "shown": 0.0633,
297
+ "right": 0.8421,
298
+ "avg_right_chars": 5.69,
299
+ "multiword_shown": 0.0067,
300
+ "multiword_right": 0.5,
301
+ "saved_with_tab": 0.0253,
302
+ "saved_word_by_word": 0.0256
303
+ },
304
+ "email/editing": {
305
+ "pauses": 300,
306
+ "shown": 0.0533,
307
+ "right": 0.875,
308
+ "avg_right_chars": 5.07,
309
+ "multiword_shown": 0.0033,
310
+ "multiword_right": 1.0,
311
+ "saved_with_tab": 0.0197,
312
+ "saved_word_by_word": 0.0197
313
+ }
314
+ }
315
+ },
316
+ "examples": [
317
+ {
318
+ "domain": "articles",
319
+ "scenario": "typing",
320
+ "before": "ROME KAVALKADE\". The segment featured acrobats, bareback rid",
321
+ "suggestion": "ers,",
322
+ "truth": "ers, clowns, tig",
323
+ "right": true
324
+ },
325
+ {
326
+ "domain": "articles",
327
+ "scenario": "typing",
328
+ "before": "lade, however, due to the club's involvement in the 2006 Ita",
329
+ "suggestion": "lian",
330
+ "truth": "lian football sc",
331
+ "right": true
332
+ },
333
+ {
334
+ "domain": "articles",
335
+ "scenario": "editing",
336
+ "before": "le tree, with a number of ecotypes, and in forms closely res",
337
+ "suggestion": "embling",
338
+ "truth": "embling other speci",
339
+ "right": true
340
+ },
341
+ {
342
+ "domain": "articles",
343
+ "scenario": "editing",
344
+ "before": "ing any kind of direct discrimination that was prominent els",
345
+ "suggestion": "ewhere in",
346
+ "truth": "ewhere in the country",
347
+ "right": true
348
+ },
349
+ {
350
+ "domain": "messages",
351
+ "scenario": "typing",
352
+ "before": "T's ability to generate coherent and contextually relevant r",
353
+ "suggestion": "esponses",
354
+ "truth": "esponses is based on",
355
+ "right": true
356
+ },
357
+ {
358
+ "domain": "messages",
359
+ "scenario": "typing",
360
+ "before": " prioritize the development of tools and models that are des",
361
+ "suggestion": "igned to",
362
+ "truth": "igned to be inclusiv",
363
+ "right": true
364
+ },
365
+ {
366
+ "domain": "messages",
367
+ "scenario": "editing",
368
+ "before": "e at which a population is growing.\nL'Hopital's rule can als",
369
+ "suggestion": "o be used to",
370
+ "truth": "o be used to solve probl",
371
+ "right": true
372
+ },
373
+ {
374
+ "domain": "messages",
375
+ "scenario": "editing",
376
+ "before": " to choose exactly which components you want to use, and cus",
377
+ "suggestion": "tomize",
378
+ "truth": "tomize your machin",
379
+ "right": true
380
+ },
381
+ {
382
+ "domain": "messages",
383
+ "scenario": "editing",
384
+ "before": " will then want to apply a small pea sized blob to the cente",
385
+ "suggestion": "r of the",
386
+ "truth": "r of the CPU. Instal",
387
+ "right": true
388
+ },
389
+ {
390
+ "domain": "email",
391
+ "scenario": "typing",
392
+ "before": "lt time but am confident that I would make an excellent addi",
393
+ "suggestion": "tion to",
394
+ "truth": "tion to your team f",
395
+ "right": true
396
+ },
397
+ {
398
+ "domain": "email",
399
+ "scenario": "typing",
400
+ "before": ". I recommend we agree. Although, we are in the content deve",
401
+ "suggestion": "lopment",
402
+ "truth": "lopment phase the t",
403
+ "right": true
404
+ },
405
+ {
406
+ "domain": "email",
407
+ "scenario": "typing",
408
+ "before": "edge system is connected to the simulation system (Tim's ide",
409
+ "suggestion": "ntity",
410
+ "truth": "a with some direc",
411
+ "right": false
412
+ },
413
+ {
414
+ "domain": "email",
415
+ "scenario": "editing",
416
+ "before": "ured products and marketing of electricity for the United St",
417
+ "suggestion": "ates",
418
+ "truth": "ates ? Actively ",
419
+ "right": true
420
+ }
421
+ ],
422
+ "latency_ms": {
423
+ "p50": 40.06,
424
+ "p95": 191.05,
425
+ "note": "Measured in this eval process, not in a browser."
426
+ }
427
+ }
v2/metrics/{tinynext-v2-float-onnx.json → tinynext-v2-first-float-onnx.json} RENAMED
File without changes
v2/metrics/tinynext-v2-first-q8-onnx.json ADDED
@@ -0,0 +1,411 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "tinynext-v2-q8-onnx",
3
+ "sees_after": true,
4
+ "cutoffs": {
5
+ "0.3": {
6
+ "articles/typing": {
7
+ "pauses": 300,
8
+ "shown": 0.1767,
9
+ "right": 0.5849,
10
+ "avg_right_chars": 6.0,
11
+ "multiword_shown": 0.0733,
12
+ "multiword_right": 0.4545,
13
+ "saved_with_tab": 0.0517,
14
+ "saved_word_by_word": 0.0575
15
+ },
16
+ "articles/editing": {
17
+ "pauses": 300,
18
+ "shown": 0.13,
19
+ "right": 0.6923,
20
+ "avg_right_chars": 5.93,
21
+ "multiword_shown": 0.0533,
22
+ "multiword_right": 0.6875,
23
+ "saved_with_tab": 0.0444,
24
+ "saved_word_by_word": 0.0464
25
+ },
26
+ "messages/typing": {
27
+ "pauses": 300,
28
+ "shown": 0.3,
29
+ "right": 0.6333,
30
+ "avg_right_chars": 7.65,
31
+ "multiword_shown": 0.1933,
32
+ "multiword_right": 0.569,
33
+ "saved_with_tab": 0.1211,
34
+ "saved_word_by_word": 0.1403
35
+ },
36
+ "messages/editing": {
37
+ "pauses": 300,
38
+ "shown": 0.2767,
39
+ "right": 0.5181,
40
+ "avg_right_chars": 6.53,
41
+ "multiword_shown": 0.14,
42
+ "multiword_right": 0.5,
43
+ "saved_with_tab": 0.0781,
44
+ "saved_word_by_word": 0.0919
45
+ },
46
+ "email/typing": {
47
+ "pauses": 300,
48
+ "shown": 0.2333,
49
+ "right": 0.5857,
50
+ "avg_right_chars": 5.46,
51
+ "multiword_shown": 0.0667,
52
+ "multiword_right": 0.45,
53
+ "saved_with_tab": 0.0622,
54
+ "saved_word_by_word": 0.0714
55
+ },
56
+ "email/editing": {
57
+ "pauses": 300,
58
+ "shown": 0.2333,
59
+ "right": 0.6,
60
+ "avg_right_chars": 5.57,
61
+ "multiword_shown": 0.0933,
62
+ "multiword_right": 0.5357,
63
+ "saved_with_tab": 0.065,
64
+ "saved_word_by_word": 0.0703
65
+ }
66
+ },
67
+ "0.4": {
68
+ "articles/typing": {
69
+ "pauses": 300,
70
+ "shown": 0.1467,
71
+ "right": 0.6818,
72
+ "avg_right_chars": 6.0,
73
+ "multiword_shown": 0.0567,
74
+ "multiword_right": 0.5882,
75
+ "saved_with_tab": 0.05,
76
+ "saved_word_by_word": 0.0531
77
+ },
78
+ "articles/editing": {
79
+ "pauses": 300,
80
+ "shown": 0.0967,
81
+ "right": 0.7586,
82
+ "avg_right_chars": 5.82,
83
+ "multiword_shown": 0.0367,
84
+ "multiword_right": 0.9091,
85
+ "saved_with_tab": 0.0356,
86
+ "saved_word_by_word": 0.0358
87
+ },
88
+ "messages/typing": {
89
+ "pauses": 300,
90
+ "shown": 0.24,
91
+ "right": 0.7361,
92
+ "avg_right_chars": 7.26,
93
+ "multiword_shown": 0.1333,
94
+ "multiword_right": 0.65,
95
+ "saved_with_tab": 0.1069,
96
+ "saved_word_by_word": 0.1167
97
+ },
98
+ "messages/editing": {
99
+ "pauses": 300,
100
+ "shown": 0.17,
101
+ "right": 0.6275,
102
+ "avg_right_chars": 6.38,
103
+ "multiword_shown": 0.0733,
104
+ "multiword_right": 0.6364,
105
+ "saved_with_tab": 0.0567,
106
+ "saved_word_by_word": 0.0658
107
+ },
108
+ "email/typing": {
109
+ "pauses": 300,
110
+ "shown": 0.1867,
111
+ "right": 0.75,
112
+ "avg_right_chars": 5.48,
113
+ "multiword_shown": 0.0467,
114
+ "multiword_right": 0.5714,
115
+ "saved_with_tab": 0.0639,
116
+ "saved_word_by_word": 0.0678
117
+ },
118
+ "email/editing": {
119
+ "pauses": 300,
120
+ "shown": 0.1667,
121
+ "right": 0.68,
122
+ "avg_right_chars": 5.26,
123
+ "multiword_shown": 0.06,
124
+ "multiword_right": 0.6111,
125
+ "saved_with_tab": 0.0497,
126
+ "saved_word_by_word": 0.0528
127
+ }
128
+ },
129
+ "0.5": {
130
+ "articles/typing": {
131
+ "pauses": 300,
132
+ "shown": 0.11,
133
+ "right": 0.7879,
134
+ "avg_right_chars": 5.81,
135
+ "multiword_shown": 0.0333,
136
+ "multiword_right": 0.7,
137
+ "saved_with_tab": 0.0419,
138
+ "saved_word_by_word": 0.0425
139
+ },
140
+ "articles/editing": {
141
+ "pauses": 300,
142
+ "shown": 0.0833,
143
+ "right": 0.8,
144
+ "avg_right_chars": 5.4,
145
+ "multiword_shown": 0.0267,
146
+ "multiword_right": 0.875,
147
+ "saved_with_tab": 0.03,
148
+ "saved_word_by_word": 0.0303
149
+ },
150
+ "messages/typing": {
151
+ "pauses": 300,
152
+ "shown": 0.18,
153
+ "right": 0.7963,
154
+ "avg_right_chars": 6.98,
155
+ "multiword_shown": 0.1,
156
+ "multiword_right": 0.7,
157
+ "saved_with_tab": 0.0833,
158
+ "saved_word_by_word": 0.09
159
+ },
160
+ "messages/editing": {
161
+ "pauses": 300,
162
+ "shown": 0.1233,
163
+ "right": 0.7297,
164
+ "avg_right_chars": 6.37,
165
+ "multiword_shown": 0.0567,
166
+ "multiword_right": 0.6471,
167
+ "saved_with_tab": 0.0478,
168
+ "saved_word_by_word": 0.0561
169
+ },
170
+ "email/typing": {
171
+ "pauses": 300,
172
+ "shown": 0.14,
173
+ "right": 0.7857,
174
+ "avg_right_chars": 5.48,
175
+ "multiword_shown": 0.0267,
176
+ "multiword_right": 0.75,
177
+ "saved_with_tab": 0.0503,
178
+ "saved_word_by_word": 0.0514
179
+ },
180
+ "email/editing": {
181
+ "pauses": 300,
182
+ "shown": 0.1267,
183
+ "right": 0.7632,
184
+ "avg_right_chars": 5.07,
185
+ "multiword_shown": 0.0367,
186
+ "multiword_right": 0.5455,
187
+ "saved_with_tab": 0.0408,
188
+ "saved_word_by_word": 0.0431
189
+ }
190
+ },
191
+ "0.6": {
192
+ "articles/typing": {
193
+ "pauses": 300,
194
+ "shown": 0.0967,
195
+ "right": 0.7931,
196
+ "avg_right_chars": 5.96,
197
+ "multiword_shown": 0.0267,
198
+ "multiword_right": 0.75,
199
+ "saved_with_tab": 0.0381,
200
+ "saved_word_by_word": 0.0381
201
+ },
202
+ "articles/editing": {
203
+ "pauses": 300,
204
+ "shown": 0.05,
205
+ "right": 0.9333,
206
+ "avg_right_chars": 5.79,
207
+ "multiword_shown": 0.02,
208
+ "multiword_right": 1.0,
209
+ "saved_with_tab": 0.0225,
210
+ "saved_word_by_word": 0.0225
211
+ },
212
+ "messages/typing": {
213
+ "pauses": 300,
214
+ "shown": 0.1467,
215
+ "right": 0.8864,
216
+ "avg_right_chars": 6.54,
217
+ "multiword_shown": 0.0733,
218
+ "multiword_right": 0.8182,
219
+ "saved_with_tab": 0.0708,
220
+ "saved_word_by_word": 0.0742
221
+ },
222
+ "messages/editing": {
223
+ "pauses": 300,
224
+ "shown": 0.0767,
225
+ "right": 0.7391,
226
+ "avg_right_chars": 6.18,
227
+ "multiword_shown": 0.04,
228
+ "multiword_right": 0.5,
229
+ "saved_with_tab": 0.0292,
230
+ "saved_word_by_word": 0.0375
231
+ },
232
+ "email/typing": {
233
+ "pauses": 300,
234
+ "shown": 0.1,
235
+ "right": 0.8667,
236
+ "avg_right_chars": 5.73,
237
+ "multiword_shown": 0.02,
238
+ "multiword_right": 1.0,
239
+ "saved_with_tab": 0.0414,
240
+ "saved_word_by_word": 0.0414
241
+ },
242
+ "email/editing": {
243
+ "pauses": 300,
244
+ "shown": 0.09,
245
+ "right": 0.8889,
246
+ "avg_right_chars": 4.92,
247
+ "multiword_shown": 0.0133,
248
+ "multiword_right": 0.75,
249
+ "saved_with_tab": 0.0328,
250
+ "saved_word_by_word": 0.0339
251
+ }
252
+ },
253
+ "0.7": {
254
+ "articles/typing": {
255
+ "pauses": 300,
256
+ "shown": 0.0667,
257
+ "right": 0.95,
258
+ "avg_right_chars": 5.63,
259
+ "multiword_shown": 0.0167,
260
+ "multiword_right": 1.0,
261
+ "saved_with_tab": 0.0297,
262
+ "saved_word_by_word": 0.0297
263
+ },
264
+ "articles/editing": {
265
+ "pauses": 300,
266
+ "shown": 0.03,
267
+ "right": 0.8889,
268
+ "avg_right_chars": 5.12,
269
+ "multiword_shown": 0.0033,
270
+ "multiword_right": 1.0,
271
+ "saved_with_tab": 0.0114,
272
+ "saved_word_by_word": 0.0114
273
+ },
274
+ "messages/typing": {
275
+ "pauses": 300,
276
+ "shown": 0.1133,
277
+ "right": 0.9412,
278
+ "avg_right_chars": 6.22,
279
+ "multiword_shown": 0.05,
280
+ "multiword_right": 0.8667,
281
+ "saved_with_tab": 0.0553,
282
+ "saved_word_by_word": 0.0561
283
+ },
284
+ "messages/editing": {
285
+ "pauses": 300,
286
+ "shown": 0.0633,
287
+ "right": 0.7895,
288
+ "avg_right_chars": 5.93,
289
+ "multiword_shown": 0.03,
290
+ "multiword_right": 0.5556,
291
+ "saved_with_tab": 0.0247,
292
+ "saved_word_by_word": 0.0319
293
+ },
294
+ "email/typing": {
295
+ "pauses": 300,
296
+ "shown": 0.0667,
297
+ "right": 0.85,
298
+ "avg_right_chars": 5.41,
299
+ "multiword_shown": 0.0067,
300
+ "multiword_right": 1.0,
301
+ "saved_with_tab": 0.0256,
302
+ "saved_word_by_word": 0.0256
303
+ },
304
+ "email/editing": {
305
+ "pauses": 300,
306
+ "shown": 0.06,
307
+ "right": 0.9444,
308
+ "avg_right_chars": 4.71,
309
+ "multiword_shown": 0.0033,
310
+ "multiword_right": 1.0,
311
+ "saved_with_tab": 0.0222,
312
+ "saved_word_by_word": 0.0222
313
+ }
314
+ }
315
+ },
316
+ "examples": [
317
+ {
318
+ "domain": "articles",
319
+ "scenario": "typing",
320
+ "before": "elipads, IFR approaches, and a separate float plane dock alo",
321
+ "suggestion": "ng the",
322
+ "truth": "ng the Fraser Rive",
323
+ "right": true
324
+ },
325
+ {
326
+ "domain": "articles",
327
+ "scenario": "editing",
328
+ "before": " where he met his wife, Kisa, a graduate of the University o",
329
+ "suggestion": "f Illinois",
330
+ "truth": "f Colorado. Sasaki the",
331
+ "right": false
332
+ },
333
+ {
334
+ "domain": "articles",
335
+ "scenario": "editing",
336
+ "before": "ing any kind of direct discrimination that was prominent els",
337
+ "suggestion": "ewhere.",
338
+ "truth": "ewhere in the count",
339
+ "right": false
340
+ },
341
+ {
342
+ "domain": "messages",
343
+ "scenario": "typing",
344
+ "before": "T's ability to generate coherent and contextually relevant r",
345
+ "suggestion": "esponses",
346
+ "truth": "esponses is based on",
347
+ "right": true
348
+ },
349
+ {
350
+ "domain": "messages",
351
+ "scenario": "typing",
352
+ "before": " prioritize the development of tools and models that are des",
353
+ "suggestion": "igned to",
354
+ "truth": "igned to be inclusiv",
355
+ "right": true
356
+ },
357
+ {
358
+ "domain": "messages",
359
+ "scenario": "editing",
360
+ "before": "e at which a population is growing.\nL'Hopital's rule can als",
361
+ "suggestion": "o be used to",
362
+ "truth": "o be used to solve probl",
363
+ "right": true
364
+ },
365
+ {
366
+ "domain": "email",
367
+ "scenario": "typing",
368
+ "before": "lt time but am confident that I would make an excellent addi",
369
+ "suggestion": "tion to",
370
+ "truth": "tion to your team f",
371
+ "right": true
372
+ },
373
+ {
374
+ "domain": "email",
375
+ "scenario": "typing",
376
+ "before": ". I recommend we agree. Although, we are in the content deve",
377
+ "suggestion": "lopment",
378
+ "truth": "lopment phase the t",
379
+ "right": true
380
+ },
381
+ {
382
+ "domain": "email",
383
+ "scenario": "editing",
384
+ "before": "ured products and marketing of electricity for the United St",
385
+ "suggestion": "ates",
386
+ "truth": "ates ? Actively ",
387
+ "right": true
388
+ },
389
+ {
390
+ "domain": "email",
391
+ "scenario": "editing",
392
+ "before": "I'm not sure what you are looking for. The Early Termination",
393
+ "suggestion": " Log",
394
+ "truth": " Date is one of ",
395
+ "right": false
396
+ },
397
+ {
398
+ "domain": "email",
399
+ "scenario": "editing",
400
+ "before": " action is necessary as contractually we are coming to the e",
401
+ "suggestion": "nd of",
402
+ "truth": "nd of the accepta",
403
+ "right": true
404
+ }
405
+ ],
406
+ "latency_ms": {
407
+ "p50": 25.25,
408
+ "p95": 75.94,
409
+ "note": "Measured in this eval process, not in a browser."
410
+ }
411
+ }
v2/metrics/tinynext-v2-first.json ADDED
@@ -0,0 +1,347 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "tinynext-v2",
3
+ "sees_after": true,
4
+ "cutoffs": {
5
+ "0.3": {
6
+ "articles/typing": {
7
+ "pauses": 1000,
8
+ "shown": 0.188,
9
+ "right": 0.617,
10
+ "avg_right_chars": 5.91,
11
+ "multiword_shown": 0.067,
12
+ "multiword_right": 0.5373,
13
+ "saved_with_tab": 0.0572,
14
+ "saved_word_by_word": 0.0626
15
+ },
16
+ "articles/editing": {
17
+ "pauses": 1000,
18
+ "shown": 0.148,
19
+ "right": 0.6081,
20
+ "avg_right_chars": 5.94,
21
+ "multiword_shown": 0.058,
22
+ "multiword_right": 0.5862,
23
+ "saved_with_tab": 0.0446,
24
+ "saved_word_by_word": 0.0491
25
+ },
26
+ "messages/typing": {
27
+ "pauses": 1000,
28
+ "shown": 0.293,
29
+ "right": 0.5563,
30
+ "avg_right_chars": 7.71,
31
+ "multiword_shown": 0.163,
32
+ "multiword_right": 0.546,
33
+ "saved_with_tab": 0.1047,
34
+ "saved_word_by_word": 0.1232
35
+ },
36
+ "messages/editing": {
37
+ "pauses": 1000,
38
+ "shown": 0.276,
39
+ "right": 0.6159,
40
+ "avg_right_chars": 7.49,
41
+ "multiword_shown": 0.152,
42
+ "multiword_right": 0.6118,
43
+ "saved_with_tab": 0.1061,
44
+ "saved_word_by_word": 0.1183
45
+ },
46
+ "email/typing": {
47
+ "pauses": 1000,
48
+ "shown": 0.237,
49
+ "right": 0.5443,
50
+ "avg_right_chars": 6.09,
51
+ "multiword_shown": 0.084,
52
+ "multiword_right": 0.3929,
53
+ "saved_with_tab": 0.0655,
54
+ "saved_word_by_word": 0.0744
55
+ },
56
+ "email/editing": {
57
+ "pauses": 1000,
58
+ "shown": 0.22,
59
+ "right": 0.5273,
60
+ "avg_right_chars": 5.77,
61
+ "multiword_shown": 0.093,
62
+ "multiword_right": 0.4194,
63
+ "saved_with_tab": 0.0558,
64
+ "saved_word_by_word": 0.0645
65
+ }
66
+ },
67
+ "0.4": {
68
+ "articles/typing": {
69
+ "pauses": 1000,
70
+ "shown": 0.14,
71
+ "right": 0.7214,
72
+ "avg_right_chars": 5.94,
73
+ "multiword_shown": 0.043,
74
+ "multiword_right": 0.6744,
75
+ "saved_with_tab": 0.05,
76
+ "saved_word_by_word": 0.0522
77
+ },
78
+ "articles/editing": {
79
+ "pauses": 1000,
80
+ "shown": 0.102,
81
+ "right": 0.6667,
82
+ "avg_right_chars": 5.65,
83
+ "multiword_shown": 0.034,
84
+ "multiword_right": 0.7353,
85
+ "saved_with_tab": 0.032,
86
+ "saved_word_by_word": 0.0341
87
+ },
88
+ "messages/typing": {
89
+ "pauses": 1000,
90
+ "shown": 0.212,
91
+ "right": 0.6934,
92
+ "avg_right_chars": 7.19,
93
+ "multiword_shown": 0.105,
94
+ "multiword_right": 0.6571,
95
+ "saved_with_tab": 0.0881,
96
+ "saved_word_by_word": 0.0958
97
+ },
98
+ "messages/editing": {
99
+ "pauses": 1000,
100
+ "shown": 0.194,
101
+ "right": 0.7113,
102
+ "avg_right_chars": 6.83,
103
+ "multiword_shown": 0.097,
104
+ "multiword_right": 0.6907,
105
+ "saved_with_tab": 0.0785,
106
+ "saved_word_by_word": 0.086
107
+ },
108
+ "email/typing": {
109
+ "pauses": 1000,
110
+ "shown": 0.172,
111
+ "right": 0.686,
112
+ "avg_right_chars": 5.86,
113
+ "multiword_shown": 0.053,
114
+ "multiword_right": 0.434,
115
+ "saved_with_tab": 0.0576,
116
+ "saved_word_by_word": 0.0618
117
+ },
118
+ "email/editing": {
119
+ "pauses": 1000,
120
+ "shown": 0.147,
121
+ "right": 0.6395,
122
+ "avg_right_chars": 5.48,
123
+ "multiword_shown": 0.057,
124
+ "multiword_right": 0.5439,
125
+ "saved_with_tab": 0.0429,
126
+ "saved_word_by_word": 0.0459
127
+ }
128
+ },
129
+ "0.5": {
130
+ "articles/typing": {
131
+ "pauses": 1000,
132
+ "shown": 0.106,
133
+ "right": 0.7642,
134
+ "avg_right_chars": 5.9,
135
+ "multiword_shown": 0.034,
136
+ "multiword_right": 0.7353,
137
+ "saved_with_tab": 0.0398,
138
+ "saved_word_by_word": 0.0411
139
+ },
140
+ "articles/editing": {
141
+ "pauses": 1000,
142
+ "shown": 0.078,
143
+ "right": 0.7692,
144
+ "avg_right_chars": 5.55,
145
+ "multiword_shown": 0.022,
146
+ "multiword_right": 0.8636,
147
+ "saved_with_tab": 0.0278,
148
+ "saved_word_by_word": 0.0282
149
+ },
150
+ "messages/typing": {
151
+ "pauses": 1000,
152
+ "shown": 0.163,
153
+ "right": 0.7791,
154
+ "avg_right_chars": 6.86,
155
+ "multiword_shown": 0.078,
156
+ "multiword_right": 0.7179,
157
+ "saved_with_tab": 0.0726,
158
+ "saved_word_by_word": 0.078
159
+ },
160
+ "messages/editing": {
161
+ "pauses": 1000,
162
+ "shown": 0.146,
163
+ "right": 0.7877,
164
+ "avg_right_chars": 6.63,
165
+ "multiword_shown": 0.071,
166
+ "multiword_right": 0.7465,
167
+ "saved_with_tab": 0.0635,
168
+ "saved_word_by_word": 0.0679
169
+ },
170
+ "email/typing": {
171
+ "pauses": 1000,
172
+ "shown": 0.113,
173
+ "right": 0.7876,
174
+ "avg_right_chars": 5.84,
175
+ "multiword_shown": 0.03,
176
+ "multiword_right": 0.6,
177
+ "saved_with_tab": 0.0433,
178
+ "saved_word_by_word": 0.0452
179
+ },
180
+ "email/editing": {
181
+ "pauses": 1000,
182
+ "shown": 0.104,
183
+ "right": 0.7788,
184
+ "avg_right_chars": 5.32,
185
+ "multiword_shown": 0.03,
186
+ "multiword_right": 0.7,
187
+ "saved_with_tab": 0.0359,
188
+ "saved_word_by_word": 0.0369
189
+ }
190
+ },
191
+ "0.6": {
192
+ "articles/typing": {
193
+ "pauses": 1000,
194
+ "shown": 0.084,
195
+ "right": 0.8571,
196
+ "avg_right_chars": 5.86,
197
+ "multiword_shown": 0.022,
198
+ "multiword_right": 0.8636,
199
+ "saved_with_tab": 0.0352,
200
+ "saved_word_by_word": 0.0353
201
+ },
202
+ "articles/editing": {
203
+ "pauses": 1000,
204
+ "shown": 0.056,
205
+ "right": 0.8571,
206
+ "avg_right_chars": 5.62,
207
+ "multiword_shown": 0.016,
208
+ "multiword_right": 0.9375,
209
+ "saved_with_tab": 0.0225,
210
+ "saved_word_by_word": 0.0227
211
+ },
212
+ "messages/typing": {
213
+ "pauses": 1000,
214
+ "shown": 0.123,
215
+ "right": 0.8455,
216
+ "avg_right_chars": 6.73,
217
+ "multiword_shown": 0.06,
218
+ "multiword_right": 0.8167,
219
+ "saved_with_tab": 0.0583,
220
+ "saved_word_by_word": 0.061
221
+ },
222
+ "messages/editing": {
223
+ "pauses": 1000,
224
+ "shown": 0.108,
225
+ "right": 0.8426,
226
+ "avg_right_chars": 6.31,
227
+ "multiword_shown": 0.052,
228
+ "multiword_right": 0.7308,
229
+ "saved_with_tab": 0.0478,
230
+ "saved_word_by_word": 0.0512
231
+ },
232
+ "email/typing": {
233
+ "pauses": 1000,
234
+ "shown": 0.076,
235
+ "right": 0.8816,
236
+ "avg_right_chars": 5.79,
237
+ "multiword_shown": 0.016,
238
+ "multiword_right": 0.8125,
239
+ "saved_with_tab": 0.0323,
240
+ "saved_word_by_word": 0.0324
241
+ },
242
+ "email/editing": {
243
+ "pauses": 1000,
244
+ "shown": 0.071,
245
+ "right": 0.8873,
246
+ "avg_right_chars": 5.25,
247
+ "multiword_shown": 0.014,
248
+ "multiword_right": 0.8571,
249
+ "saved_with_tab": 0.0276,
250
+ "saved_word_by_word": 0.0279
251
+ }
252
+ },
253
+ "0.7": {
254
+ "articles/typing": {
255
+ "pauses": 1000,
256
+ "shown": 0.059,
257
+ "right": 0.9153,
258
+ "avg_right_chars": 5.76,
259
+ "multiword_shown": 0.018,
260
+ "multiword_right": 0.9444,
261
+ "saved_with_tab": 0.0259,
262
+ "saved_word_by_word": 0.0261
263
+ },
264
+ "articles/editing": {
265
+ "pauses": 1000,
266
+ "shown": 0.035,
267
+ "right": 0.8286,
268
+ "avg_right_chars": 5.34,
269
+ "multiword_shown": 0.007,
270
+ "multiword_right": 0.8571,
271
+ "saved_with_tab": 0.0129,
272
+ "saved_word_by_word": 0.0132
273
+ },
274
+ "messages/typing": {
275
+ "pauses": 1000,
276
+ "shown": 0.099,
277
+ "right": 0.9293,
278
+ "avg_right_chars": 6.37,
279
+ "multiword_shown": 0.042,
280
+ "multiword_right": 0.9048,
281
+ "saved_with_tab": 0.0488,
282
+ "saved_word_by_word": 0.0492
283
+ },
284
+ "messages/editing": {
285
+ "pauses": 1000,
286
+ "shown": 0.086,
287
+ "right": 0.8837,
288
+ "avg_right_chars": 5.97,
289
+ "multiword_shown": 0.036,
290
+ "multiword_right": 0.7222,
291
+ "saved_with_tab": 0.0378,
292
+ "saved_word_by_word": 0.0408
293
+ },
294
+ "email/typing": {
295
+ "pauses": 1000,
296
+ "shown": 0.05,
297
+ "right": 0.84,
298
+ "avg_right_chars": 5.45,
299
+ "multiword_shown": 0.006,
300
+ "multiword_right": 0.5,
301
+ "saved_with_tab": 0.0191,
302
+ "saved_word_by_word": 0.0192
303
+ },
304
+ "email/editing": {
305
+ "pauses": 1000,
306
+ "shown": 0.054,
307
+ "right": 0.9074,
308
+ "avg_right_chars": 5.16,
309
+ "multiword_shown": 0.01,
310
+ "multiword_right": 0.9,
311
+ "saved_with_tab": 0.0211,
312
+ "saved_word_by_word": 0.0211
313
+ }
314
+ }
315
+ },
316
+ "examples": [
317
+ {
318
+ "domain": "articles",
319
+ "scenario": "editing",
320
+ "before": " the Spaghetti Western film genre) and Hate Boat, were publi",
321
+ "suggestion": "shed",
322
+ "truth": "shed by Comico C",
323
+ "right": true
324
+ },
325
+ {
326
+ "domain": "messages",
327
+ "scenario": "editing",
328
+ "before": "e optimal level of production and pricing for businesses. L'",
329
+ "suggestion": "Hpital's Rule",
330
+ "truth": "Hpital's Rule can help an",
331
+ "right": true
332
+ },
333
+ {
334
+ "domain": "email",
335
+ "scenario": "editing",
336
+ "before": "ured products and marketing of electricity for the United St",
337
+ "suggestion": "ates",
338
+ "truth": "ates ? Actively ",
339
+ "right": true
340
+ }
341
+ ],
342
+ "latency_ms": {
343
+ "p50": 16.88,
344
+ "p95": 43.43,
345
+ "note": "Measured in this eval process, not in a browser."
346
+ }
347
+ }
v2/metrics/tinynext-v2-q8-onnx.json CHANGED
@@ -5,187 +5,187 @@
5
  "0.3": {
6
  "articles/typing": {
7
  "pauses": 300,
8
- "shown": 0.1767,
9
- "right": 0.5849,
10
- "avg_right_chars": 6.0,
11
- "multiword_shown": 0.0733,
12
- "multiword_right": 0.4545,
13
- "saved_with_tab": 0.0517,
14
- "saved_word_by_word": 0.0575
15
  },
16
  "articles/editing": {
17
  "pauses": 300,
18
- "shown": 0.13,
19
- "right": 0.6923,
20
- "avg_right_chars": 5.93,
21
- "multiword_shown": 0.0533,
22
- "multiword_right": 0.6875,
23
- "saved_with_tab": 0.0444,
24
- "saved_word_by_word": 0.0464
25
  },
26
  "messages/typing": {
27
  "pauses": 300,
28
- "shown": 0.3,
29
- "right": 0.6333,
30
- "avg_right_chars": 7.65,
31
- "multiword_shown": 0.1933,
32
- "multiword_right": 0.569,
33
- "saved_with_tab": 0.1211,
34
- "saved_word_by_word": 0.1403
35
  },
36
  "messages/editing": {
37
  "pauses": 300,
38
- "shown": 0.2767,
39
- "right": 0.5181,
40
- "avg_right_chars": 6.53,
41
- "multiword_shown": 0.14,
42
- "multiword_right": 0.5,
43
- "saved_with_tab": 0.0781,
44
- "saved_word_by_word": 0.0919
45
  },
46
  "email/typing": {
47
  "pauses": 300,
48
- "shown": 0.2333,
49
- "right": 0.5857,
50
- "avg_right_chars": 5.46,
51
- "multiword_shown": 0.0667,
52
- "multiword_right": 0.45,
53
- "saved_with_tab": 0.0622,
54
- "saved_word_by_word": 0.0714
55
  },
56
  "email/editing": {
57
  "pauses": 300,
58
- "shown": 0.2333,
59
- "right": 0.6,
60
- "avg_right_chars": 5.57,
61
- "multiword_shown": 0.0933,
62
- "multiword_right": 0.5357,
63
- "saved_with_tab": 0.065,
64
- "saved_word_by_word": 0.0703
65
  }
66
  },
67
  "0.4": {
68
  "articles/typing": {
69
  "pauses": 300,
70
- "shown": 0.1467,
71
- "right": 0.6818,
72
- "avg_right_chars": 6.0,
73
- "multiword_shown": 0.0567,
74
- "multiword_right": 0.5882,
75
- "saved_with_tab": 0.05,
76
- "saved_word_by_word": 0.0531
77
  },
78
  "articles/editing": {
79
  "pauses": 300,
80
- "shown": 0.0967,
81
- "right": 0.7586,
82
- "avg_right_chars": 5.82,
83
- "multiword_shown": 0.0367,
84
- "multiword_right": 0.9091,
85
- "saved_with_tab": 0.0356,
86
- "saved_word_by_word": 0.0358
87
  },
88
  "messages/typing": {
89
  "pauses": 300,
90
- "shown": 0.24,
91
- "right": 0.7361,
92
- "avg_right_chars": 7.26,
93
- "multiword_shown": 0.1333,
94
- "multiword_right": 0.65,
95
- "saved_with_tab": 0.1069,
96
- "saved_word_by_word": 0.1167
97
  },
98
  "messages/editing": {
99
  "pauses": 300,
100
- "shown": 0.17,
101
- "right": 0.6275,
102
- "avg_right_chars": 6.38,
103
- "multiword_shown": 0.0733,
104
- "multiword_right": 0.6364,
105
- "saved_with_tab": 0.0567,
106
- "saved_word_by_word": 0.0658
107
  },
108
  "email/typing": {
109
  "pauses": 300,
110
- "shown": 0.1867,
111
- "right": 0.75,
112
- "avg_right_chars": 5.48,
113
- "multiword_shown": 0.0467,
114
- "multiword_right": 0.5714,
115
- "saved_with_tab": 0.0639,
116
- "saved_word_by_word": 0.0678
117
  },
118
  "email/editing": {
119
  "pauses": 300,
120
- "shown": 0.1667,
121
- "right": 0.68,
122
- "avg_right_chars": 5.26,
123
- "multiword_shown": 0.06,
124
- "multiword_right": 0.6111,
125
- "saved_with_tab": 0.0497,
126
- "saved_word_by_word": 0.0528
127
  }
128
  },
129
  "0.5": {
130
  "articles/typing": {
131
  "pauses": 300,
132
- "shown": 0.11,
133
- "right": 0.7879,
134
- "avg_right_chars": 5.81,
135
- "multiword_shown": 0.0333,
136
- "multiword_right": 0.7,
137
  "saved_with_tab": 0.0419,
138
- "saved_word_by_word": 0.0425
139
  },
140
  "articles/editing": {
141
  "pauses": 300,
142
  "shown": 0.0833,
143
- "right": 0.8,
144
- "avg_right_chars": 5.4,
145
  "multiword_shown": 0.0267,
146
- "multiword_right": 0.875,
147
- "saved_with_tab": 0.03,
148
- "saved_word_by_word": 0.0303
149
  },
150
  "messages/typing": {
151
  "pauses": 300,
152
- "shown": 0.18,
153
- "right": 0.7963,
154
- "avg_right_chars": 6.98,
155
- "multiword_shown": 0.1,
156
- "multiword_right": 0.7,
157
- "saved_with_tab": 0.0833,
158
- "saved_word_by_word": 0.09
159
  },
160
  "messages/editing": {
161
  "pauses": 300,
162
- "shown": 0.1233,
163
- "right": 0.7297,
164
- "avg_right_chars": 6.37,
165
- "multiword_shown": 0.0567,
166
- "multiword_right": 0.6471,
167
- "saved_with_tab": 0.0478,
168
- "saved_word_by_word": 0.0561
169
  },
170
  "email/typing": {
171
  "pauses": 300,
172
- "shown": 0.14,
173
- "right": 0.7857,
174
- "avg_right_chars": 5.48,
175
- "multiword_shown": 0.0267,
176
- "multiword_right": 0.75,
177
- "saved_with_tab": 0.0503,
178
- "saved_word_by_word": 0.0514
179
  },
180
  "email/editing": {
181
  "pauses": 300,
182
- "shown": 0.1267,
183
- "right": 0.7632,
184
  "avg_right_chars": 5.07,
185
- "multiword_shown": 0.0367,
186
- "multiword_right": 0.5455,
187
- "saved_with_tab": 0.0408,
188
- "saved_word_by_word": 0.0431
189
  }
190
  },
191
  "0.6": {
@@ -193,150 +193,134 @@
193
  "pauses": 300,
194
  "shown": 0.0967,
195
  "right": 0.7931,
196
- "avg_right_chars": 5.96,
197
- "multiword_shown": 0.0267,
198
- "multiword_right": 0.75,
199
- "saved_with_tab": 0.0381,
200
- "saved_word_by_word": 0.0381
201
  },
202
  "articles/editing": {
203
  "pauses": 300,
204
- "shown": 0.05,
205
- "right": 0.9333,
206
- "avg_right_chars": 5.79,
207
  "multiword_shown": 0.02,
208
  "multiword_right": 1.0,
209
- "saved_with_tab": 0.0225,
210
- "saved_word_by_word": 0.0225
211
  },
212
  "messages/typing": {
213
  "pauses": 300,
214
- "shown": 0.1467,
215
- "right": 0.8864,
216
- "avg_right_chars": 6.54,
217
- "multiword_shown": 0.0733,
218
- "multiword_right": 0.8182,
219
- "saved_with_tab": 0.0708,
220
- "saved_word_by_word": 0.0742
221
  },
222
  "messages/editing": {
223
  "pauses": 300,
224
- "shown": 0.0767,
225
- "right": 0.7391,
226
- "avg_right_chars": 6.18,
227
- "multiword_shown": 0.04,
228
- "multiword_right": 0.5,
229
- "saved_with_tab": 0.0292,
230
- "saved_word_by_word": 0.0375
231
  },
232
  "email/typing": {
233
  "pauses": 300,
234
- "shown": 0.1,
235
- "right": 0.8667,
236
- "avg_right_chars": 5.73,
237
- "multiword_shown": 0.02,
238
  "multiword_right": 1.0,
239
- "saved_with_tab": 0.0414,
240
- "saved_word_by_word": 0.0414
241
  },
242
  "email/editing": {
243
  "pauses": 300,
244
- "shown": 0.09,
245
- "right": 0.8889,
246
- "avg_right_chars": 4.92,
247
- "multiword_shown": 0.0133,
248
- "multiword_right": 0.75,
249
- "saved_with_tab": 0.0328,
250
- "saved_word_by_word": 0.0339
251
  }
252
  },
253
  "0.7": {
254
  "articles/typing": {
255
  "pauses": 300,
256
- "shown": 0.0667,
257
- "right": 0.95,
258
- "avg_right_chars": 5.63,
259
- "multiword_shown": 0.0167,
260
- "multiword_right": 1.0,
261
  "saved_with_tab": 0.0297,
262
- "saved_word_by_word": 0.0297
263
  },
264
  "articles/editing": {
265
  "pauses": 300,
266
- "shown": 0.03,
267
- "right": 0.8889,
268
- "avg_right_chars": 5.12,
269
- "multiword_shown": 0.0033,
270
  "multiword_right": 1.0,
271
- "saved_with_tab": 0.0114,
272
- "saved_word_by_word": 0.0114
273
  },
274
  "messages/typing": {
275
  "pauses": 300,
276
- "shown": 0.1133,
277
- "right": 0.9412,
278
- "avg_right_chars": 6.22,
279
- "multiword_shown": 0.05,
280
- "multiword_right": 0.8667,
281
- "saved_with_tab": 0.0553,
282
- "saved_word_by_word": 0.0561
283
  },
284
  "messages/editing": {
285
  "pauses": 300,
286
- "shown": 0.0633,
287
- "right": 0.7895,
288
- "avg_right_chars": 5.93,
289
- "multiword_shown": 0.03,
290
- "multiword_right": 0.5556,
291
- "saved_with_tab": 0.0247,
292
- "saved_word_by_word": 0.0319
293
  },
294
  "email/typing": {
295
  "pauses": 300,
296
- "shown": 0.0667,
297
- "right": 0.85,
298
- "avg_right_chars": 5.41,
299
  "multiword_shown": 0.0067,
300
  "multiword_right": 1.0,
301
- "saved_with_tab": 0.0256,
302
- "saved_word_by_word": 0.0256
303
  },
304
  "email/editing": {
305
  "pauses": 300,
306
- "shown": 0.06,
307
- "right": 0.9444,
308
- "avg_right_chars": 4.71,
309
- "multiword_shown": 0.0033,
310
- "multiword_right": 1.0,
311
- "saved_with_tab": 0.0222,
312
- "saved_word_by_word": 0.0222
313
  }
314
  }
315
  },
316
  "examples": [
317
- {
318
- "domain": "articles",
319
- "scenario": "typing",
320
- "before": "elipads, IFR approaches, and a separate float plane dock alo",
321
- "suggestion": "ng the",
322
- "truth": "ng the Fraser Rive",
323
- "right": true
324
- },
325
  {
326
  "domain": "articles",
327
  "scenario": "editing",
328
- "before": " where he met his wife, Kisa, a graduate of the University o",
329
- "suggestion": "f Illinois",
330
- "truth": "f Colorado. Sasaki the",
331
- "right": false
332
- },
333
- {
334
- "domain": "articles",
335
- "scenario": "editing",
336
- "before": "ing any kind of direct discrimination that was prominent els",
337
- "suggestion": "ewhere.",
338
- "truth": "ewhere in the count",
339
- "right": false
340
  },
341
  {
342
  "domain": "messages",
@@ -358,8 +342,24 @@
358
  "domain": "messages",
359
  "scenario": "editing",
360
  "before": "e at which a population is growing.\nL'Hopital's rule can als",
361
- "suggestion": "o be used to",
362
  "truth": "o be used to solve probl",
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
363
  "right": true
364
  },
365
  {
@@ -389,9 +389,17 @@
389
  {
390
  "domain": "email",
391
  "scenario": "editing",
392
- "before": "I'm not sure what you are looking for. The Early Termination",
393
- "suggestion": " Log",
394
- "truth": " Date is one of ",
 
 
 
 
 
 
 
 
395
  "right": false
396
  },
397
  {
@@ -404,8 +412,8 @@
404
  }
405
  ],
406
  "latency_ms": {
407
- "p50": 25.25,
408
- "p95": 75.94,
409
  "note": "Measured in this eval process, not in a browser."
410
  }
411
  }
 
5
  "0.3": {
6
  "articles/typing": {
7
  "pauses": 300,
8
+ "shown": 0.2067,
9
+ "right": 0.5323,
10
+ "avg_right_chars": 6.18,
11
+ "multiword_shown": 0.0967,
12
+ "multiword_right": 0.5172,
13
+ "saved_with_tab": 0.0567,
14
+ "saved_word_by_word": 0.0647
15
  },
16
  "articles/editing": {
17
  "pauses": 300,
18
+ "shown": 0.1467,
19
+ "right": 0.7273,
20
+ "avg_right_chars": 6.06,
21
+ "multiword_shown": 0.0733,
22
+ "multiword_right": 0.7727,
23
+ "saved_with_tab": 0.0539,
24
+ "saved_word_by_word": 0.0572
25
  },
26
  "messages/typing": {
27
  "pauses": 300,
28
+ "shown": 0.3533,
29
+ "right": 0.566,
30
+ "avg_right_chars": 8.38,
31
+ "multiword_shown": 0.2133,
32
+ "multiword_right": 0.5938,
33
+ "saved_with_tab": 0.1397,
34
+ "saved_word_by_word": 0.1631
35
  },
36
  "messages/editing": {
37
  "pauses": 300,
38
+ "shown": 0.3033,
39
+ "right": 0.5165,
40
+ "avg_right_chars": 7.04,
41
+ "multiword_shown": 0.15,
42
+ "multiword_right": 0.4667,
43
+ "saved_with_tab": 0.0919,
44
+ "saved_word_by_word": 0.1108
45
  },
46
  "email/typing": {
47
  "pauses": 300,
48
+ "shown": 0.2433,
49
+ "right": 0.6164,
50
+ "avg_right_chars": 6.0,
51
+ "multiword_shown": 0.07,
52
+ "multiword_right": 0.4286,
53
+ "saved_with_tab": 0.075,
54
+ "saved_word_by_word": 0.0842
55
  },
56
  "email/editing": {
57
  "pauses": 300,
58
+ "shown": 0.2133,
59
+ "right": 0.625,
60
+ "avg_right_chars": 5.33,
61
+ "multiword_shown": 0.08,
62
+ "multiword_right": 0.375,
63
+ "saved_with_tab": 0.0592,
64
+ "saved_word_by_word": 0.0664
65
  }
66
  },
67
  "0.4": {
68
  "articles/typing": {
69
  "pauses": 300,
70
+ "shown": 0.1567,
71
+ "right": 0.617,
72
+ "avg_right_chars": 6.45,
73
+ "multiword_shown": 0.0667,
74
+ "multiword_right": 0.6,
75
+ "saved_with_tab": 0.0519,
76
+ "saved_word_by_word": 0.0556
77
  },
78
  "articles/editing": {
79
  "pauses": 300,
80
+ "shown": 0.1033,
81
+ "right": 0.871,
82
+ "avg_right_chars": 5.48,
83
+ "multiword_shown": 0.0333,
84
+ "multiword_right": 0.9,
85
+ "saved_with_tab": 0.0411,
86
+ "saved_word_by_word": 0.0414
87
  },
88
  "messages/typing": {
89
  "pauses": 300,
90
+ "shown": 0.2533,
91
+ "right": 0.6711,
92
+ "avg_right_chars": 8.37,
93
+ "multiword_shown": 0.15,
94
+ "multiword_right": 0.6667,
95
+ "saved_with_tab": 0.1186,
96
+ "saved_word_by_word": 0.1319
97
  },
98
  "messages/editing": {
99
  "pauses": 300,
100
+ "shown": 0.2433,
101
+ "right": 0.6027,
102
+ "avg_right_chars": 6.52,
103
+ "multiword_shown": 0.1133,
104
+ "multiword_right": 0.5,
105
+ "saved_with_tab": 0.0797,
106
+ "saved_word_by_word": 0.0947
107
  },
108
  "email/typing": {
109
  "pauses": 300,
110
+ "shown": 0.1967,
111
+ "right": 0.6949,
112
+ "avg_right_chars": 6.07,
113
+ "multiword_shown": 0.05,
114
+ "multiword_right": 0.6,
115
+ "saved_with_tab": 0.0692,
116
+ "saved_word_by_word": 0.0728
117
  },
118
  "email/editing": {
119
  "pauses": 300,
120
+ "shown": 0.17,
121
+ "right": 0.7255,
122
+ "avg_right_chars": 5.24,
123
+ "multiword_shown": 0.0567,
124
+ "multiword_right": 0.4706,
125
+ "saved_with_tab": 0.0539,
126
+ "saved_word_by_word": 0.0583
127
  }
128
  },
129
  "0.5": {
130
  "articles/typing": {
131
  "pauses": 300,
132
+ "shown": 0.1133,
133
+ "right": 0.7059,
134
+ "avg_right_chars": 6.29,
135
+ "multiword_shown": 0.0433,
136
+ "multiword_right": 0.6923,
137
  "saved_with_tab": 0.0419,
138
+ "saved_word_by_word": 0.0436
139
  },
140
  "articles/editing": {
141
  "pauses": 300,
142
  "shown": 0.0833,
143
+ "right": 0.96,
144
+ "avg_right_chars": 5.46,
145
  "multiword_shown": 0.0267,
146
+ "multiword_right": 1.0,
147
+ "saved_with_tab": 0.0364,
148
+ "saved_word_by_word": 0.0364
149
  },
150
  "messages/typing": {
151
  "pauses": 300,
152
+ "shown": 0.1933,
153
+ "right": 0.7586,
154
+ "avg_right_chars": 7.91,
155
+ "multiword_shown": 0.1133,
156
+ "multiword_right": 0.7059,
157
+ "saved_with_tab": 0.0967,
158
+ "saved_word_by_word": 0.1067
159
  },
160
  "messages/editing": {
161
  "pauses": 300,
162
+ "shown": 0.1533,
163
+ "right": 0.6087,
164
+ "avg_right_chars": 6.36,
165
+ "multiword_shown": 0.0733,
166
+ "multiword_right": 0.5455,
167
+ "saved_with_tab": 0.0494,
168
+ "saved_word_by_word": 0.0625
169
  },
170
  "email/typing": {
171
  "pauses": 300,
172
+ "shown": 0.13,
173
+ "right": 0.8462,
174
+ "avg_right_chars": 5.85,
175
+ "multiword_shown": 0.0367,
176
+ "multiword_right": 0.8182,
177
+ "saved_with_tab": 0.0536,
178
+ "saved_word_by_word": 0.0547
179
  },
180
  "email/editing": {
181
  "pauses": 300,
182
+ "shown": 0.12,
183
+ "right": 0.7778,
184
  "avg_right_chars": 5.07,
185
+ "multiword_shown": 0.0267,
186
+ "multiword_right": 0.5,
187
+ "saved_with_tab": 0.0394,
188
+ "saved_word_by_word": 0.0419
189
  }
190
  },
191
  "0.6": {
 
193
  "pauses": 300,
194
  "shown": 0.0967,
195
  "right": 0.7931,
196
+ "avg_right_chars": 6.0,
197
+ "multiword_shown": 0.0333,
198
+ "multiword_right": 0.8,
199
+ "saved_with_tab": 0.0383,
200
+ "saved_word_by_word": 0.0389
201
  },
202
  "articles/editing": {
203
  "pauses": 300,
204
+ "shown": 0.0733,
205
+ "right": 1.0,
206
+ "avg_right_chars": 5.41,
207
  "multiword_shown": 0.02,
208
  "multiword_right": 1.0,
209
+ "saved_with_tab": 0.0331,
210
+ "saved_word_by_word": 0.0331
211
  },
212
  "messages/typing": {
213
  "pauses": 300,
214
+ "shown": 0.16,
215
+ "right": 0.8958,
216
+ "avg_right_chars": 7.16,
217
+ "multiword_shown": 0.0767,
218
+ "multiword_right": 0.8696,
219
+ "saved_with_tab": 0.0856,
220
+ "saved_word_by_word": 0.0886
221
  },
222
  "messages/editing": {
223
  "pauses": 300,
224
+ "shown": 0.1067,
225
+ "right": 0.6875,
226
+ "avg_right_chars": 6.0,
227
+ "multiword_shown": 0.05,
228
+ "multiword_right": 0.6,
229
+ "saved_with_tab": 0.0367,
230
+ "saved_word_by_word": 0.0453
231
  },
232
  "email/typing": {
233
  "pauses": 300,
234
+ "shown": 0.1067,
235
+ "right": 0.9062,
236
+ "avg_right_chars": 5.38,
237
+ "multiword_shown": 0.0133,
238
  "multiword_right": 1.0,
239
+ "saved_with_tab": 0.0433,
240
+ "saved_word_by_word": 0.0433
241
  },
242
  "email/editing": {
243
  "pauses": 300,
244
+ "shown": 0.0833,
245
+ "right": 0.88,
246
+ "avg_right_chars": 4.86,
247
+ "multiword_shown": 0.01,
248
+ "multiword_right": 0.3333,
249
+ "saved_with_tab": 0.0297,
250
+ "saved_word_by_word": 0.0308
251
  }
252
  },
253
  "0.7": {
254
  "articles/typing": {
255
  "pauses": 300,
256
+ "shown": 0.0733,
257
+ "right": 0.8182,
258
+ "avg_right_chars": 5.94,
259
+ "multiword_shown": 0.0233,
260
+ "multiword_right": 0.7143,
261
  "saved_with_tab": 0.0297,
262
+ "saved_word_by_word": 0.0303
263
  },
264
  "articles/editing": {
265
  "pauses": 300,
266
+ "shown": 0.05,
267
+ "right": 1.0,
268
+ "avg_right_chars": 5.47,
269
+ "multiword_shown": 0.0133,
270
  "multiword_right": 1.0,
271
+ "saved_with_tab": 0.0228,
272
+ "saved_word_by_word": 0.0228
273
  },
274
  "messages/typing": {
275
  "pauses": 300,
276
+ "shown": 0.1333,
277
+ "right": 0.925,
278
+ "avg_right_chars": 6.57,
279
+ "multiword_shown": 0.06,
280
+ "multiword_right": 0.8889,
281
+ "saved_with_tab": 0.0675,
282
+ "saved_word_by_word": 0.0689
283
  },
284
  "messages/editing": {
285
  "pauses": 300,
286
+ "shown": 0.0733,
287
+ "right": 0.8182,
288
+ "avg_right_chars": 5.94,
289
+ "multiword_shown": 0.0233,
290
+ "multiword_right": 0.8571,
291
+ "saved_with_tab": 0.0297,
292
+ "saved_word_by_word": 0.0325
293
  },
294
  "email/typing": {
295
  "pauses": 300,
296
+ "shown": 0.0833,
297
+ "right": 0.92,
298
+ "avg_right_chars": 5.04,
299
  "multiword_shown": 0.0067,
300
  "multiword_right": 1.0,
301
+ "saved_with_tab": 0.0322,
302
+ "saved_word_by_word": 0.0322
303
  },
304
  "email/editing": {
305
  "pauses": 300,
306
+ "shown": 0.0733,
307
+ "right": 0.9091,
308
+ "avg_right_chars": 4.65,
309
+ "multiword_shown": 0.0067,
310
+ "multiword_right": 0.0,
311
+ "saved_with_tab": 0.0258,
312
+ "saved_word_by_word": 0.0269
313
  }
314
  }
315
  },
316
  "examples": [
 
 
 
 
 
 
 
 
317
  {
318
  "domain": "articles",
319
  "scenario": "editing",
320
+ "before": "le tree, with a number of ecotypes, and in forms closely res",
321
+ "suggestion": "embling",
322
+ "truth": "embling other speci",
323
+ "right": true
 
 
 
 
 
 
 
 
324
  },
325
  {
326
  "domain": "messages",
 
342
  "domain": "messages",
343
  "scenario": "editing",
344
  "before": "e at which a population is growing.\nL'Hopital's rule can als",
345
+ "suggestion": "o be used in",
346
  "truth": "o be used to solve probl",
347
+ "right": false
348
+ },
349
+ {
350
+ "domain": "messages",
351
+ "scenario": "editing",
352
+ "before": " to choose exactly which components you want to use, and cus",
353
+ "suggestion": "tomize",
354
+ "truth": "tomize your machin",
355
+ "right": true
356
+ },
357
+ {
358
+ "domain": "messages",
359
+ "scenario": "editing",
360
+ "before": "vel of production and pricing for businesses. L'Hpital's Rul",
361
+ "suggestion": "e can",
362
+ "truth": "e can help analyz",
363
  "right": true
364
  },
365
  {
 
389
  {
390
  "domain": "email",
391
  "scenario": "editing",
392
+ "before": "tives and structured products ? Ability to effectively work ",
393
+ "suggestion": "with",
394
+ "truth": "with marketers p",
395
+ "right": true
396
+ },
397
+ {
398
+ "domain": "email",
399
+ "scenario": "editing",
400
+ "before": "Phillip/Tim/Yevgeny\nPlease review th",
401
+ "suggestion": "e attached",
402
+ "truth": "e two attachments belo",
403
  "right": false
404
  },
405
  {
 
412
  }
413
  ],
414
  "latency_ms": {
415
+ "p50": 13.45,
416
+ "p95": 31.85,
417
  "note": "Measured in this eval process, not in a browser."
418
  }
419
  }
v2/metrics/tinynext-v2.json CHANGED
@@ -1,315 +1,315 @@
1
  {
2
- "model": "tinynext-v2",
3
  "sees_after": true,
4
  "cutoffs": {
5
  "0.3": {
6
  "articles/typing": {
7
  "pauses": 1000,
8
- "shown": 0.188,
9
- "right": 0.617,
10
- "avg_right_chars": 5.91,
11
- "multiword_shown": 0.067,
12
- "multiword_right": 0.5373,
13
- "saved_with_tab": 0.0572,
14
- "saved_word_by_word": 0.0626
15
  },
16
  "articles/editing": {
17
  "pauses": 1000,
18
- "shown": 0.148,
19
- "right": 0.6081,
20
- "avg_right_chars": 5.94,
21
- "multiword_shown": 0.058,
22
- "multiword_right": 0.5862,
23
- "saved_with_tab": 0.0446,
24
- "saved_word_by_word": 0.0491
25
  },
26
  "messages/typing": {
27
  "pauses": 1000,
28
- "shown": 0.293,
29
- "right": 0.5563,
30
- "avg_right_chars": 7.71,
31
- "multiword_shown": 0.163,
32
- "multiword_right": 0.546,
33
- "saved_with_tab": 0.1047,
34
- "saved_word_by_word": 0.1232
35
  },
36
  "messages/editing": {
37
  "pauses": 1000,
38
- "shown": 0.276,
39
- "right": 0.6159,
40
- "avg_right_chars": 7.49,
41
- "multiword_shown": 0.152,
42
- "multiword_right": 0.6118,
43
- "saved_with_tab": 0.1061,
44
- "saved_word_by_word": 0.1183
45
  },
46
  "email/typing": {
47
  "pauses": 1000,
48
- "shown": 0.237,
49
- "right": 0.5443,
50
- "avg_right_chars": 6.09,
51
- "multiword_shown": 0.084,
52
- "multiword_right": 0.3929,
53
- "saved_with_tab": 0.0655,
54
- "saved_word_by_word": 0.0744
55
  },
56
  "email/editing": {
57
  "pauses": 1000,
58
- "shown": 0.22,
59
- "right": 0.5273,
60
- "avg_right_chars": 5.77,
61
- "multiword_shown": 0.093,
62
- "multiword_right": 0.4194,
63
- "saved_with_tab": 0.0558,
64
- "saved_word_by_word": 0.0645
65
  }
66
  },
67
  "0.4": {
68
  "articles/typing": {
69
  "pauses": 1000,
70
- "shown": 0.14,
71
- "right": 0.7214,
72
- "avg_right_chars": 5.94,
73
- "multiword_shown": 0.043,
74
- "multiword_right": 0.6744,
75
- "saved_with_tab": 0.05,
76
- "saved_word_by_word": 0.0522
77
  },
78
  "articles/editing": {
79
  "pauses": 1000,
80
- "shown": 0.102,
81
- "right": 0.6667,
82
- "avg_right_chars": 5.65,
83
  "multiword_shown": 0.034,
84
- "multiword_right": 0.7353,
85
- "saved_with_tab": 0.032,
86
- "saved_word_by_word": 0.0341
87
  },
88
  "messages/typing": {
89
  "pauses": 1000,
90
- "shown": 0.212,
91
- "right": 0.6934,
92
- "avg_right_chars": 7.19,
93
- "multiword_shown": 0.105,
94
- "multiword_right": 0.6571,
95
- "saved_with_tab": 0.0881,
96
- "saved_word_by_word": 0.0958
97
  },
98
  "messages/editing": {
99
  "pauses": 1000,
100
- "shown": 0.194,
101
- "right": 0.7113,
102
- "avg_right_chars": 6.83,
103
- "multiword_shown": 0.097,
104
- "multiword_right": 0.6907,
105
- "saved_with_tab": 0.0785,
106
- "saved_word_by_word": 0.086
107
  },
108
  "email/typing": {
109
  "pauses": 1000,
110
- "shown": 0.172,
111
- "right": 0.686,
112
- "avg_right_chars": 5.86,
113
- "multiword_shown": 0.053,
114
- "multiword_right": 0.434,
115
- "saved_with_tab": 0.0576,
116
- "saved_word_by_word": 0.0618
117
  },
118
  "email/editing": {
119
  "pauses": 1000,
120
- "shown": 0.147,
121
- "right": 0.6395,
122
- "avg_right_chars": 5.48,
123
- "multiword_shown": 0.057,
124
- "multiword_right": 0.5439,
125
- "saved_with_tab": 0.0429,
126
- "saved_word_by_word": 0.0459
127
  }
128
  },
129
  "0.5": {
130
  "articles/typing": {
131
  "pauses": 1000,
132
- "shown": 0.106,
133
- "right": 0.7642,
134
- "avg_right_chars": 5.9,
135
- "multiword_shown": 0.034,
136
- "multiword_right": 0.7353,
137
- "saved_with_tab": 0.0398,
138
- "saved_word_by_word": 0.0411
139
  },
140
  "articles/editing": {
141
  "pauses": 1000,
142
- "shown": 0.078,
143
- "right": 0.7692,
144
- "avg_right_chars": 5.55,
145
- "multiword_shown": 0.022,
146
- "multiword_right": 0.8636,
147
- "saved_with_tab": 0.0278,
148
- "saved_word_by_word": 0.0282
149
  },
150
  "messages/typing": {
151
  "pauses": 1000,
152
- "shown": 0.163,
153
- "right": 0.7791,
154
- "avg_right_chars": 6.86,
155
- "multiword_shown": 0.078,
156
- "multiword_right": 0.7179,
157
- "saved_with_tab": 0.0726,
158
- "saved_word_by_word": 0.078
159
  },
160
  "messages/editing": {
161
  "pauses": 1000,
162
- "shown": 0.146,
163
- "right": 0.7877,
164
- "avg_right_chars": 6.63,
165
- "multiword_shown": 0.071,
166
- "multiword_right": 0.7465,
167
- "saved_with_tab": 0.0635,
168
- "saved_word_by_word": 0.0679
169
  },
170
  "email/typing": {
171
  "pauses": 1000,
172
- "shown": 0.113,
173
- "right": 0.7876,
174
- "avg_right_chars": 5.84,
175
- "multiword_shown": 0.03,
176
- "multiword_right": 0.6,
177
- "saved_with_tab": 0.0433,
178
- "saved_word_by_word": 0.0452
179
  },
180
  "email/editing": {
181
  "pauses": 1000,
182
- "shown": 0.104,
183
- "right": 0.7788,
184
- "avg_right_chars": 5.32,
185
- "multiword_shown": 0.03,
186
- "multiword_right": 0.7,
187
- "saved_with_tab": 0.0359,
188
- "saved_word_by_word": 0.0369
189
  }
190
  },
191
  "0.6": {
192
  "articles/typing": {
193
  "pauses": 1000,
194
- "shown": 0.084,
195
- "right": 0.8571,
196
- "avg_right_chars": 5.86,
197
- "multiword_shown": 0.022,
198
- "multiword_right": 0.8636,
199
- "saved_with_tab": 0.0352,
200
- "saved_word_by_word": 0.0353
201
  },
202
  "articles/editing": {
203
  "pauses": 1000,
204
- "shown": 0.056,
205
- "right": 0.8571,
206
- "avg_right_chars": 5.62,
207
- "multiword_shown": 0.016,
208
- "multiword_right": 0.9375,
209
- "saved_with_tab": 0.0225,
210
- "saved_word_by_word": 0.0227
211
  },
212
  "messages/typing": {
213
  "pauses": 1000,
214
- "shown": 0.123,
215
- "right": 0.8455,
216
- "avg_right_chars": 6.73,
217
- "multiword_shown": 0.06,
218
- "multiword_right": 0.8167,
219
- "saved_with_tab": 0.0583,
220
- "saved_word_by_word": 0.061
221
  },
222
  "messages/editing": {
223
  "pauses": 1000,
224
- "shown": 0.108,
225
- "right": 0.8426,
226
- "avg_right_chars": 6.31,
227
- "multiword_shown": 0.052,
228
- "multiword_right": 0.7308,
229
- "saved_with_tab": 0.0478,
230
- "saved_word_by_word": 0.0512
231
  },
232
  "email/typing": {
233
  "pauses": 1000,
234
- "shown": 0.076,
235
- "right": 0.8816,
236
- "avg_right_chars": 5.79,
237
- "multiword_shown": 0.016,
238
- "multiword_right": 0.8125,
239
- "saved_with_tab": 0.0323,
240
- "saved_word_by_word": 0.0324
241
  },
242
  "email/editing": {
243
  "pauses": 1000,
244
- "shown": 0.071,
245
- "right": 0.8873,
246
- "avg_right_chars": 5.25,
247
- "multiword_shown": 0.014,
248
- "multiword_right": 0.8571,
249
- "saved_with_tab": 0.0276,
250
- "saved_word_by_word": 0.0279
251
  }
252
  },
253
  "0.7": {
254
  "articles/typing": {
255
  "pauses": 1000,
256
- "shown": 0.059,
257
- "right": 0.9153,
258
- "avg_right_chars": 5.76,
259
- "multiword_shown": 0.018,
260
- "multiword_right": 0.9444,
261
- "saved_with_tab": 0.0259,
262
- "saved_word_by_word": 0.0261
263
  },
264
  "articles/editing": {
265
  "pauses": 1000,
266
- "shown": 0.035,
267
- "right": 0.8286,
268
- "avg_right_chars": 5.34,
269
- "multiword_shown": 0.007,
270
- "multiword_right": 0.8571,
271
- "saved_with_tab": 0.0129,
272
- "saved_word_by_word": 0.0132
273
  },
274
  "messages/typing": {
275
  "pauses": 1000,
276
- "shown": 0.099,
277
- "right": 0.9293,
278
- "avg_right_chars": 6.37,
279
- "multiword_shown": 0.042,
280
- "multiword_right": 0.9048,
281
- "saved_with_tab": 0.0488,
282
- "saved_word_by_word": 0.0492
283
  },
284
  "messages/editing": {
285
  "pauses": 1000,
286
- "shown": 0.086,
287
- "right": 0.8837,
288
- "avg_right_chars": 5.97,
289
- "multiword_shown": 0.036,
290
- "multiword_right": 0.7222,
291
- "saved_with_tab": 0.0378,
292
- "saved_word_by_word": 0.0408
293
  },
294
  "email/typing": {
295
  "pauses": 1000,
296
- "shown": 0.05,
297
- "right": 0.84,
298
- "avg_right_chars": 5.45,
299
- "multiword_shown": 0.006,
300
- "multiword_right": 0.5,
301
- "saved_with_tab": 0.0191,
302
- "saved_word_by_word": 0.0192
303
  },
304
  "email/editing": {
305
  "pauses": 1000,
306
- "shown": 0.054,
307
- "right": 0.9074,
308
- "avg_right_chars": 5.16,
309
- "multiword_shown": 0.01,
310
- "multiword_right": 0.9,
311
- "saved_with_tab": 0.0211,
312
- "saved_word_by_word": 0.0211
313
  }
314
  }
315
  },
@@ -322,12 +322,36 @@
322
  "truth": "shed by Comico C",
323
  "right": true
324
  },
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
325
  {
326
  "domain": "messages",
327
  "scenario": "editing",
328
  "before": "e optimal level of production and pricing for businesses. L'",
329
- "suggestion": "Hpital's Rule",
330
- "truth": "Hpital's Rule can help an",
331
  "right": true
332
  },
333
  {
@@ -340,8 +364,8 @@
340
  }
341
  ],
342
  "latency_ms": {
343
- "p50": 16.88,
344
- "p95": 43.43,
345
  "note": "Measured in this eval process, not in a browser."
346
  }
347
  }
 
1
  {
2
+ "model": "tinynext-v2-3b",
3
  "sees_after": true,
4
  "cutoffs": {
5
  "0.3": {
6
  "articles/typing": {
7
  "pauses": 1000,
8
+ "shown": 0.204,
9
+ "right": 0.5882,
10
+ "avg_right_chars": 6.04,
11
+ "multiword_shown": 0.086,
12
+ "multiword_right": 0.5465,
13
+ "saved_with_tab": 0.0604,
14
+ "saved_word_by_word": 0.0679
15
  },
16
  "articles/editing": {
17
  "pauses": 1000,
18
+ "shown": 0.152,
19
+ "right": 0.6382,
20
+ "avg_right_chars": 5.86,
21
+ "multiword_shown": 0.062,
22
+ "multiword_right": 0.629,
23
+ "saved_with_tab": 0.0473,
24
+ "saved_word_by_word": 0.0516
25
  },
26
  "messages/typing": {
27
  "pauses": 1000,
28
+ "shown": 0.324,
29
+ "right": 0.5463,
30
+ "avg_right_chars": 7.8,
31
+ "multiword_shown": 0.18,
32
+ "multiword_right": 0.5333,
33
+ "saved_with_tab": 0.115,
34
+ "saved_word_by_word": 0.1404
35
  },
36
  "messages/editing": {
37
  "pauses": 1000,
38
+ "shown": 0.317,
39
+ "right": 0.5584,
40
+ "avg_right_chars": 7.47,
41
+ "multiword_shown": 0.173,
42
+ "multiword_right": 0.5376,
43
+ "saved_with_tab": 0.1102,
44
+ "saved_word_by_word": 0.1328
45
  },
46
  "email/typing": {
47
  "pauses": 1000,
48
+ "shown": 0.234,
49
+ "right": 0.5598,
50
+ "avg_right_chars": 6.03,
51
+ "multiword_shown": 0.087,
52
+ "multiword_right": 0.4023,
53
+ "saved_with_tab": 0.0658,
54
+ "saved_word_by_word": 0.0746
55
  },
56
  "email/editing": {
57
  "pauses": 1000,
58
+ "shown": 0.213,
59
+ "right": 0.5915,
60
+ "avg_right_chars": 5.63,
61
+ "multiword_shown": 0.078,
62
+ "multiword_right": 0.4231,
63
+ "saved_with_tab": 0.0591,
64
+ "saved_word_by_word": 0.0653
65
  }
66
  },
67
  "0.4": {
68
  "articles/typing": {
69
  "pauses": 1000,
70
+ "shown": 0.149,
71
+ "right": 0.698,
72
+ "avg_right_chars": 6.12,
73
+ "multiword_shown": 0.057,
74
+ "multiword_right": 0.614,
75
+ "saved_with_tab": 0.0531,
76
+ "saved_word_by_word": 0.0563
77
  },
78
  "articles/editing": {
79
  "pauses": 1000,
80
+ "shown": 0.107,
81
+ "right": 0.7103,
82
+ "avg_right_chars": 5.72,
83
  "multiword_shown": 0.034,
84
+ "multiword_right": 0.6765,
85
+ "saved_with_tab": 0.0362,
86
+ "saved_word_by_word": 0.0384
87
  },
88
  "messages/typing": {
89
  "pauses": 1000,
90
+ "shown": 0.251,
91
+ "right": 0.6255,
92
+ "avg_right_chars": 7.7,
93
+ "multiword_shown": 0.133,
94
+ "multiword_right": 0.6015,
95
+ "saved_with_tab": 0.1008,
96
+ "saved_word_by_word": 0.1159
97
  },
98
  "messages/editing": {
99
  "pauses": 1000,
100
+ "shown": 0.239,
101
+ "right": 0.6569,
102
+ "avg_right_chars": 7.29,
103
+ "multiword_shown": 0.127,
104
+ "multiword_right": 0.6378,
105
+ "saved_with_tab": 0.0954,
106
+ "saved_word_by_word": 0.1078
107
  },
108
  "email/typing": {
109
  "pauses": 1000,
110
+ "shown": 0.162,
111
+ "right": 0.679,
112
+ "avg_right_chars": 5.84,
113
+ "multiword_shown": 0.049,
114
+ "multiword_right": 0.4898,
115
+ "saved_with_tab": 0.0535,
116
+ "saved_word_by_word": 0.0575
117
  },
118
  "email/editing": {
119
  "pauses": 1000,
120
+ "shown": 0.165,
121
+ "right": 0.6727,
122
+ "avg_right_chars": 5.56,
123
+ "multiword_shown": 0.055,
124
+ "multiword_right": 0.5818,
125
+ "saved_with_tab": 0.0514,
126
+ "saved_word_by_word": 0.0545
127
  }
128
  },
129
  "0.5": {
130
  "articles/typing": {
131
  "pauses": 1000,
132
+ "shown": 0.101,
133
+ "right": 0.7822,
134
+ "avg_right_chars": 5.86,
135
+ "multiword_shown": 0.035,
136
+ "multiword_right": 0.7143,
137
+ "saved_with_tab": 0.0386,
138
+ "saved_word_by_word": 0.04
139
  },
140
  "articles/editing": {
141
  "pauses": 1000,
142
+ "shown": 0.08,
143
+ "right": 0.825,
144
+ "avg_right_chars": 5.61,
145
+ "multiword_shown": 0.024,
146
+ "multiword_right": 0.8333,
147
+ "saved_with_tab": 0.0308,
148
+ "saved_word_by_word": 0.0317
149
  },
150
  "messages/typing": {
151
  "pauses": 1000,
152
+ "shown": 0.193,
153
+ "right": 0.7461,
154
+ "avg_right_chars": 7.31,
155
+ "multiword_shown": 0.094,
156
+ "multiword_right": 0.7021,
157
+ "saved_with_tab": 0.0877,
158
+ "saved_word_by_word": 0.0969
159
  },
160
  "messages/editing": {
161
  "pauses": 1000,
162
+ "shown": 0.175,
163
+ "right": 0.76,
164
+ "avg_right_chars": 6.87,
165
+ "multiword_shown": 0.091,
166
+ "multiword_right": 0.7033,
167
+ "saved_with_tab": 0.0762,
168
+ "saved_word_by_word": 0.0843
169
  },
170
  "email/typing": {
171
  "pauses": 1000,
172
+ "shown": 0.112,
173
+ "right": 0.7768,
174
+ "avg_right_chars": 5.54,
175
+ "multiword_shown": 0.035,
176
+ "multiword_right": 0.5143,
177
+ "saved_with_tab": 0.0402,
178
+ "saved_word_by_word": 0.0427
179
  },
180
  "email/editing": {
181
  "pauses": 1000,
182
+ "shown": 0.118,
183
+ "right": 0.7797,
184
+ "avg_right_chars": 5.5,
185
+ "multiword_shown": 0.031,
186
+ "multiword_right": 0.7097,
187
+ "saved_with_tab": 0.0422,
188
+ "saved_word_by_word": 0.0435
189
  }
190
  },
191
  "0.6": {
192
  "articles/typing": {
193
  "pauses": 1000,
194
+ "shown": 0.076,
195
+ "right": 0.8026,
196
+ "avg_right_chars": 5.77,
197
+ "multiword_shown": 0.026,
198
+ "multiword_right": 0.7692,
199
+ "saved_with_tab": 0.0293,
200
+ "saved_word_by_word": 0.0303
201
  },
202
  "articles/editing": {
203
  "pauses": 1000,
204
+ "shown": 0.059,
205
+ "right": 0.8983,
206
+ "avg_right_chars": 5.45,
207
+ "multiword_shown": 0.014,
208
+ "multiword_right": 0.7857,
209
+ "saved_with_tab": 0.0241,
210
+ "saved_word_by_word": 0.0244
211
  },
212
  "messages/typing": {
213
  "pauses": 1000,
214
+ "shown": 0.152,
215
+ "right": 0.8421,
216
+ "avg_right_chars": 7.22,
217
+ "multiword_shown": 0.068,
218
+ "multiword_right": 0.8382,
219
+ "saved_with_tab": 0.077,
220
+ "saved_word_by_word": 0.0795
221
  },
222
  "messages/editing": {
223
  "pauses": 1000,
224
+ "shown": 0.141,
225
+ "right": 0.8369,
226
+ "avg_right_chars": 6.39,
227
+ "multiword_shown": 0.065,
228
+ "multiword_right": 0.7538,
229
+ "saved_with_tab": 0.0628,
230
+ "saved_word_by_word": 0.0669
231
  },
232
  "email/typing": {
233
  "pauses": 1000,
234
+ "shown": 0.086,
235
+ "right": 0.8256,
236
+ "avg_right_chars": 5.23,
237
+ "multiword_shown": 0.015,
238
+ "multiword_right": 0.5333,
239
+ "saved_with_tab": 0.0309,
240
+ "saved_word_by_word": 0.0322
241
  },
242
  "email/editing": {
243
  "pauses": 1000,
244
+ "shown": 0.084,
245
+ "right": 0.8452,
246
+ "avg_right_chars": 5.34,
247
+ "multiword_shown": 0.016,
248
+ "multiword_right": 0.75,
249
+ "saved_with_tab": 0.0316,
250
+ "saved_word_by_word": 0.0319
251
  }
252
  },
253
  "0.7": {
254
  "articles/typing": {
255
  "pauses": 1000,
256
+ "shown": 0.052,
257
+ "right": 0.8654,
258
+ "avg_right_chars": 6.09,
259
+ "multiword_shown": 0.017,
260
+ "multiword_right": 0.8235,
261
+ "saved_with_tab": 0.0228,
262
+ "saved_word_by_word": 0.0232
263
  },
264
  "articles/editing": {
265
  "pauses": 1000,
266
+ "shown": 0.043,
267
+ "right": 0.9302,
268
+ "avg_right_chars": 5.25,
269
+ "multiword_shown": 0.01,
270
+ "multiword_right": 0.9,
271
+ "saved_with_tab": 0.0175,
272
+ "saved_word_by_word": 0.0177
273
  },
274
  "messages/typing": {
275
  "pauses": 1000,
276
+ "shown": 0.117,
277
+ "right": 0.8718,
278
+ "avg_right_chars": 6.64,
279
+ "multiword_shown": 0.045,
280
+ "multiword_right": 0.8889,
281
+ "saved_with_tab": 0.0564,
282
+ "saved_word_by_word": 0.0579
283
  },
284
  "messages/editing": {
285
  "pauses": 1000,
286
+ "shown": 0.105,
287
+ "right": 0.8857,
288
+ "avg_right_chars": 5.98,
289
+ "multiword_shown": 0.038,
290
+ "multiword_right": 0.8684,
291
+ "saved_with_tab": 0.0463,
292
+ "saved_word_by_word": 0.0476
293
  },
294
  "email/typing": {
295
  "pauses": 1000,
296
+ "shown": 0.058,
297
+ "right": 0.8103,
298
+ "avg_right_chars": 5.19,
299
+ "multiword_shown": 0.007,
300
+ "multiword_right": 0.4286,
301
+ "saved_with_tab": 0.0203,
302
+ "saved_word_by_word": 0.0206
303
  },
304
  "email/editing": {
305
  "pauses": 1000,
306
+ "shown": 0.066,
307
+ "right": 0.8788,
308
+ "avg_right_chars": 4.98,
309
+ "multiword_shown": 0.008,
310
+ "multiword_right": 0.5,
311
+ "saved_with_tab": 0.0241,
312
+ "saved_word_by_word": 0.0244
313
  }
314
  }
315
  },
 
322
  "truth": "shed by Comico C",
323
  "right": true
324
  },
325
+ {
326
+ "domain": "articles",
327
+ "scenario": "editing",
328
+ "before": "rked on a tour of Canada with I Hate Sally, Hiretsukan, GFK,",
329
+ "suggestion": " and",
330
+ "truth": " and DeadPoets a",
331
+ "right": true
332
+ },
333
+ {
334
+ "domain": "messages",
335
+ "scenario": "typing",
336
+ "before": " be completely benevolent and will always act in our best in",
337
+ "suggestion": "terests.",
338
+ "truth": "terests, ensuring th",
339
+ "right": false
340
+ },
341
+ {
342
+ "domain": "messages",
343
+ "scenario": "typing",
344
+ "before": "onsidered new. Those decisions make the graphs as neat as po",
345
+ "suggestion": "ssible.",
346
+ "truth": "ssible, and it is t",
347
+ "right": false
348
+ },
349
  {
350
  "domain": "messages",
351
  "scenario": "editing",
352
  "before": "e optimal level of production and pricing for businesses. L'",
353
+ "suggestion": "Hpital's Rule can",
354
+ "truth": "Hpital's Rule can help analyz",
355
  "right": true
356
  },
357
  {
 
364
  }
365
  ],
366
  "latency_ms": {
367
+ "p50": 15.38,
368
+ "p95": 42.67,
369
  "note": "Measured in this eval process, not in a browser."
370
  }
371
  }
v2/model.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:517b40693de7ff4618dbd6fe267416059e2b94cfa93d9e3a89c044008380b10f
3
- size 60813575
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d720a8d92844daca8da7820debd0be2c22d064b33de16ccbcfa69adf79307f13
3
+ size 60813561
v2/model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:6a8218faeefdd282109f42f22a019d28b19182bfa041c8f49039546726e63fea
3
  size 50161328
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0ffd7eaaee87262d76e9b401514eee2449e72b7aae899579b227c114d4a37366
3
  size 50161328
v2/model_int8.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:e247edf2479b5a5073f33fb418a450c1d6ba82efcb54e06be2a8b728dd7df65e
3
- size 23328690
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:34f405276119c07d523af9a5e43354ab50acbaa0e2be167166320401a3d35a2c
3
+ size 15638872
v2/report.json CHANGED
@@ -6,21 +6,22 @@
6
  "heads": 5,
7
  "ffn": 864,
8
  "seq": 512,
9
- "tokens": 1500000000.0,
10
  "val_tokens": 4000000.0,
11
  "epochs": 1.0,
12
  "batch": 512,
13
  "lr": 0.003,
14
  "fim_rate": 0.5,
15
  "mix": "fineweb=0.6,smoltalk=0.3,enron=0.1",
16
- "fineweb_files": 2,
17
  "smoltalk_files": 9,
18
  "tokenizer_docs": 200000,
19
  "eval_pauses": 1000,
20
  "log_every": 50,
21
  "repo": "cfahlgren1/tinynext",
22
- "folder": "v2",
23
- "name": "tinynext-v2",
 
24
  "trackio_space": "cfahlgren1/tinynext-trackio",
25
  "stage": "train",
26
  "data_repo": "cfahlgren1/tinynext-data",
@@ -30,1205 +31,2360 @@
30
  "parameters": 12539200,
31
  "data": {
32
  "fineweb": {
33
- "docs": 1062484,
34
- "train_tokens": 900000000
35
  },
36
  "smoltalk": {
37
- "docs": 671427,
38
- "train_tokens": 450000000
39
  },
40
  "enron": {
41
  "docs": 268120,
42
- "train_tokens": 150000000
43
  }
44
  },
45
  "data_minutes": 0.2,
46
- "train_minutes": 28.1,
47
  "history": [
48
  {
49
  "step": 0,
50
- "train_loss": 9.0697,
51
- "lr": 2.6e-05,
52
- "tokens_per_second": 13248,
53
- "minutes": 0.33,
54
  "mfu_percent": 0.3,
55
  "gpu_util_percent": 98,
56
- "gpu_memory_gb": 70.5,
57
- "val_loss": 9.0682
58
  },
59
  {
60
  "step": 50,
61
- "train_loss": 7.0154,
62
- "lr": 0.001342,
63
- "tokens_per_second": 932232,
64
- "minutes": 0.61,
65
- "mfu_percent": 22.5,
66
  "gpu_util_percent": 100,
67
- "gpu_memory_gb": 79.1
68
  },
69
  {
70
  "step": 100,
71
- "train_loss": 6.0698,
72
- "lr": 0.002656,
73
- "tokens_per_second": 926160,
74
- "minutes": 0.85,
75
  "mfu_percent": 22.3,
76
  "gpu_util_percent": 100,
77
- "gpu_memory_gb": 79.1
78
  },
79
  {
80
  "step": 150,
81
- "train_loss": 5.5539,
82
- "lr": 0.002995,
83
- "tokens_per_second": 923370,
84
- "minutes": 1.09,
85
  "mfu_percent": 22.3,
86
  "gpu_util_percent": 100,
87
- "gpu_memory_gb": 79.1
88
  },
89
  {
90
  "step": 200,
91
- "train_loss": 5.0756,
92
- "lr": 0.002992,
93
- "tokens_per_second": 922463,
94
- "minutes": 1.32,
95
- "mfu_percent": 22.2,
96
  "gpu_util_percent": 100,
97
- "gpu_memory_gb": 79.1
98
  },
99
  {
100
  "step": 250,
101
- "train_loss": 4.6591,
102
- "lr": 0.002987,
103
- "tokens_per_second": 923079,
104
- "minutes": 1.56,
105
  "mfu_percent": 22.3,
106
  "gpu_util_percent": 100,
107
- "gpu_memory_gb": 79.1
108
  },
109
  {
110
  "step": 300,
111
- "train_loss": 4.3203,
112
- "lr": 0.002982,
113
- "tokens_per_second": 922918,
114
- "minutes": 1.8,
115
  "mfu_percent": 22.3,
116
  "gpu_util_percent": 100,
117
- "gpu_memory_gb": 79.1
118
  },
119
  {
120
  "step": 350,
121
- "train_loss": 4.0751,
122
- "lr": 0.002975,
123
- "tokens_per_second": 922585,
124
- "minutes": 2.03,
125
- "mfu_percent": 22.2,
126
  "gpu_util_percent": 100,
127
- "gpu_memory_gb": 79.1
128
  },
129
  {
130
  "step": 400,
131
- "train_loss": 3.8986,
132
- "lr": 0.002967,
133
- "tokens_per_second": 923709,
134
- "minutes": 2.27,
135
- "mfu_percent": 22.3,
136
  "gpu_util_percent": 100,
137
- "gpu_memory_gb": 79.1
138
  },
139
  {
140
  "step": 450,
141
- "train_loss": 3.7565,
142
- "lr": 0.002959,
143
- "tokens_per_second": 923903,
144
- "minutes": 2.51,
145
- "mfu_percent": 22.3,
146
  "gpu_util_percent": 100,
147
- "gpu_memory_gb": 79.1
148
  },
149
  {
150
  "step": 500,
151
- "train_loss": 3.6807,
152
- "lr": 0.002949,
153
- "tokens_per_second": 923528,
154
- "minutes": 2.74,
155
- "mfu_percent": 22.3,
156
  "gpu_util_percent": 100,
157
- "gpu_memory_gb": 79.1,
158
- "val_loss": 3.7221
159
  },
160
  {
161
  "step": 550,
162
- "train_loss": 3.6681,
163
- "lr": 0.002938,
164
- "tokens_per_second": 922910,
165
- "minutes": 3.03,
166
- "mfu_percent": 22.3,
167
  "gpu_util_percent": 100,
168
- "gpu_memory_gb": 79.1
169
  },
170
  {
171
  "step": 600,
172
- "train_loss": 3.6275,
173
- "lr": 0.002927,
174
- "tokens_per_second": 921856,
175
- "minutes": 3.26,
176
  "mfu_percent": 22.2,
177
  "gpu_util_percent": 100,
178
- "gpu_memory_gb": 79.1
179
  },
180
  {
181
  "step": 650,
182
- "train_loss": 3.5235,
183
- "lr": 0.002914,
184
- "tokens_per_second": 920847,
185
- "minutes": 3.5,
186
  "mfu_percent": 22.2,
187
  "gpu_util_percent": 100,
188
- "gpu_memory_gb": 79.1
189
  },
190
  {
191
  "step": 700,
192
- "train_loss": 3.5007,
193
- "lr": 0.002901,
194
- "tokens_per_second": 920478,
195
- "minutes": 3.74,
196
  "mfu_percent": 22.2,
197
  "gpu_util_percent": 100,
198
- "gpu_memory_gb": 79.1
199
  },
200
  {
201
  "step": 750,
202
- "train_loss": 3.5063,
203
- "lr": 0.002886,
204
- "tokens_per_second": 920413,
205
- "minutes": 3.97,
206
  "mfu_percent": 22.2,
207
  "gpu_util_percent": 100,
208
- "gpu_memory_gb": 79.1
209
  },
210
  {
211
  "step": 800,
212
- "train_loss": 3.4786,
213
- "lr": 0.002871,
214
- "tokens_per_second": 919895,
215
- "minutes": 4.21,
216
  "mfu_percent": 22.2,
217
  "gpu_util_percent": 100,
218
- "gpu_memory_gb": 79.1
219
  },
220
  {
221
  "step": 850,
222
- "train_loss": 3.4053,
223
- "lr": 0.002855,
224
- "tokens_per_second": 919668,
225
- "minutes": 4.45,
226
  "mfu_percent": 22.2,
227
  "gpu_util_percent": 100,
228
- "gpu_memory_gb": 79.1
229
  },
230
  {
231
  "step": 900,
232
- "train_loss": 3.4465,
233
- "lr": 0.002837,
234
- "tokens_per_second": 919590,
235
- "minutes": 4.69,
236
  "mfu_percent": 22.2,
237
  "gpu_util_percent": 100,
238
- "gpu_memory_gb": 79.1
239
  },
240
  {
241
  "step": 950,
242
- "train_loss": 3.3967,
243
- "lr": 0.002819,
244
- "tokens_per_second": 919159,
245
- "minutes": 4.92,
246
  "mfu_percent": 22.2,
247
  "gpu_util_percent": 100,
248
- "gpu_memory_gb": 79.1
249
  },
250
  {
251
  "step": 1000,
252
- "train_loss": 3.3874,
253
- "lr": 0.0028,
254
- "tokens_per_second": 918998,
255
- "minutes": 5.16,
256
  "mfu_percent": 22.2,
257
  "gpu_util_percent": 100,
258
- "gpu_memory_gb": 79.1,
259
- "val_loss": 3.4027
260
  },
261
  {
262
  "step": 1050,
263
- "train_loss": 3.3679,
264
- "lr": 0.002781,
265
- "tokens_per_second": 919250,
266
- "minutes": 5.45,
267
  "mfu_percent": 22.2,
268
  "gpu_util_percent": 100,
269
- "gpu_memory_gb": 79.1
270
  },
271
  {
272
  "step": 1100,
273
- "train_loss": 3.335,
274
- "lr": 0.00276,
275
- "tokens_per_second": 919050,
276
- "minutes": 5.68,
277
  "mfu_percent": 22.2,
278
  "gpu_util_percent": 100,
279
- "gpu_memory_gb": 79.1
280
  },
281
  {
282
  "step": 1150,
283
- "train_loss": 3.365,
284
- "lr": 0.002738,
285
- "tokens_per_second": 918695,
286
- "minutes": 5.92,
287
  "mfu_percent": 22.2,
288
  "gpu_util_percent": 100,
289
- "gpu_memory_gb": 79.1
290
  },
291
  {
292
  "step": 1200,
293
- "train_loss": 3.3143,
294
- "lr": 0.002716,
295
- "tokens_per_second": 918833,
296
- "minutes": 6.16,
297
  "mfu_percent": 22.2,
298
  "gpu_util_percent": 100,
299
- "gpu_memory_gb": 79.1
300
  },
301
  {
302
  "step": 1250,
303
- "train_loss": 3.2921,
304
- "lr": 0.002693,
305
- "tokens_per_second": 918924,
306
- "minutes": 6.4,
307
  "mfu_percent": 22.2,
308
  "gpu_util_percent": 100,
309
- "gpu_memory_gb": 79.1
310
  },
311
  {
312
  "step": 1300,
313
- "train_loss": 3.3627,
314
- "lr": 0.002669,
315
- "tokens_per_second": 919045,
316
  "minutes": 6.64,
317
  "mfu_percent": 22.2,
318
  "gpu_util_percent": 100,
319
- "gpu_memory_gb": 79.1
320
  },
321
  {
322
  "step": 1350,
323
- "train_loss": 3.3008,
324
- "lr": 0.002644,
325
- "tokens_per_second": 919010,
326
- "minutes": 6.87,
327
  "mfu_percent": 22.2,
328
  "gpu_util_percent": 100,
329
- "gpu_memory_gb": 79.1
330
  },
331
  {
332
  "step": 1400,
333
- "train_loss": 3.2552,
334
- "lr": 0.002618,
335
- "tokens_per_second": 919072,
336
- "minutes": 7.11,
337
  "mfu_percent": 22.2,
338
  "gpu_util_percent": 100,
339
- "gpu_memory_gb": 79.1
340
  },
341
  {
342
  "step": 1450,
343
- "train_loss": 3.2611,
344
- "lr": 0.002592,
345
- "tokens_per_second": 919238,
346
  "minutes": 7.35,
347
  "mfu_percent": 22.2,
348
  "gpu_util_percent": 100,
349
- "gpu_memory_gb": 79.1
350
  },
351
  {
352
  "step": 1500,
353
- "train_loss": 3.2515,
354
- "lr": 0.002565,
355
- "tokens_per_second": 919356,
356
  "minutes": 7.59,
357
  "mfu_percent": 22.2,
358
  "gpu_util_percent": 100,
359
- "gpu_memory_gb": 79.1,
360
- "val_loss": 3.2835
361
  },
362
  {
363
  "step": 1550,
364
- "train_loss": 3.2331,
365
- "lr": 0.002538,
366
- "tokens_per_second": 919186,
367
  "minutes": 7.87,
368
  "mfu_percent": 22.2,
369
  "gpu_util_percent": 100,
370
- "gpu_memory_gb": 79.1
371
  },
372
  {
373
  "step": 1600,
374
- "train_loss": 3.3018,
375
- "lr": 0.002509,
376
- "tokens_per_second": 919216,
377
  "minutes": 8.11,
378
  "mfu_percent": 22.2,
379
  "gpu_util_percent": 100,
380
- "gpu_memory_gb": 79.1
381
  },
382
  {
383
  "step": 1650,
384
- "train_loss": 3.2305,
385
- "lr": 0.00248,
386
- "tokens_per_second": 918761,
387
  "minutes": 8.35,
388
  "mfu_percent": 22.2,
389
  "gpu_util_percent": 100,
390
- "gpu_memory_gb": 79.1
391
  },
392
  {
393
  "step": 1700,
394
- "train_loss": 3.2899,
395
- "lr": 0.002451,
396
- "tokens_per_second": 918918,
397
- "minutes": 8.58,
398
  "mfu_percent": 22.2,
399
  "gpu_util_percent": 100,
400
- "gpu_memory_gb": 79.1
401
  },
402
  {
403
  "step": 1750,
404
- "train_loss": 3.2422,
405
- "lr": 0.002421,
406
- "tokens_per_second": 918821,
407
  "minutes": 8.82,
408
  "mfu_percent": 22.2,
409
  "gpu_util_percent": 100,
410
- "gpu_memory_gb": 79.1
411
  },
412
  {
413
  "step": 1800,
414
- "train_loss": 3.2045,
415
- "lr": 0.00239,
416
- "tokens_per_second": 918803,
417
  "minutes": 9.06,
418
  "mfu_percent": 22.2,
419
  "gpu_util_percent": 100,
420
- "gpu_memory_gb": 79.1
421
  },
422
  {
423
  "step": 1850,
424
- "train_loss": 3.2057,
425
- "lr": 0.002358,
426
- "tokens_per_second": 918327,
427
  "minutes": 9.3,
428
- "mfu_percent": 22.1,
429
  "gpu_util_percent": 100,
430
- "gpu_memory_gb": 79.1
431
  },
432
  {
433
  "step": 1900,
434
- "train_loss": 3.1794,
435
- "lr": 0.002327,
436
- "tokens_per_second": 918460,
437
- "minutes": 9.54,
438
- "mfu_percent": 22.1,
439
  "gpu_util_percent": 100,
440
- "gpu_memory_gb": 79.1
441
  },
442
  {
443
  "step": 1950,
444
- "train_loss": 3.1661,
445
- "lr": 0.002294,
446
- "tokens_per_second": 918253,
447
  "minutes": 9.77,
448
- "mfu_percent": 22.1,
449
  "gpu_util_percent": 100,
450
- "gpu_memory_gb": 79.1
451
  },
452
  {
453
  "step": 2000,
454
- "train_loss": 3.1771,
455
- "lr": 0.002261,
456
- "tokens_per_second": 918444,
457
  "minutes": 10.01,
458
- "mfu_percent": 22.1,
459
  "gpu_util_percent": 100,
460
- "gpu_memory_gb": 79.1,
461
- "val_loss": 3.2099
462
  },
463
  {
464
  "step": 2050,
465
- "train_loss": 3.1498,
466
- "lr": 0.002228,
467
- "tokens_per_second": 919237,
468
- "minutes": 10.3,
469
  "mfu_percent": 22.2,
470
  "gpu_util_percent": 100,
471
- "gpu_memory_gb": 79.1
472
  },
473
  {
474
  "step": 2100,
475
- "train_loss": 3.164,
476
- "lr": 0.002194,
477
- "tokens_per_second": 920523,
478
  "minutes": 10.53,
479
  "mfu_percent": 22.2,
480
  "gpu_util_percent": 100,
481
- "gpu_memory_gb": 79.1
482
  },
483
  {
484
  "step": 2150,
485
- "train_loss": 3.1412,
486
- "lr": 0.00216,
487
- "tokens_per_second": 921392,
488
  "minutes": 10.77,
489
  "mfu_percent": 22.2,
490
  "gpu_util_percent": 100,
491
- "gpu_memory_gb": 79.1
492
  },
493
  {
494
  "step": 2200,
495
- "train_loss": 3.1425,
496
- "lr": 0.002125,
497
- "tokens_per_second": 921489,
498
- "minutes": 11.01,
499
  "mfu_percent": 22.2,
500
  "gpu_util_percent": 100,
501
- "gpu_memory_gb": 79.1
502
  },
503
  {
504
  "step": 2250,
505
- "train_loss": 3.1489,
506
- "lr": 0.00209,
507
- "tokens_per_second": 921290,
508
  "minutes": 11.24,
509
  "mfu_percent": 22.2,
510
  "gpu_util_percent": 100,
511
- "gpu_memory_gb": 79.1
512
  },
513
  {
514
  "step": 2300,
515
- "train_loss": 3.1575,
516
- "lr": 0.002055,
517
- "tokens_per_second": 919970,
518
  "minutes": 11.48,
519
  "mfu_percent": 22.2,
520
  "gpu_util_percent": 100,
521
- "gpu_memory_gb": 79.1
522
  },
523
  {
524
  "step": 2350,
525
- "train_loss": 3.1278,
526
- "lr": 0.00202,
527
- "tokens_per_second": 919468,
528
- "minutes": 11.72,
529
  "mfu_percent": 22.2,
530
  "gpu_util_percent": 100,
531
- "gpu_memory_gb": 79.1
532
  },
533
  {
534
  "step": 2400,
535
- "train_loss": 3.1516,
536
- "lr": 0.001984,
537
- "tokens_per_second": 919200,
538
- "minutes": 11.96,
539
  "mfu_percent": 22.2,
540
  "gpu_util_percent": 100,
541
- "gpu_memory_gb": 79.1
542
  },
543
  {
544
  "step": 2450,
545
- "train_loss": 3.1655,
546
- "lr": 0.001948,
547
- "tokens_per_second": 918867,
548
- "minutes": 12.2,
549
  "mfu_percent": 22.2,
550
  "gpu_util_percent": 100,
551
- "gpu_memory_gb": 79.1
552
  },
553
  {
554
  "step": 2500,
555
- "train_loss": 3.1927,
556
- "lr": 0.001911,
557
- "tokens_per_second": 918862,
558
- "minutes": 12.43,
559
  "mfu_percent": 22.2,
560
  "gpu_util_percent": 100,
561
- "gpu_memory_gb": 79.1,
562
- "val_loss": 3.1622
563
  },
564
  {
565
  "step": 2550,
566
- "train_loss": 3.1609,
567
- "lr": 0.001875,
568
- "tokens_per_second": 918878,
569
- "minutes": 12.72,
570
  "mfu_percent": 22.2,
571
  "gpu_util_percent": 100,
572
- "gpu_memory_gb": 79.1
573
  },
574
  {
575
  "step": 2600,
576
- "train_loss": 3.1732,
577
- "lr": 0.001838,
578
- "tokens_per_second": 919151,
579
- "minutes": 12.96,
580
  "mfu_percent": 22.2,
581
  "gpu_util_percent": 100,
582
- "gpu_memory_gb": 79.1
583
  },
584
  {
585
  "step": 2650,
586
- "train_loss": 3.0875,
587
- "lr": 0.001801,
588
- "tokens_per_second": 919485,
589
- "minutes": 13.19,
590
  "mfu_percent": 22.2,
591
  "gpu_util_percent": 100,
592
- "gpu_memory_gb": 79.1
593
  },
594
  {
595
  "step": 2700,
596
- "train_loss": 3.1417,
597
- "lr": 0.001764,
598
- "tokens_per_second": 919430,
599
- "minutes": 13.43,
600
  "mfu_percent": 22.2,
601
  "gpu_util_percent": 100,
602
- "gpu_memory_gb": 79.1
603
  },
604
  {
605
  "step": 2750,
606
- "train_loss": 3.1589,
607
- "lr": 0.001727,
608
- "tokens_per_second": 919254,
609
- "minutes": 13.67,
610
  "mfu_percent": 22.2,
611
  "gpu_util_percent": 100,
612
- "gpu_memory_gb": 79.1
613
  },
614
  {
615
  "step": 2800,
616
- "train_loss": 3.052,
617
- "lr": 0.00169,
618
- "tokens_per_second": 919224,
619
- "minutes": 13.91,
620
  "mfu_percent": 22.2,
621
  "gpu_util_percent": 100,
622
- "gpu_memory_gb": 79.1
623
  },
624
  {
625
  "step": 2850,
626
- "train_loss": 3.0945,
627
- "lr": 0.001653,
628
- "tokens_per_second": 919252,
629
- "minutes": 14.14,
630
  "mfu_percent": 22.2,
631
  "gpu_util_percent": 100,
632
- "gpu_memory_gb": 79.1
633
  },
634
  {
635
  "step": 2900,
636
- "train_loss": 3.0436,
637
- "lr": 0.001616,
638
- "tokens_per_second": 919765,
639
- "minutes": 14.38,
640
  "mfu_percent": 22.2,
641
  "gpu_util_percent": 100,
642
- "gpu_memory_gb": 79.1
643
  },
644
  {
645
  "step": 2950,
646
- "train_loss": 3.0525,
647
- "lr": 0.001579,
648
- "tokens_per_second": 919937,
649
- "minutes": 14.62,
650
  "mfu_percent": 22.2,
651
  "gpu_util_percent": 100,
652
- "gpu_memory_gb": 79.1
653
  },
654
  {
655
  "step": 3000,
656
- "train_loss": 3.019,
657
- "lr": 0.001542,
658
- "tokens_per_second": 919762,
659
- "minutes": 14.86,
660
  "mfu_percent": 22.2,
661
  "gpu_util_percent": 100,
662
- "gpu_memory_gb": 79.1,
663
- "val_loss": 3.116
664
  },
665
  {
666
  "step": 3050,
667
- "train_loss": 3.13,
668
- "lr": 0.001505,
669
- "tokens_per_second": 920124,
670
- "minutes": 15.14,
671
  "mfu_percent": 22.2,
672
  "gpu_util_percent": 100,
673
- "gpu_memory_gb": 79.1
674
  },
675
  {
676
  "step": 3100,
677
- "train_loss": 3.0723,
678
- "lr": 0.001468,
679
- "tokens_per_second": 920009,
680
- "minutes": 15.38,
681
  "mfu_percent": 22.2,
682
  "gpu_util_percent": 100,
683
- "gpu_memory_gb": 79.1
684
  },
685
  {
686
  "step": 3150,
687
- "train_loss": 3.0721,
688
- "lr": 0.001431,
689
- "tokens_per_second": 919773,
690
- "minutes": 15.62,
691
  "mfu_percent": 22.2,
692
  "gpu_util_percent": 100,
693
- "gpu_memory_gb": 79.1
694
  },
695
  {
696
  "step": 3200,
697
- "train_loss": 3.082,
698
- "lr": 0.001395,
699
- "tokens_per_second": 919868,
700
- "minutes": 15.85,
701
  "mfu_percent": 22.2,
702
  "gpu_util_percent": 100,
703
- "gpu_memory_gb": 79.1
704
  },
705
  {
706
  "step": 3250,
707
- "train_loss": 3.1011,
708
- "lr": 0.001358,
709
- "tokens_per_second": 920018,
710
- "minutes": 16.09,
711
  "mfu_percent": 22.2,
712
  "gpu_util_percent": 100,
713
- "gpu_memory_gb": 79.1
714
  },
715
  {
716
  "step": 3300,
717
- "train_loss": 3.0397,
718
- "lr": 0.001322,
719
- "tokens_per_second": 919962,
720
- "minutes": 16.33,
721
  "mfu_percent": 22.2,
722
  "gpu_util_percent": 100,
723
- "gpu_memory_gb": 79.1
724
  },
725
  {
726
  "step": 3350,
727
- "train_loss": 3.0473,
728
- "lr": 0.001286,
729
- "tokens_per_second": 919990,
730
- "minutes": 16.57,
731
  "mfu_percent": 22.2,
732
  "gpu_util_percent": 100,
733
- "gpu_memory_gb": 79.1
734
  },
735
  {
736
  "step": 3400,
737
- "train_loss": 3.0615,
738
- "lr": 0.001251,
739
- "tokens_per_second": 923200,
740
- "minutes": 16.8,
741
- "mfu_percent": 22.3,
742
  "gpu_util_percent": 100,
743
- "gpu_memory_gb": 79.1
744
  },
745
  {
746
  "step": 3450,
747
- "train_loss": 3.0738,
748
- "lr": 0.001215,
749
- "tokens_per_second": 926033,
750
- "minutes": 17.04,
751
- "mfu_percent": 22.3,
752
  "gpu_util_percent": 100,
753
- "gpu_memory_gb": 79.1
754
  },
755
  {
756
  "step": 3500,
757
- "train_loss": 2.991,
758
- "lr": 0.00118,
759
- "tokens_per_second": 925099,
760
- "minutes": 17.27,
761
- "mfu_percent": 22.3,
762
  "gpu_util_percent": 100,
763
- "gpu_memory_gb": 79.1,
764
- "val_loss": 3.0811
765
  },
766
  {
767
  "step": 3550,
768
- "train_loss": 3.0385,
769
- "lr": 0.001146,
770
- "tokens_per_second": 924905,
771
- "minutes": 17.56,
772
- "mfu_percent": 22.3,
773
  "gpu_util_percent": 100,
774
- "gpu_memory_gb": 79.1
775
  },
776
  {
777
  "step": 3600,
778
- "train_loss": 3.0725,
779
- "lr": 0.001111,
780
- "tokens_per_second": 924717,
781
- "minutes": 17.79,
782
- "mfu_percent": 22.3,
783
  "gpu_util_percent": 100,
784
- "gpu_memory_gb": 79.1
785
  },
786
  {
787
  "step": 3650,
788
- "train_loss": 3.0169,
789
- "lr": 0.001077,
790
- "tokens_per_second": 925574,
791
- "minutes": 18.03,
792
- "mfu_percent": 22.3,
793
  "gpu_util_percent": 100,
794
- "gpu_memory_gb": 79.1
795
  },
796
  {
797
  "step": 3700,
798
- "train_loss": 3.0391,
799
- "lr": 0.001044,
800
- "tokens_per_second": 926820,
801
- "minutes": 18.27,
802
- "mfu_percent": 22.3,
803
  "gpu_util_percent": 100,
804
- "gpu_memory_gb": 79.1
805
  },
806
  {
807
  "step": 3750,
808
- "train_loss": 3.0948,
809
- "lr": 0.001011,
810
- "tokens_per_second": 926733,
811
- "minutes": 18.5,
812
- "mfu_percent": 22.3,
813
  "gpu_util_percent": 100,
814
- "gpu_memory_gb": 79.1
815
  },
816
  {
817
  "step": 3800,
818
- "train_loss": 3.0155,
819
- "lr": 0.000979,
820
- "tokens_per_second": 926219,
821
- "minutes": 18.74,
822
- "mfu_percent": 22.3,
823
  "gpu_util_percent": 100,
824
- "gpu_memory_gb": 79.1
825
  },
826
  {
827
  "step": 3850,
828
- "train_loss": 3.0926,
829
- "lr": 0.000947,
830
- "tokens_per_second": 924590,
831
  "minutes": 18.97,
832
- "mfu_percent": 22.3,
833
  "gpu_util_percent": 100,
834
- "gpu_memory_gb": 79.1
835
  },
836
  {
837
  "step": 3900,
838
- "train_loss": 3.0346,
839
- "lr": 0.000915,
840
- "tokens_per_second": 924607,
841
  "minutes": 19.21,
842
- "mfu_percent": 22.3,
843
  "gpu_util_percent": 100,
844
- "gpu_memory_gb": 79.1
845
  },
846
  {
847
  "step": 3950,
848
- "train_loss": 3.0385,
849
- "lr": 0.000884,
850
- "tokens_per_second": 923474,
851
- "minutes": 19.45,
852
- "mfu_percent": 22.3,
853
  "gpu_util_percent": 100,
854
- "gpu_memory_gb": 79.1
855
  },
856
  {
857
  "step": 4000,
858
- "train_loss": 2.9716,
859
- "lr": 0.000854,
860
- "tokens_per_second": 922129,
861
  "minutes": 19.68,
862
  "mfu_percent": 22.2,
863
  "gpu_util_percent": 100,
864
- "gpu_memory_gb": 79.1,
865
- "val_loss": 3.0478
866
  },
867
  {
868
  "step": 4050,
869
- "train_loss": 2.9443,
870
- "lr": 0.000824,
871
- "tokens_per_second": 920762,
872
  "minutes": 19.97,
873
  "mfu_percent": 22.2,
874
  "gpu_util_percent": 100,
875
- "gpu_memory_gb": 79.1
876
  },
877
  {
878
  "step": 4100,
879
- "train_loss": 2.9772,
880
- "lr": 0.000795,
881
- "tokens_per_second": 920133,
882
  "minutes": 20.2,
883
  "mfu_percent": 22.2,
884
  "gpu_util_percent": 100,
885
- "gpu_memory_gb": 79.1
886
  },
887
  {
888
  "step": 4150,
889
- "train_loss": 3.0015,
890
- "lr": 0.000767,
891
- "tokens_per_second": 919900,
892
  "minutes": 20.44,
893
  "mfu_percent": 22.2,
894
  "gpu_util_percent": 100,
895
- "gpu_memory_gb": 79.1
896
  },
897
  {
898
  "step": 4200,
899
- "train_loss": 3.0204,
900
- "lr": 0.000739,
901
- "tokens_per_second": 920172,
902
  "minutes": 20.68,
903
  "mfu_percent": 22.2,
904
  "gpu_util_percent": 100,
905
- "gpu_memory_gb": 79.1
906
  },
907
  {
908
  "step": 4250,
909
- "train_loss": 3.0105,
910
- "lr": 0.000712,
911
- "tokens_per_second": 923826,
912
  "minutes": 20.92,
913
- "mfu_percent": 22.3,
914
  "gpu_util_percent": 100,
915
- "gpu_memory_gb": 79.1
916
  },
917
  {
918
  "step": 4300,
919
- "train_loss": 3.0586,
920
- "lr": 0.000686,
921
- "tokens_per_second": 926224,
922
  "minutes": 21.15,
923
- "mfu_percent": 22.3,
924
  "gpu_util_percent": 100,
925
- "gpu_memory_gb": 79.1
926
  },
927
  {
928
  "step": 4350,
929
- "train_loss": 3.0093,
930
- "lr": 0.00066,
931
- "tokens_per_second": 925730,
932
  "minutes": 21.39,
933
- "mfu_percent": 22.3,
934
  "gpu_util_percent": 100,
935
- "gpu_memory_gb": 79.1
936
  },
937
  {
938
  "step": 4400,
939
- "train_loss": 3.0369,
940
- "lr": 0.000635,
941
- "tokens_per_second": 926216,
942
- "minutes": 21.62,
943
- "mfu_percent": 22.3,
944
  "gpu_util_percent": 100,
945
- "gpu_memory_gb": 79.1
946
  },
947
  {
948
  "step": 4450,
949
- "train_loss": 2.9631,
950
- "lr": 0.000611,
951
- "tokens_per_second": 926599,
952
  "minutes": 21.86,
953
- "mfu_percent": 22.3,
954
  "gpu_util_percent": 100,
955
- "gpu_memory_gb": 79.1
956
  },
957
  {
958
  "step": 4500,
959
- "train_loss": 3.0019,
960
- "lr": 0.000588,
961
- "tokens_per_second": 925142,
962
  "minutes": 22.1,
963
- "mfu_percent": 22.3,
964
  "gpu_util_percent": 100,
965
- "gpu_memory_gb": 79.1,
966
- "val_loss": 3.0207
967
  },
968
  {
969
  "step": 4550,
970
- "train_loss": 2.963,
971
- "lr": 0.000565,
972
- "tokens_per_second": 923970,
973
- "minutes": 22.38,
974
- "mfu_percent": 22.3,
975
  "gpu_util_percent": 100,
976
- "gpu_memory_gb": 79.1
977
  },
978
  {
979
  "step": 4600,
980
- "train_loss": 3.0014,
981
- "lr": 0.000544,
982
- "tokens_per_second": 924314,
983
  "minutes": 22.62,
984
- "mfu_percent": 22.3,
985
  "gpu_util_percent": 100,
986
- "gpu_memory_gb": 79.1
987
  },
988
  {
989
  "step": 4650,
990
- "train_loss": 3.0158,
991
- "lr": 0.000523,
992
- "tokens_per_second": 926032,
993
- "minutes": 22.85,
994
- "mfu_percent": 22.3,
995
  "gpu_util_percent": 100,
996
- "gpu_memory_gb": 79.1
997
  },
998
  {
999
  "step": 4700,
1000
- "train_loss": 2.9781,
1001
- "lr": 0.000503,
1002
- "tokens_per_second": 926272,
1003
- "minutes": 23.09,
1004
- "mfu_percent": 22.3,
1005
  "gpu_util_percent": 100,
1006
- "gpu_memory_gb": 79.1
1007
  },
1008
  {
1009
  "step": 4750,
1010
- "train_loss": 3.0033,
1011
- "lr": 0.000484,
1012
- "tokens_per_second": 926281,
1013
- "minutes": 23.32,
1014
- "mfu_percent": 22.3,
1015
  "gpu_util_percent": 100,
1016
- "gpu_memory_gb": 79.1
1017
  },
1018
  {
1019
  "step": 4800,
1020
- "train_loss": 2.9712,
1021
- "lr": 0.000465,
1022
- "tokens_per_second": 925900,
1023
- "minutes": 23.56,
1024
- "mfu_percent": 22.3,
1025
  "gpu_util_percent": 100,
1026
- "gpu_memory_gb": 79.1
1027
  },
1028
  {
1029
  "step": 4850,
1030
- "train_loss": 2.9531,
1031
- "lr": 0.000448,
1032
- "tokens_per_second": 925757,
1033
- "minutes": 23.79,
1034
- "mfu_percent": 22.3,
1035
  "gpu_util_percent": 100,
1036
- "gpu_memory_gb": 79.1
1037
  },
1038
  {
1039
  "step": 4900,
1040
- "train_loss": 2.995,
1041
- "lr": 0.000432,
1042
- "tokens_per_second": 925796,
1043
- "minutes": 24.03,
1044
- "mfu_percent": 22.3,
1045
  "gpu_util_percent": 100,
1046
- "gpu_memory_gb": 79.1
1047
  },
1048
  {
1049
  "step": 4950,
1050
- "train_loss": 2.9912,
1051
- "lr": 0.000416,
1052
- "tokens_per_second": 925597,
1053
- "minutes": 24.27,
1054
- "mfu_percent": 22.3,
1055
  "gpu_util_percent": 100,
1056
- "gpu_memory_gb": 79.1
1057
  },
1058
  {
1059
  "step": 5000,
1060
- "train_loss": 2.9481,
1061
- "lr": 0.000401,
1062
- "tokens_per_second": 925036,
1063
- "minutes": 24.5,
1064
- "mfu_percent": 22.3,
1065
  "gpu_util_percent": 100,
1066
- "gpu_memory_gb": 79.1,
1067
- "val_loss": 3.0002
1068
  },
1069
  {
1070
  "step": 5050,
1071
- "train_loss": 3.0146,
1072
- "lr": 0.000388,
1073
- "tokens_per_second": 924886,
1074
- "minutes": 24.79,
1075
  "mfu_percent": 22.3,
1076
  "gpu_util_percent": 100,
1077
- "gpu_memory_gb": 79.1
1078
  },
1079
  {
1080
  "step": 5100,
1081
- "train_loss": 2.9885,
1082
- "lr": 0.000375,
1083
- "tokens_per_second": 924816,
1084
- "minutes": 25.02,
1085
  "mfu_percent": 22.3,
1086
  "gpu_util_percent": 100,
1087
- "gpu_memory_gb": 79.1
1088
  },
1089
  {
1090
  "step": 5150,
1091
- "train_loss": 3.0184,
1092
- "lr": 0.000363,
1093
- "tokens_per_second": 924852,
1094
- "minutes": 25.26,
1095
- "mfu_percent": 22.3,
1096
  "gpu_util_percent": 100,
1097
- "gpu_memory_gb": 79.1
1098
  },
1099
  {
1100
  "step": 5200,
1101
- "train_loss": 2.9669,
1102
- "lr": 0.000353,
1103
- "tokens_per_second": 925481,
1104
- "minutes": 25.5,
1105
- "mfu_percent": 22.3,
1106
  "gpu_util_percent": 100,
1107
- "gpu_memory_gb": 79.1
1108
  },
1109
  {
1110
  "step": 5250,
1111
- "train_loss": 2.9598,
1112
- "lr": 0.000343,
1113
- "tokens_per_second": 926674,
1114
- "minutes": 25.73,
1115
- "mfu_percent": 22.3,
1116
  "gpu_util_percent": 100,
1117
- "gpu_memory_gb": 79.1
1118
  },
1119
  {
1120
  "step": 5300,
1121
- "train_loss": 2.9644,
1122
- "lr": 0.000334,
1123
- "tokens_per_second": 926065,
1124
- "minutes": 25.97,
1125
- "mfu_percent": 22.3,
1126
  "gpu_util_percent": 100,
1127
- "gpu_memory_gb": 79.1
1128
  },
1129
  {
1130
  "step": 5350,
1131
- "train_loss": 2.9615,
1132
- "lr": 0.000326,
1133
- "tokens_per_second": 926198,
1134
- "minutes": 26.2,
1135
- "mfu_percent": 22.3,
1136
  "gpu_util_percent": 100,
1137
- "gpu_memory_gb": 79.1
1138
  },
1139
  {
1140
  "step": 5400,
1141
- "train_loss": 2.9874,
1142
- "lr": 0.000319,
1143
- "tokens_per_second": 926366,
1144
- "minutes": 26.44,
1145
- "mfu_percent": 22.3,
1146
  "gpu_util_percent": 100,
1147
- "gpu_memory_gb": 79.1
1148
  },
1149
  {
1150
  "step": 5450,
1151
- "train_loss": 2.9591,
1152
- "lr": 0.000314,
1153
- "tokens_per_second": 926756,
1154
- "minutes": 26.67,
1155
- "mfu_percent": 22.3,
1156
  "gpu_util_percent": 100,
1157
- "gpu_memory_gb": 79.1
1158
  },
1159
  {
1160
  "step": 5500,
1161
- "train_loss": 2.9528,
1162
- "lr": 0.000309,
1163
- "tokens_per_second": 925140,
1164
- "minutes": 26.91,
1165
- "mfu_percent": 22.3,
1166
  "gpu_util_percent": 100,
1167
- "gpu_memory_gb": 79.1,
1168
- "val_loss": 2.9883
1169
  },
1170
  {
1171
  "step": 5550,
1172
- "train_loss": 3.0098,
1173
- "lr": 0.000305,
1174
- "tokens_per_second": 925097,
1175
- "minutes": 27.19,
1176
- "mfu_percent": 22.3,
1177
  "gpu_util_percent": 100,
1178
- "gpu_memory_gb": 79.1
1179
  },
1180
  {
1181
  "step": 5600,
1182
- "train_loss": 3.0141,
1183
- "lr": 0.000302,
1184
- "tokens_per_second": 925982,
1185
- "minutes": 27.43,
1186
- "mfu_percent": 22.3,
1187
  "gpu_util_percent": 100,
1188
- "gpu_memory_gb": 79.1
1189
  },
1190
  {
1191
  "step": 5650,
1192
- "train_loss": 2.9269,
1193
- "lr": 0.000301,
1194
- "tokens_per_second": 927048,
1195
- "minutes": 27.67,
1196
- "mfu_percent": 22.4,
1197
  "gpu_util_percent": 100,
1198
- "gpu_memory_gb": 79.1
1199
  },
1200
  {
1201
  "step": 5700,
1202
- "train_loss": 2.9439,
1203
- "lr": 0.0003,
1204
- "tokens_per_second": 926122,
1205
- "minutes": 27.9,
1206
- "mfu_percent": 22.3,
1207
  "gpu_util_percent": 100,
1208
- "gpu_memory_gb": 79.1
1209
  },
1210
  {
1211
- "step": 5709,
1212
- "train_loss": 2.969,
1213
- "lr": 0.0003,
1214
- "tokens_per_second": 925339,
1215
- "minutes": 27.94,
1216
- "mfu_percent": 22.3,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1217
  "gpu_util_percent": 100,
1218
- "gpu_memory_gb": 79.1,
1219
- "val_loss": 2.9852
1220
  }
1221
  ],
1222
  "exports": {
 
 
1223
  "float": {
1224
  "megabytes": 60.81,
1225
- "same_top_token": true,
1226
- "cached_step_ms_p50_cpu": 2.882
 
1227
  },
1228
  "int8": {
1229
- "megabytes": 23.33,
1230
- "same_top_token": false,
1231
- "cached_step_ms_p50_cpu": 2.125
 
1232
  }
1233
  }
1234
  }
 
6
  "heads": 5,
7
  "ffn": 864,
8
  "seq": 512,
9
+ "tokens": 3000000000.0,
10
  "val_tokens": 4000000.0,
11
  "epochs": 1.0,
12
  "batch": 512,
13
  "lr": 0.003,
14
  "fim_rate": 0.5,
15
  "mix": "fineweb=0.6,smoltalk=0.3,enron=0.1",
16
+ "fineweb_files": 4,
17
  "smoltalk_files": 9,
18
  "tokenizer_docs": 200000,
19
  "eval_pauses": 1000,
20
  "log_every": 50,
21
  "repo": "cfahlgren1/tinynext",
22
+ "folder": "v2/experiments/tinynext-v2-3b",
23
+ "eval_folder": "v2",
24
+ "name": "tinynext-v2-3b",
25
  "trackio_space": "cfahlgren1/tinynext-trackio",
26
  "stage": "train",
27
  "data_repo": "cfahlgren1/tinynext-data",
 
31
  "parameters": 12539200,
32
  "data": {
33
  "fineweb": {
34
+ "docs": 2122073,
35
+ "train_tokens": 1800000000
36
  },
37
  "smoltalk": {
38
+ "docs": 1041801,
39
+ "train_tokens": 900000000
40
  },
41
  "enron": {
42
  "docs": 268120,
43
+ "train_tokens": 300000000
44
  }
45
  },
46
  "data_minutes": 0.2,
47
+ "train_minutes": 55.8,
48
  "history": [
49
  {
50
  "step": 0,
51
+ "train_loss": 9.0702,
52
+ "lr": 1.3e-05,
53
+ "tokens_per_second": 12800,
54
+ "minutes": 0.34,
55
  "mfu_percent": 0.3,
56
  "gpu_util_percent": 98,
57
+ "gpu_memory_gb": 76.5,
58
+ "val_loss": 9.0663
59
  },
60
  {
61
  "step": 50,
62
+ "train_loss": 7.215,
63
+ "lr": 0.000671,
64
+ "tokens_per_second": 930958,
65
+ "minutes": 0.62,
66
+ "mfu_percent": 22.4,
67
  "gpu_util_percent": 100,
68
+ "gpu_memory_gb": 85.1
69
  },
70
  {
71
  "step": 100,
72
+ "train_loss": 6.1306,
73
+ "lr": 0.001329,
74
+ "tokens_per_second": 925320,
75
+ "minutes": 0.86,
76
  "mfu_percent": 22.3,
77
  "gpu_util_percent": 100,
78
+ "gpu_memory_gb": 85.1
79
  },
80
  {
81
  "step": 150,
82
+ "train_loss": 5.5618,
83
+ "lr": 0.001986,
84
+ "tokens_per_second": 924266,
85
+ "minutes": 1.1,
86
  "mfu_percent": 22.3,
87
  "gpu_util_percent": 100,
88
+ "gpu_memory_gb": 85.1
89
  },
90
  {
91
  "step": 200,
92
+ "train_loss": 5.1433,
93
+ "lr": 0.002643,
94
+ "tokens_per_second": 923447,
95
+ "minutes": 1.33,
96
+ "mfu_percent": 22.3,
97
  "gpu_util_percent": 100,
98
+ "gpu_memory_gb": 85.1
99
  },
100
  {
101
  "step": 250,
102
+ "train_loss": 4.6259,
103
+ "lr": 0.002997,
104
+ "tokens_per_second": 923503,
105
+ "minutes": 1.57,
106
  "mfu_percent": 22.3,
107
  "gpu_util_percent": 100,
108
+ "gpu_memory_gb": 85.1
109
  },
110
  {
111
  "step": 300,
112
+ "train_loss": 4.2702,
113
+ "lr": 0.002995,
114
+ "tokens_per_second": 923163,
115
+ "minutes": 1.81,
116
  "mfu_percent": 22.3,
117
  "gpu_util_percent": 100,
118
+ "gpu_memory_gb": 85.1
119
  },
120
  {
121
  "step": 350,
122
+ "train_loss": 4.0271,
123
+ "lr": 0.002994,
124
+ "tokens_per_second": 922754,
125
+ "minutes": 2.04,
126
+ "mfu_percent": 22.3,
127
  "gpu_util_percent": 100,
128
+ "gpu_memory_gb": 85.1
129
  },
130
  {
131
  "step": 400,
132
+ "train_loss": 3.8697,
133
+ "lr": 0.002992,
134
+ "tokens_per_second": 922255,
135
+ "minutes": 2.28,
136
+ "mfu_percent": 22.2,
137
  "gpu_util_percent": 100,
138
+ "gpu_memory_gb": 85.1
139
  },
140
  {
141
  "step": 450,
142
+ "train_loss": 3.6918,
143
+ "lr": 0.00299,
144
+ "tokens_per_second": 921900,
145
+ "minutes": 2.52,
146
+ "mfu_percent": 22.2,
147
  "gpu_util_percent": 100,
148
+ "gpu_memory_gb": 85.1
149
  },
150
  {
151
  "step": 500,
152
+ "train_loss": 3.6502,
153
+ "lr": 0.002987,
154
+ "tokens_per_second": 921603,
155
+ "minutes": 2.75,
156
+ "mfu_percent": 22.2,
157
  "gpu_util_percent": 100,
158
+ "gpu_memory_gb": 85.1,
159
+ "val_loss": 3.6813
160
  },
161
  {
162
  "step": 550,
163
+ "train_loss": 3.6069,
164
+ "lr": 0.002985,
165
+ "tokens_per_second": 922139,
166
+ "minutes": 3.04,
167
+ "mfu_percent": 22.2,
168
  "gpu_util_percent": 100,
169
+ "gpu_memory_gb": 85.1
170
  },
171
  {
172
  "step": 600,
173
+ "train_loss": 3.6009,
174
+ "lr": 0.002982,
175
+ "tokens_per_second": 921841,
176
+ "minutes": 3.28,
177
  "mfu_percent": 22.2,
178
  "gpu_util_percent": 100,
179
+ "gpu_memory_gb": 85.1
180
  },
181
  {
182
  "step": 650,
183
+ "train_loss": 3.5049,
184
+ "lr": 0.002978,
185
+ "tokens_per_second": 921515,
186
+ "minutes": 3.51,
187
  "mfu_percent": 22.2,
188
  "gpu_util_percent": 100,
189
+ "gpu_memory_gb": 85.1
190
  },
191
  {
192
  "step": 700,
193
+ "train_loss": 3.4728,
194
+ "lr": 0.002975,
195
+ "tokens_per_second": 921918,
196
+ "minutes": 3.75,
197
  "mfu_percent": 22.2,
198
  "gpu_util_percent": 100,
199
+ "gpu_memory_gb": 85.1
200
  },
201
  {
202
  "step": 750,
203
+ "train_loss": 3.4877,
204
+ "lr": 0.002971,
205
+ "tokens_per_second": 921386,
206
+ "minutes": 3.99,
207
  "mfu_percent": 22.2,
208
  "gpu_util_percent": 100,
209
+ "gpu_memory_gb": 85.1
210
  },
211
  {
212
  "step": 800,
213
+ "train_loss": 3.4264,
214
+ "lr": 0.002967,
215
+ "tokens_per_second": 921002,
216
+ "minutes": 4.22,
217
  "mfu_percent": 22.2,
218
  "gpu_util_percent": 100,
219
+ "gpu_memory_gb": 85.1
220
  },
221
  {
222
  "step": 850,
223
+ "train_loss": 3.3612,
224
+ "lr": 0.002963,
225
+ "tokens_per_second": 920889,
226
+ "minutes": 4.46,
227
  "mfu_percent": 22.2,
228
  "gpu_util_percent": 100,
229
+ "gpu_memory_gb": 85.1
230
  },
231
  {
232
  "step": 900,
233
+ "train_loss": 3.3516,
234
+ "lr": 0.002959,
235
+ "tokens_per_second": 920832,
236
+ "minutes": 4.7,
237
  "mfu_percent": 22.2,
238
  "gpu_util_percent": 100,
239
+ "gpu_memory_gb": 85.1
240
  },
241
  {
242
  "step": 950,
243
+ "train_loss": 3.3742,
244
+ "lr": 0.002954,
245
+ "tokens_per_second": 920862,
246
+ "minutes": 4.94,
247
  "mfu_percent": 22.2,
248
  "gpu_util_percent": 100,
249
+ "gpu_memory_gb": 85.1
250
  },
251
  {
252
  "step": 1000,
253
+ "train_loss": 3.3632,
254
+ "lr": 0.002949,
255
+ "tokens_per_second": 920554,
256
+ "minutes": 5.17,
257
  "mfu_percent": 22.2,
258
  "gpu_util_percent": 100,
259
+ "gpu_memory_gb": 85.1,
260
+ "val_loss": 3.3796
261
  },
262
  {
263
  "step": 1050,
264
+ "train_loss": 3.3061,
265
+ "lr": 0.002944,
266
+ "tokens_per_second": 920467,
267
+ "minutes": 5.46,
268
  "mfu_percent": 22.2,
269
  "gpu_util_percent": 100,
270
+ "gpu_memory_gb": 85.1
271
  },
272
  {
273
  "step": 1100,
274
+ "train_loss": 3.3351,
275
+ "lr": 0.002939,
276
+ "tokens_per_second": 921248,
277
+ "minutes": 5.69,
278
  "mfu_percent": 22.2,
279
  "gpu_util_percent": 100,
280
+ "gpu_memory_gb": 85.1
281
  },
282
  {
283
  "step": 1150,
284
+ "train_loss": 3.3238,
285
+ "lr": 0.002933,
286
+ "tokens_per_second": 921674,
287
+ "minutes": 5.93,
288
  "mfu_percent": 22.2,
289
  "gpu_util_percent": 100,
290
+ "gpu_memory_gb": 85.1
291
  },
292
  {
293
  "step": 1200,
294
+ "train_loss": 3.3208,
295
+ "lr": 0.002927,
296
+ "tokens_per_second": 921693,
297
+ "minutes": 6.17,
298
  "mfu_percent": 22.2,
299
  "gpu_util_percent": 100,
300
+ "gpu_memory_gb": 85.1
301
  },
302
  {
303
  "step": 1250,
304
+ "train_loss": 3.286,
305
+ "lr": 0.002921,
306
+ "tokens_per_second": 921893,
307
+ "minutes": 6.41,
308
  "mfu_percent": 22.2,
309
  "gpu_util_percent": 100,
310
+ "gpu_memory_gb": 85.1
311
  },
312
  {
313
  "step": 1300,
314
+ "train_loss": 3.2941,
315
+ "lr": 0.002914,
316
+ "tokens_per_second": 922496,
317
  "minutes": 6.64,
318
  "mfu_percent": 22.2,
319
  "gpu_util_percent": 100,
320
+ "gpu_memory_gb": 85.1
321
  },
322
  {
323
  "step": 1350,
324
+ "train_loss": 3.2475,
325
+ "lr": 0.002908,
326
+ "tokens_per_second": 922535,
327
+ "minutes": 6.88,
328
  "mfu_percent": 22.2,
329
  "gpu_util_percent": 100,
330
+ "gpu_memory_gb": 85.1
331
  },
332
  {
333
  "step": 1400,
334
+ "train_loss": 3.21,
335
+ "lr": 0.002901,
336
+ "tokens_per_second": 921917,
337
+ "minutes": 7.12,
338
  "mfu_percent": 22.2,
339
  "gpu_util_percent": 100,
340
+ "gpu_memory_gb": 85.1
341
  },
342
  {
343
  "step": 1450,
344
+ "train_loss": 3.2507,
345
+ "lr": 0.002894,
346
+ "tokens_per_second": 921722,
347
  "minutes": 7.35,
348
  "mfu_percent": 22.2,
349
  "gpu_util_percent": 100,
350
+ "gpu_memory_gb": 85.1
351
  },
352
  {
353
  "step": 1500,
354
+ "train_loss": 3.2548,
355
+ "lr": 0.002887,
356
+ "tokens_per_second": 920926,
357
  "minutes": 7.59,
358
  "mfu_percent": 22.2,
359
  "gpu_util_percent": 100,
360
+ "gpu_memory_gb": 85.1,
361
+ "val_loss": 3.2689
362
  },
363
  {
364
  "step": 1550,
365
+ "train_loss": 3.2187,
366
+ "lr": 0.002879,
367
+ "tokens_per_second": 920076,
368
  "minutes": 7.87,
369
  "mfu_percent": 22.2,
370
  "gpu_util_percent": 100,
371
+ "gpu_memory_gb": 85.1
372
  },
373
  {
374
  "step": 1600,
375
+ "train_loss": 3.2435,
376
+ "lr": 0.002871,
377
+ "tokens_per_second": 920390,
378
  "minutes": 8.11,
379
  "mfu_percent": 22.2,
380
  "gpu_util_percent": 100,
381
+ "gpu_memory_gb": 85.1
382
  },
383
  {
384
  "step": 1650,
385
+ "train_loss": 3.1958,
386
+ "lr": 0.002863,
387
+ "tokens_per_second": 920901,
388
  "minutes": 8.35,
389
  "mfu_percent": 22.2,
390
  "gpu_util_percent": 100,
391
+ "gpu_memory_gb": 85.1
392
  },
393
  {
394
  "step": 1700,
395
+ "train_loss": 3.1797,
396
+ "lr": 0.002855,
397
+ "tokens_per_second": 920930,
398
+ "minutes": 8.59,
399
  "mfu_percent": 22.2,
400
  "gpu_util_percent": 100,
401
+ "gpu_memory_gb": 85.1
402
  },
403
  {
404
  "step": 1750,
405
+ "train_loss": 3.2016,
406
+ "lr": 0.002846,
407
+ "tokens_per_second": 921134,
408
  "minutes": 8.82,
409
  "mfu_percent": 22.2,
410
  "gpu_util_percent": 100,
411
+ "gpu_memory_gb": 85.1
412
  },
413
  {
414
  "step": 1800,
415
+ "train_loss": 3.1736,
416
+ "lr": 0.002838,
417
+ "tokens_per_second": 922208,
418
  "minutes": 9.06,
419
  "mfu_percent": 22.2,
420
  "gpu_util_percent": 100,
421
+ "gpu_memory_gb": 85.1
422
  },
423
  {
424
  "step": 1850,
425
+ "train_loss": 3.1912,
426
+ "lr": 0.002829,
427
+ "tokens_per_second": 922453,
428
  "minutes": 9.3,
429
+ "mfu_percent": 22.2,
430
  "gpu_util_percent": 100,
431
+ "gpu_memory_gb": 85.1
432
  },
433
  {
434
  "step": 1900,
435
+ "train_loss": 3.1827,
436
+ "lr": 0.00282,
437
+ "tokens_per_second": 922660,
438
+ "minutes": 9.53,
439
+ "mfu_percent": 22.2,
440
  "gpu_util_percent": 100,
441
+ "gpu_memory_gb": 85.1
442
  },
443
  {
444
  "step": 1950,
445
+ "train_loss": 3.202,
446
+ "lr": 0.00281,
447
+ "tokens_per_second": 922623,
448
  "minutes": 9.77,
449
+ "mfu_percent": 22.2,
450
  "gpu_util_percent": 100,
451
+ "gpu_memory_gb": 85.1
452
  },
453
  {
454
  "step": 2000,
455
+ "train_loss": 3.2288,
456
+ "lr": 0.002801,
457
+ "tokens_per_second": 922474,
458
  "minutes": 10.01,
459
+ "mfu_percent": 22.2,
460
  "gpu_util_percent": 100,
461
+ "gpu_memory_gb": 85.1,
462
+ "val_loss": 3.208
463
  },
464
  {
465
  "step": 2050,
466
+ "train_loss": 3.1591,
467
+ "lr": 0.002791,
468
+ "tokens_per_second": 922197,
469
+ "minutes": 10.29,
470
  "mfu_percent": 22.2,
471
  "gpu_util_percent": 100,
472
+ "gpu_memory_gb": 85.1
473
  },
474
  {
475
  "step": 2100,
476
+ "train_loss": 3.2074,
477
+ "lr": 0.002781,
478
+ "tokens_per_second": 922026,
479
  "minutes": 10.53,
480
  "mfu_percent": 22.2,
481
  "gpu_util_percent": 100,
482
+ "gpu_memory_gb": 85.1
483
  },
484
  {
485
  "step": 2150,
486
+ "train_loss": 3.1784,
487
+ "lr": 0.002771,
488
+ "tokens_per_second": 922145,
489
  "minutes": 10.77,
490
  "mfu_percent": 22.2,
491
  "gpu_util_percent": 100,
492
+ "gpu_memory_gb": 85.1
493
  },
494
  {
495
  "step": 2200,
496
+ "train_loss": 3.1416,
497
+ "lr": 0.00276,
498
+ "tokens_per_second": 921690,
499
+ "minutes": 11.0,
500
  "mfu_percent": 22.2,
501
  "gpu_util_percent": 100,
502
+ "gpu_memory_gb": 85.1
503
  },
504
  {
505
  "step": 2250,
506
+ "train_loss": 3.1531,
507
+ "lr": 0.002749,
508
+ "tokens_per_second": 921458,
509
  "minutes": 11.24,
510
  "mfu_percent": 22.2,
511
  "gpu_util_percent": 100,
512
+ "gpu_memory_gb": 85.1
513
  },
514
  {
515
  "step": 2300,
516
+ "train_loss": 3.1148,
517
+ "lr": 0.002738,
518
+ "tokens_per_second": 920727,
519
  "minutes": 11.48,
520
  "mfu_percent": 22.2,
521
  "gpu_util_percent": 100,
522
+ "gpu_memory_gb": 85.1
523
  },
524
  {
525
  "step": 2350,
526
+ "train_loss": 3.1005,
527
+ "lr": 0.002727,
528
+ "tokens_per_second": 921746,
529
+ "minutes": 11.71,
530
  "mfu_percent": 22.2,
531
  "gpu_util_percent": 100,
532
+ "gpu_memory_gb": 85.1
533
  },
534
  {
535
  "step": 2400,
536
+ "train_loss": 3.2112,
537
+ "lr": 0.002716,
538
+ "tokens_per_second": 922555,
539
+ "minutes": 11.95,
540
  "mfu_percent": 22.2,
541
  "gpu_util_percent": 100,
542
+ "gpu_memory_gb": 85.1
543
  },
544
  {
545
  "step": 2450,
546
+ "train_loss": 3.1437,
547
+ "lr": 0.002705,
548
+ "tokens_per_second": 922219,
549
+ "minutes": 12.19,
550
  "mfu_percent": 22.2,
551
  "gpu_util_percent": 100,
552
+ "gpu_memory_gb": 85.1
553
  },
554
  {
555
  "step": 2500,
556
+ "train_loss": 3.114,
557
+ "lr": 0.002693,
558
+ "tokens_per_second": 922404,
559
+ "minutes": 12.42,
560
  "mfu_percent": 22.2,
561
  "gpu_util_percent": 100,
562
+ "gpu_memory_gb": 85.1,
563
+ "val_loss": 3.1664
564
  },
565
  {
566
  "step": 2550,
567
+ "train_loss": 3.1538,
568
+ "lr": 0.002681,
569
+ "tokens_per_second": 922093,
570
+ "minutes": 12.71,
571
  "mfu_percent": 22.2,
572
  "gpu_util_percent": 100,
573
+ "gpu_memory_gb": 85.1
574
  },
575
  {
576
  "step": 2600,
577
+ "train_loss": 3.1874,
578
+ "lr": 0.002669,
579
+ "tokens_per_second": 921696,
580
+ "minutes": 12.95,
581
  "mfu_percent": 22.2,
582
  "gpu_util_percent": 100,
583
+ "gpu_memory_gb": 85.1
584
  },
585
  {
586
  "step": 2650,
587
+ "train_loss": 3.1451,
588
+ "lr": 0.002657,
589
+ "tokens_per_second": 920726,
590
+ "minutes": 13.18,
591
  "mfu_percent": 22.2,
592
  "gpu_util_percent": 100,
593
+ "gpu_memory_gb": 85.1
594
  },
595
  {
596
  "step": 2700,
597
+ "train_loss": 3.1138,
598
+ "lr": 0.002644,
599
+ "tokens_per_second": 920915,
600
+ "minutes": 13.42,
601
  "mfu_percent": 22.2,
602
  "gpu_util_percent": 100,
603
+ "gpu_memory_gb": 85.1
604
  },
605
  {
606
  "step": 2750,
607
+ "train_loss": 3.1194,
608
+ "lr": 0.002632,
609
+ "tokens_per_second": 920910,
610
+ "minutes": 13.66,
611
  "mfu_percent": 22.2,
612
  "gpu_util_percent": 100,
613
+ "gpu_memory_gb": 85.1
614
  },
615
  {
616
  "step": 2800,
617
+ "train_loss": 3.0835,
618
+ "lr": 0.002619,
619
+ "tokens_per_second": 920819,
620
+ "minutes": 13.89,
621
  "mfu_percent": 22.2,
622
  "gpu_util_percent": 100,
623
+ "gpu_memory_gb": 85.1
624
  },
625
  {
626
  "step": 2850,
627
+ "train_loss": 3.1095,
628
+ "lr": 0.002606,
629
+ "tokens_per_second": 920420,
630
+ "minutes": 14.13,
631
  "mfu_percent": 22.2,
632
  "gpu_util_percent": 100,
633
+ "gpu_memory_gb": 85.1
634
  },
635
  {
636
  "step": 2900,
637
+ "train_loss": 3.1067,
638
+ "lr": 0.002593,
639
+ "tokens_per_second": 920261,
640
+ "minutes": 14.37,
641
  "mfu_percent": 22.2,
642
  "gpu_util_percent": 100,
643
+ "gpu_memory_gb": 85.1
644
  },
645
  {
646
  "step": 2950,
647
+ "train_loss": 3.0488,
648
+ "lr": 0.002579,
649
+ "tokens_per_second": 921100,
650
+ "minutes": 14.61,
651
  "mfu_percent": 22.2,
652
  "gpu_util_percent": 100,
653
+ "gpu_memory_gb": 85.1
654
  },
655
  {
656
  "step": 3000,
657
+ "train_loss": 3.1089,
658
+ "lr": 0.002566,
659
+ "tokens_per_second": 921915,
660
+ "minutes": 14.84,
661
  "mfu_percent": 22.2,
662
  "gpu_util_percent": 100,
663
+ "gpu_memory_gb": 85.1,
664
+ "val_loss": 3.1334
665
  },
666
  {
667
  "step": 3050,
668
+ "train_loss": 3.0736,
669
+ "lr": 0.002552,
670
+ "tokens_per_second": 922085,
671
+ "minutes": 15.13,
672
  "mfu_percent": 22.2,
673
  "gpu_util_percent": 100,
674
+ "gpu_memory_gb": 85.1
675
  },
676
  {
677
  "step": 3100,
678
+ "train_loss": 3.0915,
679
+ "lr": 0.002538,
680
+ "tokens_per_second": 921538,
681
+ "minutes": 15.36,
682
  "mfu_percent": 22.2,
683
  "gpu_util_percent": 100,
684
+ "gpu_memory_gb": 85.1
685
  },
686
  {
687
  "step": 3150,
688
+ "train_loss": 3.0751,
689
+ "lr": 0.002524,
690
+ "tokens_per_second": 921498,
691
+ "minutes": 15.6,
692
  "mfu_percent": 22.2,
693
  "gpu_util_percent": 100,
694
+ "gpu_memory_gb": 85.1
695
  },
696
  {
697
  "step": 3200,
698
+ "train_loss": 3.1093,
699
+ "lr": 0.00251,
700
+ "tokens_per_second": 920747,
701
+ "minutes": 15.84,
702
  "mfu_percent": 22.2,
703
  "gpu_util_percent": 100,
704
+ "gpu_memory_gb": 85.1
705
  },
706
  {
707
  "step": 3250,
708
+ "train_loss": 3.1425,
709
+ "lr": 0.002495,
710
+ "tokens_per_second": 920376,
711
+ "minutes": 16.08,
712
  "mfu_percent": 22.2,
713
  "gpu_util_percent": 100,
714
+ "gpu_memory_gb": 85.1
715
  },
716
  {
717
  "step": 3300,
718
+ "train_loss": 3.0762,
719
+ "lr": 0.002481,
720
+ "tokens_per_second": 920083,
721
+ "minutes": 16.31,
722
  "mfu_percent": 22.2,
723
  "gpu_util_percent": 100,
724
+ "gpu_memory_gb": 85.1
725
  },
726
  {
727
  "step": 3350,
728
+ "train_loss": 3.1273,
729
+ "lr": 0.002466,
730
+ "tokens_per_second": 919889,
731
+ "minutes": 16.55,
732
  "mfu_percent": 22.2,
733
  "gpu_util_percent": 100,
734
+ "gpu_memory_gb": 85.1
735
  },
736
  {
737
  "step": 3400,
738
+ "train_loss": 3.1,
739
+ "lr": 0.002451,
740
+ "tokens_per_second": 921166,
741
+ "minutes": 16.79,
742
+ "mfu_percent": 22.2,
743
  "gpu_util_percent": 100,
744
+ "gpu_memory_gb": 85.1
745
  },
746
  {
747
  "step": 3450,
748
+ "train_loss": 3.1428,
749
+ "lr": 0.002436,
750
+ "tokens_per_second": 921093,
751
+ "minutes": 17.02,
752
+ "mfu_percent": 22.2,
753
  "gpu_util_percent": 100,
754
+ "gpu_memory_gb": 85.1
755
  },
756
  {
757
  "step": 3500,
758
+ "train_loss": 3.0703,
759
+ "lr": 0.002421,
760
+ "tokens_per_second": 920938,
761
+ "minutes": 17.26,
762
+ "mfu_percent": 22.2,
763
  "gpu_util_percent": 100,
764
+ "gpu_memory_gb": 85.1,
765
+ "val_loss": 3.1114
766
  },
767
  {
768
  "step": 3550,
769
+ "train_loss": 3.1048,
770
+ "lr": 0.002406,
771
+ "tokens_per_second": 921446,
772
+ "minutes": 17.55,
773
+ "mfu_percent": 22.2,
774
  "gpu_util_percent": 100,
775
+ "gpu_memory_gb": 85.1
776
  },
777
  {
778
  "step": 3600,
779
+ "train_loss": 3.0576,
780
+ "lr": 0.00239,
781
+ "tokens_per_second": 921917,
782
+ "minutes": 17.78,
783
+ "mfu_percent": 22.2,
784
  "gpu_util_percent": 100,
785
+ "gpu_memory_gb": 85.1
786
  },
787
  {
788
  "step": 3650,
789
+ "train_loss": 3.0815,
790
+ "lr": 0.002375,
791
+ "tokens_per_second": 921369,
792
+ "minutes": 18.02,
793
+ "mfu_percent": 22.2,
794
  "gpu_util_percent": 100,
795
+ "gpu_memory_gb": 85.1
796
  },
797
  {
798
  "step": 3700,
799
+ "train_loss": 3.102,
800
+ "lr": 0.002359,
801
+ "tokens_per_second": 921425,
802
+ "minutes": 18.26,
803
+ "mfu_percent": 22.2,
804
  "gpu_util_percent": 100,
805
+ "gpu_memory_gb": 85.1
806
  },
807
  {
808
  "step": 3750,
809
+ "train_loss": 3.0362,
810
+ "lr": 0.002343,
811
+ "tokens_per_second": 921350,
812
+ "minutes": 18.49,
813
+ "mfu_percent": 22.2,
814
  "gpu_util_percent": 100,
815
+ "gpu_memory_gb": 85.1
816
  },
817
  {
818
  "step": 3800,
819
+ "train_loss": 3.0936,
820
+ "lr": 0.002327,
821
+ "tokens_per_second": 920994,
822
+ "minutes": 18.73,
823
+ "mfu_percent": 22.2,
824
  "gpu_util_percent": 100,
825
+ "gpu_memory_gb": 85.1
826
  },
827
  {
828
  "step": 3850,
829
+ "train_loss": 3.0723,
830
+ "lr": 0.002311,
831
+ "tokens_per_second": 920745,
832
  "minutes": 18.97,
833
+ "mfu_percent": 22.2,
834
  "gpu_util_percent": 100,
835
+ "gpu_memory_gb": 85.1
836
  },
837
  {
838
  "step": 3900,
839
+ "train_loss": 3.0621,
840
+ "lr": 0.002295,
841
+ "tokens_per_second": 920434,
842
  "minutes": 19.21,
843
+ "mfu_percent": 22.2,
844
  "gpu_util_percent": 100,
845
+ "gpu_memory_gb": 85.1
846
  },
847
  {
848
  "step": 3950,
849
+ "train_loss": 3.1088,
850
+ "lr": 0.002278,
851
+ "tokens_per_second": 919905,
852
+ "minutes": 19.44,
853
+ "mfu_percent": 22.2,
854
  "gpu_util_percent": 100,
855
+ "gpu_memory_gb": 85.1
856
  },
857
  {
858
  "step": 4000,
859
+ "train_loss": 3.019,
860
+ "lr": 0.002262,
861
+ "tokens_per_second": 919709,
862
  "minutes": 19.68,
863
  "mfu_percent": 22.2,
864
  "gpu_util_percent": 100,
865
+ "gpu_memory_gb": 85.1,
866
+ "val_loss": 3.0871
867
  },
868
  {
869
  "step": 4050,
870
+ "train_loss": 3.0537,
871
+ "lr": 0.002245,
872
+ "tokens_per_second": 919817,
873
  "minutes": 19.97,
874
  "mfu_percent": 22.2,
875
  "gpu_util_percent": 100,
876
+ "gpu_memory_gb": 85.1
877
  },
878
  {
879
  "step": 4100,
880
+ "train_loss": 3.0237,
881
+ "lr": 0.002228,
882
+ "tokens_per_second": 919826,
883
  "minutes": 20.2,
884
  "mfu_percent": 22.2,
885
  "gpu_util_percent": 100,
886
+ "gpu_memory_gb": 85.1
887
  },
888
  {
889
  "step": 4150,
890
+ "train_loss": 3.0252,
891
+ "lr": 0.002212,
892
+ "tokens_per_second": 920702,
893
  "minutes": 20.44,
894
  "mfu_percent": 22.2,
895
  "gpu_util_percent": 100,
896
+ "gpu_memory_gb": 85.1
897
  },
898
  {
899
  "step": 4200,
900
+ "train_loss": 2.9956,
901
+ "lr": 0.002195,
902
+ "tokens_per_second": 920999,
903
  "minutes": 20.68,
904
  "mfu_percent": 22.2,
905
  "gpu_util_percent": 100,
906
+ "gpu_memory_gb": 85.1
907
  },
908
  {
909
  "step": 4250,
910
+ "train_loss": 3.0576,
911
+ "lr": 0.002178,
912
+ "tokens_per_second": 920918,
913
  "minutes": 20.92,
914
+ "mfu_percent": 22.2,
915
  "gpu_util_percent": 100,
916
+ "gpu_memory_gb": 85.1
917
  },
918
  {
919
  "step": 4300,
920
+ "train_loss": 3.0944,
921
+ "lr": 0.00216,
922
+ "tokens_per_second": 921070,
923
  "minutes": 21.15,
924
+ "mfu_percent": 22.2,
925
  "gpu_util_percent": 100,
926
+ "gpu_memory_gb": 85.1
927
  },
928
  {
929
  "step": 4350,
930
+ "train_loss": 3.0655,
931
+ "lr": 0.002143,
932
+ "tokens_per_second": 921341,
933
  "minutes": 21.39,
934
+ "mfu_percent": 22.2,
935
  "gpu_util_percent": 100,
936
+ "gpu_memory_gb": 85.1
937
  },
938
  {
939
  "step": 4400,
940
+ "train_loss": 3.0756,
941
+ "lr": 0.002126,
942
+ "tokens_per_second": 920880,
943
+ "minutes": 21.63,
944
+ "mfu_percent": 22.2,
945
  "gpu_util_percent": 100,
946
+ "gpu_memory_gb": 85.1
947
  },
948
  {
949
  "step": 4450,
950
+ "train_loss": 3.0513,
951
+ "lr": 0.002108,
952
+ "tokens_per_second": 921004,
953
  "minutes": 21.86,
954
+ "mfu_percent": 22.2,
955
  "gpu_util_percent": 100,
956
+ "gpu_memory_gb": 85.1
957
  },
958
  {
959
  "step": 4500,
960
+ "train_loss": 3.0553,
961
+ "lr": 0.002091,
962
+ "tokens_per_second": 921256,
963
  "minutes": 22.1,
964
+ "mfu_percent": 22.2,
965
  "gpu_util_percent": 100,
966
+ "gpu_memory_gb": 85.1,
967
+ "val_loss": 3.0698
968
  },
969
  {
970
  "step": 4550,
971
+ "train_loss": 2.967,
972
+ "lr": 0.002073,
973
+ "tokens_per_second": 920961,
974
+ "minutes": 22.39,
975
+ "mfu_percent": 22.2,
976
  "gpu_util_percent": 100,
977
+ "gpu_memory_gb": 85.1
978
  },
979
  {
980
  "step": 4600,
981
+ "train_loss": 3.0567,
982
+ "lr": 0.002056,
983
+ "tokens_per_second": 920043,
984
  "minutes": 22.62,
985
+ "mfu_percent": 22.2,
986
  "gpu_util_percent": 100,
987
+ "gpu_memory_gb": 85.1
988
  },
989
  {
990
  "step": 4650,
991
+ "train_loss": 3.0559,
992
+ "lr": 0.002038,
993
+ "tokens_per_second": 919645,
994
+ "minutes": 22.86,
995
+ "mfu_percent": 22.2,
996
  "gpu_util_percent": 100,
997
+ "gpu_memory_gb": 85.1
998
  },
999
  {
1000
  "step": 4700,
1001
+ "train_loss": 3.0584,
1002
+ "lr": 0.00202,
1003
+ "tokens_per_second": 920036,
1004
+ "minutes": 23.1,
1005
+ "mfu_percent": 22.2,
1006
  "gpu_util_percent": 100,
1007
+ "gpu_memory_gb": 85.1
1008
  },
1009
  {
1010
  "step": 4750,
1011
+ "train_loss": 3.0321,
1012
+ "lr": 0.002002,
1013
+ "tokens_per_second": 920584,
1014
+ "minutes": 23.34,
1015
+ "mfu_percent": 22.2,
1016
  "gpu_util_percent": 100,
1017
+ "gpu_memory_gb": 85.1
1018
  },
1019
  {
1020
  "step": 4800,
1021
+ "train_loss": 2.9678,
1022
+ "lr": 0.001984,
1023
+ "tokens_per_second": 920212,
1024
+ "minutes": 23.57,
1025
+ "mfu_percent": 22.2,
1026
  "gpu_util_percent": 100,
1027
+ "gpu_memory_gb": 85.1
1028
  },
1029
  {
1030
  "step": 4850,
1031
+ "train_loss": 2.9969,
1032
+ "lr": 0.001966,
1033
+ "tokens_per_second": 921129,
1034
+ "minutes": 23.81,
1035
+ "mfu_percent": 22.2,
1036
  "gpu_util_percent": 100,
1037
+ "gpu_memory_gb": 85.1
1038
  },
1039
  {
1040
  "step": 4900,
1041
+ "train_loss": 3.0317,
1042
+ "lr": 0.001948,
1043
+ "tokens_per_second": 921336,
1044
+ "minutes": 24.05,
1045
+ "mfu_percent": 22.2,
1046
  "gpu_util_percent": 100,
1047
+ "gpu_memory_gb": 85.1
1048
  },
1049
  {
1050
  "step": 4950,
1051
+ "train_loss": 3.0314,
1052
+ "lr": 0.00193,
1053
+ "tokens_per_second": 921674,
1054
+ "minutes": 24.28,
1055
+ "mfu_percent": 22.2,
1056
  "gpu_util_percent": 100,
1057
+ "gpu_memory_gb": 85.1
1058
  },
1059
  {
1060
  "step": 5000,
1061
+ "train_loss": 3.0467,
1062
+ "lr": 0.001912,
1063
+ "tokens_per_second": 922491,
1064
+ "minutes": 24.52,
1065
+ "mfu_percent": 22.2,
1066
  "gpu_util_percent": 100,
1067
+ "gpu_memory_gb": 85.1,
1068
+ "val_loss": 3.0507
1069
  },
1070
  {
1071
  "step": 5050,
1072
+ "train_loss": 3.0208,
1073
+ "lr": 0.001894,
1074
+ "tokens_per_second": 922889,
1075
+ "minutes": 24.8,
1076
  "mfu_percent": 22.3,
1077
  "gpu_util_percent": 100,
1078
+ "gpu_memory_gb": 85.1
1079
  },
1080
  {
1081
  "step": 5100,
1082
+ "train_loss": 2.9957,
1083
+ "lr": 0.001875,
1084
+ "tokens_per_second": 922887,
1085
+ "minutes": 25.04,
1086
  "mfu_percent": 22.3,
1087
  "gpu_util_percent": 100,
1088
+ "gpu_memory_gb": 85.1
1089
  },
1090
  {
1091
  "step": 5150,
1092
+ "train_loss": 3.0355,
1093
+ "lr": 0.001857,
1094
+ "tokens_per_second": 922331,
1095
+ "minutes": 25.28,
1096
+ "mfu_percent": 22.2,
1097
  "gpu_util_percent": 100,
1098
+ "gpu_memory_gb": 85.1
1099
  },
1100
  {
1101
  "step": 5200,
1102
+ "train_loss": 3.0702,
1103
+ "lr": 0.001839,
1104
+ "tokens_per_second": 922142,
1105
+ "minutes": 25.51,
1106
+ "mfu_percent": 22.2,
1107
  "gpu_util_percent": 100,
1108
+ "gpu_memory_gb": 85.1
1109
  },
1110
  {
1111
  "step": 5250,
1112
+ "train_loss": 3.0055,
1113
+ "lr": 0.00182,
1114
+ "tokens_per_second": 922504,
1115
+ "minutes": 25.75,
1116
+ "mfu_percent": 22.2,
1117
  "gpu_util_percent": 100,
1118
+ "gpu_memory_gb": 85.1
1119
  },
1120
  {
1121
  "step": 5300,
1122
+ "train_loss": 2.9704,
1123
+ "lr": 0.001802,
1124
+ "tokens_per_second": 922274,
1125
+ "minutes": 25.99,
1126
+ "mfu_percent": 22.2,
1127
  "gpu_util_percent": 100,
1128
+ "gpu_memory_gb": 85.1
1129
  },
1130
  {
1131
  "step": 5350,
1132
+ "train_loss": 2.9492,
1133
+ "lr": 0.001783,
1134
+ "tokens_per_second": 921682,
1135
+ "minutes": 26.23,
1136
+ "mfu_percent": 22.2,
1137
  "gpu_util_percent": 100,
1138
+ "gpu_memory_gb": 85.1
1139
  },
1140
  {
1141
  "step": 5400,
1142
+ "train_loss": 3.0025,
1143
+ "lr": 0.001765,
1144
+ "tokens_per_second": 921085,
1145
+ "minutes": 26.46,
1146
+ "mfu_percent": 22.2,
1147
  "gpu_util_percent": 100,
1148
+ "gpu_memory_gb": 85.1
1149
  },
1150
  {
1151
  "step": 5450,
1152
+ "train_loss": 2.9997,
1153
+ "lr": 0.001746,
1154
+ "tokens_per_second": 921161,
1155
+ "minutes": 26.7,
1156
+ "mfu_percent": 22.2,
1157
  "gpu_util_percent": 100,
1158
+ "gpu_memory_gb": 85.1
1159
  },
1160
  {
1161
  "step": 5500,
1162
+ "train_loss": 3.012,
1163
+ "lr": 0.001728,
1164
+ "tokens_per_second": 921331,
1165
+ "minutes": 26.94,
1166
+ "mfu_percent": 22.2,
1167
  "gpu_util_percent": 100,
1168
+ "gpu_memory_gb": 85.1,
1169
+ "val_loss": 3.0312
1170
  },
1171
  {
1172
  "step": 5550,
1173
+ "train_loss": 3.0025,
1174
+ "lr": 0.001709,
1175
+ "tokens_per_second": 921305,
1176
+ "minutes": 27.22,
1177
+ "mfu_percent": 22.2,
1178
  "gpu_util_percent": 100,
1179
+ "gpu_memory_gb": 85.1
1180
  },
1181
  {
1182
  "step": 5600,
1183
+ "train_loss": 2.9965,
1184
+ "lr": 0.001691,
1185
+ "tokens_per_second": 921367,
1186
+ "minutes": 27.46,
1187
+ "mfu_percent": 22.2,
1188
  "gpu_util_percent": 100,
1189
+ "gpu_memory_gb": 85.1
1190
  },
1191
  {
1192
  "step": 5650,
1193
+ "train_loss": 3.0261,
1194
+ "lr": 0.001672,
1195
+ "tokens_per_second": 921494,
1196
+ "minutes": 27.7,
1197
+ "mfu_percent": 22.2,
1198
  "gpu_util_percent": 100,
1199
+ "gpu_memory_gb": 85.1
1200
  },
1201
  {
1202
  "step": 5700,
1203
+ "train_loss": 2.9857,
1204
+ "lr": 0.001654,
1205
+ "tokens_per_second": 921931,
1206
+ "minutes": 27.93,
1207
+ "mfu_percent": 22.2,
1208
  "gpu_util_percent": 100,
1209
+ "gpu_memory_gb": 85.1
1210
  },
1211
  {
1212
+ "step": 5750,
1213
+ "train_loss": 2.9916,
1214
+ "lr": 0.001635,
1215
+ "tokens_per_second": 921797,
1216
+ "minutes": 28.17,
1217
+ "mfu_percent": 22.2,
1218
+ "gpu_util_percent": 100,
1219
+ "gpu_memory_gb": 85.1
1220
+ },
1221
+ {
1222
+ "step": 5800,
1223
+ "train_loss": 2.9868,
1224
+ "lr": 0.001616,
1225
+ "tokens_per_second": 921701,
1226
+ "minutes": 28.41,
1227
+ "mfu_percent": 22.2,
1228
+ "gpu_util_percent": 100,
1229
+ "gpu_memory_gb": 85.1
1230
+ },
1231
+ {
1232
+ "step": 5850,
1233
+ "train_loss": 3.0135,
1234
+ "lr": 0.001598,
1235
+ "tokens_per_second": 921157,
1236
+ "minutes": 28.64,
1237
+ "mfu_percent": 22.2,
1238
+ "gpu_util_percent": 100,
1239
+ "gpu_memory_gb": 85.1
1240
+ },
1241
+ {
1242
+ "step": 5900,
1243
+ "train_loss": 3.0171,
1244
+ "lr": 0.001579,
1245
+ "tokens_per_second": 921347,
1246
+ "minutes": 28.88,
1247
+ "mfu_percent": 22.2,
1248
+ "gpu_util_percent": 100,
1249
+ "gpu_memory_gb": 85.1
1250
+ },
1251
+ {
1252
+ "step": 5950,
1253
+ "train_loss": 3.0381,
1254
+ "lr": 0.001561,
1255
+ "tokens_per_second": 921107,
1256
+ "minutes": 29.12,
1257
+ "mfu_percent": 22.2,
1258
+ "gpu_util_percent": 100,
1259
+ "gpu_memory_gb": 85.1
1260
+ },
1261
+ {
1262
+ "step": 6000,
1263
+ "train_loss": 3.0234,
1264
+ "lr": 0.001542,
1265
+ "tokens_per_second": 920618,
1266
+ "minutes": 29.35,
1267
+ "mfu_percent": 22.2,
1268
+ "gpu_util_percent": 100,
1269
+ "gpu_memory_gb": 85.1,
1270
+ "val_loss": 3.014
1271
+ },
1272
+ {
1273
+ "step": 6050,
1274
+ "train_loss": 2.9671,
1275
+ "lr": 0.001524,
1276
+ "tokens_per_second": 920459,
1277
+ "minutes": 29.64,
1278
+ "mfu_percent": 22.2,
1279
+ "gpu_util_percent": 100,
1280
+ "gpu_memory_gb": 85.1
1281
+ },
1282
+ {
1283
+ "step": 6100,
1284
+ "train_loss": 3.0432,
1285
+ "lr": 0.001505,
1286
+ "tokens_per_second": 919920,
1287
+ "minutes": 29.88,
1288
+ "mfu_percent": 22.2,
1289
+ "gpu_util_percent": 100,
1290
+ "gpu_memory_gb": 85.1
1291
+ },
1292
+ {
1293
+ "step": 6150,
1294
+ "train_loss": 3.0124,
1295
+ "lr": 0.001487,
1296
+ "tokens_per_second": 920638,
1297
+ "minutes": 30.11,
1298
+ "mfu_percent": 22.2,
1299
+ "gpu_util_percent": 100,
1300
+ "gpu_memory_gb": 85.1
1301
+ },
1302
+ {
1303
+ "step": 6200,
1304
+ "train_loss": 3.0045,
1305
+ "lr": 0.001468,
1306
+ "tokens_per_second": 919978,
1307
+ "minutes": 30.35,
1308
+ "mfu_percent": 22.2,
1309
+ "gpu_util_percent": 100,
1310
+ "gpu_memory_gb": 85.1
1311
+ },
1312
+ {
1313
+ "step": 6250,
1314
+ "train_loss": 3.0096,
1315
+ "lr": 0.00145,
1316
+ "tokens_per_second": 919568,
1317
+ "minutes": 30.59,
1318
+ "mfu_percent": 22.2,
1319
+ "gpu_util_percent": 100,
1320
+ "gpu_memory_gb": 85.1
1321
+ },
1322
+ {
1323
+ "step": 6300,
1324
+ "train_loss": 2.9723,
1325
+ "lr": 0.001432,
1326
+ "tokens_per_second": 919292,
1327
+ "minutes": 30.83,
1328
+ "mfu_percent": 22.2,
1329
+ "gpu_util_percent": 100,
1330
+ "gpu_memory_gb": 85.1
1331
+ },
1332
+ {
1333
+ "step": 6350,
1334
+ "train_loss": 2.9286,
1335
+ "lr": 0.001413,
1336
+ "tokens_per_second": 919775,
1337
+ "minutes": 31.06,
1338
+ "mfu_percent": 22.2,
1339
+ "gpu_util_percent": 100,
1340
+ "gpu_memory_gb": 85.1
1341
+ },
1342
+ {
1343
+ "step": 6400,
1344
+ "train_loss": 3.0349,
1345
+ "lr": 0.001395,
1346
+ "tokens_per_second": 920505,
1347
+ "minutes": 31.3,
1348
+ "mfu_percent": 22.2,
1349
+ "gpu_util_percent": 100,
1350
+ "gpu_memory_gb": 85.1
1351
+ },
1352
+ {
1353
+ "step": 6450,
1354
+ "train_loss": 2.9374,
1355
+ "lr": 0.001377,
1356
+ "tokens_per_second": 920417,
1357
+ "minutes": 31.54,
1358
+ "mfu_percent": 22.2,
1359
+ "gpu_util_percent": 100,
1360
+ "gpu_memory_gb": 85.1
1361
+ },
1362
+ {
1363
+ "step": 6500,
1364
+ "train_loss": 2.9625,
1365
+ "lr": 0.001359,
1366
+ "tokens_per_second": 919773,
1367
+ "minutes": 31.78,
1368
+ "mfu_percent": 22.2,
1369
+ "gpu_util_percent": 100,
1370
+ "gpu_memory_gb": 85.1,
1371
+ "val_loss": 2.9999
1372
+ },
1373
+ {
1374
+ "step": 6550,
1375
+ "train_loss": 2.9994,
1376
+ "lr": 0.001341,
1377
+ "tokens_per_second": 919974,
1378
+ "minutes": 32.06,
1379
+ "mfu_percent": 22.2,
1380
+ "gpu_util_percent": 100,
1381
+ "gpu_memory_gb": 85.1
1382
+ },
1383
+ {
1384
+ "step": 6600,
1385
+ "train_loss": 2.9899,
1386
+ "lr": 0.001323,
1387
+ "tokens_per_second": 920286,
1388
+ "minutes": 32.3,
1389
+ "mfu_percent": 22.2,
1390
+ "gpu_util_percent": 100,
1391
+ "gpu_memory_gb": 85.1
1392
+ },
1393
+ {
1394
+ "step": 6650,
1395
+ "train_loss": 2.9663,
1396
+ "lr": 0.001305,
1397
+ "tokens_per_second": 920398,
1398
+ "minutes": 32.54,
1399
+ "mfu_percent": 22.2,
1400
+ "gpu_util_percent": 100,
1401
+ "gpu_memory_gb": 85.1
1402
+ },
1403
+ {
1404
+ "step": 6700,
1405
+ "train_loss": 2.9463,
1406
+ "lr": 0.001287,
1407
+ "tokens_per_second": 921337,
1408
+ "minutes": 32.77,
1409
+ "mfu_percent": 22.2,
1410
+ "gpu_util_percent": 100,
1411
+ "gpu_memory_gb": 85.1
1412
+ },
1413
+ {
1414
+ "step": 6750,
1415
+ "train_loss": 2.9973,
1416
+ "lr": 0.001269,
1417
+ "tokens_per_second": 921370,
1418
+ "minutes": 33.01,
1419
+ "mfu_percent": 22.2,
1420
+ "gpu_util_percent": 100,
1421
+ "gpu_memory_gb": 85.1
1422
+ },
1423
+ {
1424
+ "step": 6800,
1425
+ "train_loss": 2.9206,
1426
+ "lr": 0.001251,
1427
+ "tokens_per_second": 920742,
1428
+ "minutes": 33.25,
1429
+ "mfu_percent": 22.2,
1430
+ "gpu_util_percent": 100,
1431
+ "gpu_memory_gb": 85.1
1432
+ },
1433
+ {
1434
+ "step": 6850,
1435
+ "train_loss": 2.9574,
1436
+ "lr": 0.001233,
1437
+ "tokens_per_second": 920284,
1438
+ "minutes": 33.48,
1439
+ "mfu_percent": 22.2,
1440
+ "gpu_util_percent": 100,
1441
+ "gpu_memory_gb": 85.1
1442
+ },
1443
+ {
1444
+ "step": 6900,
1445
+ "train_loss": 2.9734,
1446
+ "lr": 0.001216,
1447
+ "tokens_per_second": 919909,
1448
+ "minutes": 33.72,
1449
+ "mfu_percent": 22.2,
1450
+ "gpu_util_percent": 100,
1451
+ "gpu_memory_gb": 85.1
1452
+ },
1453
+ {
1454
+ "step": 6950,
1455
+ "train_loss": 2.992,
1456
+ "lr": 0.001198,
1457
+ "tokens_per_second": 919380,
1458
+ "minutes": 33.96,
1459
+ "mfu_percent": 22.2,
1460
+ "gpu_util_percent": 100,
1461
+ "gpu_memory_gb": 85.1
1462
+ },
1463
+ {
1464
+ "step": 7000,
1465
+ "train_loss": 2.9482,
1466
+ "lr": 0.001181,
1467
+ "tokens_per_second": 919305,
1468
+ "minutes": 34.2,
1469
+ "mfu_percent": 22.2,
1470
+ "gpu_util_percent": 100,
1471
+ "gpu_memory_gb": 85.1,
1472
+ "val_loss": 2.9834
1473
+ },
1474
+ {
1475
+ "step": 7050,
1476
+ "train_loss": 2.9944,
1477
+ "lr": 0.001163,
1478
+ "tokens_per_second": 919402,
1479
+ "minutes": 34.48,
1480
+ "mfu_percent": 22.2,
1481
+ "gpu_util_percent": 100,
1482
+ "gpu_memory_gb": 85.1
1483
+ },
1484
+ {
1485
+ "step": 7100,
1486
+ "train_loss": 2.9649,
1487
+ "lr": 0.001146,
1488
+ "tokens_per_second": 919490,
1489
+ "minutes": 34.72,
1490
+ "mfu_percent": 22.2,
1491
+ "gpu_util_percent": 100,
1492
+ "gpu_memory_gb": 85.1
1493
+ },
1494
+ {
1495
+ "step": 7150,
1496
+ "train_loss": 2.9533,
1497
+ "lr": 0.001129,
1498
+ "tokens_per_second": 919450,
1499
+ "minutes": 34.96,
1500
+ "mfu_percent": 22.2,
1501
+ "gpu_util_percent": 100,
1502
+ "gpu_memory_gb": 85.1
1503
+ },
1504
+ {
1505
+ "step": 7200,
1506
+ "train_loss": 2.9467,
1507
+ "lr": 0.001112,
1508
+ "tokens_per_second": 919809,
1509
+ "minutes": 35.19,
1510
+ "mfu_percent": 22.2,
1511
+ "gpu_util_percent": 100,
1512
+ "gpu_memory_gb": 85.1
1513
+ },
1514
+ {
1515
+ "step": 7250,
1516
+ "train_loss": 2.9482,
1517
+ "lr": 0.001095,
1518
+ "tokens_per_second": 920118,
1519
+ "minutes": 35.43,
1520
+ "mfu_percent": 22.2,
1521
+ "gpu_util_percent": 100,
1522
+ "gpu_memory_gb": 85.1
1523
+ },
1524
+ {
1525
+ "step": 7300,
1526
+ "train_loss": 2.9288,
1527
+ "lr": 0.001078,
1528
+ "tokens_per_second": 920161,
1529
+ "minutes": 35.67,
1530
+ "mfu_percent": 22.2,
1531
+ "gpu_util_percent": 100,
1532
+ "gpu_memory_gb": 85.1
1533
+ },
1534
+ {
1535
+ "step": 7350,
1536
+ "train_loss": 2.9711,
1537
+ "lr": 0.001061,
1538
+ "tokens_per_second": 920166,
1539
+ "minutes": 35.91,
1540
+ "mfu_percent": 22.2,
1541
+ "gpu_util_percent": 100,
1542
+ "gpu_memory_gb": 85.1
1543
+ },
1544
+ {
1545
+ "step": 7400,
1546
+ "train_loss": 2.9711,
1547
+ "lr": 0.001045,
1548
+ "tokens_per_second": 920031,
1549
+ "minutes": 36.14,
1550
+ "mfu_percent": 22.2,
1551
+ "gpu_util_percent": 100,
1552
+ "gpu_memory_gb": 85.1
1553
+ },
1554
+ {
1555
+ "step": 7450,
1556
+ "train_loss": 2.9619,
1557
+ "lr": 0.001028,
1558
+ "tokens_per_second": 920234,
1559
+ "minutes": 36.38,
1560
+ "mfu_percent": 22.2,
1561
+ "gpu_util_percent": 100,
1562
+ "gpu_memory_gb": 85.1
1563
+ },
1564
+ {
1565
+ "step": 7500,
1566
+ "train_loss": 2.9199,
1567
+ "lr": 0.001012,
1568
+ "tokens_per_second": 920421,
1569
+ "minutes": 36.62,
1570
+ "mfu_percent": 22.2,
1571
+ "gpu_util_percent": 100,
1572
+ "gpu_memory_gb": 85.1,
1573
+ "val_loss": 2.9683
1574
+ },
1575
+ {
1576
+ "step": 7550,
1577
+ "train_loss": 2.9091,
1578
+ "lr": 0.000995,
1579
+ "tokens_per_second": 921260,
1580
+ "minutes": 36.9,
1581
+ "mfu_percent": 22.2,
1582
+ "gpu_util_percent": 100,
1583
+ "gpu_memory_gb": 85.1
1584
+ },
1585
+ {
1586
+ "step": 7600,
1587
+ "train_loss": 2.9747,
1588
+ "lr": 0.000979,
1589
+ "tokens_per_second": 920906,
1590
+ "minutes": 37.14,
1591
+ "mfu_percent": 22.2,
1592
+ "gpu_util_percent": 100,
1593
+ "gpu_memory_gb": 85.1
1594
+ },
1595
+ {
1596
+ "step": 7650,
1597
+ "train_loss": 2.9445,
1598
+ "lr": 0.000963,
1599
+ "tokens_per_second": 920371,
1600
+ "minutes": 37.38,
1601
+ "mfu_percent": 22.2,
1602
+ "gpu_util_percent": 100,
1603
+ "gpu_memory_gb": 85.1
1604
+ },
1605
+ {
1606
+ "step": 7700,
1607
+ "train_loss": 2.9593,
1608
+ "lr": 0.000947,
1609
+ "tokens_per_second": 920018,
1610
+ "minutes": 37.62,
1611
+ "mfu_percent": 22.2,
1612
+ "gpu_util_percent": 100,
1613
+ "gpu_memory_gb": 85.1
1614
+ },
1615
+ {
1616
+ "step": 7750,
1617
+ "train_loss": 2.9826,
1618
+ "lr": 0.000931,
1619
+ "tokens_per_second": 920997,
1620
+ "minutes": 37.85,
1621
+ "mfu_percent": 22.2,
1622
+ "gpu_util_percent": 100,
1623
+ "gpu_memory_gb": 85.1
1624
+ },
1625
+ {
1626
+ "step": 7800,
1627
+ "train_loss": 2.933,
1628
+ "lr": 0.000916,
1629
+ "tokens_per_second": 921462,
1630
+ "minutes": 38.09,
1631
+ "mfu_percent": 22.2,
1632
+ "gpu_util_percent": 100,
1633
+ "gpu_memory_gb": 85.1
1634
+ },
1635
+ {
1636
+ "step": 7850,
1637
+ "train_loss": 2.9202,
1638
+ "lr": 0.0009,
1639
+ "tokens_per_second": 921036,
1640
+ "minutes": 38.33,
1641
+ "mfu_percent": 22.2,
1642
+ "gpu_util_percent": 100,
1643
+ "gpu_memory_gb": 85.1
1644
+ },
1645
+ {
1646
+ "step": 7900,
1647
+ "train_loss": 2.9152,
1648
+ "lr": 0.000885,
1649
+ "tokens_per_second": 920538,
1650
+ "minutes": 38.56,
1651
+ "mfu_percent": 22.2,
1652
+ "gpu_util_percent": 100,
1653
+ "gpu_memory_gb": 85.1
1654
+ },
1655
+ {
1656
+ "step": 7950,
1657
+ "train_loss": 2.926,
1658
+ "lr": 0.00087,
1659
+ "tokens_per_second": 919615,
1660
+ "minutes": 38.8,
1661
+ "mfu_percent": 22.2,
1662
+ "gpu_util_percent": 100,
1663
+ "gpu_memory_gb": 85.1
1664
+ },
1665
+ {
1666
+ "step": 8000,
1667
+ "train_loss": 2.8966,
1668
+ "lr": 0.000855,
1669
+ "tokens_per_second": 919288,
1670
+ "minutes": 39.04,
1671
+ "mfu_percent": 22.2,
1672
+ "gpu_util_percent": 100,
1673
+ "gpu_memory_gb": 85.1,
1674
+ "val_loss": 2.9544
1675
+ },
1676
+ {
1677
+ "step": 8050,
1678
+ "train_loss": 2.9517,
1679
+ "lr": 0.00084,
1680
+ "tokens_per_second": 919884,
1681
+ "minutes": 39.32,
1682
+ "mfu_percent": 22.2,
1683
+ "gpu_util_percent": 100,
1684
+ "gpu_memory_gb": 85.1
1685
+ },
1686
+ {
1687
+ "step": 8100,
1688
+ "train_loss": 2.9257,
1689
+ "lr": 0.000825,
1690
+ "tokens_per_second": 921250,
1691
+ "minutes": 39.56,
1692
+ "mfu_percent": 22.2,
1693
+ "gpu_util_percent": 100,
1694
+ "gpu_memory_gb": 85.1
1695
+ },
1696
+ {
1697
+ "step": 8150,
1698
+ "train_loss": 2.9299,
1699
+ "lr": 0.00081,
1700
+ "tokens_per_second": 921425,
1701
+ "minutes": 39.8,
1702
+ "mfu_percent": 22.2,
1703
+ "gpu_util_percent": 100,
1704
+ "gpu_memory_gb": 85.1
1705
+ },
1706
+ {
1707
+ "step": 8200,
1708
+ "train_loss": 2.934,
1709
+ "lr": 0.000796,
1710
+ "tokens_per_second": 921296,
1711
+ "minutes": 40.04,
1712
+ "mfu_percent": 22.2,
1713
+ "gpu_util_percent": 100,
1714
+ "gpu_memory_gb": 85.1
1715
+ },
1716
+ {
1717
+ "step": 8250,
1718
+ "train_loss": 2.8999,
1719
+ "lr": 0.000782,
1720
+ "tokens_per_second": 920909,
1721
+ "minutes": 40.27,
1722
+ "mfu_percent": 22.2,
1723
+ "gpu_util_percent": 100,
1724
+ "gpu_memory_gb": 85.1
1725
+ },
1726
+ {
1727
+ "step": 8300,
1728
+ "train_loss": 2.9048,
1729
+ "lr": 0.000767,
1730
+ "tokens_per_second": 920173,
1731
+ "minutes": 40.51,
1732
+ "mfu_percent": 22.2,
1733
+ "gpu_util_percent": 100,
1734
+ "gpu_memory_gb": 85.1
1735
+ },
1736
+ {
1737
+ "step": 8350,
1738
+ "train_loss": 2.8714,
1739
+ "lr": 0.000753,
1740
+ "tokens_per_second": 919853,
1741
+ "minutes": 40.75,
1742
+ "mfu_percent": 22.2,
1743
+ "gpu_util_percent": 100,
1744
+ "gpu_memory_gb": 85.1
1745
+ },
1746
+ {
1747
+ "step": 8400,
1748
+ "train_loss": 2.8722,
1749
+ "lr": 0.00074,
1750
+ "tokens_per_second": 919551,
1751
+ "minutes": 40.98,
1752
+ "mfu_percent": 22.2,
1753
+ "gpu_util_percent": 100,
1754
+ "gpu_memory_gb": 85.1
1755
+ },
1756
+ {
1757
+ "step": 8450,
1758
+ "train_loss": 2.9693,
1759
+ "lr": 0.000726,
1760
+ "tokens_per_second": 920679,
1761
+ "minutes": 41.22,
1762
+ "mfu_percent": 22.2,
1763
+ "gpu_util_percent": 100,
1764
+ "gpu_memory_gb": 85.1
1765
+ },
1766
+ {
1767
+ "step": 8500,
1768
+ "train_loss": 2.9393,
1769
+ "lr": 0.000713,
1770
+ "tokens_per_second": 921776,
1771
+ "minutes": 41.46,
1772
+ "mfu_percent": 22.2,
1773
+ "gpu_util_percent": 100,
1774
+ "gpu_memory_gb": 85.1,
1775
+ "val_loss": 2.9392
1776
+ },
1777
+ {
1778
+ "step": 8550,
1779
+ "train_loss": 2.8964,
1780
+ "lr": 0.000699,
1781
+ "tokens_per_second": 920913,
1782
+ "minutes": 41.74,
1783
+ "mfu_percent": 22.2,
1784
+ "gpu_util_percent": 100,
1785
+ "gpu_memory_gb": 85.1
1786
+ },
1787
+ {
1788
+ "step": 8600,
1789
+ "train_loss": 2.9192,
1790
+ "lr": 0.000686,
1791
+ "tokens_per_second": 919992,
1792
+ "minutes": 41.98,
1793
+ "mfu_percent": 22.2,
1794
+ "gpu_util_percent": 100,
1795
+ "gpu_memory_gb": 85.1
1796
+ },
1797
+ {
1798
+ "step": 8650,
1799
+ "train_loss": 2.9288,
1800
+ "lr": 0.000673,
1801
+ "tokens_per_second": 919914,
1802
+ "minutes": 42.22,
1803
+ "mfu_percent": 22.2,
1804
+ "gpu_util_percent": 100,
1805
+ "gpu_memory_gb": 85.1
1806
+ },
1807
+ {
1808
+ "step": 8700,
1809
+ "train_loss": 2.8246,
1810
+ "lr": 0.000661,
1811
+ "tokens_per_second": 919626,
1812
+ "minutes": 42.46,
1813
+ "mfu_percent": 22.2,
1814
+ "gpu_util_percent": 100,
1815
+ "gpu_memory_gb": 85.1
1816
+ },
1817
+ {
1818
+ "step": 8750,
1819
+ "train_loss": 2.9473,
1820
+ "lr": 0.000648,
1821
+ "tokens_per_second": 919635,
1822
+ "minutes": 42.69,
1823
+ "mfu_percent": 22.2,
1824
+ "gpu_util_percent": 100,
1825
+ "gpu_memory_gb": 85.1
1826
+ },
1827
+ {
1828
+ "step": 8800,
1829
+ "train_loss": 2.8781,
1830
+ "lr": 0.000636,
1831
+ "tokens_per_second": 921027,
1832
+ "minutes": 42.93,
1833
+ "mfu_percent": 22.2,
1834
+ "gpu_util_percent": 100,
1835
+ "gpu_memory_gb": 85.1
1836
+ },
1837
+ {
1838
+ "step": 8850,
1839
+ "train_loss": 2.8393,
1840
+ "lr": 0.000624,
1841
+ "tokens_per_second": 921693,
1842
+ "minutes": 43.17,
1843
+ "mfu_percent": 22.2,
1844
+ "gpu_util_percent": 100,
1845
+ "gpu_memory_gb": 85.1
1846
+ },
1847
+ {
1848
+ "step": 8900,
1849
+ "train_loss": 2.9105,
1850
+ "lr": 0.000612,
1851
+ "tokens_per_second": 922213,
1852
+ "minutes": 43.4,
1853
+ "mfu_percent": 22.2,
1854
+ "gpu_util_percent": 100,
1855
+ "gpu_memory_gb": 85.1
1856
+ },
1857
+ {
1858
+ "step": 8950,
1859
+ "train_loss": 2.9286,
1860
+ "lr": 0.0006,
1861
+ "tokens_per_second": 923184,
1862
+ "minutes": 43.64,
1863
+ "mfu_percent": 22.3,
1864
+ "gpu_util_percent": 100,
1865
+ "gpu_memory_gb": 85.1
1866
+ },
1867
+ {
1868
+ "step": 9000,
1869
+ "train_loss": 2.8668,
1870
+ "lr": 0.000588,
1871
+ "tokens_per_second": 923153,
1872
+ "minutes": 43.88,
1873
+ "mfu_percent": 22.3,
1874
+ "gpu_util_percent": 100,
1875
+ "gpu_memory_gb": 85.1,
1876
+ "val_loss": 2.9286
1877
+ },
1878
+ {
1879
+ "step": 9050,
1880
+ "train_loss": 2.9063,
1881
+ "lr": 0.000577,
1882
+ "tokens_per_second": 922088,
1883
+ "minutes": 44.16,
1884
+ "mfu_percent": 22.2,
1885
+ "gpu_util_percent": 100,
1886
+ "gpu_memory_gb": 85.1
1887
+ },
1888
+ {
1889
+ "step": 9100,
1890
+ "train_loss": 2.9428,
1891
+ "lr": 0.000566,
1892
+ "tokens_per_second": 921296,
1893
+ "minutes": 44.4,
1894
+ "mfu_percent": 22.2,
1895
+ "gpu_util_percent": 100,
1896
+ "gpu_memory_gb": 85.1
1897
+ },
1898
+ {
1899
+ "step": 9150,
1900
+ "train_loss": 2.9146,
1901
+ "lr": 0.000555,
1902
+ "tokens_per_second": 920478,
1903
+ "minutes": 44.64,
1904
+ "mfu_percent": 22.2,
1905
+ "gpu_util_percent": 100,
1906
+ "gpu_memory_gb": 85.1
1907
+ },
1908
+ {
1909
+ "step": 9200,
1910
+ "train_loss": 2.9337,
1911
+ "lr": 0.000544,
1912
+ "tokens_per_second": 919857,
1913
+ "minutes": 44.87,
1914
+ "mfu_percent": 22.2,
1915
+ "gpu_util_percent": 100,
1916
+ "gpu_memory_gb": 85.1
1917
+ },
1918
+ {
1919
+ "step": 9250,
1920
+ "train_loss": 2.8898,
1921
+ "lr": 0.000533,
1922
+ "tokens_per_second": 919677,
1923
+ "minutes": 45.11,
1924
+ "mfu_percent": 22.2,
1925
+ "gpu_util_percent": 100,
1926
+ "gpu_memory_gb": 85.1
1927
+ },
1928
+ {
1929
+ "step": 9300,
1930
+ "train_loss": 2.8907,
1931
+ "lr": 0.000523,
1932
+ "tokens_per_second": 919018,
1933
+ "minutes": 45.35,
1934
+ "mfu_percent": 22.2,
1935
+ "gpu_util_percent": 100,
1936
+ "gpu_memory_gb": 85.1
1937
+ },
1938
+ {
1939
+ "step": 9350,
1940
+ "train_loss": 2.9104,
1941
+ "lr": 0.000513,
1942
+ "tokens_per_second": 919151,
1943
+ "minutes": 45.59,
1944
+ "mfu_percent": 22.2,
1945
+ "gpu_util_percent": 100,
1946
+ "gpu_memory_gb": 85.1
1947
+ },
1948
+ {
1949
+ "step": 9400,
1950
+ "train_loss": 2.9346,
1951
+ "lr": 0.000503,
1952
+ "tokens_per_second": 919820,
1953
+ "minutes": 45.82,
1954
+ "mfu_percent": 22.2,
1955
+ "gpu_util_percent": 100,
1956
+ "gpu_memory_gb": 85.1
1957
+ },
1958
+ {
1959
+ "step": 9450,
1960
+ "train_loss": 2.864,
1961
+ "lr": 0.000493,
1962
+ "tokens_per_second": 920435,
1963
+ "minutes": 46.06,
1964
+ "mfu_percent": 22.2,
1965
+ "gpu_util_percent": 100,
1966
+ "gpu_memory_gb": 85.1
1967
+ },
1968
+ {
1969
+ "step": 9500,
1970
+ "train_loss": 2.8496,
1971
+ "lr": 0.000484,
1972
+ "tokens_per_second": 920689,
1973
+ "minutes": 46.3,
1974
+ "mfu_percent": 22.2,
1975
+ "gpu_util_percent": 100,
1976
+ "gpu_memory_gb": 85.1,
1977
+ "val_loss": 2.9166
1978
+ },
1979
+ {
1980
+ "step": 9550,
1981
+ "train_loss": 2.9794,
1982
+ "lr": 0.000475,
1983
+ "tokens_per_second": 920761,
1984
+ "minutes": 46.58,
1985
+ "mfu_percent": 22.2,
1986
+ "gpu_util_percent": 100,
1987
+ "gpu_memory_gb": 85.1
1988
+ },
1989
+ {
1990
+ "step": 9600,
1991
+ "train_loss": 2.8557,
1992
+ "lr": 0.000466,
1993
+ "tokens_per_second": 921094,
1994
+ "minutes": 46.82,
1995
+ "mfu_percent": 22.2,
1996
+ "gpu_util_percent": 100,
1997
+ "gpu_memory_gb": 85.1
1998
+ },
1999
+ {
2000
+ "step": 9650,
2001
+ "train_loss": 2.8992,
2002
+ "lr": 0.000457,
2003
+ "tokens_per_second": 921528,
2004
+ "minutes": 47.06,
2005
+ "mfu_percent": 22.2,
2006
+ "gpu_util_percent": 100,
2007
+ "gpu_memory_gb": 85.1
2008
+ },
2009
+ {
2010
+ "step": 9700,
2011
+ "train_loss": 2.835,
2012
+ "lr": 0.000448,
2013
+ "tokens_per_second": 920716,
2014
+ "minutes": 47.29,
2015
+ "mfu_percent": 22.2,
2016
+ "gpu_util_percent": 100,
2017
+ "gpu_memory_gb": 85.1
2018
+ },
2019
+ {
2020
+ "step": 9750,
2021
+ "train_loss": 2.9188,
2022
+ "lr": 0.00044,
2023
+ "tokens_per_second": 920417,
2024
+ "minutes": 47.53,
2025
+ "mfu_percent": 22.2,
2026
+ "gpu_util_percent": 100,
2027
+ "gpu_memory_gb": 85.1
2028
+ },
2029
+ {
2030
+ "step": 9800,
2031
+ "train_loss": 2.925,
2032
+ "lr": 0.000432,
2033
+ "tokens_per_second": 920052,
2034
+ "minutes": 47.77,
2035
+ "mfu_percent": 22.2,
2036
+ "gpu_util_percent": 100,
2037
+ "gpu_memory_gb": 85.1
2038
+ },
2039
+ {
2040
+ "step": 9850,
2041
+ "train_loss": 2.9457,
2042
+ "lr": 0.000424,
2043
+ "tokens_per_second": 919561,
2044
+ "minutes": 48.01,
2045
+ "mfu_percent": 22.2,
2046
+ "gpu_util_percent": 100,
2047
+ "gpu_memory_gb": 85.1
2048
+ },
2049
+ {
2050
+ "step": 9900,
2051
+ "train_loss": 2.9236,
2052
+ "lr": 0.000416,
2053
+ "tokens_per_second": 919301,
2054
+ "minutes": 48.24,
2055
+ "mfu_percent": 22.2,
2056
+ "gpu_util_percent": 100,
2057
+ "gpu_memory_gb": 85.1
2058
+ },
2059
+ {
2060
+ "step": 9950,
2061
+ "train_loss": 2.8542,
2062
+ "lr": 0.000409,
2063
+ "tokens_per_second": 919176,
2064
+ "minutes": 48.48,
2065
+ "mfu_percent": 22.2,
2066
+ "gpu_util_percent": 100,
2067
+ "gpu_memory_gb": 85.1
2068
+ },
2069
+ {
2070
+ "step": 10000,
2071
+ "train_loss": 2.8907,
2072
+ "lr": 0.000402,
2073
+ "tokens_per_second": 919788,
2074
+ "minutes": 48.72,
2075
+ "mfu_percent": 22.2,
2076
+ "gpu_util_percent": 100,
2077
+ "gpu_memory_gb": 85.1,
2078
+ "val_loss": 2.9086
2079
+ },
2080
+ {
2081
+ "step": 10050,
2082
+ "train_loss": 2.7981,
2083
+ "lr": 0.000395,
2084
+ "tokens_per_second": 920621,
2085
+ "minutes": 49.0,
2086
+ "mfu_percent": 22.2,
2087
+ "gpu_util_percent": 100,
2088
+ "gpu_memory_gb": 85.1
2089
+ },
2090
+ {
2091
+ "step": 10100,
2092
+ "train_loss": 2.8218,
2093
+ "lr": 0.000388,
2094
+ "tokens_per_second": 920907,
2095
+ "minutes": 49.24,
2096
+ "mfu_percent": 22.2,
2097
+ "gpu_util_percent": 100,
2098
+ "gpu_memory_gb": 85.1
2099
+ },
2100
+ {
2101
+ "step": 10150,
2102
+ "train_loss": 2.8762,
2103
+ "lr": 0.000382,
2104
+ "tokens_per_second": 921101,
2105
+ "minutes": 49.48,
2106
+ "mfu_percent": 22.2,
2107
+ "gpu_util_percent": 100,
2108
+ "gpu_memory_gb": 85.1
2109
+ },
2110
+ {
2111
+ "step": 10200,
2112
+ "train_loss": 2.8884,
2113
+ "lr": 0.000375,
2114
+ "tokens_per_second": 920843,
2115
+ "minutes": 49.72,
2116
+ "mfu_percent": 22.2,
2117
+ "gpu_util_percent": 100,
2118
+ "gpu_memory_gb": 85.1
2119
+ },
2120
+ {
2121
+ "step": 10250,
2122
+ "train_loss": 2.8617,
2123
+ "lr": 0.000369,
2124
+ "tokens_per_second": 921002,
2125
+ "minutes": 49.95,
2126
+ "mfu_percent": 22.2,
2127
+ "gpu_util_percent": 100,
2128
+ "gpu_memory_gb": 85.1
2129
+ },
2130
+ {
2131
+ "step": 10300,
2132
+ "train_loss": 2.8414,
2133
+ "lr": 0.000364,
2134
+ "tokens_per_second": 921151,
2135
+ "minutes": 50.19,
2136
+ "mfu_percent": 22.2,
2137
+ "gpu_util_percent": 100,
2138
+ "gpu_memory_gb": 85.1
2139
+ },
2140
+ {
2141
+ "step": 10350,
2142
+ "train_loss": 2.8439,
2143
+ "lr": 0.000358,
2144
+ "tokens_per_second": 920309,
2145
+ "minutes": 50.43,
2146
+ "mfu_percent": 22.2,
2147
+ "gpu_util_percent": 100,
2148
+ "gpu_memory_gb": 85.1
2149
+ },
2150
+ {
2151
+ "step": 10400,
2152
+ "train_loss": 2.8992,
2153
+ "lr": 0.000353,
2154
+ "tokens_per_second": 919693,
2155
+ "minutes": 50.66,
2156
+ "mfu_percent": 22.2,
2157
+ "gpu_util_percent": 100,
2158
+ "gpu_memory_gb": 85.1
2159
+ },
2160
+ {
2161
+ "step": 10450,
2162
+ "train_loss": 2.8711,
2163
+ "lr": 0.000348,
2164
+ "tokens_per_second": 919270,
2165
+ "minutes": 50.9,
2166
+ "mfu_percent": 22.2,
2167
+ "gpu_util_percent": 100,
2168
+ "gpu_memory_gb": 85.1
2169
+ },
2170
+ {
2171
+ "step": 10500,
2172
+ "train_loss": 2.8533,
2173
+ "lr": 0.000343,
2174
+ "tokens_per_second": 920712,
2175
+ "minutes": 51.14,
2176
+ "mfu_percent": 22.2,
2177
+ "gpu_util_percent": 100,
2178
+ "gpu_memory_gb": 85.1,
2179
+ "val_loss": 2.9005
2180
+ },
2181
+ {
2182
+ "step": 10550,
2183
+ "train_loss": 2.8638,
2184
+ "lr": 0.000338,
2185
+ "tokens_per_second": 921492,
2186
+ "minutes": 51.42,
2187
+ "mfu_percent": 22.2,
2188
+ "gpu_util_percent": 100,
2189
+ "gpu_memory_gb": 85.1
2190
+ },
2191
+ {
2192
+ "step": 10600,
2193
+ "train_loss": 2.8673,
2194
+ "lr": 0.000334,
2195
+ "tokens_per_second": 921466,
2196
+ "minutes": 51.66,
2197
+ "mfu_percent": 22.2,
2198
+ "gpu_util_percent": 100,
2199
+ "gpu_memory_gb": 85.1
2200
+ },
2201
+ {
2202
+ "step": 10650,
2203
+ "train_loss": 2.8849,
2204
+ "lr": 0.00033,
2205
+ "tokens_per_second": 920910,
2206
+ "minutes": 51.9,
2207
+ "mfu_percent": 22.2,
2208
+ "gpu_util_percent": 100,
2209
+ "gpu_memory_gb": 85.1
2210
+ },
2211
+ {
2212
+ "step": 10700,
2213
+ "train_loss": 2.8652,
2214
+ "lr": 0.000326,
2215
+ "tokens_per_second": 920277,
2216
+ "minutes": 52.14,
2217
+ "mfu_percent": 22.2,
2218
+ "gpu_util_percent": 100,
2219
+ "gpu_memory_gb": 85.1
2220
+ },
2221
+ {
2222
+ "step": 10750,
2223
+ "train_loss": 2.8523,
2224
+ "lr": 0.000323,
2225
+ "tokens_per_second": 919903,
2226
+ "minutes": 52.37,
2227
+ "mfu_percent": 22.2,
2228
+ "gpu_util_percent": 100,
2229
+ "gpu_memory_gb": 85.1
2230
+ },
2231
+ {
2232
+ "step": 10800,
2233
+ "train_loss": 2.8797,
2234
+ "lr": 0.00032,
2235
+ "tokens_per_second": 919671,
2236
+ "minutes": 52.61,
2237
+ "mfu_percent": 22.2,
2238
+ "gpu_util_percent": 100,
2239
+ "gpu_memory_gb": 85.1
2240
+ },
2241
+ {
2242
+ "step": 10850,
2243
+ "train_loss": 2.858,
2244
+ "lr": 0.000317,
2245
+ "tokens_per_second": 919122,
2246
+ "minutes": 52.85,
2247
+ "mfu_percent": 22.2,
2248
+ "gpu_util_percent": 100,
2249
+ "gpu_memory_gb": 85.1
2250
+ },
2251
+ {
2252
+ "step": 10900,
2253
+ "train_loss": 2.8607,
2254
+ "lr": 0.000314,
2255
+ "tokens_per_second": 919161,
2256
+ "minutes": 53.09,
2257
+ "mfu_percent": 22.2,
2258
+ "gpu_util_percent": 100,
2259
+ "gpu_memory_gb": 85.1
2260
+ },
2261
+ {
2262
+ "step": 10950,
2263
+ "train_loss": 2.8733,
2264
+ "lr": 0.000311,
2265
+ "tokens_per_second": 920086,
2266
+ "minutes": 53.32,
2267
+ "mfu_percent": 22.2,
2268
+ "gpu_util_percent": 100,
2269
+ "gpu_memory_gb": 85.1
2270
+ },
2271
+ {
2272
+ "step": 11000,
2273
+ "train_loss": 2.8539,
2274
+ "lr": 0.000309,
2275
+ "tokens_per_second": 920688,
2276
+ "minutes": 53.56,
2277
+ "mfu_percent": 22.2,
2278
+ "gpu_util_percent": 100,
2279
+ "gpu_memory_gb": 85.1,
2280
+ "val_loss": 2.8962
2281
+ },
2282
+ {
2283
+ "step": 11050,
2284
+ "train_loss": 2.8992,
2285
+ "lr": 0.000307,
2286
+ "tokens_per_second": 920928,
2287
+ "minutes": 53.84,
2288
+ "mfu_percent": 22.2,
2289
+ "gpu_util_percent": 100,
2290
+ "gpu_memory_gb": 85.1
2291
+ },
2292
+ {
2293
+ "step": 11100,
2294
+ "train_loss": 2.8486,
2295
+ "lr": 0.000305,
2296
+ "tokens_per_second": 920127,
2297
+ "minutes": 54.08,
2298
+ "mfu_percent": 22.2,
2299
+ "gpu_util_percent": 100,
2300
+ "gpu_memory_gb": 85.1
2301
+ },
2302
+ {
2303
+ "step": 11150,
2304
+ "train_loss": 2.8515,
2305
+ "lr": 0.000304,
2306
+ "tokens_per_second": 919904,
2307
+ "minutes": 54.32,
2308
+ "mfu_percent": 22.2,
2309
+ "gpu_util_percent": 100,
2310
+ "gpu_memory_gb": 85.1
2311
+ },
2312
+ {
2313
+ "step": 11200,
2314
+ "train_loss": 2.8647,
2315
+ "lr": 0.000302,
2316
+ "tokens_per_second": 919524,
2317
+ "minutes": 54.56,
2318
+ "mfu_percent": 22.2,
2319
+ "gpu_util_percent": 100,
2320
+ "gpu_memory_gb": 85.1
2321
+ },
2322
+ {
2323
+ "step": 11250,
2324
+ "train_loss": 2.8809,
2325
+ "lr": 0.000301,
2326
+ "tokens_per_second": 920127,
2327
+ "minutes": 54.79,
2328
+ "mfu_percent": 22.2,
2329
+ "gpu_util_percent": 100,
2330
+ "gpu_memory_gb": 85.1
2331
+ },
2332
+ {
2333
+ "step": 11300,
2334
+ "train_loss": 2.8752,
2335
+ "lr": 0.000301,
2336
+ "tokens_per_second": 921273,
2337
+ "minutes": 55.03,
2338
+ "mfu_percent": 22.2,
2339
+ "gpu_util_percent": 100,
2340
+ "gpu_memory_gb": 85.1
2341
+ },
2342
+ {
2343
+ "step": 11350,
2344
+ "train_loss": 2.9045,
2345
+ "lr": 0.0003,
2346
+ "tokens_per_second": 920772,
2347
+ "minutes": 55.27,
2348
+ "mfu_percent": 22.2,
2349
+ "gpu_util_percent": 100,
2350
+ "gpu_memory_gb": 85.1
2351
+ },
2352
+ {
2353
+ "step": 11400,
2354
+ "train_loss": 2.7934,
2355
+ "lr": 0.0003,
2356
+ "tokens_per_second": 920858,
2357
+ "minutes": 55.51,
2358
+ "mfu_percent": 22.2,
2359
+ "gpu_util_percent": 100,
2360
+ "gpu_memory_gb": 85.1
2361
+ },
2362
+ {
2363
+ "step": 11420,
2364
+ "train_loss": 2.8882,
2365
+ "lr": 0.0003,
2366
+ "tokens_per_second": 921250,
2367
+ "minutes": 55.6,
2368
+ "mfu_percent": 22.2,
2369
  "gpu_util_percent": 100,
2370
+ "gpu_memory_gb": 85.1,
2371
+ "val_loss": 2.8934
2372
  }
2373
  ],
2374
  "exports": {
2375
+ "parity_prompts": 300,
2376
+ "parity_steps_per_prompt": 5,
2377
  "float": {
2378
  "megabytes": 60.81,
2379
+ "same_first_token": 1.0,
2380
+ "same_five_greedy_tokens": 1.0,
2381
+ "cached_step_ms_p50_cpu": 5.544
2382
  },
2383
  "int8": {
2384
+ "megabytes": 15.64,
2385
+ "same_first_token": 0.0,
2386
+ "same_five_greedy_tokens": 0.0,
2387
+ "cached_step_ms_p50_cpu": 4.808
2388
  }
2389
  }
2390
  }
v2/tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
v2/web/graph.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:e63c34d5f6882b6c9d7dda768ea6009edc831f26f5e7a800161a54a204f521d7
3
- size 194807
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8ceff8e62001770e7f692f666b1cbd3d0bbc563c343ccc84f7a52792271121dd
3
+ size 194793
v2/web/parity.json CHANGED
@@ -1,16 +1,20 @@
1
  {
2
  "parity_prompts": 300,
3
  "parity_steps_per_prompt": 5,
 
 
 
 
 
 
4
  "float": {
5
  "megabytes": 60.81,
6
  "same_first_token": 1.0,
7
- "same_five_greedy_tokens": 1.0,
8
- "cached_step_ms_p50_cpu": 1.2
9
  },
10
- "int8": {
11
- "megabytes": 15.64,
12
- "same_first_token": 0.55,
13
- "same_five_greedy_tokens": 0.33,
14
- "cached_step_ms_p50_cpu": 0.71
15
  }
16
  }
 
1
  {
2
  "parity_prompts": 300,
3
  "parity_steps_per_prompt": 5,
4
+ "web_sizes": {
5
+ "graph_mb": 0.195,
6
+ "weights_q8_mb": 13.32,
7
+ "expanded_mb": 50.14,
8
+ "tensors": 33
9
+ },
10
  "float": {
11
  "megabytes": 60.81,
12
  "same_first_token": 1.0,
13
+ "same_five_greedy_tokens": 1.0
 
14
  },
15
+ "web_q8": {
16
+ "megabytes": 13.32,
17
+ "same_first_token": 0.96,
18
+ "same_five_greedy_tokens": 0.8967
 
19
  }
20
  }
v2/web/tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
v2/web/weights.json CHANGED
@@ -1 +1 @@
1
- {"block": 64, "f32_bytes": 50135040, "tensors": [{"name": "model.embed.weight", "shape": [8192, 320], "count": 2621440, "f32_offset": 0, "q_offset": 0, "scale_offset": 2621440}, {"name": "onnx::MatMul_1554", "shape": [320, 960], "count": 307200, "f32_offset": 10485760, "q_offset": 2785280, "scale_offset": 3092480}, {"name": "onnx::MatMul_1571", "shape": [320, 320], "count": 102400, "f32_offset": 11714560, "q_offset": 3111680, "scale_offset": 3214080}, {"name": "onnx::MatMul_1572", "shape": [320, 1728], "count": 552960, "f32_offset": 12124160, "q_offset": 3220480, "scale_offset": 3773440}, {"name": "onnx::MatMul_1573", "shape": [864, 320], "count": 276480, "f32_offset": 14336000, "q_offset": 3808000, "scale_offset": 4084480}, {"name": "onnx::MatMul_1574", "shape": [320, 960], "count": 307200, "f32_offset": 15441920, "q_offset": 4101760, "scale_offset": 4408960}, {"name": "onnx::MatMul_1591", "shape": [320, 320], "count": 102400, "f32_offset": 16670720, "q_offset": 4428160, "scale_offset": 4530560}, {"name": "onnx::MatMul_1592", "shape": [320, 1728], "count": 552960, "f32_offset": 17080320, "q_offset": 4536960, "scale_offset": 5089920}, {"name": "onnx::MatMul_1593", "shape": [864, 320], "count": 276480, "f32_offset": 19292160, "q_offset": 5124480, "scale_offset": 5400960}, {"name": "onnx::MatMul_1594", "shape": [320, 960], "count": 307200, "f32_offset": 20398080, "q_offset": 5418240, "scale_offset": 5725440}, {"name": "onnx::MatMul_1611", "shape": [320, 320], "count": 102400, "f32_offset": 21626880, "q_offset": 5744640, "scale_offset": 5847040}, {"name": "onnx::MatMul_1612", "shape": [320, 1728], "count": 552960, "f32_offset": 22036480, "q_offset": 5853440, "scale_offset": 6406400}, {"name": "onnx::MatMul_1613", "shape": [864, 320], "count": 276480, "f32_offset": 24248320, "q_offset": 6440960, "scale_offset": 6717440}, {"name": "onnx::MatMul_1614", "shape": [320, 960], "count": 307200, "f32_offset": 25354240, "q_offset": 6734720, "scale_offset": 7041920}, {"name": "onnx::MatMul_1631", "shape": [320, 320], "count": 102400, "f32_offset": 26583040, "q_offset": 7061120, "scale_offset": 7163520}, {"name": "onnx::MatMul_1632", "shape": [320, 1728], "count": 552960, "f32_offset": 26992640, "q_offset": 7169920, "scale_offset": 7722880}, {"name": "onnx::MatMul_1633", "shape": [864, 320], "count": 276480, "f32_offset": 29204480, "q_offset": 7757440, "scale_offset": 8033920}, {"name": "onnx::MatMul_1634", "shape": [320, 960], "count": 307200, "f32_offset": 30310400, "q_offset": 8051200, "scale_offset": 8358400}, {"name": "onnx::MatMul_1651", "shape": [320, 320], "count": 102400, "f32_offset": 31539200, "q_offset": 8377600, "scale_offset": 8480000}, {"name": "onnx::MatMul_1652", "shape": [320, 1728], "count": 552960, "f32_offset": 31948800, "q_offset": 8486400, "scale_offset": 9039360}, {"name": "onnx::MatMul_1653", "shape": [864, 320], "count": 276480, "f32_offset": 34160640, "q_offset": 9073920, "scale_offset": 9350400}, {"name": "onnx::MatMul_1654", "shape": [320, 960], "count": 307200, "f32_offset": 35266560, "q_offset": 9367680, "scale_offset": 9674880}, {"name": "onnx::MatMul_1671", "shape": [320, 320], "count": 102400, "f32_offset": 36495360, "q_offset": 9694080, "scale_offset": 9796480}, {"name": "onnx::MatMul_1672", "shape": [320, 1728], "count": 552960, "f32_offset": 36904960, "q_offset": 9802880, "scale_offset": 10355840}, {"name": "onnx::MatMul_1673", "shape": [864, 320], "count": 276480, "f32_offset": 39116800, "q_offset": 10390400, "scale_offset": 10666880}, {"name": "onnx::MatMul_1674", "shape": [320, 960], "count": 307200, "f32_offset": 40222720, "q_offset": 10684160, "scale_offset": 10991360}, {"name": "onnx::MatMul_1691", "shape": [320, 320], "count": 102400, "f32_offset": 41451520, "q_offset": 11010560, "scale_offset": 11112960}, {"name": "onnx::MatMul_1692", "shape": [320, 1728], "count": 552960, "f32_offset": 41861120, "q_offset": 11119360, "scale_offset": 11672320}, {"name": "onnx::MatMul_1693", "shape": [864, 320], "count": 276480, "f32_offset": 44072960, "q_offset": 11706880, "scale_offset": 11983360}, {"name": "onnx::MatMul_1694", "shape": [320, 960], "count": 307200, "f32_offset": 45178880, "q_offset": 12000640, "scale_offset": 12307840}, {"name": "onnx::MatMul_1711", "shape": [320, 320], "count": 102400, "f32_offset": 46407680, "q_offset": 12327040, "scale_offset": 12429440}, {"name": "onnx::MatMul_1712", "shape": [320, 1728], "count": 552960, "f32_offset": 46817280, "q_offset": 12435840, "scale_offset": 12988800}, {"name": "onnx::MatMul_1713", "shape": [864, 320], "count": 276480, "f32_offset": 49029120, "q_offset": 13023360, "scale_offset": 13299840}]}
 
1
+ {"block": 64, "f32_bytes": 50135040, "tensors": [{"name": "onnx::MatMul_1554", "shape": [320, 960], "count": 307200, "f32_offset": 0, "q_offset": 0, "scale_offset": 307200}, {"name": "onnx::MatMul_1571", "shape": [320, 320], "count": 102400, "f32_offset": 1228800, "q_offset": 326400, "scale_offset": 428800}, {"name": "onnx::MatMul_1572", "shape": [320, 1728], "count": 552960, "f32_offset": 1638400, "q_offset": 435200, "scale_offset": 988160}, {"name": "onnx::MatMul_1573", "shape": [864, 320], "count": 276480, "f32_offset": 3850240, "q_offset": 1022720, "scale_offset": 1299200}, {"name": "onnx::MatMul_1574", "shape": [320, 960], "count": 307200, "f32_offset": 4956160, "q_offset": 1316480, "scale_offset": 1623680}, {"name": "onnx::MatMul_1591", "shape": [320, 320], "count": 102400, "f32_offset": 6184960, "q_offset": 1642880, "scale_offset": 1745280}, {"name": "onnx::MatMul_1592", "shape": [320, 1728], "count": 552960, "f32_offset": 6594560, "q_offset": 1751680, "scale_offset": 2304640}, {"name": "onnx::MatMul_1593", "shape": [864, 320], "count": 276480, "f32_offset": 8806400, "q_offset": 2339200, "scale_offset": 2615680}, {"name": "onnx::MatMul_1594", "shape": [320, 960], "count": 307200, "f32_offset": 9912320, "q_offset": 2632960, "scale_offset": 2940160}, {"name": "onnx::MatMul_1611", "shape": [320, 320], "count": 102400, "f32_offset": 11141120, "q_offset": 2959360, "scale_offset": 3061760}, {"name": "onnx::MatMul_1612", "shape": [320, 1728], "count": 552960, "f32_offset": 11550720, "q_offset": 3068160, "scale_offset": 3621120}, {"name": "onnx::MatMul_1613", "shape": [864, 320], "count": 276480, "f32_offset": 13762560, "q_offset": 3655680, "scale_offset": 3932160}, {"name": "onnx::MatMul_1614", "shape": [320, 960], "count": 307200, "f32_offset": 14868480, "q_offset": 3949440, "scale_offset": 4256640}, {"name": "onnx::MatMul_1631", "shape": [320, 320], "count": 102400, "f32_offset": 16097280, "q_offset": 4275840, "scale_offset": 4378240}, {"name": "onnx::MatMul_1632", "shape": [320, 1728], "count": 552960, "f32_offset": 16506880, "q_offset": 4384640, "scale_offset": 4937600}, {"name": "onnx::MatMul_1633", "shape": [864, 320], "count": 276480, "f32_offset": 18718720, "q_offset": 4972160, "scale_offset": 5248640}, {"name": "onnx::MatMul_1634", "shape": [320, 960], "count": 307200, "f32_offset": 19824640, "q_offset": 5265920, "scale_offset": 5573120}, {"name": "onnx::MatMul_1651", "shape": [320, 320], "count": 102400, "f32_offset": 21053440, "q_offset": 5592320, "scale_offset": 5694720}, {"name": "onnx::MatMul_1652", "shape": [320, 1728], "count": 552960, "f32_offset": 21463040, "q_offset": 5701120, "scale_offset": 6254080}, {"name": "onnx::MatMul_1653", "shape": [864, 320], "count": 276480, "f32_offset": 23674880, "q_offset": 6288640, "scale_offset": 6565120}, {"name": "onnx::MatMul_1654", "shape": [320, 960], "count": 307200, "f32_offset": 24780800, "q_offset": 6582400, "scale_offset": 6889600}, {"name": "onnx::MatMul_1671", "shape": [320, 320], "count": 102400, "f32_offset": 26009600, "q_offset": 6908800, "scale_offset": 7011200}, {"name": "onnx::MatMul_1672", "shape": [320, 1728], "count": 552960, "f32_offset": 26419200, "q_offset": 7017600, "scale_offset": 7570560}, {"name": "onnx::MatMul_1673", "shape": [864, 320], "count": 276480, "f32_offset": 28631040, "q_offset": 7605120, "scale_offset": 7881600}, {"name": "onnx::MatMul_1674", "shape": [320, 960], "count": 307200, "f32_offset": 29736960, "q_offset": 7898880, "scale_offset": 8206080}, {"name": "onnx::MatMul_1691", "shape": [320, 320], "count": 102400, "f32_offset": 30965760, "q_offset": 8225280, "scale_offset": 8327680}, {"name": "onnx::MatMul_1692", "shape": [320, 1728], "count": 552960, "f32_offset": 31375360, "q_offset": 8334080, "scale_offset": 8887040}, {"name": "onnx::MatMul_1693", "shape": [864, 320], "count": 276480, "f32_offset": 33587200, "q_offset": 8921600, "scale_offset": 9198080}, {"name": "onnx::MatMul_1694", "shape": [320, 960], "count": 307200, "f32_offset": 34693120, "q_offset": 9215360, "scale_offset": 9522560}, {"name": "onnx::MatMul_1711", "shape": [320, 320], "count": 102400, "f32_offset": 35921920, "q_offset": 9541760, "scale_offset": 9644160}, {"name": "onnx::MatMul_1712", "shape": [320, 1728], "count": 552960, "f32_offset": 36331520, "q_offset": 9650560, "scale_offset": 10203520}, {"name": "onnx::MatMul_1713", "shape": [864, 320], "count": 276480, "f32_offset": 38543360, "q_offset": 10238080, "scale_offset": 10514560}, {"name": "onnx::MatMul_1714", "shape": [320, 8192], "count": 2621440, "f32_offset": 39649280, "q_offset": 10531840, "scale_offset": 13153280}]}
v2/web/weights.q8 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b74d19232900fda1c0e1cdcd1113a4825cc05874805796e138edf778b8053dde
3
  size 13317120
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:38496443cb6e543fa4040fb7db26924dc417ddf058aab71eec0a6e2d62675dd9
3
  size 13317120