Upload task output zeus-ld-intercode-1780895708

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,209 @@
+---
+base_model: None
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/cache/models/Jordansky--augmented-f560e4e6ee71e78d
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": null,
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 512,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "o_proj",
+    "down_proj",
+    "gate_proj",
+    "v_proj",
+    "k_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b4be5b24b8812ddaab4b195a33f1a2aebc96f26e57e88f0bd80c3cb3296acb6
+size 1057033224

added_tokens.json

@@ -0,0 +1,28 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja

@@ -0,0 +1,61 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

loss.txt

@@ -0,0 +1 @@
+1029,no_eval

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4
+size 11422654

tokenizer_config.json

@@ -0,0 +1,239 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 1010000,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

trainer_state.json

@@ -0,0 +1,1879 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.4059171597633136,
+  "eval_steps": 500,
+  "global_step": 1029,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0019723865877712033,
+      "grad_norm": 12.696123123168945,
+      "learning_rate": 6.748346456692913e-07,
+      "loss": 1.357,
+      "mean_token_accuracy": 0.7549497842788696,
+      "num_tokens": 180021.0,
+      "step": 5
+    },
+    {
+      "epoch": 0.0039447731755424065,
+      "grad_norm": 8.591487884521484,
+      "learning_rate": 1.5183779527559055e-06,
+      "loss": 1.2424,
+      "mean_token_accuracy": 0.772154700756073,
+      "num_tokens": 368724.0,
+      "step": 10
+    },
+    {
+      "epoch": 0.005917159763313609,
+      "grad_norm": 4.445512771606445,
+      "learning_rate": 2.36192125984252e-06,
+      "loss": 1.1188,
+      "mean_token_accuracy": 0.775861632823944,
+      "num_tokens": 553092.0,
+      "step": 15
+    },
+    {
+      "epoch": 0.007889546351084813,
+      "grad_norm": 2.918858051300049,
+      "learning_rate": 3.2054645669291342e-06,
+      "loss": 0.9275,
+      "mean_token_accuracy": 0.7919561505317688,
+      "num_tokens": 737811.0,
+      "step": 20
+    },
+    {
+      "epoch": 0.009861932938856016,
+      "grad_norm": 2.3222744464874268,
+      "learning_rate": 4.049007874015748e-06,
+      "loss": 0.7511,
+      "mean_token_accuracy": 0.8142940044403076,
+      "num_tokens": 921059.0,
+      "step": 25
+    },
+    {
+      "epoch": 0.011834319526627219,
+      "grad_norm": 2.116729259490967,
+      "learning_rate": 4.8925511811023625e-06,
+      "loss": 0.5497,
+      "mean_token_accuracy": 0.8610829710960388,
+      "num_tokens": 1104899.0,
+      "step": 30
+    },
+    {
+      "epoch": 0.013806706114398421,
+      "grad_norm": 1.6199977397918701,
+      "learning_rate": 5.736094488188977e-06,
+      "loss": 0.3611,
+      "mean_token_accuracy": 0.9114585757255554,
+      "num_tokens": 1292329.0,
+      "step": 35
+    },
+    {
+      "epoch": 0.015779092702169626,
+      "grad_norm": 1.066279649734497,
+      "learning_rate": 6.579637795275591e-06,
+      "loss": 0.2119,
+      "mean_token_accuracy": 0.947631013393402,
+      "num_tokens": 1481360.0,
+      "step": 40
+    },
+    {
+      "epoch": 0.01775147928994083,
+      "grad_norm": 0.9729903936386108,
+      "learning_rate": 7.4231811023622055e-06,
+      "loss": 0.1271,
+      "mean_token_accuracy": 0.969427740573883,
+      "num_tokens": 1665126.0,
+      "step": 45
+    },
+    {
+      "epoch": 0.01972386587771203,
+      "grad_norm": 0.3553341329097748,
+      "learning_rate": 8.266724409448819e-06,
+      "loss": 0.0621,
+      "mean_token_accuracy": 0.982380473613739,
+      "num_tokens": 1847070.0,
+      "step": 50
+    },
+    {
+      "epoch": 0.021696252465483234,
+      "grad_norm": 0.2363748699426651,
+      "learning_rate": 9.110267716535434e-06,
+      "loss": 0.0326,
+      "mean_token_accuracy": 0.9890782356262207,
+      "num_tokens": 2037437.0,
+      "step": 55
+    },
+    {
+      "epoch": 0.023668639053254437,
+      "grad_norm": 0.2311471402645111,
+      "learning_rate": 9.953811023622048e-06,
+      "loss": 0.0478,
+      "mean_token_accuracy": 0.9853112936019898,
+      "num_tokens": 2222734.0,
+      "step": 60
+    },
+    {
+      "epoch": 0.02564102564102564,
+      "grad_norm": 0.24362501502037048,
+      "learning_rate": 1.0797354330708661e-05,
+      "loss": 0.0276,
+      "mean_token_accuracy": 0.9898479461669922,
+      "num_tokens": 2407969.0,
+      "step": 65
+    },
+    {
+      "epoch": 0.027613412228796843,
+      "grad_norm": 0.15771926939487457,
+      "learning_rate": 1.1640897637795276e-05,
+      "loss": 0.0481,
+      "mean_token_accuracy": 0.9850168347358703,
+      "num_tokens": 2595011.0,
+      "step": 70
+    },
+    {
+      "epoch": 0.029585798816568046,
+      "grad_norm": 0.20371706783771515,
+      "learning_rate": 1.248444094488189e-05,
+      "loss": 0.0249,
+      "mean_token_accuracy": 0.9908480167388916,
+      "num_tokens": 2785644.0,
+      "step": 75
+    },
+    {
+      "epoch": 0.03155818540433925,
+      "grad_norm": 0.2078867405653,
+      "learning_rate": 1.3327984251968505e-05,
+      "loss": 0.0347,
+      "mean_token_accuracy": 0.9884193658828735,
+      "num_tokens": 2970750.0,
+      "step": 80
+    },
+    {
+      "epoch": 0.03353057199211045,
+      "grad_norm": 0.2615292966365814,
+      "learning_rate": 1.417152755905512e-05,
+      "loss": 0.0213,
+      "mean_token_accuracy": 0.9919293880462646,
+      "num_tokens": 3158759.0,
+      "step": 85
+    },
+    {
+      "epoch": 0.03550295857988166,
+      "grad_norm": 0.19127176702022552,
+      "learning_rate": 1.5015070866141732e-05,
+      "loss": 0.0262,
+      "mean_token_accuracy": 0.9905860424041748,
+      "num_tokens": 3343326.0,
+      "step": 90
+    },
+    {
+      "epoch": 0.03747534516765286,
+      "grad_norm": 0.27283141016960144,
+      "learning_rate": 1.5858614173228347e-05,
+      "loss": 0.0199,
+      "mean_token_accuracy": 0.9921495199203492,
+      "num_tokens": 3530057.0,
+      "step": 95
+    },
+    {
+      "epoch": 0.03944773175542406,
+      "grad_norm": 0.15445998311042786,
+      "learning_rate": 1.670215748031496e-05,
+      "loss": 0.0186,
+      "mean_token_accuracy": 0.9925162434577942,
+      "num_tokens": 3711503.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.04142011834319527,
+      "grad_norm": 0.14937782287597656,
+      "learning_rate": 1.7545700787401574e-05,
+      "loss": 0.021,
+      "mean_token_accuracy": 0.9921832203865051,
+      "num_tokens": 3899179.0,
+      "step": 105
+    },
+    {
+      "epoch": 0.04339250493096647,
+      "grad_norm": 0.1225525289773941,
+      "learning_rate": 1.838924409448819e-05,
+      "loss": 0.0201,
+      "mean_token_accuracy": 0.9920901179313659,
+      "num_tokens": 4086923.0,
+      "step": 110
+    },
+    {
+      "epoch": 0.045364891518737675,
+      "grad_norm": 0.4743281602859497,
+      "learning_rate": 1.9232787401574804e-05,
+      "loss": 0.0168,
+      "mean_token_accuracy": 0.993110179901123,
+      "num_tokens": 4274776.0,
+      "step": 115
+    },
+    {
+      "epoch": 0.047337278106508875,
+      "grad_norm": 0.10754459351301193,
+      "learning_rate": 2.007633070866142e-05,
+      "loss": 0.0182,
+      "mean_token_accuracy": 0.9926936984062195,
+      "num_tokens": 4458441.0,
+      "step": 120
+    },
+    {
+      "epoch": 0.04930966469428008,
+      "grad_norm": 0.31404948234558105,
+      "learning_rate": 2.0919874015748035e-05,
+      "loss": 0.0186,
+      "mean_token_accuracy": 0.9924424409866333,
+      "num_tokens": 4644925.0,
+      "step": 125
+    },
+    {
+      "epoch": 0.05128205128205128,
+      "grad_norm": 0.09206202626228333,
+      "learning_rate": 2.1763417322834647e-05,
+      "loss": 0.0166,
+      "mean_token_accuracy": 0.993120801448822,
+      "num_tokens": 4828090.0,
+      "step": 130
+    },
+    {
+      "epoch": 0.05325443786982249,
+      "grad_norm": 0.10214517265558243,
+      "learning_rate": 2.260696062992126e-05,
+      "loss": 0.0167,
+      "mean_token_accuracy": 0.9927790641784668,
+      "num_tokens": 5013289.0,
+      "step": 135
+    },
+    {
+      "epoch": 0.055226824457593686,
+      "grad_norm": 0.07818103581666946,
+      "learning_rate": 2.3450503937007877e-05,
+      "loss": 0.0151,
+      "mean_token_accuracy": 0.9935371398925781,
+      "num_tokens": 5202360.0,
+      "step": 140
+    },
+    {
+      "epoch": 0.05719921104536489,
+      "grad_norm": 0.09127297252416611,
+      "learning_rate": 2.429404724409449e-05,
+      "loss": 0.0152,
+      "mean_token_accuracy": 0.9934893608093261,
+      "num_tokens": 5394495.0,
+      "step": 145
+    },
+    {
+      "epoch": 0.05917159763313609,
+      "grad_norm": 0.11136125028133392,
+      "learning_rate": 2.51375905511811e-05,
+      "loss": 0.0145,
+      "mean_token_accuracy": 0.993546438217163,
+      "num_tokens": 5581147.0,
+      "step": 150
+    },
+    {
+      "epoch": 0.0611439842209073,
+      "grad_norm": 0.13068991899490356,
+      "learning_rate": 2.598113385826772e-05,
+      "loss": 0.0141,
+      "mean_token_accuracy": 0.9935914158821106,
+      "num_tokens": 5764996.0,
+      "step": 155
+    },
+    {
+      "epoch": 0.0631163708086785,
+      "grad_norm": 0.16084076464176178,
+      "learning_rate": 2.682467716535433e-05,
+      "loss": 0.0163,
+      "mean_token_accuracy": 0.9931087493896484,
+      "num_tokens": 5949665.0,
+      "step": 160
+    },
+    {
+      "epoch": 0.0650887573964497,
+      "grad_norm": 0.2447124868631363,
+      "learning_rate": 2.7668220472440946e-05,
+      "loss": 0.0181,
+      "mean_token_accuracy": 0.9928769588470459,
+      "num_tokens": 6134903.0,
+      "step": 165
+    },
+    {
+      "epoch": 0.0670611439842209,
+      "grad_norm": 0.4701608419418335,
+      "learning_rate": 2.851176377952756e-05,
+      "loss": 0.0146,
+      "mean_token_accuracy": 0.9936342835426331,
+      "num_tokens": 6322514.0,
+      "step": 170
+    },
+    {
+      "epoch": 0.06903353057199212,
+      "grad_norm": 0.11707167327404022,
+      "learning_rate": 2.9355307086614176e-05,
+      "loss": 0.0145,
+      "mean_token_accuracy": 0.9934436202049255,
+      "num_tokens": 6501636.0,
+      "step": 175
+    },
+    {
+      "epoch": 0.07100591715976332,
+      "grad_norm": 0.0810714140534401,
+      "learning_rate": 3.0198850393700788e-05,
+      "loss": 0.0147,
+      "mean_token_accuracy": 0.9936310172080993,
+      "num_tokens": 6683353.0,
+      "step": 180
+    },
+    {
+      "epoch": 0.07297830374753451,
+      "grad_norm": 0.14196588099002838,
+      "learning_rate": 3.1042393700787403e-05,
+      "loss": 0.0137,
+      "mean_token_accuracy": 0.9938475370407105,
+      "num_tokens": 6869907.0,
+      "step": 185
+    },
+    {
+      "epoch": 0.07495069033530571,
+      "grad_norm": 0.058894768357276917,
+      "learning_rate": 3.188593700787402e-05,
+      "loss": 0.0147,
+      "mean_token_accuracy": 0.993359363079071,
+      "num_tokens": 7053971.0,
+      "step": 190
+    },
+    {
+      "epoch": 0.07692307692307693,
+      "grad_norm": 0.08509636670351028,
+      "learning_rate": 3.2729480314960634e-05,
+      "loss": 0.0139,
+      "mean_token_accuracy": 0.9935320854187012,
+      "num_tokens": 7236944.0,
+      "step": 195
+    },
+    {
+      "epoch": 0.07889546351084813,
+      "grad_norm": 0.11753793805837631,
+      "learning_rate": 3.357302362204725e-05,
+      "loss": 0.0138,
+      "mean_token_accuracy": 0.9938223838806153,
+      "num_tokens": 7420563.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.08086785009861933,
+      "grad_norm": 0.07417895644903183,
+      "learning_rate": 3.4416566929133864e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.993987250328064,
+      "num_tokens": 7605745.0,
+      "step": 205
+    },
+    {
+      "epoch": 0.08284023668639054,
+      "grad_norm": 0.07740958780050278,
+      "learning_rate": 3.526011023622047e-05,
+      "loss": 0.0135,
+      "mean_token_accuracy": 0.9936949133872985,
+      "num_tokens": 7791888.0,
+      "step": 210
+    },
+    {
+      "epoch": 0.08481262327416174,
+      "grad_norm": 0.11897961050271988,
+      "learning_rate": 3.610365354330709e-05,
+      "loss": 0.0136,
+      "mean_token_accuracy": 0.9938376903533935,
+      "num_tokens": 7975433.0,
+      "step": 215
+    },
+    {
+      "epoch": 0.08678500986193294,
+      "grad_norm": 0.05374155938625336,
+      "learning_rate": 3.69471968503937e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9941465139389039,
+      "num_tokens": 8160428.0,
+      "step": 220
+    },
+    {
+      "epoch": 0.08875739644970414,
+      "grad_norm": 0.06223868951201439,
+      "learning_rate": 3.779074015748032e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9940422892570495,
+      "num_tokens": 8350378.0,
+      "step": 225
+    },
+    {
+      "epoch": 0.09072978303747535,
+      "grad_norm": 0.12495646625757217,
+      "learning_rate": 3.8634283464566926e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9938552856445313,
+      "num_tokens": 8536865.0,
+      "step": 230
+    },
+    {
+      "epoch": 0.09270216962524655,
+      "grad_norm": 0.0632530227303505,
+      "learning_rate": 3.947782677165355e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.9934528350830079,
+      "num_tokens": 8719053.0,
+      "step": 235
+    },
+    {
+      "epoch": 0.09467455621301775,
+      "grad_norm": 0.09418715536594391,
+      "learning_rate": 4.032137007874016e-05,
+      "loss": 0.0142,
+      "mean_token_accuracy": 0.9937668204307556,
+      "num_tokens": 8902707.0,
+      "step": 240
+    },
+    {
+      "epoch": 0.09664694280078895,
+      "grad_norm": 0.0670342892408371,
+      "learning_rate": 4.116491338582677e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9940184712409973,
+      "num_tokens": 9092697.0,
+      "step": 245
+    },
+    {
+      "epoch": 0.09861932938856016,
+      "grad_norm": 0.0770217701792717,
+      "learning_rate": 4.200845669291339e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9939847946166992,
+      "num_tokens": 9279993.0,
+      "step": 250
+    },
+    {
+      "epoch": 0.10059171597633136,
+      "grad_norm": 0.06905656307935715,
+      "learning_rate": 4.2852e-05,
+      "loss": 0.0135,
+      "mean_token_accuracy": 0.9939971566200256,
+      "num_tokens": 9466630.0,
+      "step": 255
+    },
+    {
+      "epoch": 0.10256410256410256,
+      "grad_norm": 0.1345818042755127,
+      "learning_rate": 4.285191452506468e-05,
+      "loss": 0.0139,
+      "mean_token_accuracy": 0.9937587738037109,
+      "num_tokens": 9649918.0,
+      "step": 260
+    },
+    {
+      "epoch": 0.10453648915187377,
+      "grad_norm": 0.047634709626436234,
+      "learning_rate": 4.2851658101168e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9939332246780396,
+      "num_tokens": 9832310.0,
+      "step": 265
+    },
+    {
+      "epoch": 0.10650887573964497,
+      "grad_norm": 0.03853986784815788,
+      "learning_rate": 4.2851230731037846e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9940977811813354,
+      "num_tokens": 10021113.0,
+      "step": 270
+    },
+    {
+      "epoch": 0.10848126232741617,
+      "grad_norm": 0.09243371337652206,
+      "learning_rate": 4.285063241922065e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9940739035606384,
+      "num_tokens": 10206684.0,
+      "step": 275
+    },
+    {
+      "epoch": 0.11045364891518737,
+      "grad_norm": 0.18149180710315704,
+      "learning_rate": 4.284986317208133e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.9938218593597412,
+      "num_tokens": 10392345.0,
+      "step": 280
+    },
+    {
+      "epoch": 0.11242603550295859,
+      "grad_norm": 0.33121660351753235,
+      "learning_rate": 4.284892299780329e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9939525723457336,
+      "num_tokens": 10582134.0,
+      "step": 285
+    },
+    {
+      "epoch": 0.11439842209072978,
+      "grad_norm": 0.06855133920907974,
+      "learning_rate": 4.284781190638823e-05,
+      "loss": 0.0142,
+      "mean_token_accuracy": 0.9938366293907166,
+      "num_tokens": 10764689.0,
+      "step": 290
+    },
+    {
+      "epoch": 0.11637080867850098,
+      "grad_norm": 0.2645736038684845,
+      "learning_rate": 4.2846529909656116e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9939942121505737,
+      "num_tokens": 10954315.0,
+      "step": 295
+    },
+    {
+      "epoch": 0.11834319526627218,
+      "grad_norm": 1.9121037721633911,
+      "learning_rate": 4.2845077021245034e-05,
+      "loss": 0.0158,
+      "mean_token_accuracy": 0.9931493401527405,
+      "num_tokens": 11135269.0,
+      "step": 300
+    },
+    {
+      "epoch": 0.1203155818540434,
+      "grad_norm": 1.1927855014801025,
+      "learning_rate": 4.284345325661103e-05,
+      "loss": 0.0225,
+      "mean_token_accuracy": 0.9921814203262329,
+      "num_tokens": 11316327.0,
+      "step": 305
+    },
+    {
+      "epoch": 0.1222879684418146,
+      "grad_norm": 1.997202754020691,
+      "learning_rate": 4.2841658633028e-05,
+      "loss": 0.0203,
+      "mean_token_accuracy": 0.992283034324646,
+      "num_tokens": 11502512.0,
+      "step": 310
+    },
+    {
+      "epoch": 0.1242603550295858,
+      "grad_norm": 0.1343405544757843,
+      "learning_rate": 4.28396931695874e-05,
+      "loss": 0.0147,
+      "mean_token_accuracy": 0.9935072779655456,
+      "num_tokens": 11685247.0,
+      "step": 315
+    },
+    {
+      "epoch": 0.126232741617357,
+      "grad_norm": 0.24693335592746735,
+      "learning_rate": 4.283755688719814e-05,
+      "loss": 0.0144,
+      "mean_token_accuracy": 0.9937364459037781,
+      "num_tokens": 11871657.0,
+      "step": 320
+    },
+    {
+      "epoch": 0.1282051282051282,
+      "grad_norm": 0.6627845168113708,
+      "learning_rate": 4.283524980858635e-05,
+      "loss": 0.0195,
+      "mean_token_accuracy": 0.9922019481658936,
+      "num_tokens": 12059485.0,
+      "step": 325
+    },
+    {
+      "epoch": 0.1301775147928994,
+      "grad_norm": 0.10230821371078491,
+      "learning_rate": 4.283277195829508e-05,
+      "loss": 0.0147,
+      "mean_token_accuracy": 0.9933422088623047,
+      "num_tokens": 12246508.0,
+      "step": 330
+    },
+    {
+      "epoch": 0.13214990138067062,
+      "grad_norm": 0.06060272827744484,
+      "learning_rate": 4.28301233626841e-05,
+      "loss": 0.0177,
+      "mean_token_accuracy": 0.9932606101036072,
+      "num_tokens": 12431003.0,
+      "step": 335
+    },
+    {
+      "epoch": 0.1341222879684418,
+      "grad_norm": 0.07024521380662918,
+      "learning_rate": 4.282730404992958e-05,
+      "loss": 0.0153,
+      "mean_token_accuracy": 0.9935808539390564,
+      "num_tokens": 12613759.0,
+      "step": 340
+    },
+    {
+      "epoch": 0.13609467455621302,
+      "grad_norm": 0.043302495032548904,
+      "learning_rate": 4.28243140500238e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9942602157592774,
+      "num_tokens": 12799120.0,
+      "step": 345
+    },
+    {
+      "epoch": 0.13806706114398423,
+      "grad_norm": 0.03011777251958847,
+      "learning_rate": 4.2821153394774885e-05,
+      "loss": 0.0139,
+      "mean_token_accuracy": 0.9939547777175903,
+      "num_tokens": 12980783.0,
+      "step": 350
+    },
+    {
+      "epoch": 0.14003944773175542,
+      "grad_norm": 0.03735549375414848,
+      "learning_rate": 4.2817822117806354e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9940328359603882,
+      "num_tokens": 13164781.0,
+      "step": 355
+    },
+    {
+      "epoch": 0.14201183431952663,
+      "grad_norm": 0.10837282240390778,
+      "learning_rate": 4.281432025455686e-05,
+      "loss": 0.015,
+      "mean_token_accuracy": 0.9938132286071777,
+      "num_tokens": 13353794.0,
+      "step": 360
+    },
+    {
+      "epoch": 0.14398422090729784,
+      "grad_norm": 0.07957315444946289,
+      "learning_rate": 4.2810647842279786e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.993849265575409,
+      "num_tokens": 13533412.0,
+      "step": 365
+    },
+    {
+      "epoch": 0.14595660749506903,
+      "grad_norm": 0.04022190719842911,
+      "learning_rate": 4.2806804920042826e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.994051730632782,
+      "num_tokens": 13718680.0,
+      "step": 370
+    },
+    {
+      "epoch": 0.14792899408284024,
+      "grad_norm": 0.028754152357578278,
+      "learning_rate": 4.2802791528727586e-05,
+      "loss": 0.0137,
+      "mean_token_accuracy": 0.9939261198043823,
+      "num_tokens": 13904671.0,
+      "step": 375
+    },
+    {
+      "epoch": 0.14990138067061143,
+      "grad_norm": 0.07314479351043701,
+      "learning_rate": 4.279860771102917e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9940339088439941,
+      "num_tokens": 14089839.0,
+      "step": 380
+    },
+    {
+      "epoch": 0.15187376725838264,
+      "grad_norm": 0.028541339561343193,
+      "learning_rate": 4.279425351145567e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9940938830375672,
+      "num_tokens": 14277196.0,
+      "step": 385
+    },
+    {
+      "epoch": 0.15384615384615385,
+      "grad_norm": 0.07783432304859161,
+      "learning_rate": 4.278972897632776e-05,
+      "loss": 0.0138,
+      "mean_token_accuracy": 0.9940650939941407,
+      "num_tokens": 14466798.0,
+      "step": 390
+    },
+    {
+      "epoch": 0.15581854043392504,
+      "grad_norm": 0.02617865428328514,
+      "learning_rate": 4.278503415377816e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9941792964935303,
+      "num_tokens": 14652136.0,
+      "step": 395
+    },
+    {
+      "epoch": 0.15779092702169625,
+      "grad_norm": 0.035784874111413956,
+      "learning_rate": 4.278016909375112e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.994103193283081,
+      "num_tokens": 14838618.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.15976331360946747,
+      "grad_norm": 0.02981606312096119,
+      "learning_rate": 4.277513384800191e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9940651178359985,
+      "num_tokens": 15022996.0,
+      "step": 405
+    },
+    {
+      "epoch": 0.16173570019723865,
+      "grad_norm": 0.01930910535156727,
+      "learning_rate": 4.276992847009628e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9941300868988037,
+      "num_tokens": 15202971.0,
+      "step": 410
+    },
+    {
+      "epoch": 0.16370808678500987,
+      "grad_norm": 0.02543552592396736,
+      "learning_rate": 4.276455301540986e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9941654682159424,
+      "num_tokens": 15384455.0,
+      "step": 415
+    },
+    {
+      "epoch": 0.16568047337278108,
+      "grad_norm": 0.02898615412414074,
+      "learning_rate": 4.275900754112757e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9940505146980285,
+      "num_tokens": 15571156.0,
+      "step": 420
+    },
+    {
+      "epoch": 0.16765285996055226,
+      "grad_norm": 0.025841962546110153,
+      "learning_rate": 4.275329210624306e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9941076755523681,
+      "num_tokens": 15752950.0,
+      "step": 425
+    },
+    {
+      "epoch": 0.16962524654832348,
+      "grad_norm": 0.02423620969057083,
+      "learning_rate": 4.2747406771557995e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.994234275817871,
+      "num_tokens": 15940566.0,
+      "step": 430
+    },
+    {
+      "epoch": 0.17159763313609466,
+      "grad_norm": 0.054586950689554214,
+      "learning_rate": 4.274135159968152e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9940789937973022,
+      "num_tokens": 16123322.0,
+      "step": 435
+    },
+    {
+      "epoch": 0.17357001972386588,
+      "grad_norm": 0.03049752116203308,
+      "learning_rate": 4.27351266550295e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9942823052406311,
+      "num_tokens": 16312606.0,
+      "step": 440
+    },
+    {
+      "epoch": 0.1755424063116371,
+      "grad_norm": 0.03726842626929283,
+      "learning_rate": 4.272873200382387e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9941863536834716,
+      "num_tokens": 16497900.0,
+      "step": 445
+    },
+    {
+      "epoch": 0.17751479289940827,
+      "grad_norm": 0.02811609022319317,
+      "learning_rate": 4.272216771409193e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9940958976745605,
+      "num_tokens": 16684994.0,
+      "step": 450
+    },
+    {
+      "epoch": 0.1794871794871795,
+      "grad_norm": 0.0249259602278471,
+      "learning_rate": 4.271543385566564e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9941407680511475,
+      "num_tokens": 16870817.0,
+      "step": 455
+    },
+    {
+      "epoch": 0.1814595660749507,
+      "grad_norm": 0.027562299743294716,
+      "learning_rate": 4.2708530500180844e-05,
+      "loss": 0.0141,
+      "mean_token_accuracy": 0.9937734007835388,
+      "num_tokens": 17056130.0,
+      "step": 460
+    },
+    {
+      "epoch": 0.1834319526627219,
+      "grad_norm": 0.032294247299432755,
+      "learning_rate": 4.2701457721076526e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.993969738483429,
+      "num_tokens": 17242933.0,
+      "step": 465
+    },
+    {
+      "epoch": 0.1854043392504931,
+      "grad_norm": 0.02657519280910492,
+      "learning_rate": 4.269421559359402e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9942268848419189,
+      "num_tokens": 17431102.0,
+      "step": 470
+    },
+    {
+      "epoch": 0.1873767258382643,
+      "grad_norm": 0.03130607306957245,
+      "learning_rate": 4.268680419477621e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9940387964248657,
+      "num_tokens": 17617262.0,
+      "step": 475
+    },
+    {
+      "epoch": 0.1893491124260355,
+      "grad_norm": 0.025990553200244904,
+      "learning_rate": 4.267922360346673e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.993549394607544,
+      "num_tokens": 17798021.0,
+      "step": 480
+    },
+    {
+      "epoch": 0.1913214990138067,
+      "grad_norm": 0.02206920087337494,
+      "learning_rate": 4.2671473900309116e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9942124843597412,
+      "num_tokens": 17981812.0,
+      "step": 485
+    },
+    {
+      "epoch": 0.1932938856015779,
+      "grad_norm": 0.019280826672911644,
+      "learning_rate": 4.2663555167745916e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9941383838653565,
+      "num_tokens": 18169369.0,
+      "step": 490
+    },
+    {
+      "epoch": 0.1952662721893491,
+      "grad_norm": 0.033744603395462036,
+      "learning_rate": 4.265546749001785e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9941118478775024,
+      "num_tokens": 18354700.0,
+      "step": 495
+    },
+    {
+      "epoch": 0.19723865877712032,
+      "grad_norm": 0.07425432652235031,
+      "learning_rate": 4.264721095316292e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9939664959907532,
+      "num_tokens": 18539201.0,
+      "step": 500
+    },
+    {
+      "epoch": 0.1992110453648915,
+      "grad_norm": 0.045775558799505234,
+      "learning_rate": 4.2638785645015455e-05,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.9941562175750732,
+      "num_tokens": 18724128.0,
+      "step": 505
+    },
+    {
+      "epoch": 0.20118343195266272,
+      "grad_norm": 0.020204922184348106,
+      "learning_rate": 4.263019165520522e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9938268899917603,
+      "num_tokens": 18910572.0,
+      "step": 510
+    },
+    {
+      "epoch": 0.20315581854043394,
+      "grad_norm": 0.024817291647195816,
+      "learning_rate": 4.262142907515642e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9938831329345703,
+      "num_tokens": 19094735.0,
+      "step": 515
+    },
+    {
+      "epoch": 0.20512820512820512,
+      "grad_norm": 0.027676964178681374,
+      "learning_rate": 4.261249799808676e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9942035794258117,
+      "num_tokens": 19281983.0,
+      "step": 520
+    },
+    {
+      "epoch": 0.20710059171597633,
+      "grad_norm": 0.03616034984588623,
+      "learning_rate": 4.2603398519006456e-05,
+      "loss": 0.0124,
+      "mean_token_accuracy": 0.9943036675453186,
+      "num_tokens": 19467872.0,
+      "step": 525
+    },
+    {
+      "epoch": 0.20907297830374755,
+      "grad_norm": 0.01999801956117153,
+      "learning_rate": 4.2594130734717186e-05,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.9941059827804566,
+      "num_tokens": 19654634.0,
+      "step": 530
+    },
+    {
+      "epoch": 0.21104536489151873,
+      "grad_norm": 0.026613615453243256,
+      "learning_rate": 4.258469474381111e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9941833734512329,
+      "num_tokens": 19840288.0,
+      "step": 535
+    },
+    {
+      "epoch": 0.21301775147928995,
+      "grad_norm": 0.027081554755568504,
+      "learning_rate": 4.257509064666977e-05,
+      "loss": 0.0123,
+      "mean_token_accuracy": 0.9943787336349488,
+      "num_tokens": 20026691.0,
+      "step": 540
+    },
+    {
+      "epoch": 0.21499013806706113,
+      "grad_norm": 0.04074582830071449,
+      "learning_rate": 4.256531854546308e-05,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.9939507722854615,
+      "num_tokens": 20208224.0,
+      "step": 545
+    },
+    {
+      "epoch": 0.21696252465483234,
+      "grad_norm": 0.021070433780550957,
+      "learning_rate": 4.2555378544148184e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9938449859619141,
+      "num_tokens": 20394106.0,
+      "step": 550
+    },
+    {
+      "epoch": 0.21893491124260356,
+      "grad_norm": 0.024537397548556328,
+      "learning_rate": 4.254527074846838e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9939658999443054,
+      "num_tokens": 20579070.0,
+      "step": 555
+    },
+    {
+      "epoch": 0.22090729783037474,
+      "grad_norm": 0.018611876294016838,
+      "learning_rate": 4.253499526595198e-05,
+      "loss": 0.0123,
+      "mean_token_accuracy": 0.994249427318573,
+      "num_tokens": 20767501.0,
+      "step": 560
+    },
+    {
+      "epoch": 0.22287968441814596,
+      "grad_norm": 0.026016313582658768,
+      "learning_rate": 4.25245522059112e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9940187215805054,
+      "num_tokens": 20951278.0,
+      "step": 565
+    },
+    {
+      "epoch": 0.22485207100591717,
+      "grad_norm": 0.022571122273802757,
+      "learning_rate": 4.251394167944095e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9939440011978149,
+      "num_tokens": 21140033.0,
+      "step": 570
+    },
+    {
+      "epoch": 0.22682445759368836,
+      "grad_norm": 0.03602802753448486,
+      "learning_rate": 4.250316379941768e-05,
+      "loss": 0.0124,
+      "mean_token_accuracy": 0.9941883087158203,
+      "num_tokens": 21328704.0,
+      "step": 575
+    },
+    {
+      "epoch": 0.22879684418145957,
+      "grad_norm": 0.018570076674222946,
+      "learning_rate": 4.249221868049818e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9939632892608643,
+      "num_tokens": 21510942.0,
+      "step": 580
+    },
+    {
+      "epoch": 0.23076923076923078,
+      "grad_norm": 0.02790108323097229,
+      "learning_rate": 4.2481106439118345e-05,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.9941455602645874,
+      "num_tokens": 21693866.0,
+      "step": 585
+    },
+    {
+      "epoch": 0.23274161735700197,
+      "grad_norm": 0.025027867406606674,
+      "learning_rate": 4.2469827193491936e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9940923094749451,
+      "num_tokens": 21879295.0,
+      "step": 590
+    },
+    {
+      "epoch": 0.23471400394477318,
+      "grad_norm": 0.025109363719820976,
+      "learning_rate": 4.2458381063609363e-05,
+      "loss": 0.0146,
+      "mean_token_accuracy": 0.9934637308120727,
+      "num_tokens": 22063809.0,
+      "step": 595
+    },
+    {
+      "epoch": 0.23668639053254437,
+      "grad_norm": 0.024112723767757416,
+      "learning_rate": 4.244676817123635e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9940780162811279,
+      "num_tokens": 22247806.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.23865877712031558,
+      "grad_norm": 0.02184889279305935,
+      "learning_rate": 4.243498863991267e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9940764546394348,
+      "num_tokens": 22431733.0,
+      "step": 605
+    },
+    {
+      "epoch": 0.2406311637080868,
+      "grad_norm": 0.018762197345495224,
+      "learning_rate": 4.242304259495084e-05,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.9940377354621888,
+      "num_tokens": 22621405.0,
+      "step": 610
+    },
+    {
+      "epoch": 0.24260355029585798,
+      "grad_norm": 0.03053358569741249,
+      "learning_rate": 4.2410930163434756e-05,
+      "loss": 0.0123,
+      "mean_token_accuracy": 0.9944116353988648,
+      "num_tokens": 22805682.0,
+      "step": 615
+    },
+    {
+      "epoch": 0.2445759368836292,
+      "grad_norm": 0.0349368192255497,
+      "learning_rate": 4.239865147421837e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9942061543464661,
+      "num_tokens": 22994102.0,
+      "step": 620
+    },
+    {
+      "epoch": 0.2465483234714004,
+      "grad_norm": 0.025624720379710197,
+      "learning_rate": 4.238620665792434e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9941583752632142,
+      "num_tokens": 23177702.0,
+      "step": 625
+    },
+    {
+      "epoch": 0.2485207100591716,
+      "grad_norm": 0.029787929728627205,
+      "learning_rate": 4.2373595846942535e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9941216349601746,
+      "num_tokens": 23359555.0,
+      "step": 630
+    },
+    {
+      "epoch": 0.2504930966469428,
+      "grad_norm": 0.02572852000594139,
+      "learning_rate": 4.236081917542878e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9942123293876648,
+      "num_tokens": 23544940.0,
+      "step": 635
+    },
+    {
+      "epoch": 0.252465483234714,
+      "grad_norm": 0.044698312878608704,
+      "learning_rate": 4.234787677930331e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9941087961196899,
+      "num_tokens": 23731096.0,
+      "step": 640
+    },
+    {
+      "epoch": 0.25443786982248523,
+      "grad_norm": 30.425779342651367,
+      "learning_rate": 4.2334768796249364e-05,
+      "loss": 0.0159,
+      "mean_token_accuracy": 0.9939328193664551,
+      "num_tokens": 23915652.0,
+      "step": 645
+    },
+    {
+      "epoch": 0.2564102564102564,
+      "grad_norm": 0.44328176975250244,
+      "learning_rate": 4.2321495365711744e-05,
+      "loss": 0.0243,
+      "mean_token_accuracy": 0.9924443960189819,
+      "num_tokens": 24093231.0,
+      "step": 650
+    },
+    {
+      "epoch": 0.2583826429980276,
+      "grad_norm": 0.04782794043421745,
+      "learning_rate": 4.230805662889528e-05,
+      "loss": 0.0143,
+      "mean_token_accuracy": 0.9938549757003784,
+      "num_tokens": 24278504.0,
+      "step": 655
+    },
+    {
+      "epoch": 0.2603550295857988,
+      "grad_norm": 0.13285517692565918,
+      "learning_rate": 4.229445272876338e-05,
+      "loss": 0.0136,
+      "mean_token_accuracy": 0.9938465833663941,
+      "num_tokens": 24460968.0,
+      "step": 660
+    },
+    {
+      "epoch": 0.26232741617357,
+      "grad_norm": 0.0991918072104454,
+      "learning_rate": 4.228068381003647e-05,
+      "loss": 0.0154,
+      "mean_token_accuracy": 0.9932482481002808,
+      "num_tokens": 24647103.0,
+      "step": 665
+    },
+    {
+      "epoch": 0.26429980276134124,
+      "grad_norm": 0.46482768654823303,
+      "learning_rate": 4.226675001919047e-05,
+      "loss": 0.0153,
+      "mean_token_accuracy": 0.9935938835144043,
+      "num_tokens": 24828744.0,
+      "step": 670
+    },
+    {
+      "epoch": 0.26627218934911245,
+      "grad_norm": 0.05104758217930794,
+      "learning_rate": 4.225265150445525e-05,
+      "loss": 0.0136,
+      "mean_token_accuracy": 0.9940480709075927,
+      "num_tokens": 25016751.0,
+      "step": 675
+    },
+    {
+      "epoch": 0.2682445759368836,
+      "grad_norm": 0.02835351601243019,
+      "learning_rate": 4.223838841581303e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9939886689186096,
+      "num_tokens": 25197789.0,
+      "step": 680
+    },
+    {
+      "epoch": 0.2702169625246548,
+      "grad_norm": 0.02049642615020275,
+      "learning_rate": 4.222396090499678e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9940541744232178,
+      "num_tokens": 25380186.0,
+      "step": 685
+    },
+    {
+      "epoch": 0.27218934911242604,
+      "grad_norm": 0.05688883364200592,
+      "learning_rate": 4.220936912548864e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9941863417625427,
+      "num_tokens": 25563728.0,
+      "step": 690
+    },
+    {
+      "epoch": 0.27416173570019725,
+      "grad_norm": 0.03041102923452854,
+      "learning_rate": 4.2194613232518255e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9938729286193848,
+      "num_tokens": 25747493.0,
+      "step": 695
+    },
+    {
+      "epoch": 0.27613412228796846,
+      "grad_norm": 0.022827785462141037,
+      "learning_rate": 4.217969338306114e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9939754962921142,
+      "num_tokens": 25933349.0,
+      "step": 700
+    },
+    {
+      "epoch": 0.2781065088757396,
+      "grad_norm": 0.02409464307129383,
+      "learning_rate": 4.2164609735837e-05,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.9940469980239868,
+      "num_tokens": 26116953.0,
+      "step": 705
+    },
+    {
+      "epoch": 0.28007889546351084,
+      "grad_norm": 0.021490557119250298,
+      "learning_rate": 4.214936245130805e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9941638350486756,
+      "num_tokens": 26301647.0,
+      "step": 710
+    },
+    {
+      "epoch": 0.28205128205128205,
+      "grad_norm": 0.02634543552994728,
+      "learning_rate": 4.2133951691677306e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9942031621932983,
+      "num_tokens": 26488583.0,
+      "step": 715
+    },
+    {
+      "epoch": 0.28402366863905326,
+      "grad_norm": 0.01704087294638157,
+      "learning_rate": 4.211837762088684e-05,
+      "loss": 0.0123,
+      "mean_token_accuracy": 0.9944246053695679,
+      "num_tokens": 26678077.0,
+      "step": 720
+    },
+    {
+      "epoch": 0.2859960552268245,
+      "grad_norm": 0.026759184896945953,
+      "learning_rate": 4.2102640404616094e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.9937499523162842,
+      "num_tokens": 26860728.0,
+      "step": 725
+    },
+    {
+      "epoch": 0.2879684418145957,
+      "grad_norm": 0.027147632092237473,
+      "learning_rate": 4.208674021028002e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9942026376724243,
+      "num_tokens": 27047732.0,
+      "step": 730
+    },
+    {
+      "epoch": 0.28994082840236685,
+      "grad_norm": 0.022147085517644882,
+      "learning_rate": 4.207067720702742e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9942293286323547,
+      "num_tokens": 27233802.0,
+      "step": 735
+    },
+    {
+      "epoch": 0.29191321499013806,
+      "grad_norm": 0.037467051297426224,
+      "learning_rate": 4.205445156573903e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9941612005233764,
+      "num_tokens": 27418734.0,
+      "step": 740
+    },
+    {
+      "epoch": 0.2938856015779093,
+      "grad_norm": 0.02817392349243164,
+      "learning_rate": 4.2038063459025794e-05,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.9941320419311523,
+      "num_tokens": 27602218.0,
+      "step": 745
+    },
+    {
+      "epoch": 0.2958579881656805,
+      "grad_norm": 0.02613888494670391,
+      "learning_rate": 4.202151306122695e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9943588614463806,
+      "num_tokens": 27787524.0,
+      "step": 750
+    },
+    {
+      "epoch": 0.2978303747534517,
+      "grad_norm": 0.023414531722664833,
+      "learning_rate": 4.200480054840826e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.994118595123291,
+      "num_tokens": 27972322.0,
+      "step": 755
+    },
+    {
+      "epoch": 0.29980276134122286,
+      "grad_norm": 0.03286295756697655,
+      "learning_rate": 4.198792609836004e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9941413998603821,
+      "num_tokens": 28155797.0,
+      "step": 760
+    },
+    {
+      "epoch": 0.30177514792899407,
+      "grad_norm": 0.027933163568377495,
+      "learning_rate": 4.1970889890595355e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9940989732742309,
+      "num_tokens": 28338176.0,
+      "step": 765
+    },
+    {
+      "epoch": 0.3037475345167653,
+      "grad_norm": 0.026723258197307587,
+      "learning_rate": 4.195369210634807e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9941996097564697,
+      "num_tokens": 28522574.0,
+      "step": 770
+    },
+    {
+      "epoch": 0.3057199211045365,
+      "grad_norm": 0.020453844219446182,
+      "learning_rate": 4.193633292857092e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9939847469329834,
+      "num_tokens": 28707610.0,
+      "step": 775
+    },
+    {
+      "epoch": 0.3076923076923077,
+      "grad_norm": 0.04157761111855507,
+      "learning_rate": 4.191881254193358e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.993830943107605,
+      "num_tokens": 28890795.0,
+      "step": 780
+    },
+    {
+      "epoch": 0.3096646942800789,
+      "grad_norm": 0.03104614093899727,
+      "learning_rate": 4.190113113282066e-05,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.9942401647567749,
+      "num_tokens": 29076723.0,
+      "step": 785
+    },
+    {
+      "epoch": 0.3116370808678501,
+      "grad_norm": 0.022480012848973274,
+      "learning_rate": 4.188328888932979e-05,
+      "loss": 0.0124,
+      "mean_token_accuracy": 0.9942922234535218,
+      "num_tokens": 29262650.0,
+      "step": 790
+    },
+    {
+      "epoch": 0.3136094674556213,
+      "grad_norm": 0.028224948793649673,
+      "learning_rate": 4.1865286001269556e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9941365480422973,
+      "num_tokens": 29451937.0,
+      "step": 795
+    },
+    {
+      "epoch": 0.3155818540433925,
+      "grad_norm": 0.020351704210042953,
+      "learning_rate": 4.184712266015752e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9941185712814331,
+      "num_tokens": 29636313.0,
+      "step": 800
+    },
+    {
+      "epoch": 0.3175542406311637,
+      "grad_norm": 0.023821154609322548,
+      "learning_rate": 4.182879905921816e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9941583514213562,
+      "num_tokens": 29822875.0,
+      "step": 805
+    },
+    {
+      "epoch": 0.31952662721893493,
+      "grad_norm": 0.017884748056530952,
+      "learning_rate": 4.181031539338083e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9940088272094727,
+      "num_tokens": 30007756.0,
+      "step": 810
+    },
+    {
+      "epoch": 0.3214990138067061,
+      "grad_norm": 0.01931493543088436,
+      "learning_rate": 4.1791671859277685e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.994010066986084,
+      "num_tokens": 30190743.0,
+      "step": 815
+    },
+    {
+      "epoch": 0.3234714003944773,
+      "grad_norm": 0.03150451183319092,
+      "learning_rate": 4.177286865524156e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9941862225532532,
+      "num_tokens": 30376129.0,
+      "step": 820
+    },
+    {
+      "epoch": 0.3254437869822485,
+      "grad_norm": 0.013357733376324177,
+      "learning_rate": 4.175390598130391e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9942962050437927,
+      "num_tokens": 30562067.0,
+      "step": 825
+    },
+    {
+      "epoch": 0.32741617357001973,
+      "grad_norm": 0.028592921793460846,
+      "learning_rate": 4.1734784039192636e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9940245032310486,
+      "num_tokens": 30743304.0,
+      "step": 830
+    },
+    {
+      "epoch": 0.32938856015779094,
+      "grad_norm": 0.022661888971924782,
+      "learning_rate": 4.171550303232998e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9938766360282898,
+      "num_tokens": 30928289.0,
+      "step": 835
+    },
+    {
+      "epoch": 0.33136094674556216,
+      "grad_norm": 0.033437468111515045,
+      "learning_rate": 4.169606316583033e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9941809058189393,
+      "num_tokens": 31116161.0,
+      "step": 840
+    },
+    {
+      "epoch": 0.3333333333333333,
+      "grad_norm": 0.025233056396245956,
+      "learning_rate": 4.167646464649803e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.994313097000122,
+      "num_tokens": 31301580.0,
+      "step": 845
+    },
+    {
+      "epoch": 0.33530571992110453,
+      "grad_norm": 0.024270253255963326,
+      "learning_rate": 4.165670768282523e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9939130067825317,
+      "num_tokens": 31489968.0,
+      "step": 850
+    },
+    {
+      "epoch": 0.33727810650887574,
+      "grad_norm": 0.019101927056908607,
+      "learning_rate": 4.1636792484989604e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9939839720726014,
+      "num_tokens": 31676714.0,
+      "step": 855
+    },
+    {
+      "epoch": 0.33925049309664695,
+      "grad_norm": 0.023301871493458748,
+      "learning_rate": 4.1616719264852176e-05,
+      "loss": 0.0123,
+      "mean_token_accuracy": 0.9941732168197632,
+      "num_tokens": 31861683.0,
+      "step": 860
+    },
+    {
+      "epoch": 0.34122287968441817,
+      "grad_norm": 0.02081654965877533,
+      "learning_rate": 4.1596488235955015e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9940586805343627,
+      "num_tokens": 32049155.0,
+      "step": 865
+    },
+    {
+      "epoch": 0.3431952662721893,
+      "grad_norm": 0.02621420845389366,
+      "learning_rate": 4.157609961351899e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9941748380661011,
+      "num_tokens": 32234796.0,
+      "step": 870
+    },
+    {
+      "epoch": 0.34516765285996054,
+      "grad_norm": 0.016854939982295036,
+      "learning_rate": 4.155555361444148e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9941337823867797,
+      "num_tokens": 32415708.0,
+      "step": 875
+    },
+    {
+      "epoch": 0.34714003944773175,
+      "grad_norm": 0.019724294543266296,
+      "learning_rate": 4.153485045729405e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9940037846565246,
+      "num_tokens": 32603243.0,
+      "step": 880
+    },
+    {
+      "epoch": 0.34911242603550297,
+      "grad_norm": 0.024253640323877335,
+      "learning_rate": 4.151399036232016e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9942573547363281,
+      "num_tokens": 32788971.0,
+      "step": 885
+    },
+    {
+      "epoch": 0.3510848126232742,
+      "grad_norm": 0.015055987983942032,
+      "learning_rate": 4.1492973551432764e-05,
+      "loss": 0.0124,
+      "mean_token_accuracy": 0.9942630171775818,
+      "num_tokens": 32979230.0,
+      "step": 890
+    },
+    {
+      "epoch": 0.3530571992110454,
+      "grad_norm": 0.012248920276761055,
+      "learning_rate": 4.147180024821203e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9940227746963501,
+      "num_tokens": 33164537.0,
+      "step": 895
+    },
+    {
+      "epoch": 0.35502958579881655,
+      "grad_norm": 0.019068609923124313,
+      "learning_rate": 4.145047067790287e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9940544724464416,
+      "num_tokens": 33350130.0,
+      "step": 900
+    },
+    {
+      "epoch": 0.35700197238658776,
+      "grad_norm": 0.019035546109080315,
+      "learning_rate": 4.142898506741261e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9941940903663635,
+      "num_tokens": 33536490.0,
+      "step": 905
+    },
+    {
+      "epoch": 0.358974358974359,
+      "grad_norm": 0.030201325193047523,
+      "learning_rate": 4.1407343645308565e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.9935310006141662,
+      "num_tokens": 33719980.0,
+      "step": 910
+    },
+    {
+      "epoch": 0.3609467455621302,
+      "grad_norm": 0.02158587984740734,
+      "learning_rate": 4.138554664181559e-05,
+      "loss": 0.0124,
+      "mean_token_accuracy": 0.994192922115326,
+      "num_tokens": 33904317.0,
+      "step": 915
+    },
+    {
+      "epoch": 0.3629191321499014,
+      "grad_norm": 0.01816326193511486,
+      "learning_rate": 4.1363594288813646e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9941263437271118,
+      "num_tokens": 34093341.0,
+      "step": 920
+    },
+    {
+      "epoch": 0.36489151873767256,
+      "grad_norm": 0.013991765677928925,
+      "learning_rate": 4.13414868198353e-05,
+      "loss": 0.0123,
+      "mean_token_accuracy": 0.99427170753479,
+      "num_tokens": 34278248.0,
+      "step": 925
+    },
+    {
+      "epoch": 0.3668639053254438,
+      "grad_norm": 0.020902413874864578,
+      "learning_rate": 4.131922447006331e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.994118845462799,
+      "num_tokens": 34466100.0,
+      "step": 930
+    },
+    {
+      "epoch": 0.368836291913215,
+      "grad_norm": 0.01593414507806301,
+      "learning_rate": 4.1296807476328036e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9943223476409913,
+      "num_tokens": 34651696.0,
+      "step": 935
+    },
+    {
+      "epoch": 0.3708086785009862,
+      "grad_norm": 0.030721718445420265,
+      "learning_rate": 4.1274236077105e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9940510272979737,
+      "num_tokens": 34835622.0,
+      "step": 940
+    },
+    {
+      "epoch": 0.3727810650887574,
+      "grad_norm": 0.015614481642842293,
+      "learning_rate": 4.1251510512512305e-05,
+      "loss": 0.0123,
+      "mean_token_accuracy": 0.994101345539093,
+      "num_tokens": 35021150.0,
+      "step": 945
+    },
+    {
+      "epoch": 0.3747534516765286,
+      "grad_norm": 0.016186527907848358,
+      "learning_rate": 4.122863102430806e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9940269470214844,
+      "num_tokens": 35205592.0,
+      "step": 950
+    },
+    {
+      "epoch": 0.3767258382642998,
+      "grad_norm": 0.013813738711178303,
+      "learning_rate": 4.120559785588789e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9939452648162842,
+      "num_tokens": 35390894.0,
+      "step": 955
+    },
+    {
+      "epoch": 0.378698224852071,
+      "grad_norm": 0.033841319382190704,
+      "learning_rate": 4.1182411252282255e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9942763090133667,
+      "num_tokens": 35578752.0,
+      "step": 960
+    },
+    {
+      "epoch": 0.3806706114398422,
+      "grad_norm": 0.016619008034467697,
+      "learning_rate": 4.115907146015389e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9941715478897095,
+      "num_tokens": 35762874.0,
+      "step": 965
+    },
+    {
+      "epoch": 0.3826429980276134,
+      "grad_norm": 0.012615431100130081,
+      "learning_rate": 4.1135578727795184e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9939143896102905,
+      "num_tokens": 35950724.0,
+      "step": 970
+    },
+    {
+      "epoch": 0.38461538461538464,
+      "grad_norm": 0.03185771405696869,
+      "learning_rate": 4.111193330512552e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.993620765209198,
+      "num_tokens": 36132223.0,
+      "step": 975
+    },
+    {
+      "epoch": 0.3865877712031558,
+      "grad_norm": 0.0167035274207592,
+      "learning_rate": 4.1088135443688624e-05,
+      "loss": 0.0124,
+      "mean_token_accuracy": 0.9940300226211548,
+      "num_tokens": 36319685.0,
+      "step": 980
+    },
+    {
+      "epoch": 0.388560157790927,
+      "grad_norm": 0.04164189472794533,
+      "learning_rate": 4.1064185396649894e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9942732572555542,
+      "num_tokens": 36507537.0,
+      "step": 985
+    },
+    {
+      "epoch": 0.3905325443786982,
+      "grad_norm": 0.025893423706293106,
+      "learning_rate": 4.104008341879371e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9938620328903198,
+      "num_tokens": 36692826.0,
+      "step": 990
+    },
+    {
+      "epoch": 0.39250493096646943,
+      "grad_norm": 0.03625747561454773,
+      "learning_rate": 4.1015829766520684e-05,
+      "loss": 0.0124,
+      "mean_token_accuracy": 0.9942021846771241,
+      "num_tokens": 36880255.0,
+      "step": 995
+    },
+    {
+      "epoch": 0.39447731755424065,
+      "grad_norm": 0.0189555361866951,
+      "learning_rate": 4.0991424697845015e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9939949154853821,
+      "num_tokens": 37062438.0,
+      "step": 1000
+    },
+    {
+      "epoch": 0.39644970414201186,
+      "grad_norm": 0.01576905883848667,
+      "learning_rate": 4.0966868472391645e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.994293999671936,
+      "num_tokens": 37247090.0,
+      "step": 1005
+    },
+    {
+      "epoch": 0.398422090729783,
+      "grad_norm": 0.016341818496584892,
+      "learning_rate": 4.094216135139357e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9940265059471131,
+      "num_tokens": 37431333.0,
+      "step": 1010
+    },
+    {
+      "epoch": 0.40039447731755423,
+      "grad_norm": 0.019754884764552116,
+      "learning_rate": 4.0917303597689037e-05,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.9942436695098877,
+      "num_tokens": 37613486.0,
+      "step": 1015
+    },
+    {
+      "epoch": 0.40236686390532544,
+      "grad_norm": 0.016370050609111786,
+      "learning_rate": 4.089229547571874e-05,
+      "loss": 0.0143,
+      "mean_token_accuracy": 0.9936313033103943,
+      "num_tokens": 37797416.0,
+      "step": 1020
+    },
+    {
+      "epoch": 0.40433925049309666,
+      "grad_norm": 0.014701160602271557,
+      "learning_rate": 4.086713725152302e-05,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9941794633865356,
+      "num_tokens": 37981781.0,
+      "step": 1025
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 5070,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 8.9173167078323e+17,
+  "train_batch_size": 20,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f0080dd025b486e2f6572865dedc7d19204763e1dec6c0dc9ca9441ae15bd81
+size 6545