Add files using upload-large-folder tool

README.md

@@ -0,0 +1,16 @@
+---
+library_name: peft
+tags:
+- OPSD
+- lora
+---
+
+# OPSD
+
+147 formal LoRA adapters and frozen-v3 evaluation reports.
+
+Training completed before evaluation; all 162 inference jobs and 64,800 record evaluations completed before publication.
+
+Test dataset: HEART77/vigil-frozen-v3-test, revision `ba161c6fbb2829631ac9fef67ea296f0c4c1fbc4`.
+
+See `reports/frozen_v3/RESULTS.md`, `summary.json`, and `provenance/protocol.json`. Test images, manifests, and raw per-item predictions remain in private storage.

runs/single_internvl/removal/L01_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/single_internvl/removal/L01_s13/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.1.mlp.up_proj",
+    "model.language_model.layers.1.self_attn.o_proj",
+    "model.language_model.layers.1.self_attn.q_proj",
+    "model.language_model.layers.1.self_attn.v_proj",
+    "model.language_model.layers.1.mlp.down_proj",
+    "model.language_model.layers.1.mlp.gate_proj",
+    "model.language_model.layers.1.self_attn.k_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/single_internvl/removal/L01_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 1,
+    "rank": 16,
+    "trainable_parameters": 1212416
+  }
+}

runs/single_internvl/removal/L01_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 0.40592190623283386, "loss_scale": 4.0, "retries": 0, "seconds": 2.5, "max_memory_gb": 19.26, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 0.36344754695892334, "loss_scale": 4.0, "retries": 0, "seconds": 4.3, "max_memory_gb": 19.28, "ce": 0.7818918777375075, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 1.3991551399230957, "loss_scale": 4.0, "retries": 0, "seconds": 6.4, "max_memory_gb": 19.28, "ce": 1.3810374652966857, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 0.7788308262825012, "loss_scale": 4.0, "retries": 0, "seconds": 8.6, "max_memory_gb": 19.28, "ce": 0.7543257179404463, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.0141468970105052, "loss_scale": 4.0, "retries": 0, "seconds": 10.7, "max_memory_gb": 19.28, "ce": 0.040175008167352644, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 0.3695380687713623, "loss_scale": 4.0, "retries": 0, "seconds": 12.8, "max_memory_gb": 19.28, "ce": 0.43776411323051434, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 1.1101232767105103, "loss_scale": 4.0, "retries": 0, "seconds": 14.9, "max_memory_gb": 19.28, "ce": 1.0263768752174656, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 1.054341435432434, "loss_scale": 4.0, "retries": 0, "seconds": 16.7, "max_memory_gb": 19.28, "ce": 0.9989463137462735, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 0.8179892301559448, "loss_scale": 4.0, "retries": 0, "seconds": 18.5, "max_memory_gb": 19.28, "ce": 0.716429965570569, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.12168001383543015, "loss_scale": 4.0, "retries": 0, "seconds": 20.4, "max_memory_gb": 19.29, "ce": 0.1929677892330801, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.9725871682167053, "loss_scale": 4.0, "retries": 0, "seconds": 22.3, "max_memory_gb": 19.29, "ce": 0.8364583739239606, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 0.7289850115776062, "loss_scale": 4.0, "retries": 0, "seconds": 24.1, "max_memory_gb": 19.29, "ce": 0.5560933909491723, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 0.5041403770446777, "loss_scale": 4.0, "retries": 0, "seconds": 25.9, "max_memory_gb": 19.29, "ce": 0.6450286247472832, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 1.1493346691131592, "loss_scale": 4.0, "retries": 0, "seconds": 27.7, "max_memory_gb": 19.29, "ce": 1.0117005808278918, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 0.9112585783004761, "loss_scale": 4.0, "retries": 0, "seconds": 29.6, "max_memory_gb": 19.29, "ce": 0.6802241793232042, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.3645963966846466, "loss_scale": 4.0, "retries": 0, "seconds": 31.4, "max_memory_gb": 19.29, "ce": 0.3167726145038614, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.4588906764984131, "loss_scale": 4.0, "retries": 0, "seconds": 33.2, "max_memory_gb": 19.29, "ce": 0.572909342150524, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 1.2925015687942505, "loss_scale": 4.0, "retries": 0, "seconds": 35.0, "max_memory_gb": 19.31, "ce": 0.7748111893670284, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 2.1017236709594727, "loss_scale": 4.0, "retries": 0, "seconds": 36.9, "max_memory_gb": 19.31, "ce": 1.0824327822774649, "rank": 0.0}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.010016602464020252, "loss_scale": 4.0, "retries": 0, "seconds": 38.7, "max_memory_gb": 19.31, "ce": 0.041236602683056844, "rank": 0.1208396777510643}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 1.1086032390594482, "loss_scale": 4.0, "retries": 0, "seconds": 40.5, "max_memory_gb": 19.31, "ce": 0.5981180348571797, "rank": 0.0}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 1.2421575784683228, "loss_scale": 4.0, "retries": 0, "seconds": 42.3, "max_memory_gb": 19.31, "ce": 0.7009174292252283, "rank": 0.3551759123802185}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.3917088508605957, "loss_scale": 4.0, "retries": 0, "seconds": 44.2, "max_memory_gb": 19.31, "ce": 0.7998873066612759, "rank": 0.011396646499633789}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 1.2479617595672607, "loss_scale": 4.0, "retries": 0, "seconds": 46.0, "max_memory_gb": 19.31, "ce": 0.8195599215105176, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 1.2586431503295898, "loss_scale": 4.0, "retries": 0, "seconds": 47.8, "max_memory_gb": 19.31, "ce": 0.9151941442032694, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 1.1077830791473389, "loss_scale": 4.0, "retries": 0, "seconds": 49.7, "max_memory_gb": 19.31, "ce": 0.7663088738918304, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 1.3017511367797852, "loss_scale": 4.0, "retries": 0, "seconds": 51.6, "max_memory_gb": 19.31, "ce": 0.7385220820651739, "rank": 0.0}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 0.7846518754959106, "loss_scale": 4.0, "retries": 0, "seconds": 53.4, "max_memory_gb": 19.31, "ce": 0.8932427188847214, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 1.1619844436645508, "loss_scale": 4.0, "retries": 0, "seconds": 55.2, "max_memory_gb": 19.31, "ce": 0.5808665056247264, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 1.1104190349578857, "loss_scale": 4.0, "retries": 0, "seconds": 57.0, "max_memory_gb": 19.31, "ce": 0.5064625118830008, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 1.6948169469833374, "loss_scale": 4.0, "retries": 0, "seconds": 58.9, "max_memory_gb": 19.31, "ce": 0.8119220368098468, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 1.8314310312271118, "loss_scale": 4.0, "retries": 0, "seconds": 60.7, "max_memory_gb": 19.31, "ce": 0.7884359443560243, "rank": 0.037193551659584045}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.6582241058349609, "loss_scale": 4.0, "retries": 0, "seconds": 62.6, "max_memory_gb": 19.31, "ce": 0.40641301858704537, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.7061072587966919, "loss_scale": 4.0, "retries": 0, "seconds": 64.4, "max_memory_gb": 19.31, "ce": 0.43351518824238156, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.9400355815887451, "loss_scale": 4.0, "retries": 0, "seconds": 66.2, "max_memory_gb": 19.31, "ce": 0.3809940833489236, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 0.6247174143791199, "loss_scale": 4.0, "retries": 0, "seconds": 68.0, "max_memory_gb": 19.31, "ce": 0.4302608587167924, "rank": 0.032392606139183044}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.7612232565879822, "loss_scale": 4.0, "retries": 0, "seconds": 69.9, "max_memory_gb": 19.31, "ce": 0.7083904566534329, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.7006694078445435, "loss_scale": 4.0, "retries": 0, "seconds": 71.7, "max_memory_gb": 19.31, "ce": 0.6813920848071575, "rank": 0.06770891696214676}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 0.3616948425769806, "loss_scale": 4.0, "retries": 0, "seconds": 73.5, "max_memory_gb": 19.31, "ce": 0.4527928082316066, "rank": 0.12486670166254044}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.5384841561317444, "loss_scale": 4.0, "retries": 0, "seconds": 75.3, "max_memory_gb": 19.31, "ce": 0.33248997008558945, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.5210649967193604, "loss_scale": 4.0, "retries": 0, "seconds": 77.1, "max_memory_gb": 19.31, "ce": 0.479469593759859, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.31517481803894043, "loss_scale": 4.0, "retries": 0, "seconds": 78.9, "max_memory_gb": 19.31, "ce": 0.2387531903659692, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.7554526925086975, "loss_scale": 4.0, "retries": 0, "seconds": 80.7, "max_memory_gb": 19.31, "ce": 0.33331166005518753, "rank": 0.17465374618768692}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 1.0665444135665894, "loss_scale": 4.0, "retries": 0, "seconds": 82.5, "max_memory_gb": 19.31, "ce": 0.7403724914584018, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 0.5883893370628357, "loss_scale": 4.0, "retries": 0, "seconds": 84.3, "max_memory_gb": 19.31, "ce": 0.7926418727147393, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.4314138889312744, "loss_scale": 4.0, "retries": 0, "seconds": 86.1, "max_memory_gb": 19.31, "ce": 0.43780967593193054, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.46660417318344116, "loss_scale": 4.0, "retries": 0, "seconds": 88.0, "max_memory_gb": 19.31, "ce": 0.26962551317410544, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.6566321849822998, "loss_scale": 4.0, "retries": 0, "seconds": 89.8, "max_memory_gb": 19.31, "ce": 0.4504003395559266, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.539411723613739, "loss_scale": 4.0, "retries": 0, "seconds": 91.6, "max_memory_gb": 19.31, "ce": 0.42483852969962754, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.7484670877456665, "loss_scale": 4.0, "retries": 0, "seconds": 93.4, "max_memory_gb": 19.31, "ce": 0.5282506507865037, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 0.9668408036231995, "loss_scale": 4.0, "retries": 0, "seconds": 95.2, "max_memory_gb": 19.31, "ce": 0.6159717908558378, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.7208146452903748, "loss_scale": 4.0, "retries": 0, "seconds": 97.0, "max_memory_gb": 19.31, "ce": 0.4658133091797936, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 0.8892861604690552, "loss_scale": 4.0, "retries": 0, "seconds": 98.8, "max_memory_gb": 19.31, "ce": 0.5682745957747102, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 0.43626976013183594, "loss_scale": 4.0, "retries": 0, "seconds": 100.6, "max_memory_gb": 19.31, "ce": 0.5296333220321685, "rank": 0.0007812976837158203}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.07273999601602554, "loss_scale": 4.0, "retries": 0, "seconds": 102.4, "max_memory_gb": 19.31, "ce": 0.2199955745745683, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.32026100158691406, "loss_scale": 4.0, "retries": 0, "seconds": 104.2, "max_memory_gb": 19.31, "ce": 0.4979811115190387, "rank": 0.06718770414590836}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.1452859342098236, "loss_scale": 4.0, "retries": 0, "seconds": 106.0, "max_memory_gb": 19.31, "ce": 0.18102607928449288, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.29244452714920044, "loss_scale": 4.0, "retries": 0, "seconds": 107.8, "max_memory_gb": 19.31, "ce": 0.3049686987651512, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.1007750928401947, "loss_scale": 4.0, "retries": 0, "seconds": 109.8, "max_memory_gb": 19.31, "ce": 0.15379801084054634, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.1384604424238205, "loss_scale": 4.0, "retries": 0, "seconds": 111.6, "max_memory_gb": 19.31, "ce": 0.27864634457364446, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.48464295268058777, "loss_scale": 4.0, "retries": 0, "seconds": 113.4, "max_memory_gb": 19.31, "ce": 0.6361057665199041, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.2016584873199463, "loss_scale": 4.0, "retries": 0, "seconds": 115.2, "max_memory_gb": 19.31, "ce": 0.18675681126478594, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.1933012455701828, "loss_scale": 4.0, "retries": 0, "seconds": 117.0, "max_memory_gb": 19.31, "ce": 0.35231770237442106, "rank": 0.07322118431329727}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.22679272294044495, "loss_scale": 4.0, "retries": 0, "seconds": 118.8, "max_memory_gb": 19.31, "ce": 0.3595613136421889, "rank": 0.11976394057273865}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 0.6003723740577698, "loss_scale": 4.0, "retries": 0, "seconds": 120.7, "max_memory_gb": 19.31, "ce": 0.5789170991629362, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.24977464973926544, "loss_scale": 4.0, "retries": 0, "seconds": 122.5, "max_memory_gb": 19.31, "ce": 0.26374749204597947, "rank": 0.12445572018623352}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.3167632818222046, "loss_scale": 4.0, "retries": 0, "seconds": 124.3, "max_memory_gb": 19.31, "ce": 0.37260169236105867, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.14639894664287567, "loss_scale": 4.0, "retries": 0, "seconds": 126.1, "max_memory_gb": 19.31, "ce": 0.16567068321910483, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.5318487882614136, "loss_scale": 4.0, "retries": 0, "seconds": 127.9, "max_memory_gb": 19.31, "ce": 0.6095046477857977, "rank": 0.01920653134584427}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.25295835733413696, "loss_scale": 4.0, "retries": 0, "seconds": 129.7, "max_memory_gb": 19.31, "ce": 0.2644439910945948, "rank": 0.12921026349067688}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.4292335510253906, "loss_scale": 4.0, "retries": 0, "seconds": 131.6, "max_memory_gb": 19.31, "ce": 0.3828245178410725, "rank": 0.030864804983139038}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.2965637147426605, "loss_scale": 4.0, "retries": 0, "seconds": 133.4, "max_memory_gb": 19.31, "ce": 0.537155169993639, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 0.7054654955863953, "loss_scale": 4.0, "retries": 0, "seconds": 135.2, "max_memory_gb": 19.31, "ce": 0.30398173673893325, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.3900926411151886, "loss_scale": 4.0, "retries": 0, "seconds": 137.0, "max_memory_gb": 19.31, "ce": 0.3363196582299679, "rank": 0.16274848580360413}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.38821008801460266, "loss_scale": 4.0, "retries": 0, "seconds": 138.8, "max_memory_gb": 19.31, "ce": 0.7675008794758469, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 0.245627298951149, "loss_scale": 4.0, "retries": 0, "seconds": 140.6, "max_memory_gb": 19.31, "ce": 0.47871756326640025, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.16239924728870392, "loss_scale": 4.0, "retries": 0, "seconds": 142.4, "max_memory_gb": 19.31, "ce": 0.3504665078144171, "rank": 0.0817294716835022}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.1838541030883789, "loss_scale": 4.0, "retries": 0, "seconds": 144.2, "max_memory_gb": 19.31, "ce": 0.2903674147601123, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.25790268182754517, "loss_scale": 4.0, "retries": 0, "seconds": 145.9, "max_memory_gb": 19.31, "ce": 0.15818016626235476, "rank": 0.1027480810880661}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.2105238139629364, "loss_scale": 4.0, "retries": 0, "seconds": 147.7, "max_memory_gb": 19.31, "ce": 0.2803927052977997, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.3115900456905365, "loss_scale": 4.0, "retries": 0, "seconds": 149.5, "max_memory_gb": 19.31, "ce": 0.4248777867760509, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.36762332916259766, "loss_scale": 4.0, "retries": 0, "seconds": 151.4, "max_memory_gb": 19.31, "ce": 0.3739104426203994, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 0.2914287745952606, "loss_scale": 4.0, "retries": 0, "seconds": 153.2, "max_memory_gb": 19.31, "ce": 0.6183468459494179, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.1212158352136612, "loss_scale": 4.0, "retries": 0, "seconds": 155.0, "max_memory_gb": 19.31, "ce": 0.13584373885555578, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.8536118268966675, "loss_scale": 4.0, "retries": 0, "seconds": 156.8, "max_memory_gb": 19.31, "ce": 0.08175593795726854, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.7244089245796204, "loss_scale": 4.0, "retries": 0, "seconds": 158.7, "max_memory_gb": 19.31, "ce": 0.4202517695957795, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.30414295196533203, "loss_scale": 4.0, "retries": 0, "seconds": 160.4, "max_memory_gb": 19.31, "ce": 0.450413357510115, "rank": 0.009706921875476837}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.1613537222146988, "loss_scale": 4.0, "retries": 0, "seconds": 162.3, "max_memory_gb": 19.31, "ce": 0.25229025912722136, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.07636202126741409, "loss_scale": 4.0, "retries": 0, "seconds": 164.0, "max_memory_gb": 19.31, "ce": 0.04956017698214055, "rank": 0.16678066551685333}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.3262903690338135, "loss_scale": 4.0, "retries": 0, "seconds": 165.8, "max_memory_gb": 19.31, "ce": 0.3891130508563947, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 1.083946704864502, "loss_scale": 4.0, "retries": 0, "seconds": 167.6, "max_memory_gb": 19.31, "ce": 0.5264686799273477, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 0.9123865365982056, "loss_scale": 4.0, "retries": 0, "seconds": 169.5, "max_memory_gb": 19.31, "ce": 0.42275113198093095, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 0.47980234026908875, "loss_scale": 4.0, "retries": 0, "seconds": 171.3, "max_memory_gb": 19.31, "ce": 0.3645461001433432, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 1.1351947784423828, "loss_scale": 4.0, "retries": 0, "seconds": 173.1, "max_memory_gb": 19.31, "ce": 0.5240755912382156, "rank": 0.030618295073509216}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 1.0062243938446045, "loss_scale": 4.0, "retries": 0, "seconds": 174.9, "max_memory_gb": 19.31, "ce": 0.8162761994171888, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 0.8757691383361816, "loss_scale": 4.0, "retries": 0, "seconds": 176.7, "max_memory_gb": 19.31, "ce": 0.9096461817098316, "rank": 0.5468819290399551}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.6444434523582458, "loss_scale": 4.0, "retries": 0, "seconds": 178.5, "max_memory_gb": 19.31, "ce": 0.1866251177561935, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.1835479587316513, "loss_scale": 4.0, "retries": 0, "seconds": 180.3, "max_memory_gb": 19.31, "ce": 0.28093283164889726, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.33303776383399963, "loss_scale": 4.0, "retries": 0, "seconds": 182.1, "max_memory_gb": 19.31, "ce": 0.060410236616917246, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.5115317702293396, "loss_scale": 4.0, "retries": 0, "seconds": 183.9, "max_memory_gb": 19.31, "ce": 0.49273527204059064, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.22455260157585144, "loss_scale": 4.0, "retries": 0, "seconds": 185.8, "max_memory_gb": 19.31, "ce": 0.3277510083280504, "rank": 0.1171407699584961}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.15911193192005157, "loss_scale": 4.0, "retries": 0, "seconds": 187.6, "max_memory_gb": 19.31, "ce": 0.2986096358508803, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.1458621323108673, "loss_scale": 4.0, "retries": 0, "seconds": 189.4, "max_memory_gb": 19.31, "ce": 0.25799373613517673, "rank": 0.09455118328332901}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.05587594956159592, "loss_scale": 4.0, "retries": 0, "seconds": 191.1, "max_memory_gb": 19.31, "ce": 0.14815567000914598, "rank": 0.12461426854133606}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.47198349237442017, "loss_scale": 4.0, "retries": 0, "seconds": 193.0, "max_memory_gb": 19.31, "ce": 0.45699186995625496, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.20295070111751556, "loss_scale": 4.0, "retries": 0, "seconds": 194.8, "max_memory_gb": 19.31, "ce": 0.3202244611675269, "rank": 0.03279662877321243}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 1.6665109395980835, "loss_scale": 4.0, "retries": 0, "seconds": 196.7, "max_memory_gb": 19.31, "ce": 0.9328791693769745, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 0.2557063102722168, "loss_scale": 4.0, "retries": 0, "seconds": 198.5, "max_memory_gb": 19.31, "ce": 0.22210977065878978, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 0.2884180247783661, "loss_scale": 4.0, "retries": 0, "seconds": 200.3, "max_memory_gb": 19.31, "ce": 0.5665930206887424, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 1.8824983835220337, "loss_scale": 4.0, "retries": 0, "seconds": 202.1, "max_memory_gb": 19.31, "ce": 0.49791506095789373, "rank": 0.1599670797586441}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.3113999366760254, "loss_scale": 4.0, "retries": 0, "seconds": 204.0, "max_memory_gb": 19.31, "ce": 0.32039916011854075, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.1683175414800644, "loss_scale": 4.0, "retries": 0, "seconds": 205.8, "max_memory_gb": 19.31, "ce": 0.20294732378533809, "rank": 0.09723980724811554}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.19055336713790894, "loss_scale": 4.0, "retries": 0, "seconds": 207.6, "max_memory_gb": 19.31, "ce": 0.2881030768621713, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.2399868667125702, "loss_scale": 4.0, "retries": 0, "seconds": 209.5, "max_memory_gb": 19.31, "ce": 0.4666197875048965, "rank": 0.09933970868587494}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.18105772137641907, "loss_scale": 4.0, "retries": 0, "seconds": 211.3, "max_memory_gb": 19.31, "ce": 0.23102165812815656, "rank": 0.05080440640449524}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.6359997987747192, "loss_scale": 4.0, "retries": 0, "seconds": 213.2, "max_memory_gb": 19.31, "ce": 0.28672988180915127, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.9363471269607544, "loss_scale": 4.0, "retries": 0, "seconds": 215.0, "max_memory_gb": 19.32, "ce": 0.34975115966881276, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 0.40459275245666504, "loss_scale": 4.0, "retries": 0, "seconds": 216.8, "max_memory_gb": 19.32, "ce": 0.35172750646597706, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.12079533189535141, "loss_scale": 4.0, "retries": 0, "seconds": 218.6, "max_memory_gb": 19.32, "ce": 0.1474592200229381, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 1.120408058166504, "loss_scale": 4.0, "retries": 0, "seconds": 220.6, "max_memory_gb": 19.32, "ce": 0.8171109643881209, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.18062296509742737, "loss_scale": 4.0, "retries": 0, "seconds": 222.4, "max_memory_gb": 19.32, "ce": 0.18992279446320026, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.1802021563053131, "loss_scale": 4.0, "retries": 0, "seconds": 224.3, "max_memory_gb": 19.32, "ce": 0.22836451749390108, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.5508224368095398, "loss_scale": 4.0, "retries": 0, "seconds": 226.1, "max_memory_gb": 19.32, "ce": 0.2633881024084985, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 0.8381392955780029, "loss_scale": 4.0, "retries": 0, "seconds": 228.0, "max_memory_gb": 19.32, "ce": 0.767589655588381, "rank": 0.24686729907989502}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.3167914152145386, "loss_scale": 4.0, "retries": 0, "seconds": 229.9, "max_memory_gb": 19.32, "ce": 0.28075647227524314, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 0.8201347589492798, "loss_scale": 4.0, "retries": 0, "seconds": 231.7, "max_memory_gb": 19.32, "ce": 0.7303085895255208, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.24201802909374237, "loss_scale": 4.0, "retries": 0, "seconds": 233.5, "max_memory_gb": 19.32, "ce": 0.1335820956740008, "rank": 0.0031963884830474854}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 0.8645281791687012, "loss_scale": 4.0, "retries": 0, "seconds": 235.3, "max_memory_gb": 19.32, "ce": 0.6533466030959971, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 0.24991977214813232, "loss_scale": 4.0, "retries": 0, "seconds": 237.1, "max_memory_gb": 19.32, "ce": 0.44816427817568183, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 0.3236270546913147, "loss_scale": 4.0, "retries": 0, "seconds": 238.9, "max_memory_gb": 19.32, "ce": 0.5295100678049494, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 0.8512251973152161, "loss_scale": 4.0, "retries": 0, "seconds": 240.7, "max_memory_gb": 19.32, "ce": 0.24401543734711595, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 0.41413936018943787, "loss_scale": 4.0, "retries": 0, "seconds": 242.5, "max_memory_gb": 19.32, "ce": 0.5080489180945733, "rank": 0.10073179751634598}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 0.42306894063949585, "loss_scale": 4.0, "retries": 0, "seconds": 244.5, "max_memory_gb": 19.32, "ce": 0.6903891663532704, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.5956385731697083, "loss_scale": 4.0, "retries": 0, "seconds": 246.3, "max_memory_gb": 19.32, "ce": 0.252192398818579, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 0.352979451417923, "loss_scale": 4.0, "retries": 0, "seconds": 248.2, "max_memory_gb": 19.32, "ce": 0.38931160303764045, "rank": 0.003233380615711212}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 1.280977487564087, "loss_scale": 4.0, "retries": 0, "seconds": 250.0, "max_memory_gb": 19.32, "ce": 0.2954423698829487, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.17128877341747284, "loss_scale": 4.0, "retries": 0, "seconds": 252.0, "max_memory_gb": 19.32, "ce": 0.2184133361442946, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 0.8119917511940002, "loss_scale": 4.0, "retries": 0, "seconds": 253.8, "max_memory_gb": 19.32, "ce": 0.39297481975518167, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.6219630241394043, "loss_scale": 4.0, "retries": 0, "seconds": 255.7, "max_memory_gb": 19.32, "ce": 0.4630586258863332, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.17169809341430664, "loss_scale": 4.0, "retries": 0, "seconds": 257.5, "max_memory_gb": 19.32, "ce": 0.2134622723024222, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.1983896791934967, "loss_scale": 4.0, "retries": 0, "seconds": 259.4, "max_memory_gb": 19.32, "ce": 0.20867509020899888, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.37429380416870117, "loss_scale": 4.0, "retries": 0, "seconds": 261.2, "max_memory_gb": 19.32, "ce": 0.30515119515621336, "rank": 0.09259506314992905}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 1.1712480783462524, "loss_scale": 4.0, "retries": 0, "seconds": 263.0, "max_memory_gb": 19.32, "ce": 0.5432860272721882, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.9957643747329712, "loss_scale": 4.0, "retries": 0, "seconds": 264.8, "max_memory_gb": 19.32, "ce": 0.3780719470232725, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.10820425301790237, "loss_scale": 4.0, "retries": 0, "seconds": 266.7, "max_memory_gb": 19.32, "ce": 0.20224674353994487, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.12347738444805145, "loss_scale": 4.0, "retries": 0, "seconds": 268.5, "max_memory_gb": 19.32, "ce": 0.18758350491407327, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 0.5357276201248169, "loss_scale": 4.0, "retries": 0, "seconds": 270.3, "max_memory_gb": 19.32, "ce": 0.8291092629369814, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.144830584526062, "loss_scale": 4.0, "retries": 0, "seconds": 272.1, "max_memory_gb": 19.32, "ce": 0.12832073472782213, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.1712813526391983, "loss_scale": 4.0, "retries": 0, "seconds": 273.9, "max_memory_gb": 19.32, "ce": 0.1281146741239354, "rank": 0.12531554698944092}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.29761025309562683, "loss_scale": 4.0, "retries": 0, "seconds": 275.8, "max_memory_gb": 19.32, "ce": 0.3314167254138738, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.06442123651504517, "loss_scale": 4.0, "retries": 0, "seconds": 277.6, "max_memory_gb": 19.32, "ce": 0.1648841198584705, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 0.22121980786323547, "loss_scale": 4.0, "retries": 0, "seconds": 279.4, "max_memory_gb": 19.32, "ce": 0.37884153400864307, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.1875600665807724, "loss_scale": 4.0, "retries": 0, "seconds": 281.3, "max_memory_gb": 19.32, "ce": 0.2859301030412098, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.7791773676872253, "loss_scale": 4.0, "retries": 0, "seconds": 283.1, "max_memory_gb": 19.32, "ce": 0.3903549069073051, "rank": 0.10373024642467499}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 0.3750467598438263, "loss_scale": 4.0, "retries": 0, "seconds": 284.9, "max_memory_gb": 19.32, "ce": 0.3689140360784222, "rank": 0.05427846312522888}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.5074861645698547, "loss_scale": 4.0, "retries": 0, "seconds": 286.8, "max_memory_gb": 19.32, "ce": 0.14902920423810428, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.5665552020072937, "loss_scale": 4.0, "retries": 0, "seconds": 288.5, "max_memory_gb": 19.32, "ce": 0.153109860533732, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 0.3438061773777008, "loss_scale": 4.0, "retries": 0, "seconds": 290.4, "max_memory_gb": 19.32, "ce": 0.30915794201064273, "rank": 0.0032129138708114624}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.232133686542511, "loss_scale": 4.0, "retries": 0, "seconds": 292.2, "max_memory_gb": 19.45, "ce": 0.25247777760523604, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 0.7284208536148071, "loss_scale": 4.0, "retries": 0, "seconds": 294.0, "max_memory_gb": 19.45, "ce": 0.4719165792921558, "rank": 0.07217305898666382}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 0.40608304738998413, "loss_scale": 4.0, "retries": 0, "seconds": 295.8, "max_memory_gb": 19.45, "ce": 0.20190843827549543, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.42090466618537903, "loss_scale": 4.0, "retries": 0, "seconds": 297.6, "max_memory_gb": 19.45, "ce": 0.22526264019688824, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 1.440641164779663, "loss_scale": 4.0, "retries": 0, "seconds": 299.4, "max_memory_gb": 19.45, "ce": 0.6815463330131024, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 1.1556954383850098, "loss_scale": 4.0, "retries": 0, "seconds": 301.3, "max_memory_gb": 19.45, "ce": 0.28488085040589795, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.058992352336645126, "loss_scale": 4.0, "retries": 0, "seconds": 303.1, "max_memory_gb": 19.45, "ce": 0.1189584113744786, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 1.1604126691818237, "loss_scale": 4.0, "retries": 0, "seconds": 305.0, "max_memory_gb": 19.45, "ce": 0.507716550026089, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.24264369904994965, "loss_scale": 4.0, "retries": 0, "seconds": 306.8, "max_memory_gb": 19.45, "ce": 0.27879373962059617, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.14709767699241638, "loss_scale": 4.0, "retries": 0, "seconds": 308.6, "max_memory_gb": 19.45, "ce": 0.2215954294806579, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 0.44985413551330566, "loss_scale": 4.0, "retries": 0, "seconds": 310.6, "max_memory_gb": 19.45, "ce": 0.6636706795543432, "rank": 0.13001646101474762}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.24966084957122803, "loss_scale": 4.0, "retries": 0, "seconds": 312.4, "max_memory_gb": 19.45, "ce": 0.3161497827095445, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.2616180181503296, "loss_scale": 4.0, "retries": 0, "seconds": 314.3, "max_memory_gb": 19.45, "ce": 0.2681300118588297, "rank": 0.03139045089483261}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.2689417898654938, "loss_scale": 4.0, "retries": 0, "seconds": 316.2, "max_memory_gb": 19.45, "ce": 0.3862062598345801, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.2914855182170868, "loss_scale": 4.0, "retries": 0, "seconds": 317.9, "max_memory_gb": 19.45, "ce": 0.33870344213210046, "rank": 0.06502124667167664}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.2242196947336197, "loss_scale": 4.0, "retries": 0, "seconds": 319.7, "max_memory_gb": 19.45, "ce": 0.262047889284986, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 2.776348352432251, "loss_scale": 4.0, "retries": 0, "seconds": 321.6, "max_memory_gb": 19.45, "ce": 0.7512544058263302, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.33944782614707947, "loss_scale": 4.0, "retries": 0, "seconds": 323.4, "max_memory_gb": 19.45, "ce": 0.3294498010072857, "rank": 0.1769171804189682}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 0.3462626338005066, "loss_scale": 4.0, "retries": 0, "seconds": 325.2, "max_memory_gb": 19.45, "ce": 0.4631744680227712, "rank": 0.1250041276216507}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 1.1450239419937134, "loss_scale": 4.0, "retries": 0, "seconds": 327.0, "max_memory_gb": 19.45, "ce": 0.18995307352452073, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 0.4877725839614868, "loss_scale": 4.0, "retries": 0, "seconds": 328.8, "max_memory_gb": 19.45, "ce": 0.6377574895350335, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.19480833411216736, "loss_scale": 4.0, "retries": 0, "seconds": 330.6, "max_memory_gb": 19.45, "ce": 0.3264166129520163, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.25356221199035645, "loss_scale": 4.0, "retries": 0, "seconds": 332.5, "max_memory_gb": 19.45, "ce": 0.40405370166990906, "rank": 0.11836646497249603}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 2.1581594944000244, "loss_scale": 4.0, "retries": 0, "seconds": 334.3, "max_memory_gb": 19.45, "ce": 0.18805047661226126, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.11547576636075974, "loss_scale": 4.0, "retries": 0, "seconds": 336.1, "max_memory_gb": 19.45, "ce": 0.19405132826796034, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.14768151938915253, "loss_scale": 4.0, "retries": 0, "seconds": 338.0, "max_memory_gb": 19.45, "ce": 0.22397452876384705, "rank": 0.21317403763532639}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.34086737036705017, "loss_scale": 4.0, "retries": 0, "seconds": 339.8, "max_memory_gb": 19.45, "ce": 0.35199551680125296, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 1.2823668718338013, "loss_scale": 4.0, "retries": 0, "seconds": 341.7, "max_memory_gb": 19.45, "ce": 0.3745486254993011, "rank": 0.08177260309457779}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.45032405853271484, "loss_scale": 4.0, "retries": 0, "seconds": 343.5, "max_memory_gb": 19.45, "ce": 0.6011145627126098, "rank": 0.13312870264053345}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.1115115880966187, "loss_scale": 4.0, "retries": 0, "seconds": 345.3, "max_memory_gb": 19.45, "ce": 0.3154245875948618, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 1.8970586061477661, "loss_scale": 4.0, "retries": 0, "seconds": 347.1, "max_memory_gb": 19.45, "ce": 0.5939468389842659, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.2872411012649536, "loss_scale": 4.0, "retries": 0, "seconds": 349.0, "max_memory_gb": 19.45, "ce": 0.41935682465555146, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 3.184696674346924, "loss_scale": 4.0, "retries": 0, "seconds": 350.7, "max_memory_gb": 19.45, "ce": 0.3126186196714116, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 0.5748406648635864, "loss_scale": 4.0, "retries": 0, "seconds": 352.6, "max_memory_gb": 19.45, "ce": 0.6765283655840904, "rank": 0.0040934085845947266}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.24772827327251434, "loss_scale": 4.0, "retries": 0, "seconds": 354.4, "max_memory_gb": 19.45, "ce": 0.28172808030103624, "rank": 0.12481611222028732}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.22709572315216064, "loss_scale": 4.0, "retries": 0, "seconds": 356.2, "max_memory_gb": 19.45, "ce": 0.13254247776785633, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.28917160630226135, "loss_scale": 4.0, "retries": 0, "seconds": 358.1, "max_memory_gb": 19.45, "ce": 0.4139360477565788, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.22922293841838837, "loss_scale": 4.0, "retries": 0, "seconds": 360.0, "max_memory_gb": 19.45, "ce": 0.3453557935563367, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.6576845049858093, "loss_scale": 4.0, "retries": 0, "seconds": 361.8, "max_memory_gb": 19.45, "ce": 0.42518917325651273, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.7399536967277527, "loss_scale": 4.0, "retries": 0, "seconds": 363.6, "max_memory_gb": 19.45, "ce": 0.21285070072190138, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 0.5291104316711426, "loss_scale": 4.0, "retries": 0, "seconds": 365.4, "max_memory_gb": 19.45, "ce": 0.46440916863502935, "rank": 0.06330450624227524}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 1.2647960186004639, "loss_scale": 4.0, "retries": 0, "seconds": 367.3, "max_memory_gb": 19.45, "ce": 0.4977582274732413, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.3800576329231262, "loss_scale": 4.0, "retries": 0, "seconds": 369.1, "max_memory_gb": 19.45, "ce": 0.2517201117807417, "rank": 0.07968184351921082}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.08229101449251175, "loss_scale": 4.0, "retries": 0, "seconds": 370.9, "max_memory_gb": 19.45, "ce": 0.13429237492346147, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.24201638996601105, "loss_scale": 4.0, "retries": 0, "seconds": 372.7, "max_memory_gb": 19.45, "ce": 0.2383951305237133, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.2734195291996002, "loss_scale": 4.0, "retries": 0, "seconds": 374.5, "max_memory_gb": 19.45, "ce": 0.40981736907269806, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 0.17465372383594513, "loss_scale": 4.0, "retries": 0, "seconds": 376.3, "max_memory_gb": 19.45, "ce": 0.3145608165032172, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.17759987711906433, "loss_scale": 4.0, "retries": 0, "seconds": 378.2, "max_memory_gb": 19.45, "ce": 0.3372169402673535, "rank": 0.08100275695323944}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.0482158362865448, "loss_scale": 4.0, "retries": 0, "seconds": 380.0, "max_memory_gb": 19.45, "ce": 0.06988950526283588, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.235432967543602, "loss_scale": 4.0, "retries": 0, "seconds": 381.9, "max_memory_gb": 19.45, "ce": 0.28748516738323815, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.32002565264701843, "loss_scale": 4.0, "retries": 0, "seconds": 383.7, "max_memory_gb": 19.45, "ce": 0.2898050213989336, "rank": 0.19920413196086884}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 0.8068851232528687, "loss_scale": 4.0, "retries": 0, "seconds": 385.5, "max_memory_gb": 19.45, "ce": 0.5710261146614357, "rank": 0.14166587591171265}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.3463784158229828, "loss_scale": 4.0, "retries": 0, "seconds": 387.3, "max_memory_gb": 19.45, "ce": 0.3531001113005914, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 0.3232337236404419, "loss_scale": 4.0, "retries": 0, "seconds": 389.2, "max_memory_gb": 19.45, "ce": 0.4457376260543242, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.3818279802799225, "loss_scale": 4.0, "retries": 0, "seconds": 391.1, "max_memory_gb": 19.45, "ce": 0.4011570021684747, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 0.6328138709068298, "loss_scale": 4.0, "retries": 0, "seconds": 392.9, "max_memory_gb": 19.45, "ce": 0.4963851754400821, "rank": 0.1613505482673645}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 1.0178014039993286, "loss_scale": 4.0, "retries": 0, "seconds": 394.8, "max_memory_gb": 19.45, "ce": 0.6040830553647538, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.17908944189548492, "loss_scale": 4.0, "retries": 0, "seconds": 396.6, "max_memory_gb": 19.45, "ce": 0.13091767273726873, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.2016948014497757, "loss_scale": 4.0, "retries": 0, "seconds": 398.3, "max_memory_gb": 19.45, "ce": 0.19077631773689063, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 0.9696150422096252, "loss_scale": 4.0, "retries": 0, "seconds": 400.0, "max_memory_gb": 19.45, "ce": 0.36050675297155976, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.26512497663497925, "loss_scale": 4.0, "retries": 0, "seconds": 401.6, "max_memory_gb": 19.45, "ce": 0.13272723595400748, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.1484983265399933, "loss_scale": 4.0, "retries": 0, "seconds": 403.3, "max_memory_gb": 19.45, "ce": 0.1880517380841411, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.2655772864818573, "loss_scale": 4.0, "retries": 0, "seconds": 405.0, "max_memory_gb": 19.45, "ce": 0.46295536682009697, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.24320940673351288, "loss_scale": 4.0, "retries": 0, "seconds": 406.6, "max_memory_gb": 19.45, "ce": 0.3332467137261119, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 0.2576617896556854, "loss_scale": 4.0, "retries": 0, "seconds": 408.3, "max_memory_gb": 19.45, "ce": 0.06864381962805055, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.1539723426103592, "loss_scale": 4.0, "retries": 0, "seconds": 409.9, "max_memory_gb": 19.45, "ce": 0.22796667356305989, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.171442911028862, "loss_scale": 4.0, "retries": 0, "seconds": 411.6, "max_memory_gb": 19.45, "ce": 0.2548979125713231, "rank": 0.05804113298654556}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 1.0088025331497192, "loss_scale": 4.0, "retries": 0, "seconds": 413.3, "max_memory_gb": 19.45, "ce": 0.2545439042587532, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.3158963918685913, "loss_scale": 4.0, "retries": 0, "seconds": 414.9, "max_memory_gb": 19.45, "ce": 0.22699045208719326, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.12724952399730682, "loss_scale": 4.0, "retries": 0, "seconds": 416.6, "max_memory_gb": 19.45, "ce": 0.1419896688166773, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.2371964007616043, "loss_scale": 4.0, "retries": 0, "seconds": 418.2, "max_memory_gb": 19.45, "ce": 0.16931805592321325, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 1.7615810632705688, "loss_scale": 4.0, "retries": 0, "seconds": 419.9, "max_memory_gb": 19.45, "ce": 0.5870256164926104, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.12425887584686279, "loss_scale": 4.0, "retries": 0, "seconds": 421.5, "max_memory_gb": 19.45, "ce": 0.14908091907454946, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.2326740175485611, "loss_scale": 4.0, "retries": 0, "seconds": 423.2, "max_memory_gb": 19.45, "ce": 0.4452290900517255, "rank": 0.02852059155702591}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.42931506037712097, "loss_scale": 4.0, "retries": 0, "seconds": 424.9, "max_memory_gb": 19.45, "ce": 0.3049736252432922, "rank": 0.00865403562784195}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.3620966076850891, "loss_scale": 4.0, "retries": 0, "seconds": 426.5, "max_memory_gb": 19.45, "ce": 0.13180751763866283, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 1.3211952447891235, "loss_scale": 4.0, "retries": 0, "seconds": 428.2, "max_memory_gb": 19.45, "ce": 0.4554861411525053, "rank": 0.03967825323343277}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.12403732538223267, "loss_scale": 4.0, "retries": 0, "seconds": 429.8, "max_memory_gb": 19.45, "ce": 0.13674281771091046, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.3269234597682953, "loss_scale": 4.0, "retries": 0, "seconds": 431.5, "max_memory_gb": 19.45, "ce": 0.2854757207715011, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.4232649803161621, "loss_scale": 4.0, "retries": 0, "seconds": 433.1, "max_memory_gb": 19.45, "ce": 0.3447082934435457, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.35908517241477966, "loss_scale": 4.0, "retries": 0, "seconds": 434.8, "max_memory_gb": 19.45, "ce": 0.38311101516592316, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 0.3575441837310791, "loss_scale": 4.0, "retries": 0, "seconds": 436.4, "max_memory_gb": 19.45, "ce": 0.32218267941789236, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.5399430990219116, "loss_scale": 4.0, "retries": 0, "seconds": 438.1, "max_memory_gb": 19.45, "ce": 0.13431719823893218, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.09469609707593918, "loss_scale": 4.0, "retries": 0, "seconds": 439.8, "max_memory_gb": 19.45, "ce": 0.10853020156355342, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.1984127163887024, "loss_scale": 4.0, "retries": 0, "seconds": 441.5, "max_memory_gb": 19.45, "ce": 0.31385496121947654, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.32127800583839417, "loss_scale": 4.0, "retries": 0, "seconds": 443.1, "max_memory_gb": 19.45, "ce": 0.2589682945981622, "rank": 0.046023741364479065}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 1.0004361867904663, "loss_scale": 4.0, "retries": 0, "seconds": 444.8, "max_memory_gb": 19.45, "ce": 0.42060368531383574, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.15513357520103455, "loss_scale": 4.0, "retries": 0, "seconds": 446.5, "max_memory_gb": 19.45, "ce": 0.19731371247326024, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.5730578303337097, "loss_scale": 4.0, "retries": 0, "seconds": 448.1, "max_memory_gb": 19.45, "ce": 0.24841289707501346, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.15482792258262634, "loss_scale": 4.0, "retries": 0, "seconds": 449.8, "max_memory_gb": 19.45, "ce": 0.2655737061868422, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.2109987586736679, "loss_scale": 4.0, "retries": 0, "seconds": 451.5, "max_memory_gb": 19.45, "ce": 0.32338087214156985, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.20954978466033936, "loss_scale": 4.0, "retries": 0, "seconds": 453.2, "max_memory_gb": 19.45, "ce": 0.25606998480361653, "rank": 0.0}

runs/single_internvl/removal/L02_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/single_internvl/removal/L02_s13/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.2.mlp.gate_proj",
+    "model.language_model.layers.2.mlp.up_proj",
+    "model.language_model.layers.2.self_attn.o_proj",
+    "model.language_model.layers.2.self_attn.k_proj",
+    "model.language_model.layers.2.self_attn.q_proj",
+    "model.language_model.layers.2.mlp.down_proj",
+    "model.language_model.layers.2.self_attn.v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/single_internvl/removal/L02_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 2,
+    "rank": 16,
+    "trainable_parameters": 1212416
+  }
+}

runs/single_internvl/removal/L02_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 0.4815939664840698, "loss_scale": 4.0, "retries": 0, "seconds": 3.0, "max_memory_gb": 19.17, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 0.4193096458911896, "loss_scale": 4.0, "retries": 0, "seconds": 4.9, "max_memory_gb": 19.19, "ce": 0.7802844818324957, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 1.7109901905059814, "loss_scale": 4.0, "retries": 0, "seconds": 6.9, "max_memory_gb": 19.19, "ce": 1.3811025014147162, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 0.9609407782554626, "loss_scale": 4.0, "retries": 0, "seconds": 8.9, "max_memory_gb": 19.19, "ce": 0.7541647365287645, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.01567627303302288, "loss_scale": 4.0, "retries": 0, "seconds": 10.8, "max_memory_gb": 19.19, "ce": 0.04018468969843525, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 0.41979020833969116, "loss_scale": 4.0, "retries": 0, "seconds": 12.7, "max_memory_gb": 19.19, "ce": 0.43492158720619045, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 1.3757244348526, "loss_scale": 4.0, "retries": 0, "seconds": 14.7, "max_memory_gb": 19.19, "ce": 1.027275827646008, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 1.2565181255340576, "loss_scale": 4.0, "retries": 0, "seconds": 16.7, "max_memory_gb": 19.19, "ce": 1.0019140904769301, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 0.9597170948982239, "loss_scale": 4.0, "retries": 0, "seconds": 18.6, "max_memory_gb": 19.19, "ce": 0.7153039355762303, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.13126236200332642, "loss_scale": 4.0, "retries": 0, "seconds": 20.6, "max_memory_gb": 19.2, "ce": 0.19294103993161116, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 1.1419047117233276, "loss_scale": 4.0, "retries": 0, "seconds": 22.5, "max_memory_gb": 19.2, "ce": 0.8336179434409132, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 0.8571864366531372, "loss_scale": 4.0, "retries": 0, "seconds": 24.4, "max_memory_gb": 19.2, "ce": 0.5564799394505826, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 0.600639820098877, "loss_scale": 4.0, "retries": 0, "seconds": 26.3, "max_memory_gb": 19.2, "ce": 0.6464266303628392, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 1.3361161947250366, "loss_scale": 4.0, "retries": 0, "seconds": 28.2, "max_memory_gb": 19.2, "ce": 1.0045891380868852, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 1.0900087356567383, "loss_scale": 4.0, "retries": 0, "seconds": 30.0, "max_memory_gb": 19.2, "ce": 0.6747653705326684, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.4059518575668335, "loss_scale": 4.0, "retries": 0, "seconds": 31.9, "max_memory_gb": 19.2, "ce": 0.31389052318263566, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.5313112735748291, "loss_scale": 4.0, "retries": 0, "seconds": 33.9, "max_memory_gb": 19.2, "ce": 0.5702260763137019, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 1.4363837242126465, "loss_scale": 4.0, "retries": 0, "seconds": 35.9, "max_memory_gb": 19.22, "ce": 0.765407617414894, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 2.3935601711273193, "loss_scale": 4.0, "retries": 0, "seconds": 37.8, "max_memory_gb": 19.22, "ce": 1.0647613955661654, "rank": 0.0}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.010894471779465675, "loss_scale": 4.0, "retries": 0, "seconds": 39.7, "max_memory_gb": 19.22, "ce": 0.04120926247014722, "rank": 0.12091289460659027}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 1.2688422203063965, "loss_scale": 4.0, "retries": 0, "seconds": 41.6, "max_memory_gb": 19.22, "ce": 0.5825637139187165, "rank": 0.0}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 1.3941508531570435, "loss_scale": 4.0, "retries": 0, "seconds": 43.6, "max_memory_gb": 19.22, "ce": 0.687364899997192, "rank": 0.3505646586418152}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.4978418350219727, "loss_scale": 4.0, "retries": 0, "seconds": 45.6, "max_memory_gb": 19.22, "ce": 0.7871776353126734, "rank": 0.017970174551010132}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 1.3543856143951416, "loss_scale": 4.0, "retries": 0, "seconds": 47.5, "max_memory_gb": 19.22, "ce": 0.803424658253789, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 1.3695749044418335, "loss_scale": 4.0, "retries": 0, "seconds": 49.5, "max_memory_gb": 19.22, "ce": 0.897726557283022, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 1.1743727922439575, "loss_scale": 4.0, "retries": 0, "seconds": 51.5, "max_memory_gb": 19.22, "ce": 0.7525512762367725, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 1.4176228046417236, "loss_scale": 4.0, "retries": 0, "seconds": 53.5, "max_memory_gb": 19.22, "ce": 0.7120045820847736, "rank": 0.0}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 0.698060154914856, "loss_scale": 4.0, "retries": 0, "seconds": 55.5, "max_memory_gb": 19.22, "ce": 0.8876944780349731, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 1.2000170946121216, "loss_scale": 4.0, "retries": 0, "seconds": 57.4, "max_memory_gb": 19.22, "ce": 0.5576293242629617, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 1.1815687417984009, "loss_scale": 4.0, "retries": 0, "seconds": 59.3, "max_memory_gb": 19.22, "ce": 0.4897790460813667, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 1.7674254179000854, "loss_scale": 4.0, "retries": 0, "seconds": 61.3, "max_memory_gb": 19.22, "ce": 0.7885617417050526, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 1.9186447858810425, "loss_scale": 4.0, "retries": 0, "seconds": 63.2, "max_memory_gb": 19.22, "ce": 0.7509774658828974, "rank": 0.038848526775836945}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.6751506328582764, "loss_scale": 4.0, "retries": 0, "seconds": 65.1, "max_memory_gb": 19.22, "ce": 0.39310963376192376, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.7301095724105835, "loss_scale": 4.0, "retries": 0, "seconds": 67.1, "max_memory_gb": 19.22, "ce": 0.4187826936122292, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.9349300265312195, "loss_scale": 4.0, "retries": 0, "seconds": 69.0, "max_memory_gb": 19.22, "ce": 0.3570751355146058, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 0.6379501223564148, "loss_scale": 4.0, "retries": 0, "seconds": 71.0, "max_memory_gb": 19.22, "ce": 0.4152814981134725, "rank": 0.027927398681640625}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.7395309209823608, "loss_scale": 4.0, "retries": 0, "seconds": 72.9, "max_memory_gb": 19.22, "ce": 0.6942046977055725, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.6586531400680542, "loss_scale": 4.0, "retries": 0, "seconds": 74.9, "max_memory_gb": 19.22, "ce": 0.6677412372082472, "rank": 0.0673891231417656}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 0.4141908288002014, "loss_scale": 4.0, "retries": 0, "seconds": 76.9, "max_memory_gb": 19.22, "ce": 0.44759739852452185, "rank": 0.1248626559972763}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.4829695522785187, "loss_scale": 4.0, "retries": 0, "seconds": 78.8, "max_memory_gb": 19.22, "ce": 0.3192907291268057, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.6297574043273926, "loss_scale": 4.0, "retries": 0, "seconds": 80.8, "max_memory_gb": 19.22, "ce": 0.48438244475983083, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.2394160032272339, "loss_scale": 4.0, "retries": 0, "seconds": 82.6, "max_memory_gb": 19.22, "ce": 0.2320537656196393, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.6246753334999084, "loss_scale": 4.0, "retries": 0, "seconds": 84.5, "max_memory_gb": 19.22, "ce": 0.3064493826168473, "rank": 0.18748872727155685}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 1.0152392387390137, "loss_scale": 4.0, "retries": 0, "seconds": 86.5, "max_memory_gb": 19.22, "ce": 0.7370669126967186, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 0.5481081008911133, "loss_scale": 4.0, "retries": 0, "seconds": 88.4, "max_memory_gb": 19.22, "ce": 0.7896322963060811, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.4171431362628937, "loss_scale": 4.0, "retries": 0, "seconds": 90.3, "max_memory_gb": 19.22, "ce": 0.43666799273341894, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.3989890515804291, "loss_scale": 4.0, "retries": 0, "seconds": 92.2, "max_memory_gb": 19.22, "ce": 0.2582358338404447, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.5947293639183044, "loss_scale": 4.0, "retries": 0, "seconds": 94.2, "max_memory_gb": 19.22, "ce": 0.43149392813211307, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.3818880021572113, "loss_scale": 4.0, "retries": 0, "seconds": 96.2, "max_memory_gb": 19.22, "ce": 0.40889825581689365, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.6199989318847656, "loss_scale": 4.0, "retries": 0, "seconds": 98.1, "max_memory_gb": 19.22, "ce": 0.5105803364604071, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 1.0676915645599365, "loss_scale": 4.0, "retries": 0, "seconds": 99.9, "max_memory_gb": 19.22, "ce": 0.6084188287059078, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.6855641007423401, "loss_scale": 4.0, "retries": 0, "seconds": 101.8, "max_memory_gb": 19.22, "ce": 0.44793774569188827, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 0.9277637600898743, "loss_scale": 4.0, "retries": 0, "seconds": 103.7, "max_memory_gb": 19.22, "ce": 0.5875408542342484, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 0.37448650598526, "loss_scale": 4.0, "retries": 0, "seconds": 105.6, "max_memory_gb": 19.22, "ce": 0.5313065749942325, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.09293600916862488, "loss_scale": 4.0, "retries": 0, "seconds": 107.5, "max_memory_gb": 19.22, "ce": 0.22067819637595676, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.3053410053253174, "loss_scale": 4.0, "retries": 0, "seconds": 109.3, "max_memory_gb": 19.22, "ce": 0.49208762566559017, "rank": 0.0781899243593216}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.11907236278057098, "loss_scale": 4.0, "retries": 0, "seconds": 111.2, "max_memory_gb": 19.22, "ce": 0.1796349241230928, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.3051099181175232, "loss_scale": 4.0, "retries": 0, "seconds": 113.2, "max_memory_gb": 19.22, "ce": 0.2977004419299192, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.14148657023906708, "loss_scale": 4.0, "retries": 0, "seconds": 115.0, "max_memory_gb": 19.22, "ce": 0.1553968081534549, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.17172500491142273, "loss_scale": 4.0, "retries": 0, "seconds": 116.7, "max_memory_gb": 19.22, "ce": 0.2810907387465704, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.44653019309043884, "loss_scale": 4.0, "retries": 0, "seconds": 118.7, "max_memory_gb": 19.22, "ce": 0.6325446162372828, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.2472439557313919, "loss_scale": 4.0, "retries": 0, "seconds": 120.6, "max_memory_gb": 19.22, "ce": 0.18703554416424595, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.20326276123523712, "loss_scale": 4.0, "retries": 0, "seconds": 122.5, "max_memory_gb": 19.22, "ce": 0.35293348086997867, "rank": 0.07072587311267853}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.23505596816539764, "loss_scale": 4.0, "retries": 0, "seconds": 124.5, "max_memory_gb": 19.22, "ce": 0.35851032321806997, "rank": 0.11752728372812271}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 0.5523808598518372, "loss_scale": 4.0, "retries": 0, "seconds": 126.4, "max_memory_gb": 19.22, "ce": 0.5822484621312469, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.32676881551742554, "loss_scale": 4.0, "retries": 0, "seconds": 128.3, "max_memory_gb": 19.22, "ce": 0.2609684283456204, "rank": 0.12444749474525452}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.28305479884147644, "loss_scale": 4.0, "retries": 0, "seconds": 130.3, "max_memory_gb": 19.22, "ce": 0.3637023625196889, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.13872958719730377, "loss_scale": 4.0, "retries": 0, "seconds": 132.4, "max_memory_gb": 19.22, "ce": 0.1635458423425007, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.49487462639808655, "loss_scale": 4.0, "retries": 0, "seconds": 134.3, "max_memory_gb": 19.22, "ce": 0.5994542613625526, "rank": 0.020927824079990387}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.2274283468723297, "loss_scale": 4.0, "retries": 0, "seconds": 136.1, "max_memory_gb": 19.22, "ce": 0.25900148176879156, "rank": 0.12562549114227295}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.3596228063106537, "loss_scale": 4.0, "retries": 0, "seconds": 137.9, "max_memory_gb": 19.22, "ce": 0.3699012405472786, "rank": 0.024740569293498993}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.26560571789741516, "loss_scale": 4.0, "retries": 0, "seconds": 139.9, "max_memory_gb": 19.22, "ce": 0.5315895907115191, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 0.6137079000473022, "loss_scale": 4.0, "retries": 0, "seconds": 141.8, "max_memory_gb": 19.22, "ce": 0.311617529258001, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.3812532424926758, "loss_scale": 4.0, "retries": 0, "seconds": 143.7, "max_memory_gb": 19.22, "ce": 0.3266976463710307, "rank": 0.158510223031044}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.8273828625679016, "loss_scale": 4.0, "retries": 0, "seconds": 145.7, "max_memory_gb": 19.22, "ce": 0.7598117282614112, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 0.242418110370636, "loss_scale": 4.0, "retries": 0, "seconds": 147.7, "max_memory_gb": 19.22, "ce": 0.4697158207127359, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.20186053216457367, "loss_scale": 4.0, "retries": 0, "seconds": 149.7, "max_memory_gb": 19.22, "ce": 0.3497527298241039, "rank": 0.06987204402685165}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.23455949127674103, "loss_scale": 4.0, "retries": 0, "seconds": 151.6, "max_memory_gb": 19.22, "ce": 0.2881608604438952, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.23447830975055695, "loss_scale": 4.0, "retries": 0, "seconds": 153.6, "max_memory_gb": 19.22, "ce": 0.15424745157815778, "rank": 0.10303416103124619}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.21247711777687073, "loss_scale": 4.0, "retries": 0, "seconds": 155.6, "max_memory_gb": 19.22, "ce": 0.2752120460277183, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.31333062052726746, "loss_scale": 4.0, "retries": 0, "seconds": 158.1, "max_memory_gb": 19.22, "ce": 0.42197345092426986, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.3474826514720917, "loss_scale": 4.0, "retries": 0, "seconds": 160.4, "max_memory_gb": 19.22, "ce": 0.3688325481925858, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 0.3044486343860626, "loss_scale": 4.0, "retries": 0, "seconds": 162.4, "max_memory_gb": 19.22, "ce": 0.6149900018499466, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.12588562071323395, "loss_scale": 4.0, "retries": 0, "seconds": 164.9, "max_memory_gb": 19.22, "ce": 0.1309622567110864, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.8101552724838257, "loss_scale": 4.0, "retries": 0, "seconds": 167.1, "max_memory_gb": 19.22, "ce": 0.07814502709311455, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.6876180768013, "loss_scale": 4.0, "retries": 0, "seconds": 169.4, "max_memory_gb": 19.22, "ce": 0.4358478223439306, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.3141985833644867, "loss_scale": 4.0, "retries": 0, "seconds": 171.6, "max_memory_gb": 19.22, "ce": 0.44330611724581104, "rank": 0.004551947116851807}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.180338054895401, "loss_scale": 4.0, "retries": 0, "seconds": 174.1, "max_memory_gb": 19.22, "ce": 0.25003878031384374, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.08403287827968597, "loss_scale": 4.0, "retries": 0, "seconds": 176.6, "max_memory_gb": 19.22, "ce": 0.04997064955955466, "rank": 0.16595566272735596}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.36188265681266785, "loss_scale": 4.0, "retries": 0, "seconds": 179.0, "max_memory_gb": 19.22, "ce": 0.38107322725409176, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 0.8326702117919922, "loss_scale": 4.0, "retries": 0, "seconds": 181.4, "max_memory_gb": 19.22, "ce": 0.5063350652635563, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 0.7859653234481812, "loss_scale": 4.0, "retries": 0, "seconds": 183.5, "max_memory_gb": 19.22, "ce": 0.4379925225939587, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 0.476014107465744, "loss_scale": 4.0, "retries": 0, "seconds": 185.4, "max_memory_gb": 19.22, "ce": 0.3574374362360686, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 1.1507002115249634, "loss_scale": 4.0, "retries": 0, "seconds": 187.3, "max_memory_gb": 19.22, "ce": 0.5014596863184124, "rank": 0.02655906230211258}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 0.9385671019554138, "loss_scale": 4.0, "retries": 0, "seconds": 189.1, "max_memory_gb": 19.22, "ce": 0.8425525827333331, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 0.8689064383506775, "loss_scale": 4.0, "retries": 0, "seconds": 191.2, "max_memory_gb": 19.22, "ce": 0.8997623072937131, "rank": 0.545691043138504}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.6250796914100647, "loss_scale": 4.0, "retries": 0, "seconds": 193.2, "max_memory_gb": 19.22, "ce": 0.19558967248303816, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.24038536846637726, "loss_scale": 4.0, "retries": 0, "seconds": 195.4, "max_memory_gb": 19.22, "ce": 0.27822357337208814, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.34339702129364014, "loss_scale": 4.0, "retries": 0, "seconds": 197.3, "max_memory_gb": 19.22, "ce": 0.06003466217589448, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.428598016500473, "loss_scale": 4.0, "retries": 0, "seconds": 199.2, "max_memory_gb": 19.22, "ce": 0.4937247863272205, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.23746612668037415, "loss_scale": 4.0, "retries": 0, "seconds": 201.2, "max_memory_gb": 19.22, "ce": 0.3162567590479739, "rank": 0.11642998456954956}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.17477242648601532, "loss_scale": 4.0, "retries": 0, "seconds": 203.3, "max_memory_gb": 19.22, "ce": 0.2923809419735335, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.15897701680660248, "loss_scale": 4.0, "retries": 0, "seconds": 205.7, "max_memory_gb": 19.22, "ce": 0.25681582598735986, "rank": 0.09445348381996155}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.06716334074735641, "loss_scale": 4.0, "retries": 0, "seconds": 207.5, "max_memory_gb": 19.22, "ce": 0.14771429626853205, "rank": 0.12450793385505676}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.4717456102371216, "loss_scale": 4.0, "retries": 0, "seconds": 209.5, "max_memory_gb": 19.22, "ce": 0.4613266873639077, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.22845184803009033, "loss_scale": 4.0, "retries": 0, "seconds": 211.4, "max_memory_gb": 19.22, "ce": 0.3102479098015465, "rank": 0.02630656212568283}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 1.6168195009231567, "loss_scale": 4.0, "retries": 0, "seconds": 213.4, "max_memory_gb": 19.22, "ce": 0.9028420438226021, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 0.2450954020023346, "loss_scale": 4.0, "retries": 0, "seconds": 215.2, "max_memory_gb": 19.22, "ce": 0.2162666743133741, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 0.2996680736541748, "loss_scale": 4.0, "retries": 0, "seconds": 217.2, "max_memory_gb": 19.22, "ce": 0.5593322815839201, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 1.850001573562622, "loss_scale": 4.0, "retries": 0, "seconds": 219.2, "max_memory_gb": 19.22, "ce": 0.5191299833822995, "rank": 0.16320154070854187}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.3276801109313965, "loss_scale": 4.0, "retries": 0, "seconds": 221.1, "max_memory_gb": 19.22, "ce": 0.3124709624607931, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.19005751609802246, "loss_scale": 4.0, "retries": 0, "seconds": 223.1, "max_memory_gb": 19.22, "ce": 0.19849093881202862, "rank": 0.09519626945257187}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.20693695545196533, "loss_scale": 4.0, "retries": 0, "seconds": 225.1, "max_memory_gb": 19.22, "ce": 0.28752328563132323, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.22889983654022217, "loss_scale": 4.0, "retries": 0, "seconds": 227.5, "max_memory_gb": 19.22, "ce": 0.46631603431887925, "rank": 0.10065532475709915}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.18811075389385223, "loss_scale": 4.0, "retries": 0, "seconds": 229.7, "max_memory_gb": 19.22, "ce": 0.22359714602862368, "rank": 0.041939832270145416}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.5255821943283081, "loss_scale": 4.0, "retries": 0, "seconds": 231.9, "max_memory_gb": 19.22, "ce": 0.2856685219157953, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.8288289308547974, "loss_scale": 4.0, "retries": 0, "seconds": 233.9, "max_memory_gb": 19.23, "ce": 0.34340535425872076, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 0.33559495210647583, "loss_scale": 4.0, "retries": 0, "seconds": 236.0, "max_memory_gb": 19.23, "ce": 0.33116975447774166, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.13361221551895142, "loss_scale": 4.0, "retries": 0, "seconds": 237.9, "max_memory_gb": 19.23, "ce": 0.1484054398479202, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 0.8717948794364929, "loss_scale": 4.0, "retries": 0, "seconds": 239.9, "max_memory_gb": 19.23, "ce": 0.8278198500629514, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.1535162627696991, "loss_scale": 4.0, "retries": 0, "seconds": 241.8, "max_memory_gb": 19.23, "ce": 0.1834873916614015, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.20188592374324799, "loss_scale": 4.0, "retries": 0, "seconds": 243.9, "max_memory_gb": 19.23, "ce": 0.22847694600204704, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.5742639899253845, "loss_scale": 4.0, "retries": 0, "seconds": 245.9, "max_memory_gb": 19.23, "ce": 0.26233970129396766, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 0.741313636302948, "loss_scale": 4.0, "retries": 0, "seconds": 248.0, "max_memory_gb": 19.23, "ce": 0.7543683836556738, "rank": 0.2390628457069397}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.37237367033958435, "loss_scale": 4.0, "retries": 0, "seconds": 249.9, "max_memory_gb": 19.23, "ce": 0.2835328763030702, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 0.7430710196495056, "loss_scale": 4.0, "retries": 0, "seconds": 251.8, "max_memory_gb": 19.23, "ce": 0.7339011547155678, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.1944931447505951, "loss_scale": 4.0, "retries": 0, "seconds": 253.7, "max_memory_gb": 19.23, "ce": 0.13485664298787015, "rank": 0.027950547635555267}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 0.897223174571991, "loss_scale": 4.0, "retries": 0, "seconds": 256.1, "max_memory_gb": 19.23, "ce": 0.6397324190184008, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 0.2733939588069916, "loss_scale": 4.0, "retries": 0, "seconds": 258.9, "max_memory_gb": 19.23, "ce": 0.43901497730985284, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 0.32359275221824646, "loss_scale": 4.0, "retries": 0, "seconds": 260.8, "max_memory_gb": 19.23, "ce": 0.515605286025675, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 0.7134895920753479, "loss_scale": 4.0, "retries": 0, "seconds": 262.7, "max_memory_gb": 19.23, "ce": 0.24131353336269967, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 0.4565916955471039, "loss_scale": 4.0, "retries": 0, "seconds": 264.6, "max_memory_gb": 19.23, "ce": 0.4944567346938129, "rank": 0.09736371785402298}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 0.4328828454017639, "loss_scale": 4.0, "retries": 0, "seconds": 266.5, "max_memory_gb": 19.23, "ce": 0.6748652241658419, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.5186700224876404, "loss_scale": 4.0, "retries": 0, "seconds": 268.4, "max_memory_gb": 19.23, "ce": 0.2629761758121276, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 0.30721521377563477, "loss_scale": 4.0, "retries": 0, "seconds": 270.4, "max_memory_gb": 19.23, "ce": 0.37265695119276643, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 1.028083086013794, "loss_scale": 4.0, "retries": 0, "seconds": 272.3, "max_memory_gb": 19.23, "ce": 0.33318531227996573, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.1648659110069275, "loss_scale": 4.0, "retries": 0, "seconds": 274.3, "max_memory_gb": 19.23, "ce": 0.2091470659361221, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 0.8187541961669922, "loss_scale": 4.0, "retries": 0, "seconds": 276.3, "max_memory_gb": 19.23, "ce": 0.3829780104570091, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.5759442448616028, "loss_scale": 4.0, "retries": 0, "seconds": 278.2, "max_memory_gb": 19.23, "ce": 0.4542218480055453, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.17219239473342896, "loss_scale": 4.0, "retries": 0, "seconds": 280.2, "max_memory_gb": 19.23, "ce": 0.20847139185934793, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.20913410186767578, "loss_scale": 4.0, "retries": 0, "seconds": 282.1, "max_memory_gb": 19.23, "ce": 0.20473196328384802, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.44452399015426636, "loss_scale": 4.0, "retries": 0, "seconds": 284.1, "max_memory_gb": 19.23, "ce": 0.3070152559375856, "rank": 0.1047181561589241}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 0.8914951682090759, "loss_scale": 4.0, "retries": 0, "seconds": 286.1, "max_memory_gb": 19.23, "ce": 0.5510949024301226, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 1.042665719985962, "loss_scale": 4.0, "retries": 0, "seconds": 288.1, "max_memory_gb": 19.23, "ce": 0.4097807567450218, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.09885472059249878, "loss_scale": 4.0, "retries": 0, "seconds": 290.1, "max_memory_gb": 19.23, "ce": 0.19936298612992687, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.13144679367542267, "loss_scale": 4.0, "retries": 0, "seconds": 292.2, "max_memory_gb": 19.23, "ce": 0.18303702821140178, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 0.512596607208252, "loss_scale": 4.0, "retries": 0, "seconds": 294.1, "max_memory_gb": 19.23, "ce": 0.8049810028605862, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.1333722621202469, "loss_scale": 4.0, "retries": 0, "seconds": 296.0, "max_memory_gb": 19.23, "ce": 0.1274936668837654, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.20054298639297485, "loss_scale": 4.0, "retries": 0, "seconds": 298.0, "max_memory_gb": 19.23, "ce": 0.12681524982326664, "rank": 0.12532655894756317}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.2967732548713684, "loss_scale": 4.0, "retries": 0, "seconds": 300.0, "max_memory_gb": 19.23, "ce": 0.3260911542456597, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.06971834599971771, "loss_scale": 4.0, "retries": 0, "seconds": 302.2, "max_memory_gb": 19.23, "ce": 0.16249140916625038, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 0.2317570149898529, "loss_scale": 4.0, "retries": 0, "seconds": 304.3, "max_memory_gb": 19.23, "ce": 0.36389874493761454, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.18927250802516937, "loss_scale": 4.0, "retries": 0, "seconds": 306.6, "max_memory_gb": 19.23, "ce": 0.2812632326185849, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 1.7137680053710938, "loss_scale": 4.0, "retries": 0, "seconds": 308.9, "max_memory_gb": 19.23, "ce": 0.45332778384909034, "rank": 0.09536339342594147}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 0.3969351649284363, "loss_scale": 4.0, "retries": 0, "seconds": 310.9, "max_memory_gb": 19.23, "ce": 0.36681099356246705, "rank": 0.066017746925354}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.42595505714416504, "loss_scale": 4.0, "retries": 0, "seconds": 312.8, "max_memory_gb": 19.23, "ce": 0.14194252859670087, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.5494235157966614, "loss_scale": 4.0, "retries": 0, "seconds": 314.7, "max_memory_gb": 19.23, "ce": 0.15157640542747686, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 0.36138737201690674, "loss_scale": 4.0, "retries": 0, "seconds": 316.7, "max_memory_gb": 19.23, "ce": 0.3167638201848604, "rank": 0.0071667879819869995}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.22154133021831512, "loss_scale": 4.0, "retries": 0, "seconds": 318.6, "max_memory_gb": 19.36, "ce": 0.24475564376916736, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 0.6971162557601929, "loss_scale": 4.0, "retries": 0, "seconds": 320.5, "max_memory_gb": 19.36, "ce": 0.4308109477860853, "rank": 0.058072082698345184}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 0.3326045572757721, "loss_scale": 4.0, "retries": 0, "seconds": 322.4, "max_memory_gb": 19.36, "ce": 0.20382331692962907, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.38331201672554016, "loss_scale": 4.0, "retries": 0, "seconds": 324.4, "max_memory_gb": 19.36, "ce": 0.22040585743889096, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 0.9390133023262024, "loss_scale": 4.0, "retries": 0, "seconds": 326.4, "max_memory_gb": 19.36, "ce": 0.5602559660328552, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.40641406178474426, "loss_scale": 4.0, "retries": 0, "seconds": 328.4, "max_memory_gb": 19.36, "ce": 0.26578041177708656, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.062214791774749756, "loss_scale": 4.0, "retries": 0, "seconds": 330.4, "max_memory_gb": 19.36, "ce": 0.11661315039236797, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 1.0644645690917969, "loss_scale": 4.0, "retries": 0, "seconds": 332.4, "max_memory_gb": 19.36, "ce": 0.5007079905481078, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.3079162538051605, "loss_scale": 4.0, "retries": 0, "seconds": 334.3, "max_memory_gb": 19.36, "ce": 0.2766256283503026, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.15043407678604126, "loss_scale": 4.0, "retries": 0, "seconds": 336.2, "max_memory_gb": 19.36, "ce": 0.21478926981217228, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 0.4845055043697357, "loss_scale": 4.0, "retries": 0, "seconds": 338.2, "max_memory_gb": 19.36, "ce": 0.6473620589822531, "rank": 0.13069675862789154}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.21279926598072052, "loss_scale": 4.0, "retries": 0, "seconds": 340.1, "max_memory_gb": 19.36, "ce": 0.3055944500520127, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.27648642659187317, "loss_scale": 4.0, "retries": 0, "seconds": 342.1, "max_memory_gb": 19.36, "ce": 0.263254246773613, "rank": 0.04018884152173996}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.2940986156463623, "loss_scale": 4.0, "retries": 0, "seconds": 344.2, "max_memory_gb": 19.36, "ce": 0.3692210044246167, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.3162703812122345, "loss_scale": 4.0, "retries": 0, "seconds": 346.1, "max_memory_gb": 19.36, "ce": 0.3273878716863692, "rank": 0.08590389043092728}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.27208957076072693, "loss_scale": 4.0, "retries": 0, "seconds": 348.0, "max_memory_gb": 19.36, "ce": 0.2616493736841221, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 2.498347043991089, "loss_scale": 4.0, "retries": 0, "seconds": 350.0, "max_memory_gb": 19.36, "ce": 0.750804009847343, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.36527732014656067, "loss_scale": 4.0, "retries": 0, "seconds": 351.9, "max_memory_gb": 19.36, "ce": 0.31513730535516515, "rank": 0.1554998904466629}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 0.37729623913764954, "loss_scale": 4.0, "retries": 0, "seconds": 353.8, "max_memory_gb": 19.36, "ce": 0.45053085508334334, "rank": 0.12498755753040314}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 1.2518975734710693, "loss_scale": 4.0, "retries": 0, "seconds": 355.8, "max_memory_gb": 19.36, "ce": 0.26969630472012796, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 0.6145650744438171, "loss_scale": 4.0, "retries": 0, "seconds": 357.6, "max_memory_gb": 19.36, "ce": 0.6222061177504656, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.1975184828042984, "loss_scale": 4.0, "retries": 0, "seconds": 359.6, "max_memory_gb": 19.36, "ce": 0.31788430287269875, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.2744672894477844, "loss_scale": 4.0, "retries": 0, "seconds": 361.6, "max_memory_gb": 19.36, "ce": 0.38850470283068717, "rank": 0.11728943139314651}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 0.759063720703125, "loss_scale": 4.0, "retries": 0, "seconds": 363.5, "max_memory_gb": 19.36, "ce": 0.1886363974226697, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.12405700236558914, "loss_scale": 4.0, "retries": 0, "seconds": 365.5, "max_memory_gb": 19.36, "ce": 0.18960087041341467, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.1644810140132904, "loss_scale": 4.0, "retries": 0, "seconds": 367.5, "max_memory_gb": 19.36, "ce": 0.21771275082801367, "rank": 0.21210798621177673}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.310162216424942, "loss_scale": 4.0, "retries": 0, "seconds": 369.5, "max_memory_gb": 19.36, "ce": 0.33566399686969817, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 0.844744086265564, "loss_scale": 4.0, "retries": 0, "seconds": 371.5, "max_memory_gb": 19.36, "ce": 0.3539159292849945, "rank": 0.06477659940719604}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.4189724922180176, "loss_scale": 4.0, "retries": 0, "seconds": 373.5, "max_memory_gb": 19.36, "ce": 0.5694613233208656, "rank": 0.13336610794067383}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 0.8827145099639893, "loss_scale": 4.0, "retries": 0, "seconds": 375.4, "max_memory_gb": 19.36, "ce": 0.33771102767605043, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 1.0871751308441162, "loss_scale": 4.0, "retries": 0, "seconds": 377.3, "max_memory_gb": 19.36, "ce": 0.6492025512270629, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.2731454372406006, "loss_scale": 4.0, "retries": 0, "seconds": 379.2, "max_memory_gb": 19.36, "ce": 0.4094958580681123, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 2.109480857849121, "loss_scale": 4.0, "retries": 0, "seconds": 381.1, "max_memory_gb": 19.36, "ce": 0.27237896732549416, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 0.5696210265159607, "loss_scale": 4.0, "retries": 0, "seconds": 383.1, "max_memory_gb": 19.36, "ce": 0.6465660473331809, "rank": 0.0004299730062484741}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.24707356095314026, "loss_scale": 4.0, "retries": 0, "seconds": 385.1, "max_memory_gb": 19.36, "ce": 0.26875025514527806, "rank": 0.12460722029209137}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.18672135472297668, "loss_scale": 4.0, "retries": 0, "seconds": 387.0, "max_memory_gb": 19.36, "ce": 0.12897139017877635, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.3164486289024353, "loss_scale": 4.0, "retries": 0, "seconds": 389.0, "max_memory_gb": 19.36, "ce": 0.40456175373401493, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.22577859461307526, "loss_scale": 4.0, "retries": 0, "seconds": 391.0, "max_memory_gb": 19.36, "ce": 0.3289936823402968, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.594179093837738, "loss_scale": 4.0, "retries": 0, "seconds": 393.0, "max_memory_gb": 19.36, "ce": 0.4101587761833798, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.30717188119888306, "loss_scale": 4.0, "retries": 0, "seconds": 394.9, "max_memory_gb": 19.36, "ce": 0.2121642335987417, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 0.48061421513557434, "loss_scale": 4.0, "retries": 0, "seconds": 396.8, "max_memory_gb": 19.36, "ce": 0.43273583010886796, "rank": 0.05852556973695755}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 1.4347972869873047, "loss_scale": 4.0, "retries": 0, "seconds": 398.8, "max_memory_gb": 19.36, "ce": 0.5346365425502881, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.3258460760116577, "loss_scale": 4.0, "retries": 0, "seconds": 400.7, "max_memory_gb": 19.36, "ce": 0.248645011393819, "rank": 0.07897843420505524}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.10104916989803314, "loss_scale": 4.0, "retries": 0, "seconds": 402.6, "max_memory_gb": 19.36, "ce": 0.13031366508221254, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.28842324018478394, "loss_scale": 4.0, "retries": 0, "seconds": 404.6, "max_memory_gb": 19.36, "ce": 0.23407996611786075, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.26510706543922424, "loss_scale": 4.0, "retries": 0, "seconds": 406.6, "max_memory_gb": 19.36, "ce": 0.393443611334078, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 0.15650005638599396, "loss_scale": 4.0, "retries": 0, "seconds": 408.4, "max_memory_gb": 19.36, "ce": 0.3108211178914644, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.21444804966449738, "loss_scale": 4.0, "retries": 0, "seconds": 410.4, "max_memory_gb": 19.36, "ce": 0.325238146693664, "rank": 0.0726865604519844}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.05490979179739952, "loss_scale": 4.0, "retries": 0, "seconds": 412.2, "max_memory_gb": 19.36, "ce": 0.070884818356717, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.25256484746932983, "loss_scale": 4.0, "retries": 0, "seconds": 414.3, "max_memory_gb": 19.36, "ce": 0.2739713311984815, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.3113827705383301, "loss_scale": 4.0, "retries": 0, "seconds": 416.2, "max_memory_gb": 19.36, "ce": 0.2703122925013304, "rank": 0.17955727130174637}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 0.48389118909835815, "loss_scale": 4.0, "retries": 0, "seconds": 418.1, "max_memory_gb": 19.36, "ce": 0.5511911270677956, "rank": 0.12689396739006042}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.4723488688468933, "loss_scale": 4.0, "retries": 0, "seconds": 420.1, "max_memory_gb": 19.36, "ce": 0.33744383935118094, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 0.24558313190937042, "loss_scale": 4.0, "retries": 0, "seconds": 422.0, "max_memory_gb": 19.36, "ce": 0.43086106958799064, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.3328668177127838, "loss_scale": 4.0, "retries": 0, "seconds": 424.0, "max_memory_gb": 19.36, "ce": 0.3715870853338856, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 0.8839820027351379, "loss_scale": 4.0, "retries": 0, "seconds": 425.9, "max_memory_gb": 19.36, "ce": 0.46887942998000653, "rank": 0.11007729172706604}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 0.9792357683181763, "loss_scale": 4.0, "retries": 0, "seconds": 427.8, "max_memory_gb": 19.36, "ce": 0.6065514370202436, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.34653761982917786, "loss_scale": 4.0, "retries": 0, "seconds": 429.8, "max_memory_gb": 19.36, "ce": 0.132376669353107, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.21356192231178284, "loss_scale": 4.0, "retries": 0, "seconds": 431.7, "max_memory_gb": 19.36, "ce": 0.1816086970793549, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 0.9315308332443237, "loss_scale": 4.0, "retries": 0, "seconds": 433.7, "max_memory_gb": 19.36, "ce": 0.33450416289269924, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.14940983057022095, "loss_scale": 4.0, "retries": 0, "seconds": 435.6, "max_memory_gb": 19.36, "ce": 0.11586370364602772, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.19472186267375946, "loss_scale": 4.0, "retries": 0, "seconds": 437.5, "max_memory_gb": 19.36, "ce": 0.1824402055608516, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.31553226709365845, "loss_scale": 4.0, "retries": 0, "seconds": 439.5, "max_memory_gb": 19.36, "ce": 0.44184046844020486, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.25903216004371643, "loss_scale": 4.0, "retries": 0, "seconds": 441.5, "max_memory_gb": 19.36, "ce": 0.3116035441344138, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 0.2586883008480072, "loss_scale": 4.0, "retries": 0, "seconds": 443.4, "max_memory_gb": 19.36, "ce": 0.06604004415567033, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.1798987239599228, "loss_scale": 4.0, "retries": 0, "seconds": 445.4, "max_memory_gb": 19.36, "ce": 0.22013207884447183, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.20949453115463257, "loss_scale": 4.0, "retries": 0, "seconds": 447.4, "max_memory_gb": 19.36, "ce": 0.24274068884551525, "rank": 0.03620723634958267}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 1.0021202564239502, "loss_scale": 4.0, "retries": 0, "seconds": 449.4, "max_memory_gb": 19.36, "ce": 0.2732981464068871, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.3747673034667969, "loss_scale": 4.0, "retries": 0, "seconds": 451.3, "max_memory_gb": 19.36, "ce": 0.2259450200945139, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.13139787316322327, "loss_scale": 4.0, "retries": 0, "seconds": 453.3, "max_memory_gb": 19.36, "ce": 0.1385041187168099, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.5764058828353882, "loss_scale": 4.0, "retries": 0, "seconds": 455.2, "max_memory_gb": 19.36, "ce": 0.16607349796686321, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 1.7564846277236938, "loss_scale": 4.0, "retries": 0, "seconds": 457.0, "max_memory_gb": 19.36, "ce": 0.5024184982175939, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.14301137626171112, "loss_scale": 4.0, "retries": 0, "seconds": 458.9, "max_memory_gb": 19.36, "ce": 0.14398591994904564, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.252755343914032, "loss_scale": 4.0, "retries": 0, "seconds": 460.9, "max_memory_gb": 19.36, "ce": 0.43706685036886483, "rank": 0.03325986862182617}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.23193636536598206, "loss_scale": 4.0, "retries": 0, "seconds": 462.8, "max_memory_gb": 19.36, "ce": 0.30013199415407144, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.3917531967163086, "loss_scale": 4.0, "retries": 0, "seconds": 464.8, "max_memory_gb": 19.36, "ce": 0.14713187103734526, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 0.9874169230461121, "loss_scale": 4.0, "retries": 0, "seconds": 466.7, "max_memory_gb": 19.36, "ce": 0.36193708207429154, "rank": 0.0296541228890419}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.14337027072906494, "loss_scale": 4.0, "retries": 0, "seconds": 468.6, "max_memory_gb": 19.36, "ce": 0.12918876799994905, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.3292771279811859, "loss_scale": 4.0, "retries": 0, "seconds": 470.5, "max_memory_gb": 19.36, "ce": 0.2813966420289944, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.4572426676750183, "loss_scale": 4.0, "retries": 0, "seconds": 472.5, "max_memory_gb": 19.36, "ce": 0.32911665574647486, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.6578199863433838, "loss_scale": 4.0, "retries": 0, "seconds": 474.4, "max_memory_gb": 19.36, "ce": 0.37581598962424323, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 0.3584192395210266, "loss_scale": 4.0, "retries": 0, "seconds": 476.3, "max_memory_gb": 19.36, "ce": 0.3101437270961469, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.4498022198677063, "loss_scale": 4.0, "retries": 0, "seconds": 478.2, "max_memory_gb": 19.36, "ce": 0.12358897751801123, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.09406177699565887, "loss_scale": 4.0, "retries": 0, "seconds": 480.2, "max_memory_gb": 19.36, "ce": 0.10374609600694384, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.22488005459308624, "loss_scale": 4.0, "retries": 0, "seconds": 482.1, "max_memory_gb": 19.36, "ce": 0.29899447163916193, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.31114500761032104, "loss_scale": 4.0, "retries": 0, "seconds": 484.1, "max_memory_gb": 19.36, "ce": 0.25305491918697953, "rank": 0.04318663477897644}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 0.9668372869491577, "loss_scale": 4.0, "retries": 0, "seconds": 486.0, "max_memory_gb": 19.36, "ce": 0.42072327714413404, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.16395363211631775, "loss_scale": 4.0, "retries": 0, "seconds": 488.0, "max_memory_gb": 19.36, "ce": 0.18786091900437896, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.5007453560829163, "loss_scale": 4.0, "retries": 0, "seconds": 489.9, "max_memory_gb": 19.36, "ce": 0.2442198511944298, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.28923845291137695, "loss_scale": 4.0, "retries": 0, "seconds": 491.8, "max_memory_gb": 19.36, "ce": 0.2599181671976112, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.2731456160545349, "loss_scale": 4.0, "retries": 0, "seconds": 493.8, "max_memory_gb": 19.36, "ce": 0.3128208201378584, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.2220398634672165, "loss_scale": 4.0, "retries": 0, "seconds": 495.8, "max_memory_gb": 19.36, "ce": 0.2483573119243374, "rank": 0.0}

runs/single_internvl/removal/L07_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/single_internvl/removal/L07_s13/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.7.mlp.up_proj",
+    "model.language_model.layers.7.self_attn.k_proj",
+    "model.language_model.layers.7.self_attn.v_proj",
+    "model.language_model.layers.7.self_attn.o_proj",
+    "model.language_model.layers.7.mlp.gate_proj",
+    "model.language_model.layers.7.mlp.down_proj",
+    "model.language_model.layers.7.self_attn.q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/single_internvl/removal/L07_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 7,
+    "rank": 16,
+    "trainable_parameters": 1212416
+  }
+}

runs/single_internvl/removal/L07_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 0.7280412912368774, "loss_scale": 4.0, "retries": 0, "seconds": 2.7, "max_memory_gb": 18.73, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 0.6602698564529419, "loss_scale": 4.0, "retries": 0, "seconds": 4.5, "max_memory_gb": 18.74, "ce": 0.7805676178068097, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 2.659924030303955, "loss_scale": 4.0, "retries": 0, "seconds": 6.3, "max_memory_gb": 18.74, "ce": 1.3769186092540622, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 1.4639161825180054, "loss_scale": 4.0, "retries": 0, "seconds": 8.1, "max_memory_gb": 18.74, "ce": 0.7504135421741012, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.026584956794977188, "loss_scale": 4.0, "retries": 0, "seconds": 9.8, "max_memory_gb": 18.74, "ce": 0.04013556132758822, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 0.7140323519706726, "loss_scale": 4.0, "retries": 0, "seconds": 11.5, "max_memory_gb": 18.74, "ce": 0.43252502498216927, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 2.034420967102051, "loss_scale": 4.0, "retries": 0, "seconds": 13.3, "max_memory_gb": 18.74, "ce": 1.002125917289959, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 2.000894784927368, "loss_scale": 4.0, "retries": 0, "seconds": 15.1, "max_memory_gb": 18.74, "ce": 0.9630236197263002, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 1.4736469984054565, "loss_scale": 4.0, "retries": 0, "seconds": 16.8, "max_memory_gb": 18.74, "ce": 0.6837471011094749, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.21299917995929718, "loss_scale": 4.0, "retries": 0, "seconds": 18.6, "max_memory_gb": 18.75, "ce": 0.18705425973166712, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 1.8919966220855713, "loss_scale": 4.0, "retries": 0, "seconds": 20.4, "max_memory_gb": 18.75, "ce": 0.7540003604008234, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 1.4279640913009644, "loss_scale": 4.0, "retries": 0, "seconds": 22.1, "max_memory_gb": 18.75, "ce": 0.489805129232991, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 1.0017684698104858, "loss_scale": 4.0, "retries": 0, "seconds": 23.9, "max_memory_gb": 18.75, "ce": 0.594200123767223, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 1.9835431575775146, "loss_scale": 4.0, "retries": 0, "seconds": 25.6, "max_memory_gb": 18.75, "ce": 0.8676203191280365, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 1.5000241994857788, "loss_scale": 4.0, "retries": 0, "seconds": 27.3, "max_memory_gb": 18.75, "ce": 0.5506463064161835, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.6412835121154785, "loss_scale": 4.0, "retries": 0, "seconds": 29.1, "max_memory_gb": 18.75, "ce": 0.2603170577567653, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.7510098814964294, "loss_scale": 4.0, "retries": 0, "seconds": 30.8, "max_memory_gb": 18.75, "ce": 0.4972954538898193, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 1.7639445066452026, "loss_scale": 4.0, "retries": 0, "seconds": 32.6, "max_memory_gb": 18.77, "ce": 0.5290465912839863, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 2.088456392288208, "loss_scale": 4.0, "retries": 0, "seconds": 34.4, "max_memory_gb": 18.77, "ce": 0.6806325577199459, "rank": 0.0647321566939354}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.023156076669692993, "loss_scale": 4.0, "retries": 0, "seconds": 36.1, "max_memory_gb": 18.77, "ce": 0.04142289507944952, "rank": 0.11896459013223648}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.7694600820541382, "loss_scale": 4.0, "retries": 0, "seconds": 37.9, "max_memory_gb": 18.77, "ce": 0.36985271544835996, "rank": 0.005864545702934265}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.928100049495697, "loss_scale": 4.0, "retries": 0, "seconds": 39.7, "max_memory_gb": 18.77, "ce": 0.4777041482157074, "rank": 0.20051993429660797}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.423018455505371, "loss_scale": 4.0, "retries": 0, "seconds": 41.5, "max_memory_gb": 18.77, "ce": 0.5847351587940466, "rank": 0.1282394826412201}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 0.7235305309295654, "loss_scale": 4.0, "retries": 0, "seconds": 43.2, "max_memory_gb": 18.77, "ce": 0.5662764166481793, "rank": 0.1332586407661438}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 0.7527705430984497, "loss_scale": 4.0, "retries": 0, "seconds": 45.0, "max_memory_gb": 18.77, "ce": 0.6522515063043102, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 0.3783283233642578, "loss_scale": 4.0, "retries": 0, "seconds": 46.8, "max_memory_gb": 18.77, "ce": 0.5687171258032322, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 0.7332257032394409, "loss_scale": 4.0, "retries": 0, "seconds": 48.5, "max_memory_gb": 18.77, "ce": 0.48477028206252726, "rank": 0.10858781635761261}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 0.5605731010437012, "loss_scale": 4.0, "retries": 0, "seconds": 50.3, "max_memory_gb": 18.77, "ce": 0.7889201468788087, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 0.39750587940216064, "loss_scale": 4.0, "retries": 0, "seconds": 52.1, "max_memory_gb": 18.77, "ce": 0.3949824875453487, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 0.46463337540626526, "loss_scale": 4.0, "retries": 0, "seconds": 53.8, "max_memory_gb": 18.77, "ce": 0.2851874969956043, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 0.6683490872383118, "loss_scale": 4.0, "retries": 0, "seconds": 55.6, "max_memory_gb": 18.77, "ce": 0.5149045090656728, "rank": 0.10256955027580261}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 0.4897187650203705, "loss_scale": 4.0, "retries": 0, "seconds": 57.4, "max_memory_gb": 18.77, "ce": 0.4661796623840928, "rank": 0.06674288213253021}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.2667846977710724, "loss_scale": 4.0, "retries": 0, "seconds": 59.2, "max_memory_gb": 18.77, "ce": 0.2766895333188586, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.2652001678943634, "loss_scale": 4.0, "retries": 0, "seconds": 60.9, "max_memory_gb": 18.77, "ce": 0.31729647352040047, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.24838726222515106, "loss_scale": 4.0, "retries": 0, "seconds": 62.7, "max_memory_gb": 18.77, "ce": 0.21963925270392792, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 0.30566734075546265, "loss_scale": 4.0, "retries": 0, "seconds": 64.5, "max_memory_gb": 18.77, "ce": 0.3208600001526065, "rank": 0.024831220507621765}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.5174288749694824, "loss_scale": 4.0, "retries": 0, "seconds": 66.2, "max_memory_gb": 18.77, "ce": 0.5513968727027532, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.42072421312332153, "loss_scale": 4.0, "retries": 0, "seconds": 68.0, "max_memory_gb": 18.77, "ce": 0.5647797137498856, "rank": 0.02986779808998108}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 0.453694224357605, "loss_scale": 4.0, "retries": 0, "seconds": 69.7, "max_memory_gb": 18.77, "ce": 0.3910710971613298, "rank": 0.1247577890753746}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.23914726078510284, "loss_scale": 4.0, "retries": 0, "seconds": 71.5, "max_memory_gb": 18.77, "ce": 0.25175533832225483, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.7165244221687317, "loss_scale": 4.0, "retries": 0, "seconds": 73.2, "max_memory_gb": 18.77, "ce": 0.4574205061653629, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.14840032160282135, "loss_scale": 4.0, "retries": 0, "seconds": 75.0, "max_memory_gb": 18.77, "ce": 0.20401940080046188, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.23532640933990479, "loss_scale": 4.0, "retries": 0, "seconds": 76.7, "max_memory_gb": 18.77, "ce": 0.2178090986162715, "rank": 0.2061884105205536}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 1.3850127458572388, "loss_scale": 4.0, "retries": 0, "seconds": 78.4, "max_memory_gb": 18.77, "ce": 0.6749528407635808, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 0.8105053305625916, "loss_scale": 4.0, "retries": 0, "seconds": 80.2, "max_memory_gb": 18.77, "ce": 0.7331260507344268, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.503091037273407, "loss_scale": 4.0, "retries": 0, "seconds": 81.9, "max_memory_gb": 18.77, "ce": 0.3683112794533372, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.1424015760421753, "loss_scale": 4.0, "retries": 0, "seconds": 83.7, "max_memory_gb": 18.77, "ce": 0.20921688797534443, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.24158917367458344, "loss_scale": 4.0, "retries": 0, "seconds": 85.5, "max_memory_gb": 18.77, "ce": 0.33154915168415755, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.32838472723960876, "loss_scale": 4.0, "retries": 0, "seconds": 87.2, "max_memory_gb": 18.77, "ce": 0.34219912509433925, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.4821896255016327, "loss_scale": 4.0, "retries": 0, "seconds": 89.0, "max_memory_gb": 18.77, "ce": 0.3860217905639729, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 1.1632877588272095, "loss_scale": 4.0, "retries": 0, "seconds": 90.7, "max_memory_gb": 18.77, "ce": 0.5507958502512338, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.34394872188568115, "loss_scale": 4.0, "retries": 0, "seconds": 92.5, "max_memory_gb": 18.77, "ce": 0.3127290584961884, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 0.9976670742034912, "loss_scale": 4.0, "retries": 0, "seconds": 94.2, "max_memory_gb": 18.77, "ce": 0.5275446649175137, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 0.5351318120956421, "loss_scale": 4.0, "retries": 0, "seconds": 96.0, "max_memory_gb": 18.77, "ce": 0.44963908745558, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.18034417927265167, "loss_scale": 4.0, "retries": 0, "seconds": 97.7, "max_memory_gb": 18.77, "ce": 0.20189078706607688, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.5815595984458923, "loss_scale": 4.0, "retries": 0, "seconds": 99.4, "max_memory_gb": 18.77, "ce": 0.42205657809972763, "rank": 0.03328074514865875}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.13580675423145294, "loss_scale": 4.0, "retries": 0, "seconds": 101.2, "max_memory_gb": 18.77, "ce": 0.15843629717710428, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.30091023445129395, "loss_scale": 4.0, "retries": 0, "seconds": 102.9, "max_memory_gb": 18.77, "ce": 0.24102613753348123, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.15119369328022003, "loss_scale": 4.0, "retries": 0, "seconds": 104.7, "max_memory_gb": 18.77, "ce": 0.14277975669392617, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.19894419610500336, "loss_scale": 4.0, "retries": 0, "seconds": 106.4, "max_memory_gb": 18.77, "ce": 0.2534833053359762, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.5477577447891235, "loss_scale": 4.0, "retries": 0, "seconds": 108.2, "max_memory_gb": 18.77, "ce": 0.5287255644798279, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.23923088610172272, "loss_scale": 4.0, "retries": 0, "seconds": 109.9, "max_memory_gb": 18.77, "ce": 0.1683380277827382, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.32069623470306396, "loss_scale": 4.0, "retries": 0, "seconds": 111.7, "max_memory_gb": 18.77, "ce": 0.3055110811255872, "rank": 0.07356894016265869}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.2410830557346344, "loss_scale": 4.0, "retries": 0, "seconds": 113.5, "max_memory_gb": 18.77, "ce": 0.3086685196030885, "rank": 0.10780102014541626}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 0.5675851106643677, "loss_scale": 4.0, "retries": 0, "seconds": 115.3, "max_memory_gb": 18.77, "ce": 0.5009991298429668, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.5022217631340027, "loss_scale": 4.0, "retries": 0, "seconds": 117.0, "max_memory_gb": 18.77, "ce": 0.2568494655495215, "rank": 0.12447106093168259}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.23841454088687897, "loss_scale": 4.0, "retries": 0, "seconds": 118.8, "max_memory_gb": 18.77, "ce": 0.28973617329029366, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.1404077261686325, "loss_scale": 4.0, "retries": 0, "seconds": 120.6, "max_memory_gb": 18.77, "ce": 0.13852236140883178, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.6999439597129822, "loss_scale": 4.0, "retries": 0, "seconds": 122.3, "max_memory_gb": 18.77, "ce": 0.5524504974018782, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.2715592682361603, "loss_scale": 4.0, "retries": 0, "seconds": 124.0, "max_memory_gb": 18.77, "ce": 0.20031576804467477, "rank": 0.12249092757701874}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.2813583016395569, "loss_scale": 4.0, "retries": 0, "seconds": 125.8, "max_memory_gb": 18.77, "ce": 0.2891924888463109, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.3479881286621094, "loss_scale": 4.0, "retries": 0, "seconds": 127.6, "max_memory_gb": 18.77, "ce": 0.4445349029265344, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 0.5992375612258911, "loss_scale": 4.0, "retries": 0, "seconds": 129.3, "max_memory_gb": 18.77, "ce": 0.2826482262753416, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.40255528688430786, "loss_scale": 4.0, "retries": 0, "seconds": 131.0, "max_memory_gb": 18.77, "ce": 0.246445934411895, "rank": 0.08896329998970032}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.4683055579662323, "loss_scale": 4.0, "retries": 0, "seconds": 132.8, "max_memory_gb": 18.77, "ce": 0.7224365759757347, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 0.3839787244796753, "loss_scale": 4.0, "retries": 0, "seconds": 134.5, "max_memory_gb": 18.77, "ce": 0.4328866378054954, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.21490035951137543, "loss_scale": 4.0, "retries": 0, "seconds": 136.2, "max_memory_gb": 18.77, "ce": 0.30048823793185875, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.2594383656978607, "loss_scale": 4.0, "retries": 0, "seconds": 138.0, "max_memory_gb": 18.77, "ce": 0.23647434017038904, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.21432200074195862, "loss_scale": 4.0, "retries": 0, "seconds": 139.7, "max_memory_gb": 18.77, "ce": 0.12255162222572835, "rank": 0.09725994616746902}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.1909199357032776, "loss_scale": 4.0, "retries": 0, "seconds": 141.4, "max_memory_gb": 18.77, "ce": 0.2210982307960876, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.39018774032592773, "loss_scale": 4.0, "retries": 0, "seconds": 143.1, "max_memory_gb": 18.77, "ce": 0.3473365142708644, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.2755439579486847, "loss_scale": 4.0, "retries": 0, "seconds": 144.8, "max_memory_gb": 18.77, "ce": 0.28746672184206545, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 0.43855100870132446, "loss_scale": 4.0, "retries": 0, "seconds": 146.6, "max_memory_gb": 18.77, "ce": 0.5228791828558315, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.16063326597213745, "loss_scale": 4.0, "retries": 0, "seconds": 148.4, "max_memory_gb": 18.77, "ce": 0.10164174525198177, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.46250173449516296, "loss_scale": 4.0, "retries": 0, "seconds": 150.1, "max_memory_gb": 18.77, "ce": 0.05177525667386362, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.7271418571472168, "loss_scale": 4.0, "retries": 0, "seconds": 151.9, "max_memory_gb": 18.77, "ce": 0.3453686269931495, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.43832117319107056, "loss_scale": 4.0, "retries": 0, "seconds": 153.6, "max_memory_gb": 18.77, "ce": 0.36900530887942296, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.1763792634010315, "loss_scale": 4.0, "retries": 0, "seconds": 155.4, "max_memory_gb": 18.77, "ce": 0.21496751453378238, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.13252466917037964, "loss_scale": 4.0, "retries": 0, "seconds": 157.1, "max_memory_gb": 18.77, "ce": 0.04943653874943266, "rank": 0.16419656574726105}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.5035068392753601, "loss_scale": 4.0, "retries": 0, "seconds": 158.8, "max_memory_gb": 18.77, "ce": 0.3236761100124568, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 1.1087614297866821, "loss_scale": 4.0, "retries": 0, "seconds": 160.6, "max_memory_gb": 18.77, "ce": 0.5145831517875195, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 0.9764909744262695, "loss_scale": 4.0, "retries": 0, "seconds": 162.3, "max_memory_gb": 18.77, "ce": 0.34679743430388044, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 0.5391830205917358, "loss_scale": 4.0, "retries": 0, "seconds": 164.1, "max_memory_gb": 18.77, "ce": 0.2898589884862304, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 0.8174962997436523, "loss_scale": 4.0, "retries": 0, "seconds": 165.8, "max_memory_gb": 18.77, "ce": 0.35322882852051407, "rank": 0.03462009131908417}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 1.3357110023498535, "loss_scale": 4.0, "retries": 0, "seconds": 167.6, "max_memory_gb": 18.77, "ce": 0.7448838681448251, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 1.3037971258163452, "loss_scale": 4.0, "retries": 0, "seconds": 169.3, "max_memory_gb": 18.77, "ce": 0.7270673887105659, "rank": 0.46038419008255005}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.7044829726219177, "loss_scale": 4.0, "retries": 0, "seconds": 171.1, "max_memory_gb": 18.77, "ce": 0.15110594214638695, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.3329790532588959, "loss_scale": 4.0, "retries": 0, "seconds": 172.9, "max_memory_gb": 18.77, "ce": 0.2148233067127876, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.16954845190048218, "loss_scale": 4.0, "retries": 0, "seconds": 174.6, "max_memory_gb": 18.77, "ce": 0.045731222693575546, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.6073898077011108, "loss_scale": 4.0, "retries": 0, "seconds": 176.4, "max_memory_gb": 18.77, "ce": 0.39756009774282575, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.320043683052063, "loss_scale": 4.0, "retries": 0, "seconds": 178.2, "max_memory_gb": 18.77, "ce": 0.23202117541222833, "rank": 0.10033781826496124}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.49404317140579224, "loss_scale": 4.0, "retries": 0, "seconds": 180.0, "max_memory_gb": 18.77, "ce": 0.23680552508449182, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.20817679166793823, "loss_scale": 4.0, "retries": 0, "seconds": 181.7, "max_memory_gb": 18.77, "ce": 0.21918858074059244, "rank": 0.09437908232212067}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.17209365963935852, "loss_scale": 4.0, "retries": 0, "seconds": 183.4, "max_memory_gb": 18.77, "ce": 0.13365240101120435, "rank": 0.12172213196754456}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.5968709588050842, "loss_scale": 4.0, "retries": 0, "seconds": 185.2, "max_memory_gb": 18.77, "ce": 0.389633585116826, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.420850932598114, "loss_scale": 4.0, "retries": 0, "seconds": 186.9, "max_memory_gb": 18.77, "ce": 0.21563402315950952, "rank": 0.010237768292427063}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 2.6713156700134277, "loss_scale": 4.0, "retries": 0, "seconds": 188.7, "max_memory_gb": 18.77, "ce": 0.6670213990109914, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 0.46442869305610657, "loss_scale": 4.0, "retries": 0, "seconds": 190.5, "max_memory_gb": 18.77, "ce": 0.18301573699864093, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 0.6713674664497375, "loss_scale": 4.0, "retries": 0, "seconds": 192.3, "max_memory_gb": 18.77, "ce": 0.4459338572341949, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 1.8162727355957031, "loss_scale": 4.0, "retries": 0, "seconds": 194.0, "max_memory_gb": 18.77, "ce": 0.37230109237134457, "rank": 0.13489370048046112}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.476835161447525, "loss_scale": 4.0, "retries": 0, "seconds": 195.8, "max_memory_gb": 18.77, "ce": 0.23555324773769826, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.6910537481307983, "loss_scale": 4.0, "retries": 0, "seconds": 197.7, "max_memory_gb": 18.77, "ce": 0.1346832040580921, "rank": 0.0005516707897186279}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.30080872774124146, "loss_scale": 4.0, "retries": 0, "seconds": 199.5, "max_memory_gb": 18.77, "ce": 0.23143155232537538, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.6045554876327515, "loss_scale": 4.0, "retries": 0, "seconds": 201.2, "max_memory_gb": 18.77, "ce": 0.3974019580055028, "rank": 0.08783002197742462}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.4384866952896118, "loss_scale": 4.0, "retries": 0, "seconds": 203.0, "max_memory_gb": 18.77, "ce": 0.15208866851753555, "rank": 0.02554171532392502}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.5818803310394287, "loss_scale": 4.0, "retries": 0, "seconds": 204.8, "max_memory_gb": 18.77, "ce": 0.25674775935476646, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.8614542484283447, "loss_scale": 4.0, "retries": 0, "seconds": 206.5, "max_memory_gb": 18.78, "ce": 0.24114419912802987, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 0.4969964027404785, "loss_scale": 4.0, "retries": 0, "seconds": 208.3, "max_memory_gb": 18.78, "ce": 0.3022136723011499, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.2615656852722168, "loss_scale": 4.0, "retries": 0, "seconds": 210.1, "max_memory_gb": 18.78, "ce": 0.11242500680236844, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 1.5802820920944214, "loss_scale": 4.0, "retries": 0, "seconds": 211.8, "max_memory_gb": 18.78, "ce": 0.6388813925150316, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.3321360647678375, "loss_scale": 4.0, "retries": 0, "seconds": 213.6, "max_memory_gb": 18.78, "ce": 0.140509397500864, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.748989999294281, "loss_scale": 4.0, "retries": 0, "seconds": 215.3, "max_memory_gb": 18.78, "ce": 0.16187287903449032, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.7211767435073853, "loss_scale": 4.0, "retries": 0, "seconds": 217.1, "max_memory_gb": 18.78, "ce": 0.1977252479409799, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.3898913860321045, "loss_scale": 4.0, "retries": 0, "seconds": 218.9, "max_memory_gb": 18.78, "ce": 0.6632041225966532, "rank": 0.16011685132980347}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.5554641485214233, "loss_scale": 4.0, "retries": 0, "seconds": 220.6, "max_memory_gb": 18.78, "ce": 0.22529925291019026, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 1.6289114952087402, "loss_scale": 4.0, "retries": 0, "seconds": 222.4, "max_memory_gb": 18.78, "ce": 0.6134890313260257, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.31103235483169556, "loss_scale": 4.0, "retries": 0, "seconds": 224.1, "max_memory_gb": 18.78, "ce": 0.10006988232998992, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 1.211030125617981, "loss_scale": 4.0, "retries": 0, "seconds": 225.9, "max_memory_gb": 18.78, "ce": 0.4725538119673729, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 0.8651493787765503, "loss_scale": 4.0, "retries": 0, "seconds": 227.6, "max_memory_gb": 18.78, "ce": 0.30809942400082946, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 0.7761979699134827, "loss_scale": 4.0, "retries": 0, "seconds": 229.4, "max_memory_gb": 18.78, "ce": 0.3921062024892308, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 1.3280538320541382, "loss_scale": 4.0, "retries": 0, "seconds": 231.1, "max_memory_gb": 18.78, "ce": 0.2621643472084543, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 1.3577158451080322, "loss_scale": 4.0, "retries": 0, "seconds": 232.9, "max_memory_gb": 18.78, "ce": 0.3215600337716751, "rank": 0.05654802918434143}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 0.9878774285316467, "loss_scale": 4.0, "retries": 0, "seconds": 234.6, "max_memory_gb": 18.78, "ce": 0.5854957303963602, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.6583898663520813, "loss_scale": 4.0, "retries": 0, "seconds": 236.4, "max_memory_gb": 18.78, "ce": 0.2383890933647308, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 1.092604160308838, "loss_scale": 4.0, "retries": 0, "seconds": 238.1, "max_memory_gb": 18.78, "ce": 0.15574512112652883, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 1.1293048858642578, "loss_scale": 4.0, "retries": 0, "seconds": 239.8, "max_memory_gb": 18.78, "ce": 0.16954795383207966, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.7755581140518188, "loss_scale": 4.0, "retries": 0, "seconds": 241.6, "max_memory_gb": 18.78, "ce": 0.0983172075211769, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 1.0199673175811768, "loss_scale": 4.0, "retries": 0, "seconds": 243.4, "max_memory_gb": 18.78, "ce": 0.20879603526555002, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 1.3908113241195679, "loss_scale": 4.0, "retries": 0, "seconds": 245.2, "max_memory_gb": 18.78, "ce": 0.25033921777503565, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.7291543483734131, "loss_scale": 4.0, "retries": 0, "seconds": 247.0, "max_memory_gb": 18.78, "ce": 0.07500768347927078, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.717124879360199, "loss_scale": 4.0, "retries": 0, "seconds": 248.8, "max_memory_gb": 18.78, "ce": 0.05704826964210952, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 1.4488458633422852, "loss_scale": 4.0, "retries": 0, "seconds": 250.5, "max_memory_gb": 18.78, "ce": 0.16797741113987286, "rank": 0.06932565569877625}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 2.157060384750366, "loss_scale": 4.0, "retries": 0, "seconds": 252.3, "max_memory_gb": 18.78, "ce": 0.4538728598731723, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 1.3180241584777832, "loss_scale": 4.0, "retries": 0, "seconds": 254.0, "max_memory_gb": 18.78, "ce": 0.1886895394709427, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.5960597991943359, "loss_scale": 4.0, "retries": 0, "seconds": 255.8, "max_memory_gb": 18.78, "ce": 0.07672045039635123, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.6959083676338196, "loss_scale": 4.0, "retries": 0, "seconds": 257.6, "max_memory_gb": 18.78, "ce": 0.08492542096064426, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 1.7795952558517456, "loss_scale": 4.0, "retries": 0, "seconds": 259.3, "max_memory_gb": 18.78, "ce": 0.6976776312476431, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.27704906463623047, "loss_scale": 4.0, "retries": 0, "seconds": 261.1, "max_memory_gb": 18.78, "ce": 0.03504453918958461, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.5607136487960815, "loss_scale": 4.0, "retries": 0, "seconds": 262.8, "max_memory_gb": 18.78, "ce": 0.05443080281474977, "rank": 0.12525877356529236}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.7530529499053955, "loss_scale": 4.0, "retries": 0, "seconds": 264.6, "max_memory_gb": 18.78, "ce": 0.10290447215083987, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.8118016719818115, "loss_scale": 4.0, "retries": 0, "seconds": 266.4, "max_memory_gb": 18.78, "ce": 0.11807182375287084, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 1.046296238899231, "loss_scale": 4.0, "retries": 0, "seconds": 268.1, "max_memory_gb": 18.78, "ce": 0.14973095167306383, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 1.7838878631591797, "loss_scale": 4.0, "retries": 0, "seconds": 269.9, "max_memory_gb": 18.78, "ce": 0.13850131720846548, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 1.3448587656021118, "loss_scale": 4.0, "retries": 0, "seconds": 271.6, "max_memory_gb": 18.78, "ce": 0.14158729332848452, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 1.543885350227356, "loss_scale": 4.0, "retries": 0, "seconds": 273.4, "max_memory_gb": 18.78, "ce": 0.1937542891926114, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.6397945284843445, "loss_scale": 4.0, "retries": 0, "seconds": 275.2, "max_memory_gb": 18.78, "ce": 0.04509010799097268, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.46352115273475647, "loss_scale": 4.0, "retries": 0, "seconds": 276.9, "max_memory_gb": 18.78, "ce": 0.05372893292860681, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 6.871656894683838, "loss_scale": 4.0, "retries": 0, "seconds": 278.6, "max_memory_gb": 18.78, "ce": 0.2969523923848101, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 1.1735036373138428, "loss_scale": 4.0, "retries": 0, "seconds": 280.4, "max_memory_gb": 18.89, "ce": 0.08481440496325376, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 1.2286643981933594, "loss_scale": 4.0, "retries": 0, "seconds": 282.2, "max_memory_gb": 18.89, "ce": 0.18381457299983595, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 0.6732577085494995, "loss_scale": 4.0, "retries": 0, "seconds": 283.9, "max_memory_gb": 18.89, "ce": 0.10088842694949562, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.9462505578994751, "loss_scale": 4.0, "retries": 0, "seconds": 285.6, "max_memory_gb": 18.89, "ce": 0.11092818266661197, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 3.8792433738708496, "loss_scale": 4.0, "retries": 0, "seconds": 287.4, "max_memory_gb": 18.89, "ce": 0.26808318693656474, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 1.0658299922943115, "loss_scale": 4.0, "retries": 0, "seconds": 289.1, "max_memory_gb": 18.89, "ce": 0.07246697532536928, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.6980513334274292, "loss_scale": 4.0, "retries": 0, "seconds": 290.8, "max_memory_gb": 18.89, "ce": 0.039257726329196885, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 2.4181969165802, "loss_scale": 4.0, "retries": 0, "seconds": 292.4, "max_memory_gb": 18.89, "ce": 0.39036581740219845, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.40363404154777527, "loss_scale": 4.0, "retries": 0, "seconds": 294.0, "max_memory_gb": 18.89, "ce": 0.03478825889760628, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.5652129054069519, "loss_scale": 4.0, "retries": 0, "seconds": 295.5, "max_memory_gb": 18.89, "ce": 0.043335993419532315, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 1.9662724733352661, "loss_scale": 4.0, "retries": 0, "seconds": 297.1, "max_memory_gb": 18.89, "ce": 0.4641529658692889, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 1.1214745044708252, "loss_scale": 4.0, "retries": 0, "seconds": 298.7, "max_memory_gb": 18.89, "ce": 0.05975367553151045, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 1.8954530954360962, "loss_scale": 4.0, "retries": 0, "seconds": 300.3, "max_memory_gb": 18.89, "ce": 0.1576367190770327, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.9603800177574158, "loss_scale": 4.0, "retries": 0, "seconds": 301.9, "max_memory_gb": 18.89, "ce": 0.05806291487533599, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.9373462200164795, "loss_scale": 4.0, "retries": 0, "seconds": 303.4, "max_memory_gb": 18.89, "ce": 0.11752054785029031, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 1.238145351409912, "loss_scale": 4.0, "retries": 0, "seconds": 305.0, "max_memory_gb": 18.89, "ce": 0.0942039188180388, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 4.115416526794434, "loss_scale": 4.0, "retries": 0, "seconds": 306.5, "max_memory_gb": 18.89, "ce": 0.5261651044711471, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 2.261594295501709, "loss_scale": 4.0, "retries": 0, "seconds": 308.1, "max_memory_gb": 18.89, "ce": 0.03662852333582123, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 1.9963022470474243, "loss_scale": 4.0, "retries": 0, "seconds": 309.7, "max_memory_gb": 18.89, "ce": 0.3494410601997515, "rank": 0.12497936189174652}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 1.415209412574768, "loss_scale": 4.0, "retries": 0, "seconds": 311.2, "max_memory_gb": 18.89, "ce": 0.1247053485858487, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 1.6455867290496826, "loss_scale": 4.0, "retries": 0, "seconds": 312.8, "max_memory_gb": 18.89, "ce": 0.45785952038113464, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.6701025366783142, "loss_scale": 4.0, "retries": 0, "seconds": 314.3, "max_memory_gb": 18.89, "ce": 0.05634937944705598, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.8562345504760742, "loss_scale": 4.0, "retries": 0, "seconds": 315.9, "max_memory_gb": 18.89, "ce": 0.05602616400574334, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 1.0548415184020996, "loss_scale": 4.0, "retries": 0, "seconds": 317.5, "max_memory_gb": 18.89, "ce": 0.08048345016868552, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.5866363048553467, "loss_scale": 4.0, "retries": 0, "seconds": 319.1, "max_memory_gb": 18.89, "ce": 0.02814048635354993, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.2703908383846283, "loss_scale": 4.0, "retries": 0, "seconds": 320.6, "max_memory_gb": 18.89, "ce": 0.022995871165221615, "rank": 0.21379158645868301}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 1.210810661315918, "loss_scale": 4.0, "retries": 0, "seconds": 322.2, "max_memory_gb": 18.89, "ce": 0.10598033874703106, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 1.4232003688812256, "loss_scale": 4.0, "retries": 0, "seconds": 323.8, "max_memory_gb": 18.89, "ce": 0.11285570411428125, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 1.1909970045089722, "loss_scale": 4.0, "retries": 0, "seconds": 325.4, "max_memory_gb": 18.89, "ce": 0.22118855189182796, "rank": 0.11973848193883896}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.4251434803009033, "loss_scale": 4.0, "retries": 0, "seconds": 327.0, "max_memory_gb": 18.89, "ce": 0.28221494599051766, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 2.763176918029785, "loss_scale": 4.0, "retries": 0, "seconds": 328.5, "max_memory_gb": 18.89, "ce": 0.5068515813181875, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 1.3207494020462036, "loss_scale": 4.0, "retries": 0, "seconds": 330.1, "max_memory_gb": 18.89, "ce": 0.037731225391326007, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 2.7080602645874023, "loss_scale": 4.0, "retries": 0, "seconds": 331.6, "max_memory_gb": 18.89, "ce": 0.25611564976316004, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.5187851190567017, "loss_scale": 4.0, "retries": 0, "seconds": 333.2, "max_memory_gb": 18.89, "ce": 0.4803357917116955, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.5491951107978821, "loss_scale": 4.0, "retries": 0, "seconds": 334.8, "max_memory_gb": 18.89, "ce": 0.03267999015224632, "rank": 0.12500202655792236}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.48027387261390686, "loss_scale": 4.0, "retries": 0, "seconds": 336.3, "max_memory_gb": 18.89, "ce": 0.05109669481316814, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 1.3618403673171997, "loss_scale": 4.0, "retries": 0, "seconds": 337.9, "max_memory_gb": 18.89, "ce": 0.1504674850966694, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.979463517665863, "loss_scale": 4.0, "retries": 0, "seconds": 339.5, "max_memory_gb": 18.89, "ce": 0.056670072171868924, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 1.1361547708511353, "loss_scale": 4.0, "retries": 0, "seconds": 341.1, "max_memory_gb": 18.89, "ce": 0.2670027100321022, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.47596001625061035, "loss_scale": 4.0, "retries": 0, "seconds": 342.7, "max_memory_gb": 18.89, "ce": 0.06983153617329663, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 2.899630069732666, "loss_scale": 4.0, "retries": 0, "seconds": 344.3, "max_memory_gb": 18.89, "ce": 0.17065846281548147, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 2.020488977432251, "loss_scale": 4.0, "retries": 0, "seconds": 345.8, "max_memory_gb": 18.89, "ce": 0.35851735885080416, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.7191687226295471, "loss_scale": 4.0, "retries": 0, "seconds": 347.4, "max_memory_gb": 18.89, "ce": 0.0789688380809821, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.30568352341651917, "loss_scale": 4.0, "retries": 0, "seconds": 348.9, "max_memory_gb": 18.89, "ce": 0.022261551345650332, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.22786159813404083, "loss_scale": 4.0, "retries": 0, "seconds": 350.5, "max_memory_gb": 18.89, "ce": 0.025865120551316068, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 1.0945582389831543, "loss_scale": 4.0, "retries": 0, "seconds": 352.1, "max_memory_gb": 18.89, "ce": 0.05066691181855276, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 9.57360553741455, "loss_scale": 4.0, "retries": 0, "seconds": 353.6, "max_memory_gb": 18.89, "ce": 0.287236648335238, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.810791015625, "loss_scale": 4.0, "retries": 0, "seconds": 355.2, "max_memory_gb": 18.89, "ce": 0.04335619449147998, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.23035722970962524, "loss_scale": 4.0, "retries": 0, "seconds": 356.8, "max_memory_gb": 18.89, "ce": 0.05914806369037251, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.4221954047679901, "loss_scale": 4.0, "retries": 0, "seconds": 358.4, "max_memory_gb": 18.89, "ce": 0.02482702678298665, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.766227126121521, "loss_scale": 4.0, "retries": 0, "seconds": 360.0, "max_memory_gb": 18.89, "ce": 0.11100014671501413, "rank": 0.12485962361097336}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 2.5294079780578613, "loss_scale": 4.0, "retries": 0, "seconds": 361.5, "max_memory_gb": 18.89, "ce": 0.46468512920728244, "rank": 0.14375156164169312}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.6824791431427002, "loss_scale": 4.0, "retries": 0, "seconds": 363.1, "max_memory_gb": 18.89, "ce": 0.15755013321177103, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 1.3860135078430176, "loss_scale": 4.0, "retries": 0, "seconds": 364.7, "max_memory_gb": 18.89, "ce": 0.2522824544867035, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.6857856512069702, "loss_scale": 4.0, "retries": 0, "seconds": 366.3, "max_memory_gb": 18.89, "ce": 0.20198620151131763, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 3.164492607116699, "loss_scale": 4.0, "retries": 0, "seconds": 367.9, "max_memory_gb": 18.89, "ce": 0.2879702347836428, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 2.5996651649475098, "loss_scale": 4.0, "retries": 0, "seconds": 369.5, "max_memory_gb": 18.89, "ce": 0.5860653765048482, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.5065958499908447, "loss_scale": 4.0, "retries": 0, "seconds": 371.0, "max_memory_gb": 18.89, "ce": 0.03482003135013656, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.23067453503608704, "loss_scale": 4.0, "retries": 0, "seconds": 372.6, "max_memory_gb": 18.89, "ce": 0.020135859711444937, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 0.7044851183891296, "loss_scale": 4.0, "retries": 0, "seconds": 374.2, "max_memory_gb": 18.89, "ce": 0.17325934924883768, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.2603335678577423, "loss_scale": 4.0, "retries": 0, "seconds": 375.7, "max_memory_gb": 18.89, "ce": 0.03404516447130845, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 1.1513606309890747, "loss_scale": 4.0, "retries": 0, "seconds": 377.3, "max_memory_gb": 18.89, "ce": 0.08039422906040272, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 1.131055474281311, "loss_scale": 4.0, "retries": 0, "seconds": 378.9, "max_memory_gb": 18.89, "ce": 0.1251026678364724, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 1.3126778602600098, "loss_scale": 4.0, "retries": 0, "seconds": 380.5, "max_memory_gb": 18.89, "ce": 0.05751183558277262, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 0.6687389612197876, "loss_scale": 4.0, "retries": 0, "seconds": 382.0, "max_memory_gb": 18.89, "ce": 0.04216309747152991, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.5864762663841248, "loss_scale": 4.0, "retries": 0, "seconds": 383.6, "max_memory_gb": 18.89, "ce": 0.0505765689904365, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.8190295696258545, "loss_scale": 4.0, "retries": 0, "seconds": 385.2, "max_memory_gb": 18.89, "ce": 0.03920721646500169, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.9438312649726868, "loss_scale": 4.0, "retries": 0, "seconds": 386.8, "max_memory_gb": 18.89, "ce": 0.08825548886579782, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.6905035376548767, "loss_scale": 4.0, "retries": 0, "seconds": 388.3, "max_memory_gb": 18.89, "ce": 0.06395789142788999, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.8711163401603699, "loss_scale": 4.0, "retries": 0, "seconds": 389.9, "max_memory_gb": 18.89, "ce": 0.040706297610995534, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.3813354969024658, "loss_scale": 4.0, "retries": 0, "seconds": 391.4, "max_memory_gb": 18.89, "ce": 0.054109724929730874, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 2.7822368144989014, "loss_scale": 4.0, "retries": 0, "seconds": 393.0, "max_memory_gb": 18.89, "ce": 0.46323246086831205, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.5189505219459534, "loss_scale": 4.0, "retries": 0, "seconds": 394.5, "max_memory_gb": 18.89, "ce": 0.045684920631174464, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 1.543986201286316, "loss_scale": 4.0, "retries": 0, "seconds": 396.1, "max_memory_gb": 18.89, "ce": 0.05398104678897653, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.9620116353034973, "loss_scale": 4.0, "retries": 0, "seconds": 397.7, "max_memory_gb": 18.89, "ce": 0.05429482180988998, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.6709856390953064, "loss_scale": 4.0, "retries": 0, "seconds": 399.2, "max_memory_gb": 18.89, "ce": 0.053733609509436064, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 1.7369097471237183, "loss_scale": 4.0, "retries": 0, "seconds": 400.8, "max_memory_gb": 18.89, "ce": 0.2707445339201513, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.24671104550361633, "loss_scale": 4.0, "retries": 0, "seconds": 402.4, "max_memory_gb": 18.89, "ce": 0.03236644098524266, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.5379636883735657, "loss_scale": 4.0, "retries": 0, "seconds": 403.9, "max_memory_gb": 18.89, "ce": 0.0581105103410664, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.5830275416374207, "loss_scale": 4.0, "retries": 0, "seconds": 405.5, "max_memory_gb": 18.89, "ce": 0.050865684024756774, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.939509391784668, "loss_scale": 4.0, "retries": 0, "seconds": 407.0, "max_memory_gb": 18.89, "ce": 0.1565592972910963, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 2.3364224433898926, "loss_scale": 4.0, "retries": 0, "seconds": 408.6, "max_memory_gb": 18.89, "ce": 0.11412292425802661, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 1.3207147121429443, "loss_scale": 4.0, "retries": 0, "seconds": 410.2, "max_memory_gb": 18.89, "ce": 0.09873768941429262, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.4341607093811035, "loss_scale": 4.0, "retries": 0, "seconds": 411.7, "max_memory_gb": 18.89, "ce": 0.025318105640508293, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.5943960547447205, "loss_scale": 4.0, "retries": 0, "seconds": 413.3, "max_memory_gb": 18.89, "ce": 0.023614312099198287, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.2847386300563812, "loss_scale": 4.0, "retries": 0, "seconds": 414.9, "max_memory_gb": 18.89, "ce": 0.03605866864381824, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 1.6012688875198364, "loss_scale": 4.0, "retries": 0, "seconds": 416.4, "max_memory_gb": 18.89, "ce": 0.3131775876390748, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.3967633843421936, "loss_scale": 4.0, "retries": 0, "seconds": 418.0, "max_memory_gb": 18.89, "ce": 0.021076891953271115, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 10.96203899383545, "loss_scale": 4.0, "retries": 0, "seconds": 419.6, "max_memory_gb": 18.89, "ce": 0.3237349515090955, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 1.6874895095825195, "loss_scale": 4.0, "retries": 0, "seconds": 421.1, "max_memory_gb": 18.89, "ce": 0.049384636207832955, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.8962497711181641, "loss_scale": 4.0, "retries": 0, "seconds": 422.7, "max_memory_gb": 18.89, "ce": 0.04442092060344294, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.7396087050437927, "loss_scale": 4.0, "retries": 0, "seconds": 424.3, "max_memory_gb": 18.89, "ce": 0.044071072108636145, "rank": 0.0}

runs/single_internvl/removal/L09_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/single_internvl/removal/L09_s13/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.9.self_attn.q_proj",
+    "model.language_model.layers.9.mlp.gate_proj",
+    "model.language_model.layers.9.self_attn.k_proj",
+    "model.language_model.layers.9.mlp.up_proj",
+    "model.language_model.layers.9.self_attn.v_proj",
+    "model.language_model.layers.9.mlp.down_proj",
+    "model.language_model.layers.9.self_attn.o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/single_internvl/removal/L09_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 9,
+    "rank": 16,
+    "trainable_parameters": 1212416
+  }
+}

runs/single_internvl/removal/L09_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 0.7543995976448059, "loss_scale": 4.0, "retries": 0, "seconds": 2.7, "max_memory_gb": 18.55, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 0.6416022181510925, "loss_scale": 4.0, "retries": 0, "seconds": 4.2, "max_memory_gb": 18.56, "ce": 0.7814205223439785, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 2.710921049118042, "loss_scale": 4.0, "retries": 0, "seconds": 5.8, "max_memory_gb": 18.56, "ce": 1.3828201200813055, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 1.5249555110931396, "loss_scale": 4.0, "retries": 0, "seconds": 7.3, "max_memory_gb": 18.56, "ce": 0.7545164185357862, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.02311723120510578, "loss_scale": 4.0, "retries": 0, "seconds": 8.9, "max_memory_gb": 18.56, "ce": 0.040148938823222124, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 0.7228443026542664, "loss_scale": 4.0, "retries": 0, "seconds": 10.4, "max_memory_gb": 18.56, "ce": 0.4324813985003857, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 2.1337873935699463, "loss_scale": 4.0, "retries": 0, "seconds": 12.0, "max_memory_gb": 18.56, "ce": 1.003888204042596, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 2.0813872814178467, "loss_scale": 4.0, "retries": 0, "seconds": 13.6, "max_memory_gb": 18.56, "ce": 0.9661723282188177, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 1.5647242069244385, "loss_scale": 4.0, "retries": 0, "seconds": 15.1, "max_memory_gb": 18.56, "ce": 0.6755358916707337, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.21872130036354065, "loss_scale": 4.0, "retries": 0, "seconds": 16.7, "max_memory_gb": 18.57, "ce": 0.18699902802472934, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 2.007392406463623, "loss_scale": 4.0, "retries": 0, "seconds": 18.4, "max_memory_gb": 18.57, "ce": 0.7475600971811218, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 1.515763282775879, "loss_scale": 4.0, "retries": 0, "seconds": 20.0, "max_memory_gb": 18.57, "ce": 0.4829831558818114, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 1.0506727695465088, "loss_scale": 4.0, "retries": 0, "seconds": 21.5, "max_memory_gb": 18.57, "ce": 0.5859361175444064, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 2.032449722290039, "loss_scale": 4.0, "retries": 0, "seconds": 23.1, "max_memory_gb": 18.57, "ce": 0.8455384206026793, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 1.4873822927474976, "loss_scale": 4.0, "retries": 0, "seconds": 24.7, "max_memory_gb": 18.57, "ce": 0.5313351931533816, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.668851375579834, "loss_scale": 4.0, "retries": 0, "seconds": 26.2, "max_memory_gb": 18.57, "ce": 0.24676874429860618, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.7347823977470398, "loss_scale": 4.0, "retries": 0, "seconds": 27.8, "max_memory_gb": 18.57, "ce": 0.48004995126393624, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 1.533940076828003, "loss_scale": 4.0, "retries": 0, "seconds": 29.3, "max_memory_gb": 18.58, "ce": 0.48240949245519005, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 1.6043577194213867, "loss_scale": 4.0, "retries": 0, "seconds": 30.9, "max_memory_gb": 18.58, "ce": 0.6132125733420253, "rank": 0.07705918699502945}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.024001849815249443, "loss_scale": 4.0, "retries": 0, "seconds": 32.5, "max_memory_gb": 18.58, "ce": 0.041344011266119196, "rank": 0.11870098114013672}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.4871397614479065, "loss_scale": 4.0, "retries": 0, "seconds": 34.0, "max_memory_gb": 18.58, "ce": 0.33722275119180267, "rank": 0.036650970578193665}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.6302933096885681, "loss_scale": 4.0, "retries": 0, "seconds": 35.6, "max_memory_gb": 18.58, "ce": 0.4460374590707943, "rank": 0.17618878185749054}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.421855092048645, "loss_scale": 4.0, "retries": 0, "seconds": 37.2, "max_memory_gb": 18.58, "ce": 0.5573755651657848, "rank": 0.12814076244831085}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 0.5248526334762573, "loss_scale": 4.0, "retries": 0, "seconds": 38.7, "max_memory_gb": 18.58, "ce": 0.5364871786441654, "rank": 0.13481393456459045}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 0.5808108448982239, "loss_scale": 4.0, "retries": 0, "seconds": 40.3, "max_memory_gb": 18.58, "ce": 0.6171847608347889, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 0.3036867380142212, "loss_scale": 4.0, "retries": 0, "seconds": 41.9, "max_memory_gb": 18.58, "ce": 0.5506936104502529, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 0.30763277411460876, "loss_scale": 4.0, "retries": 0, "seconds": 43.4, "max_memory_gb": 18.58, "ce": 0.4685051072301576, "rank": 0.27109185606241226}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 0.5616766810417175, "loss_scale": 4.0, "retries": 0, "seconds": 45.0, "max_memory_gb": 18.58, "ce": 0.7720589018426836, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 0.3862423300743103, "loss_scale": 4.0, "retries": 0, "seconds": 46.6, "max_memory_gb": 18.58, "ce": 0.37936915626050904, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 0.32243627309799194, "loss_scale": 4.0, "retries": 0, "seconds": 48.1, "max_memory_gb": 18.58, "ce": 0.2596255997668777, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 0.6121974587440491, "loss_scale": 4.0, "retries": 0, "seconds": 49.7, "max_memory_gb": 18.58, "ce": 0.4963022917509079, "rank": 0.10278445482254028}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 0.38696014881134033, "loss_scale": 4.0, "retries": 0, "seconds": 51.3, "max_memory_gb": 18.58, "ce": 0.4474957458442077, "rank": 0.060065120458602905}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.2211615890264511, "loss_scale": 4.0, "retries": 0, "seconds": 52.8, "max_memory_gb": 18.58, "ce": 0.26666323497192934, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.2591150999069214, "loss_scale": 4.0, "retries": 0, "seconds": 54.4, "max_memory_gb": 18.58, "ce": 0.3076986506675894, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.21283546090126038, "loss_scale": 4.0, "retries": 0, "seconds": 55.9, "max_memory_gb": 18.58, "ce": 0.20442772578826407, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 0.3014560341835022, "loss_scale": 4.0, "retries": 0, "seconds": 57.5, "max_memory_gb": 18.58, "ce": 0.30796779569936916, "rank": 0.03170394152402878}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.4543769657611847, "loss_scale": 4.0, "retries": 0, "seconds": 59.1, "max_memory_gb": 18.58, "ce": 0.5264338240376674, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.45662736892700195, "loss_scale": 4.0, "retries": 0, "seconds": 60.6, "max_memory_gb": 18.58, "ce": 0.5400587837211788, "rank": 0.03278350830078125}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 0.5377368330955505, "loss_scale": 4.0, "retries": 0, "seconds": 62.2, "max_memory_gb": 18.58, "ce": 0.3799314603820676, "rank": 0.1247239038348198}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.24263536930084229, "loss_scale": 4.0, "retries": 0, "seconds": 63.7, "max_memory_gb": 18.58, "ce": 0.24087118557508802, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.7530121207237244, "loss_scale": 4.0, "retries": 0, "seconds": 65.3, "max_memory_gb": 18.58, "ce": 0.450658291898435, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.17641596496105194, "loss_scale": 4.0, "retries": 0, "seconds": 66.8, "max_memory_gb": 18.58, "ce": 0.19917097085999558, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.22540098428726196, "loss_scale": 4.0, "retries": 0, "seconds": 68.4, "max_memory_gb": 18.58, "ce": 0.20815965939254966, "rank": 0.2057780846953392}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 1.4628568887710571, "loss_scale": 4.0, "retries": 0, "seconds": 69.9, "max_memory_gb": 18.58, "ce": 0.6568334736512043, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 0.8801754713058472, "loss_scale": 4.0, "retries": 0, "seconds": 71.5, "max_memory_gb": 18.58, "ce": 0.7015722946380265, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.4774005711078644, "loss_scale": 4.0, "retries": 0, "seconds": 73.0, "max_memory_gb": 18.58, "ce": 0.3545247479341924, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.14322303235530853, "loss_scale": 4.0, "retries": 0, "seconds": 74.6, "max_memory_gb": 18.58, "ce": 0.20268770295660943, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.2599898874759674, "loss_scale": 4.0, "retries": 0, "seconds": 76.2, "max_memory_gb": 18.58, "ce": 0.31598146248143166, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.3700732886791229, "loss_scale": 4.0, "retries": 0, "seconds": 77.8, "max_memory_gb": 18.58, "ce": 0.3164969601421035, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.4879654049873352, "loss_scale": 4.0, "retries": 0, "seconds": 79.3, "max_memory_gb": 18.58, "ce": 0.36545178802043665, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 1.1846070289611816, "loss_scale": 4.0, "retries": 0, "seconds": 80.9, "max_memory_gb": 18.58, "ce": 0.5179121986511745, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.3666009306907654, "loss_scale": 4.0, "retries": 0, "seconds": 82.5, "max_memory_gb": 18.58, "ce": 0.2908782253944082, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 1.0369997024536133, "loss_scale": 4.0, "retries": 0, "seconds": 84.0, "max_memory_gb": 18.58, "ce": 0.5162904473254457, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 0.6289287805557251, "loss_scale": 4.0, "retries": 0, "seconds": 85.6, "max_memory_gb": 18.58, "ce": 0.42725256754783913, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.20874762535095215, "loss_scale": 4.0, "retries": 0, "seconds": 87.1, "max_memory_gb": 18.58, "ce": 0.19004251995647792, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.6473643183708191, "loss_scale": 4.0, "retries": 0, "seconds": 88.7, "max_memory_gb": 18.58, "ce": 0.39179440937004983, "rank": 0.04721163213253021}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.18161818385124207, "loss_scale": 4.0, "retries": 0, "seconds": 90.2, "max_memory_gb": 18.58, "ce": 0.15228776982985437, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.43808355927467346, "loss_scale": 4.0, "retries": 0, "seconds": 91.8, "max_memory_gb": 18.58, "ce": 0.2240643408440519, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.18386439979076385, "loss_scale": 4.0, "retries": 0, "seconds": 93.4, "max_memory_gb": 18.58, "ce": 0.13865662722673733, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.2437189221382141, "loss_scale": 4.0, "retries": 0, "seconds": 94.9, "max_memory_gb": 18.58, "ce": 0.244141903182026, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.5332450866699219, "loss_scale": 4.0, "retries": 0, "seconds": 96.5, "max_memory_gb": 18.58, "ce": 0.5107823621947318, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.273278146982193, "loss_scale": 4.0, "retries": 0, "seconds": 98.0, "max_memory_gb": 18.58, "ce": 0.1613027299172245, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.38301295042037964, "loss_scale": 4.0, "retries": 0, "seconds": 99.6, "max_memory_gb": 18.58, "ce": 0.2838007982354611, "rank": 0.0657183974981308}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.3089781105518341, "loss_scale": 4.0, "retries": 0, "seconds": 101.2, "max_memory_gb": 18.58, "ce": 0.2914643825497478, "rank": 0.10708414763212204}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 0.6200557947158813, "loss_scale": 4.0, "retries": 0, "seconds": 102.8, "max_memory_gb": 18.58, "ce": 0.4804286570288241, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.5639777779579163, "loss_scale": 4.0, "retries": 0, "seconds": 104.3, "max_memory_gb": 18.58, "ce": 0.26631213016207766, "rank": 0.12434913963079453}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.26879435777664185, "loss_scale": 4.0, "retries": 0, "seconds": 105.9, "max_memory_gb": 18.58, "ce": 0.26592805262771435, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.15845631062984467, "loss_scale": 4.0, "retries": 0, "seconds": 107.5, "max_memory_gb": 18.58, "ce": 0.12961186798929702, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.8575606346130371, "loss_scale": 4.0, "retries": 0, "seconds": 109.0, "max_memory_gb": 18.58, "ce": 0.5252500108908862, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.27169880270957947, "loss_scale": 4.0, "retries": 0, "seconds": 110.6, "max_memory_gb": 18.58, "ce": 0.18745781644247472, "rank": 0.12246641516685486}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.3345454931259155, "loss_scale": 4.0, "retries": 0, "seconds": 112.1, "max_memory_gb": 18.58, "ce": 0.2594764248242427, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.40286606550216675, "loss_scale": 4.0, "retries": 0, "seconds": 113.7, "max_memory_gb": 18.58, "ce": 0.41542033036239445, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 0.620808482170105, "loss_scale": 4.0, "retries": 0, "seconds": 115.2, "max_memory_gb": 18.58, "ce": 0.2725189244665671, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.2590886652469635, "loss_scale": 4.0, "retries": 0, "seconds": 116.8, "max_memory_gb": 18.58, "ce": 0.23428629586851457, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.6934884786605835, "loss_scale": 4.0, "retries": 0, "seconds": 118.6, "max_memory_gb": 18.58, "ce": 0.7087785127514508, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 0.5126237869262695, "loss_scale": 4.0, "retries": 0, "seconds": 120.4, "max_memory_gb": 18.58, "ce": 0.41275040910113603, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.33915916085243225, "loss_scale": 4.0, "retries": 0, "seconds": 122.1, "max_memory_gb": 18.58, "ce": 0.27546848374186084, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.33778682351112366, "loss_scale": 4.0, "retries": 0, "seconds": 123.8, "max_memory_gb": 18.58, "ce": 0.2294466638704762, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.20751775801181793, "loss_scale": 4.0, "retries": 0, "seconds": 125.6, "max_memory_gb": 18.58, "ce": 0.10864625619069557, "rank": 0.08695998787879944}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.25385189056396484, "loss_scale": 4.0, "retries": 0, "seconds": 127.3, "max_memory_gb": 18.58, "ce": 0.19832119000784587, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.5212660431861877, "loss_scale": 4.0, "retries": 0, "seconds": 129.1, "max_memory_gb": 18.58, "ce": 0.3089844831265509, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.3386540412902832, "loss_scale": 4.0, "retries": 0, "seconds": 130.8, "max_memory_gb": 18.58, "ce": 0.25648196728434414, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 0.6311894655227661, "loss_scale": 4.0, "retries": 0, "seconds": 132.6, "max_memory_gb": 18.58, "ce": 0.4758852444792865, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.24657544493675232, "loss_scale": 4.0, "retries": 0, "seconds": 134.3, "max_memory_gb": 18.58, "ce": 0.09120387763687177, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.4354928731918335, "loss_scale": 4.0, "retries": 0, "seconds": 136.1, "max_memory_gb": 18.58, "ce": 0.045226340254885145, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.7115737795829773, "loss_scale": 4.0, "retries": 0, "seconds": 137.8, "max_memory_gb": 18.58, "ce": 0.32020136271603405, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.5655451416969299, "loss_scale": 4.0, "retries": 0, "seconds": 139.5, "max_memory_gb": 18.58, "ce": 0.3344766911031911, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.37863972783088684, "loss_scale": 4.0, "retries": 0, "seconds": 141.3, "max_memory_gb": 18.58, "ce": 0.19351738109253347, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.19548627734184265, "loss_scale": 4.0, "retries": 0, "seconds": 143.0, "max_memory_gb": 18.58, "ce": 0.04806159864347137, "rank": 0.1594434678554535}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.7532394528388977, "loss_scale": 4.0, "retries": 0, "seconds": 144.7, "max_memory_gb": 18.58, "ce": 0.28826080047292635, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 1.3999477624893188, "loss_scale": 4.0, "retries": 0, "seconds": 146.5, "max_memory_gb": 18.58, "ce": 0.5226982820458943, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 1.0311373472213745, "loss_scale": 4.0, "retries": 0, "seconds": 148.2, "max_memory_gb": 18.58, "ce": 0.3112851196419797, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 0.671779215335846, "loss_scale": 4.0, "retries": 0, "seconds": 150.0, "max_memory_gb": 18.58, "ce": 0.24972512177191675, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 0.8623047471046448, "loss_scale": 4.0, "retries": 0, "seconds": 151.7, "max_memory_gb": 18.58, "ce": 0.2930115811759606, "rank": 0.03974246233701706}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 1.7161544561386108, "loss_scale": 4.0, "retries": 0, "seconds": 153.5, "max_memory_gb": 18.58, "ce": 0.7003710276912898, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 1.6682885885238647, "loss_scale": 4.0, "retries": 0, "seconds": 155.2, "max_memory_gb": 18.58, "ce": 0.6316474254708737, "rank": 0.4280747026205063}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.4806556701660156, "loss_scale": 4.0, "retries": 0, "seconds": 157.0, "max_memory_gb": 18.58, "ce": 0.1210110621759668, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.4767279028892517, "loss_scale": 4.0, "retries": 0, "seconds": 158.7, "max_memory_gb": 18.58, "ce": 0.16302741669642273, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.14305171370506287, "loss_scale": 4.0, "retries": 0, "seconds": 160.4, "max_memory_gb": 18.58, "ce": 0.039614889732547454, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.9237738251686096, "loss_scale": 4.0, "retries": 0, "seconds": 162.2, "max_memory_gb": 18.58, "ce": 0.33644724963232875, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.9285507798194885, "loss_scale": 4.0, "retries": 0, "seconds": 164.0, "max_memory_gb": 18.58, "ce": 0.16651424826704897, "rank": 0.04755371809005737}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.9534357190132141, "loss_scale": 4.0, "retries": 0, "seconds": 165.7, "max_memory_gb": 18.58, "ce": 0.17036294334684499, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.5063989758491516, "loss_scale": 4.0, "retries": 0, "seconds": 167.5, "max_memory_gb": 18.58, "ce": 0.1843034202902345, "rank": 0.07354094088077545}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.3011360466480255, "loss_scale": 4.0, "retries": 0, "seconds": 169.1, "max_memory_gb": 18.58, "ce": 0.111441146073048, "rank": 0.12473249435424805}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 1.051462173461914, "loss_scale": 4.0, "retries": 0, "seconds": 170.9, "max_memory_gb": 18.58, "ce": 0.27290861308574677, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.8317955136299133, "loss_scale": 4.0, "retries": 0, "seconds": 172.6, "max_memory_gb": 18.58, "ce": 0.12441386005957611, "rank": 0.013904042541980743}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 3.284635066986084, "loss_scale": 4.0, "retries": 0, "seconds": 174.4, "max_memory_gb": 18.58, "ce": 0.4421225530841184, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 0.9601719379425049, "loss_scale": 4.0, "retries": 0, "seconds": 176.1, "max_memory_gb": 18.58, "ce": 0.15033719036728144, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 1.2685720920562744, "loss_scale": 4.0, "retries": 0, "seconds": 177.9, "max_memory_gb": 18.58, "ce": 0.30476398835889995, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 1.9594980478286743, "loss_scale": 4.0, "retries": 0, "seconds": 179.6, "max_memory_gb": 18.58, "ce": 0.24345143209211528, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.8307963609695435, "loss_scale": 4.0, "retries": 0, "seconds": 181.4, "max_memory_gb": 18.58, "ce": 0.13904483437363524, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.7300843000411987, "loss_scale": 4.0, "retries": 0, "seconds": 183.1, "max_memory_gb": 18.58, "ce": 0.047814471246965695, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.5879043936729431, "loss_scale": 4.0, "retries": 0, "seconds": 184.9, "max_memory_gb": 18.58, "ce": 0.15567166125401855, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 1.1425201892852783, "loss_scale": 4.0, "retries": 0, "seconds": 186.6, "max_memory_gb": 18.58, "ce": 0.3188790201675147, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.7709266543388367, "loss_scale": 4.0, "retries": 0, "seconds": 188.4, "max_memory_gb": 18.58, "ce": 0.04224138213612605, "rank": 0.00992131233215332}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.7626210451126099, "loss_scale": 4.0, "retries": 0, "seconds": 190.1, "max_memory_gb": 18.58, "ce": 0.21950002931043855, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.947064220905304, "loss_scale": 4.0, "retries": 0, "seconds": 191.8, "max_memory_gb": 18.59, "ce": 0.14105960370034154, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 1.2632253170013428, "loss_scale": 4.0, "retries": 0, "seconds": 193.6, "max_memory_gb": 18.59, "ce": 0.23055260236105823, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.5949615836143494, "loss_scale": 4.0, "retries": 0, "seconds": 195.3, "max_memory_gb": 18.59, "ce": 0.03646732199104008, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 2.8944664001464844, "loss_scale": 4.0, "retries": 0, "seconds": 197.1, "max_memory_gb": 18.59, "ce": 0.47137270320490643, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.4908672273159027, "loss_scale": 4.0, "retries": 0, "seconds": 198.8, "max_memory_gb": 18.59, "ce": 0.0663078216834947, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.7267575860023499, "loss_scale": 4.0, "retries": 0, "seconds": 200.5, "max_memory_gb": 18.59, "ce": 0.09805659938501776, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 1.2826615571975708, "loss_scale": 4.0, "retries": 0, "seconds": 202.3, "max_memory_gb": 18.59, "ce": 0.13325490918941796, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.8771839141845703, "loss_scale": 4.0, "retries": 0, "seconds": 204.0, "max_memory_gb": 18.59, "ce": 0.551762698458333, "rank": 0.0}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.7576889395713806, "loss_scale": 4.0, "retries": 0, "seconds": 205.7, "max_memory_gb": 18.59, "ce": 0.14827236751443706, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 2.752695322036743, "loss_scale": 4.0, "retries": 0, "seconds": 207.4, "max_memory_gb": 18.59, "ce": 0.3694095353712328, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.5775076150894165, "loss_scale": 4.0, "retries": 0, "seconds": 209.2, "max_memory_gb": 18.59, "ce": 0.021021468307480973, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 2.2891764640808105, "loss_scale": 4.0, "retries": 0, "seconds": 210.9, "max_memory_gb": 18.59, "ce": 0.28545005022806436, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 1.8248380422592163, "loss_scale": 4.0, "retries": 0, "seconds": 212.6, "max_memory_gb": 18.59, "ce": 0.11149697765358724, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 4.171327590942383, "loss_scale": 4.0, "retries": 0, "seconds": 214.4, "max_memory_gb": 18.59, "ce": 0.2405056491261348, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 2.5104925632476807, "loss_scale": 4.0, "retries": 0, "seconds": 216.1, "max_memory_gb": 18.59, "ce": 0.24837408552411944, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 4.709761142730713, "loss_scale": 4.0, "retries": 0, "seconds": 217.8, "max_memory_gb": 18.59, "ce": 0.24831716104654333, "rank": 0.026130713522434235}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 2.5611038208007812, "loss_scale": 4.0, "retries": 0, "seconds": 219.5, "max_memory_gb": 18.59, "ce": 0.4867795640020631, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 1.2199972867965698, "loss_scale": 4.0, "retries": 0, "seconds": 221.3, "max_memory_gb": 18.59, "ce": 0.18310562090312033, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 1.9315632581710815, "loss_scale": 4.0, "retries": 0, "seconds": 223.0, "max_memory_gb": 18.59, "ce": 0.06954068315098993, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 1.214080572128296, "loss_scale": 4.0, "retries": 0, "seconds": 224.7, "max_memory_gb": 18.59, "ce": 0.07630430511926534, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.4210933446884155, "loss_scale": 4.0, "retries": 0, "seconds": 226.5, "max_memory_gb": 18.59, "ce": 0.027974047261523083, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 2.600457191467285, "loss_scale": 4.0, "retries": 0, "seconds": 228.2, "max_memory_gb": 18.59, "ce": 0.14145167759852484, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 1.257625699043274, "loss_scale": 4.0, "retries": 0, "seconds": 230.0, "max_memory_gb": 18.59, "ce": 0.08729730907361954, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 1.0175340175628662, "loss_scale": 4.0, "retries": 0, "seconds": 231.8, "max_memory_gb": 18.59, "ce": 0.024379843795031775, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.2970535457134247, "loss_scale": 4.0, "retries": 0, "seconds": 233.6, "max_memory_gb": 18.59, "ce": 0.016794077506347094, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 1.0021778345108032, "loss_scale": 4.0, "retries": 0, "seconds": 235.4, "max_memory_gb": 18.59, "ce": 0.059635044512106106, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 2.954472541809082, "loss_scale": 4.0, "retries": 0, "seconds": 237.2, "max_memory_gb": 18.59, "ce": 0.39040609447192764, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 1.692773461341858, "loss_scale": 4.0, "retries": 0, "seconds": 238.9, "max_memory_gb": 18.59, "ce": 0.11199899116763845, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.46306636929512024, "loss_scale": 4.0, "retries": 0, "seconds": 240.7, "max_memory_gb": 18.59, "ce": 0.02101874131403747, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.29075154662132263, "loss_scale": 4.0, "retries": 0, "seconds": 242.4, "max_memory_gb": 18.59, "ce": 0.025546834825945552, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 4.2325263023376465, "loss_scale": 4.0, "retries": 0, "seconds": 244.2, "max_memory_gb": 18.59, "ce": 0.6912002578901593, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.39263948798179626, "loss_scale": 4.0, "retries": 0, "seconds": 245.9, "max_memory_gb": 18.59, "ce": 0.011657189079869568, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 1.6277316808700562, "loss_scale": 4.0, "retries": 0, "seconds": 247.6, "max_memory_gb": 18.59, "ce": 0.03365121188835474, "rank": 0.12522903084754944}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 1.5451048612594604, "loss_scale": 4.0, "retries": 0, "seconds": 249.4, "max_memory_gb": 18.59, "ce": 0.076481170282932, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 1.5350239276885986, "loss_scale": 4.0, "retries": 0, "seconds": 251.1, "max_memory_gb": 18.59, "ce": 0.041901562608472886, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 2.1270382404327393, "loss_scale": 4.0, "retries": 0, "seconds": 252.9, "max_memory_gb": 18.59, "ce": 0.08590223408100428, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 2.4576637744903564, "loss_scale": 4.0, "retries": 0, "seconds": 254.6, "max_memory_gb": 18.59, "ce": 0.09017638156183239, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 1.4209280014038086, "loss_scale": 4.0, "retries": 0, "seconds": 256.3, "max_memory_gb": 18.59, "ce": 0.0749380689157988, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 3.2090444564819336, "loss_scale": 4.0, "retries": 0, "seconds": 258.1, "max_memory_gb": 18.59, "ce": 0.17067891575061367, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 8.215410232543945, "loss_scale": 4.0, "retries": 0, "seconds": 259.8, "max_memory_gb": 18.59, "ce": 0.0544306554565992, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.3772270381450653, "loss_scale": 4.0, "retries": 0, "seconds": 261.6, "max_memory_gb": 18.59, "ce": 0.023656158158701146, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 13.567941665649414, "loss_scale": 4.0, "retries": 0, "seconds": 263.3, "max_memory_gb": 18.59, "ce": 0.6206497665771167, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.6200212240219116, "loss_scale": 4.0, "retries": 0, "seconds": 265.1, "max_memory_gb": 18.7, "ce": 0.020706810137198772, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 1.721535086631775, "loss_scale": 4.0, "retries": 0, "seconds": 266.8, "max_memory_gb": 18.7, "ce": 0.18225443673145492, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 0.6055293679237366, "loss_scale": 4.0, "retries": 0, "seconds": 268.6, "max_memory_gb": 18.7, "ce": 0.06606870534415066, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.8994426131248474, "loss_scale": 4.0, "retries": 0, "seconds": 270.3, "max_memory_gb": 18.7, "ce": 0.04863754073448945, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 2.312920093536377, "loss_scale": 4.0, "retries": 0, "seconds": 272.0, "max_memory_gb": 18.7, "ce": 0.195081090379972, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 3.688744306564331, "loss_scale": 4.0, "retries": 0, "seconds": 273.8, "max_memory_gb": 18.7, "ce": 0.08307071555736911, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.2113742232322693, "loss_scale": 4.0, "retries": 0, "seconds": 275.6, "max_memory_gb": 18.7, "ce": 0.014303327494417317, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 3.558340549468994, "loss_scale": 4.0, "retries": 0, "seconds": 277.3, "max_memory_gb": 18.7, "ce": 0.4326562324567931, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 1.64910888671875, "loss_scale": 4.0, "retries": 0, "seconds": 279.1, "max_memory_gb": 18.7, "ce": 0.023423623279086314, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.32677584886550903, "loss_scale": 4.0, "retries": 0, "seconds": 280.8, "max_memory_gb": 18.7, "ce": 0.01355771480302792, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 2.5130648612976074, "loss_scale": 4.0, "retries": 0, "seconds": 282.6, "max_memory_gb": 18.7, "ce": 0.4282032222399721, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.8300533294677734, "loss_scale": 4.0, "retries": 0, "seconds": 284.3, "max_memory_gb": 18.7, "ce": 0.012340030909172128, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 1.8463306427001953, "loss_scale": 4.0, "retries": 0, "seconds": 286.1, "max_memory_gb": 18.7, "ce": 0.10475202710449594, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.4384109079837799, "loss_scale": 4.0, "retries": 0, "seconds": 287.9, "max_memory_gb": 18.7, "ce": 0.019694187853019685, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.9431163668632507, "loss_scale": 4.0, "retries": 0, "seconds": 289.6, "max_memory_gb": 18.7, "ce": 0.0517203774797963, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 1.018452525138855, "loss_scale": 4.0, "retries": 0, "seconds": 291.3, "max_memory_gb": 18.7, "ce": 0.055564306331689295, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 6.636721611022949, "loss_scale": 4.0, "retries": 0, "seconds": 293.0, "max_memory_gb": 18.7, "ce": 0.4161482908821199, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.19607335329055786, "loss_scale": 4.0, "retries": 0, "seconds": 294.8, "max_memory_gb": 18.7, "ce": 0.007064996723784134, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 2.8538992404937744, "loss_scale": 4.0, "retries": 0, "seconds": 296.5, "max_memory_gb": 18.7, "ce": 0.2956946274553047, "rank": 0.12493453919887543}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 0.956466019153595, "loss_scale": 4.0, "retries": 0, "seconds": 298.2, "max_memory_gb": 18.7, "ce": 0.09720158301934134, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 2.506632089614868, "loss_scale": 4.0, "retries": 0, "seconds": 299.9, "max_memory_gb": 18.7, "ce": 0.36743179540985693, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.9948664307594299, "loss_scale": 4.0, "retries": 0, "seconds": 301.7, "max_memory_gb": 18.7, "ce": 0.04237953534902772, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.9980185627937317, "loss_scale": 4.0, "retries": 0, "seconds": 303.4, "max_memory_gb": 18.7, "ce": 0.027373231598176062, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 1.4388618469238281, "loss_scale": 4.0, "retries": 0, "seconds": 305.1, "max_memory_gb": 18.7, "ce": 0.07493838468053582, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.5880610942840576, "loss_scale": 4.0, "retries": 0, "seconds": 306.9, "max_memory_gb": 18.7, "ce": 0.009994940171281996, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 1.419072151184082, "loss_scale": 4.0, "retries": 0, "seconds": 308.6, "max_memory_gb": 18.7, "ce": 0.021565675336660206, "rank": 0.1931474283337593}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 1.511480689048767, "loss_scale": 4.0, "retries": 0, "seconds": 310.4, "max_memory_gb": 18.7, "ce": 0.09634963662483642, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 1.023084044456482, "loss_scale": 4.0, "retries": 0, "seconds": 312.1, "max_memory_gb": 18.7, "ce": 0.05303637853830878, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 1.6645289659500122, "loss_scale": 4.0, "retries": 0, "seconds": 313.9, "max_memory_gb": 18.7, "ce": 0.1944109307223698, "rank": 0.14116686582565308}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.553003191947937, "loss_scale": 4.0, "retries": 0, "seconds": 315.6, "max_memory_gb": 18.7, "ce": 0.24165654086164068, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 3.867243766784668, "loss_scale": 4.0, "retries": 0, "seconds": 317.3, "max_memory_gb": 18.7, "ce": 0.4326672101742588, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.39119192957878113, "loss_scale": 4.0, "retries": 0, "seconds": 319.1, "max_memory_gb": 18.7, "ce": 0.010696259247197304, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 3.447802782058716, "loss_scale": 4.0, "retries": 0, "seconds": 320.8, "max_memory_gb": 18.7, "ce": 0.2779179939025198, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 2.3928892612457275, "loss_scale": 4.0, "retries": 0, "seconds": 322.5, "max_memory_gb": 18.7, "ce": 0.47537971234851284, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.40686747431755066, "loss_scale": 4.0, "retries": 0, "seconds": 324.3, "max_memory_gb": 18.7, "ce": 0.0074625007991926395, "rank": 0.1249937191605568}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.8779225945472717, "loss_scale": 4.0, "retries": 0, "seconds": 326.0, "max_memory_gb": 18.7, "ce": 0.04084922730544349, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 2.7937843799591064, "loss_scale": 4.0, "retries": 0, "seconds": 327.7, "max_memory_gb": 18.7, "ce": 0.11866571516293334, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 1.2559932470321655, "loss_scale": 4.0, "retries": 0, "seconds": 329.5, "max_memory_gb": 18.7, "ce": 0.028818718010370503, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 1.4414153099060059, "loss_scale": 4.0, "retries": 0, "seconds": 331.3, "max_memory_gb": 18.7, "ce": 0.23362062140222406, "rank": 0.01877322793006897}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 1.3002506494522095, "loss_scale": 4.0, "retries": 0, "seconds": 333.0, "max_memory_gb": 18.7, "ce": 0.058686459524324164, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 4.591490745544434, "loss_scale": 4.0, "retries": 0, "seconds": 334.8, "max_memory_gb": 18.7, "ce": 0.10092101457121316, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 1.5835415124893188, "loss_scale": 4.0, "retries": 0, "seconds": 336.5, "max_memory_gb": 18.7, "ce": 0.2940264851495158, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 1.0536357164382935, "loss_scale": 4.0, "retries": 0, "seconds": 338.3, "max_memory_gb": 18.7, "ce": 0.07540821247857821, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.20387710630893707, "loss_scale": 4.0, "retries": 0, "seconds": 339.9, "max_memory_gb": 18.7, "ce": 0.005112571354857209, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.24892321228981018, "loss_scale": 4.0, "retries": 0, "seconds": 341.7, "max_memory_gb": 18.7, "ce": 0.011468681659607682, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 3.8767333030700684, "loss_scale": 4.0, "retries": 0, "seconds": 343.4, "max_memory_gb": 18.7, "ce": 0.0334848009806592, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 19.60105323791504, "loss_scale": 4.0, "retries": 0, "seconds": 345.2, "max_memory_gb": 18.7, "ce": 0.40587184257219633, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.9321170449256897, "loss_scale": 4.0, "retries": 0, "seconds": 346.9, "max_memory_gb": 18.7, "ce": 0.016821638082546997, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.422160804271698, "loss_scale": 4.0, "retries": 0, "seconds": 348.6, "max_memory_gb": 18.7, "ce": 0.04763737169923843, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.33723658323287964, "loss_scale": 4.0, "retries": 0, "seconds": 350.4, "max_memory_gb": 18.7, "ce": 0.0061857426899223356, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 1.0254932641983032, "loss_scale": 4.0, "retries": 0, "seconds": 352.1, "max_memory_gb": 18.7, "ce": 0.09163807475852082, "rank": 0.12425046414136887}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 3.2618398666381836, "loss_scale": 4.0, "retries": 0, "seconds": 353.9, "max_memory_gb": 18.7, "ce": 0.49260379060979176, "rank": 0.0}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.9955758452415466, "loss_scale": 4.0, "retries": 0, "seconds": 355.6, "max_memory_gb": 18.7, "ce": 0.14516113090940053, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 2.2430014610290527, "loss_scale": 4.0, "retries": 0, "seconds": 357.4, "max_memory_gb": 18.7, "ce": 0.30327545958425617, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.8501502275466919, "loss_scale": 4.0, "retries": 0, "seconds": 359.2, "max_memory_gb": 18.7, "ce": 0.20182899930478015, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 3.790506362915039, "loss_scale": 4.0, "retries": 0, "seconds": 360.9, "max_memory_gb": 18.7, "ce": 0.285087730975647, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 3.6315577030181885, "loss_scale": 4.0, "retries": 0, "seconds": 362.7, "max_memory_gb": 18.7, "ce": 0.6279572168823506, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.8864130973815918, "loss_scale": 4.0, "retries": 0, "seconds": 364.4, "max_memory_gb": 18.7, "ce": 0.03021424219286928, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.29694393277168274, "loss_scale": 4.0, "retries": 0, "seconds": 366.2, "max_memory_gb": 18.7, "ce": 0.004566016466924339, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 1.550883412361145, "loss_scale": 4.0, "retries": 0, "seconds": 367.9, "max_memory_gb": 18.7, "ce": 0.16193472570739686, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.31426382064819336, "loss_scale": 4.0, "retries": 0, "seconds": 369.6, "max_memory_gb": 18.7, "ce": 0.023761248059599893, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 1.464160442352295, "loss_scale": 4.0, "retries": 0, "seconds": 371.3, "max_memory_gb": 18.7, "ce": 0.05058957319715773, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 1.5560804605484009, "loss_scale": 4.0, "retries": 0, "seconds": 373.1, "max_memory_gb": 18.7, "ce": 0.08976838493254036, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 1.3241064548492432, "loss_scale": 4.0, "retries": 0, "seconds": 374.9, "max_memory_gb": 18.7, "ce": 0.027848335545058944, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 2.2933194637298584, "loss_scale": 4.0, "retries": 0, "seconds": 376.6, "max_memory_gb": 18.7, "ce": 0.07853811919176223, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.4656839370727539, "loss_scale": 4.0, "retries": 0, "seconds": 378.3, "max_memory_gb": 18.7, "ce": 0.030788503278017743, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 1.259965181350708, "loss_scale": 4.0, "retries": 0, "seconds": 380.0, "max_memory_gb": 18.7, "ce": 0.023419558401656104, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.945668637752533, "loss_scale": 4.0, "retries": 0, "seconds": 381.8, "max_memory_gb": 18.7, "ce": 0.06234394840339519, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.8804069757461548, "loss_scale": 4.0, "retries": 0, "seconds": 383.5, "max_memory_gb": 18.7, "ce": 0.030323025570396567, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.5732883810997009, "loss_scale": 4.0, "retries": 0, "seconds": 385.2, "max_memory_gb": 18.7, "ce": 0.014870840941512142, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.5291584134101868, "loss_scale": 4.0, "retries": 0, "seconds": 387.0, "max_memory_gb": 18.7, "ce": 0.047934035130310804, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 4.25327205657959, "loss_scale": 4.0, "retries": 0, "seconds": 388.6, "max_memory_gb": 18.7, "ce": 0.46752650318376254, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.898131251335144, "loss_scale": 4.0, "retries": 0, "seconds": 390.3, "max_memory_gb": 18.7, "ce": 0.04153807175316615, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 2.0612621307373047, "loss_scale": 4.0, "retries": 0, "seconds": 392.1, "max_memory_gb": 18.7, "ce": 0.027596775093115866, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 1.2815977334976196, "loss_scale": 4.0, "retries": 0, "seconds": 393.9, "max_memory_gb": 18.7, "ce": 0.02409704548335867, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.7979245185852051, "loss_scale": 4.0, "retries": 0, "seconds": 395.6, "max_memory_gb": 18.7, "ce": 0.03701874051330378, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 2.2043561935424805, "loss_scale": 4.0, "retries": 0, "seconds": 397.3, "max_memory_gb": 18.7, "ce": 0.23289976492742426, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.1965366005897522, "loss_scale": 4.0, "retries": 0, "seconds": 399.0, "max_memory_gb": 18.7, "ce": 0.011454761725872231, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 3.9235804080963135, "loss_scale": 4.0, "retries": 0, "seconds": 400.8, "max_memory_gb": 18.7, "ce": 0.05642776681634132, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.5975024700164795, "loss_scale": 4.0, "retries": 0, "seconds": 402.5, "max_memory_gb": 18.7, "ce": 0.026983332380041247, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 1.1423717737197876, "loss_scale": 4.0, "retries": 0, "seconds": 404.2, "max_memory_gb": 18.7, "ce": 0.11378230302943848, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 1.997336745262146, "loss_scale": 4.0, "retries": 0, "seconds": 405.9, "max_memory_gb": 18.7, "ce": 0.08641817345414893, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 1.6496323347091675, "loss_scale": 4.0, "retries": 0, "seconds": 407.7, "max_memory_gb": 18.7, "ce": 0.10635815050272868, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.32952091097831726, "loss_scale": 4.0, "retries": 0, "seconds": 409.4, "max_memory_gb": 18.7, "ce": 0.014275260932663514, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.3816908001899719, "loss_scale": 4.0, "retries": 0, "seconds": 411.2, "max_memory_gb": 18.7, "ce": 0.004906526344257145, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.28472062945365906, "loss_scale": 4.0, "retries": 0, "seconds": 412.9, "max_memory_gb": 18.7, "ce": 0.023373901945888065, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 1.729025959968567, "loss_scale": 4.0, "retries": 0, "seconds": 414.7, "max_memory_gb": 18.7, "ce": 0.33487480070471065, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.13845276832580566, "loss_scale": 4.0, "retries": 0, "seconds": 416.4, "max_memory_gb": 18.7, "ce": 0.0048102162345458055, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 8.140676498413086, "loss_scale": 4.0, "retries": 0, "seconds": 418.1, "max_memory_gb": 18.7, "ce": 0.15525174633717143, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.7860394716262817, "loss_scale": 4.0, "retries": 0, "seconds": 419.8, "max_memory_gb": 18.7, "ce": 0.02222868055832805, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.5870615243911743, "loss_scale": 4.0, "retries": 0, "seconds": 421.6, "max_memory_gb": 18.7, "ce": 0.016921360627748072, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.5509512424468994, "loss_scale": 4.0, "retries": 0, "seconds": 423.3, "max_memory_gb": 18.7, "ce": 0.022710363184160087, "rank": 0.0}

runs/single_internvl/removal/L11_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/single_internvl/removal/L11_s13/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.11.self_attn.k_proj",
+    "model.language_model.layers.11.self_attn.q_proj",
+    "model.language_model.layers.11.self_attn.o_proj",
+    "model.language_model.layers.11.self_attn.v_proj",
+    "model.language_model.layers.11.mlp.up_proj",
+    "model.language_model.layers.11.mlp.gate_proj",
+    "model.language_model.layers.11.mlp.down_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/single_internvl/removal/L11_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 11,
+    "rank": 16,
+    "trainable_parameters": 1212416
+  }
+}

runs/single_internvl/removal/L11_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 0.8583835363388062, "loss_scale": 4.0, "retries": 0, "seconds": 2.3, "max_memory_gb": 18.37, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 0.7312538623809814, "loss_scale": 4.0, "retries": 0, "seconds": 3.8, "max_memory_gb": 18.38, "ce": 0.7821123394860479, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 2.7896690368652344, "loss_scale": 4.0, "retries": 0, "seconds": 5.4, "max_memory_gb": 18.38, "ce": 1.3802493680268526, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 1.5515573024749756, "loss_scale": 4.0, "retries": 0, "seconds": 6.9, "max_memory_gb": 18.38, "ce": 0.7522179709703778, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.024985255673527718, "loss_scale": 4.0, "retries": 0, "seconds": 8.5, "max_memory_gb": 18.38, "ce": 0.04016260440039332, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 0.8062506914138794, "loss_scale": 4.0, "retries": 0, "seconds": 10.0, "max_memory_gb": 18.38, "ce": 0.4327859465338406, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 2.2404019832611084, "loss_scale": 4.0, "retries": 0, "seconds": 11.5, "max_memory_gb": 18.38, "ce": 1.0015089469470695, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 2.219301700592041, "loss_scale": 4.0, "retries": 0, "seconds": 13.1, "max_memory_gb": 18.38, "ce": 0.9611502727493644, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 1.6578450202941895, "loss_scale": 4.0, "retries": 0, "seconds": 14.6, "max_memory_gb": 18.38, "ce": 0.6736251921392977, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.24618017673492432, "loss_scale": 4.0, "retries": 0, "seconds": 16.1, "max_memory_gb": 18.39, "ce": 0.18572181192575954, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 2.0930185317993164, "loss_scale": 4.0, "retries": 0, "seconds": 17.7, "max_memory_gb": 18.39, "ce": 0.7382279995363206, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 1.532218337059021, "loss_scale": 4.0, "retries": 0, "seconds": 19.3, "max_memory_gb": 18.39, "ce": 0.4763216118362834, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 1.0905665159225464, "loss_scale": 4.0, "retries": 0, "seconds": 20.8, "max_memory_gb": 18.39, "ce": 0.5774984821200633, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 2.1294479370117188, "loss_scale": 4.0, "retries": 0, "seconds": 22.3, "max_memory_gb": 18.39, "ce": 0.8290050039067864, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 1.4887118339538574, "loss_scale": 4.0, "retries": 0, "seconds": 23.9, "max_memory_gb": 18.39, "ce": 0.517184425115147, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.6804677248001099, "loss_scale": 4.0, "retries": 0, "seconds": 25.4, "max_memory_gb": 18.39, "ce": 0.23792082120053237, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.7941493988037109, "loss_scale": 4.0, "retries": 0, "seconds": 26.9, "max_memory_gb": 18.39, "ce": 0.47091457129863556, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 1.4345842599868774, "loss_scale": 4.0, "retries": 0, "seconds": 28.5, "max_memory_gb": 18.4, "ce": 0.4571814418886788, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 1.4478727579116821, "loss_scale": 4.0, "retries": 0, "seconds": 30.0, "max_memory_gb": 18.4, "ce": 0.5896463189274073, "rank": 0.08144105970859528}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.02612871117889881, "loss_scale": 4.0, "retries": 0, "seconds": 31.5, "max_memory_gb": 18.4, "ce": 0.04126762959640473, "rank": 0.11984061449766159}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.3885555863380432, "loss_scale": 4.0, "retries": 0, "seconds": 33.1, "max_memory_gb": 18.4, "ce": 0.32524072899104794, "rank": 0.045568063855171204}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.5244256258010864, "loss_scale": 4.0, "retries": 0, "seconds": 34.7, "max_memory_gb": 18.4, "ce": 0.4371236486331327, "rank": 0.16702193021774292}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.9865769147872925, "loss_scale": 4.0, "retries": 0, "seconds": 36.2, "max_memory_gb": 18.4, "ce": 0.5377287059977789, "rank": 0.12618087232112885}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 0.4601915776729584, "loss_scale": 4.0, "retries": 0, "seconds": 37.7, "max_memory_gb": 18.4, "ce": 0.523694243049249, "rank": 0.13302886486053467}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 0.5563321113586426, "loss_scale": 4.0, "retries": 0, "seconds": 39.3, "max_memory_gb": 18.4, "ce": 0.6120486503787106, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 0.29462480545043945, "loss_scale": 4.0, "retries": 0, "seconds": 40.9, "max_memory_gb": 18.4, "ce": 0.5453642283100635, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 0.28729555010795593, "loss_scale": 4.0, "retries": 0, "seconds": 42.4, "max_memory_gb": 18.4, "ce": 0.46258722682250664, "rank": 0.28429999202489853}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 0.64333575963974, "loss_scale": 4.0, "retries": 0, "seconds": 44.0, "max_memory_gb": 18.4, "ce": 0.770089341327548, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 0.544593095779419, "loss_scale": 4.0, "retries": 0, "seconds": 45.5, "max_memory_gb": 18.4, "ce": 0.38132081244839355, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 0.29051199555397034, "loss_scale": 4.0, "retries": 0, "seconds": 47.0, "max_memory_gb": 18.4, "ce": 0.2538471767102237, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 0.6199575662612915, "loss_scale": 4.0, "retries": 0, "seconds": 48.6, "max_memory_gb": 18.4, "ce": 0.48376742168329656, "rank": 0.10241959989070892}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 0.3597923219203949, "loss_scale": 4.0, "retries": 0, "seconds": 50.2, "max_memory_gb": 18.4, "ce": 0.4399727461859584, "rank": 0.06148791313171387}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.2334100604057312, "loss_scale": 4.0, "retries": 0, "seconds": 51.7, "max_memory_gb": 18.4, "ce": 0.2661782763316296, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.2597646415233612, "loss_scale": 4.0, "retries": 0, "seconds": 53.3, "max_memory_gb": 18.4, "ce": 0.3063270371167164, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.20165830850601196, "loss_scale": 4.0, "retries": 0, "seconds": 54.8, "max_memory_gb": 18.4, "ce": 0.20309903034649324, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 0.3412189185619354, "loss_scale": 4.0, "retries": 0, "seconds": 56.4, "max_memory_gb": 18.4, "ce": 0.31161391705973074, "rank": 0.028541654348373413}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.4953363835811615, "loss_scale": 4.0, "retries": 0, "seconds": 57.9, "max_memory_gb": 18.4, "ce": 0.5244853220065124, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.41445425152778625, "loss_scale": 4.0, "retries": 0, "seconds": 59.4, "max_memory_gb": 18.4, "ce": 0.5416842373088002, "rank": 0.026775076985359192}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 0.5482542514801025, "loss_scale": 4.0, "retries": 0, "seconds": 61.0, "max_memory_gb": 18.4, "ce": 0.3788248448399827, "rank": 0.12489809095859528}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.2505883276462555, "loss_scale": 4.0, "retries": 0, "seconds": 62.5, "max_memory_gb": 18.4, "ce": 0.2370156926481286, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.9041618704795837, "loss_scale": 4.0, "retries": 0, "seconds": 64.0, "max_memory_gb": 18.4, "ce": 0.4518239774624817, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.185525044798851, "loss_scale": 4.0, "retries": 0, "seconds": 65.6, "max_memory_gb": 18.4, "ce": 0.19932049565250054, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.22765003144741058, "loss_scale": 4.0, "retries": 0, "seconds": 67.1, "max_memory_gb": 18.4, "ce": 0.20757864193001296, "rank": 0.20281939953565598}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 1.7114897966384888, "loss_scale": 4.0, "retries": 0, "seconds": 68.6, "max_memory_gb": 18.4, "ce": 0.6394938917983382, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 1.0144599676132202, "loss_scale": 4.0, "retries": 0, "seconds": 70.2, "max_memory_gb": 18.4, "ce": 0.6899393553030677, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.5410241484642029, "loss_scale": 4.0, "retries": 0, "seconds": 71.7, "max_memory_gb": 18.4, "ce": 0.34431388886878267, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.15766723453998566, "loss_scale": 4.0, "retries": 0, "seconds": 73.2, "max_memory_gb": 18.4, "ce": 0.2032859390892554, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.23753513395786285, "loss_scale": 4.0, "retries": 0, "seconds": 74.8, "max_memory_gb": 18.4, "ce": 0.3128515320131555, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.39974308013916016, "loss_scale": 4.0, "retries": 0, "seconds": 76.3, "max_memory_gb": 18.4, "ce": 0.31997470029091346, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.4905294179916382, "loss_scale": 4.0, "retries": 0, "seconds": 77.9, "max_memory_gb": 18.4, "ce": 0.3548133016411157, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 1.4015729427337646, "loss_scale": 4.0, "retries": 0, "seconds": 79.4, "max_memory_gb": 18.4, "ce": 0.5103737394310883, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.40285900235176086, "loss_scale": 4.0, "retries": 0, "seconds": 80.9, "max_memory_gb": 18.4, "ce": 0.2858925772452494, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 1.0485901832580566, "loss_scale": 4.0, "retries": 0, "seconds": 82.5, "max_memory_gb": 18.4, "ce": 0.4900573423365131, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 0.597388505935669, "loss_scale": 4.0, "retries": 0, "seconds": 84.1, "max_memory_gb": 18.4, "ce": 0.41572645807173103, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.24954374134540558, "loss_scale": 4.0, "retries": 0, "seconds": 85.6, "max_memory_gb": 18.4, "ce": 0.19098117778776214, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.4834832549095154, "loss_scale": 4.0, "retries": 0, "seconds": 87.1, "max_memory_gb": 18.4, "ce": 0.3910846459912136, "rank": 0.08693327009677887}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.18899501860141754, "loss_scale": 4.0, "retries": 0, "seconds": 88.6, "max_memory_gb": 18.4, "ce": 0.14977715915301815, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.3966715931892395, "loss_scale": 4.0, "retries": 0, "seconds": 90.2, "max_memory_gb": 18.4, "ce": 0.21325821902428288, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.20542998611927032, "loss_scale": 4.0, "retries": 0, "seconds": 91.7, "max_memory_gb": 18.4, "ce": 0.13911788706900552, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.27997270226478577, "loss_scale": 4.0, "retries": 0, "seconds": 93.2, "max_memory_gb": 18.4, "ce": 0.23719084478216246, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.6807587146759033, "loss_scale": 4.0, "retries": 0, "seconds": 94.8, "max_memory_gb": 18.4, "ce": 0.5057586701586843, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.26885131001472473, "loss_scale": 4.0, "retries": 0, "seconds": 96.3, "max_memory_gb": 18.4, "ce": 0.15665111085399985, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.49896523356437683, "loss_scale": 4.0, "retries": 0, "seconds": 97.9, "max_memory_gb": 18.4, "ce": 0.27056417614221573, "rank": 0.05839700996875763}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.44238975644111633, "loss_scale": 4.0, "retries": 0, "seconds": 99.4, "max_memory_gb": 18.4, "ce": 0.27980081061832607, "rank": 0.09513188898563385}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 0.6995773315429688, "loss_scale": 4.0, "retries": 0, "seconds": 101.0, "max_memory_gb": 18.4, "ce": 0.4739794423803687, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.5709779858589172, "loss_scale": 4.0, "retries": 0, "seconds": 102.5, "max_memory_gb": 18.4, "ce": 0.26243247139086634, "rank": 0.12436024844646454}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.41352787613868713, "loss_scale": 4.0, "retries": 0, "seconds": 104.1, "max_memory_gb": 18.4, "ce": 0.2514077515807003, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.23639683425426483, "loss_scale": 4.0, "retries": 0, "seconds": 105.7, "max_memory_gb": 18.4, "ce": 0.12452777370708645, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.9779086709022522, "loss_scale": 4.0, "retries": 0, "seconds": 107.2, "max_memory_gb": 18.4, "ce": 0.5319581794319674, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.34519121050834656, "loss_scale": 4.0, "retries": 0, "seconds": 108.8, "max_memory_gb": 18.4, "ce": 0.18181447184178978, "rank": 0.12163931131362915}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.49111708998680115, "loss_scale": 4.0, "retries": 0, "seconds": 110.3, "max_memory_gb": 18.4, "ce": 0.23683618444920285, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.7019501328468323, "loss_scale": 4.0, "retries": 0, "seconds": 111.9, "max_memory_gb": 18.4, "ce": 0.3858687112806365, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 0.809353232383728, "loss_scale": 4.0, "retries": 0, "seconds": 113.4, "max_memory_gb": 18.4, "ce": 0.2669152989255963, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.6131088733673096, "loss_scale": 4.0, "retries": 0, "seconds": 114.9, "max_memory_gb": 18.4, "ce": 0.20964905991058913, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.6756067872047424, "loss_scale": 4.0, "retries": 0, "seconds": 116.5, "max_memory_gb": 18.4, "ce": 0.7078139354125597, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 0.6372622847557068, "loss_scale": 4.0, "retries": 0, "seconds": 118.0, "max_memory_gb": 18.4, "ce": 0.4214854253223166, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.570737361907959, "loss_scale": 4.0, "retries": 0, "seconds": 119.6, "max_memory_gb": 18.4, "ce": 0.24976244040590245, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.4662134349346161, "loss_scale": 4.0, "retries": 0, "seconds": 121.1, "max_memory_gb": 18.4, "ce": 0.18939696143206675, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.3597126603126526, "loss_scale": 4.0, "retries": 0, "seconds": 122.6, "max_memory_gb": 18.4, "ce": 0.09242980149429059, "rank": 0.07779373973608017}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.3526899814605713, "loss_scale": 4.0, "retries": 0, "seconds": 124.1, "max_memory_gb": 18.4, "ce": 0.1879776823670909, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.8656448721885681, "loss_scale": 4.0, "retries": 0, "seconds": 125.7, "max_memory_gb": 18.4, "ce": 0.24230783456005156, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.8205500841140747, "loss_scale": 4.0, "retries": 0, "seconds": 127.2, "max_memory_gb": 18.4, "ce": 0.2114457506686449, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 1.1715937852859497, "loss_scale": 4.0, "retries": 0, "seconds": 128.8, "max_memory_gb": 18.4, "ce": 0.41825919671100564, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.5432417392730713, "loss_scale": 4.0, "retries": 0, "seconds": 130.3, "max_memory_gb": 18.4, "ce": 0.06295867679864386, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.3421182930469513, "loss_scale": 4.0, "retries": 0, "seconds": 131.9, "max_memory_gb": 18.4, "ce": 0.03352386032929644, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 1.0243386030197144, "loss_scale": 4.0, "retries": 0, "seconds": 133.5, "max_memory_gb": 18.4, "ce": 0.2049099049763754, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 1.0982744693756104, "loss_scale": 4.0, "retries": 0, "seconds": 135.0, "max_memory_gb": 18.4, "ce": 0.27745556532318005, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.6296706795692444, "loss_scale": 4.0, "retries": 0, "seconds": 136.5, "max_memory_gb": 18.4, "ce": 0.13252424998790957, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.2954011857509613, "loss_scale": 4.0, "retries": 0, "seconds": 138.0, "max_memory_gb": 18.4, "ce": 0.04031906773161609, "rank": 0.12841733545064926}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 1.0288946628570557, "loss_scale": 4.0, "retries": 0, "seconds": 139.7, "max_memory_gb": 18.4, "ce": 0.17458916388568468, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 1.9096249341964722, "loss_scale": 4.0, "retries": 0, "seconds": 141.2, "max_memory_gb": 18.4, "ce": 0.5748893131385557, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 1.4083904027938843, "loss_scale": 4.0, "retries": 0, "seconds": 142.8, "max_memory_gb": 18.4, "ce": 0.2542498469538259, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 0.851468026638031, "loss_scale": 4.0, "retries": 0, "seconds": 144.3, "max_memory_gb": 18.4, "ce": 0.12665213691070676, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 1.215029239654541, "loss_scale": 4.0, "retries": 0, "seconds": 145.9, "max_memory_gb": 18.4, "ce": 0.17100595100782812, "rank": 0.03903473913669586}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 2.1764252185821533, "loss_scale": 4.0, "retries": 0, "seconds": 147.4, "max_memory_gb": 18.4, "ce": 0.6703197844326496, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 5.763444423675537, "loss_scale": 4.0, "retries": 0, "seconds": 149.0, "max_memory_gb": 18.4, "ce": 0.6706760253291577, "rank": 0.39340928196907043}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.4874674677848816, "loss_scale": 4.0, "retries": 0, "seconds": 150.5, "max_memory_gb": 18.4, "ce": 0.0626620977272978, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 3.7626376152038574, "loss_scale": 4.0, "retries": 0, "seconds": 152.0, "max_memory_gb": 18.4, "ce": 0.16665721647950704, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.18142499029636383, "loss_scale": 4.0, "retries": 0, "seconds": 153.6, "max_memory_gb": 18.4, "ce": 0.025568676363945997, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 1.2178068161010742, "loss_scale": 4.0, "retries": 0, "seconds": 155.1, "max_memory_gb": 18.4, "ce": 0.2058668837416917, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.9369432330131531, "loss_scale": 4.0, "retries": 0, "seconds": 156.7, "max_memory_gb": 18.4, "ce": 0.05024644191144034, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.8463483452796936, "loss_scale": 4.0, "retries": 0, "seconds": 158.2, "max_memory_gb": 18.4, "ce": 0.07671890867641196, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 1.6507610082626343, "loss_scale": 4.0, "retries": 0, "seconds": 159.7, "max_memory_gb": 18.4, "ce": 0.11091892873810139, "rank": 0.05726593732833862}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.5033245086669922, "loss_scale": 4.0, "retries": 0, "seconds": 161.2, "max_memory_gb": 18.4, "ce": 0.0678223983177304, "rank": 0.12497423589229584}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 1.0917091369628906, "loss_scale": 4.0, "retries": 0, "seconds": 162.8, "max_memory_gb": 18.4, "ce": 0.17442192533053458, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.6522143483161926, "loss_scale": 4.0, "retries": 0, "seconds": 164.3, "max_memory_gb": 18.4, "ce": 0.046625694318208843, "rank": 0.00869530439376831}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 3.3292922973632812, "loss_scale": 4.0, "retries": 0, "seconds": 165.8, "max_memory_gb": 18.4, "ce": 0.3395870413396551, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 0.8942374587059021, "loss_scale": 4.0, "retries": 0, "seconds": 167.3, "max_memory_gb": 18.4, "ce": 0.09698947129800217, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 1.4977611303329468, "loss_scale": 4.0, "retries": 0, "seconds": 168.9, "max_memory_gb": 18.4, "ce": 0.24246670783031732, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 1.5991029739379883, "loss_scale": 4.0, "retries": 0, "seconds": 170.4, "max_memory_gb": 18.4, "ce": 0.13368442677892745, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.8648316860198975, "loss_scale": 4.0, "retries": 0, "seconds": 172.0, "max_memory_gb": 18.4, "ce": 0.09902750394758186, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.19909872114658356, "loss_scale": 4.0, "retries": 0, "seconds": 173.5, "max_memory_gb": 18.4, "ce": 0.015489222067117225, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.7617664933204651, "loss_scale": 4.0, "retries": 0, "seconds": 175.0, "max_memory_gb": 18.4, "ce": 0.11666628812236013, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 1.2816386222839355, "loss_scale": 4.0, "retries": 0, "seconds": 176.6, "max_memory_gb": 18.4, "ce": 0.23564637699746527, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.19543714821338654, "loss_scale": 4.0, "retries": 0, "seconds": 178.1, "max_memory_gb": 18.4, "ce": 0.019290988580905832, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.5573917031288147, "loss_scale": 4.0, "retries": 0, "seconds": 179.6, "max_memory_gb": 18.4, "ce": 0.18603385816641094, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 1.0398086309432983, "loss_scale": 4.0, "retries": 0, "seconds": 181.2, "max_memory_gb": 18.41, "ce": 0.08705207506136503, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 0.9870986342430115, "loss_scale": 4.0, "retries": 0, "seconds": 182.7, "max_memory_gb": 18.41, "ce": 0.24702586192142917, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.21603700518608093, "loss_scale": 4.0, "retries": 0, "seconds": 184.2, "max_memory_gb": 18.41, "ce": 0.020559849743222003, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 2.417628049850464, "loss_scale": 4.0, "retries": 0, "seconds": 185.8, "max_memory_gb": 18.41, "ce": 0.4148678944184212, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.39127451181411743, "loss_scale": 4.0, "retries": 0, "seconds": 187.3, "max_memory_gb": 18.41, "ce": 0.05974576426251588, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 5.459457874298096, "loss_scale": 4.0, "retries": 0, "seconds": 188.9, "max_memory_gb": 18.41, "ce": 0.14818463184201391, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.7855747938156128, "loss_scale": 4.0, "retries": 0, "seconds": 190.4, "max_memory_gb": 18.41, "ce": 0.08804880734533072, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 2.367288827896118, "loss_scale": 4.0, "retries": 0, "seconds": 191.9, "max_memory_gb": 18.41, "ce": 0.5553893024771241, "rank": 0.022742927074432373}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.8083525896072388, "loss_scale": 4.0, "retries": 0, "seconds": 193.4, "max_memory_gb": 18.41, "ce": 0.1505300764401909, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 1.905909538269043, "loss_scale": 4.0, "retries": 0, "seconds": 194.9, "max_memory_gb": 18.41, "ce": 0.40451738762203604, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.7549667954444885, "loss_scale": 4.0, "retries": 0, "seconds": 196.4, "max_memory_gb": 18.41, "ce": 0.026437130209160387, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 2.5576841831207275, "loss_scale": 4.0, "retries": 0, "seconds": 198.0, "max_memory_gb": 18.41, "ce": 0.30215756373945624, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 1.2220443487167358, "loss_scale": 4.0, "retries": 0, "seconds": 199.5, "max_memory_gb": 18.41, "ce": 0.10760041140019894, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 1.2757693529129028, "loss_scale": 4.0, "retries": 0, "seconds": 201.0, "max_memory_gb": 18.41, "ce": 0.20502800702524837, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 2.6909515857696533, "loss_scale": 4.0, "retries": 0, "seconds": 202.6, "max_memory_gb": 18.41, "ce": 0.330578462977428, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 4.671360015869141, "loss_scale": 4.0, "retries": 0, "seconds": 204.1, "max_memory_gb": 18.41, "ce": 0.22505797358644486, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 1.554489016532898, "loss_scale": 4.0, "retries": 0, "seconds": 205.6, "max_memory_gb": 18.41, "ce": 0.4369536084413994, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.8569634556770325, "loss_scale": 4.0, "retries": 0, "seconds": 207.2, "max_memory_gb": 18.41, "ce": 0.17779143092502636, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 0.9274177551269531, "loss_scale": 4.0, "retries": 0, "seconds": 208.7, "max_memory_gb": 18.41, "ce": 0.05959832269581966, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 0.6662150025367737, "loss_scale": 4.0, "retries": 0, "seconds": 210.4, "max_memory_gb": 18.41, "ce": 0.05062375518900808, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.27017420530319214, "loss_scale": 4.0, "retries": 0, "seconds": 212.2, "max_memory_gb": 18.41, "ce": 0.02387463123886846, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 2.015021562576294, "loss_scale": 4.0, "retries": 0, "seconds": 214.0, "max_memory_gb": 18.41, "ce": 0.13327566266525537, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.7411991357803345, "loss_scale": 4.0, "retries": 0, "seconds": 215.8, "max_memory_gb": 18.41, "ce": 0.05410690631833859, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.1748104989528656, "loss_scale": 4.0, "retries": 0, "seconds": 217.5, "max_memory_gb": 18.41, "ce": 0.013978465009131469, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.18799340724945068, "loss_scale": 4.0, "retries": 0, "seconds": 219.3, "max_memory_gb": 18.41, "ce": 0.015157265050220303, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.6918195486068726, "loss_scale": 4.0, "retries": 0, "seconds": 221.0, "max_memory_gb": 18.41, "ce": 0.057104465311567765, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 2.2151708602905273, "loss_scale": 4.0, "retries": 0, "seconds": 222.8, "max_memory_gb": 18.41, "ce": 0.41899796166580927, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 1.174047827720642, "loss_scale": 4.0, "retries": 0, "seconds": 224.5, "max_memory_gb": 18.41, "ce": 0.11109890177613124, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.19493229687213898, "loss_scale": 4.0, "retries": 0, "seconds": 226.2, "max_memory_gb": 18.41, "ce": 0.012502032051997958, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.25273680686950684, "loss_scale": 4.0, "retries": 0, "seconds": 228.0, "max_memory_gb": 18.41, "ce": 0.022936933368328027, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 2.3046529293060303, "loss_scale": 4.0, "retries": 0, "seconds": 229.7, "max_memory_gb": 18.41, "ce": 0.6594373138286755, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.10749872028827667, "loss_scale": 4.0, "retries": 0, "seconds": 231.4, "max_memory_gb": 18.41, "ce": 0.007539096176969906, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.28835415840148926, "loss_scale": 4.0, "retries": 0, "seconds": 233.1, "max_memory_gb": 18.41, "ce": 0.01631575066858204, "rank": 0.12537193298339844}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.36240726709365845, "loss_scale": 4.0, "retries": 0, "seconds": 234.9, "max_memory_gb": 18.41, "ce": 0.06439582971506752, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.7373208403587341, "loss_scale": 4.0, "retries": 0, "seconds": 236.6, "max_memory_gb": 18.41, "ce": 0.03951935544500884, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 1.4610923528671265, "loss_scale": 4.0, "retries": 0, "seconds": 238.4, "max_memory_gb": 18.41, "ce": 0.05689534631164861, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 4.464324474334717, "loss_scale": 4.0, "retries": 0, "seconds": 240.1, "max_memory_gb": 18.41, "ce": 0.12174470879654109, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.8262391090393066, "loss_scale": 4.0, "retries": 0, "seconds": 241.8, "max_memory_gb": 18.41, "ce": 0.06898272899707081, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 2.132598876953125, "loss_scale": 4.0, "retries": 0, "seconds": 243.6, "max_memory_gb": 18.41, "ce": 0.15067465752144926, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.7413672208786011, "loss_scale": 4.0, "retries": 0, "seconds": 245.3, "max_memory_gb": 18.41, "ce": 0.026776240020808473, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.28121834993362427, "loss_scale": 4.0, "retries": 0, "seconds": 247.0, "max_memory_gb": 18.41, "ce": 0.01879793801344931, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 13.866918563842773, "loss_scale": 4.0, "retries": 0, "seconds": 248.8, "max_memory_gb": 18.41, "ce": 0.680615598372242, "rank": 0.018645405769348145}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.2884502410888672, "loss_scale": 4.0, "retries": 0, "seconds": 250.5, "max_memory_gb": 18.51, "ce": 0.014778290071262745, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 1.073072910308838, "loss_scale": 4.0, "retries": 0, "seconds": 252.3, "max_memory_gb": 18.51, "ce": 0.1767231674020877, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 0.4956936538219452, "loss_scale": 4.0, "retries": 0, "seconds": 254.0, "max_memory_gb": 18.51, "ce": 0.06855613024526974, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.752465546131134, "loss_scale": 4.0, "retries": 0, "seconds": 255.5, "max_memory_gb": 18.51, "ce": 0.03319426917914825, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 2.08720064163208, "loss_scale": 4.0, "retries": 0, "seconds": 257.0, "max_memory_gb": 18.51, "ce": 0.18813836434856057, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.7307220697402954, "loss_scale": 4.0, "retries": 0, "seconds": 258.6, "max_memory_gb": 18.51, "ce": 0.057626401216111844, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.12870167195796967, "loss_scale": 4.0, "retries": 0, "seconds": 260.2, "max_memory_gb": 18.51, "ce": 0.012254280194611056, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 2.5093178749084473, "loss_scale": 4.0, "retries": 0, "seconds": 261.7, "max_memory_gb": 18.51, "ce": 0.4343464117409894, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.1579945832490921, "loss_scale": 4.0, "retries": 0, "seconds": 263.3, "max_memory_gb": 18.51, "ce": 0.009618064315873198, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.36926865577697754, "loss_scale": 4.0, "retries": 0, "seconds": 264.8, "max_memory_gb": 18.51, "ce": 0.0210735496802954, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 2.3714194297790527, "loss_scale": 4.0, "retries": 0, "seconds": 266.4, "max_memory_gb": 18.51, "ce": 0.41187244189495686, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.12327158451080322, "loss_scale": 4.0, "retries": 0, "seconds": 268.0, "max_memory_gb": 18.51, "ce": 0.005186550157304737, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 1.2746148109436035, "loss_scale": 4.0, "retries": 0, "seconds": 269.5, "max_memory_gb": 18.51, "ce": 0.08484014592431777, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.2787781357765198, "loss_scale": 4.0, "retries": 0, "seconds": 271.1, "max_memory_gb": 18.51, "ce": 0.016594421773334034, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.982721745967865, "loss_scale": 4.0, "retries": 0, "seconds": 272.6, "max_memory_gb": 18.51, "ce": 0.047490887736785226, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.7142497301101685, "loss_scale": 4.0, "retries": 0, "seconds": 274.2, "max_memory_gb": 18.51, "ce": 0.05016422873131887, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 4.742164134979248, "loss_scale": 4.0, "retries": 0, "seconds": 275.7, "max_memory_gb": 18.51, "ce": 0.32501873883302324, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.9378211498260498, "loss_scale": 4.0, "retries": 0, "seconds": 277.3, "max_memory_gb": 18.51, "ce": 0.014152065541566117, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 1.5056267976760864, "loss_scale": 4.0, "retries": 0, "seconds": 278.8, "max_memory_gb": 18.51, "ce": 0.2958981066794877, "rank": 0.12499796599149704}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 1.0273370742797852, "loss_scale": 4.0, "retries": 0, "seconds": 280.4, "max_memory_gb": 18.51, "ce": 0.09137106098933145, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 1.805677890777588, "loss_scale": 4.0, "retries": 0, "seconds": 281.9, "max_memory_gb": 18.51, "ce": 0.3927326784382785, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.6874398589134216, "loss_scale": 4.0, "retries": 0, "seconds": 283.4, "max_memory_gb": 18.51, "ce": 0.031276104535209015, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.5265793800354004, "loss_scale": 4.0, "retries": 0, "seconds": 285.0, "max_memory_gb": 18.51, "ce": 0.022837484648334794, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 1.229752540588379, "loss_scale": 4.0, "retries": 0, "seconds": 286.5, "max_memory_gb": 18.51, "ce": 0.07313159127079416, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.21390098333358765, "loss_scale": 4.0, "retries": 0, "seconds": 288.1, "max_memory_gb": 18.51, "ce": 0.006096994987274229, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.6728028655052185, "loss_scale": 4.0, "retries": 0, "seconds": 289.7, "max_memory_gb": 18.51, "ce": 0.014127095329286021, "rank": 0.1893458589911461}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.7289735078811646, "loss_scale": 4.0, "retries": 0, "seconds": 291.2, "max_memory_gb": 18.51, "ce": 0.05877849404896551, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 0.8446484208106995, "loss_scale": 4.0, "retries": 0, "seconds": 292.8, "max_memory_gb": 18.51, "ce": 0.045775883362239256, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 1.2378017902374268, "loss_scale": 4.0, "retries": 0, "seconds": 294.3, "max_memory_gb": 18.51, "ce": 0.1936156686861068, "rank": 0.13864150643348694}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.4200687408447266, "loss_scale": 4.0, "retries": 0, "seconds": 295.9, "max_memory_gb": 18.51, "ce": 0.20560929603925615, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 2.7394051551818848, "loss_scale": 4.0, "retries": 0, "seconds": 297.4, "max_memory_gb": 18.51, "ce": 0.4665820673108101, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.688746988773346, "loss_scale": 4.0, "retries": 0, "seconds": 299.0, "max_memory_gb": 18.51, "ce": 0.016282962569675874, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 2.476367235183716, "loss_scale": 4.0, "retries": 0, "seconds": 300.5, "max_memory_gb": 18.51, "ce": 0.2716506407596171, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.9148615598678589, "loss_scale": 4.0, "retries": 0, "seconds": 302.0, "max_memory_gb": 18.51, "ce": 0.46154125523753464, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.13491514325141907, "loss_scale": 4.0, "retries": 0, "seconds": 303.6, "max_memory_gb": 18.51, "ce": 0.004414341723531834, "rank": 0.12500755488872528}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.8087231516838074, "loss_scale": 4.0, "retries": 0, "seconds": 305.1, "max_memory_gb": 18.51, "ce": 0.0497856078291079, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 1.8814637660980225, "loss_scale": 4.0, "retries": 0, "seconds": 306.7, "max_memory_gb": 18.51, "ce": 0.07508260858048743, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.9227007627487183, "loss_scale": 4.0, "retries": 0, "seconds": 308.3, "max_memory_gb": 18.51, "ce": 0.03337222539175855, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 1.1726181507110596, "loss_scale": 4.0, "retries": 0, "seconds": 309.8, "max_memory_gb": 18.51, "ce": 0.2273748822772177, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.4230288863182068, "loss_scale": 4.0, "retries": 0, "seconds": 311.4, "max_memory_gb": 18.51, "ce": 0.05699597822240321, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 2.2385575771331787, "loss_scale": 4.0, "retries": 0, "seconds": 312.9, "max_memory_gb": 18.51, "ce": 0.13698888519138563, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 1.1834666728973389, "loss_scale": 4.0, "retries": 0, "seconds": 314.5, "max_memory_gb": 18.51, "ce": 0.2767884342028992, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.7915499806404114, "loss_scale": 4.0, "retries": 0, "seconds": 316.0, "max_memory_gb": 18.51, "ce": 0.06622253456589533, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.37433987855911255, "loss_scale": 4.0, "retries": 0, "seconds": 317.5, "max_memory_gb": 18.51, "ce": 0.00841825307588806, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.21985043585300446, "loss_scale": 4.0, "retries": 0, "seconds": 319.1, "max_memory_gb": 18.51, "ce": 0.01194153493270278, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 4.648649215698242, "loss_scale": 4.0, "retries": 0, "seconds": 320.6, "max_memory_gb": 18.51, "ce": 0.044479756586952135, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 5.288998126983643, "loss_scale": 4.0, "retries": 0, "seconds": 322.2, "max_memory_gb": 18.51, "ce": 0.1788858757354319, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.8576745986938477, "loss_scale": 4.0, "retries": 0, "seconds": 323.7, "max_memory_gb": 18.51, "ce": 0.019241838841480785, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.2996992766857147, "loss_scale": 4.0, "retries": 0, "seconds": 325.3, "max_memory_gb": 18.51, "ce": 0.044758881456800736, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.1272125542163849, "loss_scale": 4.0, "retries": 0, "seconds": 326.8, "max_memory_gb": 18.51, "ce": 0.005073234657174908, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.9661572575569153, "loss_scale": 4.0, "retries": 0, "seconds": 328.4, "max_memory_gb": 18.51, "ce": 0.11396566816256382, "rank": 0.12492847442626953}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 2.261906623840332, "loss_scale": 4.0, "retries": 0, "seconds": 330.0, "max_memory_gb": 18.51, "ce": 0.4888515558277504, "rank": 0.0}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.7010340094566345, "loss_scale": 4.0, "retries": 0, "seconds": 331.5, "max_memory_gb": 18.51, "ce": 0.1464519999062759, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 1.5855655670166016, "loss_scale": 4.0, "retries": 0, "seconds": 333.1, "max_memory_gb": 18.51, "ce": 0.2990057994138624, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.7402868866920471, "loss_scale": 4.0, "retries": 0, "seconds": 334.7, "max_memory_gb": 18.51, "ce": 0.19179651054218994, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 1.9905028343200684, "loss_scale": 4.0, "retries": 0, "seconds": 336.2, "max_memory_gb": 18.51, "ce": 0.2685901269715032, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 2.55580997467041, "loss_scale": 4.0, "retries": 0, "seconds": 337.8, "max_memory_gb": 18.51, "ce": 0.577414574483555, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.5487525463104248, "loss_scale": 4.0, "retries": 0, "seconds": 339.3, "max_memory_gb": 18.51, "ce": 0.02311130828456953, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.141025111079216, "loss_scale": 4.0, "retries": 0, "seconds": 340.9, "max_memory_gb": 18.51, "ce": 0.0038008195824659197, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 1.0060514211654663, "loss_scale": 4.0, "retries": 0, "seconds": 342.4, "max_memory_gb": 18.51, "ce": 0.16111055194051005, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.3519757390022278, "loss_scale": 4.0, "retries": 0, "seconds": 344.0, "max_memory_gb": 18.51, "ce": 0.02763230480832135, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.968418538570404, "loss_scale": 4.0, "retries": 0, "seconds": 345.5, "max_memory_gb": 18.51, "ce": 0.0637907908512716, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.789656400680542, "loss_scale": 4.0, "retries": 0, "seconds": 347.1, "max_memory_gb": 18.51, "ce": 0.09817222975834738, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.8677237033843994, "loss_scale": 4.0, "retries": 0, "seconds": 348.7, "max_memory_gb": 18.51, "ce": 0.028634404836338945, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 2.2139813899993896, "loss_scale": 4.0, "retries": 0, "seconds": 350.2, "max_memory_gb": 18.51, "ce": 0.08434616947488394, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.35234278440475464, "loss_scale": 4.0, "retries": 0, "seconds": 351.8, "max_memory_gb": 18.51, "ce": 0.03492858757817885, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.6011795401573181, "loss_scale": 4.0, "retries": 0, "seconds": 353.3, "max_memory_gb": 18.51, "ce": 0.03110975114032044, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.6362974643707275, "loss_scale": 4.0, "retries": 0, "seconds": 354.9, "max_memory_gb": 18.51, "ce": 0.04326525268697878, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.8142428398132324, "loss_scale": 4.0, "retries": 0, "seconds": 356.4, "max_memory_gb": 18.51, "ce": 0.03859929211103008, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.2829609811306, "loss_scale": 4.0, "retries": 0, "seconds": 357.9, "max_memory_gb": 18.51, "ce": 0.016252249264653074, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.5768660306930542, "loss_scale": 4.0, "retries": 0, "seconds": 359.5, "max_memory_gb": 18.51, "ce": 0.050972630520845996, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 2.7114720344543457, "loss_scale": 4.0, "retries": 0, "seconds": 361.0, "max_memory_gb": 18.51, "ce": 0.47354183087009005, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.5403403043746948, "loss_scale": 4.0, "retries": 0, "seconds": 362.5, "max_memory_gb": 18.51, "ce": 0.029866721893995418, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.7786666750907898, "loss_scale": 4.0, "retries": 0, "seconds": 364.1, "max_memory_gb": 18.51, "ce": 0.027998109202599153, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.7617272734642029, "loss_scale": 4.0, "retries": 0, "seconds": 365.6, "max_memory_gb": 18.51, "ce": 0.021337557896913495, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.5326805710792542, "loss_scale": 4.0, "retries": 0, "seconds": 367.1, "max_memory_gb": 18.51, "ce": 0.026315326238545822, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 1.9426493644714355, "loss_scale": 4.0, "retries": 0, "seconds": 368.7, "max_memory_gb": 18.51, "ce": 0.2478843845638039, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.31208860874176025, "loss_scale": 4.0, "retries": 0, "seconds": 370.2, "max_memory_gb": 18.51, "ce": 0.015079347528626386, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.927801787853241, "loss_scale": 4.0, "retries": 0, "seconds": 371.8, "max_memory_gb": 18.51, "ce": 0.04590213357005268, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.5418986082077026, "loss_scale": 4.0, "retries": 0, "seconds": 373.3, "max_memory_gb": 18.51, "ce": 0.023765129189996514, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.8658542037010193, "loss_scale": 4.0, "retries": 0, "seconds": 374.8, "max_memory_gb": 18.51, "ce": 0.10305102671554778, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 1.3235585689544678, "loss_scale": 4.0, "retries": 0, "seconds": 376.4, "max_memory_gb": 18.51, "ce": 0.055184431537782075, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 1.546392798423767, "loss_scale": 4.0, "retries": 0, "seconds": 377.9, "max_memory_gb": 18.51, "ce": 0.12672578779165633, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.22000586986541748, "loss_scale": 4.0, "retries": 0, "seconds": 379.5, "max_memory_gb": 18.51, "ce": 0.008697676373230934, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.20273809134960175, "loss_scale": 4.0, "retries": 0, "seconds": 381.0, "max_memory_gb": 18.51, "ce": 0.005337634778697975, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.3512783646583557, "loss_scale": 4.0, "retries": 0, "seconds": 382.6, "max_memory_gb": 18.51, "ce": 0.024113549352478003, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 1.585718035697937, "loss_scale": 4.0, "retries": 0, "seconds": 384.1, "max_memory_gb": 18.51, "ce": 0.35302130997297354, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.08028322458267212, "loss_scale": 4.0, "retries": 0, "seconds": 385.7, "max_memory_gb": 18.51, "ce": 0.0033566545589565067, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 3.4426000118255615, "loss_scale": 4.0, "retries": 0, "seconds": 387.2, "max_memory_gb": 18.51, "ce": 0.12306802075386258, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.6367363929748535, "loss_scale": 4.0, "retries": 0, "seconds": 388.8, "max_memory_gb": 18.51, "ce": 0.01983653641946148, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.6084651947021484, "loss_scale": 4.0, "retries": 0, "seconds": 390.3, "max_memory_gb": 18.51, "ce": 0.019251440215157345, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.7310636639595032, "loss_scale": 4.0, "retries": 0, "seconds": 391.9, "max_memory_gb": 18.51, "ce": 0.025604342539736535, "rank": 0.0}

runs/single_internvl/removal/L16_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/single_internvl/removal/L16_s13/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.16.self_attn.q_proj",
+    "model.language_model.layers.16.mlp.gate_proj",
+    "model.language_model.layers.16.self_attn.v_proj",
+    "model.language_model.layers.16.mlp.up_proj",
+    "model.language_model.layers.16.self_attn.o_proj",
+    "model.language_model.layers.16.self_attn.k_proj",
+    "model.language_model.layers.16.mlp.down_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/single_internvl/removal/L16_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 16,
+    "rank": 16,
+    "trainable_parameters": 1212416
+  }
+}

runs/single_internvl/removal/L16_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 1.1436179876327515, "loss_scale": 4.0, "retries": 0, "seconds": 2.7, "max_memory_gb": 17.92, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 1.0349295139312744, "loss_scale": 4.0, "retries": 0, "seconds": 4.4, "max_memory_gb": 17.94, "ce": 0.7807847931289871, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 3.9168179035186768, "loss_scale": 4.0, "retries": 0, "seconds": 6.1, "max_memory_gb": 17.94, "ce": 1.380730865523219, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 2.2446398735046387, "loss_scale": 4.0, "retries": 0, "seconds": 7.9, "max_memory_gb": 17.94, "ce": 0.7489903689020139, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.03197513148188591, "loss_scale": 4.0, "retries": 0, "seconds": 9.7, "max_memory_gb": 17.94, "ce": 0.040141397718798544, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 1.1331133842468262, "loss_scale": 4.0, "retries": 0, "seconds": 11.5, "max_memory_gb": 17.94, "ce": 0.42848081691772677, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 3.1694839000701904, "loss_scale": 4.0, "retries": 0, "seconds": 13.2, "max_memory_gb": 17.94, "ce": 0.9820652292673913, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 3.0035810470581055, "loss_scale": 4.0, "retries": 0, "seconds": 15.2, "max_memory_gb": 17.94, "ce": 0.9387150881811976, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 2.347097396850586, "loss_scale": 4.0, "retries": 0, "seconds": 17.2, "max_memory_gb": 17.94, "ce": 0.6479787928983569, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.31481456756591797, "loss_scale": 4.0, "retries": 0, "seconds": 19.0, "max_memory_gb": 17.94, "ce": 0.18163860133790877, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 2.8800692558288574, "loss_scale": 4.0, "retries": 0, "seconds": 20.7, "max_memory_gb": 17.94, "ce": 0.690124271561217, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 2.074636936187744, "loss_scale": 4.0, "retries": 0, "seconds": 22.5, "max_memory_gb": 17.94, "ce": 0.43383421843009273, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 1.4416131973266602, "loss_scale": 4.0, "retries": 0, "seconds": 24.2, "max_memory_gb": 17.94, "ce": 0.540999826936968, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 2.5093164443969727, "loss_scale": 4.0, "retries": 0, "seconds": 25.9, "max_memory_gb": 17.94, "ce": 0.7387380860745907, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 1.6144665479660034, "loss_scale": 4.0, "retries": 0, "seconds": 27.6, "max_memory_gb": 17.94, "ce": 0.4530394970349789, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.7198825478553772, "loss_scale": 4.0, "retries": 0, "seconds": 29.3, "max_memory_gb": 17.94, "ce": 0.20903433828789275, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.8437013030052185, "loss_scale": 4.0, "retries": 0, "seconds": 31.0, "max_memory_gb": 17.94, "ce": 0.43902485742000863, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 0.9374105334281921, "loss_scale": 4.0, "retries": 0, "seconds": 32.7, "max_memory_gb": 17.95, "ce": 0.3789659531757934, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 0.6746107935905457, "loss_scale": 4.0, "retries": 0, "seconds": 34.5, "max_memory_gb": 17.95, "ce": 0.5112604065798223, "rank": 0.15610408782958984}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.0325070321559906, "loss_scale": 4.0, "retries": 0, "seconds": 36.3, "max_memory_gb": 17.95, "ce": 0.04053727049722511, "rank": 0.12104397267103195}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.26806384325027466, "loss_scale": 4.0, "retries": 0, "seconds": 38.0, "max_memory_gb": 17.95, "ce": 0.3067839929699403, "rank": 0.058147408068180084}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.4350499212741852, "loss_scale": 4.0, "retries": 0, "seconds": 39.7, "max_memory_gb": 17.95, "ce": 0.4152903985159355, "rank": 0.14839938282966614}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 3.976499319076538, "loss_scale": 4.0, "retries": 0, "seconds": 41.4, "max_memory_gb": 17.95, "ce": 0.5230240636719827, "rank": 0.1208050549030304}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 0.37581631541252136, "loss_scale": 4.0, "retries": 0, "seconds": 43.1, "max_memory_gb": 17.95, "ce": 0.5009460556320846, "rank": 0.127047598361969}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 0.7171021103858948, "loss_scale": 4.0, "retries": 0, "seconds": 44.8, "max_memory_gb": 17.95, "ce": 0.6019479384631268, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 0.3981475234031677, "loss_scale": 4.0, "retries": 0, "seconds": 46.7, "max_memory_gb": 17.95, "ce": 0.5321328985737637, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 0.4800596833229065, "loss_scale": 4.0, "retries": 0, "seconds": 48.4, "max_memory_gb": 17.95, "ce": 0.46453661280247616, "rank": 0.3083985596895218}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 0.9459040760993958, "loss_scale": 4.0, "retries": 0, "seconds": 50.2, "max_memory_gb": 17.95, "ce": 0.7565001230686903, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 0.6639293432235718, "loss_scale": 4.0, "retries": 0, "seconds": 51.9, "max_memory_gb": 17.95, "ce": 0.38895953103201464, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 0.31921589374542236, "loss_scale": 4.0, "retries": 0, "seconds": 53.7, "max_memory_gb": 17.95, "ce": 0.24196980744318353, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 0.7836763262748718, "loss_scale": 4.0, "retries": 0, "seconds": 55.4, "max_memory_gb": 17.95, "ce": 0.4831209594849497, "rank": 0.10046841204166412}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 0.434878945350647, "loss_scale": 4.0, "retries": 0, "seconds": 57.2, "max_memory_gb": 17.95, "ce": 0.4264558104914613, "rank": 0.06818735599517822}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.3184138536453247, "loss_scale": 4.0, "retries": 0, "seconds": 58.9, "max_memory_gb": 17.95, "ce": 0.25873075803974643, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.3439009487628937, "loss_scale": 4.0, "retries": 0, "seconds": 60.6, "max_memory_gb": 17.95, "ce": 0.29835538096449454, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.25525665283203125, "loss_scale": 4.0, "retries": 0, "seconds": 62.3, "max_memory_gb": 17.95, "ce": 0.1913852697070979, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 0.3860713541507721, "loss_scale": 4.0, "retries": 0, "seconds": 64.2, "max_memory_gb": 17.95, "ce": 0.29488482265151106, "rank": 0.0194561704993248}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.6586690545082092, "loss_scale": 4.0, "retries": 0, "seconds": 65.9, "max_memory_gb": 17.95, "ce": 0.5165954934200272, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.5771510601043701, "loss_scale": 4.0, "retries": 0, "seconds": 67.7, "max_memory_gb": 17.95, "ce": 0.5098576843738556, "rank": 0.03614763915538788}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 0.6739535331726074, "loss_scale": 4.0, "retries": 0, "seconds": 69.4, "max_memory_gb": 17.95, "ce": 0.3652919123705942, "rank": 0.1248285174369812}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.34876149892807007, "loss_scale": 4.0, "retries": 0, "seconds": 71.1, "max_memory_gb": 17.95, "ce": 0.22275410440488486, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.9572052955627441, "loss_scale": 4.0, "retries": 0, "seconds": 72.8, "max_memory_gb": 17.95, "ce": 0.4437949104467407, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.27369388937950134, "loss_scale": 4.0, "retries": 0, "seconds": 74.5, "max_memory_gb": 17.95, "ce": 0.18722974388219882, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.30126553773880005, "loss_scale": 4.0, "retries": 0, "seconds": 76.2, "max_memory_gb": 17.95, "ce": 0.19544343523739371, "rank": 0.1987466663122177}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 2.5861916542053223, "loss_scale": 4.0, "retries": 0, "seconds": 77.8, "max_memory_gb": 17.95, "ce": 0.5907522870693356, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 1.3883168697357178, "loss_scale": 4.0, "retries": 0, "seconds": 79.5, "max_memory_gb": 17.95, "ce": 0.6573829776025377, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.6766360998153687, "loss_scale": 4.0, "retries": 0, "seconds": 81.2, "max_memory_gb": 17.95, "ce": 0.31341161008458585, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.24943704903125763, "loss_scale": 4.0, "retries": 0, "seconds": 83.0, "max_memory_gb": 17.95, "ce": 0.18271509744226933, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.38244345784187317, "loss_scale": 4.0, "retries": 0, "seconds": 84.8, "max_memory_gb": 17.95, "ce": 0.2858134987181984, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.48565465211868286, "loss_scale": 4.0, "retries": 0, "seconds": 86.5, "max_memory_gb": 17.95, "ce": 0.28751152060794993, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.6608637571334839, "loss_scale": 4.0, "retries": 0, "seconds": 88.2, "max_memory_gb": 17.95, "ce": 0.32788793939107563, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 1.9657607078552246, "loss_scale": 4.0, "retries": 0, "seconds": 89.8, "max_memory_gb": 17.95, "ce": 0.42985862736532, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.5053106546401978, "loss_scale": 4.0, "retries": 0, "seconds": 91.5, "max_memory_gb": 17.95, "ce": 0.260223891804344, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 1.2612708806991577, "loss_scale": 4.0, "retries": 0, "seconds": 93.2, "max_memory_gb": 17.95, "ce": 0.4238739727297798, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 0.979458212852478, "loss_scale": 4.0, "retries": 0, "seconds": 94.9, "max_memory_gb": 17.95, "ce": 0.3450947331439238, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.40476250648498535, "loss_scale": 4.0, "retries": 0, "seconds": 96.5, "max_memory_gb": 17.95, "ce": 0.1611583426274592, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.8283235430717468, "loss_scale": 4.0, "retries": 0, "seconds": 98.2, "max_memory_gb": 17.95, "ce": 0.3220456389244646, "rank": 0.07533608376979828}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.3388187289237976, "loss_scale": 4.0, "retries": 0, "seconds": 99.8, "max_memory_gb": 17.95, "ce": 0.12827344855759293, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.5028573870658875, "loss_scale": 4.0, "retries": 0, "seconds": 101.6, "max_memory_gb": 17.95, "ce": 0.16534100084390957, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.41121906042099, "loss_scale": 4.0, "retries": 0, "seconds": 103.2, "max_memory_gb": 17.95, "ce": 0.12237774601089768, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.5136573314666748, "loss_scale": 4.0, "retries": 0, "seconds": 104.9, "max_memory_gb": 17.95, "ce": 0.20561883482150733, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.7773415446281433, "loss_scale": 4.0, "retries": 0, "seconds": 106.7, "max_memory_gb": 17.95, "ce": 0.4214118766831234, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.38345733284950256, "loss_scale": 4.0, "retries": 0, "seconds": 108.3, "max_memory_gb": 17.95, "ce": 0.12000022944994271, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.8556653261184692, "loss_scale": 4.0, "retries": 0, "seconds": 110.0, "max_memory_gb": 17.95, "ce": 0.17930128559237346, "rank": 0.0}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.7871785759925842, "loss_scale": 4.0, "retries": 0, "seconds": 111.8, "max_memory_gb": 17.95, "ce": 0.17996069020591676, "rank": 0.0}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 1.0776638984680176, "loss_scale": 4.0, "retries": 0, "seconds": 113.5, "max_memory_gb": 17.95, "ce": 0.32205187482759356, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.8464495539665222, "loss_scale": 4.0, "retries": 0, "seconds": 115.2, "max_memory_gb": 17.95, "ce": 0.24137338617856585, "rank": 0.12434186041355133}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.8727827668190002, "loss_scale": 4.0, "retries": 0, "seconds": 117.0, "max_memory_gb": 17.95, "ce": 0.133832948631607, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.376003623008728, "loss_scale": 4.0, "retries": 0, "seconds": 118.9, "max_memory_gb": 17.95, "ce": 0.07674896392563824, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 1.504111647605896, "loss_scale": 4.0, "retries": 0, "seconds": 120.7, "max_memory_gb": 17.95, "ce": 0.4755306311417371, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.8205890655517578, "loss_scale": 4.0, "retries": 0, "seconds": 122.4, "max_memory_gb": 17.95, "ce": 0.10373433763743378, "rank": 0.06838081777095795}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.7789586782455444, "loss_scale": 4.0, "retries": 0, "seconds": 124.1, "max_memory_gb": 17.95, "ce": 0.11754866182491241, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 1.1839741468429565, "loss_scale": 4.0, "retries": 0, "seconds": 125.9, "max_memory_gb": 17.95, "ce": 0.22666920023038983, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 1.1341679096221924, "loss_scale": 4.0, "retries": 0, "seconds": 127.5, "max_memory_gb": 17.95, "ce": 0.19823233153874753, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.7593621015548706, "loss_scale": 4.0, "retries": 0, "seconds": 129.3, "max_memory_gb": 17.95, "ce": 0.13262892942930193, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 1.5414549112319946, "loss_scale": 4.0, "retries": 0, "seconds": 131.0, "max_memory_gb": 17.95, "ce": 0.7325289761647582, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 2.110198974609375, "loss_scale": 4.0, "retries": 0, "seconds": 132.7, "max_memory_gb": 17.95, "ce": 0.40036386318388395, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 1.0278812646865845, "loss_scale": 4.0, "retries": 0, "seconds": 134.4, "max_memory_gb": 17.95, "ce": 0.09591245799674653, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 1.8252625465393066, "loss_scale": 4.0, "retries": 0, "seconds": 136.1, "max_memory_gb": 17.95, "ce": 0.13265314680393203, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.579498827457428, "loss_scale": 4.0, "retries": 0, "seconds": 137.8, "max_memory_gb": 17.95, "ce": 0.02593838488610345, "rank": 0.015399396419525146}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.762902021408081, "loss_scale": 4.0, "retries": 0, "seconds": 139.5, "max_memory_gb": 17.95, "ce": 0.07301748525242147, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.6877476572990417, "loss_scale": 4.0, "retries": 0, "seconds": 141.2, "max_memory_gb": 17.95, "ce": 0.07785933325067163, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 3.1328091621398926, "loss_scale": 4.0, "retries": 0, "seconds": 143.0, "max_memory_gb": 17.95, "ce": 0.13929252969683148, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 1.6540261507034302, "loss_scale": 4.0, "retries": 0, "seconds": 144.7, "max_memory_gb": 17.95, "ce": 0.2939920750445708, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.3097141981124878, "loss_scale": 4.0, "retries": 0, "seconds": 146.4, "max_memory_gb": 17.95, "ce": 0.02112476182833234, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.16963091492652893, "loss_scale": 4.0, "retries": 0, "seconds": 148.1, "max_memory_gb": 17.95, "ce": 0.008267936134416232, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 1.0504249334335327, "loss_scale": 4.0, "retries": 0, "seconds": 149.8, "max_memory_gb": 17.95, "ce": 0.08829861236154102, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 1.5811206102371216, "loss_scale": 4.0, "retries": 0, "seconds": 151.4, "max_memory_gb": 17.95, "ce": 0.17249302194431948, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.8901858329772949, "loss_scale": 4.0, "retries": 0, "seconds": 153.2, "max_memory_gb": 17.95, "ce": 0.06037593443579681, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.20800574123859406, "loss_scale": 4.0, "retries": 0, "seconds": 154.8, "max_memory_gb": 17.95, "ce": 0.01840280597025412, "rank": 0.12499521672725677}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.8873358368873596, "loss_scale": 4.0, "retries": 0, "seconds": 156.5, "max_memory_gb": 17.95, "ce": 0.04783708025206579, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 2.7826550006866455, "loss_scale": 4.0, "retries": 0, "seconds": 158.2, "max_memory_gb": 17.95, "ce": 0.5847469808195456, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 1.8421401977539062, "loss_scale": 4.0, "retries": 0, "seconds": 159.9, "max_memory_gb": 17.95, "ce": 0.15139773667920053, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 1.6340042352676392, "loss_scale": 4.0, "retries": 0, "seconds": 161.6, "max_memory_gb": 17.95, "ce": 0.07955079065868631, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 0.7568002343177795, "loss_scale": 4.0, "retries": 0, "seconds": 163.4, "max_memory_gb": 17.95, "ce": 0.045743657829007134, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 2.7676279544830322, "loss_scale": 4.0, "retries": 0, "seconds": 165.1, "max_memory_gb": 17.95, "ce": 0.4663136576418765, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 8.220417022705078, "loss_scale": 4.0, "retries": 0, "seconds": 166.8, "max_memory_gb": 17.95, "ce": 0.6635004502022639, "rank": 0.32918164134025574}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.7121272087097168, "loss_scale": 4.0, "retries": 0, "seconds": 168.5, "max_memory_gb": 17.95, "ce": 0.020487078465521336, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 5.2618865966796875, "loss_scale": 4.0, "retries": 0, "seconds": 170.3, "max_memory_gb": 17.95, "ce": 0.13967418365064077, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.8809877038002014, "loss_scale": 4.0, "retries": 0, "seconds": 171.9, "max_memory_gb": 17.95, "ce": 0.016384284649802794, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 1.4545197486877441, "loss_scale": 4.0, "retries": 0, "seconds": 173.7, "max_memory_gb": 17.95, "ce": 0.14686968627211172, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.518003523349762, "loss_scale": 4.0, "retries": 0, "seconds": 175.5, "max_memory_gb": 17.95, "ce": 0.02383773215842666, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 1.0843342542648315, "loss_scale": 4.0, "retries": 0, "seconds": 177.2, "max_memory_gb": 17.95, "ce": 0.04476981174229877, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.7159887552261353, "loss_scale": 4.0, "retries": 0, "seconds": 179.0, "max_memory_gb": 17.95, "ce": 0.025885448442750203, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 4.487462043762207, "loss_scale": 4.0, "retries": 0, "seconds": 180.6, "max_memory_gb": 17.95, "ce": 0.08896828563342751, "rank": 0.12495770305395126}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.8566101789474487, "loss_scale": 4.0, "retries": 0, "seconds": 182.2, "max_memory_gb": 17.95, "ce": 0.10517831009929068, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.8056060671806335, "loss_scale": 4.0, "retries": 0, "seconds": 183.8, "max_memory_gb": 17.95, "ce": 0.016606160439550877, "rank": 0.0}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 2.393240213394165, "loss_scale": 4.0, "retries": 0, "seconds": 185.5, "max_memory_gb": 17.95, "ce": 0.1615153535922218, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 2.2790610790252686, "loss_scale": 4.0, "retries": 0, "seconds": 187.3, "max_memory_gb": 17.95, "ce": 0.14941526908660308, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 2.0827009677886963, "loss_scale": 4.0, "retries": 0, "seconds": 189.0, "max_memory_gb": 17.95, "ce": 0.19067741351318546, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.92840975522995, "loss_scale": 4.0, "retries": 0, "seconds": 190.7, "max_memory_gb": 17.95, "ce": 0.0774993066152092, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.7625800371170044, "loss_scale": 4.0, "retries": 0, "seconds": 192.4, "max_memory_gb": 17.95, "ce": 0.056210325655683846, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.07338646054267883, "loss_scale": 4.0, "retries": 0, "seconds": 194.2, "max_memory_gb": 17.95, "ce": 0.0034030379601972527, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.8476897478103638, "loss_scale": 4.0, "retries": 0, "seconds": 195.9, "max_memory_gb": 17.95, "ce": 0.07595163524820236, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 1.5032949447631836, "loss_scale": 4.0, "retries": 0, "seconds": 197.7, "max_memory_gb": 17.95, "ce": 0.20734233051916817, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.23355990648269653, "loss_scale": 4.0, "retries": 0, "seconds": 199.3, "max_memory_gb": 17.95, "ce": 0.010406051274912897, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.589320719242096, "loss_scale": 4.0, "retries": 0, "seconds": 201.1, "max_memory_gb": 17.95, "ce": 0.16490724442337523, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.7949571013450623, "loss_scale": 4.0, "retries": 0, "seconds": 202.7, "max_memory_gb": 17.96, "ce": 0.02734322014657664, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 1.5392934083938599, "loss_scale": 4.0, "retries": 0, "seconds": 204.5, "max_memory_gb": 17.96, "ce": 0.24918905578397244, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.182085320353508, "loss_scale": 4.0, "retries": 0, "seconds": 206.2, "max_memory_gb": 17.96, "ce": 0.009616467910745996, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 3.545584201812744, "loss_scale": 4.0, "retries": 0, "seconds": 207.9, "max_memory_gb": 17.96, "ce": 0.3384404452826857, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 2.482783317565918, "loss_scale": 4.0, "retries": 0, "seconds": 209.7, "max_memory_gb": 17.96, "ce": 0.08853605989861535, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 1.179930567741394, "loss_scale": 4.0, "retries": 0, "seconds": 211.3, "max_memory_gb": 17.96, "ce": 0.11696663769544102, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 1.0104482173919678, "loss_scale": 4.0, "retries": 0, "seconds": 213.0, "max_memory_gb": 17.96, "ce": 0.0692871361534344, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.9266808032989502, "loss_scale": 4.0, "retries": 0, "seconds": 214.7, "max_memory_gb": 17.96, "ce": 0.5720868141779647, "rank": 0.0}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.7885400056838989, "loss_scale": 4.0, "retries": 0, "seconds": 216.4, "max_memory_gb": 17.96, "ce": 0.15342422071262263, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 2.7528200149536133, "loss_scale": 4.0, "retries": 0, "seconds": 218.1, "max_memory_gb": 17.96, "ce": 0.4719627324520843, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.8825628161430359, "loss_scale": 4.0, "retries": 0, "seconds": 219.8, "max_memory_gb": 17.96, "ce": 0.02076387269607949, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 1.6693754196166992, "loss_scale": 4.0, "retries": 0, "seconds": 221.5, "max_memory_gb": 17.96, "ce": 0.21996422899246681, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 1.2144259214401245, "loss_scale": 4.0, "retries": 0, "seconds": 223.2, "max_memory_gb": 17.96, "ce": 0.0608965868523228, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 1.8029146194458008, "loss_scale": 4.0, "retries": 0, "seconds": 224.9, "max_memory_gb": 17.96, "ce": 0.16365870491426904, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 3.09175181388855, "loss_scale": 4.0, "retries": 0, "seconds": 226.6, "max_memory_gb": 17.96, "ce": 0.33179004772682674, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 2.75349760055542, "loss_scale": 4.0, "retries": 0, "seconds": 228.2, "max_memory_gb": 17.96, "ce": 0.13798442333609273, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 1.5509408712387085, "loss_scale": 4.0, "retries": 0, "seconds": 229.9, "max_memory_gb": 17.96, "ce": 0.4031428583912202, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 1.6290028095245361, "loss_scale": 4.0, "retries": 0, "seconds": 231.6, "max_memory_gb": 17.96, "ce": 0.1346363581803871, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 0.5822528004646301, "loss_scale": 4.0, "retries": 0, "seconds": 233.3, "max_memory_gb": 17.96, "ce": 0.020888721992378123, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 1.078113079071045, "loss_scale": 4.0, "retries": 0, "seconds": 235.0, "max_memory_gb": 17.96, "ce": 0.059454762613313505, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.6088624000549316, "loss_scale": 4.0, "retries": 0, "seconds": 236.7, "max_memory_gb": 17.96, "ce": 0.03714517849221011, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 1.6960523128509521, "loss_scale": 4.0, "retries": 0, "seconds": 238.5, "max_memory_gb": 17.96, "ce": 0.10256793259759434, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.7093878984451294, "loss_scale": 4.0, "retries": 0, "seconds": 240.2, "max_memory_gb": 17.96, "ce": 0.02766222216632741, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.14672283828258514, "loss_scale": 4.0, "retries": 0, "seconds": 242.0, "max_memory_gb": 17.96, "ce": 0.007153373132496199, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.15216201543807983, "loss_scale": 4.0, "retries": 0, "seconds": 243.7, "max_memory_gb": 17.96, "ce": 0.008782875314864214, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.4965536594390869, "loss_scale": 4.0, "retries": 0, "seconds": 245.3, "max_memory_gb": 17.96, "ce": 0.03407812229306728, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 3.1548805236816406, "loss_scale": 4.0, "retries": 0, "seconds": 247.0, "max_memory_gb": 17.96, "ce": 0.39169223562703337, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 1.0135164260864258, "loss_scale": 4.0, "retries": 0, "seconds": 248.8, "max_memory_gb": 17.96, "ce": 0.05226680135820061, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.11312408745288849, "loss_scale": 4.0, "retries": 0, "seconds": 250.6, "max_memory_gb": 17.96, "ce": 0.00412457928632648, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.20413914322853088, "loss_scale": 4.0, "retries": 0, "seconds": 252.3, "max_memory_gb": 17.96, "ce": 0.0116689483729715, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 2.5286946296691895, "loss_scale": 4.0, "retries": 0, "seconds": 254.0, "max_memory_gb": 17.96, "ce": 0.5981695221271366, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.09557293355464935, "loss_scale": 4.0, "retries": 0, "seconds": 255.6, "max_memory_gb": 17.96, "ce": 0.0048822997464412765, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.11040588468313217, "loss_scale": 4.0, "retries": 0, "seconds": 257.3, "max_memory_gb": 17.96, "ce": 0.0075790989485540194, "rank": 0.12486720085144043}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.3363838493824005, "loss_scale": 4.0, "retries": 0, "seconds": 259.1, "max_memory_gb": 17.96, "ce": 0.05356814199330984, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.5893744826316833, "loss_scale": 4.0, "retries": 0, "seconds": 260.8, "max_memory_gb": 17.96, "ce": 0.021468774508321076, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 4.998746871948242, "loss_scale": 4.0, "retries": 0, "seconds": 262.5, "max_memory_gb": 17.96, "ce": 0.10685397903398552, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 5.170038223266602, "loss_scale": 4.0, "retries": 0, "seconds": 264.2, "max_memory_gb": 17.96, "ce": 0.14725372289831284, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.7913944721221924, "loss_scale": 4.0, "retries": 0, "seconds": 266.0, "max_memory_gb": 17.96, "ce": 0.0452986201053136, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 1.9000370502471924, "loss_scale": 4.0, "retries": 0, "seconds": 267.7, "max_memory_gb": 17.96, "ce": 0.13304550339398702, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.1862204670906067, "loss_scale": 4.0, "retries": 0, "seconds": 269.4, "max_memory_gb": 17.96, "ce": 0.005567610242451337, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.1925111562013626, "loss_scale": 4.0, "retries": 0, "seconds": 271.1, "max_memory_gb": 17.96, "ce": 0.011358291356373229, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 15.762581825256348, "loss_scale": 4.0, "retries": 0, "seconds": 272.8, "max_memory_gb": 17.96, "ce": 0.9858590550320514, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.49218621850013733, "loss_scale": 4.0, "retries": 0, "seconds": 274.5, "max_memory_gb": 18.03, "ce": 0.017740109546139138, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 1.1222636699676514, "loss_scale": 4.0, "retries": 0, "seconds": 276.2, "max_memory_gb": 18.03, "ce": 0.17519712926150532, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 1.383125901222229, "loss_scale": 4.0, "retries": 0, "seconds": 277.9, "max_memory_gb": 18.03, "ce": 0.07631991209564148, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.9682809710502625, "loss_scale": 4.0, "retries": 0, "seconds": 279.5, "max_memory_gb": 18.03, "ce": 0.05968412394031475, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 1.9433268308639526, "loss_scale": 4.0, "retries": 0, "seconds": 281.2, "max_memory_gb": 18.03, "ce": 0.14462546189315617, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.45094114542007446, "loss_scale": 4.0, "retries": 0, "seconds": 282.8, "max_memory_gb": 18.03, "ce": 0.043481856326252455, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.12347523868083954, "loss_scale": 4.0, "retries": 0, "seconds": 284.6, "max_memory_gb": 18.03, "ce": 0.010063619592074247, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 2.9058897495269775, "loss_scale": 4.0, "retries": 0, "seconds": 286.3, "max_memory_gb": 18.03, "ce": 0.42935996684536804, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.14470098912715912, "loss_scale": 4.0, "retries": 0, "seconds": 288.1, "max_memory_gb": 18.03, "ce": 0.006711228721542284, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.18034793436527252, "loss_scale": 4.0, "retries": 0, "seconds": 290.0, "max_memory_gb": 18.03, "ce": 0.007370872008323204, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 2.3789055347442627, "loss_scale": 4.0, "retries": 0, "seconds": 291.7, "max_memory_gb": 18.03, "ce": 0.4420224913264974, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.057306014001369476, "loss_scale": 4.0, "retries": 0, "seconds": 293.5, "max_memory_gb": 18.03, "ce": 0.0017578981717178976, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 1.2463833093643188, "loss_scale": 4.0, "retries": 0, "seconds": 295.2, "max_memory_gb": 18.03, "ce": 0.08538945409168264, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.16739432513713837, "loss_scale": 4.0, "retries": 0, "seconds": 297.0, "max_memory_gb": 18.03, "ce": 0.00874397990264697, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.7270529866218567, "loss_scale": 4.0, "retries": 0, "seconds": 298.7, "max_memory_gb": 18.03, "ce": 0.031194041534035932, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 1.1971346139907837, "loss_scale": 4.0, "retries": 0, "seconds": 300.3, "max_memory_gb": 18.03, "ce": 0.038122970292533864, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 3.2804746627807617, "loss_scale": 4.0, "retries": 0, "seconds": 302.0, "max_memory_gb": 18.03, "ce": 0.20236933373234933, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.17568013072013855, "loss_scale": 4.0, "retries": 0, "seconds": 303.7, "max_memory_gb": 18.03, "ce": 0.004088821284312871, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 1.6789796352386475, "loss_scale": 4.0, "retries": 0, "seconds": 305.4, "max_memory_gb": 18.03, "ce": 0.2828630994845298, "rank": 0.12501735985279083}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 0.8284793496131897, "loss_scale": 4.0, "retries": 0, "seconds": 307.1, "max_memory_gb": 18.03, "ce": 0.08218412719725166, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 1.5688693523406982, "loss_scale": 4.0, "retries": 0, "seconds": 308.8, "max_memory_gb": 18.03, "ce": 0.42002088506478685, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.8174856901168823, "loss_scale": 4.0, "retries": 0, "seconds": 310.6, "max_memory_gb": 18.03, "ce": 0.03714470966224326, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.4253508448600769, "loss_scale": 4.0, "retries": 0, "seconds": 312.3, "max_memory_gb": 18.03, "ce": 0.01893673035192478, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 0.817215085029602, "loss_scale": 4.0, "retries": 0, "seconds": 314.0, "max_memory_gb": 18.03, "ce": 0.07051601725470391, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.17141281068325043, "loss_scale": 4.0, "retries": 0, "seconds": 315.7, "max_memory_gb": 18.03, "ce": 0.003932591214493186, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.7805998921394348, "loss_scale": 4.0, "retries": 0, "seconds": 317.4, "max_memory_gb": 18.03, "ce": 0.018487936288693163, "rank": 0.19596075266599655}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 1.3048179149627686, "loss_scale": 4.0, "retries": 0, "seconds": 319.2, "max_memory_gb": 18.03, "ce": 0.08063357148785144, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 1.1065776348114014, "loss_scale": 4.0, "retries": 0, "seconds": 320.9, "max_memory_gb": 18.03, "ce": 0.04762916306845, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 1.4588388204574585, "loss_scale": 4.0, "retries": 0, "seconds": 322.7, "max_memory_gb": 18.03, "ce": 0.1978781204088591, "rank": 0.14794960618019104}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.5314759016036987, "loss_scale": 4.0, "retries": 0, "seconds": 324.4, "max_memory_gb": 18.03, "ce": 0.20386585266942348, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 2.7866454124450684, "loss_scale": 4.0, "retries": 0, "seconds": 326.0, "max_memory_gb": 18.03, "ce": 0.40360452653840184, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.14923252165317535, "loss_scale": 4.0, "retries": 0, "seconds": 327.8, "max_memory_gb": 18.03, "ce": 0.004260560726834228, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 2.110001564025879, "loss_scale": 4.0, "retries": 0, "seconds": 329.5, "max_memory_gb": 18.03, "ce": 0.20499969378579408, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.5235440731048584, "loss_scale": 4.0, "retries": 0, "seconds": 331.2, "max_memory_gb": 18.03, "ce": 0.42313599832050386, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.09560520946979523, "loss_scale": 4.0, "retries": 0, "seconds": 332.9, "max_memory_gb": 18.03, "ce": 0.0031176217544270912, "rank": 0.12499642372131348}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.4559430778026581, "loss_scale": 4.0, "retries": 0, "seconds": 334.6, "max_memory_gb": 18.03, "ce": 0.04290138734359061, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 1.7433849573135376, "loss_scale": 4.0, "retries": 0, "seconds": 336.4, "max_memory_gb": 18.03, "ce": 0.06182139001884934, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.9876763224601746, "loss_scale": 4.0, "retries": 0, "seconds": 338.2, "max_memory_gb": 18.03, "ce": 0.0210067394887119, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 1.1139222383499146, "loss_scale": 4.0, "retries": 0, "seconds": 339.9, "max_memory_gb": 18.03, "ce": 0.21235375705873594, "rank": 0.02750183641910553}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.5690248012542725, "loss_scale": 4.0, "retries": 0, "seconds": 341.6, "max_memory_gb": 18.03, "ce": 0.060443952097557485, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 2.507930278778076, "loss_scale": 4.0, "retries": 0, "seconds": 343.3, "max_memory_gb": 18.03, "ce": 0.07850681243871804, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 1.104411005973816, "loss_scale": 4.0, "retries": 0, "seconds": 345.1, "max_memory_gb": 18.03, "ce": 0.2425455759948818, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.6492277383804321, "loss_scale": 4.0, "retries": 0, "seconds": 346.8, "max_memory_gb": 18.03, "ce": 0.06015747194032883, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.15000256896018982, "loss_scale": 4.0, "retries": 0, "seconds": 348.5, "max_memory_gb": 18.03, "ce": 0.0040605738568046945, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.1093292087316513, "loss_scale": 4.0, "retries": 0, "seconds": 350.2, "max_memory_gb": 18.03, "ce": 0.005453366749861743, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 1.0286331176757812, "loss_scale": 4.0, "retries": 0, "seconds": 351.9, "max_memory_gb": 18.03, "ce": 0.01436744113379973, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 13.742362976074219, "loss_scale": 4.0, "retries": 0, "seconds": 353.7, "max_memory_gb": 18.03, "ce": 0.29431436751019646, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 1.046218991279602, "loss_scale": 4.0, "retries": 0, "seconds": 355.5, "max_memory_gb": 18.03, "ce": 0.011852200928842649, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.337576687335968, "loss_scale": 4.0, "retries": 0, "seconds": 357.2, "max_memory_gb": 18.03, "ce": 0.04771123411410372, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.25773364305496216, "loss_scale": 4.0, "retries": 0, "seconds": 358.9, "max_memory_gb": 18.03, "ce": 0.004481735377339646, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.6819812655448914, "loss_scale": 4.0, "retries": 0, "seconds": 360.7, "max_memory_gb": 18.03, "ce": 0.10843178355207783, "rank": 0.1250070035457611}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 2.7704672813415527, "loss_scale": 4.0, "retries": 0, "seconds": 362.4, "max_memory_gb": 18.03, "ce": 0.4967917439153098, "rank": 0.027808547019958496}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.8300899267196655, "loss_scale": 4.0, "retries": 0, "seconds": 364.2, "max_memory_gb": 18.03, "ce": 0.1299679868061503, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 1.8223453760147095, "loss_scale": 4.0, "retries": 0, "seconds": 365.9, "max_memory_gb": 18.03, "ce": 0.3075221265680739, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.6987003684043884, "loss_scale": 4.0, "retries": 0, "seconds": 367.5, "max_memory_gb": 18.03, "ce": 0.18387352199897578, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 3.348066806793213, "loss_scale": 4.0, "retries": 0, "seconds": 369.2, "max_memory_gb": 18.03, "ce": 0.2922649373740569, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 2.5310018062591553, "loss_scale": 4.0, "retries": 0, "seconds": 370.8, "max_memory_gb": 18.03, "ce": 0.5013706978497794, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 1.1709833145141602, "loss_scale": 4.0, "retries": 0, "seconds": 372.6, "max_memory_gb": 18.03, "ce": 0.033094069416620187, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.0502275787293911, "loss_scale": 4.0, "retries": 0, "seconds": 374.4, "max_memory_gb": 18.03, "ce": 0.001517024029908498, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 1.1461431980133057, "loss_scale": 4.0, "retries": 0, "seconds": 376.1, "max_memory_gb": 18.03, "ce": 0.1890546061185887, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.36937686800956726, "loss_scale": 4.0, "retries": 0, "seconds": 377.9, "max_memory_gb": 18.03, "ce": 0.02997199145556806, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 1.0992672443389893, "loss_scale": 4.0, "retries": 0, "seconds": 379.6, "max_memory_gb": 18.03, "ce": 0.05438124395641353, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 1.0235939025878906, "loss_scale": 4.0, "retries": 0, "seconds": 381.3, "max_memory_gb": 18.03, "ce": 0.0987011737197463, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 1.2368919849395752, "loss_scale": 4.0, "retries": 0, "seconds": 383.1, "max_memory_gb": 18.03, "ce": 0.04006234658663743, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 1.5117641687393188, "loss_scale": 4.0, "retries": 0, "seconds": 384.9, "max_memory_gb": 18.03, "ce": 0.08284073825416272, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.7299569249153137, "loss_scale": 4.0, "retries": 0, "seconds": 386.6, "max_memory_gb": 18.03, "ce": 0.05429392695077695, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 1.0592896938323975, "loss_scale": 4.0, "retries": 0, "seconds": 388.3, "max_memory_gb": 18.03, "ce": 0.04748950920111383, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.46701574325561523, "loss_scale": 4.0, "retries": 0, "seconds": 390.1, "max_memory_gb": 18.03, "ce": 0.024070632927760016, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.6041505336761475, "loss_scale": 4.0, "retries": 0, "seconds": 391.8, "max_memory_gb": 18.03, "ce": 0.027146668467139534, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.5110667943954468, "loss_scale": 4.0, "retries": 0, "seconds": 393.5, "max_memory_gb": 18.03, "ce": 0.020483505504671484, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 1.3033493757247925, "loss_scale": 4.0, "retries": 0, "seconds": 395.2, "max_memory_gb": 18.03, "ce": 0.0631419087667382, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 2.939439058303833, "loss_scale": 4.0, "retries": 0, "seconds": 396.9, "max_memory_gb": 18.03, "ce": 0.5079677784524392, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.7626463174819946, "loss_scale": 4.0, "retries": 0, "seconds": 398.6, "max_memory_gb": 18.03, "ce": 0.03320598747632175, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 1.6855394840240479, "loss_scale": 4.0, "retries": 0, "seconds": 400.3, "max_memory_gb": 18.03, "ce": 0.04032044930681877, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 1.938006043434143, "loss_scale": 4.0, "retries": 0, "seconds": 402.0, "max_memory_gb": 18.03, "ce": 0.0333436863256793, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.4925629496574402, "loss_scale": 4.0, "retries": 0, "seconds": 403.7, "max_memory_gb": 18.03, "ce": 0.03293077811213152, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 2.2888221740722656, "loss_scale": 4.0, "retries": 0, "seconds": 405.4, "max_memory_gb": 18.03, "ce": 0.27413622015023975, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.17035581171512604, "loss_scale": 4.0, "retries": 0, "seconds": 407.1, "max_memory_gb": 18.03, "ce": 0.011506043596455129, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.32905858755111694, "loss_scale": 4.0, "retries": 0, "seconds": 408.8, "max_memory_gb": 18.03, "ce": 0.035367750730074476, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.4931158423423767, "loss_scale": 4.0, "retries": 0, "seconds": 410.5, "max_memory_gb": 18.03, "ce": 0.030104267293154408, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.5025555491447449, "loss_scale": 4.0, "retries": 0, "seconds": 412.2, "max_memory_gb": 18.03, "ce": 0.10638998086506035, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 3.4851396083831787, "loss_scale": 4.0, "retries": 0, "seconds": 413.9, "max_memory_gb": 18.03, "ce": 0.1622880301074474, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 1.52577805519104, "loss_scale": 4.0, "retries": 0, "seconds": 415.7, "max_memory_gb": 18.03, "ce": 0.1574590904260731, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.32666832208633423, "loss_scale": 4.0, "retries": 0, "seconds": 417.3, "max_memory_gb": 18.03, "ce": 0.015112455133021285, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.13414521515369415, "loss_scale": 4.0, "retries": 0, "seconds": 419.0, "max_memory_gb": 18.03, "ce": 0.002907186006268603, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.27691033482551575, "loss_scale": 4.0, "retries": 0, "seconds": 420.7, "max_memory_gb": 18.03, "ce": 0.022034851436728786, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 1.3843523263931274, "loss_scale": 4.0, "retries": 0, "seconds": 422.4, "max_memory_gb": 18.03, "ce": 0.3456016260533943, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.05666644871234894, "loss_scale": 4.0, "retries": 0, "seconds": 424.1, "max_memory_gb": 18.03, "ce": 0.002017725795667502, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 6.215066909790039, "loss_scale": 4.0, "retries": 0, "seconds": 425.9, "max_memory_gb": 18.03, "ce": 0.19093623260232562, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.8380308747291565, "loss_scale": 4.0, "retries": 0, "seconds": 427.5, "max_memory_gb": 18.03, "ce": 0.025092952921113465, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.7380418181419373, "loss_scale": 4.0, "retries": 0, "seconds": 429.3, "max_memory_gb": 18.03, "ce": 0.018742907828709576, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.2891964614391327, "loss_scale": 4.0, "retries": 0, "seconds": 431.0, "max_memory_gb": 18.03, "ce": 0.016099260254122782, "rank": 0.0}

runs/single_internvl/removal/L20_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/single_internvl/removal/L20_s13/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.20.mlp.down_proj",
+    "model.language_model.layers.20.self_attn.o_proj",
+    "model.language_model.layers.20.self_attn.k_proj",
+    "model.language_model.layers.20.self_attn.q_proj",
+    "model.language_model.layers.20.self_attn.v_proj",
+    "model.language_model.layers.20.mlp.up_proj",
+    "model.language_model.layers.20.mlp.gate_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/single_internvl/removal/L20_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 20,
+    "rank": 16,
+    "trainable_parameters": 1212416
+  }
+}

runs/single_internvl/removal/L20_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 1.3248170614242554, "loss_scale": 4.0, "retries": 0, "seconds": 2.8, "max_memory_gb": 17.57, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 1.2384719848632812, "loss_scale": 4.0, "retries": 0, "seconds": 4.7, "max_memory_gb": 17.58, "ce": 0.7824874501420709, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 5.109800815582275, "loss_scale": 4.0, "retries": 0, "seconds": 6.4, "max_memory_gb": 17.58, "ce": 1.3777108415961266, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 2.956238031387329, "loss_scale": 4.0, "retries": 0, "seconds": 8.1, "max_memory_gb": 17.58, "ce": 0.7476939852676878, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.02965516783297062, "loss_scale": 4.0, "retries": 0, "seconds": 9.8, "max_memory_gb": 17.58, "ce": 0.04015987499860785, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 1.4094467163085938, "loss_scale": 4.0, "retries": 0, "seconds": 11.4, "max_memory_gb": 17.58, "ce": 0.42626966666284716, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 4.162441730499268, "loss_scale": 4.0, "retries": 0, "seconds": 13.0, "max_memory_gb": 17.58, "ce": 0.9658956021648919, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 4.144965648651123, "loss_scale": 4.0, "retries": 0, "seconds": 14.7, "max_memory_gb": 17.58, "ce": 0.9171806639060378, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 2.9902961254119873, "loss_scale": 4.0, "retries": 0, "seconds": 16.3, "max_memory_gb": 17.58, "ce": 0.6354889604263008, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.38416096568107605, "loss_scale": 4.0, "retries": 0, "seconds": 18.0, "max_memory_gb": 17.58, "ce": 0.17693274789780844, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 3.7722349166870117, "loss_scale": 4.0, "retries": 0, "seconds": 19.7, "max_memory_gb": 17.58, "ce": 0.6574110079745878, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 2.425395965576172, "loss_scale": 4.0, "retries": 0, "seconds": 21.3, "max_memory_gb": 17.58, "ce": 0.41981448248225206, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 1.8928906917572021, "loss_scale": 4.0, "retries": 0, "seconds": 23.0, "max_memory_gb": 17.58, "ce": 0.5102844374159758, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 2.77675461769104, "loss_scale": 4.0, "retries": 0, "seconds": 24.6, "max_memory_gb": 17.58, "ce": 0.6920175421983004, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 1.8077713251113892, "loss_scale": 4.0, "retries": 0, "seconds": 26.3, "max_memory_gb": 17.58, "ce": 0.4187801948808101, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.7565835118293762, "loss_scale": 4.0, "retries": 0, "seconds": 27.9, "max_memory_gb": 17.58, "ce": 0.18777976187993772, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.8144195079803467, "loss_scale": 4.0, "retries": 0, "seconds": 29.5, "max_memory_gb": 17.58, "ce": 0.41520002750621643, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 0.5559887290000916, "loss_scale": 4.0, "retries": 0, "seconds": 31.2, "max_memory_gb": 17.59, "ce": 0.3506679717756924, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 0.4364861845970154, "loss_scale": 4.0, "retries": 0, "seconds": 32.8, "max_memory_gb": 17.59, "ce": 0.4795566856628284, "rank": 0.16515236347913742}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.029646212235093117, "loss_scale": 4.0, "retries": 0, "seconds": 34.5, "max_memory_gb": 17.59, "ce": 0.04050248508065124, "rank": 0.12145307660102844}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.23729142546653748, "loss_scale": 4.0, "retries": 0, "seconds": 36.1, "max_memory_gb": 17.59, "ce": 0.3003283169127826, "rank": 0.05966964364051819}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.3447206914424896, "loss_scale": 4.0, "retries": 0, "seconds": 37.8, "max_memory_gb": 17.59, "ce": 0.42022433687816374, "rank": 0.14157499372959137}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.7777454853057861, "loss_scale": 4.0, "retries": 0, "seconds": 39.5, "max_memory_gb": 17.59, "ce": 0.5261195447565115, "rank": 0.11903730779886246}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 0.33806443214416504, "loss_scale": 4.0, "retries": 0, "seconds": 41.1, "max_memory_gb": 17.59, "ce": 0.4926909860223532, "rank": 0.12483417242765427}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 0.757274329662323, "loss_scale": 4.0, "retries": 0, "seconds": 42.8, "max_memory_gb": 17.59, "ce": 0.6047996080160374, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 0.39827030897140503, "loss_scale": 4.0, "retries": 0, "seconds": 44.5, "max_memory_gb": 17.59, "ce": 0.5366712024551816, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 0.535404622554779, "loss_scale": 4.0, "retries": 0, "seconds": 46.1, "max_memory_gb": 17.59, "ce": 0.46777628404743155, "rank": 0.31206562370061874}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 0.9332696199417114, "loss_scale": 4.0, "retries": 0, "seconds": 47.8, "max_memory_gb": 17.59, "ce": 0.74329175776802, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 0.6849414706230164, "loss_scale": 4.0, "retries": 0, "seconds": 49.5, "max_memory_gb": 17.59, "ce": 0.39736983738839626, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 0.26995357871055603, "loss_scale": 4.0, "retries": 0, "seconds": 51.2, "max_memory_gb": 17.59, "ce": 0.23434928274946287, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 0.7613170146942139, "loss_scale": 4.0, "retries": 0, "seconds": 52.9, "max_memory_gb": 17.59, "ce": 0.4772600980941206, "rank": 0.09859605133533478}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 0.33627185225486755, "loss_scale": 4.0, "retries": 0, "seconds": 54.5, "max_memory_gb": 17.59, "ce": 0.42011804605135694, "rank": 0.07078618556261063}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.42322227358818054, "loss_scale": 4.0, "retries": 0, "seconds": 56.2, "max_memory_gb": 17.59, "ce": 0.2613613706198521, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.3695281147956848, "loss_scale": 4.0, "retries": 0, "seconds": 57.9, "max_memory_gb": 17.59, "ce": 0.295534817942098, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.2578314244747162, "loss_scale": 4.0, "retries": 0, "seconds": 59.5, "max_memory_gb": 17.59, "ce": 0.19558078048794414, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 0.44412994384765625, "loss_scale": 4.0, "retries": 0, "seconds": 61.2, "max_memory_gb": 17.59, "ce": 0.30144979849865194, "rank": 0.021687105298042297}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.602846086025238, "loss_scale": 4.0, "retries": 0, "seconds": 62.8, "max_memory_gb": 17.59, "ce": 0.502499881433323, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.46697595715522766, "loss_scale": 4.0, "retries": 0, "seconds": 64.4, "max_memory_gb": 17.59, "ce": 0.5242161434143782, "rank": 0.020389646291732788}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 0.6818740367889404, "loss_scale": 4.0, "retries": 0, "seconds": 66.1, "max_memory_gb": 17.59, "ce": 0.3584217115640058, "rank": 0.12494106590747833}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.33185046911239624, "loss_scale": 4.0, "retries": 0, "seconds": 67.8, "max_memory_gb": 17.59, "ce": 0.22227050220317324, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.7958202362060547, "loss_scale": 4.0, "retries": 0, "seconds": 69.5, "max_memory_gb": 17.59, "ce": 0.4280650581640657, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.24461588263511658, "loss_scale": 4.0, "retries": 0, "seconds": 71.2, "max_memory_gb": 17.59, "ce": 0.1830150668756687, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.30144640803337097, "loss_scale": 4.0, "retries": 0, "seconds": 72.8, "max_memory_gb": 17.59, "ce": 0.19797057245159522, "rank": 0.20554489642381668}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 2.4830162525177, "loss_scale": 4.0, "retries": 0, "seconds": 74.5, "max_memory_gb": 17.59, "ce": 0.550724946133414, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 1.4131746292114258, "loss_scale": 4.0, "retries": 0, "seconds": 76.2, "max_memory_gb": 17.59, "ce": 0.6487055391189642, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.6518539190292358, "loss_scale": 4.0, "retries": 0, "seconds": 77.8, "max_memory_gb": 17.59, "ce": 0.29653711180435494, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.26562952995300293, "loss_scale": 4.0, "retries": 0, "seconds": 79.5, "max_memory_gb": 17.59, "ce": 0.18197981097910088, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.377778023481369, "loss_scale": 4.0, "retries": 0, "seconds": 81.1, "max_memory_gb": 17.59, "ce": 0.28210455729276873, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.5118017792701721, "loss_scale": 4.0, "retries": 0, "seconds": 82.8, "max_memory_gb": 17.59, "ce": 0.285062307066255, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.7390931844711304, "loss_scale": 4.0, "retries": 0, "seconds": 84.4, "max_memory_gb": 17.59, "ce": 0.3128355125154485, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 2.4755969047546387, "loss_scale": 4.0, "retries": 0, "seconds": 86.1, "max_memory_gb": 17.59, "ce": 0.3736811113085423, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.5147281289100647, "loss_scale": 4.0, "retries": 0, "seconds": 87.9, "max_memory_gb": 17.59, "ce": 0.24924184726842213, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 1.0420324802398682, "loss_scale": 4.0, "retries": 0, "seconds": 89.6, "max_memory_gb": 17.59, "ce": 0.3881203413475305, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 1.003934621810913, "loss_scale": 4.0, "retries": 0, "seconds": 91.2, "max_memory_gb": 17.59, "ce": 0.32720599288586527, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.40308648347854614, "loss_scale": 4.0, "retries": 0, "seconds": 92.8, "max_memory_gb": 17.59, "ce": 0.15514672332210466, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.8541567921638489, "loss_scale": 4.0, "retries": 0, "seconds": 94.5, "max_memory_gb": 17.59, "ce": 0.3066054426599294, "rank": 0.06062159687280655}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.29986417293548584, "loss_scale": 4.0, "retries": 0, "seconds": 96.1, "max_memory_gb": 17.59, "ce": 0.11495981673942879, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.5297765135765076, "loss_scale": 4.0, "retries": 0, "seconds": 97.7, "max_memory_gb": 17.59, "ce": 0.1489888787473319, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.4651786983013153, "loss_scale": 4.0, "retries": 0, "seconds": 99.4, "max_memory_gb": 17.59, "ce": 0.12531248219602276, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.6232925653457642, "loss_scale": 4.0, "retries": 0, "seconds": 101.1, "max_memory_gb": 17.59, "ce": 0.18970175505091902, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.847366213798523, "loss_scale": 4.0, "retries": 0, "seconds": 102.8, "max_memory_gb": 17.59, "ce": 0.41328949184389785, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.35418668389320374, "loss_scale": 4.0, "retries": 0, "seconds": 104.5, "max_memory_gb": 17.59, "ce": 0.09996887863962911, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.7452492713928223, "loss_scale": 4.0, "retries": 0, "seconds": 106.1, "max_memory_gb": 17.59, "ce": 0.17076358903432265, "rank": 0.0}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.7108998894691467, "loss_scale": 4.0, "retries": 0, "seconds": 107.8, "max_memory_gb": 17.59, "ce": 0.15674594382289797, "rank": 0.06574442237615585}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 1.1840893030166626, "loss_scale": 4.0, "retries": 0, "seconds": 109.5, "max_memory_gb": 17.59, "ce": 0.26780672720633447, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.7047975063323975, "loss_scale": 4.0, "retries": 0, "seconds": 111.1, "max_memory_gb": 17.59, "ce": 0.19848640740065093, "rank": 0.12441647052764893}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.6788487434387207, "loss_scale": 4.0, "retries": 0, "seconds": 112.8, "max_memory_gb": 17.59, "ce": 0.1043201195425354, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.3467685878276825, "loss_scale": 4.0, "retries": 0, "seconds": 114.5, "max_memory_gb": 17.59, "ce": 0.05560625926045759, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 1.6751816272735596, "loss_scale": 4.0, "retries": 0, "seconds": 116.1, "max_memory_gb": 17.59, "ce": 0.47516433300916106, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.5448243021965027, "loss_scale": 4.0, "retries": 0, "seconds": 117.8, "max_memory_gb": 17.59, "ce": 0.08064008312067017, "rank": 0.10392063111066818}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.6383655071258545, "loss_scale": 4.0, "retries": 0, "seconds": 119.4, "max_memory_gb": 17.59, "ce": 0.08112880953785861, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.7016156911849976, "loss_scale": 4.0, "retries": 0, "seconds": 121.1, "max_memory_gb": 17.59, "ce": 0.18093226419296116, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 0.9035090804100037, "loss_scale": 4.0, "retries": 0, "seconds": 122.6, "max_memory_gb": 17.59, "ce": 0.16497334863561264, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.6130346059799194, "loss_scale": 4.0, "retries": 0, "seconds": 124.3, "max_memory_gb": 17.59, "ce": 0.13540600313581308, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 1.5631300210952759, "loss_scale": 4.0, "retries": 0, "seconds": 125.9, "max_memory_gb": 17.59, "ce": 0.7225477027241141, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 1.3470247983932495, "loss_scale": 4.0, "retries": 0, "seconds": 127.6, "max_memory_gb": 17.59, "ce": 0.36954968952341005, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.3494624197483063, "loss_scale": 4.0, "retries": 0, "seconds": 129.2, "max_memory_gb": 17.59, "ce": 0.04855290683917701, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.501125693321228, "loss_scale": 4.0, "retries": 0, "seconds": 130.8, "max_memory_gb": 17.59, "ce": 0.06979235334983969, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.2794855833053589, "loss_scale": 4.0, "retries": 0, "seconds": 132.4, "max_memory_gb": 17.59, "ce": 0.021353352535697923, "rank": 0.04793989658355713}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.33319729566574097, "loss_scale": 4.0, "retries": 0, "seconds": 134.1, "max_memory_gb": 17.59, "ce": 0.044923746906988526, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.606386661529541, "loss_scale": 4.0, "retries": 0, "seconds": 135.6, "max_memory_gb": 17.59, "ce": 0.07302828691899776, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 1.0947648286819458, "loss_scale": 4.0, "retries": 0, "seconds": 137.3, "max_memory_gb": 17.59, "ce": 0.07840179314371198, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 1.208918571472168, "loss_scale": 4.0, "retries": 0, "seconds": 138.9, "max_memory_gb": 17.59, "ce": 0.27233125105885847, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.3624802231788635, "loss_scale": 4.0, "retries": 0, "seconds": 140.6, "max_memory_gb": 17.59, "ce": 0.031966509057838266, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.08097346872091293, "loss_scale": 4.0, "retries": 0, "seconds": 142.2, "max_memory_gb": 17.59, "ce": 0.0063312816196230415, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.9475054740905762, "loss_scale": 4.0, "retries": 0, "seconds": 143.8, "max_memory_gb": 17.59, "ce": 0.09608010668307543, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.8988863229751587, "loss_scale": 4.0, "retries": 0, "seconds": 145.4, "max_memory_gb": 17.59, "ce": 0.13006153090827866, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.43743932247161865, "loss_scale": 4.0, "retries": 0, "seconds": 147.0, "max_memory_gb": 17.59, "ce": 0.059440484503738844, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.1973978728055954, "loss_scale": 4.0, "retries": 0, "seconds": 148.7, "max_memory_gb": 17.59, "ce": 0.023544599513115827, "rank": 0.12499880790710449}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.8198279142379761, "loss_scale": 4.0, "retries": 0, "seconds": 150.3, "max_memory_gb": 17.59, "ce": 0.06744522568624234, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 2.825993537902832, "loss_scale": 4.0, "retries": 0, "seconds": 152.0, "max_memory_gb": 17.59, "ce": 0.6226940463639039, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 1.3958381414413452, "loss_scale": 4.0, "retries": 0, "seconds": 153.6, "max_memory_gb": 17.59, "ce": 0.09657777488837382, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 0.5882135033607483, "loss_scale": 4.0, "retries": 0, "seconds": 155.2, "max_memory_gb": 17.59, "ce": 0.06184629298513755, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 0.49391597509384155, "loss_scale": 4.0, "retries": 0, "seconds": 156.9, "max_memory_gb": 17.59, "ce": 0.0372162583807949, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 2.6204137802124023, "loss_scale": 4.0, "retries": 0, "seconds": 158.5, "max_memory_gb": 17.59, "ce": 0.46699163017910905, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 3.6199800968170166, "loss_scale": 4.0, "retries": 0, "seconds": 160.2, "max_memory_gb": 17.59, "ce": 0.4767184806987643, "rank": 0.2681138515472412}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.3099839687347412, "loss_scale": 4.0, "retries": 0, "seconds": 161.9, "max_memory_gb": 17.59, "ce": 0.021839680273842532, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 2.3806235790252686, "loss_scale": 4.0, "retries": 0, "seconds": 163.5, "max_memory_gb": 17.59, "ce": 0.101331433874293, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.10423846542835236, "loss_scale": 4.0, "retries": 0, "seconds": 165.1, "max_memory_gb": 17.59, "ce": 0.01168227733819549, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.868965208530426, "loss_scale": 4.0, "retries": 0, "seconds": 166.8, "max_memory_gb": 17.59, "ce": 0.1356794231251115, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.39196497201919556, "loss_scale": 4.0, "retries": 0, "seconds": 168.5, "max_memory_gb": 17.59, "ce": 0.03173230669199256, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 1.017507791519165, "loss_scale": 4.0, "retries": 0, "seconds": 170.1, "max_memory_gb": 17.59, "ce": 0.08712510484474478, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.3097819983959198, "loss_scale": 4.0, "retries": 0, "seconds": 171.8, "max_memory_gb": 17.59, "ce": 0.024059477051196154, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.36719995737075806, "loss_scale": 4.0, "retries": 0, "seconds": 173.3, "max_memory_gb": 17.59, "ce": 0.0359476719477243, "rank": 0.12487136572599411}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.5467187762260437, "loss_scale": 4.0, "retries": 0, "seconds": 175.0, "max_memory_gb": 17.59, "ce": 0.0924426605051849, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.3512974977493286, "loss_scale": 4.0, "retries": 0, "seconds": 176.6, "max_memory_gb": 17.59, "ce": 0.02080597434542142, "rank": 0.0}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 1.1715149879455566, "loss_scale": 4.0, "retries": 0, "seconds": 178.3, "max_memory_gb": 17.59, "ce": 0.10674543466484465, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 2.4426019191741943, "loss_scale": 4.0, "retries": 0, "seconds": 179.9, "max_memory_gb": 17.59, "ce": 0.2369475675313879, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 1.1270232200622559, "loss_scale": 4.0, "retries": 0, "seconds": 181.6, "max_memory_gb": 17.59, "ce": 0.2130773335811682, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.9256296157836914, "loss_scale": 4.0, "retries": 0, "seconds": 183.2, "max_memory_gb": 17.59, "ce": 0.04827260549063794, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.4112365245819092, "loss_scale": 4.0, "retries": 0, "seconds": 184.9, "max_memory_gb": 17.59, "ce": 0.04634200019972923, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.07654301077127457, "loss_scale": 4.0, "retries": 0, "seconds": 186.5, "max_memory_gb": 17.59, "ce": 0.009759374461282277, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.4079480767250061, "loss_scale": 4.0, "retries": 0, "seconds": 188.2, "max_memory_gb": 17.59, "ce": 0.06491232560074423, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.7702516317367554, "loss_scale": 4.0, "retries": 0, "seconds": 189.9, "max_memory_gb": 17.59, "ce": 0.1642963382328162, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.359970360994339, "loss_scale": 4.0, "retries": 0, "seconds": 191.5, "max_memory_gb": 17.59, "ce": 0.010285147509421222, "rank": 0.02061677724123001}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.3699680268764496, "loss_scale": 4.0, "retries": 0, "seconds": 193.1, "max_memory_gb": 17.59, "ce": 0.14758534595239325, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.4168228805065155, "loss_scale": 4.0, "retries": 0, "seconds": 194.8, "max_memory_gb": 17.6, "ce": 0.022096936547313817, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 1.208003282546997, "loss_scale": 4.0, "retries": 0, "seconds": 196.5, "max_memory_gb": 17.6, "ce": 0.27592476655263454, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.09904158860445023, "loss_scale": 4.0, "retries": 0, "seconds": 198.1, "max_memory_gb": 17.6, "ce": 0.009314706386248872, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 2.7244908809661865, "loss_scale": 4.0, "retries": 0, "seconds": 199.8, "max_memory_gb": 17.6, "ce": 0.2316901021549711, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.5128899812698364, "loss_scale": 4.0, "retries": 0, "seconds": 201.4, "max_memory_gb": 17.6, "ce": 0.08052139037044981, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 1.3598326444625854, "loss_scale": 4.0, "retries": 0, "seconds": 203.2, "max_memory_gb": 17.6, "ce": 0.15241009384226345, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.6233360767364502, "loss_scale": 4.0, "retries": 0, "seconds": 204.8, "max_memory_gb": 17.6, "ce": 0.07702596983290277, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.4059157371520996, "loss_scale": 4.0, "retries": 0, "seconds": 206.6, "max_memory_gb": 17.6, "ce": 0.5916175217716955, "rank": 0.0}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.6439822912216187, "loss_scale": 4.0, "retries": 0, "seconds": 208.2, "max_memory_gb": 17.6, "ce": 0.17829090634768363, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 1.8266113996505737, "loss_scale": 4.0, "retries": 0, "seconds": 209.8, "max_memory_gb": 17.6, "ce": 0.5517441836418584, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.4318140745162964, "loss_scale": 4.0, "retries": 0, "seconds": 211.5, "max_memory_gb": 17.6, "ce": 0.015599805974034098, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 0.9903481006622314, "loss_scale": 4.0, "retries": 0, "seconds": 213.2, "max_memory_gb": 17.6, "ce": 0.2000095826369943, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 0.7854935526847839, "loss_scale": 4.0, "retries": 0, "seconds": 215.0, "max_memory_gb": 17.6, "ce": 0.0743751452973811, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 0.8568882346153259, "loss_scale": 4.0, "retries": 0, "seconds": 216.7, "max_memory_gb": 17.6, "ce": 0.17114273291372228, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 3.0113346576690674, "loss_scale": 4.0, "retries": 0, "seconds": 218.4, "max_memory_gb": 17.6, "ce": 0.3385119961021701, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 1.7646310329437256, "loss_scale": 4.0, "retries": 0, "seconds": 220.0, "max_memory_gb": 17.6, "ce": 0.1814502396609896, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 1.4647905826568604, "loss_scale": 4.0, "retries": 0, "seconds": 221.7, "max_memory_gb": 17.6, "ce": 0.43059577367966995, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.7669066786766052, "loss_scale": 4.0, "retries": 0, "seconds": 223.3, "max_memory_gb": 17.6, "ce": 0.13209213728259783, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 0.3245958089828491, "loss_scale": 4.0, "retries": 0, "seconds": 224.9, "max_memory_gb": 17.6, "ce": 0.028579757825355045, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 0.6928563117980957, "loss_scale": 4.0, "retries": 0, "seconds": 226.5, "max_memory_gb": 17.6, "ce": 0.05322021059691906, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.6731410622596741, "loss_scale": 4.0, "retries": 0, "seconds": 228.2, "max_memory_gb": 17.6, "ce": 0.05203045713278698, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 1.2928298711776733, "loss_scale": 4.0, "retries": 0, "seconds": 229.9, "max_memory_gb": 17.6, "ce": 0.12207354862766806, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.5076701641082764, "loss_scale": 4.0, "retries": 0, "seconds": 231.6, "max_memory_gb": 17.6, "ce": 0.037042719501187094, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.10690862685441971, "loss_scale": 4.0, "retries": 0, "seconds": 233.4, "max_memory_gb": 17.6, "ce": 0.010468310718806606, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.11347521841526031, "loss_scale": 4.0, "retries": 0, "seconds": 235.1, "max_memory_gb": 17.6, "ce": 0.01223999149078736, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.4943484365940094, "loss_scale": 4.0, "retries": 0, "seconds": 236.7, "max_memory_gb": 17.6, "ce": 0.0427435208184761, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 2.440189838409424, "loss_scale": 4.0, "retries": 0, "seconds": 238.4, "max_memory_gb": 17.6, "ce": 0.26871575303835016, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.8709987998008728, "loss_scale": 4.0, "retries": 0, "seconds": 240.0, "max_memory_gb": 17.6, "ce": 0.07384078775066882, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.11388131231069565, "loss_scale": 4.0, "retries": 0, "seconds": 241.7, "max_memory_gb": 17.6, "ce": 0.0067133471438864945, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.14201655983924866, "loss_scale": 4.0, "retries": 0, "seconds": 243.3, "max_memory_gb": 17.6, "ce": 0.016816207440569997, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 1.8831576108932495, "loss_scale": 4.0, "retries": 0, "seconds": 245.0, "max_memory_gb": 17.6, "ce": 0.56737774818248, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.09581546485424042, "loss_scale": 4.0, "retries": 0, "seconds": 246.6, "max_memory_gb": 17.6, "ce": 0.00811127827364544, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.09087590873241425, "loss_scale": 4.0, "retries": 0, "seconds": 248.3, "max_memory_gb": 17.6, "ce": 0.010425260028569028, "rank": 0.12477973848581314}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.30501580238342285, "loss_scale": 4.0, "retries": 0, "seconds": 249.9, "max_memory_gb": 17.6, "ce": 0.05750883455039002, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.8945126533508301, "loss_scale": 4.0, "retries": 0, "seconds": 251.7, "max_memory_gb": 17.6, "ce": 0.032457682583299174, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 1.2139075994491577, "loss_scale": 4.0, "retries": 0, "seconds": 253.3, "max_memory_gb": 17.6, "ce": 0.07410403071844485, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 1.0436882972717285, "loss_scale": 4.0, "retries": 0, "seconds": 255.0, "max_memory_gb": 17.6, "ce": 0.12929106383126054, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.6537646055221558, "loss_scale": 4.0, "retries": 0, "seconds": 256.7, "max_memory_gb": 17.6, "ce": 0.06143422590321279, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 1.50250244140625, "loss_scale": 4.0, "retries": 0, "seconds": 258.3, "max_memory_gb": 17.6, "ce": 0.1270891885878882, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.07454925030469894, "loss_scale": 4.0, "retries": 0, "seconds": 260.0, "max_memory_gb": 17.6, "ce": 0.004867368535542482, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.13234902918338776, "loss_scale": 4.0, "retries": 0, "seconds": 261.7, "max_memory_gb": 17.6, "ce": 0.012097843460651347, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 7.809901237487793, "loss_scale": 4.0, "retries": 0, "seconds": 263.3, "max_memory_gb": 17.6, "ce": 0.6469625197914866, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.21433056890964508, "loss_scale": 4.0, "retries": 0, "seconds": 265.0, "max_memory_gb": 17.66, "ce": 0.015130584448343143, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 0.73319411277771, "loss_scale": 4.0, "retries": 0, "seconds": 266.7, "max_memory_gb": 17.66, "ce": 0.16626685482333414, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 0.9549387693405151, "loss_scale": 4.0, "retries": 0, "seconds": 268.4, "max_memory_gb": 17.66, "ce": 0.10028422738650988, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.8540410399436951, "loss_scale": 4.0, "retries": 0, "seconds": 270.0, "max_memory_gb": 17.66, "ce": 0.06377842818255886, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 1.104090690612793, "loss_scale": 4.0, "retries": 0, "seconds": 271.6, "max_memory_gb": 17.66, "ce": 0.12934907263843343, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.7758826613426208, "loss_scale": 4.0, "retries": 0, "seconds": 273.3, "max_memory_gb": 17.66, "ce": 0.06589042568884906, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.1277836114168167, "loss_scale": 4.0, "retries": 0, "seconds": 275.0, "max_memory_gb": 17.66, "ce": 0.012441735840184265, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 2.306980609893799, "loss_scale": 4.0, "retries": 0, "seconds": 276.7, "max_memory_gb": 17.66, "ce": 0.39501382331945933, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.2591552138328552, "loss_scale": 4.0, "retries": 0, "seconds": 278.8, "max_memory_gb": 17.66, "ce": 0.01724589109653607, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.18383586406707764, "loss_scale": 4.0, "retries": 0, "seconds": 280.6, "max_memory_gb": 17.66, "ce": 0.012980338178749662, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 1.0632152557373047, "loss_scale": 4.0, "retries": 0, "seconds": 282.4, "max_memory_gb": 17.66, "ce": 0.35163400595774874, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.10967288166284561, "loss_scale": 4.0, "retries": 0, "seconds": 284.0, "max_memory_gb": 17.66, "ce": 0.005776506844085816, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.7709283232688904, "loss_scale": 4.0, "retries": 0, "seconds": 285.7, "max_memory_gb": 17.66, "ce": 0.0863336573252127, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.24423445761203766, "loss_scale": 4.0, "retries": 0, "seconds": 287.4, "max_memory_gb": 17.66, "ce": 0.02258660578809213, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.4410126209259033, "loss_scale": 4.0, "retries": 0, "seconds": 289.0, "max_memory_gb": 17.66, "ce": 0.03946826500759926, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.3554149270057678, "loss_scale": 4.0, "retries": 0, "seconds": 290.6, "max_memory_gb": 17.66, "ce": 0.02438337010039504, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 2.222033977508545, "loss_scale": 4.0, "retries": 0, "seconds": 292.3, "max_memory_gb": 17.66, "ce": 0.17104794818442315, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.20237882435321808, "loss_scale": 4.0, "retries": 0, "seconds": 294.0, "max_memory_gb": 17.66, "ce": 0.010050334745756118, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 0.9382278919219971, "loss_scale": 4.0, "retries": 0, "seconds": 295.6, "max_memory_gb": 17.66, "ce": 0.2743527168313449, "rank": 0.1250115931034088}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 0.37304040789604187, "loss_scale": 4.0, "retries": 0, "seconds": 297.3, "max_memory_gb": 17.66, "ce": 0.052845013691694476, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 1.1740138530731201, "loss_scale": 4.0, "retries": 0, "seconds": 298.8, "max_memory_gb": 17.66, "ce": 0.4107948995347215, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.428648442029953, "loss_scale": 4.0, "retries": 0, "seconds": 300.5, "max_memory_gb": 17.66, "ce": 0.038249150798947085, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.467080295085907, "loss_scale": 4.0, "retries": 0, "seconds": 302.1, "max_memory_gb": 17.66, "ce": 0.03697050596383633, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 0.9071320295333862, "loss_scale": 4.0, "retries": 0, "seconds": 303.8, "max_memory_gb": 17.66, "ce": 0.09585957341187168, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.18613246083259583, "loss_scale": 4.0, "retries": 0, "seconds": 305.5, "max_memory_gb": 17.66, "ce": 0.006920430740819938, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.4217962920665741, "loss_scale": 4.0, "retries": 0, "seconds": 307.1, "max_memory_gb": 17.66, "ce": 0.018030691863259563, "rank": 0.20165814459323883}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.7113214731216431, "loss_scale": 4.0, "retries": 0, "seconds": 308.8, "max_memory_gb": 17.66, "ce": 0.0757856732707296, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 1.1080610752105713, "loss_scale": 4.0, "retries": 0, "seconds": 310.4, "max_memory_gb": 17.66, "ce": 0.07093630506392401, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.833413302898407, "loss_scale": 4.0, "retries": 0, "seconds": 312.2, "max_memory_gb": 17.66, "ce": 0.1674217802574276, "rank": 0.13978828489780426}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.364965558052063, "loss_scale": 4.0, "retries": 0, "seconds": 313.8, "max_memory_gb": 17.66, "ce": 0.21850280452645165, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 1.881251573562622, "loss_scale": 4.0, "retries": 0, "seconds": 315.4, "max_memory_gb": 17.66, "ce": 0.39655227913317503, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.24480804800987244, "loss_scale": 4.0, "retries": 0, "seconds": 317.0, "max_memory_gb": 17.66, "ce": 0.011470006282252143, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 1.8252686262130737, "loss_scale": 4.0, "retries": 0, "seconds": 318.6, "max_memory_gb": 17.66, "ce": 0.23156089476833586, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.0739343166351318, "loss_scale": 4.0, "retries": 0, "seconds": 320.3, "max_memory_gb": 17.66, "ce": 0.4144069765898166, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.0976019874215126, "loss_scale": 4.0, "retries": 0, "seconds": 321.9, "max_memory_gb": 17.66, "ce": 0.005884563058316417, "rank": 0.1250099092721939}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.68837970495224, "loss_scale": 4.0, "retries": 0, "seconds": 323.6, "max_memory_gb": 17.66, "ce": 0.06472094654600369, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.9393793940544128, "loss_scale": 4.0, "retries": 0, "seconds": 325.4, "max_memory_gb": 17.66, "ce": 0.08176569271381595, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.4187259376049042, "loss_scale": 4.0, "retries": 0, "seconds": 327.1, "max_memory_gb": 17.66, "ce": 0.02974016827374726, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.8604319095611572, "loss_scale": 4.0, "retries": 0, "seconds": 328.8, "max_memory_gb": 17.66, "ce": 0.2178933883733407, "rank": 0.014837250113487244}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.34226173162460327, "loss_scale": 4.0, "retries": 0, "seconds": 330.4, "max_memory_gb": 17.66, "ce": 0.060831408387457486, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 1.1244175434112549, "loss_scale": 4.0, "retries": 0, "seconds": 332.1, "max_memory_gb": 17.66, "ce": 0.10405615691524872, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 1.118219017982483, "loss_scale": 4.0, "retries": 0, "seconds": 333.7, "max_memory_gb": 17.66, "ce": 0.25722983686137013, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.4324241280555725, "loss_scale": 4.0, "retries": 0, "seconds": 335.4, "max_memory_gb": 17.66, "ce": 0.05308719135064166, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.07010610401630402, "loss_scale": 4.0, "retries": 0, "seconds": 336.9, "max_memory_gb": 17.66, "ce": 0.004829055309301111, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.09233681112527847, "loss_scale": 4.0, "retries": 0, "seconds": 338.6, "max_memory_gb": 17.66, "ce": 0.008411809641984291, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 1.0117180347442627, "loss_scale": 4.0, "retries": 0, "seconds": 340.2, "max_memory_gb": 17.66, "ce": 0.03919389910879545, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 1.971084713935852, "loss_scale": 4.0, "retries": 0, "seconds": 341.8, "max_memory_gb": 17.66, "ce": 0.13578222421620012, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.43989095091819763, "loss_scale": 4.0, "retries": 0, "seconds": 343.5, "max_memory_gb": 17.66, "ce": 0.02712203138435143, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.49579647183418274, "loss_scale": 4.0, "retries": 0, "seconds": 345.2, "max_memory_gb": 17.66, "ce": 0.05984798472127295, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.266875296831131, "loss_scale": 4.0, "retries": 0, "seconds": 346.8, "max_memory_gb": 17.66, "ce": 0.010397195669611392, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.5997568964958191, "loss_scale": 4.0, "retries": 0, "seconds": 348.5, "max_memory_gb": 17.66, "ce": 0.12195433741726447, "rank": 0.12498421221971512}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 1.717735767364502, "loss_scale": 4.0, "retries": 0, "seconds": 350.2, "max_memory_gb": 17.66, "ce": 0.4577398502142387, "rank": 0.04472362995147705}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.622533917427063, "loss_scale": 4.0, "retries": 0, "seconds": 352.0, "max_memory_gb": 17.66, "ce": 0.1498198062836309, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 1.0189340114593506, "loss_scale": 4.0, "retries": 0, "seconds": 353.6, "max_memory_gb": 17.66, "ce": 0.22417831704296987, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.6404550075531006, "loss_scale": 4.0, "retries": 0, "seconds": 355.3, "max_memory_gb": 17.66, "ce": 0.19482947760843672, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 1.4570974111557007, "loss_scale": 4.0, "retries": 0, "seconds": 356.9, "max_memory_gb": 17.66, "ce": 0.2687730121779168, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 2.155031442642212, "loss_scale": 4.0, "retries": 0, "seconds": 358.6, "max_memory_gb": 17.66, "ce": 0.5255648316669976, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.4418899416923523, "loss_scale": 4.0, "retries": 0, "seconds": 360.3, "max_memory_gb": 17.66, "ce": 0.02687826453257003, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.22704139351844788, "loss_scale": 4.0, "retries": 0, "seconds": 362.0, "max_memory_gb": 17.66, "ce": 0.006642885786277475, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 0.8362876176834106, "loss_scale": 4.0, "retries": 0, "seconds": 363.6, "max_memory_gb": 17.66, "ce": 0.18192309133155504, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.241973415017128, "loss_scale": 4.0, "retries": 0, "seconds": 365.2, "max_memory_gb": 17.66, "ce": 0.0292405947370753, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.7459161877632141, "loss_scale": 4.0, "retries": 0, "seconds": 366.9, "max_memory_gb": 17.66, "ce": 0.0655706167626704, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.5205881595611572, "loss_scale": 4.0, "retries": 0, "seconds": 368.5, "max_memory_gb": 17.66, "ce": 0.07551232711557532, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.5471490025520325, "loss_scale": 4.0, "retries": 0, "seconds": 370.2, "max_memory_gb": 17.66, "ce": 0.0446941941154364, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 2.571727752685547, "loss_scale": 4.0, "retries": 0, "seconds": 371.9, "max_memory_gb": 17.66, "ce": 0.1158635839674389, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.5518234372138977, "loss_scale": 4.0, "retries": 0, "seconds": 373.5, "max_memory_gb": 17.66, "ce": 0.04990617829025723, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.3266841173171997, "loss_scale": 4.0, "retries": 0, "seconds": 375.1, "max_memory_gb": 17.66, "ce": 0.030862281601002906, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.7777577042579651, "loss_scale": 4.0, "retries": 0, "seconds": 376.8, "max_memory_gb": 17.66, "ce": 0.05128454511032032, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.4303368330001831, "loss_scale": 4.0, "retries": 0, "seconds": 378.4, "max_memory_gb": 17.66, "ce": 0.03291999828797998, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.2832413613796234, "loss_scale": 4.0, "retries": 0, "seconds": 380.0, "max_memory_gb": 17.66, "ce": 0.02092448013900139, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.6307923197746277, "loss_scale": 4.0, "retries": 0, "seconds": 381.7, "max_memory_gb": 17.66, "ce": 0.05671387243455683, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 2.4850783348083496, "loss_scale": 4.0, "retries": 0, "seconds": 383.3, "max_memory_gb": 17.66, "ce": 0.4516256202769, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.3671855330467224, "loss_scale": 4.0, "retries": 0, "seconds": 384.9, "max_memory_gb": 17.66, "ce": 0.040106864043991663, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.6332973837852478, "loss_scale": 4.0, "retries": 0, "seconds": 386.6, "max_memory_gb": 17.66, "ce": 0.03616780724405544, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.5271810293197632, "loss_scale": 4.0, "retries": 0, "seconds": 388.3, "max_memory_gb": 17.66, "ce": 0.037678749482438434, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.3912912905216217, "loss_scale": 4.0, "retries": 0, "seconds": 389.9, "max_memory_gb": 17.66, "ce": 0.025819283040618757, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 1.8639312982559204, "loss_scale": 4.0, "retries": 0, "seconds": 391.6, "max_memory_gb": 17.66, "ce": 0.28084941432462074, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.16203975677490234, "loss_scale": 4.0, "retries": 0, "seconds": 393.2, "max_memory_gb": 17.66, "ce": 0.014540371679686359, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.3585990369319916, "loss_scale": 4.0, "retries": 0, "seconds": 394.8, "max_memory_gb": 17.66, "ce": 0.050916834094095975, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.28667864203453064, "loss_scale": 4.0, "retries": 0, "seconds": 396.5, "max_memory_gb": 17.66, "ce": 0.02488683997762564, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.43557316064834595, "loss_scale": 4.0, "retries": 0, "seconds": 398.1, "max_memory_gb": 17.66, "ce": 0.11085074035509024, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 1.4120827913284302, "loss_scale": 4.0, "retries": 0, "seconds": 399.7, "max_memory_gb": 17.66, "ce": 0.11297066865154193, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 1.3704029321670532, "loss_scale": 4.0, "retries": 0, "seconds": 401.4, "max_memory_gb": 17.66, "ce": 0.19206664834382536, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.13208764791488647, "loss_scale": 4.0, "retries": 0, "seconds": 403.1, "max_memory_gb": 17.66, "ce": 0.01153864123261883, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.14857076108455658, "loss_scale": 4.0, "retries": 0, "seconds": 404.7, "max_memory_gb": 17.66, "ce": 0.006603846246434841, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.2618470788002014, "loss_scale": 4.0, "retries": 0, "seconds": 406.4, "max_memory_gb": 17.66, "ce": 0.02957349539065035, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 1.1723034381866455, "loss_scale": 4.0, "retries": 0, "seconds": 408.1, "max_memory_gb": 17.66, "ce": 0.3245813661196735, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.0501294881105423, "loss_scale": 4.0, "retries": 0, "seconds": 409.8, "max_memory_gb": 17.66, "ce": 0.0036395347142388346, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 1.5242877006530762, "loss_scale": 4.0, "retries": 0, "seconds": 411.4, "max_memory_gb": 17.66, "ce": 0.11592487756252012, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.3850555419921875, "loss_scale": 4.0, "retries": 0, "seconds": 413.0, "max_memory_gb": 17.66, "ce": 0.02236746377820964, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.32157886028289795, "loss_scale": 4.0, "retries": 0, "seconds": 414.6, "max_memory_gb": 17.66, "ce": 0.01810374978231266, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.20994672179222107, "loss_scale": 4.0, "retries": 0, "seconds": 416.2, "max_memory_gb": 17.66, "ce": 0.01836231892957585, "rank": 0.0}

runs/single_internvl/removal/L25_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/single_internvl/removal/L25_s13/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.25.self_attn.k_proj",
+    "model.language_model.layers.25.self_attn.o_proj",
+    "model.language_model.layers.25.self_attn.v_proj",
+    "model.language_model.layers.25.mlp.gate_proj",
+    "model.language_model.layers.25.mlp.up_proj",
+    "model.language_model.layers.25.self_attn.q_proj",
+    "model.language_model.layers.25.mlp.down_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/single_internvl/removal/L25_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 25,
+    "rank": 16,
+    "trainable_parameters": 1212416
+  }
+}

runs/single_internvl/removal/L25_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 0.5093048214912415, "loss_scale": 4.0, "retries": 0, "seconds": 2.5, "max_memory_gb": 17.12, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 0.48893874883651733, "loss_scale": 4.0, "retries": 0, "seconds": 4.1, "max_memory_gb": 17.13, "ce": 0.7801791587589832, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 1.8283525705337524, "loss_scale": 4.0, "retries": 0, "seconds": 5.8, "max_memory_gb": 17.13, "ce": 1.380540213547647, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 1.0929348468780518, "loss_scale": 4.0, "retries": 0, "seconds": 7.4, "max_memory_gb": 17.13, "ce": 0.7527540817845875, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.017614008858799934, "loss_scale": 4.0, "retries": 0, "seconds": 9.2, "max_memory_gb": 17.13, "ce": 0.04009930468237144, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 0.5331562161445618, "loss_scale": 4.0, "retries": 0, "seconds": 10.8, "max_memory_gb": 17.13, "ce": 0.43495201927726157, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 1.6290277242660522, "loss_scale": 4.0, "retries": 0, "seconds": 12.4, "max_memory_gb": 17.13, "ce": 1.0216896246856777, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 1.727509617805481, "loss_scale": 4.0, "retries": 0, "seconds": 14.0, "max_memory_gb": 17.13, "ce": 0.9936003750190139, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 1.4059903621673584, "loss_scale": 4.0, "retries": 0, "seconds": 15.7, "max_memory_gb": 17.13, "ce": 0.7079064422287047, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.23165477812290192, "loss_scale": 4.0, "retries": 0, "seconds": 17.2, "max_memory_gb": 17.13, "ce": 0.1903480777546065, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 1.6322407722473145, "loss_scale": 4.0, "retries": 0, "seconds": 18.9, "max_memory_gb": 17.13, "ce": 0.8163453773522633, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 1.0320292711257935, "loss_scale": 4.0, "retries": 0, "seconds": 20.5, "max_memory_gb": 17.13, "ce": 0.5459871031416696, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 0.996042788028717, "loss_scale": 4.0, "retries": 0, "seconds": 22.1, "max_memory_gb": 17.13, "ce": 0.6262200372002553, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 2.089630365371704, "loss_scale": 4.0, "retries": 0, "seconds": 23.7, "max_memory_gb": 17.13, "ce": 0.9628381291404366, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 1.5986205339431763, "loss_scale": 4.0, "retries": 0, "seconds": 25.3, "max_memory_gb": 17.13, "ce": 0.6377901805417423, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.7513816952705383, "loss_scale": 4.0, "retries": 0, "seconds": 26.9, "max_memory_gb": 17.13, "ce": 0.2883294641651446, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.8564352989196777, "loss_scale": 4.0, "retries": 0, "seconds": 28.6, "max_memory_gb": 17.13, "ce": 0.5385892925442022, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 2.455453395843506, "loss_scale": 4.0, "retries": 0, "seconds": 30.1, "max_memory_gb": 17.14, "ce": 0.6495776371375541, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 3.6437299251556396, "loss_scale": 4.0, "retries": 0, "seconds": 31.7, "max_memory_gb": 17.14, "ce": 0.8541146731004119, "rank": 0.00640338659286499}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.01909940503537655, "loss_scale": 4.0, "retries": 0, "seconds": 33.3, "max_memory_gb": 17.14, "ce": 0.04057951290997153, "rank": 0.12125281244516373}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 1.7889115810394287, "loss_scale": 4.0, "retries": 0, "seconds": 35.0, "max_memory_gb": 17.14, "ce": 0.4485215785598484, "rank": 0.0}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 1.8714591264724731, "loss_scale": 4.0, "retries": 0, "seconds": 36.6, "max_memory_gb": 17.14, "ce": 0.5461991321499227, "rank": 0.2559953033924103}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.5721774101257324, "loss_scale": 4.0, "retries": 0, "seconds": 38.3, "max_memory_gb": 17.14, "ce": 0.643157396018978, "rank": 0.01669713854789734}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 1.3930286169052124, "loss_scale": 4.0, "retries": 0, "seconds": 39.9, "max_memory_gb": 17.14, "ce": 0.5901082400232553, "rank": 0.023283690214157104}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 1.6391782760620117, "loss_scale": 4.0, "retries": 0, "seconds": 41.6, "max_memory_gb": 17.14, "ce": 0.696748123235011, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 0.9351484179496765, "loss_scale": 4.0, "retries": 0, "seconds": 43.3, "max_memory_gb": 17.14, "ce": 0.5794223677366972, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 1.1237982511520386, "loss_scale": 4.0, "retries": 0, "seconds": 44.9, "max_memory_gb": 17.14, "ce": 0.5119812030534376, "rank": 0.06654846668243408}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 0.5748503804206848, "loss_scale": 4.0, "retries": 0, "seconds": 46.6, "max_memory_gb": 17.14, "ce": 0.7736353462096304, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 0.7373248338699341, "loss_scale": 4.0, "retries": 0, "seconds": 48.3, "max_memory_gb": 17.14, "ce": 0.41295859427191317, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 0.7257634997367859, "loss_scale": 4.0, "retries": 0, "seconds": 49.9, "max_memory_gb": 17.14, "ce": 0.27729853102482593, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 0.5368363261222839, "loss_scale": 4.0, "retries": 0, "seconds": 51.6, "max_memory_gb": 17.14, "ce": 0.48744101054035127, "rank": 0.08983643352985382}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 0.49356886744499207, "loss_scale": 4.0, "retries": 0, "seconds": 53.2, "max_memory_gb": 17.14, "ce": 0.43067092390265316, "rank": 0.05532018840312958}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.29975685477256775, "loss_scale": 4.0, "retries": 0, "seconds": 54.9, "max_memory_gb": 17.14, "ce": 0.26532903651241213, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.3388766050338745, "loss_scale": 4.0, "retries": 0, "seconds": 56.6, "max_memory_gb": 17.14, "ce": 0.3122835132635373, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.23143425583839417, "loss_scale": 4.0, "retries": 0, "seconds": 58.2, "max_memory_gb": 17.14, "ce": 0.20260447019609273, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 0.5527716279029846, "loss_scale": 4.0, "retries": 0, "seconds": 59.9, "max_memory_gb": 17.14, "ce": 0.3084577375630033, "rank": 0.005424462258815765}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.4821767807006836, "loss_scale": 4.0, "retries": 0, "seconds": 61.5, "max_memory_gb": 17.14, "ce": 0.5091051678464282, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.4898974299430847, "loss_scale": 4.0, "retries": 0, "seconds": 63.2, "max_memory_gb": 17.14, "ce": 0.5316795235266909, "rank": 0.021342717111110687}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 0.4654505252838135, "loss_scale": 4.0, "retries": 0, "seconds": 64.9, "max_memory_gb": 17.14, "ce": 0.3782780107721919, "rank": 0.12497427314519882}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.31638333201408386, "loss_scale": 4.0, "retries": 0, "seconds": 66.5, "max_memory_gb": 17.14, "ce": 0.2261000436010363, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.48586001992225647, "loss_scale": 4.0, "retries": 0, "seconds": 68.1, "max_memory_gb": 17.14, "ce": 0.43178676487877965, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.22070345282554626, "loss_scale": 4.0, "retries": 0, "seconds": 69.8, "max_memory_gb": 17.14, "ce": 0.19142900449878653, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.28740057349205017, "loss_scale": 4.0, "retries": 0, "seconds": 71.4, "max_memory_gb": 17.14, "ce": 0.20252310805153684, "rank": 0.20697609335184097}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 0.9975202083587646, "loss_scale": 4.0, "retries": 0, "seconds": 73.0, "max_memory_gb": 17.14, "ce": 0.6221129624136665, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 0.7327554821968079, "loss_scale": 4.0, "retries": 0, "seconds": 74.6, "max_memory_gb": 17.14, "ce": 0.7054568759922404, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.4682195484638214, "loss_scale": 4.0, "retries": 0, "seconds": 76.2, "max_memory_gb": 17.14, "ce": 0.3156977033795556, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.23777690529823303, "loss_scale": 4.0, "retries": 0, "seconds": 77.9, "max_memory_gb": 17.14, "ce": 0.18637845997000113, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.36243513226509094, "loss_scale": 4.0, "retries": 0, "seconds": 79.5, "max_memory_gb": 17.14, "ce": 0.28320429255836643, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.4637232720851898, "loss_scale": 4.0, "retries": 0, "seconds": 81.1, "max_memory_gb": 17.14, "ce": 0.3070733352979005, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.8708722591400146, "loss_scale": 4.0, "retries": 0, "seconds": 82.7, "max_memory_gb": 17.14, "ce": 0.3721818518715736, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 1.0908600091934204, "loss_scale": 4.0, "retries": 0, "seconds": 84.3, "max_memory_gb": 17.14, "ce": 0.4810271214482782, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.5526512265205383, "loss_scale": 4.0, "retries": 0, "seconds": 85.9, "max_memory_gb": 17.14, "ce": 0.265915515145025, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 0.8543774485588074, "loss_scale": 4.0, "retries": 0, "seconds": 87.5, "max_memory_gb": 17.14, "ce": 0.44673560024239123, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 0.7281795144081116, "loss_scale": 4.0, "retries": 0, "seconds": 89.2, "max_memory_gb": 17.14, "ce": 0.36414667944336543, "rank": 0.00784127414226532}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.33128631114959717, "loss_scale": 4.0, "retries": 0, "seconds": 90.7, "max_memory_gb": 17.14, "ce": 0.16002728189050686, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.6701024174690247, "loss_scale": 4.0, "retries": 0, "seconds": 92.3, "max_memory_gb": 17.14, "ce": 0.3409681729390286, "rank": 0.07841365039348602}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.27575749158859253, "loss_scale": 4.0, "retries": 0, "seconds": 93.8, "max_memory_gb": 17.14, "ce": 0.12738287101819878, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.44445040822029114, "loss_scale": 4.0, "retries": 0, "seconds": 95.5, "max_memory_gb": 17.14, "ce": 0.1613052964785311, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.3066294491291046, "loss_scale": 4.0, "retries": 0, "seconds": 97.1, "max_memory_gb": 17.14, "ce": 0.1151160038280068, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.5856269001960754, "loss_scale": 4.0, "retries": 0, "seconds": 98.7, "max_memory_gb": 17.14, "ce": 0.18927471957431408, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.8536425828933716, "loss_scale": 4.0, "retries": 0, "seconds": 100.3, "max_memory_gb": 17.14, "ce": 0.42885590871446766, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.24898692965507507, "loss_scale": 4.0, "retries": 0, "seconds": 101.9, "max_memory_gb": 17.14, "ce": 0.11503240103047574, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.9842475652694702, "loss_scale": 4.0, "retries": 0, "seconds": 103.3, "max_memory_gb": 17.14, "ce": 0.18508459243457764, "rank": 0.02233315259218216}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.7160115838050842, "loss_scale": 4.0, "retries": 0, "seconds": 104.9, "max_memory_gb": 17.14, "ce": 0.17292864579940215, "rank": 0.07352680712938309}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 1.08479642868042, "loss_scale": 4.0, "retries": 0, "seconds": 106.4, "max_memory_gb": 17.14, "ce": 0.32273681834340096, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.7807782888412476, "loss_scale": 4.0, "retries": 0, "seconds": 108.1, "max_memory_gb": 17.14, "ce": 0.22143417736231186, "rank": 0.12444048374891281}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.6602402925491333, "loss_scale": 4.0, "retries": 0, "seconds": 109.7, "max_memory_gb": 17.14, "ce": 0.09755972804850899, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.3496321141719818, "loss_scale": 4.0, "retries": 0, "seconds": 111.5, "max_memory_gb": 17.14, "ce": 0.06306835335453798, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.8121251463890076, "loss_scale": 4.0, "retries": 0, "seconds": 113.1, "max_memory_gb": 17.14, "ce": 0.43791760932072066, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.4429810643196106, "loss_scale": 4.0, "retries": 0, "seconds": 114.8, "max_memory_gb": 17.14, "ce": 0.06301698835159186, "rank": 0.09210634231567383}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.5756657719612122, "loss_scale": 4.0, "retries": 0, "seconds": 116.4, "max_memory_gb": 17.14, "ce": 0.11495376245784428, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.7344136834144592, "loss_scale": 4.0, "retries": 0, "seconds": 117.8, "max_memory_gb": 17.14, "ce": 0.19389968406176195, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 0.6331436038017273, "loss_scale": 4.0, "retries": 0, "seconds": 119.4, "max_memory_gb": 17.14, "ce": 0.22897161472701555, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.33896365761756897, "loss_scale": 4.0, "retries": 0, "seconds": 121.0, "max_memory_gb": 17.14, "ce": 0.12040424482347589, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.7783538699150085, "loss_scale": 4.0, "retries": 0, "seconds": 122.6, "max_memory_gb": 17.14, "ce": 0.6939692266751081, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 0.863715410232544, "loss_scale": 4.0, "retries": 0, "seconds": 124.2, "max_memory_gb": 17.14, "ce": 0.2995559969858732, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.3518388271331787, "loss_scale": 4.0, "retries": 0, "seconds": 125.8, "max_memory_gb": 17.14, "ce": 0.07049826478032628, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.338763028383255, "loss_scale": 4.0, "retries": 0, "seconds": 127.4, "max_memory_gb": 17.14, "ce": 0.07583928032181575, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.14299508929252625, "loss_scale": 4.0, "retries": 0, "seconds": 129.0, "max_memory_gb": 17.14, "ce": 0.023733150737371034, "rank": 0.08015711605548859}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.31427469849586487, "loss_scale": 4.0, "retries": 0, "seconds": 130.7, "max_memory_gb": 17.14, "ce": 0.06027395027467719, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.38502728939056396, "loss_scale": 4.0, "retries": 0, "seconds": 132.3, "max_memory_gb": 17.14, "ce": 0.07650210673455149, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.30823832750320435, "loss_scale": 4.0, "retries": 0, "seconds": 133.8, "max_memory_gb": 17.14, "ce": 0.07125557840481633, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 1.429054856300354, "loss_scale": 4.0, "retries": 0, "seconds": 135.5, "max_memory_gb": 17.14, "ce": 0.26595151011497364, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.33114802837371826, "loss_scale": 4.0, "retries": 0, "seconds": 137.2, "max_memory_gb": 17.14, "ce": 0.030228068292672106, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.15367008745670319, "loss_scale": 4.0, "retries": 0, "seconds": 138.8, "max_memory_gb": 17.14, "ce": 0.014001330517658062, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.6320410966873169, "loss_scale": 4.0, "retries": 0, "seconds": 140.5, "max_memory_gb": 17.14, "ce": 0.1179796407232061, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.6120229363441467, "loss_scale": 4.0, "retries": 0, "seconds": 142.1, "max_memory_gb": 17.14, "ce": 0.1195291001786245, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.20956766605377197, "loss_scale": 4.0, "retries": 0, "seconds": 143.7, "max_memory_gb": 17.14, "ce": 0.05183407678077856, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.09354819357395172, "loss_scale": 4.0, "retries": 0, "seconds": 145.3, "max_memory_gb": 17.14, "ce": 0.023736188791644963, "rank": 0.12499704957008362}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.4452427625656128, "loss_scale": 4.0, "retries": 0, "seconds": 147.0, "max_memory_gb": 17.14, "ce": 0.09426305530359969, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 1.4949671030044556, "loss_scale": 4.0, "retries": 0, "seconds": 148.6, "max_memory_gb": 17.14, "ce": 0.5629148583939241, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 1.0951229333877563, "loss_scale": 4.0, "retries": 0, "seconds": 150.3, "max_memory_gb": 17.14, "ce": 0.19525935545334505, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 0.49353480339050293, "loss_scale": 4.0, "retries": 0, "seconds": 151.9, "max_memory_gb": 17.14, "ce": 0.11580703826621175, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 0.6759054660797119, "loss_scale": 4.0, "retries": 0, "seconds": 153.5, "max_memory_gb": 17.14, "ce": 0.0847241755691357, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 1.6312203407287598, "loss_scale": 4.0, "retries": 0, "seconds": 155.1, "max_memory_gb": 17.14, "ce": 0.4832953649456613, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 1.7238249778747559, "loss_scale": 4.0, "retries": 0, "seconds": 156.8, "max_memory_gb": 17.14, "ce": 0.47329270595218986, "rank": 0.32817769050598145}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.2582746148109436, "loss_scale": 4.0, "retries": 0, "seconds": 158.4, "max_memory_gb": 17.14, "ce": 0.029665737820323557, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.7474683523178101, "loss_scale": 4.0, "retries": 0, "seconds": 160.1, "max_memory_gb": 17.14, "ce": 0.07034682036101003, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.06825726479291916, "loss_scale": 4.0, "retries": 0, "seconds": 161.7, "max_memory_gb": 17.14, "ce": 0.01187226616730186, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.8783787488937378, "loss_scale": 4.0, "retries": 0, "seconds": 163.3, "max_memory_gb": 17.14, "ce": 0.12954897893359885, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.14048409461975098, "loss_scale": 4.0, "retries": 0, "seconds": 165.0, "max_memory_gb": 17.14, "ce": 0.024127373326336965, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 1.545376181602478, "loss_scale": 4.0, "retries": 0, "seconds": 166.6, "max_memory_gb": 17.14, "ce": 0.09540872943762224, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.2595088481903076, "loss_scale": 4.0, "retries": 0, "seconds": 168.3, "max_memory_gb": 17.14, "ce": 0.03224377436345094, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.14945702254772186, "loss_scale": 4.0, "retries": 0, "seconds": 169.8, "max_memory_gb": 17.14, "ce": 0.045835901648388244, "rank": 0.12495838105678558}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.4881114363670349, "loss_scale": 4.0, "retries": 0, "seconds": 171.5, "max_memory_gb": 17.14, "ce": 0.11513426215969957, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.5000390410423279, "loss_scale": 4.0, "retries": 0, "seconds": 173.2, "max_memory_gb": 17.14, "ce": 0.03424352110596374, "rank": 0.0}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 1.8647667169570923, "loss_scale": 4.0, "retries": 0, "seconds": 174.8, "max_memory_gb": 17.14, "ce": 0.15914661546958087, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 1.136386752128601, "loss_scale": 4.0, "retries": 0, "seconds": 176.5, "max_memory_gb": 17.14, "ce": 0.11244770317352959, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 0.7472736835479736, "loss_scale": 4.0, "retries": 0, "seconds": 178.1, "max_memory_gb": 17.14, "ce": 0.18606780946720392, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.7310864925384521, "loss_scale": 4.0, "retries": 0, "seconds": 179.7, "max_memory_gb": 17.14, "ce": 0.07468926790170372, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.26545360684394836, "loss_scale": 4.0, "retries": 0, "seconds": 181.4, "max_memory_gb": 17.14, "ce": 0.04850343635553145, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.06667082756757736, "loss_scale": 4.0, "retries": 0, "seconds": 183.0, "max_memory_gb": 17.14, "ce": 0.01183819765719818, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.27967891097068787, "loss_scale": 4.0, "retries": 0, "seconds": 184.6, "max_memory_gb": 17.14, "ce": 0.04622573648521211, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.5338912606239319, "loss_scale": 4.0, "retries": 0, "seconds": 186.3, "max_memory_gb": 17.14, "ce": 0.12245372257893905, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.0826970711350441, "loss_scale": 4.0, "retries": 0, "seconds": 188.0, "max_memory_gb": 17.14, "ce": 0.01203913903736975, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.37203213572502136, "loss_scale": 4.0, "retries": 0, "seconds": 189.6, "max_memory_gb": 17.14, "ce": 0.18169178537300468, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.47474944591522217, "loss_scale": 4.0, "retries": 0, "seconds": 191.1, "max_memory_gb": 17.14, "ce": 0.050598990277649136, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 0.8196220993995667, "loss_scale": 4.0, "retries": 0, "seconds": 192.8, "max_memory_gb": 17.14, "ce": 0.28283477176773886, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.08962790668010712, "loss_scale": 4.0, "retries": 0, "seconds": 194.4, "max_memory_gb": 17.14, "ce": 0.015002972906586365, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 2.04828143119812, "loss_scale": 4.0, "retries": 0, "seconds": 196.1, "max_memory_gb": 17.14, "ce": 0.312420581751212, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.26883891224861145, "loss_scale": 4.0, "retries": 0, "seconds": 197.8, "max_memory_gb": 17.14, "ce": 0.06970833094419504, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.8883512020111084, "loss_scale": 4.0, "retries": 0, "seconds": 199.4, "max_memory_gb": 17.14, "ce": 0.13131089852322475, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.3741244673728943, "loss_scale": 4.0, "retries": 0, "seconds": 201.1, "max_memory_gb": 17.14, "ce": 0.0716183687618468, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.4674322605133057, "loss_scale": 4.0, "retries": 0, "seconds": 202.7, "max_memory_gb": 17.14, "ce": 0.6582428027868445, "rank": 0.17153578996658325}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.38139811158180237, "loss_scale": 4.0, "retries": 0, "seconds": 204.3, "max_memory_gb": 17.14, "ce": 0.16122263770375866, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 1.5358834266662598, "loss_scale": 4.0, "retries": 0, "seconds": 205.8, "max_memory_gb": 17.14, "ce": 0.6155644316459075, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.13041158020496368, "loss_scale": 4.0, "retries": 0, "seconds": 207.5, "max_memory_gb": 17.14, "ce": 0.013364410036956542, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 0.9041499495506287, "loss_scale": 4.0, "retries": 0, "seconds": 209.1, "max_memory_gb": 17.14, "ce": 0.2186673878277361, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 0.6952298283576965, "loss_scale": 4.0, "retries": 0, "seconds": 210.7, "max_memory_gb": 17.14, "ce": 0.07055594722623937, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 0.7879686951637268, "loss_scale": 4.0, "retries": 0, "seconds": 212.4, "max_memory_gb": 17.14, "ce": 0.17974803362449165, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 1.4720031023025513, "loss_scale": 4.0, "retries": 0, "seconds": 214.0, "max_memory_gb": 17.14, "ce": 0.22287689486984164, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 1.3213475942611694, "loss_scale": 4.0, "retries": 0, "seconds": 215.5, "max_memory_gb": 17.14, "ce": 0.14081772439021734, "rank": 0.05351105332374573}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 1.0353087186813354, "loss_scale": 4.0, "retries": 0, "seconds": 217.2, "max_memory_gb": 17.14, "ce": 0.4720107119064778, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.5925025343894958, "loss_scale": 4.0, "retries": 0, "seconds": 218.8, "max_memory_gb": 17.14, "ce": 0.13166102338436758, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 0.47934815287590027, "loss_scale": 4.0, "retries": 0, "seconds": 220.5, "max_memory_gb": 17.14, "ce": 0.045989519043359905, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 0.39405354857444763, "loss_scale": 4.0, "retries": 0, "seconds": 222.0, "max_memory_gb": 17.14, "ce": 0.05059716292453231, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.45419785380363464, "loss_scale": 4.0, "retries": 0, "seconds": 223.6, "max_memory_gb": 17.14, "ce": 0.04133731029287446, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 0.6549954414367676, "loss_scale": 4.0, "retries": 0, "seconds": 225.3, "max_memory_gb": 17.14, "ce": 0.10070137708680704, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.3500255346298218, "loss_scale": 4.0, "retries": 0, "seconds": 227.0, "max_memory_gb": 17.14, "ce": 0.04998101439559832, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.07152301073074341, "loss_scale": 4.0, "retries": 0, "seconds": 228.6, "max_memory_gb": 17.14, "ce": 0.01514693295575853, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.07291282713413239, "loss_scale": 4.0, "retries": 0, "seconds": 230.3, "max_memory_gb": 17.14, "ce": 0.015031847378850216, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.5531790256500244, "loss_scale": 4.0, "retries": 0, "seconds": 231.9, "max_memory_gb": 17.14, "ce": 0.07292110714843147, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 1.675907015800476, "loss_scale": 4.0, "retries": 0, "seconds": 233.5, "max_memory_gb": 17.14, "ce": 0.31328228543998193, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.7586268782615662, "loss_scale": 4.0, "retries": 0, "seconds": 235.1, "max_memory_gb": 17.14, "ce": 0.10062639106763527, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.06744575500488281, "loss_scale": 4.0, "retries": 0, "seconds": 236.6, "max_memory_gb": 17.14, "ce": 0.009535704128211364, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.09027182310819626, "loss_scale": 4.0, "retries": 0, "seconds": 238.2, "max_memory_gb": 17.14, "ce": 0.020582922734320164, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 1.344934105873108, "loss_scale": 4.0, "retries": 0, "seconds": 239.8, "max_memory_gb": 17.14, "ce": 0.589150461149984, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.078172966837883, "loss_scale": 4.0, "retries": 0, "seconds": 241.4, "max_memory_gb": 17.14, "ce": 0.014063496189010039, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.08212494850158691, "loss_scale": 4.0, "retries": 0, "seconds": 243.1, "max_memory_gb": 17.14, "ce": 0.01403610041597858, "rank": 0.12463066726922989}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.23258253931999207, "loss_scale": 4.0, "retries": 0, "seconds": 244.7, "max_memory_gb": 17.14, "ce": 0.05885378274251707, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.2564374804496765, "loss_scale": 4.0, "retries": 0, "seconds": 246.3, "max_memory_gb": 17.14, "ce": 0.04114693283190718, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 0.36300674080848694, "loss_scale": 4.0, "retries": 0, "seconds": 248.0, "max_memory_gb": 17.14, "ce": 0.048575629414699506, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.5926106572151184, "loss_scale": 4.0, "retries": 0, "seconds": 249.6, "max_memory_gb": 17.14, "ce": 0.07585554225988744, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.5196222066879272, "loss_scale": 4.0, "retries": 0, "seconds": 251.2, "max_memory_gb": 17.14, "ce": 0.0591285883274395, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 1.3850206136703491, "loss_scale": 4.0, "retries": 0, "seconds": 252.9, "max_memory_gb": 17.14, "ce": 0.1713633144790947, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.09862962365150452, "loss_scale": 4.0, "retries": 0, "seconds": 254.5, "max_memory_gb": 17.14, "ce": 0.009964727802071138, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.09612838178873062, "loss_scale": 4.0, "retries": 0, "seconds": 256.1, "max_memory_gb": 17.14, "ce": 0.013322853341378504, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 2.7482101917266846, "loss_scale": 4.0, "retries": 0, "seconds": 257.7, "max_memory_gb": 17.14, "ce": 0.4219322490234845, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.19295960664749146, "loss_scale": 4.0, "retries": 0, "seconds": 259.3, "max_memory_gb": 17.18, "ce": 0.022705645234964322, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 0.49190065264701843, "loss_scale": 4.0, "retries": 0, "seconds": 260.9, "max_memory_gb": 17.18, "ce": 0.17876426936709322, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 0.8731034994125366, "loss_scale": 4.0, "retries": 0, "seconds": 262.5, "max_memory_gb": 17.18, "ce": 0.11520384213872603, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.4800969958305359, "loss_scale": 4.0, "retries": 0, "seconds": 264.1, "max_memory_gb": 17.18, "ce": 0.0676964482445328, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 1.3299500942230225, "loss_scale": 4.0, "retries": 0, "seconds": 265.7, "max_memory_gb": 17.18, "ce": 0.19195469256374054, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.37147435545921326, "loss_scale": 4.0, "retries": 0, "seconds": 267.3, "max_memory_gb": 17.18, "ce": 0.06421826619043713, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.0800343006849289, "loss_scale": 4.0, "retries": 0, "seconds": 268.9, "max_memory_gb": 17.18, "ce": 0.01598456685860583, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 1.6850254535675049, "loss_scale": 4.0, "retries": 0, "seconds": 270.4, "max_memory_gb": 17.18, "ce": 0.46132621065771673, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.15641917288303375, "loss_scale": 4.0, "retries": 0, "seconds": 272.2, "max_memory_gb": 17.18, "ce": 0.014154985721688718, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.1407299041748047, "loss_scale": 4.0, "retries": 0, "seconds": 274.0, "max_memory_gb": 17.18, "ce": 0.015117820650630165, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 0.8290445804595947, "loss_scale": 4.0, "retries": 0, "seconds": 275.8, "max_memory_gb": 17.18, "ce": 0.3118594526167726, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.05380900204181671, "loss_scale": 4.0, "retries": 0, "seconds": 277.4, "max_memory_gb": 17.18, "ce": 0.00904093151075358, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.35033103823661804, "loss_scale": 4.0, "retries": 0, "seconds": 279.1, "max_memory_gb": 17.18, "ce": 0.0717493464612744, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.18076500296592712, "loss_scale": 4.0, "retries": 0, "seconds": 280.7, "max_memory_gb": 17.18, "ce": 0.02289449617092032, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.2785208523273468, "loss_scale": 4.0, "retries": 0, "seconds": 282.3, "max_memory_gb": 17.18, "ce": 0.02938081868342124, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.5268921256065369, "loss_scale": 4.0, "retries": 0, "seconds": 283.9, "max_memory_gb": 17.18, "ce": 0.04927926862001186, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 1.9024184942245483, "loss_scale": 4.0, "retries": 0, "seconds": 285.5, "max_memory_gb": 17.18, "ce": 0.22871951837441884, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.1648005098104477, "loss_scale": 4.0, "retries": 0, "seconds": 287.2, "max_memory_gb": 17.18, "ce": 0.0126926897210069, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 0.8914054036140442, "loss_scale": 4.0, "retries": 0, "seconds": 288.7, "max_memory_gb": 17.18, "ce": 0.2909737239624519, "rank": 0.12501712143421173}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 0.3651697337627411, "loss_scale": 4.0, "retries": 0, "seconds": 290.4, "max_memory_gb": 17.18, "ce": 0.05638096225447953, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 0.8527162075042725, "loss_scale": 4.0, "retries": 0, "seconds": 292.1, "max_memory_gb": 17.18, "ce": 0.4216868837838774, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.3578925132751465, "loss_scale": 4.0, "retries": 0, "seconds": 293.7, "max_memory_gb": 17.18, "ce": 0.0395329762250185, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.2903139889240265, "loss_scale": 4.0, "retries": 0, "seconds": 295.3, "max_memory_gb": 17.18, "ce": 0.031155018863501027, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 0.6040517091751099, "loss_scale": 4.0, "retries": 0, "seconds": 297.0, "max_memory_gb": 17.18, "ce": 0.09598300782818114, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.14585308730602264, "loss_scale": 4.0, "retries": 0, "seconds": 298.6, "max_memory_gb": 17.18, "ce": 0.016317776050527755, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.1705508679151535, "loss_scale": 4.0, "retries": 0, "seconds": 300.3, "max_memory_gb": 17.18, "ce": 0.023812618395822938, "rank": 0.2093534767627716}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.39497628808021545, "loss_scale": 4.0, "retries": 0, "seconds": 301.9, "max_memory_gb": 17.18, "ce": 0.05594222496438306, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 0.7055532336235046, "loss_scale": 4.0, "retries": 0, "seconds": 303.6, "max_memory_gb": 17.18, "ce": 0.10830623078254575, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.6961323022842407, "loss_scale": 4.0, "retries": 0, "seconds": 305.3, "max_memory_gb": 17.18, "ce": 0.16964194586034864, "rank": 0.14136238396167755}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 0.9553115367889404, "loss_scale": 4.0, "retries": 0, "seconds": 306.9, "max_memory_gb": 17.18, "ce": 0.22321738778873623, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 1.2426018714904785, "loss_scale": 4.0, "retries": 0, "seconds": 308.5, "max_memory_gb": 17.18, "ce": 0.38093589288473595, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.14369934797286987, "loss_scale": 4.0, "retries": 0, "seconds": 310.2, "max_memory_gb": 17.18, "ce": 0.018712390679866076, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 1.1519980430603027, "loss_scale": 4.0, "retries": 0, "seconds": 311.8, "max_memory_gb": 17.18, "ce": 0.24994733770108724, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.0421618223190308, "loss_scale": 4.0, "retries": 0, "seconds": 313.5, "max_memory_gb": 17.18, "ce": 0.5093110977031756, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.07924240082502365, "loss_scale": 4.0, "retries": 0, "seconds": 315.2, "max_memory_gb": 17.18, "ce": 0.00814081794305821, "rank": 0.12500950694084167}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.3641881048679352, "loss_scale": 4.0, "retries": 0, "seconds": 316.9, "max_memory_gb": 17.18, "ce": 0.059478193332324736, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.5649424195289612, "loss_scale": 4.0, "retries": 0, "seconds": 318.6, "max_memory_gb": 17.18, "ce": 0.09424487504293211, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.4506683349609375, "loss_scale": 4.0, "retries": 0, "seconds": 320.3, "max_memory_gb": 17.18, "ce": 0.04458319277364353, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.8082674741744995, "loss_scale": 4.0, "retries": 0, "seconds": 322.0, "max_memory_gb": 17.18, "ce": 0.23547230049734935, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.20384182035923004, "loss_scale": 4.0, "retries": 0, "seconds": 323.5, "max_memory_gb": 17.18, "ce": 0.05932121654768707, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 1.4990991353988647, "loss_scale": 4.0, "retries": 0, "seconds": 325.2, "max_memory_gb": 17.18, "ce": 0.1455873391314526, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 0.7909298539161682, "loss_scale": 4.0, "retries": 0, "seconds": 326.9, "max_memory_gb": 17.18, "ce": 0.3238973770348821, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.22001154720783234, "loss_scale": 4.0, "retries": 0, "seconds": 328.5, "max_memory_gb": 17.18, "ce": 0.054255587889201706, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.06068495288491249, "loss_scale": 4.0, "retries": 0, "seconds": 330.2, "max_memory_gb": 17.18, "ce": 0.009385047455452877, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.22668345272541046, "loss_scale": 4.0, "retries": 0, "seconds": 331.8, "max_memory_gb": 17.18, "ce": 0.016689561554812826, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.3181777596473694, "loss_scale": 4.0, "retries": 0, "seconds": 333.5, "max_memory_gb": 17.18, "ce": 0.04827481645043008, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 0.46009182929992676, "loss_scale": 4.0, "retries": 0, "seconds": 335.1, "max_memory_gb": 17.18, "ce": 0.09823459744984575, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.36932674050331116, "loss_scale": 4.0, "retries": 0, "seconds": 336.7, "max_memory_gb": 17.18, "ce": 0.03991717543976847, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.17669059336185455, "loss_scale": 4.0, "retries": 0, "seconds": 338.4, "max_memory_gb": 17.18, "ce": 0.05008171719236998, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.14204756915569305, "loss_scale": 4.0, "retries": 0, "seconds": 340.1, "max_memory_gb": 17.18, "ce": 0.017392839037711383, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.3316093683242798, "loss_scale": 4.0, "retries": 0, "seconds": 341.8, "max_memory_gb": 17.18, "ce": 0.12097872537560761, "rank": 0.12506282329559326}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 0.9699962735176086, "loss_scale": 4.0, "retries": 0, "seconds": 343.4, "max_memory_gb": 17.18, "ce": 0.44125649097622954, "rank": 0.1639891266822815}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.44116970896720886, "loss_scale": 4.0, "retries": 0, "seconds": 345.0, "max_memory_gb": 17.18, "ce": 0.15935827800421976, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 0.7339628338813782, "loss_scale": 4.0, "retries": 0, "seconds": 346.7, "max_memory_gb": 17.18, "ce": 0.20506934894365259, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.4760630130767822, "loss_scale": 4.0, "retries": 0, "seconds": 348.3, "max_memory_gb": 17.18, "ce": 0.20382428102311678, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 1.4497900009155273, "loss_scale": 4.0, "retries": 0, "seconds": 350.0, "max_memory_gb": 17.18, "ce": 0.2976399518274775, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 1.1750322580337524, "loss_scale": 4.0, "retries": 0, "seconds": 351.6, "max_memory_gb": 17.18, "ce": 0.5346186831593513, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.3819843530654907, "loss_scale": 4.0, "retries": 0, "seconds": 353.3, "max_memory_gb": 17.18, "ce": 0.0283768200497434, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.041628871113061905, "loss_scale": 4.0, "retries": 0, "seconds": 354.8, "max_memory_gb": 17.18, "ce": 0.00791464369467576, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 0.49140554666519165, "loss_scale": 4.0, "retries": 0, "seconds": 356.4, "max_memory_gb": 17.18, "ce": 0.18734398970991606, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.12327788025140762, "loss_scale": 4.0, "retries": 0, "seconds": 358.0, "max_memory_gb": 17.18, "ce": 0.024247869076589268, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 1.0947315692901611, "loss_scale": 4.0, "retries": 0, "seconds": 359.7, "max_memory_gb": 17.18, "ce": 0.067612383638334, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.4167911112308502, "loss_scale": 4.0, "retries": 0, "seconds": 361.3, "max_memory_gb": 17.18, "ce": 0.07137770764529705, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.3924010992050171, "loss_scale": 4.0, "retries": 0, "seconds": 363.0, "max_memory_gb": 17.18, "ce": 0.04616560087015387, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 1.0096362829208374, "loss_scale": 4.0, "retries": 0, "seconds": 364.6, "max_memory_gb": 17.18, "ce": 0.07475570803580922, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.28730565309524536, "loss_scale": 4.0, "retries": 0, "seconds": 366.3, "max_memory_gb": 17.18, "ce": 0.037449140785611235, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.29487285017967224, "loss_scale": 4.0, "retries": 0, "seconds": 367.9, "max_memory_gb": 17.18, "ce": 0.030405257668462582, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.6413666009902954, "loss_scale": 4.0, "retries": 0, "seconds": 369.4, "max_memory_gb": 17.18, "ce": 0.06031445093503862, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.3737027049064636, "loss_scale": 4.0, "retries": 0, "seconds": 371.1, "max_memory_gb": 17.18, "ce": 0.050119689305574866, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.15434135496616364, "loss_scale": 4.0, "retries": 0, "seconds": 372.6, "max_memory_gb": 17.18, "ce": 0.012776979410773492, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.2321428805589676, "loss_scale": 4.0, "retries": 0, "seconds": 374.3, "max_memory_gb": 17.18, "ce": 0.04577340633113636, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 1.9994853734970093, "loss_scale": 4.0, "retries": 0, "seconds": 375.9, "max_memory_gb": 17.18, "ce": 0.5152564582167543, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.25574928522109985, "loss_scale": 4.0, "retries": 0, "seconds": 377.3, "max_memory_gb": 17.18, "ce": 0.03507228994567413, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.7464489936828613, "loss_scale": 4.0, "retries": 0, "seconds": 379.0, "max_memory_gb": 17.18, "ce": 0.06975606494233944, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.2778191864490509, "loss_scale": 4.0, "retries": 0, "seconds": 380.7, "max_memory_gb": 17.18, "ce": 0.032616328389849514, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.23047542572021484, "loss_scale": 4.0, "retries": 0, "seconds": 382.3, "max_memory_gb": 17.18, "ce": 0.024078953507341794, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 1.3554093837738037, "loss_scale": 4.0, "retries": 0, "seconds": 383.9, "max_memory_gb": 17.18, "ce": 0.34036803197159315, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.08991381525993347, "loss_scale": 4.0, "retries": 0, "seconds": 385.5, "max_memory_gb": 17.18, "ce": 0.018200690108642448, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.20112751424312592, "loss_scale": 4.0, "retries": 0, "seconds": 387.1, "max_memory_gb": 17.18, "ce": 0.05507053519249894, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.26558780670166016, "loss_scale": 4.0, "retries": 0, "seconds": 388.8, "max_memory_gb": 17.18, "ce": 0.04168297135765897, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.2490302175283432, "loss_scale": 4.0, "retries": 0, "seconds": 390.4, "max_memory_gb": 17.18, "ce": 0.11036809613142395, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 0.7122349143028259, "loss_scale": 4.0, "retries": 0, "seconds": 392.0, "max_memory_gb": 17.18, "ce": 0.06250772599378251, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.9875214695930481, "loss_scale": 4.0, "retries": 0, "seconds": 393.7, "max_memory_gb": 17.18, "ce": 0.189935178847918, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.08870340138673782, "loss_scale": 4.0, "retries": 0, "seconds": 395.4, "max_memory_gb": 17.18, "ce": 0.010333651334804017, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.11269692331552505, "loss_scale": 4.0, "retries": 0, "seconds": 397.0, "max_memory_gb": 17.18, "ce": 0.011776154118706472, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.23610657453536987, "loss_scale": 4.0, "retries": 0, "seconds": 398.6, "max_memory_gb": 17.18, "ce": 0.03413058280420955, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 0.9579800367355347, "loss_scale": 4.0, "retries": 0, "seconds": 400.2, "max_memory_gb": 17.18, "ce": 0.33368120453087613, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.04135161265730858, "loss_scale": 4.0, "retries": 0, "seconds": 401.9, "max_memory_gb": 17.18, "ce": 0.007909414975074469, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.5853040218353271, "loss_scale": 4.0, "retries": 0, "seconds": 403.5, "max_memory_gb": 17.18, "ce": 0.0788358721356417, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.27320587635040283, "loss_scale": 4.0, "retries": 0, "seconds": 405.2, "max_memory_gb": 17.18, "ce": 0.024910679931053892, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.2518347203731537, "loss_scale": 4.0, "retries": 0, "seconds": 406.8, "max_memory_gb": 17.18, "ce": 0.03195220744237304, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.4324246346950531, "loss_scale": 4.0, "retries": 0, "seconds": 408.4, "max_memory_gb": 17.18, "ce": 0.03418303043872584, "rank": 0.0}

runs/single_internvl/removal/L29_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/single_internvl/removal/L29_s13/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.29.self_attn.k_proj",
+    "model.language_model.layers.29.mlp.up_proj",
+    "model.language_model.layers.29.mlp.down_proj",
+    "model.language_model.layers.29.mlp.gate_proj",
+    "model.language_model.layers.29.self_attn.v_proj",
+    "model.language_model.layers.29.self_attn.q_proj",
+    "model.language_model.layers.29.self_attn.o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/single_internvl/removal/L29_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 29,
+    "rank": 16,
+    "trainable_parameters": 1212416
+  }
+}

runs/single_internvl/removal/L29_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 0.22705738246440887, "loss_scale": 4.0, "retries": 0, "seconds": 2.3, "max_memory_gb": 16.76, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 0.236006498336792, "loss_scale": 4.0, "retries": 0, "seconds": 3.9, "max_memory_gb": 16.77, "ce": 0.7821806777756137, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 0.6656700372695923, "loss_scale": 4.0, "retries": 0, "seconds": 5.5, "max_memory_gb": 16.77, "ce": 1.3806696003302932, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 0.34455645084381104, "loss_scale": 4.0, "retries": 0, "seconds": 7.0, "max_memory_gb": 16.77, "ce": 0.7539795556422177, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.013687686063349247, "loss_scale": 4.0, "retries": 0, "seconds": 8.6, "max_memory_gb": 16.77, "ce": 0.04008739669279748, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 0.23872290551662445, "loss_scale": 4.0, "retries": 0, "seconds": 10.1, "max_memory_gb": 16.77, "ce": 0.437514605706383, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 0.5352051854133606, "loss_scale": 4.0, "retries": 0, "seconds": 11.6, "max_memory_gb": 16.77, "ce": 1.0332397590500477, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 0.5597342252731323, "loss_scale": 4.0, "retries": 0, "seconds": 13.2, "max_memory_gb": 16.77, "ce": 1.010436525568366, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 0.4023561477661133, "loss_scale": 4.0, "retries": 0, "seconds": 14.8, "max_memory_gb": 16.77, "ce": 0.7274706773459911, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.1001758724451065, "loss_scale": 4.0, "retries": 0, "seconds": 16.2, "max_memory_gb": 16.77, "ce": 0.19419024474336766, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.5416147708892822, "loss_scale": 4.0, "retries": 0, "seconds": 17.6, "max_memory_gb": 16.77, "ce": 0.8550187596338219, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 0.40560269355773926, "loss_scale": 4.0, "retries": 0, "seconds": 19.0, "max_memory_gb": 16.77, "ce": 0.5726028580220373, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 0.38884297013282776, "loss_scale": 4.0, "retries": 0, "seconds": 20.4, "max_memory_gb": 16.77, "ce": 0.6536452381551499, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 0.6674638390541077, "loss_scale": 4.0, "retries": 0, "seconds": 21.8, "max_memory_gb": 16.77, "ce": 1.0454076477326453, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 0.5643481612205505, "loss_scale": 4.0, "retries": 0, "seconds": 23.2, "max_memory_gb": 16.77, "ce": 0.70950462927226, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.2898411452770233, "loss_scale": 4.0, "retries": 0, "seconds": 24.6, "max_memory_gb": 16.77, "ce": 0.3206272875977447, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.35908403992652893, "loss_scale": 4.0, "retries": 0, "seconds": 26.0, "max_memory_gb": 16.77, "ce": 0.5874775775264425, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 0.7934726476669312, "loss_scale": 4.0, "retries": 0, "seconds": 27.4, "max_memory_gb": 16.78, "ce": 0.8299381765464204, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 1.589768409729004, "loss_scale": 4.0, "retries": 0, "seconds": 28.8, "max_memory_gb": 16.78, "ce": 1.1620771856978536, "rank": 0.0}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.013939062133431435, "loss_scale": 4.0, "retries": 0, "seconds": 30.2, "max_memory_gb": 16.78, "ce": 0.040502182571799494, "rank": 0.12113948166370392}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.8916688561439514, "loss_scale": 4.0, "retries": 0, "seconds": 31.6, "max_memory_gb": 16.78, "ce": 0.6256251474760575, "rank": 0.0}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.9464405179023743, "loss_scale": 4.0, "retries": 0, "seconds": 33.0, "max_memory_gb": 16.78, "ce": 0.7591371633534436, "rank": 0.3904182016849518}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.1513746976852417, "loss_scale": 4.0, "retries": 0, "seconds": 34.5, "max_memory_gb": 16.78, "ce": 0.8355006427113949, "rank": 0.012885302305221558}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 1.1083571910858154, "loss_scale": 4.0, "retries": 0, "seconds": 35.9, "max_memory_gb": 16.78, "ce": 0.833688591606915, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 1.1892523765563965, "loss_scale": 4.0, "retries": 0, "seconds": 37.3, "max_memory_gb": 16.78, "ce": 0.943802831134235, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 1.0215345621109009, "loss_scale": 4.0, "retries": 0, "seconds": 38.7, "max_memory_gb": 16.78, "ce": 0.80153713747859, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 1.1514588594436646, "loss_scale": 4.0, "retries": 0, "seconds": 40.2, "max_memory_gb": 16.78, "ce": 0.8037481481369468, "rank": 0.0}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 0.7106178998947144, "loss_scale": 4.0, "retries": 0, "seconds": 41.6, "max_memory_gb": 16.78, "ce": 0.8821439475286752, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 1.1417933702468872, "loss_scale": 4.0, "retries": 0, "seconds": 43.0, "max_memory_gb": 16.78, "ce": 0.6595248642843217, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 1.1656166315078735, "loss_scale": 4.0, "retries": 0, "seconds": 44.4, "max_memory_gb": 16.78, "ce": 0.521461010164785, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 1.577561855316162, "loss_scale": 4.0, "retries": 0, "seconds": 45.8, "max_memory_gb": 16.78, "ce": 0.8359195662196726, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 1.8999277353286743, "loss_scale": 4.0, "retries": 0, "seconds": 47.2, "max_memory_gb": 16.78, "ce": 0.7673381874337792, "rank": 0.006217598915100098}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.7508261203765869, "loss_scale": 4.0, "retries": 0, "seconds": 48.6, "max_memory_gb": 16.78, "ce": 0.3974959089537151, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.7769289612770081, "loss_scale": 4.0, "retries": 0, "seconds": 50.0, "max_memory_gb": 16.78, "ce": 0.38942154356482206, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 1.0302950143814087, "loss_scale": 4.0, "retries": 0, "seconds": 51.6, "max_memory_gb": 16.78, "ce": 0.3665289812015544, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 0.9594993591308594, "loss_scale": 4.0, "retries": 0, "seconds": 53.1, "max_memory_gb": 16.78, "ce": 0.4404231714870548, "rank": 0.0}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.8421458005905151, "loss_scale": 4.0, "retries": 0, "seconds": 54.7, "max_memory_gb": 16.78, "ce": 0.6368268513178919, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.8419681191444397, "loss_scale": 4.0, "retries": 0, "seconds": 56.2, "max_memory_gb": 16.78, "ce": 0.6407836675643921, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 0.3420964777469635, "loss_scale": 4.0, "retries": 0, "seconds": 57.7, "max_memory_gb": 16.78, "ce": 0.40636611613444984, "rank": 0.12494529783725739}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.46444034576416016, "loss_scale": 4.0, "retries": 0, "seconds": 59.3, "max_memory_gb": 16.78, "ce": 0.25684368812289904, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.4098750650882721, "loss_scale": 4.0, "retries": 0, "seconds": 60.8, "max_memory_gb": 16.78, "ce": 0.45738637493923306, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.5881945490837097, "loss_scale": 4.0, "retries": 0, "seconds": 62.3, "max_memory_gb": 16.78, "ce": 0.2475081694756227, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 1.0107624530792236, "loss_scale": 4.0, "retries": 0, "seconds": 63.8, "max_memory_gb": 16.78, "ce": 0.30814024476057966, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 0.7820872068405151, "loss_scale": 4.0, "retries": 0, "seconds": 65.4, "max_memory_gb": 16.78, "ce": 0.6754278236148821, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 0.7916213274002075, "loss_scale": 4.0, "retries": 0, "seconds": 66.9, "max_memory_gb": 16.78, "ce": 0.7698280399490613, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.3931085467338562, "loss_scale": 4.0, "retries": 0, "seconds": 68.5, "max_memory_gb": 16.78, "ce": 0.35079978103749454, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.2932136058807373, "loss_scale": 4.0, "retries": 0, "seconds": 70.0, "max_memory_gb": 16.78, "ce": 0.2104867538291728, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.5347543954849243, "loss_scale": 4.0, "retries": 0, "seconds": 71.6, "max_memory_gb": 16.78, "ce": 0.335369118838571, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.6110615730285645, "loss_scale": 4.0, "retries": 0, "seconds": 73.1, "max_memory_gb": 16.78, "ce": 0.3657712705389713, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 1.1134920120239258, "loss_scale": 4.0, "retries": 0, "seconds": 74.7, "max_memory_gb": 16.78, "ce": 0.4341747780199512, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 0.7041658163070679, "loss_scale": 4.0, "retries": 0, "seconds": 76.2, "max_memory_gb": 16.78, "ce": 0.5292577890340908, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.6067470908164978, "loss_scale": 4.0, "retries": 0, "seconds": 77.8, "max_memory_gb": 16.78, "ce": 0.3119657028692018, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 0.7601370215415955, "loss_scale": 4.0, "retries": 0, "seconds": 79.3, "max_memory_gb": 16.78, "ce": 0.5107251186855137, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 0.6146304607391357, "loss_scale": 4.0, "retries": 0, "seconds": 80.8, "max_memory_gb": 16.78, "ce": 0.40525333647383377, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.27307233214378357, "loss_scale": 4.0, "retries": 0, "seconds": 82.3, "max_memory_gb": 16.78, "ce": 0.17779268525191583, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.5746673941612244, "loss_scale": 4.0, "retries": 0, "seconds": 83.9, "max_memory_gb": 16.78, "ce": 0.39595114334952086, "rank": 0.07545797526836395}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.2810024619102478, "loss_scale": 4.0, "retries": 0, "seconds": 85.4, "max_memory_gb": 16.78, "ce": 0.153979104510654, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.33914223313331604, "loss_scale": 4.0, "retries": 0, "seconds": 86.9, "max_memory_gb": 16.78, "ce": 0.19971510331743048, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.20423510670661926, "loss_scale": 4.0, "retries": 0, "seconds": 88.5, "max_memory_gb": 16.78, "ce": 0.132330898908549, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.3986726701259613, "loss_scale": 4.0, "retries": 0, "seconds": 90.0, "max_memory_gb": 16.78, "ce": 0.21067124926048564, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.6201410889625549, "loss_scale": 4.0, "retries": 0, "seconds": 91.6, "max_memory_gb": 16.78, "ce": 0.5008693456184119, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.1958378702402115, "loss_scale": 4.0, "retries": 0, "seconds": 93.1, "max_memory_gb": 16.78, "ce": 0.13692936630832264, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.5133958458900452, "loss_scale": 4.0, "retries": 0, "seconds": 94.7, "max_memory_gb": 16.78, "ce": 0.26185498072300106, "rank": 0.05511343479156494}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.5047652721405029, "loss_scale": 4.0, "retries": 0, "seconds": 96.2, "max_memory_gb": 16.78, "ce": 0.24412793084047735, "rank": 0.11091773211956024}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 0.9036202430725098, "loss_scale": 4.0, "retries": 0, "seconds": 97.8, "max_memory_gb": 16.78, "ce": 0.46657588332891464, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.6801196932792664, "loss_scale": 4.0, "retries": 0, "seconds": 99.3, "max_memory_gb": 16.78, "ce": 0.2417528109810405, "rank": 0.1243181973695755}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.6029787063598633, "loss_scale": 4.0, "retries": 0, "seconds": 100.9, "max_memory_gb": 16.78, "ce": 0.21849633925012313, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.23149645328521729, "loss_scale": 4.0, "retries": 0, "seconds": 102.5, "max_memory_gb": 16.78, "ce": 0.11258778519550106, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.7201064825057983, "loss_scale": 4.0, "retries": 0, "seconds": 104.0, "max_memory_gb": 16.78, "ce": 0.48617693432606757, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.4254215657711029, "loss_scale": 4.0, "retries": 0, "seconds": 105.6, "max_memory_gb": 16.78, "ce": 0.14509447016462218, "rank": 0.12173637747764587}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.6924136877059937, "loss_scale": 4.0, "retries": 0, "seconds": 107.1, "max_memory_gb": 16.78, "ce": 0.21815362907091185, "rank": 0.0011039525270462036}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.8962143063545227, "loss_scale": 4.0, "retries": 0, "seconds": 108.7, "max_memory_gb": 16.78, "ce": 0.328937525046058, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 0.49780207872390747, "loss_scale": 4.0, "retries": 0, "seconds": 110.2, "max_memory_gb": 16.78, "ce": 0.2796077045659331, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.508784294128418, "loss_scale": 4.0, "retries": 0, "seconds": 111.7, "max_memory_gb": 16.78, "ce": 0.1695320031412848, "rank": 0.08022058010101318}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.5351917743682861, "loss_scale": 4.0, "retries": 0, "seconds": 113.3, "max_memory_gb": 16.78, "ce": 0.7048488526488654, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 0.7402247786521912, "loss_scale": 4.0, "retries": 0, "seconds": 114.8, "max_memory_gb": 16.78, "ce": 0.3402528825827176, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.8109874129295349, "loss_scale": 4.0, "retries": 0, "seconds": 116.3, "max_memory_gb": 16.78, "ce": 0.17144758954964345, "rank": 0.01830047369003296}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.4632997512817383, "loss_scale": 4.0, "retries": 0, "seconds": 117.8, "max_memory_gb": 16.78, "ce": 0.1506055631034542, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.2140119969844818, "loss_scale": 4.0, "retries": 0, "seconds": 119.4, "max_memory_gb": 16.78, "ce": 0.05637026153544866, "rank": 0.0977579653263092}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.4283723533153534, "loss_scale": 4.0, "retries": 0, "seconds": 120.9, "max_memory_gb": 16.78, "ce": 0.12843829189841927, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.676800012588501, "loss_scale": 4.0, "retries": 0, "seconds": 122.4, "max_memory_gb": 16.78, "ce": 0.16202930366853252, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.5954376459121704, "loss_scale": 4.0, "retries": 0, "seconds": 124.0, "max_memory_gb": 16.78, "ce": 0.13645151021773927, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 1.4295306205749512, "loss_scale": 4.0, "retries": 0, "seconds": 125.5, "max_memory_gb": 16.78, "ce": 0.380974249146675, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.42095378041267395, "loss_scale": 4.0, "retries": 0, "seconds": 127.1, "max_memory_gb": 16.78, "ce": 0.04909477908950066, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.34884271025657654, "loss_scale": 4.0, "retries": 0, "seconds": 128.6, "max_memory_gb": 16.78, "ce": 0.03817076311452183, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.611648678779602, "loss_scale": 4.0, "retries": 0, "seconds": 130.2, "max_memory_gb": 16.78, "ce": 0.1798112477408722, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.7984947562217712, "loss_scale": 4.0, "retries": 0, "seconds": 131.7, "max_memory_gb": 16.78, "ce": 0.19460777408676222, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.3173632025718689, "loss_scale": 4.0, "retries": 0, "seconds": 133.3, "max_memory_gb": 16.78, "ce": 0.08649988364140881, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.16404421627521515, "loss_scale": 4.0, "retries": 0, "seconds": 134.8, "max_memory_gb": 16.78, "ce": 0.03385820166863596, "rank": 0.1547664999961853}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.45436111092567444, "loss_scale": 4.0, "retries": 0, "seconds": 136.3, "max_memory_gb": 16.78, "ce": 0.1522853876449517, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 1.0701878070831299, "loss_scale": 4.0, "retries": 0, "seconds": 137.9, "max_memory_gb": 16.78, "ce": 0.5029516269351006, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 0.9052855968475342, "loss_scale": 4.0, "retries": 0, "seconds": 139.4, "max_memory_gb": 16.78, "ce": 0.28166769688596105, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 0.45169273018836975, "loss_scale": 4.0, "retries": 0, "seconds": 141.0, "max_memory_gb": 16.78, "ce": 0.1529729224566836, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 0.9205766916275024, "loss_scale": 4.0, "retries": 0, "seconds": 142.5, "max_memory_gb": 16.78, "ce": 0.17783982306718826, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 1.7510740756988525, "loss_scale": 4.0, "retries": 0, "seconds": 144.1, "max_memory_gb": 16.78, "ce": 0.5694542856654152, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 1.3295416831970215, "loss_scale": 4.0, "retries": 0, "seconds": 145.6, "max_memory_gb": 16.78, "ce": 0.5255832298425958, "rank": 0.46035368740558624}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.45402151346206665, "loss_scale": 4.0, "retries": 0, "seconds": 147.1, "max_memory_gb": 16.78, "ce": 0.07138793239573715, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.4039345681667328, "loss_scale": 4.0, "retries": 0, "seconds": 148.7, "max_memory_gb": 16.78, "ce": 0.07696541323457495, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.1329682469367981, "loss_scale": 4.0, "retries": 0, "seconds": 150.2, "max_memory_gb": 16.78, "ce": 0.01974081931166438, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.940768301486969, "loss_scale": 4.0, "retries": 0, "seconds": 151.8, "max_memory_gb": 16.78, "ce": 0.16638686938676983, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.2852882146835327, "loss_scale": 4.0, "retries": 0, "seconds": 153.4, "max_memory_gb": 16.78, "ce": 0.031689250376075506, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 1.165181279182434, "loss_scale": 4.0, "retries": 0, "seconds": 154.9, "max_memory_gb": 16.78, "ce": 0.0945069776425953, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.25988277792930603, "loss_scale": 4.0, "retries": 0, "seconds": 156.4, "max_memory_gb": 16.78, "ce": 0.03934515767468838, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.1789223551750183, "loss_scale": 4.0, "retries": 0, "seconds": 157.9, "max_memory_gb": 16.78, "ce": 0.0493835579682127, "rank": 0.12493757158517838}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.6740671992301941, "loss_scale": 4.0, "retries": 0, "seconds": 159.5, "max_memory_gb": 16.78, "ce": 0.15209730714559555, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.5428918600082397, "loss_scale": 4.0, "retries": 0, "seconds": 161.0, "max_memory_gb": 16.78, "ce": 0.04418035286653321, "rank": 0.002617165446281433}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 3.6297459602355957, "loss_scale": 4.0, "retries": 0, "seconds": 162.6, "max_memory_gb": 16.78, "ce": 0.38124211464310065, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 0.6317566633224487, "loss_scale": 4.0, "retries": 0, "seconds": 164.1, "max_memory_gb": 16.78, "ce": 0.08542108115216251, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 0.7328076958656311, "loss_scale": 4.0, "retries": 0, "seconds": 165.7, "max_memory_gb": 16.78, "ce": 0.20564918406307697, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 1.5792745351791382, "loss_scale": 4.0, "retries": 0, "seconds": 167.2, "max_memory_gb": 16.78, "ce": 0.12741499993717298, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.38292795419692993, "loss_scale": 4.0, "retries": 0, "seconds": 168.8, "max_memory_gb": 16.78, "ce": 0.07624821210447408, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.21945877373218536, "loss_scale": 4.0, "retries": 0, "seconds": 170.4, "max_memory_gb": 16.78, "ce": 0.016247477797151078, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.49685096740722656, "loss_scale": 4.0, "retries": 0, "seconds": 171.9, "max_memory_gb": 16.78, "ce": 0.06936070203664713, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.688420832157135, "loss_scale": 4.0, "retries": 0, "seconds": 173.5, "max_memory_gb": 16.78, "ce": 0.14230191364185885, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.20160914957523346, "loss_scale": 4.0, "retries": 0, "seconds": 175.0, "max_memory_gb": 16.78, "ce": 0.02184953462710837, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.5464684963226318, "loss_scale": 4.0, "retries": 0, "seconds": 176.6, "max_memory_gb": 16.78, "ce": 0.19128370999897015, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.8281389474868774, "loss_scale": 4.0, "retries": 0, "seconds": 178.1, "max_memory_gb": 16.78, "ce": 0.07862286260569817, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 0.6492772698402405, "loss_scale": 4.0, "retries": 0, "seconds": 179.7, "max_memory_gb": 16.78, "ce": 0.24466115756968065, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.13920386135578156, "loss_scale": 4.0, "retries": 0, "seconds": 181.2, "max_memory_gb": 16.78, "ce": 0.02010808230988914, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 2.8180410861968994, "loss_scale": 4.0, "retries": 0, "seconds": 182.8, "max_memory_gb": 16.78, "ce": 0.3052805726110819, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.7371824383735657, "loss_scale": 4.0, "retries": 0, "seconds": 184.3, "max_memory_gb": 16.78, "ce": 0.08296389189126785, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 1.0779829025268555, "loss_scale": 4.0, "retries": 0, "seconds": 185.9, "max_memory_gb": 16.78, "ce": 0.12080882451118669, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.4486510157585144, "loss_scale": 4.0, "retries": 0, "seconds": 187.4, "max_memory_gb": 16.78, "ce": 0.08424561464926228, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.6325362920761108, "loss_scale": 4.0, "retries": 0, "seconds": 189.0, "max_memory_gb": 16.78, "ce": 0.6524999281245982, "rank": 0.1151198148727417}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.3715462386608124, "loss_scale": 4.0, "retries": 0, "seconds": 190.5, "max_memory_gb": 16.78, "ce": 0.1586785038234666, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 1.9701311588287354, "loss_scale": 4.0, "retries": 0, "seconds": 192.0, "max_memory_gb": 16.78, "ce": 0.6158983134082519, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.16578494012355804, "loss_scale": 4.0, "retries": 0, "seconds": 193.6, "max_memory_gb": 16.78, "ce": 0.015905673471024784, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 1.3768643140792847, "loss_scale": 4.0, "retries": 0, "seconds": 195.2, "max_memory_gb": 16.78, "ce": 0.25964878986997064, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 0.9375837445259094, "loss_scale": 4.0, "retries": 0, "seconds": 196.8, "max_memory_gb": 16.78, "ce": 0.0847143103601411, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 1.0486830472946167, "loss_scale": 4.0, "retries": 0, "seconds": 198.4, "max_memory_gb": 16.78, "ce": 0.21751831617439166, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 1.6894011497497559, "loss_scale": 4.0, "retries": 0, "seconds": 199.9, "max_memory_gb": 16.78, "ce": 0.2033559635747224, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 1.8964484930038452, "loss_scale": 4.0, "retries": 0, "seconds": 201.5, "max_memory_gb": 16.78, "ce": 0.1867852986906655, "rank": 0.07331356406211853}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 1.4569593667984009, "loss_scale": 4.0, "retries": 0, "seconds": 203.1, "max_memory_gb": 16.78, "ce": 0.43887688271934167, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.8109880685806274, "loss_scale": 4.0, "retries": 0, "seconds": 204.7, "max_memory_gb": 16.78, "ce": 0.15593633653679717, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 0.29238227009773254, "loss_scale": 4.0, "retries": 0, "seconds": 206.2, "max_memory_gb": 16.78, "ce": 0.028336078714346513, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 0.7477676272392273, "loss_scale": 4.0, "retries": 0, "seconds": 207.7, "max_memory_gb": 16.78, "ce": 0.06875496808788739, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.5760160684585571, "loss_scale": 4.0, "retries": 0, "seconds": 209.4, "max_memory_gb": 16.78, "ce": 0.03415641185711138, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 0.6443371772766113, "loss_scale": 4.0, "retries": 0, "seconds": 211.0, "max_memory_gb": 16.78, "ce": 0.09943750948878005, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.6051102876663208, "loss_scale": 4.0, "retries": 0, "seconds": 212.7, "max_memory_gb": 16.78, "ce": 0.07219488282862585, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.137943297624588, "loss_scale": 4.0, "retries": 0, "seconds": 214.3, "max_memory_gb": 16.78, "ce": 0.0198603485696367, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.1234668716788292, "loss_scale": 4.0, "retries": 0, "seconds": 216.0, "max_memory_gb": 16.78, "ce": 0.01762852033789386, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.9353194236755371, "loss_scale": 4.0, "retries": 0, "seconds": 217.5, "max_memory_gb": 16.78, "ce": 0.09471022153593367, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 2.1634695529937744, "loss_scale": 4.0, "retries": 0, "seconds": 219.1, "max_memory_gb": 16.78, "ce": 0.2710210205968906, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 1.037036657333374, "loss_scale": 4.0, "retries": 0, "seconds": 220.7, "max_memory_gb": 16.78, "ce": 0.09541414299746975, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.10305819660425186, "loss_scale": 4.0, "retries": 0, "seconds": 222.4, "max_memory_gb": 16.78, "ce": 0.012666585704209865, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.21769988536834717, "loss_scale": 4.0, "retries": 0, "seconds": 224.0, "max_memory_gb": 16.78, "ce": 0.028312200389336795, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 1.733681082725525, "loss_scale": 4.0, "retries": 0, "seconds": 225.5, "max_memory_gb": 16.78, "ce": 0.5957203221914824, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.11863616853952408, "loss_scale": 4.0, "retries": 0, "seconds": 227.0, "max_memory_gb": 16.78, "ce": 0.019842313263325195, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.10226717591285706, "loss_scale": 4.0, "retries": 0, "seconds": 228.5, "max_memory_gb": 16.78, "ce": 0.0161619589489419, "rank": 0.12517833709716797}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.29388436675071716, "loss_scale": 4.0, "retries": 0, "seconds": 230.0, "max_memory_gb": 16.78, "ce": 0.05908076590276323, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.5680608749389648, "loss_scale": 4.0, "retries": 0, "seconds": 231.4, "max_memory_gb": 16.78, "ce": 0.054874562192708254, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 0.5292342305183411, "loss_scale": 4.0, "retries": 0, "seconds": 232.7, "max_memory_gb": 16.78, "ce": 0.05554231600399362, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.8563212156295776, "loss_scale": 4.0, "retries": 0, "seconds": 234.1, "max_memory_gb": 16.78, "ce": 0.0938820648680121, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 1.0835498571395874, "loss_scale": 4.0, "retries": 0, "seconds": 235.4, "max_memory_gb": 16.78, "ce": 0.07714273314923048, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 2.0041799545288086, "loss_scale": 4.0, "retries": 0, "seconds": 236.8, "max_memory_gb": 16.78, "ce": 0.22056410391996906, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.13090592622756958, "loss_scale": 4.0, "retries": 0, "seconds": 238.2, "max_memory_gb": 16.78, "ce": 0.013776929825326079, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.22980022430419922, "loss_scale": 4.0, "retries": 0, "seconds": 239.5, "max_memory_gb": 16.78, "ce": 0.020719967636978254, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 1.9407362937927246, "loss_scale": 4.0, "retries": 0, "seconds": 240.8, "max_memory_gb": 16.78, "ce": 0.2945260865108139, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.27754276990890503, "loss_scale": 4.0, "retries": 0, "seconds": 242.2, "max_memory_gb": 16.8, "ce": 0.03613884030346526, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 0.7542209029197693, "loss_scale": 4.0, "retries": 0, "seconds": 243.6, "max_memory_gb": 16.8, "ce": 0.173373549914686, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 1.0054645538330078, "loss_scale": 4.0, "retries": 0, "seconds": 245.0, "max_memory_gb": 16.8, "ce": 0.12787208391273452, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.6108832955360413, "loss_scale": 4.0, "retries": 0, "seconds": 246.4, "max_memory_gb": 16.8, "ce": 0.07437946584468591, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 1.6733328104019165, "loss_scale": 4.0, "retries": 0, "seconds": 247.8, "max_memory_gb": 16.8, "ce": 0.21096126007614657, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.5740089416503906, "loss_scale": 4.0, "retries": 0, "seconds": 249.3, "max_memory_gb": 16.8, "ce": 0.07620846296777017, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.10162289440631866, "loss_scale": 4.0, "retries": 0, "seconds": 250.8, "max_memory_gb": 16.8, "ce": 0.01774210686380684, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 2.4022207260131836, "loss_scale": 4.0, "retries": 0, "seconds": 252.2, "max_memory_gb": 16.8, "ce": 0.413928660491365, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.2378043532371521, "loss_scale": 4.0, "retries": 0, "seconds": 253.7, "max_memory_gb": 16.8, "ce": 0.021413152921013534, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.22400905191898346, "loss_scale": 4.0, "retries": 0, "seconds": 255.1, "max_memory_gb": 16.8, "ce": 0.017296008329140022, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 0.9662556052207947, "loss_scale": 4.0, "retries": 0, "seconds": 256.6, "max_memory_gb": 16.8, "ce": 0.3239500188501552, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.09128303825855255, "loss_scale": 4.0, "retries": 0, "seconds": 258.0, "max_memory_gb": 16.8, "ce": 0.011543903188794502, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.35227328538894653, "loss_scale": 4.0, "retries": 0, "seconds": 259.5, "max_memory_gb": 16.8, "ce": 0.07043580033860053, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.24177131056785583, "loss_scale": 4.0, "retries": 0, "seconds": 261.0, "max_memory_gb": 16.8, "ce": 0.02455143118277192, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.5751469731330872, "loss_scale": 4.0, "retries": 0, "seconds": 262.4, "max_memory_gb": 16.8, "ce": 0.045467509946320206, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.5850371718406677, "loss_scale": 4.0, "retries": 0, "seconds": 263.8, "max_memory_gb": 16.8, "ce": 0.04513931712244812, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 2.8860971927642822, "loss_scale": 4.0, "retries": 0, "seconds": 265.2, "max_memory_gb": 16.8, "ce": 0.27248293487355113, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.23046061396598816, "loss_scale": 4.0, "retries": 0, "seconds": 266.7, "max_memory_gb": 16.8, "ce": 0.017029203576385044, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 1.0811340808868408, "loss_scale": 4.0, "retries": 0, "seconds": 268.1, "max_memory_gb": 16.8, "ce": 0.3126157239494205, "rank": 0.12505830824375153}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 1.0898168087005615, "loss_scale": 4.0, "retries": 0, "seconds": 269.5, "max_memory_gb": 16.8, "ce": 0.07869886196567677, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 1.0695044994354248, "loss_scale": 4.0, "retries": 0, "seconds": 271.0, "max_memory_gb": 16.8, "ce": 0.4528324717666692, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.46168550848960876, "loss_scale": 4.0, "retries": 0, "seconds": 272.4, "max_memory_gb": 16.8, "ce": 0.04454532780800946, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.4322090446949005, "loss_scale": 4.0, "retries": 0, "seconds": 273.8, "max_memory_gb": 16.8, "ce": 0.039630236191442236, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 0.9722033143043518, "loss_scale": 4.0, "retries": 0, "seconds": 275.3, "max_memory_gb": 16.8, "ce": 0.10181806984473951, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.13966163992881775, "loss_scale": 4.0, "retries": 0, "seconds": 276.7, "max_memory_gb": 16.8, "ce": 0.017636241031596, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.18407851457595825, "loss_scale": 4.0, "retries": 0, "seconds": 278.1, "max_memory_gb": 16.8, "ce": 0.021894609965784184, "rank": 0.21598920971155167}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.5213875770568848, "loss_scale": 4.0, "retries": 0, "seconds": 279.5, "max_memory_gb": 16.8, "ce": 0.061839949688874185, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 0.9502987861633301, "loss_scale": 4.0, "retries": 0, "seconds": 281.0, "max_memory_gb": 16.8, "ce": 0.11599104114338843, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.9269958138465881, "loss_scale": 4.0, "retries": 0, "seconds": 282.4, "max_memory_gb": 16.8, "ce": 0.16964178916532546, "rank": 0.14104925096035004}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.3019795417785645, "loss_scale": 4.0, "retries": 0, "seconds": 283.8, "max_memory_gb": 16.8, "ce": 0.22385787900748255, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 1.608216404914856, "loss_scale": 4.0, "retries": 0, "seconds": 285.3, "max_memory_gb": 16.8, "ce": 0.34742220159387216, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.2522124648094177, "loss_scale": 4.0, "retries": 0, "seconds": 286.7, "max_memory_gb": 16.8, "ce": 0.025486428494332358, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 1.6497955322265625, "loss_scale": 4.0, "retries": 0, "seconds": 288.1, "max_memory_gb": 16.8, "ce": 0.2001343799638562, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.530015230178833, "loss_scale": 4.0, "retries": 0, "seconds": 289.6, "max_memory_gb": 16.8, "ce": 0.5047714955289848, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.11133411526679993, "loss_scale": 4.0, "retries": 0, "seconds": 291.2, "max_memory_gb": 16.8, "ce": 0.010724710235081147, "rank": 0.12503483891487122}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.6048194766044617, "loss_scale": 4.0, "retries": 0, "seconds": 292.7, "max_memory_gb": 16.8, "ce": 0.06542376823199447, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.5520139336585999, "loss_scale": 4.0, "retries": 0, "seconds": 294.3, "max_memory_gb": 16.8, "ce": 0.10758331825491041, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.5029297471046448, "loss_scale": 4.0, "retries": 0, "seconds": 295.9, "max_memory_gb": 16.8, "ce": 0.03893508391229261, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 1.1186634302139282, "loss_scale": 4.0, "retries": 0, "seconds": 297.5, "max_memory_gb": 16.8, "ce": 0.24169424554565921, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.3071366548538208, "loss_scale": 4.0, "retries": 0, "seconds": 299.1, "max_memory_gb": 16.8, "ce": 0.06656261278840248, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 2.3504722118377686, "loss_scale": 4.0, "retries": 0, "seconds": 300.7, "max_memory_gb": 16.8, "ce": 0.1566917145100888, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 1.1371335983276367, "loss_scale": 4.0, "retries": 0, "seconds": 302.3, "max_memory_gb": 16.8, "ce": 0.33103913560626097, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.5354625582695007, "loss_scale": 4.0, "retries": 0, "seconds": 303.8, "max_memory_gb": 16.8, "ce": 0.054642939794575796, "rank": 0.0008561909198760986}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.15885204076766968, "loss_scale": 4.0, "retries": 0, "seconds": 305.4, "max_memory_gb": 16.8, "ce": 0.015653120606657467, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.20787584781646729, "loss_scale": 4.0, "retries": 0, "seconds": 307.0, "max_memory_gb": 16.8, "ce": 0.019968702930782456, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.3463091552257538, "loss_scale": 4.0, "retries": 0, "seconds": 308.6, "max_memory_gb": 16.8, "ce": 0.0492799466883298, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 0.6469545364379883, "loss_scale": 4.0, "retries": 0, "seconds": 310.1, "max_memory_gb": 16.8, "ce": 0.09508692182225786, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.39050930738449097, "loss_scale": 4.0, "retries": 0, "seconds": 311.7, "max_memory_gb": 16.8, "ce": 0.0443894675518095, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.20338447391986847, "loss_scale": 4.0, "retries": 0, "seconds": 313.3, "max_memory_gb": 16.8, "ce": 0.049701971918693744, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.2032032012939453, "loss_scale": 4.0, "retries": 0, "seconds": 314.9, "max_memory_gb": 16.8, "ce": 0.0236838084529154, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.5317086577415466, "loss_scale": 4.0, "retries": 0, "seconds": 316.4, "max_memory_gb": 16.8, "ce": 0.12489818298490718, "rank": 0.12620504200458527}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 1.206904649734497, "loss_scale": 4.0, "retries": 0, "seconds": 317.8, "max_memory_gb": 16.8, "ce": 0.4625052839255659, "rank": 0.18377095460891724}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.5408017635345459, "loss_scale": 4.0, "retries": 0, "seconds": 319.3, "max_memory_gb": 16.8, "ce": 0.14875406062128604, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 0.9617842435836792, "loss_scale": 4.0, "retries": 0, "seconds": 320.7, "max_memory_gb": 16.8, "ce": 0.2378714280494023, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.6873348355293274, "loss_scale": 4.0, "retries": 0, "seconds": 322.2, "max_memory_gb": 16.8, "ce": 0.20318018263787962, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 2.3202364444732666, "loss_scale": 4.0, "retries": 0, "seconds": 323.6, "max_memory_gb": 16.8, "ce": 0.32203216288144176, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 1.502170205116272, "loss_scale": 4.0, "retries": 0, "seconds": 325.1, "max_memory_gb": 16.8, "ce": 0.535519843986549, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.374936044216156, "loss_scale": 4.0, "retries": 0, "seconds": 326.5, "max_memory_gb": 16.8, "ce": 0.0335522273380775, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.04768132418394089, "loss_scale": 4.0, "retries": 0, "seconds": 328.0, "max_memory_gb": 16.8, "ce": 0.0074414954251551535, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 0.5838769674301147, "loss_scale": 4.0, "retries": 0, "seconds": 329.4, "max_memory_gb": 16.8, "ce": 0.2059758113900898, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.1552928388118744, "loss_scale": 4.0, "retries": 0, "seconds": 330.9, "max_memory_gb": 16.8, "ce": 0.026733480735856574, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.880544126033783, "loss_scale": 4.0, "retries": 0, "seconds": 332.3, "max_memory_gb": 16.8, "ce": 0.05078040199987299, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.6094248294830322, "loss_scale": 4.0, "retries": 0, "seconds": 333.8, "max_memory_gb": 16.8, "ce": 0.07329153837054037, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.48466476798057556, "loss_scale": 4.0, "retries": 0, "seconds": 335.3, "max_memory_gb": 16.8, "ce": 0.05057234033301938, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 1.2432962656021118, "loss_scale": 4.0, "retries": 0, "seconds": 336.7, "max_memory_gb": 16.8, "ce": 0.07071147608803585, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.497188925743103, "loss_scale": 4.0, "retries": 0, "seconds": 338.1, "max_memory_gb": 16.8, "ce": 0.05340726514259586, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.3509179353713989, "loss_scale": 4.0, "retries": 0, "seconds": 339.6, "max_memory_gb": 16.8, "ce": 0.029665453665074892, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 1.0183850526809692, "loss_scale": 4.0, "retries": 0, "seconds": 341.1, "max_memory_gb": 16.8, "ce": 0.06496373637855868, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.5181592106819153, "loss_scale": 4.0, "retries": 0, "seconds": 342.5, "max_memory_gb": 16.8, "ce": 0.059381486567872344, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.2703230381011963, "loss_scale": 4.0, "retries": 0, "seconds": 343.9, "max_memory_gb": 16.8, "ce": 0.018573018627648707, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.3215513229370117, "loss_scale": 4.0, "retries": 0, "seconds": 345.4, "max_memory_gb": 16.8, "ce": 0.053682673154980876, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 2.8088884353637695, "loss_scale": 4.0, "retries": 0, "seconds": 346.8, "max_memory_gb": 16.8, "ce": 0.485848827418522, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.5202431678771973, "loss_scale": 4.0, "retries": 0, "seconds": 348.2, "max_memory_gb": 16.8, "ce": 0.05351979171973653, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.7144002914428711, "loss_scale": 4.0, "retries": 0, "seconds": 349.7, "max_memory_gb": 16.8, "ce": 0.06131319445557892, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.3323226869106293, "loss_scale": 4.0, "retries": 0, "seconds": 351.1, "max_memory_gb": 16.8, "ce": 0.043137979082530364, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.23906102776527405, "loss_scale": 4.0, "retries": 0, "seconds": 352.6, "max_memory_gb": 16.8, "ce": 0.01727407082580612, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 2.018420934677124, "loss_scale": 4.0, "retries": 0, "seconds": 354.0, "max_memory_gb": 16.8, "ce": 0.32514211578381946, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.12928543984889984, "loss_scale": 4.0, "retries": 0, "seconds": 355.4, "max_memory_gb": 16.8, "ce": 0.022031172371498542, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.25453364849090576, "loss_scale": 4.0, "retries": 0, "seconds": 356.8, "max_memory_gb": 16.8, "ce": 0.06818883647792973, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.2645283043384552, "loss_scale": 4.0, "retries": 0, "seconds": 358.3, "max_memory_gb": 16.8, "ce": 0.05499489384237677, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.39426037669181824, "loss_scale": 4.0, "retries": 0, "seconds": 359.7, "max_memory_gb": 16.8, "ce": 0.13590637227753177, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 0.9213244915008545, "loss_scale": 4.0, "retries": 0, "seconds": 361.1, "max_memory_gb": 16.8, "ce": 0.07426619685065816, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 1.343058705329895, "loss_scale": 4.0, "retries": 0, "seconds": 362.6, "max_memory_gb": 16.8, "ce": 0.17907767747192338, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.08305712044239044, "loss_scale": 4.0, "retries": 0, "seconds": 364.0, "max_memory_gb": 16.8, "ce": 0.009699935792014003, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.15318551659584045, "loss_scale": 4.0, "retries": 0, "seconds": 365.5, "max_memory_gb": 16.8, "ce": 0.0159960458840942, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.5280833840370178, "loss_scale": 4.0, "retries": 0, "seconds": 366.9, "max_memory_gb": 16.8, "ce": 0.046055402897764, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 1.3959462642669678, "loss_scale": 4.0, "retries": 0, "seconds": 368.3, "max_memory_gb": 16.8, "ce": 0.33388541550084483, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.06807074695825577, "loss_scale": 4.0, "retries": 0, "seconds": 369.8, "max_memory_gb": 16.8, "ce": 0.008858784047333756, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.6526767015457153, "loss_scale": 4.0, "retries": 0, "seconds": 371.2, "max_memory_gb": 16.8, "ce": 0.06281339185625257, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.30216872692108154, "loss_scale": 4.0, "retries": 0, "seconds": 372.6, "max_memory_gb": 16.8, "ce": 0.02613852587819565, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.4610690772533417, "loss_scale": 4.0, "retries": 0, "seconds": 374.1, "max_memory_gb": 16.8, "ce": 0.04042122850660235, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.4199230372905731, "loss_scale": 4.0, "retries": 0, "seconds": 375.5, "max_memory_gb": 16.8, "ce": 0.0420329955231864, "rank": 0.0}

runs/single_internvl/removal/L34_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/single_internvl/removal/L34_s13/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.34.mlp.down_proj",
+    "model.language_model.layers.34.mlp.gate_proj",
+    "model.language_model.layers.34.self_attn.k_proj",
+    "model.language_model.layers.34.self_attn.v_proj",
+    "model.language_model.layers.34.self_attn.q_proj",
+    "model.language_model.layers.34.mlp.up_proj",
+    "model.language_model.layers.34.self_attn.o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/single_internvl/removal/L34_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 34,
+    "rank": 16,
+    "trainable_parameters": 1212416
+  }
+}

runs/single_internvl/removal/L34_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 0.37697282433509827, "loss_scale": 4.0, "retries": 0, "seconds": 1.9, "max_memory_gb": 16.47, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 0.36333996057510376, "loss_scale": 4.0, "retries": 0, "seconds": 3.0, "max_memory_gb": 16.47, "ce": 0.7818417346825299, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 1.0036683082580566, "loss_scale": 4.0, "retries": 0, "seconds": 4.2, "max_memory_gb": 16.47, "ce": 1.3800277495756745, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 0.504908561706543, "loss_scale": 4.0, "retries": 0, "seconds": 5.3, "max_memory_gb": 16.47, "ce": 0.753200720928362, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.01706514321267605, "loss_scale": 4.0, "retries": 0, "seconds": 6.4, "max_memory_gb": 16.47, "ce": 0.0401000810561527, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 0.2778424322605133, "loss_scale": 4.0, "retries": 0, "seconds": 7.5, "max_memory_gb": 16.47, "ce": 0.4374538242336712, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 0.7788075804710388, "loss_scale": 4.0, "retries": 0, "seconds": 8.7, "max_memory_gb": 16.47, "ce": 1.0321714658166457, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 0.9445608258247375, "loss_scale": 4.0, "retries": 0, "seconds": 9.8, "max_memory_gb": 16.47, "ce": 1.0113956909626722, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 0.6100039482116699, "loss_scale": 4.0, "retries": 0, "seconds": 10.9, "max_memory_gb": 16.47, "ce": 0.7282404857687652, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.14395664632320404, "loss_scale": 4.0, "retries": 0, "seconds": 12.0, "max_memory_gb": 16.47, "ce": 0.19443038578901906, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.8624423742294312, "loss_scale": 4.0, "retries": 0, "seconds": 13.2, "max_memory_gb": 16.47, "ce": 0.8527663172280882, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 0.5775631070137024, "loss_scale": 4.0, "retries": 0, "seconds": 14.3, "max_memory_gb": 16.47, "ce": 0.5738065013301821, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 0.6007704734802246, "loss_scale": 4.0, "retries": 0, "seconds": 15.4, "max_memory_gb": 16.47, "ce": 0.6540568474410975, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 0.9638102650642395, "loss_scale": 4.0, "retries": 0, "seconds": 16.6, "max_memory_gb": 16.47, "ce": 1.043966586701572, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 0.8501530885696411, "loss_scale": 4.0, "retries": 0, "seconds": 17.7, "max_memory_gb": 16.47, "ce": 0.7092022457311487, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.48231443762779236, "loss_scale": 4.0, "retries": 0, "seconds": 18.8, "max_memory_gb": 16.47, "ce": 0.318986185688118, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.6860455274581909, "loss_scale": 4.0, "retries": 0, "seconds": 19.9, "max_memory_gb": 16.47, "ce": 0.5834681881206052, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 1.2794744968414307, "loss_scale": 4.0, "retries": 0, "seconds": 21.0, "max_memory_gb": 16.47, "ce": 0.8209197083051549, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 2.456230640411377, "loss_scale": 4.0, "retries": 0, "seconds": 22.2, "max_memory_gb": 16.47, "ce": 1.1548096146434546, "rank": 0.0}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.01639426313340664, "loss_scale": 4.0, "retries": 0, "seconds": 23.3, "max_memory_gb": 16.47, "ce": 0.04053511290476308, "rank": 0.12113901972770691}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 1.4040708541870117, "loss_scale": 4.0, "retries": 0, "seconds": 24.4, "max_memory_gb": 16.47, "ce": 0.6264293238955361, "rank": 0.0}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 1.662851333618164, "loss_scale": 4.0, "retries": 0, "seconds": 25.6, "max_memory_gb": 16.47, "ce": 0.7443132800908643, "rank": 0.37799394130706787}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.7366935014724731, "loss_scale": 4.0, "retries": 0, "seconds": 26.7, "max_memory_gb": 16.47, "ce": 0.8229905467569552, "rank": 0.02319541573524475}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 1.7738651037216187, "loss_scale": 4.0, "retries": 0, "seconds": 27.8, "max_memory_gb": 16.47, "ce": 0.8186458954587579, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 2.140807628631592, "loss_scale": 4.0, "retries": 0, "seconds": 29.0, "max_memory_gb": 16.47, "ce": 0.9036249301643693, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 1.7865381240844727, "loss_scale": 4.0, "retries": 0, "seconds": 30.1, "max_memory_gb": 16.47, "ce": 0.763377083465457, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 1.9263668060302734, "loss_scale": 4.0, "retries": 0, "seconds": 31.4, "max_memory_gb": 16.47, "ce": 0.7591586427370203, "rank": 0.0}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 1.0858981609344482, "loss_scale": 4.0, "retries": 0, "seconds": 32.7, "max_memory_gb": 16.47, "ce": 0.8656565840356052, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 1.738990306854248, "loss_scale": 4.0, "retries": 0, "seconds": 34.0, "max_memory_gb": 16.47, "ce": 0.6459723094012588, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 2.015662670135498, "loss_scale": 4.0, "retries": 0, "seconds": 35.3, "max_memory_gb": 16.47, "ce": 0.4807385348412936, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 2.4972968101501465, "loss_scale": 4.0, "retries": 0, "seconds": 36.6, "max_memory_gb": 16.47, "ce": 0.7869028197601438, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 2.892801284790039, "loss_scale": 4.0, "retries": 0, "seconds": 37.9, "max_memory_gb": 16.47, "ce": 0.7059161504730582, "rank": 0.004937350749969482}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 1.2435386180877686, "loss_scale": 4.0, "retries": 0, "seconds": 39.1, "max_memory_gb": 16.47, "ce": 0.3597019736189395, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 1.0483580827713013, "loss_scale": 4.0, "retries": 0, "seconds": 40.2, "max_memory_gb": 16.47, "ce": 0.36524646337056765, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 1.4891506433486938, "loss_scale": 4.0, "retries": 0, "seconds": 41.3, "max_memory_gb": 16.47, "ce": 0.3100005729138502, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 1.4995447397232056, "loss_scale": 4.0, "retries": 0, "seconds": 42.5, "max_memory_gb": 16.47, "ce": 0.4273763623277773, "rank": 0.0}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 1.1326344013214111, "loss_scale": 4.0, "retries": 0, "seconds": 43.6, "max_memory_gb": 16.47, "ce": 0.5760487298248336, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 1.3268370628356934, "loss_scale": 4.0, "retries": 0, "seconds": 44.7, "max_memory_gb": 16.47, "ce": 0.6125357612036169, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 0.3543999195098877, "loss_scale": 4.0, "retries": 0, "seconds": 45.9, "max_memory_gb": 16.47, "ce": 0.3882125968375476, "rank": 0.12494726479053497}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.516063928604126, "loss_scale": 4.0, "retries": 0, "seconds": 47.0, "max_memory_gb": 16.47, "ce": 0.2348732163227396, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.5082621574401855, "loss_scale": 4.0, "retries": 0, "seconds": 48.1, "max_memory_gb": 16.47, "ce": 0.44254923303378746, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.8029446005821228, "loss_scale": 4.0, "retries": 0, "seconds": 49.2, "max_memory_gb": 16.47, "ce": 0.24273295654711546, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 1.118118166923523, "loss_scale": 4.0, "retries": 0, "seconds": 50.3, "max_memory_gb": 16.47, "ce": 0.2558760981919477, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 1.0303400754928589, "loss_scale": 4.0, "retries": 0, "seconds": 51.5, "max_memory_gb": 16.47, "ce": 0.6300190602105431, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 1.1678783893585205, "loss_scale": 4.0, "retries": 0, "seconds": 52.6, "max_memory_gb": 16.47, "ce": 0.7487031390191987, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.4480407238006592, "loss_scale": 4.0, "retries": 0, "seconds": 53.7, "max_memory_gb": 16.47, "ce": 0.3354267397080548, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.3894316256046295, "loss_scale": 4.0, "retries": 0, "seconds": 54.8, "max_memory_gb": 16.47, "ce": 0.18915487424237654, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.7497044801712036, "loss_scale": 4.0, "retries": 0, "seconds": 55.9, "max_memory_gb": 16.47, "ce": 0.3013283763139043, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.8198146820068359, "loss_scale": 4.0, "retries": 0, "seconds": 57.1, "max_memory_gb": 16.47, "ce": 0.3167969326641469, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 1.2758241891860962, "loss_scale": 4.0, "retries": 0, "seconds": 58.2, "max_memory_gb": 16.47, "ce": 0.34982589607534464, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 0.8858809471130371, "loss_scale": 4.0, "retries": 0, "seconds": 59.3, "max_memory_gb": 16.47, "ce": 0.5014352126418089, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.6670557260513306, "loss_scale": 4.0, "retries": 0, "seconds": 60.4, "max_memory_gb": 16.47, "ce": 0.2663166034471942, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 1.0014262199401855, "loss_scale": 4.0, "retries": 0, "seconds": 61.5, "max_memory_gb": 16.47, "ce": 0.49504307843744755, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 0.8236311078071594, "loss_scale": 4.0, "retries": 0, "seconds": 62.6, "max_memory_gb": 16.47, "ce": 0.3750501309405081, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.3874347507953644, "loss_scale": 4.0, "retries": 0, "seconds": 63.7, "max_memory_gb": 16.47, "ce": 0.15508591146499384, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.820077121257782, "loss_scale": 4.0, "retries": 0, "seconds": 64.8, "max_memory_gb": 16.47, "ce": 0.3332490543834865, "rank": 0.03994583338499069}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.6360047459602356, "loss_scale": 4.0, "retries": 0, "seconds": 65.9, "max_memory_gb": 16.47, "ce": 0.15280931848246837, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.41774049401283264, "loss_scale": 4.0, "retries": 0, "seconds": 67.0, "max_memory_gb": 16.47, "ce": 0.15604140435607405, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.5618591904640198, "loss_scale": 4.0, "retries": 0, "seconds": 68.2, "max_memory_gb": 16.47, "ce": 0.11329113170359051, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 1.1832886934280396, "loss_scale": 4.0, "retries": 0, "seconds": 69.3, "max_memory_gb": 16.47, "ce": 0.20346714655170217, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.799121081829071, "loss_scale": 4.0, "retries": 0, "seconds": 70.4, "max_memory_gb": 16.47, "ce": 0.44520545075647533, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.26936405897140503, "loss_scale": 4.0, "retries": 0, "seconds": 71.5, "max_memory_gb": 16.47, "ce": 0.12665179833129514, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.8390800952911377, "loss_scale": 4.0, "retries": 0, "seconds": 72.6, "max_memory_gb": 16.47, "ce": 0.19278360030148178, "rank": 0.0}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.6208792328834534, "loss_scale": 4.0, "retries": 0, "seconds": 73.8, "max_memory_gb": 16.47, "ce": 0.1897073938162066, "rank": 0.09596826881170273}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 1.658509612083435, "loss_scale": 4.0, "retries": 0, "seconds": 74.9, "max_memory_gb": 16.47, "ce": 0.41829866636544466, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 1.9510027170181274, "loss_scale": 4.0, "retries": 0, "seconds": 76.0, "max_memory_gb": 16.47, "ce": 0.27468980705680224, "rank": 0.12403365224599838}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.5905797481536865, "loss_scale": 4.0, "retries": 0, "seconds": 77.2, "max_memory_gb": 16.47, "ce": 0.15109971178753767, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.2566986680030823, "loss_scale": 4.0, "retries": 0, "seconds": 78.3, "max_memory_gb": 16.47, "ce": 0.08334301332615723, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.9773396849632263, "loss_scale": 4.0, "retries": 0, "seconds": 79.4, "max_memory_gb": 16.47, "ce": 0.39661564212292433, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.414661705493927, "loss_scale": 4.0, "retries": 0, "seconds": 80.5, "max_memory_gb": 16.47, "ce": 0.0978982317756163, "rank": 0.11627974361181259}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.6527422070503235, "loss_scale": 4.0, "retries": 0, "seconds": 81.7, "max_memory_gb": 16.47, "ce": 0.14431387384775007, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 1.08540940284729, "loss_scale": 4.0, "retries": 0, "seconds": 82.8, "max_memory_gb": 16.47, "ce": 0.24851243430748582, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 0.6349141597747803, "loss_scale": 4.0, "retries": 0, "seconds": 83.9, "max_memory_gb": 16.47, "ce": 0.270366591088532, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.43549954891204834, "loss_scale": 4.0, "retries": 0, "seconds": 85.0, "max_memory_gb": 16.47, "ce": 0.14211054017869174, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.9179470539093018, "loss_scale": 4.0, "retries": 0, "seconds": 86.1, "max_memory_gb": 16.47, "ce": 0.6504120186436921, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 1.0235298871994019, "loss_scale": 4.0, "retries": 0, "seconds": 87.3, "max_memory_gb": 16.47, "ce": 0.2624501315294765, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.5703531503677368, "loss_scale": 4.0, "retries": 0, "seconds": 88.3, "max_memory_gb": 16.47, "ce": 0.11682438067509793, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.4967241883277893, "loss_scale": 4.0, "retries": 0, "seconds": 89.4, "max_memory_gb": 16.47, "ce": 0.11845991769223474, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.2319795936346054, "loss_scale": 4.0, "retries": 0, "seconds": 90.5, "max_memory_gb": 16.47, "ce": 0.04191487850039266, "rank": 0.09327591955661774}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.4416199028491974, "loss_scale": 4.0, "retries": 0, "seconds": 91.6, "max_memory_gb": 16.47, "ce": 0.08215789577297983, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.7151677012443542, "loss_scale": 4.0, "retries": 0, "seconds": 92.7, "max_memory_gb": 16.47, "ce": 0.13983060664031655, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.29991278052330017, "loss_scale": 4.0, "retries": 0, "seconds": 93.8, "max_memory_gb": 16.47, "ce": 0.08724006067495793, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 2.084763765335083, "loss_scale": 4.0, "retries": 0, "seconds": 95.0, "max_memory_gb": 16.47, "ce": 0.29444577857793774, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.8226151466369629, "loss_scale": 4.0, "retries": 0, "seconds": 96.1, "max_memory_gb": 16.47, "ce": 0.04368204893080474, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.5176093578338623, "loss_scale": 4.0, "retries": 0, "seconds": 97.2, "max_memory_gb": 16.47, "ce": 0.05600358764422708, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 1.2416189908981323, "loss_scale": 4.0, "retries": 0, "seconds": 98.4, "max_memory_gb": 16.47, "ce": 0.18182196276029572, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.8423271775245667, "loss_scale": 4.0, "retries": 0, "seconds": 99.4, "max_memory_gb": 16.47, "ce": 0.12653345595754217, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.24914036691188812, "loss_scale": 4.0, "retries": 0, "seconds": 100.6, "max_memory_gb": 16.47, "ce": 0.06851762634687475, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.1614539623260498, "loss_scale": 4.0, "retries": 0, "seconds": 101.6, "max_memory_gb": 16.47, "ce": 0.031075716022314737, "rank": 0.14773492515087128}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.41261130571365356, "loss_scale": 4.0, "retries": 0, "seconds": 102.7, "max_memory_gb": 16.47, "ce": 0.14991454457049258, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 0.8956695795059204, "loss_scale": 4.0, "retries": 0, "seconds": 103.9, "max_memory_gb": 16.47, "ce": 0.4102355641553004, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 0.9394790530204773, "loss_scale": 4.0, "retries": 0, "seconds": 105.0, "max_memory_gb": 16.47, "ce": 0.28541965808108216, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 0.5855439305305481, "loss_scale": 4.0, "retries": 0, "seconds": 106.1, "max_memory_gb": 16.47, "ce": 0.16086902178358287, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 1.419149398803711, "loss_scale": 4.0, "retries": 0, "seconds": 107.2, "max_memory_gb": 16.47, "ce": 0.20189730884158053, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 1.777392864227295, "loss_scale": 4.0, "retries": 0, "seconds": 108.4, "max_memory_gb": 16.47, "ce": 0.5439631823101081, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 1.609268069267273, "loss_scale": 4.0, "retries": 0, "seconds": 109.5, "max_memory_gb": 16.47, "ce": 0.497226148028858, "rank": 0.3955492526292801}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.6217010021209717, "loss_scale": 4.0, "retries": 0, "seconds": 110.6, "max_memory_gb": 16.47, "ce": 0.1112474013643805, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.9965576529502869, "loss_scale": 4.0, "retries": 0, "seconds": 111.7, "max_memory_gb": 16.47, "ce": 0.09061716906580841, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.4090101420879364, "loss_scale": 4.0, "retries": 0, "seconds": 112.8, "max_memory_gb": 16.47, "ce": 0.04518565549005871, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 1.2120099067687988, "loss_scale": 4.0, "retries": 0, "seconds": 114.0, "max_memory_gb": 16.47, "ce": 0.20700035567278974, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.7200842499732971, "loss_scale": 4.0, "retries": 0, "seconds": 115.1, "max_memory_gb": 16.47, "ce": 0.048027279030065984, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 2.0659687519073486, "loss_scale": 4.0, "retries": 0, "seconds": 116.2, "max_memory_gb": 16.47, "ce": 0.11383550587925129, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.34520474076271057, "loss_scale": 4.0, "retries": 0, "seconds": 117.3, "max_memory_gb": 16.47, "ce": 0.04552893227082677, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.20903410017490387, "loss_scale": 4.0, "retries": 0, "seconds": 118.4, "max_memory_gb": 16.47, "ce": 0.04851284992037108, "rank": 0.12469123303890228}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.668154776096344, "loss_scale": 4.0, "retries": 0, "seconds": 119.5, "max_memory_gb": 16.47, "ce": 0.18220524722710252, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.6567931771278381, "loss_scale": 4.0, "retries": 0, "seconds": 120.6, "max_memory_gb": 16.47, "ce": 0.03771089453948662, "rank": 0.002963334321975708}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 2.436673402786255, "loss_scale": 4.0, "retries": 0, "seconds": 121.8, "max_memory_gb": 16.47, "ce": 0.5277618865038676, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 0.30281227827072144, "loss_scale": 4.0, "retries": 0, "seconds": 122.9, "max_memory_gb": 16.47, "ce": 0.0964548575648223, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 0.7731270790100098, "loss_scale": 4.0, "retries": 0, "seconds": 124.0, "max_memory_gb": 16.47, "ce": 0.24454095523105934, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 1.291972279548645, "loss_scale": 4.0, "retries": 0, "seconds": 125.1, "max_memory_gb": 16.47, "ce": 0.27912062099494506, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.32300716638565063, "loss_scale": 4.0, "retries": 0, "seconds": 126.3, "max_memory_gb": 16.47, "ce": 0.09244312348891981, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.17786622047424316, "loss_scale": 4.0, "retries": 0, "seconds": 127.4, "max_memory_gb": 16.47, "ce": 0.01920354437606875, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.578056275844574, "loss_scale": 4.0, "retries": 0, "seconds": 128.5, "max_memory_gb": 16.47, "ce": 0.09883112000534311, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.8005799055099487, "loss_scale": 4.0, "retries": 0, "seconds": 129.6, "max_memory_gb": 16.47, "ce": 0.1565826365549583, "rank": 0.0010785609483718872}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.3439752459526062, "loss_scale": 4.0, "retries": 0, "seconds": 130.8, "max_memory_gb": 16.47, "ce": 0.03229418510454707, "rank": 0.0695645809173584}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.6253312230110168, "loss_scale": 4.0, "retries": 0, "seconds": 131.9, "max_memory_gb": 16.47, "ce": 0.2114641066873446, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.7975348234176636, "loss_scale": 4.0, "retries": 0, "seconds": 133.0, "max_memory_gb": 16.47, "ce": 0.12377377814118518, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 0.544745147228241, "loss_scale": 4.0, "retries": 0, "seconds": 134.1, "max_memory_gb": 16.47, "ce": 0.2106686784318299, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.2850057780742645, "loss_scale": 4.0, "retries": 0, "seconds": 135.2, "max_memory_gb": 16.47, "ce": 0.046230538820964284, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 1.779310703277588, "loss_scale": 4.0, "retries": 0, "seconds": 136.4, "max_memory_gb": 16.47, "ce": 0.36983632465125993, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.33932778239250183, "loss_scale": 4.0, "retries": 0, "seconds": 137.5, "max_memory_gb": 16.47, "ce": 0.0822479846101487, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.5063629150390625, "loss_scale": 4.0, "retries": 0, "seconds": 138.6, "max_memory_gb": 16.47, "ce": 0.0916971905680839, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.8335936665534973, "loss_scale": 4.0, "retries": 0, "seconds": 139.7, "max_memory_gb": 16.47, "ce": 0.16520675166975707, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.3107722997665405, "loss_scale": 4.0, "retries": 0, "seconds": 140.9, "max_memory_gb": 16.47, "ce": 0.5719972184742801, "rank": 0.14775025844573975}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.388380229473114, "loss_scale": 4.0, "retries": 0, "seconds": 141.9, "max_memory_gb": 16.47, "ce": 0.20755541452672333, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 1.306437611579895, "loss_scale": 4.0, "retries": 0, "seconds": 143.1, "max_memory_gb": 16.47, "ce": 0.5754807913908735, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.17542123794555664, "loss_scale": 4.0, "retries": 0, "seconds": 144.2, "max_memory_gb": 16.47, "ce": 0.015437188067153329, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 1.1867859363555908, "loss_scale": 4.0, "retries": 0, "seconds": 145.3, "max_memory_gb": 16.47, "ce": 0.2730775853706291, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 1.2045732736587524, "loss_scale": 4.0, "retries": 0, "seconds": 146.4, "max_memory_gb": 16.47, "ce": 0.08507163314061472, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 1.2311270236968994, "loss_scale": 4.0, "retries": 0, "seconds": 147.5, "max_memory_gb": 16.47, "ce": 0.26927708511357196, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 0.6685683131217957, "loss_scale": 4.0, "retries": 0, "seconds": 148.6, "max_memory_gb": 16.47, "ce": 0.10419778566574678, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 2.0711803436279297, "loss_scale": 4.0, "retries": 0, "seconds": 149.7, "max_memory_gb": 16.47, "ce": 0.1616114312200807, "rank": 0.06833894550800323}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 1.0771619081497192, "loss_scale": 4.0, "retries": 0, "seconds": 150.8, "max_memory_gb": 16.47, "ce": 0.34844099488691427, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.6553912162780762, "loss_scale": 4.0, "retries": 0, "seconds": 152.0, "max_memory_gb": 16.47, "ce": 0.18641569580540818, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 0.35698312520980835, "loss_scale": 4.0, "retries": 0, "seconds": 153.1, "max_memory_gb": 16.47, "ce": 0.037046230339910835, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 0.9327713251113892, "loss_scale": 4.0, "retries": 0, "seconds": 154.1, "max_memory_gb": 16.47, "ce": 0.15512139527709223, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.27888792753219604, "loss_scale": 4.0, "retries": 0, "seconds": 155.3, "max_memory_gb": 16.47, "ce": 0.02866761150653474, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 1.3888413906097412, "loss_scale": 4.0, "retries": 0, "seconds": 156.4, "max_memory_gb": 16.47, "ce": 0.1775853431900032, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.4672420620918274, "loss_scale": 4.0, "retries": 0, "seconds": 157.6, "max_memory_gb": 16.47, "ce": 0.11615869497472886, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.20112116634845734, "loss_scale": 4.0, "retries": 0, "seconds": 158.7, "max_memory_gb": 16.47, "ce": 0.019717589420906734, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.19749265909194946, "loss_scale": 4.0, "retries": 0, "seconds": 159.9, "max_memory_gb": 16.47, "ce": 0.018817738004145212, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.7079147696495056, "loss_scale": 4.0, "retries": 0, "seconds": 161.0, "max_memory_gb": 16.47, "ce": 0.10090806575317401, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 1.557098627090454, "loss_scale": 4.0, "retries": 0, "seconds": 162.1, "max_memory_gb": 16.47, "ce": 0.30950082360504894, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.9552809596061707, "loss_scale": 4.0, "retries": 0, "seconds": 163.2, "max_memory_gb": 16.47, "ce": 0.13984733380493708, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.5325322151184082, "loss_scale": 4.0, "retries": 0, "seconds": 164.3, "max_memory_gb": 16.47, "ce": 0.02643648046432645, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.7228319644927979, "loss_scale": 4.0, "retries": 0, "seconds": 165.5, "max_memory_gb": 16.47, "ce": 0.05363310057146009, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 1.5830427408218384, "loss_scale": 4.0, "retries": 0, "seconds": 166.6, "max_memory_gb": 16.47, "ce": 0.4455763533769641, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.2514544725418091, "loss_scale": 4.0, "retries": 0, "seconds": 167.7, "max_memory_gb": 16.47, "ce": 0.03833604387182277, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.2860412299633026, "loss_scale": 4.0, "retries": 0, "seconds": 168.8, "max_memory_gb": 16.47, "ce": 0.035966094728792086, "rank": 0.1259242445230484}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.3975849449634552, "loss_scale": 4.0, "retries": 0, "seconds": 170.0, "max_memory_gb": 16.47, "ce": 0.06837291046394967, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.34855103492736816, "loss_scale": 4.0, "retries": 0, "seconds": 171.1, "max_memory_gb": 16.47, "ce": 0.050007321719022, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 0.77161705493927, "loss_scale": 4.0, "retries": 0, "seconds": 172.2, "max_memory_gb": 16.47, "ce": 0.09025262338400353, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.9732603430747986, "loss_scale": 4.0, "retries": 0, "seconds": 173.4, "max_memory_gb": 16.47, "ce": 0.0812042463257967, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 1.3983192443847656, "loss_scale": 4.0, "retries": 0, "seconds": 174.5, "max_memory_gb": 16.47, "ce": 0.17796078929677606, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 2.6553149223327637, "loss_scale": 4.0, "retries": 0, "seconds": 175.6, "max_memory_gb": 16.47, "ce": 0.15237412783608306, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.33447033166885376, "loss_scale": 4.0, "retries": 0, "seconds": 176.7, "max_memory_gb": 16.47, "ce": 0.035799507015326526, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.5152310729026794, "loss_scale": 4.0, "retries": 0, "seconds": 177.8, "max_memory_gb": 16.47, "ce": 0.0688240767485695, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 1.8496010303497314, "loss_scale": 4.0, "retries": 0, "seconds": 179.0, "max_memory_gb": 16.47, "ce": 0.28227768921351526, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.3468017876148224, "loss_scale": 4.0, "retries": 0, "seconds": 180.1, "max_memory_gb": 16.47, "ce": 0.054187047557206824, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 0.6269118785858154, "loss_scale": 4.0, "retries": 0, "seconds": 181.2, "max_memory_gb": 16.47, "ce": 0.13319567577855196, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 1.1282122135162354, "loss_scale": 4.0, "retries": 0, "seconds": 182.3, "max_memory_gb": 16.47, "ce": 0.13602144627657253, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.6078510284423828, "loss_scale": 4.0, "retries": 0, "seconds": 183.4, "max_memory_gb": 16.47, "ce": 0.1251141565735452, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 1.5528593063354492, "loss_scale": 4.0, "retries": 0, "seconds": 184.5, "max_memory_gb": 16.47, "ce": 0.1725542359345127, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.4919430613517761, "loss_scale": 4.0, "retries": 0, "seconds": 185.7, "max_memory_gb": 16.47, "ce": 0.09749445135821588, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.18611779808998108, "loss_scale": 4.0, "retries": 0, "seconds": 186.8, "max_memory_gb": 16.47, "ce": 0.022491285475553013, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 1.115303635597229, "loss_scale": 4.0, "retries": 0, "seconds": 187.9, "max_memory_gb": 16.47, "ce": 0.23022943930118345, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.6244511604309082, "loss_scale": 4.0, "retries": 0, "seconds": 189.1, "max_memory_gb": 16.47, "ce": 0.05945959925156785, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.20177966356277466, "loss_scale": 4.0, "retries": 0, "seconds": 190.2, "max_memory_gb": 16.47, "ce": 0.018802164748194627, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 1.228642225265503, "loss_scale": 4.0, "retries": 0, "seconds": 191.3, "max_memory_gb": 16.47, "ce": 0.30387939731008373, "rank": 0.02485613524913788}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.15516115725040436, "loss_scale": 4.0, "retries": 0, "seconds": 192.5, "max_memory_gb": 16.47, "ce": 0.015821544599020854, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.7074139714241028, "loss_scale": 4.0, "retries": 0, "seconds": 193.7, "max_memory_gb": 16.47, "ce": 0.09003754995501367, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.5178984999656677, "loss_scale": 4.0, "retries": 0, "seconds": 195.0, "max_memory_gb": 16.47, "ce": 0.03973349944863003, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.5363967418670654, "loss_scale": 4.0, "retries": 0, "seconds": 196.2, "max_memory_gb": 16.47, "ce": 0.04360662661201786, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.9476203918457031, "loss_scale": 4.0, "retries": 0, "seconds": 197.4, "max_memory_gb": 16.47, "ce": 0.056181842173828045, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 1.5198495388031006, "loss_scale": 4.0, "retries": 0, "seconds": 198.6, "max_memory_gb": 16.47, "ce": 0.3588478302117437, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.23689164221286774, "loss_scale": 4.0, "retries": 0, "seconds": 199.7, "max_memory_gb": 16.47, "ce": 0.015487012729863636, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 1.3912217617034912, "loss_scale": 4.0, "retries": 0, "seconds": 200.8, "max_memory_gb": 16.47, "ce": 0.27249953761929646, "rank": 0.1250317543745041}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 1.0345325469970703, "loss_scale": 4.0, "retries": 0, "seconds": 201.9, "max_memory_gb": 16.47, "ce": 0.18893296233727597, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 0.904940128326416, "loss_scale": 4.0, "retries": 0, "seconds": 203.0, "max_memory_gb": 16.47, "ce": 0.4732677418323874, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.9944871068000793, "loss_scale": 4.0, "retries": 0, "seconds": 204.1, "max_memory_gb": 16.47, "ce": 0.05835513208876364, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.5636237859725952, "loss_scale": 4.0, "retries": 0, "seconds": 205.3, "max_memory_gb": 16.47, "ce": 0.04672438351553865, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 0.567101240158081, "loss_scale": 4.0, "retries": 0, "seconds": 206.4, "max_memory_gb": 16.47, "ce": 0.09598180279135704, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.12335053831338882, "loss_scale": 4.0, "retries": 0, "seconds": 207.5, "max_memory_gb": 16.47, "ce": 0.01730666120965907, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.46027809381484985, "loss_scale": 4.0, "retries": 0, "seconds": 208.7, "max_memory_gb": 16.47, "ce": 0.0310289260596619, "rank": 0.2113511711359024}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.8926610946655273, "loss_scale": 4.0, "retries": 0, "seconds": 209.8, "max_memory_gb": 16.47, "ce": 0.09552528632048052, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 0.8736748695373535, "loss_scale": 4.0, "retries": 0, "seconds": 210.9, "max_memory_gb": 16.47, "ce": 0.16863692700644606, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 1.180155873298645, "loss_scale": 4.0, "retries": 0, "seconds": 212.1, "max_memory_gb": 16.47, "ce": 0.17748545689391904, "rank": 0.1500600278377533}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.032430648803711, "loss_scale": 4.0, "retries": 0, "seconds": 213.2, "max_memory_gb": 16.47, "ce": 0.2531946210292517, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 1.6099072694778442, "loss_scale": 4.0, "retries": 0, "seconds": 214.3, "max_memory_gb": 16.47, "ce": 0.3646751344858785, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.30085402727127075, "loss_scale": 4.0, "retries": 0, "seconds": 215.4, "max_memory_gb": 16.47, "ce": 0.033223973485291936, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 0.6243379712104797, "loss_scale": 4.0, "retries": 0, "seconds": 216.5, "max_memory_gb": 16.47, "ce": 0.10308627719496144, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.2363760471343994, "loss_scale": 4.0, "retries": 0, "seconds": 217.6, "max_memory_gb": 16.47, "ce": 0.44966652774019167, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.2820783853530884, "loss_scale": 4.0, "retries": 0, "seconds": 218.7, "max_memory_gb": 16.47, "ce": 0.015261376429407392, "rank": 0.12502455711364746}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.5177993178367615, "loss_scale": 4.0, "retries": 0, "seconds": 219.8, "max_memory_gb": 16.47, "ce": 0.06352436669112649, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.41736918687820435, "loss_scale": 4.0, "retries": 0, "seconds": 220.9, "max_memory_gb": 16.47, "ce": 0.11601673896075226, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.5571185946464539, "loss_scale": 4.0, "retries": 0, "seconds": 222.1, "max_memory_gb": 16.47, "ce": 0.046802582321106456, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.44432783126831055, "loss_scale": 4.0, "retries": 0, "seconds": 223.2, "max_memory_gb": 16.47, "ce": 0.1994148353987839, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.48132261633872986, "loss_scale": 4.0, "retries": 0, "seconds": 224.3, "max_memory_gb": 16.47, "ce": 0.0924191138765309, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 1.498591423034668, "loss_scale": 4.0, "retries": 0, "seconds": 225.5, "max_memory_gb": 16.47, "ce": 0.08379903706372716, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 0.9317373037338257, "loss_scale": 4.0, "retries": 0, "seconds": 226.6, "max_memory_gb": 16.47, "ce": 0.3587797676882474, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.4674202501773834, "loss_scale": 4.0, "retries": 0, "seconds": 227.7, "max_memory_gb": 16.47, "ce": 0.059135785544640385, "rank": 0.04021787643432617}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.17445600032806396, "loss_scale": 4.0, "retries": 0, "seconds": 228.8, "max_memory_gb": 16.47, "ce": 0.016192299724934855, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.40761950612068176, "loss_scale": 4.0, "retries": 0, "seconds": 229.9, "max_memory_gb": 16.47, "ce": 0.0459327877179021, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.35771647095680237, "loss_scale": 4.0, "retries": 0, "seconds": 231.0, "max_memory_gb": 16.47, "ce": 0.053875377372605726, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 0.6595151424407959, "loss_scale": 4.0, "retries": 0, "seconds": 232.1, "max_memory_gb": 16.47, "ce": 0.09566784778871806, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.4022265076637268, "loss_scale": 4.0, "retries": 0, "seconds": 233.3, "max_memory_gb": 16.47, "ce": 0.049924463623028714, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.22734633088111877, "loss_scale": 4.0, "retries": 0, "seconds": 234.4, "max_memory_gb": 16.47, "ce": 0.047229815914761275, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.25616079568862915, "loss_scale": 4.0, "retries": 0, "seconds": 235.5, "max_memory_gb": 16.47, "ce": 0.023402352278935723, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.5313452482223511, "loss_scale": 4.0, "retries": 0, "seconds": 236.6, "max_memory_gb": 16.47, "ce": 0.13250076785334386, "rank": 0.12616384029388428}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 1.0552879571914673, "loss_scale": 4.0, "retries": 0, "seconds": 237.7, "max_memory_gb": 16.47, "ce": 0.39907155432592845, "rank": 0.211464524269104}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.5967330932617188, "loss_scale": 4.0, "retries": 0, "seconds": 238.9, "max_memory_gb": 16.47, "ce": 0.14313358271465404, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 0.9684703946113586, "loss_scale": 4.0, "retries": 0, "seconds": 240.0, "max_memory_gb": 16.47, "ce": 0.21468290593475103, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.6585131287574768, "loss_scale": 4.0, "retries": 0, "seconds": 241.1, "max_memory_gb": 16.47, "ce": 0.1976494153641397, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 3.0073089599609375, "loss_scale": 4.0, "retries": 0, "seconds": 242.2, "max_memory_gb": 16.47, "ce": 0.3605282520657056, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 1.2880847454071045, "loss_scale": 4.0, "retries": 0, "seconds": 243.4, "max_memory_gb": 16.47, "ce": 0.4855814398470102, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.8678447604179382, "loss_scale": 4.0, "retries": 0, "seconds": 244.5, "max_memory_gb": 16.47, "ce": 0.06098696395565639, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.08010831475257874, "loss_scale": 4.0, "retries": 0, "seconds": 245.6, "max_memory_gb": 16.47, "ce": 0.007874903647461906, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 0.7185176610946655, "loss_scale": 4.0, "retries": 0, "seconds": 246.7, "max_memory_gb": 16.47, "ce": 0.289093883086025, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.3001505136489868, "loss_scale": 4.0, "retries": 0, "seconds": 247.8, "max_memory_gb": 16.47, "ce": 0.03177441160005401, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.8332198858261108, "loss_scale": 4.0, "retries": 0, "seconds": 249.0, "max_memory_gb": 16.47, "ce": 0.05167115412768908, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.6888043284416199, "loss_scale": 4.0, "retries": 0, "seconds": 250.1, "max_memory_gb": 16.47, "ce": 0.07856762774463277, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.5132201910018921, "loss_scale": 4.0, "retries": 0, "seconds": 251.2, "max_memory_gb": 16.47, "ce": 0.045233361302962294, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 0.38406315445899963, "loss_scale": 4.0, "retries": 0, "seconds": 252.3, "max_memory_gb": 16.47, "ce": 0.05465775750053581, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.4228702783584595, "loss_scale": 4.0, "retries": 0, "seconds": 253.4, "max_memory_gb": 16.47, "ce": 0.060873661903315224, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.36609160900115967, "loss_scale": 4.0, "retries": 0, "seconds": 254.6, "max_memory_gb": 16.47, "ce": 0.026316577961551957, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.7694076299667358, "loss_scale": 4.0, "retries": 0, "seconds": 255.7, "max_memory_gb": 16.47, "ce": 0.10842212364150328, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.655116617679596, "loss_scale": 4.0, "retries": 0, "seconds": 256.8, "max_memory_gb": 16.47, "ce": 0.09241615953214932, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.40513965487480164, "loss_scale": 4.0, "retries": 0, "seconds": 257.9, "max_memory_gb": 16.47, "ce": 0.03252592947683297, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.3871251046657562, "loss_scale": 4.0, "retries": 0, "seconds": 259.0, "max_memory_gb": 16.47, "ce": 0.07046998459554743, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 1.0038515329360962, "loss_scale": 4.0, "retries": 0, "seconds": 260.1, "max_memory_gb": 16.47, "ce": 0.3100561512328568, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.6405134201049805, "loss_scale": 4.0, "retries": 0, "seconds": 261.2, "max_memory_gb": 16.47, "ce": 0.06083733921695966, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.9725005030632019, "loss_scale": 4.0, "retries": 0, "seconds": 262.3, "max_memory_gb": 16.47, "ce": 0.06815970124443993, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.42570826411247253, "loss_scale": 4.0, "retries": 0, "seconds": 263.4, "max_memory_gb": 16.47, "ce": 0.0617963120166678, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.49502629041671753, "loss_scale": 4.0, "retries": 0, "seconds": 264.5, "max_memory_gb": 16.47, "ce": 0.05567442985193338, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 0.8329427242279053, "loss_scale": 4.0, "retries": 0, "seconds": 265.6, "max_memory_gb": 16.47, "ce": 0.16301850847958121, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.17644870281219482, "loss_scale": 4.0, "retries": 0, "seconds": 266.7, "max_memory_gb": 16.47, "ce": 0.028634108661208302, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.41459178924560547, "loss_scale": 4.0, "retries": 0, "seconds": 267.8, "max_memory_gb": 16.47, "ce": 0.09887482569320127, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.3981791138648987, "loss_scale": 4.0, "retries": 0, "seconds": 269.0, "max_memory_gb": 16.47, "ce": 0.0738258586898155, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.6915767788887024, "loss_scale": 4.0, "retries": 0, "seconds": 270.1, "max_memory_gb": 16.47, "ce": 0.17878472372103715, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 1.4398187398910522, "loss_scale": 4.0, "retries": 0, "seconds": 271.2, "max_memory_gb": 16.47, "ce": 0.09749470687529538, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.6550721526145935, "loss_scale": 4.0, "retries": 0, "seconds": 272.3, "max_memory_gb": 16.47, "ce": 0.163861174689373, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.130872905254364, "loss_scale": 4.0, "retries": 0, "seconds": 273.4, "max_memory_gb": 16.47, "ce": 0.012063124202541076, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.18328535556793213, "loss_scale": 4.0, "retries": 0, "seconds": 274.6, "max_memory_gb": 16.47, "ce": 0.020158613719104324, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.8864163756370544, "loss_scale": 4.0, "retries": 0, "seconds": 275.7, "max_memory_gb": 16.47, "ce": 0.07616565118951257, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 0.8956190943717957, "loss_scale": 4.0, "retries": 0, "seconds": 276.8, "max_memory_gb": 16.47, "ce": 0.29864724386425223, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.11072275042533875, "loss_scale": 4.0, "retries": 0, "seconds": 277.9, "max_memory_gb": 16.47, "ce": 0.009990259142796276, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.901702344417572, "loss_scale": 4.0, "retries": 0, "seconds": 279.1, "max_memory_gb": 16.47, "ce": 0.10896813331601152, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.48657265305519104, "loss_scale": 4.0, "retries": 0, "seconds": 280.1, "max_memory_gb": 16.47, "ce": 0.03445332067713025, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.48990482091903687, "loss_scale": 4.0, "retries": 0, "seconds": 281.3, "max_memory_gb": 16.47, "ce": 0.04633619330707006, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.3585844039916992, "loss_scale": 4.0, "retries": 0, "seconds": 282.4, "max_memory_gb": 16.47, "ce": 0.050929299686686136, "rank": 0.0}

runs/single_llava/context/L02_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "context",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/llava-1.5-7b-hf",
+  "backbone": "llava",
+  "train": "/data/OPSD/assets/v3/context.jsonl",
+  "items": 1000,
+  "steps": 500,
+  "epochs": 2,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 2e-05,
+  "scheduler": "constant",
+  "warmup_steps": 0,
+  "final_fraction": 1.0,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 2,
+    "rank": 16,
+    "trainable_parameters": 1249280
+  }
+}

runs/single_llava/context/L06_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/llava-1.5-7b-hf
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/llava-1.5-7b-hf
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/single_llava/context/L06_s13/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/llava-1.5-7b-hf",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.6.mlp.gate_proj",
+    "model.language_model.layers.6.self_attn.q_proj",
+    "model.language_model.layers.6.self_attn.o_proj",
+    "model.language_model.layers.6.mlp.down_proj",
+    "model.language_model.layers.6.self_attn.v_proj",
+    "model.language_model.layers.6.mlp.up_proj",
+    "model.language_model.layers.6.self_attn.k_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/single_llava/context/L06_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "context",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/llava-1.5-7b-hf",
+  "backbone": "llava",
+  "train": "/data/OPSD/assets/v3/context.jsonl",
+  "items": 1000,
+  "steps": 500,
+  "epochs": 2,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 2e-05,
+  "scheduler": "constant",
+  "warmup_steps": 0,
+  "final_fraction": 1.0,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 6,
+    "rank": 16,
+    "trainable_parameters": 1249280
+  }
+}

runs/single_llava/context/L10_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/llava-1.5-7b-hf
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/llava-1.5-7b-hf
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/single_llava/context/L10_s13/adapter/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/llava-1.5-7b-hf",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.10.self_attn.o_proj",
+    "model.language_model.layers.10.mlp.up_proj",
+    "model.language_model.layers.10.self_attn.v_proj",
+    "model.language_model.layers.10.self_attn.q_proj",
+    "model.language_model.layers.10.mlp.gate_proj",
+    "model.language_model.layers.10.self_attn.k_proj",
+    "model.language_model.layers.10.mlp.down_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/single_llava/context/L10_s13/config.json

@@ -0,0 +1,24 @@
+{
+  "behavior": "context",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/llava-1.5-7b-hf",
+  "backbone": "llava",
+  "train": "/data/OPSD/assets/v3/context.jsonl",
+  "items": 1000,
+  "steps": 500,
+  "epochs": 2,
+  "steps_per_epoch": 250,
+  "groups_per_step": 4,
+  "lr": 2e-05,
+  "scheduler": "constant",
+  "warmup_steps": 0,
+  "final_fraction": 1.0,
+  "weight_decay": 0.0,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "adapter": {
+    "layer": 10,
+    "rank": 16,
+    "trainable_parameters": 1249280
+  }
+}