Add files using upload-large-folder tool

runs/fixed_internvl/context/Shallow8_s42/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/context/Shallow8_s42/adapter/adapter_config.json

@@ -0,0 +1,213 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.0.mlp.down_proj": 144,
+    "model.language_model.layers.0.mlp.gate_proj": 144,
+    "model.language_model.layers.0.mlp.up_proj": 144,
+    "model.language_model.layers.0.self_attn.k_proj": 144,
+    "model.language_model.layers.0.self_attn.o_proj": 144,
+    "model.language_model.layers.0.self_attn.q_proj": 144,
+    "model.language_model.layers.0.self_attn.v_proj": 144,
+    "model.language_model.layers.1.mlp.down_proj": 144,
+    "model.language_model.layers.1.mlp.gate_proj": 144,
+    "model.language_model.layers.1.mlp.up_proj": 144,
+    "model.language_model.layers.1.self_attn.k_proj": 144,
+    "model.language_model.layers.1.self_attn.o_proj": 144,
+    "model.language_model.layers.1.self_attn.q_proj": 144,
+    "model.language_model.layers.1.self_attn.v_proj": 144,
+    "model.language_model.layers.2.mlp.down_proj": 144,
+    "model.language_model.layers.2.mlp.gate_proj": 144,
+    "model.language_model.layers.2.mlp.up_proj": 144,
+    "model.language_model.layers.2.self_attn.k_proj": 144,
+    "model.language_model.layers.2.self_attn.o_proj": 144,
+    "model.language_model.layers.2.self_attn.q_proj": 144,
+    "model.language_model.layers.2.self_attn.v_proj": 144,
+    "model.language_model.layers.3.mlp.down_proj": 144,
+    "model.language_model.layers.3.mlp.gate_proj": 144,
+    "model.language_model.layers.3.mlp.up_proj": 144,
+    "model.language_model.layers.3.self_attn.k_proj": 144,
+    "model.language_model.layers.3.self_attn.o_proj": 144,
+    "model.language_model.layers.3.self_attn.q_proj": 144,
+    "model.language_model.layers.3.self_attn.v_proj": 144,
+    "model.language_model.layers.4.mlp.down_proj": 144,
+    "model.language_model.layers.4.mlp.gate_proj": 144,
+    "model.language_model.layers.4.mlp.up_proj": 144,
+    "model.language_model.layers.4.self_attn.k_proj": 144,
+    "model.language_model.layers.4.self_attn.o_proj": 144,
+    "model.language_model.layers.4.self_attn.q_proj": 144,
+    "model.language_model.layers.4.self_attn.v_proj": 144,
+    "model.language_model.layers.5.mlp.down_proj": 144,
+    "model.language_model.layers.5.mlp.gate_proj": 144,
+    "model.language_model.layers.5.mlp.up_proj": 144,
+    "model.language_model.layers.5.self_attn.k_proj": 144,
+    "model.language_model.layers.5.self_attn.o_proj": 144,
+    "model.language_model.layers.5.self_attn.q_proj": 144,
+    "model.language_model.layers.5.self_attn.v_proj": 144,
+    "model.language_model.layers.6.mlp.down_proj": 144,
+    "model.language_model.layers.6.mlp.gate_proj": 144,
+    "model.language_model.layers.6.mlp.up_proj": 144,
+    "model.language_model.layers.6.self_attn.k_proj": 144,
+    "model.language_model.layers.6.self_attn.o_proj": 144,
+    "model.language_model.layers.6.self_attn.q_proj": 144,
+    "model.language_model.layers.6.self_attn.v_proj": 144,
+    "model.language_model.layers.7.mlp.down_proj": 144,
+    "model.language_model.layers.7.mlp.gate_proj": 144,
+    "model.language_model.layers.7.mlp.up_proj": 144,
+    "model.language_model.layers.7.self_attn.k_proj": 144,
+    "model.language_model.layers.7.self_attn.o_proj": 144,
+    "model.language_model.layers.7.self_attn.q_proj": 144,
+    "model.language_model.layers.7.self_attn.v_proj": 144
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.0.mlp.down_proj": 72,
+    "model.language_model.layers.0.mlp.gate_proj": 72,
+    "model.language_model.layers.0.mlp.up_proj": 72,
+    "model.language_model.layers.0.self_attn.k_proj": 72,
+    "model.language_model.layers.0.self_attn.o_proj": 72,
+    "model.language_model.layers.0.self_attn.q_proj": 72,
+    "model.language_model.layers.0.self_attn.v_proj": 72,
+    "model.language_model.layers.1.mlp.down_proj": 72,
+    "model.language_model.layers.1.mlp.gate_proj": 72,
+    "model.language_model.layers.1.mlp.up_proj": 72,
+    "model.language_model.layers.1.self_attn.k_proj": 72,
+    "model.language_model.layers.1.self_attn.o_proj": 72,
+    "model.language_model.layers.1.self_attn.q_proj": 72,
+    "model.language_model.layers.1.self_attn.v_proj": 72,
+    "model.language_model.layers.2.mlp.down_proj": 72,
+    "model.language_model.layers.2.mlp.gate_proj": 72,
+    "model.language_model.layers.2.mlp.up_proj": 72,
+    "model.language_model.layers.2.self_attn.k_proj": 72,
+    "model.language_model.layers.2.self_attn.o_proj": 72,
+    "model.language_model.layers.2.self_attn.q_proj": 72,
+    "model.language_model.layers.2.self_attn.v_proj": 72,
+    "model.language_model.layers.3.mlp.down_proj": 72,
+    "model.language_model.layers.3.mlp.gate_proj": 72,
+    "model.language_model.layers.3.mlp.up_proj": 72,
+    "model.language_model.layers.3.self_attn.k_proj": 72,
+    "model.language_model.layers.3.self_attn.o_proj": 72,
+    "model.language_model.layers.3.self_attn.q_proj": 72,
+    "model.language_model.layers.3.self_attn.v_proj": 72,
+    "model.language_model.layers.4.mlp.down_proj": 72,
+    "model.language_model.layers.4.mlp.gate_proj": 72,
+    "model.language_model.layers.4.mlp.up_proj": 72,
+    "model.language_model.layers.4.self_attn.k_proj": 72,
+    "model.language_model.layers.4.self_attn.o_proj": 72,
+    "model.language_model.layers.4.self_attn.q_proj": 72,
+    "model.language_model.layers.4.self_attn.v_proj": 72,
+    "model.language_model.layers.5.mlp.down_proj": 72,
+    "model.language_model.layers.5.mlp.gate_proj": 72,
+    "model.language_model.layers.5.mlp.up_proj": 72,
+    "model.language_model.layers.5.self_attn.k_proj": 72,
+    "model.language_model.layers.5.self_attn.o_proj": 72,
+    "model.language_model.layers.5.self_attn.q_proj": 72,
+    "model.language_model.layers.5.self_attn.v_proj": 72,
+    "model.language_model.layers.6.mlp.down_proj": 72,
+    "model.language_model.layers.6.mlp.gate_proj": 72,
+    "model.language_model.layers.6.mlp.up_proj": 72,
+    "model.language_model.layers.6.self_attn.k_proj": 72,
+    "model.language_model.layers.6.self_attn.o_proj": 72,
+    "model.language_model.layers.6.self_attn.q_proj": 72,
+    "model.language_model.layers.6.self_attn.v_proj": 72,
+    "model.language_model.layers.7.mlp.down_proj": 72,
+    "model.language_model.layers.7.mlp.gate_proj": 72,
+    "model.language_model.layers.7.mlp.up_proj": 72,
+    "model.language_model.layers.7.self_attn.k_proj": 72,
+    "model.language_model.layers.7.self_attn.o_proj": 72,
+    "model.language_model.layers.7.self_attn.q_proj": 72,
+    "model.language_model.layers.7.self_attn.v_proj": 72
+  },
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.5.self_attn.k_proj",
+    "model.language_model.layers.5.mlp.down_proj",
+    "model.language_model.layers.7.self_attn.v_proj",
+    "model.language_model.layers.1.self_attn.q_proj",
+    "model.language_model.layers.4.self_attn.k_proj",
+    "model.language_model.layers.6.self_attn.q_proj",
+    "model.language_model.layers.7.self_attn.k_proj",
+    "model.language_model.layers.6.self_attn.k_proj",
+    "model.language_model.layers.0.self_attn.o_proj",
+    "model.language_model.layers.1.self_attn.o_proj",
+    "model.language_model.layers.2.self_attn.k_proj",
+    "model.language_model.layers.6.self_attn.o_proj",
+    "model.language_model.layers.0.self_attn.v_proj",
+    "model.language_model.layers.4.mlp.up_proj",
+    "model.language_model.layers.3.self_attn.q_proj",
+    "model.language_model.layers.1.mlp.up_proj",
+    "model.language_model.layers.1.mlp.down_proj",
+    "model.language_model.layers.5.self_attn.q_proj",
+    "model.language_model.layers.6.self_attn.v_proj",
+    "model.language_model.layers.5.self_attn.v_proj",
+    "model.language_model.layers.7.self_attn.o_proj",
+    "model.language_model.layers.3.mlp.up_proj",
+    "model.language_model.layers.7.mlp.down_proj",
+    "model.language_model.layers.0.mlp.gate_proj",
+    "model.language_model.layers.4.self_attn.q_proj",
+    "model.language_model.layers.6.mlp.down_proj",
+    "model.language_model.layers.2.self_attn.q_proj",
+    "model.language_model.layers.3.self_attn.o_proj",
+    "model.language_model.layers.4.mlp.gate_proj",
+    "model.language_model.layers.4.self_attn.o_proj",
+    "model.language_model.layers.0.mlp.down_proj",
+    "model.language_model.layers.4.self_attn.v_proj",
+    "model.language_model.layers.3.self_attn.k_proj",
+    "model.language_model.layers.5.self_attn.o_proj",
+    "model.language_model.layers.1.mlp.gate_proj",
+    "model.language_model.layers.3.mlp.down_proj",
+    "model.language_model.layers.2.self_attn.v_proj",
+    "model.language_model.layers.0.self_attn.q_proj",
+    "model.language_model.layers.5.mlp.up_proj",
+    "model.language_model.layers.1.self_attn.v_proj",
+    "model.language_model.layers.6.mlp.up_proj",
+    "model.language_model.layers.5.mlp.gate_proj",
+    "model.language_model.layers.0.mlp.up_proj",
+    "model.language_model.layers.1.self_attn.k_proj",
+    "model.language_model.layers.3.mlp.gate_proj",
+    "model.language_model.layers.4.mlp.down_proj",
+    "model.language_model.layers.0.self_attn.k_proj",
+    "model.language_model.layers.6.mlp.gate_proj",
+    "model.language_model.layers.2.mlp.gate_proj",
+    "model.language_model.layers.2.self_attn.o_proj",
+    "model.language_model.layers.2.mlp.up_proj",
+    "model.language_model.layers.3.self_attn.v_proj",
+    "model.language_model.layers.7.self_attn.q_proj",
+    "model.language_model.layers.7.mlp.up_proj",
+    "model.language_model.layers.7.mlp.gate_proj",
+    "model.language_model.layers.2.mlp.down_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/fixed_internvl/context/Shallow8_s42/config.json

@@ -0,0 +1,43 @@
+{
+  "behavior": "context",
+  "seed": 42,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/context.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "world_size": 1,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "arm": {
+    "arm": "Shallow8",
+    "layers": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7
+    ],
+    "ranks": {
+      "0": 72,
+      "1": 72,
+      "2": 72,
+      "3": 72,
+      "4": 72,
+      "5": 72,
+      "6": 72,
+      "7": 72
+    },
+    "parameters": 43646976,
+    "budget": 43646976
+  }
+}

runs/fixed_internvl/context/Shallow8_s42/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 13.360817909240723, "loss_scale": 4.0, "retries": 0, "seconds": 2.0, "max_memory_gb": 16.79, "ce": 0.6558216065168381, "kl": 0.11560958076734096, "context": 0.7136263996362686}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 8.333952903747559, "loss_scale": 4.0, "retries": 0, "seconds": 3.1, "max_memory_gb": 17.07, "ce": 0.5917620584362169, "kl": 0.060682600241932505, "context": 0.6221033387665784}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 14.723043441772461, "loss_scale": 4.0, "retries": 0, "seconds": 4.1, "max_memory_gb": 17.08, "ce": 0.6262433074880391, "kl": 0.036605335655622184, "context": 0.6445459639653563}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 13.171707153320312, "loss_scale": 4.0, "retries": 0, "seconds": 5.1, "max_memory_gb": 17.08, "ce": 0.43537799587647896, "kl": 0.6782515849918127, "context": 0.7745037862332538}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 4.297255516052246, "loss_scale": 4.0, "retries": 0, "seconds": 6.1, "max_memory_gb": 17.08, "ce": 0.22356315766228363, "kl": 1.169863623130368, "context": 0.8084949859767221}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 7.870523929595947, "loss_scale": 4.0, "retries": 0, "seconds": 7.1, "max_memory_gb": 17.08, "ce": 0.16931212006602436, "kl": 0.324097916498431, "context": 0.3313610786572099}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 2.9887382984161377, "loss_scale": 4.0, "retries": 0, "seconds": 8.2, "max_memory_gb": 17.08, "ce": 0.17164810933172703, "kl": 0.3312494227429852, "context": 0.3372728303074837}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 2.9925179481506348, "loss_scale": 4.0, "retries": 0, "seconds": 9.2, "max_memory_gb": 17.08, "ce": 0.21266398635589212, "kl": 0.09130242344599537, "context": 0.2583151942371842}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 4.474491596221924, "loss_scale": 4.0, "retries": 0, "seconds": 10.2, "max_memory_gb": 17.08, "ce": 0.5789872417226434, "kl": 0.09850906068459153, "context": 0.628241759724915}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 11.128975868225098, "loss_scale": 4.0, "retries": 0, "seconds": 11.3, "max_memory_gb": 17.08, "ce": 0.11002099129837006, "kl": 0.3562353131128475, "context": 0.28813864645780995}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.7212154865264893, "loss_scale": 4.0, "retries": 0, "seconds": 12.2, "max_memory_gb": 17.08, "ce": 0.0026443500528330333, "kl": 0.24077180282836252, "context": 0.12303024866969281}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 7.864652633666992, "loss_scale": 4.0, "retries": 0, "seconds": 13.2, "max_memory_gb": 17.08, "ce": 0.4417824898264371, "kl": 1.1037428810959682, "context": 0.9936538923066109}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 0.7157631516456604, "loss_scale": 4.0, "retries": 0, "seconds": 14.3, "max_memory_gb": 17.08, "ce": 0.05026993766659871, "kl": 0.047651506727561355, "context": 0.07409568887669593}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 0.6710599064826965, "loss_scale": 4.0, "retries": 0, "seconds": 15.3, "max_memory_gb": 17.08, "ce": 0.025411487935343757, "kl": 0.10134041809942573, "context": 0.07608169841114432}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 6.227840900421143, "loss_scale": 4.0, "retries": 0, "seconds": 16.3, "max_memory_gb": 17.08, "ce": 0.47335879679303616, "kl": 0.016227395535679534, "context": 0.48147249198518693}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 7.855039596557617, "loss_scale": 4.0, "retries": 0, "seconds": 17.4, "max_memory_gb": 17.08, "ce": 0.29752136765932846, "kl": 0.024735589415286086, "context": 0.3098891649278812}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 1.0083720684051514, "loss_scale": 4.0, "retries": 0, "seconds": 18.3, "max_memory_gb": 17.08, "ce": 0.009607089159544557, "kl": 0.091724820289528, "context": 0.055469500104663894}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 0.7303684949874878, "loss_scale": 4.0, "retries": 0, "seconds": 19.4, "max_memory_gb": 17.08, "ce": 0.027199057396501303, "kl": 0.04552088998025283, "context": 0.04995950451120734}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 0.3213456869125366, "loss_scale": 4.0, "retries": 0, "seconds": 20.4, "max_memory_gb": 17.08, "ce": 0.00399409161082076, "kl": 0.02945993932644342, "context": 0.018724061317698215}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.4479367434978485, "loss_scale": 4.0, "retries": 0, "seconds": 21.4, "max_memory_gb": 17.08, "ce": 0.003564578015357256, "kl": 0.10335491304522293, "context": 0.05524203473760281}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 1.2764617204666138, "loss_scale": 4.0, "retries": 0, "seconds": 22.5, "max_memory_gb": 17.08, "ce": 0.003108661010628566, "kl": 0.07340076745458646, "context": 0.03980904578929767}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.6633093357086182, "loss_scale": 4.0, "retries": 0, "seconds": 23.5, "max_memory_gb": 17.08, "ce": 0.03531976592284991, "kl": 0.01797694057722765, "context": 0.04430823632810643}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 3.3674986362457275, "loss_scale": 4.0, "retries": 0, "seconds": 24.5, "max_memory_gb": 17.08, "ce": 0.1025337428873172, "kl": 0.6130956253800832, "context": 0.409081560006598}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 1.99097740650177, "loss_scale": 4.0, "retries": 0, "seconds": 25.6, "max_memory_gb": 17.08, "ce": 0.06875411088185501, "kl": 0.8060746188930352, "context": 0.47179142952518305}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 1.5490508079528809, "loss_scale": 4.0, "retries": 0, "seconds": 26.5, "max_memory_gb": 17.08, "ce": 0.12457426487526391, "kl": 0.21271509212601813, "context": 0.2309318183833966}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 1.6677800416946411, "loss_scale": 4.0, "retries": 0, "seconds": 27.5, "max_memory_gb": 17.08, "ce": 0.04411983260524721, "kl": 0.027105891355802214, "context": 0.05767277921540881}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 3.706446647644043, "loss_scale": 4.0, "retries": 0, "seconds": 28.6, "max_memory_gb": 17.08, "ce": 0.20456909920903854, "kl": 0.06839320345534361, "context": 0.23876570478023496}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 1.927957534790039, "loss_scale": 4.0, "retries": 0, "seconds": 29.6, "max_memory_gb": 17.08, "ce": 0.007455488201230764, "kl": 0.9450394971718197, "context": 0.47997523355297744}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 1.6950284242630005, "loss_scale": 4.0, "retries": 0, "seconds": 30.6, "max_memory_gb": 17.08, "ce": 0.11289158949512057, "kl": 0.04721641949436162, "context": 0.13649979923502542}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 1.9076534509658813, "loss_scale": 4.0, "retries": 0, "seconds": 31.7, "max_memory_gb": 17.08, "ce": 0.011429584497818723, "kl": 0.7192494398914278, "context": 0.37105429463554174}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 3.1511545181274414, "loss_scale": 4.0, "retries": 0, "seconds": 32.7, "max_memory_gb": 17.08, "ce": 0.6249555391987087, "kl": 0.624476895754924, "context": 0.9371939938282594}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 1.1220228672027588, "loss_scale": 4.0, "retries": 0, "seconds": 33.7, "max_memory_gb": 17.08, "ce": 0.03851207390107447, "kl": 0.017864656747406116, "context": 0.04744440271315398}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.011608117260038853, "loss_scale": 4.0, "retries": 0, "seconds": 34.8, "max_memory_gb": 17.08, "ce": 0.0005819607958983397, "kl": 0.0007189700063463533, "context": 0.0009414458254468627}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.008420020341873169, "loss_scale": 4.0, "retries": 0, "seconds": 35.7, "max_memory_gb": 17.08, "ce": 0.0003989508368249517, "kl": 0.00013291711957208463, "context": 0.00046540940456907265}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 1.2460083961486816, "loss_scale": 4.0, "retries": 0, "seconds": 36.9, "max_memory_gb": 17.08, "ce": 0.030445677799434634, "kl": 0.13036014993713252, "context": 0.09562575273957918}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 1.4093223810195923, "loss_scale": 4.0, "retries": 0, "seconds": 37.8, "max_memory_gb": 17.08, "ce": 0.13473081607844506, "kl": 0.01943348285499269, "context": 0.1444475536900427}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.6894105076789856, "loss_scale": 4.0, "retries": 0, "seconds": 38.8, "max_memory_gb": 17.08, "ce": 0.026629685344232712, "kl": 0.577821531380323, "context": 0.3155404453573283}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.6389856934547424, "loss_scale": 4.0, "retries": 0, "seconds": 39.9, "max_memory_gb": 17.08, "ce": 0.0070751209277659655, "kl": 0.21964392747031525, "context": 0.11689708265475929}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 0.8546860814094543, "loss_scale": 4.0, "retries": 0, "seconds": 40.8, "max_memory_gb": 17.08, "ce": 0.09718419919954613, "kl": 0.27993764023995027, "context": 0.23715302141499706}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.13668054342269897, "loss_scale": 4.0, "retries": 0, "seconds": 41.8, "max_memory_gb": 17.08, "ce": 0.007034500416921219, "kl": 0.004569643278045987, "context": 0.009319321581642726}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.30852964520454407, "loss_scale": 4.0, "retries": 0, "seconds": 43.0, "max_memory_gb": 17.08, "ce": 0.01204274759220425, "kl": 0.02091524316347204, "context": 0.022500368941109627}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.8840864896774292, "loss_scale": 4.0, "retries": 0, "seconds": 43.9, "max_memory_gb": 17.08, "ce": 0.020005155914986972, "kl": 0.6940483340299579, "context": 0.3670293267132365}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 6.988037586212158, "loss_scale": 4.0, "retries": 0, "seconds": 44.9, "max_memory_gb": 17.08, "ce": 0.12146661954466254, "kl": 0.19120328261487884, "context": 0.21706825296860188}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 3.812509775161743, "loss_scale": 4.0, "retries": 0, "seconds": 46.0, "max_memory_gb": 17.08, "ce": 0.5275515361354337, "kl": 0.02865177386411233, "context": 0.5418773958517704}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 5.367697238922119, "loss_scale": 4.0, "retries": 0, "seconds": 47.0, "max_memory_gb": 17.08, "ce": 0.5280047197520616, "kl": 0.04299584200634854, "context": 0.5495026558928657}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 1.7527302503585815, "loss_scale": 4.0, "retries": 0, "seconds": 48.1, "max_memory_gb": 17.08, "ce": 0.27138965085032396, "kl": 0.02077482587264967, "context": 0.281777075084392}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.1201881691813469, "loss_scale": 4.0, "retries": 0, "seconds": 49.0, "max_memory_gb": 17.08, "ce": 0.005498805083334446, "kl": 0.015493662365770433, "context": 0.013245636597275734}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 1.1813464164733887, "loss_scale": 4.0, "retries": 0, "seconds": 50.0, "max_memory_gb": 17.08, "ce": 0.09958167985314503, "kl": 0.012448208457499277, "context": 0.1058057812624611}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 1.6227105855941772, "loss_scale": 4.0, "retries": 0, "seconds": 51.1, "max_memory_gb": 17.08, "ce": 0.22349351820230368, "kl": 0.15708861332041124, "context": 0.30203780994634144}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.7935534715652466, "loss_scale": 4.0, "retries": 0, "seconds": 52.1, "max_memory_gb": 17.08, "ce": 0.026224352477584034, "kl": 0.21797279419843107, "context": 0.13521074783056974}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 2.3533272743225098, "loss_scale": 4.0, "retries": 0, "seconds": 53.0, "max_memory_gb": 17.08, "ce": 0.0659535508020781, "kl": 0.1618740108970087, "context": 0.1468905545771122}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.23347845673561096, "loss_scale": 4.0, "retries": 0, "seconds": 54.1, "max_memory_gb": 17.08, "ce": 0.013041251095273765, "kl": 0.022596157134103123, "context": 0.024339329786016606}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 0.3377764821052551, "loss_scale": 4.0, "retries": 0, "seconds": 55.1, "max_memory_gb": 17.08, "ce": 0.009291229478549212, "kl": 0.02165225089993328, "context": 0.02011735487030819}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 2.9711320400238037, "loss_scale": 4.0, "retries": 0, "seconds": 56.1, "max_memory_gb": 17.08, "ce": 0.047694957203930244, "kl": 0.12509414566738997, "context": 0.11024203000124544}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 14.141380310058594, "loss_scale": 4.0, "retries": 0, "seconds": 57.2, "max_memory_gb": 17.08, "ce": 0.1939660428906791, "kl": 0.21665135657167411, "context": 0.30229172769759316}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 1.9908124208450317, "loss_scale": 4.0, "retries": 0, "seconds": 58.1, "max_memory_gb": 17.08, "ce": 0.010176199815759901, "kl": 0.15647912208532944, "context": 0.08841575717087835}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 1.487259030342102, "loss_scale": 4.0, "retries": 0, "seconds": 59.2, "max_memory_gb": 17.08, "ce": 0.0726356774684973, "kl": 0.013737064338783966, "context": 0.079504209687002}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.7362216711044312, "loss_scale": 4.0, "retries": 0, "seconds": 60.2, "max_memory_gb": 17.08, "ce": 0.027894298429600894, "kl": 0.48012228358129505, "context": 0.2679554376518354}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.11419221013784409, "loss_scale": 4.0, "retries": 0, "seconds": 61.2, "max_memory_gb": 17.08, "ce": 0.007527314293838572, "kl": 0.0012251927942088514, "context": 0.00813991084578447}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 1.090063452720642, "loss_scale": 4.0, "retries": 0, "seconds": 62.2, "max_memory_gb": 17.08, "ce": 0.09898895531659946, "kl": 0.03759045648621395, "context": 0.11778418345784303}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.25273898243904114, "loss_scale": 4.0, "retries": 0, "seconds": 63.2, "max_memory_gb": 17.08, "ce": 0.014835931331617758, "kl": 0.09433741726388689, "context": 0.0620046385738533}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.3332825005054474, "loss_scale": 4.0, "retries": 0, "seconds": 64.2, "max_memory_gb": 17.08, "ce": 0.013820649692206644, "kl": 0.119388431085099, "context": 0.07351486332481727}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 2.330357789993286, "loss_scale": 4.0, "retries": 0, "seconds": 65.3, "max_memory_gb": 17.08, "ce": 0.1868907621537801, "kl": 0.11437679970981662, "context": 0.2440791592161986}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 1.1826115846633911, "loss_scale": 4.0, "retries": 0, "seconds": 66.3, "max_memory_gb": 17.08, "ce": 0.061950383824296296, "kl": 0.11132380124763586, "context": 0.11761228000978008}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 5.495735168457031, "loss_scale": 4.0, "retries": 0, "seconds": 67.3, "max_memory_gb": 17.08, "ce": 0.23739473056048155, "kl": 0.01625728397630155, "context": 0.2455233666114509}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 2.7780508995056152, "loss_scale": 4.0, "retries": 0, "seconds": 68.4, "max_memory_gb": 17.08, "ce": 0.04975881455902709, "kl": 0.0528156350410427, "context": 0.0761666283433442}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.2592068314552307, "loss_scale": 4.0, "retries": 0, "seconds": 69.4, "max_memory_gb": 17.08, "ce": 0.029718053818214685, "kl": 0.0020332898857304826, "context": 0.030734698884771205}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.49536120891571045, "loss_scale": 4.0, "retries": 0, "seconds": 70.5, "max_memory_gb": 17.08, "ce": 0.05887677473947406, "kl": 0.014238681847928092, "context": 0.06599611282581463}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.7579951882362366, "loss_scale": 4.0, "retries": 0, "seconds": 71.4, "max_memory_gb": 17.08, "ce": 0.05512939256732352, "kl": 0.04849705274682492, "context": 0.07937792083248496}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 4.377717018127441, "loss_scale": 4.0, "retries": 0, "seconds": 72.4, "max_memory_gb": 17.08, "ce": 0.23675186568289064, "kl": 0.15472504370154638, "context": 0.31411438950453885}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 3.931645154953003, "loss_scale": 4.0, "retries": 0, "seconds": 73.5, "max_memory_gb": 17.08, "ce": 0.6441082173259929, "kl": 0.02135294832987711, "context": 0.654784704791382}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.1263335794210434, "loss_scale": 4.0, "retries": 0, "seconds": 74.5, "max_memory_gb": 17.08, "ce": 0.01052175447694026, "kl": 0.0029106537404004484, "context": 0.011977081070654094}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 3.132357358932495, "loss_scale": 4.0, "retries": 0, "seconds": 75.5, "max_memory_gb": 17.08, "ce": 0.1875379162374884, "kl": 0.04787013362511061, "context": 0.21147297945572063}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 1.060571551322937, "loss_scale": 4.0, "retries": 0, "seconds": 76.6, "max_memory_gb": 17.08, "ce": 0.04674508888274431, "kl": 0.10321114474209026, "context": 0.09835065761581063}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 3.433879852294922, "loss_scale": 4.0, "retries": 0, "seconds": 77.6, "max_memory_gb": 17.08, "ce": 0.10734481675899588, "kl": 0.22923521765619626, "context": 0.22196242802601773}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 0.7000571489334106, "loss_scale": 4.0, "retries": 0, "seconds": 78.5, "max_memory_gb": 17.08, "ce": 0.010589034762233496, "kl": 0.1828641148749739, "context": 0.10202109394595027}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.3760168254375458, "loss_scale": 4.0, "retries": 0, "seconds": 79.7, "max_memory_gb": 17.08, "ce": 0.007601221892400645, "kl": 0.07774367180172703, "context": 0.046473055859678425}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.4650455117225647, "loss_scale": 4.0, "retries": 0, "seconds": 80.6, "max_memory_gb": 17.08, "ce": 0.01241564261727035, "kl": 0.17766391881741583, "context": 0.10124759841710329}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 2.562025308609009, "loss_scale": 4.0, "retries": 0, "seconds": 81.7, "max_memory_gb": 17.08, "ce": 0.015330440503021237, "kl": 0.3931420313929266, "context": 0.2119014562049415}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 1.7307220697402954, "loss_scale": 4.0, "retries": 0, "seconds": 82.7, "max_memory_gb": 17.08, "ce": 0.24841890994503046, "kl": 0.0359312172367936, "context": 0.2663845315873914}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 5.686016082763672, "loss_scale": 4.0, "retries": 0, "seconds": 83.7, "max_memory_gb": 17.08, "ce": 0.6078041801229119, "kl": 0.015699012375989696, "context": 0.6156536588678136}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 1.224798560142517, "loss_scale": 4.0, "retries": 0, "seconds": 84.8, "max_memory_gb": 17.08, "ce": 0.002106485379044898, "kl": 0.27487754291360034, "context": 0.13954525561712217}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 0.6768404841423035, "loss_scale": 4.0, "retries": 0, "seconds": 85.8, "max_memory_gb": 17.08, "ce": 0.09422996372450143, "kl": 0.06039774251985364, "context": 0.12442883499898016}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.815053403377533, "loss_scale": 4.0, "retries": 0, "seconds": 86.8, "max_memory_gb": 17.08, "ce": 0.07836350245634094, "kl": 0.16895844810642302, "context": 0.16284272639313713}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.5665488243103027, "loss_scale": 4.0, "retries": 0, "seconds": 87.9, "max_memory_gb": 17.08, "ce": 0.02314212882265565, "kl": 0.016302815463859588, "context": 0.031293535575969145}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 1.1818392276763916, "loss_scale": 4.0, "retries": 0, "seconds": 88.9, "max_memory_gb": 17.08, "ce": 0.024256787495687604, "kl": 0.02525893982965499, "context": 0.03688625758513808}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.49879005551338196, "loss_scale": 4.0, "retries": 0, "seconds": 89.9, "max_memory_gb": 17.08, "ce": 0.05333898910248536, "kl": 0.0221516975434497, "context": 0.06441483972230344}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 1.2962666749954224, "loss_scale": 4.0, "retries": 0, "seconds": 91.0, "max_memory_gb": 17.08, "ce": 0.04615009874396492, "kl": 0.0026709084304457065, "context": 0.04748555249534547}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.1317037045955658, "loss_scale": 4.0, "retries": 0, "seconds": 92.0, "max_memory_gb": 17.08, "ce": 0.007021045916189905, "kl": 0.014276526655521593, "context": 0.014159309153910726}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.05868742987513542, "loss_scale": 4.0, "retries": 0, "seconds": 93.1, "max_memory_gb": 17.08, "ce": 0.002565259455877822, "kl": 0.007401629794912878, "context": 0.006266074546147138}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 0.9458936452865601, "loss_scale": 4.0, "retries": 0, "seconds": 94.1, "max_memory_gb": 17.08, "ce": 0.02160298881062772, "kl": 0.11831968883052468, "context": 0.08076283347327262}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 0.3783479332923889, "loss_scale": 4.0, "retries": 0, "seconds": 95.0, "max_memory_gb": 17.08, "ce": 0.041239811573177576, "kl": 0.03324027365306392, "context": 0.057859948836266994}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 4.505658149719238, "loss_scale": 4.0, "retries": 0, "seconds": 96.1, "max_memory_gb": 17.08, "ce": 0.2562470241682604, "kl": 0.461866163299419, "context": 0.4871801028493792}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 2.505525588989258, "loss_scale": 4.0, "retries": 0, "seconds": 97.1, "max_memory_gb": 17.08, "ce": 0.11343423691869248, "kl": 0.4048475839663297, "context": 0.31585803162306547}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 0.627509593963623, "loss_scale": 4.0, "retries": 0, "seconds": 98.1, "max_memory_gb": 17.08, "ce": 0.037338850321248174, "kl": 0.09934292593970895, "context": 0.08701031282544136}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 0.5180256366729736, "loss_scale": 4.0, "retries": 0, "seconds": 99.2, "max_memory_gb": 17.08, "ce": 0.012455579999368638, "kl": 0.2570430088599096, "context": 0.14097708323970437}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.17473721504211426, "loss_scale": 4.0, "retries": 0, "seconds": 100.2, "max_memory_gb": 17.08, "ce": 0.0031999897182686254, "kl": 0.023299293650779873, "context": 0.014849636354483664}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.5009772181510925, "loss_scale": 4.0, "retries": 0, "seconds": 101.1, "max_memory_gb": 17.08, "ce": 0.016628147415758576, "kl": 0.041565352970792446, "context": 0.037410823788377456}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.7383313179016113, "loss_scale": 4.0, "retries": 0, "seconds": 102.2, "max_memory_gb": 17.08, "ce": 0.008393150026677176, "kl": 0.36962779978603066, "context": 0.19320704508572817}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.11799546331167221, "loss_scale": 4.0, "retries": 0, "seconds": 103.2, "max_memory_gb": 17.08, "ce": 0.00742333012749441, "kl": 0.006363812870404217, "context": 0.010605236631818116}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 1.2068864107131958, "loss_scale": 4.0, "retries": 0, "seconds": 104.3, "max_memory_gb": 17.08, "ce": 0.17064523897897743, "kl": 0.05617588129643991, "context": 0.1987331833006465}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.1965869665145874, "loss_scale": 4.0, "retries": 0, "seconds": 105.3, "max_memory_gb": 17.08, "ce": 0.007529590868216474, "kl": 0.03205868742315943, "context": 0.02355893458116043}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 2.9828379154205322, "loss_scale": 4.0, "retries": 0, "seconds": 106.3, "max_memory_gb": 17.08, "ce": 0.29756492120213807, "kl": 0.03343707502790494, "context": 0.31428345874883235}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 1.984705924987793, "loss_scale": 4.0, "retries": 0, "seconds": 107.4, "max_memory_gb": 17.08, "ce": 0.12607313902117312, "kl": 0.007092843872669619, "context": 0.1296195572358556}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 13.800069808959961, "loss_scale": 4.0, "retries": 0, "seconds": 108.4, "max_memory_gb": 17.08, "ce": 0.64317301736628, "kl": 0.2309104409232532, "context": 0.758628258085082}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 6.46587610244751, "loss_scale": 4.0, "retries": 0, "seconds": 109.4, "max_memory_gb": 17.08, "ce": 0.22242543796892278, "kl": 0.021425561981459396, "context": 0.2331382189586293}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 3.576908588409424, "loss_scale": 4.0, "retries": 0, "seconds": 110.5, "max_memory_gb": 17.08, "ce": 0.3472674677032046, "kl": 0.13390207532211207, "context": 0.41421851934865117}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 3.3160951137542725, "loss_scale": 4.0, "retries": 0, "seconds": 111.5, "max_memory_gb": 17.08, "ce": 0.30135184107348323, "kl": 0.01048857944260817, "context": 0.3065961269894615}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 1.7926888465881348, "loss_scale": 4.0, "retries": 0, "seconds": 112.4, "max_memory_gb": 17.08, "ce": 0.04071430419571698, "kl": 0.39024516311474144, "context": 0.23583688959479332}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 1.1376012563705444, "loss_scale": 4.0, "retries": 0, "seconds": 113.5, "max_memory_gb": 17.08, "ce": 0.028241229301784188, "kl": 0.0054142118751769885, "context": 0.03094833536306396}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.1696777641773224, "loss_scale": 4.0, "retries": 0, "seconds": 114.5, "max_memory_gb": 17.08, "ce": 0.0007242859537655022, "kl": 0.010395220519967552, "context": 0.005921896190557163}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 3.5460867881774902, "loss_scale": 4.0, "retries": 0, "seconds": 115.6, "max_memory_gb": 17.08, "ce": 0.10982869155122899, "kl": 0.1863406718475744, "context": 0.20299903495470062}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 2.1492438316345215, "loss_scale": 4.0, "retries": 0, "seconds": 116.6, "max_memory_gb": 17.08, "ce": 0.5381698430282995, "kl": 0.07611483642540406, "context": 0.5762272687861696}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.14186514914035797, "loss_scale": 4.0, "retries": 0, "seconds": 117.6, "max_memory_gb": 17.08, "ce": 0.007271054259035736, "kl": 0.0008432983086095192, "context": 0.007692703453358263}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 2.312605381011963, "loss_scale": 4.0, "retries": 0, "seconds": 118.7, "max_memory_gb": 17.08, "ce": 0.10432126611704007, "kl": 0.008820934483082965, "context": 0.10873173153959215}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.5013370513916016, "loss_scale": 4.0, "retries": 0, "seconds": 119.7, "max_memory_gb": 17.08, "ce": 0.034816639414202655, "kl": 0.0026098409070982598, "context": 0.036121560107858386}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 3.7077083587646484, "loss_scale": 4.0, "retries": 0, "seconds": 120.7, "max_memory_gb": 17.08, "ce": 0.49615364720921207, "kl": 0.024876826428226195, "context": 0.5085920334313414}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 3.0064918994903564, "loss_scale": 4.0, "retries": 0, "seconds": 121.8, "max_memory_gb": 17.08, "ce": 0.06616645796020748, "kl": 0.8941936015180545, "context": 0.5132632715249201}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.3759630620479584, "loss_scale": 4.0, "retries": 0, "seconds": 122.7, "max_memory_gb": 17.08, "ce": 0.033720210840328946, "kl": 0.03209546853986467, "context": 0.049767946318752365}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 0.4897310733795166, "loss_scale": 4.0, "retries": 0, "seconds": 123.7, "max_memory_gb": 17.08, "ce": 0.015768399916851195, "kl": 0.02025671075369928, "context": 0.0258967549452791}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 3.155677318572998, "loss_scale": 4.0, "retries": 0, "seconds": 124.8, "max_memory_gb": 17.08, "ce": 0.07697086716507329, "kl": 0.010379338936218119, "context": 0.08216053826618008}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.4443150758743286, "loss_scale": 4.0, "retries": 0, "seconds": 125.8, "max_memory_gb": 17.08, "ce": 0.004093464245670475, "kl": 0.09070446496480145, "context": 0.049445697222836316}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.15124960243701935, "loss_scale": 4.0, "retries": 0, "seconds": 127.0, "max_memory_gb": 17.08, "ce": 0.012992582902370486, "kl": 0.001319009055805509, "context": 0.013652087392983958}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.0678861141204834, "loss_scale": 4.0, "retries": 0, "seconds": 128.0, "max_memory_gb": 17.08, "ce": 0.013746911754424218, "kl": 0.4809785725964275, "context": 0.2542362036292616}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 1.8718968629837036, "loss_scale": 4.0, "retries": 0, "seconds": 129.1, "max_memory_gb": 17.08, "ce": 0.012153045783634298, "kl": 0.05569010187173262, "context": 0.039998096239287406}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 4.108522891998291, "loss_scale": 4.0, "retries": 0, "seconds": 130.2, "max_memory_gb": 17.08, "ce": 0.6697753269836539, "kl": 0.23170307022417092, "context": 0.7856268616451416}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 1.6641535758972168, "loss_scale": 4.0, "retries": 0, "seconds": 131.2, "max_memory_gb": 17.08, "ce": 0.007342210643400904, "kl": 0.18547721586764965, "context": 0.10008081630803645}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 1.5737687349319458, "loss_scale": 4.0, "retries": 0, "seconds": 132.2, "max_memory_gb": 17.08, "ce": 0.027438933393568732, "kl": 0.057577173596655484, "context": 0.05622752066119574}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 5.704034805297852, "loss_scale": 4.0, "retries": 0, "seconds": 133.3, "max_memory_gb": 17.08, "ce": 0.40313919985783286, "kl": 0.25823229208344856, "context": 0.5322553458681796}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 2.371879816055298, "loss_scale": 4.0, "retries": 0, "seconds": 134.1, "max_memory_gb": 17.08, "ce": 0.03581958713766653, "kl": 0.06004378432407975, "context": 0.06584147975081578}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 1.3916285037994385, "loss_scale": 4.0, "retries": 0, "seconds": 135.0, "max_memory_gb": 17.08, "ce": 0.1734769338509068, "kl": 0.05355975573183969, "context": 0.20025680679827929}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 0.025978688150644302, "loss_scale": 4.0, "retries": 0, "seconds": 136.0, "max_memory_gb": 17.08, "ce": 0.0013823658991896082, "kl": 0.00026207329210592434, "context": 0.001513402512500761}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 0.09858907014131546, "loss_scale": 4.0, "retries": 0, "seconds": 136.9, "max_memory_gb": 17.08, "ce": 0.006905716443725396, "kl": 0.001713177554847789, "context": 0.0077623050747206435}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.9268452525138855, "loss_scale": 4.0, "retries": 0, "seconds": 137.9, "max_memory_gb": 17.08, "ce": 0.04050174118310679, "kl": 0.07227464725247046, "context": 0.07663906866218895}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 2.2270543575286865, "loss_scale": 4.0, "retries": 0, "seconds": 138.8, "max_memory_gb": 17.08, "ce": 0.37006283225491643, "kl": 0.3408367257579812, "context": 0.5404811836488079}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 0.4120861887931824, "loss_scale": 4.0, "retries": 0, "seconds": 139.7, "max_memory_gb": 17.08, "ce": 0.018056146276649088, "kl": 0.006437815056415275, "context": 0.02127505390672013}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 7.186992168426514, "loss_scale": 4.0, "retries": 0, "seconds": 140.6, "max_memory_gb": 17.08, "ce": 0.2580349971540272, "kl": 0.26482547460182104, "context": 0.3904477315954864}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 0.18235918879508972, "loss_scale": 4.0, "retries": 0, "seconds": 141.5, "max_memory_gb": 17.08, "ce": 0.00970838381908834, "kl": 0.002753221313469112, "context": 0.011084994359407574}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 3.22788405418396, "loss_scale": 4.0, "retries": 0, "seconds": 142.4, "max_memory_gb": 17.08, "ce": 0.37536560051376, "kl": 0.0731006942021395, "context": 0.4119159476394998}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.3698105216026306, "loss_scale": 4.0, "retries": 0, "seconds": 143.4, "max_memory_gb": 17.08, "ce": 0.01891646973672323, "kl": 0.007591674455397879, "context": 0.022712306497851387}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 1.587808609008789, "loss_scale": 4.0, "retries": 0, "seconds": 144.3, "max_memory_gb": 17.08, "ce": 0.0979243801557459, "kl": 0.17969115468440577, "context": 0.18776996154338121}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.8521654009819031, "loss_scale": 4.0, "retries": 0, "seconds": 145.2, "max_memory_gb": 17.08, "ce": 0.004167782433796674, "kl": 0.3124232463824228, "context": 0.16037939983652905}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 1.5247929096221924, "loss_scale": 4.0, "retries": 0, "seconds": 146.2, "max_memory_gb": 17.08, "ce": 0.21941492601763457, "kl": 0.018002359429374337, "context": 0.22841610701289028}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 1.1440244913101196, "loss_scale": 4.0, "retries": 0, "seconds": 147.1, "max_memory_gb": 17.08, "ce": 0.005042826727731153, "kl": 0.19033036040491425, "context": 0.1002080092439428}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.17110762000083923, "loss_scale": 4.0, "retries": 0, "seconds": 148.1, "max_memory_gb": 17.08, "ce": 0.011151395265187602, "kl": 0.004323389852288528, "context": 0.013313090603332967}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.036323342472314835, "loss_scale": 4.0, "retries": 0, "seconds": 149.0, "max_memory_gb": 17.08, "ce": 0.002168463630368933, "kl": 0.0007222522581287194, "context": 0.0025295897503383458}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 2.698232889175415, "loss_scale": 4.0, "retries": 0, "seconds": 149.9, "max_memory_gb": 17.08, "ce": 0.06872706601279788, "kl": 0.009505677000561263, "context": 0.07347990732523613}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 2.452997922897339, "loss_scale": 4.0, "retries": 0, "seconds": 150.9, "max_memory_gb": 17.08, "ce": 0.25842469092458487, "kl": 0.14807094833668089, "context": 0.332460172183346}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 1.558362603187561, "loss_scale": 4.0, "retries": 0, "seconds": 151.8, "max_memory_gb": 17.08, "ce": 0.1712656288727885, "kl": 0.3549099903648312, "context": 0.3487206240533851}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 1.6423768997192383, "loss_scale": 4.0, "retries": 0, "seconds": 152.7, "max_memory_gb": 17.08, "ce": 0.00644836608262267, "kl": 0.32519811180281977, "context": 0.1690474164643092}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.045506298542022705, "loss_scale": 4.0, "retries": 0, "seconds": 153.7, "max_memory_gb": 17.08, "ce": 0.0018251204310217872, "kl": 0.0014067491702007828, "context": 0.002528495024307631}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 1.3618758916854858, "loss_scale": 4.0, "retries": 0, "seconds": 154.6, "max_memory_gb": 17.08, "ce": 0.030990308383479714, "kl": 0.019030129442398902, "context": 0.0405053726863116}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.11347194015979767, "loss_scale": 4.0, "retries": 0, "seconds": 155.5, "max_memory_gb": 17.08, "ce": 0.0027394474745960906, "kl": 0.01569632420432754, "context": 0.01058760954765603}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.18806983530521393, "loss_scale": 4.0, "retries": 0, "seconds": 156.5, "max_memory_gb": 17.08, "ce": 0.008392645453568548, "kl": 0.002378845470957458, "context": 0.009582068218151107}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 0.9173834323883057, "loss_scale": 4.0, "retries": 0, "seconds": 157.4, "max_memory_gb": 17.08, "ce": 0.09366799204144627, "kl": 0.046625652108559734, "context": 0.11698081431677565}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 1.7856428623199463, "loss_scale": 4.0, "retries": 0, "seconds": 158.4, "max_memory_gb": 17.08, "ce": 0.11967472155811265, "kl": 0.03260964353103191, "context": 0.13597954181022942}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.08523949980735779, "loss_scale": 4.0, "retries": 0, "seconds": 159.3, "max_memory_gb": 17.08, "ce": 0.004916554767987691, "kl": 0.0034215642044728156, "context": 0.006627336755627766}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 0.9221476316452026, "loss_scale": 4.0, "retries": 0, "seconds": 160.1, "max_memory_gb": 17.08, "ce": 0.013357669493416324, "kl": 0.41662181832361966, "context": 0.221668573532952}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.4490547478199005, "loss_scale": 4.0, "retries": 0, "seconds": 161.1, "max_memory_gb": 17.08, "ce": 0.01652931430726312, "kl": 0.03252178073125833, "context": 0.0327902048593387}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 1.071428894996643, "loss_scale": 4.0, "retries": 0, "seconds": 162.0, "max_memory_gb": 17.08, "ce": 0.032283293025102466, "kl": 0.08437049471748992, "context": 0.07446854167210404}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 1.7495787143707275, "loss_scale": 4.0, "retries": 0, "seconds": 162.9, "max_memory_gb": 17.08, "ce": 0.07091212586965412, "kl": 0.060121833404991776, "context": 0.10097304545342922}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.8216626048088074, "loss_scale": 4.0, "retries": 0, "seconds": 163.9, "max_memory_gb": 17.08, "ce": 0.005190993600990623, "kl": 0.020808606408536434, "context": 0.015595297096297145}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 0.4498121440410614, "loss_scale": 4.0, "retries": 0, "seconds": 164.8, "max_memory_gb": 17.08, "ce": 0.07019469933584332, "kl": 0.06544796794514696, "context": 0.1029186814557761}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 1.1238261461257935, "loss_scale": 4.0, "retries": 0, "seconds": 165.7, "max_memory_gb": 17.08, "ce": 0.10332678889972158, "kl": 0.07567788334563375, "context": 0.1411657347343862}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.3267431855201721, "loss_scale": 4.0, "retries": 0, "seconds": 166.7, "max_memory_gb": 17.08, "ce": 0.008230114035541192, "kl": 0.061970345352619916, "context": 0.039215287048136815}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 0.7959360480308533, "loss_scale": 4.0, "retries": 0, "seconds": 167.6, "max_memory_gb": 17.08, "ce": 0.017570278665516526, "kl": 0.026513810036703944, "context": 0.030827183625660837}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.11283959448337555, "loss_scale": 4.0, "retries": 0, "seconds": 168.5, "max_memory_gb": 17.08, "ce": 0.005250452923064586, "kl": 0.0022745263913748204, "context": 0.0063877162465360016}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.5096104145050049, "loss_scale": 4.0, "retries": 0, "seconds": 169.4, "max_memory_gb": 17.08, "ce": 0.05921443732222542, "kl": 0.07415803422918543, "context": 0.09629345432040282}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 4.073096752166748, "loss_scale": 4.0, "retries": 0, "seconds": 170.3, "max_memory_gb": 17.08, "ce": 0.003508210531435907, "kl": 0.21113942083684378, "context": 0.10907792096259072}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 2.24466872215271, "loss_scale": 4.0, "retries": 0, "seconds": 171.3, "max_memory_gb": 17.08, "ce": 0.02702349633909762, "kl": 0.17505195995909162, "context": 0.11454947642050683}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.8901212215423584, "loss_scale": 4.0, "retries": 0, "seconds": 172.2, "max_memory_gb": 17.08, "ce": 0.04067896181368269, "kl": 0.028166475356556475, "context": 0.05476220091804862}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 2.686448812484741, "loss_scale": 4.0, "retries": 0, "seconds": 173.1, "max_memory_gb": 17.08, "ce": 0.15479401918128133, "kl": 0.011274085837612802, "context": 0.16043105418793857}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 3.0203378200531006, "loss_scale": 4.0, "retries": 0, "seconds": 174.1, "max_memory_gb": 17.08, "ce": 0.05459122633328661, "kl": 0.11724584526382387, "context": 0.11321414913982153}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 6.967684268951416, "loss_scale": 4.0, "retries": 0, "seconds": 175.0, "max_memory_gb": 17.08, "ce": 0.10125066345790401, "kl": 0.05450394327635877, "context": 0.1285026352852583}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 0.28018683195114136, "loss_scale": 4.0, "retries": 0, "seconds": 175.8, "max_memory_gb": 17.08, "ce": 0.019506982003804296, "kl": 0.0070355019165617705, "context": 0.023024733571219258}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.9598243236541748, "loss_scale": 4.0, "retries": 0, "seconds": 176.8, "max_memory_gb": 17.08, "ce": 0.013762429356575012, "kl": 0.24107264599297196, "context": 0.13429875520523638}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 6.0299859046936035, "loss_scale": 4.0, "retries": 0, "seconds": 177.8, "max_memory_gb": 17.08, "ce": 0.17295125723467208, "kl": 0.12416519767430145, "context": 0.23503384710056707}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 0.25118425488471985, "loss_scale": 4.0, "retries": 0, "seconds": 178.7, "max_memory_gb": 17.08, "ce": 0.0026037069619633257, "kl": 0.029912170517491177, "context": 0.017559792671818286}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 0.8704926371574402, "loss_scale": 4.0, "retries": 0, "seconds": 179.6, "max_memory_gb": 17.08, "ce": 0.07283605600241572, "kl": 0.07848259283855441, "context": 0.1120773542788811}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.9002621173858643, "loss_scale": 4.0, "retries": 0, "seconds": 180.5, "max_memory_gb": 17.08, "ce": 0.19901343922538217, "kl": 0.033617321445490234, "context": 0.2158220979545149}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 5.949662685394287, "loss_scale": 4.0, "retries": 0, "seconds": 181.5, "max_memory_gb": 17.08, "ce": 0.2676018995116465, "kl": 0.18580960255349055, "context": 0.3605067169992253}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 1.6471924781799316, "loss_scale": 4.0, "retries": 0, "seconds": 182.4, "max_memory_gb": 17.08, "ce": 0.1250490321253892, "kl": 0.11131941477560758, "context": 0.1807087402266916}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.06948307901620865, "loss_scale": 4.0, "retries": 0, "seconds": 183.3, "max_memory_gb": 17.08, "ce": 0.0028516087913885713, "kl": 0.004060489998664707, "context": 0.004881853761617094}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.08198843896389008, "loss_scale": 4.0, "retries": 0, "seconds": 184.3, "max_memory_gb": 17.08, "ce": 0.004129282206122298, "kl": 0.0037889710984018166, "context": 0.006023768073646352}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.06160728260874748, "loss_scale": 4.0, "retries": 0, "seconds": 185.2, "max_memory_gb": 17.08, "ce": 0.001262561279872898, "kl": 0.001553636519020074, "context": 0.0020393795130075887}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 0.259141743183136, "loss_scale": 4.0, "retries": 0, "seconds": 186.1, "max_memory_gb": 17.08, "ce": 0.003324079909361899, "kl": 0.04772350532584824, "context": 0.027185833430849016}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 4.09207820892334, "loss_scale": 4.0, "retries": 0, "seconds": 187.1, "max_memory_gb": 17.08, "ce": 0.2388486539130099, "kl": 0.03807927180605475, "context": 0.25788828881195514}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.9974026679992676, "loss_scale": 4.0, "retries": 0, "seconds": 188.0, "max_memory_gb": 17.08, "ce": 0.03226120726321824, "kl": 0.1596033161622472, "context": 0.112062867206987}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 0.5401380658149719, "loss_scale": 4.0, "retries": 0, "seconds": 189.0, "max_memory_gb": 17.08, "ce": 0.02820234201499261, "kl": 0.035259551045783155, "context": 0.045832117524696514}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.5450451374053955, "loss_scale": 4.0, "retries": 0, "seconds": 189.9, "max_memory_gb": 17.08, "ce": 0.02484105888288468, "kl": 0.009723125745949801, "context": 0.02970262130838819}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 0.07395286113023758, "loss_scale": 4.0, "retries": 0, "seconds": 190.7, "max_memory_gb": 17.08, "ce": 0.0012456406002456788, "kl": 0.003918691058061086, "context": 0.0032049860747065395}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.8779104948043823, "loss_scale": 4.0, "retries": 0, "seconds": 191.7, "max_memory_gb": 17.08, "ce": 0.3862400155776413, "kl": 0.3376312791951932, "context": 0.5550556680973386}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.5048288106918335, "loss_scale": 4.0, "retries": 0, "seconds": 192.6, "max_memory_gb": 17.08, "ce": 0.04127110067929607, "kl": 0.04462932931710384, "context": 0.06358576533966698}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.45158207416534424, "loss_scale": 4.0, "retries": 0, "seconds": 193.5, "max_memory_gb": 17.08, "ce": 0.010360112028138246, "kl": 0.09508531829146705, "context": 0.05790277221967699}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.040291137993335724, "loss_scale": 4.0, "retries": 0, "seconds": 194.5, "max_memory_gb": 17.08, "ce": 0.0018801161204464734, "kl": 0.0007046727823762922, "context": 0.002232452476164326}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.1171773225069046, "loss_scale": 4.0, "retries": 0, "seconds": 195.4, "max_memory_gb": 17.08, "ce": 0.004795981309143826, "kl": 0.003583929119486129, "context": 0.0065879458270501345}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.4724586606025696, "loss_scale": 4.0, "retries": 0, "seconds": 196.3, "max_memory_gb": 17.08, "ce": 0.0005441301109385677, "kl": 0.04017780745198252, "context": 0.02063303413160611}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.7426279187202454, "loss_scale": 4.0, "retries": 0, "seconds": 197.3, "max_memory_gb": 17.08, "ce": 0.0023257745342561975, "kl": 0.2452271308611671, "context": 0.12493934313533828}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 2.8361494541168213, "loss_scale": 4.0, "retries": 0, "seconds": 198.2, "max_memory_gb": 17.08, "ce": 0.0704010152403498, "kl": 0.4780865693464875, "context": 0.30944430181989446}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 0.13733169436454773, "loss_scale": 4.0, "retries": 0, "seconds": 199.2, "max_memory_gb": 17.08, "ce": 0.00800962679204531, "kl": 0.0025130825888481922, "context": 0.00926616822835058}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 1.5924288034439087, "loss_scale": 4.0, "retries": 0, "seconds": 200.1, "max_memory_gb": 17.08, "ce": 0.03212905827967916, "kl": 0.027205152829992585, "context": 0.04573163569148164}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 2.1714367866516113, "loss_scale": 4.0, "retries": 0, "seconds": 201.0, "max_memory_gb": 17.08, "ce": 0.024560042467783205, "kl": 0.8855889311525971, "context": 0.4673544940305874}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.0993432104587555, "loss_scale": 4.0, "retries": 0, "seconds": 202.0, "max_memory_gb": 17.08, "ce": 0.0038454093228210695, "kl": 0.002005409891353338, "context": 0.004848114098422229}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.06700877845287323, "loss_scale": 4.0, "retries": 0, "seconds": 202.8, "max_memory_gb": 17.08, "ce": 0.003448580944677815, "kl": 0.0010940264855889836, "context": 0.003995594219304621}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 0.11106888204813004, "loss_scale": 4.0, "retries": 0, "seconds": 203.7, "max_memory_gb": 17.08, "ce": 0.004560801346087828, "kl": 0.004735024267574772, "context": 0.006928313407115638}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.7658205628395081, "loss_scale": 4.0, "retries": 0, "seconds": 204.7, "max_memory_gb": 17.08, "ce": 0.029129607137292624, "kl": 0.09931295335991308, "context": 0.07878608349710703}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.5287904739379883, "loss_scale": 4.0, "retries": 0, "seconds": 205.6, "max_memory_gb": 17.08, "ce": 0.013366246166697238, "kl": 0.02474353199113466, "context": 0.025738012802321464}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 1.0958189964294434, "loss_scale": 4.0, "retries": 0, "seconds": 206.5, "max_memory_gb": 17.08, "ce": 0.07528888180968352, "kl": 0.0653073702706024, "context": 0.10794256709050387}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.6898767948150635, "loss_scale": 4.0, "retries": 0, "seconds": 207.5, "max_memory_gb": 17.08, "ce": 0.013544079469284043, "kl": 0.015960550415911712, "context": 0.02152435516472906}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 2.192033529281616, "loss_scale": 4.0, "retries": 0, "seconds": 208.4, "max_memory_gb": 17.08, "ce": 0.03984998588566668, "kl": 0.18201788015358034, "context": 0.13085892284289002}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 2.2373359203338623, "loss_scale": 4.0, "retries": 0, "seconds": 209.4, "max_memory_gb": 17.08, "ce": 0.042332926437666174, "kl": 0.08854327405015283, "context": 0.08660456346842693}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 1.667165756225586, "loss_scale": 4.0, "retries": 0, "seconds": 210.3, "max_memory_gb": 17.08, "ce": 0.12639892200240865, "kl": 0.2572142143162637, "context": 0.2550060179855791}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 1.9377734661102295, "loss_scale": 4.0, "retries": 0, "seconds": 211.2, "max_memory_gb": 17.08, "ce": 0.41273467357677873, "kl": 0.5179879893839825, "context": 0.6717286659404635}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 0.20266729593276978, "loss_scale": 4.0, "retries": 0, "seconds": 212.2, "max_memory_gb": 17.08, "ce": 0.008581910973589402, "kl": 0.0035980179272883106, "context": 0.010380920153693296}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 3.754970073699951, "loss_scale": 4.0, "retries": 0, "seconds": 213.1, "max_memory_gb": 17.08, "ce": 0.06212193913233932, "kl": 0.022578462107048836, "context": 0.07341116829775274}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.44812798500061035, "loss_scale": 4.0, "retries": 0, "seconds": 214.0, "max_memory_gb": 17.08, "ce": 0.006564008377608843, "kl": 0.05834896946180379, "context": 0.03573849383974448}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.799283504486084, "loss_scale": 4.0, "retries": 0, "seconds": 214.9, "max_memory_gb": 17.08, "ce": 0.0041778616068768315, "kl": 0.14039484273962444, "context": 0.07437528111040592}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 7.079317569732666, "loss_scale": 4.0, "retries": 0, "seconds": 215.8, "max_memory_gb": 17.08, "ce": 0.18748714691901114, "kl": 1.110170562089479, "context": 0.7425724160857499}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.2398693561553955, "loss_scale": 4.0, "retries": 0, "seconds": 216.7, "max_memory_gb": 17.08, "ce": 0.0053658070537494496, "kl": 0.01678494228690397, "context": 0.013758278539171442}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 5.709300518035889, "loss_scale": 4.0, "retries": 0, "seconds": 217.7, "max_memory_gb": 17.08, "ce": 0.28997400336083956, "kl": 0.677538666590408, "context": 0.6287433328689076}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 3.9200439453125, "loss_scale": 4.0, "retries": 0, "seconds": 218.6, "max_memory_gb": 17.08, "ce": 0.2937418489600532, "kl": 0.0791682692051836, "context": 0.3333259937353432}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 1.2093396186828613, "loss_scale": 4.0, "retries": 0, "seconds": 219.6, "max_memory_gb": 17.08, "ce": 0.0030490929348161444, "kl": 0.03882535778575402, "context": 0.022461771077360027}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 0.26352936029434204, "loss_scale": 4.0, "retries": 0, "seconds": 220.5, "max_memory_gb": 17.08, "ce": 0.010624475835356861, "kl": 0.04331360680225771, "context": 0.032281280640745535}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.029741322621703148, "loss_scale": 4.0, "retries": 0, "seconds": 221.4, "max_memory_gb": 17.08, "ce": 0.001375162559270393, "kl": 0.00040706207732910116, "context": 0.0015786935982760042}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 2.2226929664611816, "loss_scale": 4.0, "retries": 0, "seconds": 222.4, "max_memory_gb": 17.08, "ce": 0.0077058169117663056, "kl": 0.8097705431173381, "context": 0.41259109135717154}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.6254263520240784, "loss_scale": 4.0, "retries": 0, "seconds": 223.3, "max_memory_gb": 17.08, "ce": 0.05327208456583321, "kl": 0.12702823360450566, "context": 0.116786201717332}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.13479912281036377, "loss_scale": 4.0, "retries": 0, "seconds": 224.1, "max_memory_gb": 17.08, "ce": 0.0065433500276412815, "kl": 0.015893279822194017, "context": 0.014489989844150841}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.7918363213539124, "loss_scale": 4.0, "retries": 0, "seconds": 225.1, "max_memory_gb": 17.08, "ce": 0.10801481149974279, "kl": 0.05628315362991998, "context": 0.13615638128248975}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 6.926004409790039, "loss_scale": 4.0, "retries": 0, "seconds": 226.0, "max_memory_gb": 17.08, "ce": 0.5754481391631998, "kl": 0.010452998103573918, "context": 0.580674665980041}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 0.4119584262371063, "loss_scale": 4.0, "retries": 0, "seconds": 226.9, "max_memory_gb": 17.08, "ce": 0.003967233642470092, "kl": 0.06761300853395369, "context": 0.03777373788761906}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 12.958744049072266, "loss_scale": 4.0, "retries": 0, "seconds": 227.9, "max_memory_gb": 17.08, "ce": 0.15696161275263876, "kl": 0.32497492951370077, "context": 0.31944908123114146}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.521541953086853, "loss_scale": 4.0, "retries": 0, "seconds": 228.8, "max_memory_gb": 17.08, "ce": 0.0013459642141242512, "kl": 0.08375025971599825, "context": 0.04322109464555979}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.3507992923259735, "loss_scale": 4.0, "retries": 0, "seconds": 229.8, "max_memory_gb": 17.08, "ce": 0.023321012849919498, "kl": 0.011453873616119381, "context": 0.029047949152300134}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 3.370558977127075, "loss_scale": 4.0, "retries": 0, "seconds": 230.7, "max_memory_gb": 17.08, "ce": 0.01374805939849466, "kl": 0.0744486815383425, "context": 0.05097240081522614}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 0.7647026777267456, "loss_scale": 4.0, "retries": 0, "seconds": 231.6, "max_memory_gb": 17.08, "ce": 0.06495525524951518, "kl": 0.07974110561463021, "context": 0.10482580805546604}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.6996275782585144, "loss_scale": 4.0, "retries": 0, "seconds": 232.6, "max_memory_gb": 17.08, "ce": 0.04200789594324306, "kl": 0.07899255519441795, "context": 0.08150417724391446}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 1.3963892459869385, "loss_scale": 4.0, "retries": 0, "seconds": 233.5, "max_memory_gb": 17.08, "ce": 0.004207065270747989, "kl": 0.9113011669687694, "context": 0.4598576359276194}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 7.871495723724365, "loss_scale": 4.0, "retries": 0, "seconds": 234.3, "max_memory_gb": 17.08, "ce": 0.19305832176178228, "kl": 0.15862016078244778, "context": 0.2723683947115205}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.16054686903953552, "loss_scale": 4.0, "retries": 0, "seconds": 235.3, "max_memory_gb": 17.08, "ce": 0.010816792208061088, "kl": 0.0012630787477974081, "context": 0.011448331169958692}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 8.055854797363281, "loss_scale": 4.0, "retries": 0, "seconds": 236.3, "max_memory_gb": 17.08, "ce": 0.5036520506255329, "kl": 0.4619218680309132, "context": 0.7346129864454269}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.5786089301109314, "loss_scale": 4.0, "retries": 0, "seconds": 237.1, "max_memory_gb": 17.08, "ce": 0.01771908857335802, "kl": 0.06279390116833383, "context": 0.04911603784421459}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 6.849900245666504, "loss_scale": 4.0, "retries": 0, "seconds": 238.2, "max_memory_gb": 17.08, "ce": 0.1689149147568969, "kl": 0.1474704251777439, "context": 0.24265012360410765}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 8.253273010253906, "loss_scale": 4.0, "retries": 0, "seconds": 239.0, "max_memory_gb": 17.08, "ce": 0.22476774093229324, "kl": 0.029736343338299775, "context": 0.23963591040228494}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.9884482026100159, "loss_scale": 4.0, "retries": 0, "seconds": 240.0, "max_memory_gb": 17.08, "ce": 0.06495900415757205, "kl": 0.05707996579258179, "context": 0.09349898938671686}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 1.4600783586502075, "loss_scale": 4.0, "retries": 0, "seconds": 240.9, "max_memory_gb": 17.08, "ce": 0.1103173816591152, "kl": 0.1462266592134256, "context": 0.18343071130220778}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 3.795264959335327, "loss_scale": 4.0, "retries": 0, "seconds": 241.8, "max_memory_gb": 17.08, "ce": 0.04606290586525574, "kl": 0.24593844020273536, "context": 0.1690321260248311}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.22781147062778473, "loss_scale": 4.0, "retries": 0, "seconds": 242.8, "max_memory_gb": 17.08, "ce": 0.007842853709007613, "kl": 0.003945599150029011, "context": 0.009815653043915518}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.05425867438316345, "loss_scale": 4.0, "retries": 0, "seconds": 243.7, "max_memory_gb": 17.08, "ce": 0.002553748185164295, "kl": 0.00040701979651203146, "context": 0.0027572580438572913}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.26294001936912537, "loss_scale": 4.0, "retries": 0, "seconds": 244.6, "max_memory_gb": 17.08, "ce": 0.010843654934433289, "kl": 0.00726451953232754, "context": 0.014475915260845795}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.056947045028209686, "loss_scale": 4.0, "retries": 0, "seconds": 245.6, "max_memory_gb": 17.08, "ce": 0.002465825353283435, "kl": 0.00482000060401333, "context": 0.004875825499766506}

runs/fixed_internvl/removal/Deep8_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/removal/Deep8_s13/adapter/adapter_config.json

@@ -0,0 +1,213 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.28.mlp.down_proj": 144,
+    "model.language_model.layers.28.mlp.gate_proj": 144,
+    "model.language_model.layers.28.mlp.up_proj": 144,
+    "model.language_model.layers.28.self_attn.k_proj": 144,
+    "model.language_model.layers.28.self_attn.o_proj": 144,
+    "model.language_model.layers.28.self_attn.q_proj": 144,
+    "model.language_model.layers.28.self_attn.v_proj": 144,
+    "model.language_model.layers.29.mlp.down_proj": 144,
+    "model.language_model.layers.29.mlp.gate_proj": 144,
+    "model.language_model.layers.29.mlp.up_proj": 144,
+    "model.language_model.layers.29.self_attn.k_proj": 144,
+    "model.language_model.layers.29.self_attn.o_proj": 144,
+    "model.language_model.layers.29.self_attn.q_proj": 144,
+    "model.language_model.layers.29.self_attn.v_proj": 144,
+    "model.language_model.layers.30.mlp.down_proj": 144,
+    "model.language_model.layers.30.mlp.gate_proj": 144,
+    "model.language_model.layers.30.mlp.up_proj": 144,
+    "model.language_model.layers.30.self_attn.k_proj": 144,
+    "model.language_model.layers.30.self_attn.o_proj": 144,
+    "model.language_model.layers.30.self_attn.q_proj": 144,
+    "model.language_model.layers.30.self_attn.v_proj": 144,
+    "model.language_model.layers.31.mlp.down_proj": 144,
+    "model.language_model.layers.31.mlp.gate_proj": 144,
+    "model.language_model.layers.31.mlp.up_proj": 144,
+    "model.language_model.layers.31.self_attn.k_proj": 144,
+    "model.language_model.layers.31.self_attn.o_proj": 144,
+    "model.language_model.layers.31.self_attn.q_proj": 144,
+    "model.language_model.layers.31.self_attn.v_proj": 144,
+    "model.language_model.layers.32.mlp.down_proj": 144,
+    "model.language_model.layers.32.mlp.gate_proj": 144,
+    "model.language_model.layers.32.mlp.up_proj": 144,
+    "model.language_model.layers.32.self_attn.k_proj": 144,
+    "model.language_model.layers.32.self_attn.o_proj": 144,
+    "model.language_model.layers.32.self_attn.q_proj": 144,
+    "model.language_model.layers.32.self_attn.v_proj": 144,
+    "model.language_model.layers.33.mlp.down_proj": 144,
+    "model.language_model.layers.33.mlp.gate_proj": 144,
+    "model.language_model.layers.33.mlp.up_proj": 144,
+    "model.language_model.layers.33.self_attn.k_proj": 144,
+    "model.language_model.layers.33.self_attn.o_proj": 144,
+    "model.language_model.layers.33.self_attn.q_proj": 144,
+    "model.language_model.layers.33.self_attn.v_proj": 144,
+    "model.language_model.layers.34.mlp.down_proj": 144,
+    "model.language_model.layers.34.mlp.gate_proj": 144,
+    "model.language_model.layers.34.mlp.up_proj": 144,
+    "model.language_model.layers.34.self_attn.k_proj": 144,
+    "model.language_model.layers.34.self_attn.o_proj": 144,
+    "model.language_model.layers.34.self_attn.q_proj": 144,
+    "model.language_model.layers.34.self_attn.v_proj": 144,
+    "model.language_model.layers.35.mlp.down_proj": 144,
+    "model.language_model.layers.35.mlp.gate_proj": 144,
+    "model.language_model.layers.35.mlp.up_proj": 144,
+    "model.language_model.layers.35.self_attn.k_proj": 144,
+    "model.language_model.layers.35.self_attn.o_proj": 144,
+    "model.language_model.layers.35.self_attn.q_proj": 144,
+    "model.language_model.layers.35.self_attn.v_proj": 144
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.28.mlp.down_proj": 72,
+    "model.language_model.layers.28.mlp.gate_proj": 72,
+    "model.language_model.layers.28.mlp.up_proj": 72,
+    "model.language_model.layers.28.self_attn.k_proj": 72,
+    "model.language_model.layers.28.self_attn.o_proj": 72,
+    "model.language_model.layers.28.self_attn.q_proj": 72,
+    "model.language_model.layers.28.self_attn.v_proj": 72,
+    "model.language_model.layers.29.mlp.down_proj": 72,
+    "model.language_model.layers.29.mlp.gate_proj": 72,
+    "model.language_model.layers.29.mlp.up_proj": 72,
+    "model.language_model.layers.29.self_attn.k_proj": 72,
+    "model.language_model.layers.29.self_attn.o_proj": 72,
+    "model.language_model.layers.29.self_attn.q_proj": 72,
+    "model.language_model.layers.29.self_attn.v_proj": 72,
+    "model.language_model.layers.30.mlp.down_proj": 72,
+    "model.language_model.layers.30.mlp.gate_proj": 72,
+    "model.language_model.layers.30.mlp.up_proj": 72,
+    "model.language_model.layers.30.self_attn.k_proj": 72,
+    "model.language_model.layers.30.self_attn.o_proj": 72,
+    "model.language_model.layers.30.self_attn.q_proj": 72,
+    "model.language_model.layers.30.self_attn.v_proj": 72,
+    "model.language_model.layers.31.mlp.down_proj": 72,
+    "model.language_model.layers.31.mlp.gate_proj": 72,
+    "model.language_model.layers.31.mlp.up_proj": 72,
+    "model.language_model.layers.31.self_attn.k_proj": 72,
+    "model.language_model.layers.31.self_attn.o_proj": 72,
+    "model.language_model.layers.31.self_attn.q_proj": 72,
+    "model.language_model.layers.31.self_attn.v_proj": 72,
+    "model.language_model.layers.32.mlp.down_proj": 72,
+    "model.language_model.layers.32.mlp.gate_proj": 72,
+    "model.language_model.layers.32.mlp.up_proj": 72,
+    "model.language_model.layers.32.self_attn.k_proj": 72,
+    "model.language_model.layers.32.self_attn.o_proj": 72,
+    "model.language_model.layers.32.self_attn.q_proj": 72,
+    "model.language_model.layers.32.self_attn.v_proj": 72,
+    "model.language_model.layers.33.mlp.down_proj": 72,
+    "model.language_model.layers.33.mlp.gate_proj": 72,
+    "model.language_model.layers.33.mlp.up_proj": 72,
+    "model.language_model.layers.33.self_attn.k_proj": 72,
+    "model.language_model.layers.33.self_attn.o_proj": 72,
+    "model.language_model.layers.33.self_attn.q_proj": 72,
+    "model.language_model.layers.33.self_attn.v_proj": 72,
+    "model.language_model.layers.34.mlp.down_proj": 72,
+    "model.language_model.layers.34.mlp.gate_proj": 72,
+    "model.language_model.layers.34.mlp.up_proj": 72,
+    "model.language_model.layers.34.self_attn.k_proj": 72,
+    "model.language_model.layers.34.self_attn.o_proj": 72,
+    "model.language_model.layers.34.self_attn.q_proj": 72,
+    "model.language_model.layers.34.self_attn.v_proj": 72,
+    "model.language_model.layers.35.mlp.down_proj": 72,
+    "model.language_model.layers.35.mlp.gate_proj": 72,
+    "model.language_model.layers.35.mlp.up_proj": 72,
+    "model.language_model.layers.35.self_attn.k_proj": 72,
+    "model.language_model.layers.35.self_attn.o_proj": 72,
+    "model.language_model.layers.35.self_attn.q_proj": 72,
+    "model.language_model.layers.35.self_attn.v_proj": 72
+  },
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.29.mlp.down_proj",
+    "model.language_model.layers.29.self_attn.q_proj",
+    "model.language_model.layers.34.self_attn.k_proj",
+    "model.language_model.layers.33.self_attn.q_proj",
+    "model.language_model.layers.32.mlp.down_proj",
+    "model.language_model.layers.35.self_attn.k_proj",
+    "model.language_model.layers.30.self_attn.o_proj",
+    "model.language_model.layers.35.mlp.down_proj",
+    "model.language_model.layers.34.mlp.up_proj",
+    "model.language_model.layers.28.mlp.down_proj",
+    "model.language_model.layers.30.mlp.down_proj",
+    "model.language_model.layers.34.mlp.gate_proj",
+    "model.language_model.layers.32.self_attn.v_proj",
+    "model.language_model.layers.31.self_attn.o_proj",
+    "model.language_model.layers.31.mlp.down_proj",
+    "model.language_model.layers.31.mlp.gate_proj",
+    "model.language_model.layers.30.self_attn.v_proj",
+    "model.language_model.layers.32.mlp.gate_proj",
+    "model.language_model.layers.31.self_attn.v_proj",
+    "model.language_model.layers.33.mlp.down_proj",
+    "model.language_model.layers.32.self_attn.q_proj",
+    "model.language_model.layers.28.self_attn.q_proj",
+    "model.language_model.layers.33.mlp.gate_proj",
+    "model.language_model.layers.32.self_attn.k_proj",
+    "model.language_model.layers.35.self_attn.q_proj",
+    "model.language_model.layers.28.self_attn.v_proj",
+    "model.language_model.layers.29.self_attn.v_proj",
+    "model.language_model.layers.30.mlp.gate_proj",
+    "model.language_model.layers.33.self_attn.k_proj",
+    "model.language_model.layers.34.mlp.down_proj",
+    "model.language_model.layers.28.self_attn.k_proj",
+    "model.language_model.layers.34.self_attn.q_proj",
+    "model.language_model.layers.35.self_attn.v_proj",
+    "model.language_model.layers.30.self_attn.k_proj",
+    "model.language_model.layers.29.mlp.gate_proj",
+    "model.language_model.layers.33.mlp.up_proj",
+    "model.language_model.layers.35.mlp.gate_proj",
+    "model.language_model.layers.28.mlp.gate_proj",
+    "model.language_model.layers.32.self_attn.o_proj",
+    "model.language_model.layers.33.self_attn.o_proj",
+    "model.language_model.layers.29.mlp.up_proj",
+    "model.language_model.layers.35.mlp.up_proj",
+    "model.language_model.layers.29.self_attn.k_proj",
+    "model.language_model.layers.31.mlp.up_proj",
+    "model.language_model.layers.30.self_attn.q_proj",
+    "model.language_model.layers.35.self_attn.o_proj",
+    "model.language_model.layers.28.self_attn.o_proj",
+    "model.language_model.layers.29.self_attn.o_proj",
+    "model.language_model.layers.31.self_attn.q_proj",
+    "model.language_model.layers.34.self_attn.v_proj",
+    "model.language_model.layers.32.mlp.up_proj",
+    "model.language_model.layers.30.mlp.up_proj",
+    "model.language_model.layers.33.self_attn.v_proj",
+    "model.language_model.layers.34.self_attn.o_proj",
+    "model.language_model.layers.31.self_attn.k_proj",
+    "model.language_model.layers.28.mlp.up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/fixed_internvl/removal/Deep8_s13/config.json

@@ -0,0 +1,43 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "world_size": 1,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "arm": {
+    "arm": "Deep8",
+    "layers": [
+      28,
+      29,
+      30,
+      31,
+      32,
+      33,
+      34,
+      35
+    ],
+    "ranks": {
+      "28": 72,
+      "29": 72,
+      "30": 72,
+      "31": 72,
+      "32": 72,
+      "33": 72,
+      "34": 72,
+      "35": 72
+    },
+    "parameters": 43646976,
+    "budget": 43646976
+  }
+}

runs/fixed_internvl/removal/Deep8_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 1.6960206031799316, "loss_scale": 4.0, "retries": 0, "seconds": 5.5, "max_memory_gb": 16.83, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 1.81200110912323, "loss_scale": 4.0, "retries": 0, "seconds": 10.1, "max_memory_gb": 17.17, "ce": 0.781619841716747, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 4.991434097290039, "loss_scale": 4.0, "retries": 0, "seconds": 14.7, "max_memory_gb": 17.18, "ce": 1.3613118417561054, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 2.5539052486419678, "loss_scale": 4.0, "retries": 0, "seconds": 19.3, "max_memory_gb": 17.18, "ce": 0.7234638199515757, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.08924728631973267, "loss_scale": 4.0, "retries": 0, "seconds": 23.9, "max_memory_gb": 17.18, "ce": 0.03916978449160524, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 1.4161972999572754, "loss_scale": 4.0, "retries": 0, "seconds": 28.3, "max_memory_gb": 17.18, "ce": 0.3973650020561763, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 3.484485149383545, "loss_scale": 4.0, "retries": 0, "seconds": 32.8, "max_memory_gb": 17.18, "ce": 0.8499696038816182, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 3.6550161838531494, "loss_scale": 4.0, "retries": 0, "seconds": 37.5, "max_memory_gb": 17.18, "ce": 0.7589388908818364, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 2.2897658348083496, "loss_scale": 4.0, "retries": 0, "seconds": 41.9, "max_memory_gb": 17.18, "ce": 0.5108338005375117, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.2987181544303894, "loss_scale": 4.0, "retries": 0, "seconds": 46.5, "max_memory_gb": 17.18, "ce": 0.14756144062266685, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 2.562034845352173, "loss_scale": 4.0, "retries": 0, "seconds": 50.9, "max_memory_gb": 17.18, "ce": 0.4312914837610151, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 1.6705743074417114, "loss_scale": 4.0, "retries": 0, "seconds": 55.6, "max_memory_gb": 17.18, "ce": 0.23298202701334958, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 1.0955297946929932, "loss_scale": 4.0, "retries": 0, "seconds": 60.1, "max_memory_gb": 17.18, "ce": 0.2970346204874659, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 1.4563738107681274, "loss_scale": 4.0, "retries": 0, "seconds": 64.7, "max_memory_gb": 17.18, "ce": 0.4370708139613271, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 1.0339142084121704, "loss_scale": 4.0, "retries": 0, "seconds": 69.3, "max_memory_gb": 17.18, "ce": 0.25005166047230887, "rank": 0.035745516419410706}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.2317369133234024, "loss_scale": 4.0, "retries": 0, "seconds": 73.7, "max_memory_gb": 17.18, "ce": 0.10100598282951978, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.8685653805732727, "loss_scale": 4.0, "retries": 0, "seconds": 78.2, "max_memory_gb": 17.18, "ce": 0.3067083202840877, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 0.8503016233444214, "loss_scale": 4.0, "retries": 0, "seconds": 82.8, "max_memory_gb": 17.19, "ce": 0.22639081966190133, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 0.8838106393814087, "loss_scale": 4.0, "retries": 0, "seconds": 87.3, "max_memory_gb": 17.19, "ce": 0.2484732959383109, "rank": 0.21912997961044312}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.06454605609178543, "loss_scale": 4.0, "retries": 0, "seconds": 91.7, "max_memory_gb": 17.19, "ce": 0.028506537008070154, "rank": 0.12365324050188065}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.5729688405990601, "loss_scale": 4.0, "retries": 0, "seconds": 96.2, "max_memory_gb": 17.19, "ce": 0.15465636444241682, "rank": 0.055743634700775146}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 1.1568646430969238, "loss_scale": 4.0, "retries": 0, "seconds": 100.9, "max_memory_gb": 17.19, "ce": 0.2325941760209389, "rank": 0.14083543419837952}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.771278977394104, "loss_scale": 4.0, "retries": 0, "seconds": 105.2, "max_memory_gb": 17.19, "ce": 0.1954563977969883, "rank": 0.10030236095190048}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 0.8763794898986816, "loss_scale": 4.0, "retries": 0, "seconds": 109.7, "max_memory_gb": 17.19, "ce": 0.09700414627150167, "rank": 0.11799502372741699}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 3.6795742511749268, "loss_scale": 4.0, "retries": 0, "seconds": 114.3, "max_memory_gb": 17.19, "ce": 0.22622131026582792, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 1.6075944900512695, "loss_scale": 4.0, "retries": 0, "seconds": 118.8, "max_memory_gb": 17.19, "ce": 0.15869956466485746, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 1.472715973854065, "loss_scale": 4.0, "retries": 0, "seconds": 123.4, "max_memory_gb": 17.19, "ce": 0.09267856903170468, "rank": 0.09653015434741974}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 1.4583171606063843, "loss_scale": 4.0, "retries": 0, "seconds": 127.9, "max_memory_gb": 17.19, "ce": 0.3250976701383479, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 2.0724871158599854, "loss_scale": 4.0, "retries": 0, "seconds": 132.6, "max_memory_gb": 17.19, "ce": 0.35594283952377737, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 1.524192452430725, "loss_scale": 4.0, "retries": 0, "seconds": 136.6, "max_memory_gb": 17.19, "ce": 0.045146949721129204, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 2.8094699382781982, "loss_scale": 4.0, "retries": 0, "seconds": 141.0, "max_memory_gb": 17.19, "ce": 0.4001522166072391, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 1.6185922622680664, "loss_scale": 4.0, "retries": 0, "seconds": 145.4, "max_memory_gb": 17.19, "ce": 0.06219425457675243, "rank": 0.0}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 1.2918192148208618, "loss_scale": 4.0, "retries": 0, "seconds": 149.8, "max_memory_gb": 17.19, "ce": 0.15664478011603933, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 1.0643422603607178, "loss_scale": 4.0, "retries": 0, "seconds": 154.4, "max_memory_gb": 17.19, "ce": 0.11318061840574956, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.05636649951338768, "loss_scale": 4.0, "retries": 0, "seconds": 159.0, "max_memory_gb": 17.19, "ce": 0.0054068069366621785, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 3.6430537700653076, "loss_scale": 4.0, "retries": 0, "seconds": 163.5, "max_memory_gb": 17.19, "ce": 0.2848706511867931, "rank": 0.0}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 1.012430191040039, "loss_scale": 4.0, "retries": 0, "seconds": 168.0, "max_memory_gb": 17.19, "ce": 0.2152043600945035, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 1.5217645168304443, "loss_scale": 4.0, "retries": 0, "seconds": 172.6, "max_memory_gb": 17.19, "ce": 0.15702865754428785, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 0.8591509461402893, "loss_scale": 4.0, "retries": 0, "seconds": 177.3, "max_memory_gb": 17.19, "ce": 0.22456484676513355, "rank": 0.12499773502349854}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.186946302652359, "loss_scale": 4.0, "retries": 0, "seconds": 181.9, "max_memory_gb": 17.19, "ce": 0.01613004312821431, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 1.258520483970642, "loss_scale": 4.0, "retries": 0, "seconds": 186.5, "max_memory_gb": 17.19, "ce": 0.2801388363222941, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.22896580398082733, "loss_scale": 4.0, "retries": 0, "seconds": 191.2, "max_memory_gb": 17.19, "ce": 0.019562753215723205, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.25212183594703674, "loss_scale": 4.0, "retries": 0, "seconds": 195.8, "max_memory_gb": 17.19, "ce": 0.02191549085546285, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 1.833009123802185, "loss_scale": 4.0, "retries": 0, "seconds": 200.3, "max_memory_gb": 17.19, "ce": 0.32931038468632323, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 1.336392879486084, "loss_scale": 4.0, "retries": 0, "seconds": 204.9, "max_memory_gb": 17.19, "ce": 0.2921999726968352, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.5888805389404297, "loss_scale": 4.0, "retries": 0, "seconds": 209.4, "max_memory_gb": 17.19, "ce": 0.10303274390753359, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.8612083792686462, "loss_scale": 4.0, "retries": 0, "seconds": 213.9, "max_memory_gb": 17.19, "ce": 0.051515607214241754, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.13087527453899384, "loss_scale": 4.0, "retries": 0, "seconds": 218.5, "max_memory_gb": 17.19, "ce": 0.0064674967143218964, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.2114107757806778, "loss_scale": 4.0, "retries": 0, "seconds": 222.4, "max_memory_gb": 17.19, "ce": 0.01947730420215521, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 2.0159409046173096, "loss_scale": 4.0, "retries": 0, "seconds": 226.5, "max_memory_gb": 17.19, "ce": 0.12973105800665508, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 0.8623561859130859, "loss_scale": 4.0, "retries": 0, "seconds": 231.1, "max_memory_gb": 17.19, "ce": 0.0718860338874947, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.3926038444042206, "loss_scale": 4.0, "retries": 0, "seconds": 235.4, "max_memory_gb": 17.19, "ce": 0.04312100569950417, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 1.219672679901123, "loss_scale": 4.0, "retries": 0, "seconds": 239.9, "max_memory_gb": 17.19, "ce": 0.12788867106792168, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 2.186924457550049, "loss_scale": 4.0, "retries": 0, "seconds": 244.4, "max_memory_gb": 17.19, "ce": 0.15814003235209384, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.3706612288951874, "loss_scale": 4.0, "retries": 0, "seconds": 249.1, "max_memory_gb": 17.19, "ce": 0.018164914967201184, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.646741509437561, "loss_scale": 4.0, "retries": 0, "seconds": 253.6, "max_memory_gb": 17.19, "ce": 0.03440237624454312, "rank": 0.0}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.19884039461612701, "loss_scale": 4.0, "retries": 0, "seconds": 258.0, "max_memory_gb": 17.19, "ce": 0.01620039220097169, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.04384598881006241, "loss_scale": 4.0, "retries": 0, "seconds": 264.0, "max_memory_gb": 17.19, "ce": 0.005742922525314498, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.8188601136207581, "loss_scale": 4.0, "retries": 0, "seconds": 269.1, "max_memory_gb": 17.19, "ce": 0.08702815843571443, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 1.682601809501648, "loss_scale": 4.0, "retries": 0, "seconds": 274.2, "max_memory_gb": 17.19, "ce": 0.10877964407609397, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.9019931554794312, "loss_scale": 4.0, "retries": 0, "seconds": 279.1, "max_memory_gb": 17.19, "ce": 0.21035340083108167, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.7622170448303223, "loss_scale": 4.0, "retries": 0, "seconds": 285.0, "max_memory_gb": 17.19, "ce": 0.07748050454392796, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.4170958995819092, "loss_scale": 4.0, "retries": 0, "seconds": 290.2, "max_memory_gb": 17.19, "ce": 0.027050068360040314, "rank": 0.0}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.20643077790737152, "loss_scale": 4.0, "retries": 0, "seconds": 294.7, "max_memory_gb": 17.19, "ce": 0.014741918805157184, "rank": 0.0}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 1.1918617486953735, "loss_scale": 4.0, "retries": 0, "seconds": 299.0, "max_memory_gb": 17.19, "ce": 0.13508902888679586, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 2.256488561630249, "loss_scale": 4.0, "retries": 0, "seconds": 303.7, "max_memory_gb": 17.19, "ce": 0.22102219601492834, "rank": 0.12297695130109787}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.038212478160858154, "loss_scale": 4.0, "retries": 0, "seconds": 309.1, "max_memory_gb": 17.19, "ce": 0.0038562015470233746, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.2141539752483368, "loss_scale": 4.0, "retries": 0, "seconds": 313.8, "max_memory_gb": 17.19, "ce": 0.021449182571814163, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 2.4011127948760986, "loss_scale": 4.0, "retries": 0, "seconds": 318.3, "max_memory_gb": 17.19, "ce": 0.6019497981469613, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.11597883701324463, "loss_scale": 4.0, "retries": 0, "seconds": 323.5, "max_memory_gb": 17.19, "ce": 0.00831222584565694, "rank": 0.0}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.15870428085327148, "loss_scale": 4.0, "retries": 0, "seconds": 329.5, "max_memory_gb": 17.19, "ce": 0.012095243368094089, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 1.048232078552246, "loss_scale": 4.0, "retries": 0, "seconds": 335.0, "max_memory_gb": 17.19, "ce": 0.0951283794711344, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 0.5630810260772705, "loss_scale": 4.0, "retries": 0, "seconds": 339.5, "max_memory_gb": 17.19, "ce": 0.1065291298430111, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.6782105565071106, "loss_scale": 4.0, "retries": 0, "seconds": 344.1, "max_memory_gb": 17.19, "ce": 0.1327445599101793, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 1.7802057266235352, "loss_scale": 4.0, "retries": 0, "seconds": 348.8, "max_memory_gb": 17.19, "ce": 0.7063898194326157, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 1.1257059574127197, "loss_scale": 4.0, "retries": 0, "seconds": 353.5, "max_memory_gb": 17.19, "ce": 0.24456209242453042, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.24938248097896576, "loss_scale": 4.0, "retries": 0, "seconds": 357.9, "max_memory_gb": 17.19, "ce": 0.020868475548923016, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.3042744994163513, "loss_scale": 4.0, "retries": 0, "seconds": 363.5, "max_memory_gb": 17.19, "ce": 0.03480735230277787, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.23207871615886688, "loss_scale": 4.0, "retries": 0, "seconds": 368.3, "max_memory_gb": 17.19, "ce": 0.019096009581289763, "rank": 0.03340373933315277}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.23081964254379272, "loss_scale": 4.0, "retries": 0, "seconds": 372.8, "max_memory_gb": 17.19, "ce": 0.01582804294412199, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.3937508761882782, "loss_scale": 4.0, "retries": 0, "seconds": 377.4, "max_memory_gb": 17.19, "ce": 0.04481313350129312, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 1.1510282754898071, "loss_scale": 4.0, "retries": 0, "seconds": 381.8, "max_memory_gb": 17.19, "ce": 0.13438485619235507, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 1.4571563005447388, "loss_scale": 4.0, "retries": 0, "seconds": 386.3, "max_memory_gb": 17.19, "ce": 0.1650518198039208, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 1.6874380111694336, "loss_scale": 4.0, "retries": 0, "seconds": 391.0, "max_memory_gb": 17.19, "ce": 0.07116734477972386, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.12223760038614273, "loss_scale": 4.0, "retries": 0, "seconds": 395.4, "max_memory_gb": 17.19, "ce": 0.00606555834883693, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.2644760012626648, "loss_scale": 4.0, "retries": 0, "seconds": 399.8, "max_memory_gb": 17.19, "ce": 0.03135272152212565, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.9291737079620361, "loss_scale": 4.0, "retries": 0, "seconds": 404.2, "max_memory_gb": 17.19, "ce": 0.16473256350286647, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.2861516773700714, "loss_scale": 4.0, "retries": 0, "seconds": 408.5, "max_memory_gb": 17.19, "ce": 0.0425994533161429, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.10692792385816574, "loss_scale": 4.0, "retries": 0, "seconds": 412.4, "max_memory_gb": 17.19, "ce": 0.019128514453541356, "rank": 0.1249871477484703}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.3915866017341614, "loss_scale": 4.0, "retries": 0, "seconds": 417.0, "max_memory_gb": 17.19, "ce": 0.08732481679999182, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 1.725392460823059, "loss_scale": 4.0, "retries": 0, "seconds": 421.7, "max_memory_gb": 17.19, "ce": 0.5386774947169215, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 1.139824628829956, "loss_scale": 4.0, "retries": 0, "seconds": 426.3, "max_memory_gb": 17.19, "ce": 0.18273520661682596, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 0.4734126329421997, "loss_scale": 4.0, "retries": 0, "seconds": 430.5, "max_memory_gb": 17.19, "ce": 0.09840842398625682, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 0.8734853267669678, "loss_scale": 4.0, "retries": 0, "seconds": 434.7, "max_memory_gb": 17.19, "ce": 0.08083106630056136, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 1.9279506206512451, "loss_scale": 4.0, "retries": 0, "seconds": 438.9, "max_memory_gb": 17.19, "ce": 0.42101949183506804, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 1.8297042846679688, "loss_scale": 4.0, "retries": 0, "seconds": 443.5, "max_memory_gb": 17.19, "ce": 0.4464275196842209, "rank": 0.2983452379703522}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.3692508637905121, "loss_scale": 4.0, "retries": 0, "seconds": 447.9, "max_memory_gb": 17.19, "ce": 0.04404518909700528, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.34639379382133484, "loss_scale": 4.0, "retries": 0, "seconds": 452.6, "max_memory_gb": 17.19, "ce": 0.030738127921267733, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.08836808055639267, "loss_scale": 4.0, "retries": 0, "seconds": 457.0, "max_memory_gb": 17.19, "ce": 0.011346112439696299, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.9620298743247986, "loss_scale": 4.0, "retries": 0, "seconds": 461.9, "max_memory_gb": 17.19, "ce": 0.13172246778412955, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.1640169471502304, "loss_scale": 4.0, "retries": 0, "seconds": 466.6, "max_memory_gb": 17.19, "ce": 0.023190169804365723, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.26203393936157227, "loss_scale": 4.0, "retries": 0, "seconds": 471.1, "max_memory_gb": 17.19, "ce": 0.015823934977561294, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.1474657654762268, "loss_scale": 4.0, "retries": 0, "seconds": 475.5, "max_memory_gb": 17.19, "ce": 0.012351556993962731, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.2895885407924652, "loss_scale": 4.0, "retries": 0, "seconds": 480.0, "max_memory_gb": 17.19, "ce": 0.039395187313857605, "rank": 0.12484967708587646}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.5056543350219727, "loss_scale": 4.0, "retries": 0, "seconds": 484.5, "max_memory_gb": 17.19, "ce": 0.08293792100198516, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.1389755755662918, "loss_scale": 4.0, "retries": 0, "seconds": 489.1, "max_memory_gb": 17.19, "ce": 0.005508015647137654, "rank": 0.1057635247707367}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 0.7117779850959778, "loss_scale": 4.0, "retries": 0, "seconds": 493.8, "max_memory_gb": 17.19, "ce": 0.11824499011504486, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 1.2881861925125122, "loss_scale": 4.0, "retries": 0, "seconds": 498.4, "max_memory_gb": 17.19, "ce": 0.16683825848338074, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 1.0474269390106201, "loss_scale": 4.0, "retries": 0, "seconds": 502.9, "max_memory_gb": 17.19, "ce": 0.2195171111743548, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.522599458694458, "loss_scale": 4.0, "retries": 0, "seconds": 507.6, "max_memory_gb": 17.19, "ce": 0.04266106855845919, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.22559945285320282, "loss_scale": 4.0, "retries": 0, "seconds": 512.2, "max_memory_gb": 17.19, "ce": 0.028881661230343525, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.10418542474508286, "loss_scale": 4.0, "retries": 0, "seconds": 517.1, "max_memory_gb": 17.19, "ce": 0.007419600015964534, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.3967974781990051, "loss_scale": 4.0, "retries": 0, "seconds": 522.1, "max_memory_gb": 17.19, "ce": 0.029378783405263675, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.5132375955581665, "loss_scale": 4.0, "retries": 0, "seconds": 526.8, "max_memory_gb": 17.19, "ce": 0.09090911306293492, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.2895828187465668, "loss_scale": 4.0, "retries": 0, "seconds": 531.4, "max_memory_gb": 17.19, "ce": 0.02110088632298357, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.43499431014060974, "loss_scale": 4.0, "retries": 0, "seconds": 535.8, "max_memory_gb": 17.19, "ce": 0.1397289375706805, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.2998093366622925, "loss_scale": 4.0, "retries": 0, "seconds": 540.5, "max_memory_gb": 17.19, "ce": 0.0233654413621025, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 0.889430046081543, "loss_scale": 4.0, "retries": 0, "seconds": 545.3, "max_memory_gb": 17.19, "ce": 0.2641569829722812, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.09742522239685059, "loss_scale": 4.0, "retries": 0, "seconds": 550.1, "max_memory_gb": 17.19, "ce": 0.011338992402670556, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 1.3719749450683594, "loss_scale": 4.0, "retries": 0, "seconds": 555.2, "max_memory_gb": 17.19, "ce": 0.13308736531939758, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.40326401591300964, "loss_scale": 4.0, "retries": 0, "seconds": 560.0, "max_memory_gb": 17.19, "ce": 0.0815274664792014, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.6569191217422485, "loss_scale": 4.0, "retries": 0, "seconds": 564.7, "max_memory_gb": 17.19, "ce": 0.10472898858279223, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.37123048305511475, "loss_scale": 4.0, "retries": 0, "seconds": 569.4, "max_memory_gb": 17.19, "ce": 0.05854639865356148, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.4061568975448608, "loss_scale": 4.0, "retries": 0, "seconds": 573.9, "max_memory_gb": 17.19, "ce": 0.5286254382531297, "rank": 0.06743943691253662}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.5593215227127075, "loss_scale": 4.0, "retries": 0, "seconds": 578.3, "max_memory_gb": 17.19, "ce": 0.1690505114293046, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 1.59359610080719, "loss_scale": 4.0, "retries": 0, "seconds": 582.9, "max_memory_gb": 17.19, "ce": 0.48830310175685554, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.4768345057964325, "loss_scale": 4.0, "retries": 0, "seconds": 587.5, "max_memory_gb": 17.19, "ce": 0.015280707806596183, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 0.9321281313896179, "loss_scale": 4.0, "retries": 0, "seconds": 592.1, "max_memory_gb": 17.19, "ce": 0.18686007049836917, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 1.548304557800293, "loss_scale": 4.0, "retries": 0, "seconds": 596.6, "max_memory_gb": 17.19, "ce": 0.08491856714999813, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 1.0220649242401123, "loss_scale": 4.0, "retries": 0, "seconds": 601.3, "max_memory_gb": 17.19, "ce": 0.1867344625475198, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 1.0574016571044922, "loss_scale": 4.0, "retries": 0, "seconds": 606.1, "max_memory_gb": 17.19, "ce": 0.15936702149474513, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 2.7130279541015625, "loss_scale": 4.0, "retries": 0, "seconds": 611.7, "max_memory_gb": 17.19, "ce": 0.2507234689330744, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 1.1236281394958496, "loss_scale": 4.0, "retries": 0, "seconds": 618.8, "max_memory_gb": 17.19, "ce": 0.27793684422704246, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.9295814633369446, "loss_scale": 4.0, "retries": 0, "seconds": 624.2, "max_memory_gb": 17.19, "ce": 0.20745503309035485, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 0.09352707117795944, "loss_scale": 4.0, "retries": 0, "seconds": 629.0, "max_memory_gb": 17.19, "ce": 0.007485481903586333, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 0.2806827425956726, "loss_scale": 4.0, "retries": 0, "seconds": 633.5, "max_memory_gb": 17.19, "ce": 0.028549922475804124, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.5792230367660522, "loss_scale": 4.0, "retries": 0, "seconds": 638.3, "max_memory_gb": 17.19, "ce": 0.052943405123187404, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 1.7608766555786133, "loss_scale": 4.0, "retries": 0, "seconds": 643.6, "max_memory_gb": 17.19, "ce": 0.11556862016732339, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.3287483751773834, "loss_scale": 4.0, "retries": 0, "seconds": 648.5, "max_memory_gb": 17.19, "ce": 0.03906702639915238, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.04076270014047623, "loss_scale": 4.0, "retries": 0, "seconds": 654.2, "max_memory_gb": 17.19, "ce": 0.00427121874417935, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.04282495751976967, "loss_scale": 4.0, "retries": 0, "seconds": 660.3, "max_memory_gb": 17.19, "ce": 0.005154301456059329, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.4881405532360077, "loss_scale": 4.0, "retries": 0, "seconds": 664.9, "max_memory_gb": 17.19, "ce": 0.04441492185299012, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 1.478092074394226, "loss_scale": 4.0, "retries": 0, "seconds": 669.4, "max_memory_gb": 17.19, "ce": 0.21767422900393285, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.43256014585494995, "loss_scale": 4.0, "retries": 0, "seconds": 673.1, "max_memory_gb": 17.19, "ce": 0.04507621062157341, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.07564446330070496, "loss_scale": 4.0, "retries": 0, "seconds": 677.5, "max_memory_gb": 17.19, "ce": 0.004872088089314275, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.3969058096408844, "loss_scale": 4.0, "retries": 0, "seconds": 681.7, "max_memory_gb": 17.19, "ce": 0.032451319658321154, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 1.3904576301574707, "loss_scale": 4.0, "retries": 0, "seconds": 686.1, "max_memory_gb": 17.19, "ce": 0.3958861487935792, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.18015620112419128, "loss_scale": 4.0, "retries": 0, "seconds": 690.5, "max_memory_gb": 17.19, "ce": 0.01462464553060272, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.09450389444828033, "loss_scale": 4.0, "retries": 0, "seconds": 695.0, "max_memory_gb": 17.19, "ce": 0.01102896196596248, "rank": 0.12419082224369049}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.4297712743282318, "loss_scale": 4.0, "retries": 0, "seconds": 699.6, "max_memory_gb": 17.19, "ce": 0.07531556857611577, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 2.4945261478424072, "loss_scale": 4.0, "retries": 0, "seconds": 704.2, "max_memory_gb": 17.19, "ce": 0.07702095219156035, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 0.6728220582008362, "loss_scale": 4.0, "retries": 0, "seconds": 708.7, "max_memory_gb": 17.19, "ce": 0.05358530893443003, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.927329421043396, "loss_scale": 4.0, "retries": 0, "seconds": 713.2, "max_memory_gb": 17.19, "ce": 0.09881540781896092, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.5468595623970032, "loss_scale": 4.0, "retries": 0, "seconds": 717.9, "max_memory_gb": 17.19, "ce": 0.040146001938410336, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 2.523735761642456, "loss_scale": 4.0, "retries": 0, "seconds": 722.4, "max_memory_gb": 17.19, "ce": 0.24159068898279656, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.30150291323661804, "loss_scale": 4.0, "retries": 0, "seconds": 726.9, "max_memory_gb": 17.19, "ce": 0.013093629466197854, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.18433967232704163, "loss_scale": 4.0, "retries": 0, "seconds": 731.5, "max_memory_gb": 17.19, "ce": 0.016535896565073926, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 3.8258705139160156, "loss_scale": 4.0, "retries": 0, "seconds": 736.1, "max_memory_gb": 17.19, "ce": 0.49591439372625246, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.2537427842617035, "loss_scale": 4.0, "retries": 0, "seconds": 740.7, "max_memory_gb": 17.19, "ce": 0.02291625748966908, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 0.733864963054657, "loss_scale": 4.0, "retries": 0, "seconds": 745.5, "max_memory_gb": 17.19, "ce": 0.1646432314341837, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 1.2894614934921265, "loss_scale": 4.0, "retries": 0, "seconds": 750.3, "max_memory_gb": 17.19, "ce": 0.13331963009659376, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.632707417011261, "loss_scale": 4.0, "retries": 0, "seconds": 754.8, "max_memory_gb": 17.19, "ce": 0.1026409562034587, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 1.692457914352417, "loss_scale": 4.0, "retries": 0, "seconds": 759.5, "max_memory_gb": 17.19, "ce": 0.19161013298980833, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.4905936121940613, "loss_scale": 4.0, "retries": 0, "seconds": 763.8, "max_memory_gb": 17.19, "ce": 0.08382034432042929, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.07694592326879501, "loss_scale": 4.0, "retries": 0, "seconds": 768.5, "max_memory_gb": 17.19, "ce": 0.011621316283026317, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 1.4490776062011719, "loss_scale": 4.0, "retries": 0, "seconds": 773.1, "max_memory_gb": 17.19, "ce": 0.2780028617653443, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.12767043709754944, "loss_scale": 4.0, "retries": 0, "seconds": 777.9, "max_memory_gb": 17.19, "ce": 0.00957099399965955, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.04926569387316704, "loss_scale": 4.0, "retries": 0, "seconds": 782.4, "max_memory_gb": 17.19, "ce": 0.005475287002809637, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 0.8946131467819214, "loss_scale": 4.0, "retries": 0, "seconds": 787.0, "max_memory_gb": 17.19, "ce": 0.27717809281784866, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.053948406130075455, "loss_scale": 4.0, "retries": 0, "seconds": 791.9, "max_memory_gb": 17.19, "ce": 0.0024434093711533933, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.3068870007991791, "loss_scale": 4.0, "retries": 0, "seconds": 796.8, "max_memory_gb": 17.19, "ce": 0.05871649100117793, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.18646535277366638, "loss_scale": 4.0, "retries": 0, "seconds": 801.4, "max_memory_gb": 17.19, "ce": 0.01164364944997942, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.31489166617393494, "loss_scale": 4.0, "retries": 0, "seconds": 805.6, "max_memory_gb": 17.19, "ce": 0.01633586746174842, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.7025015950202942, "loss_scale": 4.0, "retries": 0, "seconds": 810.3, "max_memory_gb": 17.19, "ce": 0.036776008499145973, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 1.3579994440078735, "loss_scale": 4.0, "retries": 0, "seconds": 815.1, "max_memory_gb": 17.19, "ce": 0.17991233885550173, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.026688575744628906, "loss_scale": 4.0, "retries": 0, "seconds": 819.7, "max_memory_gb": 17.19, "ce": 0.0017893550721055362, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 1.0816994905471802, "loss_scale": 4.0, "retries": 0, "seconds": 824.7, "max_memory_gb": 17.19, "ce": 0.34761629339254796, "rank": 0.12493220716714859}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 0.28609028458595276, "loss_scale": 4.0, "retries": 0, "seconds": 829.5, "max_memory_gb": 17.19, "ce": 0.06315181206264242, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 0.7398937940597534, "loss_scale": 4.0, "retries": 0, "seconds": 834.2, "max_memory_gb": 17.19, "ce": 0.35643701279559536, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.46639344096183777, "loss_scale": 4.0, "retries": 0, "seconds": 839.3, "max_memory_gb": 17.19, "ce": 0.028104951011300727, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.2622377276420593, "loss_scale": 4.0, "retries": 0, "seconds": 845.2, "max_memory_gb": 17.19, "ce": 0.012726858126370644, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 0.857897937297821, "loss_scale": 4.0, "retries": 0, "seconds": 850.6, "max_memory_gb": 17.19, "ce": 0.10261595079555264, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.14839443564414978, "loss_scale": 4.0, "retries": 0, "seconds": 855.5, "max_memory_gb": 17.19, "ce": 0.008149185015554394, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.23958346247673035, "loss_scale": 4.0, "retries": 0, "seconds": 860.4, "max_memory_gb": 17.19, "ce": 0.034986553706403356, "rank": 0.20766225457191467}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.6709228754043579, "loss_scale": 4.0, "retries": 0, "seconds": 865.0, "max_memory_gb": 17.19, "ce": 0.05573529190951376, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 0.7468692064285278, "loss_scale": 4.0, "retries": 0, "seconds": 869.6, "max_memory_gb": 17.19, "ce": 0.12576916419129702, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.8565695881843567, "loss_scale": 4.0, "retries": 0, "seconds": 874.5, "max_memory_gb": 17.19, "ce": 0.1389856946771033, "rank": 0.14922110736370087}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 0.9262624382972717, "loss_scale": 4.0, "retries": 0, "seconds": 879.1, "max_memory_gb": 17.19, "ce": 0.2601974939502725, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 1.310255765914917, "loss_scale": 4.0, "retries": 0, "seconds": 883.7, "max_memory_gb": 17.19, "ce": 0.33814340656931563, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.23044680058956146, "loss_scale": 4.0, "retries": 0, "seconds": 889.0, "max_memory_gb": 17.19, "ce": 0.012385853370687983, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 0.9900158643722534, "loss_scale": 4.0, "retries": 0, "seconds": 893.6, "max_memory_gb": 17.19, "ce": 0.170945272257768, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.0256959199905396, "loss_scale": 4.0, "retries": 0, "seconds": 898.2, "max_memory_gb": 17.19, "ce": 0.3881945952875867, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.023375052958726883, "loss_scale": 4.0, "retries": 0, "seconds": 903.0, "max_memory_gb": 17.19, "ce": 0.0017694640564513975, "rank": 0.12506547570228577}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.3786547780036926, "loss_scale": 4.0, "retries": 0, "seconds": 907.5, "max_memory_gb": 17.19, "ce": 0.05821258862192735, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.3457748591899872, "loss_scale": 4.0, "retries": 0, "seconds": 912.0, "max_memory_gb": 17.19, "ce": 0.06655196714780232, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.19476976990699768, "loss_scale": 4.0, "retries": 0, "seconds": 916.7, "max_memory_gb": 17.19, "ce": 0.010066436017496017, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.977434515953064, "loss_scale": 4.0, "retries": 0, "seconds": 921.4, "max_memory_gb": 17.19, "ce": 0.22746115370591724, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.41220876574516296, "loss_scale": 4.0, "retries": 0, "seconds": 925.8, "max_memory_gb": 17.19, "ce": 0.06418315784105744, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 2.8187310695648193, "loss_scale": 4.0, "retries": 0, "seconds": 930.4, "max_memory_gb": 17.19, "ce": 0.18343535019994306, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 0.7081598043441772, "loss_scale": 4.0, "retries": 0, "seconds": 935.0, "max_memory_gb": 17.19, "ce": 0.22860935351081935, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.25505876541137695, "loss_scale": 4.0, "retries": 0, "seconds": 939.4, "max_memory_gb": 17.19, "ce": 0.047092095402149425, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.0351136140525341, "loss_scale": 4.0, "retries": 0, "seconds": 943.9, "max_memory_gb": 17.19, "ce": 0.0030590077426495554, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.11041812598705292, "loss_scale": 4.0, "retries": 0, "seconds": 948.6, "max_memory_gb": 17.19, "ce": 0.009543437857246317, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.27585384249687195, "loss_scale": 4.0, "retries": 0, "seconds": 953.2, "max_memory_gb": 17.19, "ce": 0.020683270332938264, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 0.57818204164505, "loss_scale": 4.0, "retries": 0, "seconds": 957.8, "max_memory_gb": 17.19, "ce": 0.0950856316771933, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.32655537128448486, "loss_scale": 4.0, "retries": 0, "seconds": 962.5, "max_memory_gb": 17.19, "ce": 0.03646915419790275, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.27641671895980835, "loss_scale": 4.0, "retries": 0, "seconds": 966.9, "max_memory_gb": 17.19, "ce": 0.05782961654472274, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.16621290147304535, "loss_scale": 4.0, "retries": 0, "seconds": 971.5, "max_memory_gb": 17.19, "ce": 0.008953676726605408, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.43404319882392883, "loss_scale": 4.0, "retries": 0, "seconds": 976.3, "max_memory_gb": 17.19, "ce": 0.10723159652934555, "rank": 0.12499219924211502}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 1.0212846994400024, "loss_scale": 4.0, "retries": 0, "seconds": 980.8, "max_memory_gb": 17.19, "ce": 0.374537614526389, "rank": 0.11107176542282104}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.4110061526298523, "loss_scale": 4.0, "retries": 0, "seconds": 985.4, "max_memory_gb": 17.19, "ce": 0.11238864760116485, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 0.9089711308479309, "loss_scale": 4.0, "retries": 0, "seconds": 989.9, "max_memory_gb": 17.19, "ce": 0.18555024159536515, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.542027473449707, "loss_scale": 4.0, "retries": 0, "seconds": 994.6, "max_memory_gb": 17.19, "ce": 0.1302504112738916, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 1.3009746074676514, "loss_scale": 4.0, "retries": 0, "seconds": 999.0, "max_memory_gb": 17.19, "ce": 0.1869786086822387, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 1.0347089767456055, "loss_scale": 4.0, "retries": 0, "seconds": 1003.4, "max_memory_gb": 17.19, "ce": 0.3917109649092936, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.523189902305603, "loss_scale": 4.0, "retries": 0, "seconds": 1008.1, "max_memory_gb": 17.19, "ce": 0.042706817014732223, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.06284260004758835, "loss_scale": 4.0, "retries": 0, "seconds": 1012.7, "max_memory_gb": 17.19, "ce": 0.003609864638349336, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 0.598121702671051, "loss_scale": 4.0, "retries": 0, "seconds": 1017.2, "max_memory_gb": 17.19, "ce": 0.18942612651119362, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.1576223522424698, "loss_scale": 4.0, "retries": 0, "seconds": 1021.9, "max_memory_gb": 17.19, "ce": 0.025331975153676467, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.2257683426141739, "loss_scale": 4.0, "retries": 0, "seconds": 1026.4, "max_memory_gb": 17.19, "ce": 0.014967640800364279, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.395423024892807, "loss_scale": 4.0, "retries": 0, "seconds": 1031.0, "max_memory_gb": 17.19, "ce": 0.028367362695234988, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.24712367355823517, "loss_scale": 4.0, "retries": 0, "seconds": 1035.8, "max_memory_gb": 17.19, "ce": 0.026863605860512507, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 1.006341576576233, "loss_scale": 4.0, "retries": 0, "seconds": 1040.2, "max_memory_gb": 17.19, "ce": 0.08649895612165892, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.28498905897140503, "loss_scale": 4.0, "retries": 0, "seconds": 1045.0, "max_memory_gb": 17.19, "ce": 0.031461731791750935, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.15616288781166077, "loss_scale": 4.0, "retries": 0, "seconds": 1049.8, "max_memory_gb": 17.19, "ce": 0.008554475718028698, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.5341545939445496, "loss_scale": 4.0, "retries": 0, "seconds": 1054.4, "max_memory_gb": 17.19, "ce": 0.04706432206239697, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.5153979063034058, "loss_scale": 4.0, "retries": 0, "seconds": 1058.8, "max_memory_gb": 17.19, "ce": 0.05698864970509021, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.10610804706811905, "loss_scale": 4.0, "retries": 0, "seconds": 1063.3, "max_memory_gb": 17.19, "ce": 0.01037320981083667, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.3971869647502899, "loss_scale": 4.0, "retries": 0, "seconds": 1068.2, "max_memory_gb": 17.19, "ce": 0.059616470788796505, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 1.524552822113037, "loss_scale": 4.0, "retries": 0, "seconds": 1072.4, "max_memory_gb": 17.19, "ce": 0.3912537476840612, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.25172674655914307, "loss_scale": 4.0, "retries": 0, "seconds": 1076.9, "max_memory_gb": 17.19, "ce": 0.03273441530143373, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.39005473256111145, "loss_scale": 4.0, "retries": 0, "seconds": 1081.7, "max_memory_gb": 17.19, "ce": 0.031512724716094453, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.2505607306957245, "loss_scale": 4.0, "retries": 0, "seconds": 1086.3, "max_memory_gb": 17.19, "ce": 0.028542717841446574, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.40657109022140503, "loss_scale": 4.0, "retries": 0, "seconds": 1090.8, "max_memory_gb": 17.19, "ce": 0.04516646837885219, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 1.3137736320495605, "loss_scale": 4.0, "retries": 0, "seconds": 1095.5, "max_memory_gb": 17.19, "ce": 0.26879256546908437, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.12848176062107086, "loss_scale": 4.0, "retries": 0, "seconds": 1099.9, "max_memory_gb": 17.19, "ce": 0.01586347355066664, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.421267032623291, "loss_scale": 4.0, "retries": 0, "seconds": 1104.5, "max_memory_gb": 17.19, "ce": 0.0676416419717043, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.2250811755657196, "loss_scale": 4.0, "retries": 0, "seconds": 1109.1, "max_memory_gb": 17.19, "ce": 0.03172509111834643, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.38538530468940735, "loss_scale": 4.0, "retries": 0, "seconds": 1113.7, "max_memory_gb": 17.19, "ce": 0.0898574055897825, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 0.3274117410182953, "loss_scale": 4.0, "retries": 0, "seconds": 1118.1, "max_memory_gb": 17.19, "ce": 0.016743821754289456, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.672434389591217, "loss_scale": 4.0, "retries": 0, "seconds": 1122.6, "max_memory_gb": 17.19, "ce": 0.14636038946287044, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.12827254831790924, "loss_scale": 4.0, "retries": 0, "seconds": 1127.2, "max_memory_gb": 17.19, "ce": 0.012787206427560704, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.06693265587091446, "loss_scale": 4.0, "retries": 0, "seconds": 1131.7, "max_memory_gb": 17.19, "ce": 0.0029151155418958297, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.6144866347312927, "loss_scale": 4.0, "retries": 0, "seconds": 1136.2, "max_memory_gb": 17.19, "ce": 0.05002486377259174, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 0.8692854642868042, "loss_scale": 4.0, "retries": 0, "seconds": 1140.8, "max_memory_gb": 17.19, "ce": 0.27865288024560186, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.05273432657122612, "loss_scale": 4.0, "retries": 0, "seconds": 1145.3, "max_memory_gb": 17.19, "ce": 0.0028384878249880785, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.8034098148345947, "loss_scale": 4.0, "retries": 0, "seconds": 1149.7, "max_memory_gb": 17.19, "ce": 0.05843233461030195, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.11430784314870834, "loss_scale": 4.0, "retries": 0, "seconds": 1154.1, "max_memory_gb": 17.19, "ce": 0.008955398735793096, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.17805621027946472, "loss_scale": 4.0, "retries": 0, "seconds": 1158.5, "max_memory_gb": 17.19, "ce": 0.015019058964981014, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.3403255343437195, "loss_scale": 4.0, "retries": 0, "seconds": 1162.9, "max_memory_gb": 17.19, "ce": 0.02251124089912082, "rank": 0.0}

runs/fixed_internvl/removal/Deep8_s42/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/removal/Deep8_s42/adapter/adapter_config.json

@@ -0,0 +1,213 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.28.mlp.down_proj": 144,
+    "model.language_model.layers.28.mlp.gate_proj": 144,
+    "model.language_model.layers.28.mlp.up_proj": 144,
+    "model.language_model.layers.28.self_attn.k_proj": 144,
+    "model.language_model.layers.28.self_attn.o_proj": 144,
+    "model.language_model.layers.28.self_attn.q_proj": 144,
+    "model.language_model.layers.28.self_attn.v_proj": 144,
+    "model.language_model.layers.29.mlp.down_proj": 144,
+    "model.language_model.layers.29.mlp.gate_proj": 144,
+    "model.language_model.layers.29.mlp.up_proj": 144,
+    "model.language_model.layers.29.self_attn.k_proj": 144,
+    "model.language_model.layers.29.self_attn.o_proj": 144,
+    "model.language_model.layers.29.self_attn.q_proj": 144,
+    "model.language_model.layers.29.self_attn.v_proj": 144,
+    "model.language_model.layers.30.mlp.down_proj": 144,
+    "model.language_model.layers.30.mlp.gate_proj": 144,
+    "model.language_model.layers.30.mlp.up_proj": 144,
+    "model.language_model.layers.30.self_attn.k_proj": 144,
+    "model.language_model.layers.30.self_attn.o_proj": 144,
+    "model.language_model.layers.30.self_attn.q_proj": 144,
+    "model.language_model.layers.30.self_attn.v_proj": 144,
+    "model.language_model.layers.31.mlp.down_proj": 144,
+    "model.language_model.layers.31.mlp.gate_proj": 144,
+    "model.language_model.layers.31.mlp.up_proj": 144,
+    "model.language_model.layers.31.self_attn.k_proj": 144,
+    "model.language_model.layers.31.self_attn.o_proj": 144,
+    "model.language_model.layers.31.self_attn.q_proj": 144,
+    "model.language_model.layers.31.self_attn.v_proj": 144,
+    "model.language_model.layers.32.mlp.down_proj": 144,
+    "model.language_model.layers.32.mlp.gate_proj": 144,
+    "model.language_model.layers.32.mlp.up_proj": 144,
+    "model.language_model.layers.32.self_attn.k_proj": 144,
+    "model.language_model.layers.32.self_attn.o_proj": 144,
+    "model.language_model.layers.32.self_attn.q_proj": 144,
+    "model.language_model.layers.32.self_attn.v_proj": 144,
+    "model.language_model.layers.33.mlp.down_proj": 144,
+    "model.language_model.layers.33.mlp.gate_proj": 144,
+    "model.language_model.layers.33.mlp.up_proj": 144,
+    "model.language_model.layers.33.self_attn.k_proj": 144,
+    "model.language_model.layers.33.self_attn.o_proj": 144,
+    "model.language_model.layers.33.self_attn.q_proj": 144,
+    "model.language_model.layers.33.self_attn.v_proj": 144,
+    "model.language_model.layers.34.mlp.down_proj": 144,
+    "model.language_model.layers.34.mlp.gate_proj": 144,
+    "model.language_model.layers.34.mlp.up_proj": 144,
+    "model.language_model.layers.34.self_attn.k_proj": 144,
+    "model.language_model.layers.34.self_attn.o_proj": 144,
+    "model.language_model.layers.34.self_attn.q_proj": 144,
+    "model.language_model.layers.34.self_attn.v_proj": 144,
+    "model.language_model.layers.35.mlp.down_proj": 144,
+    "model.language_model.layers.35.mlp.gate_proj": 144,
+    "model.language_model.layers.35.mlp.up_proj": 144,
+    "model.language_model.layers.35.self_attn.k_proj": 144,
+    "model.language_model.layers.35.self_attn.o_proj": 144,
+    "model.language_model.layers.35.self_attn.q_proj": 144,
+    "model.language_model.layers.35.self_attn.v_proj": 144
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.28.mlp.down_proj": 72,
+    "model.language_model.layers.28.mlp.gate_proj": 72,
+    "model.language_model.layers.28.mlp.up_proj": 72,
+    "model.language_model.layers.28.self_attn.k_proj": 72,
+    "model.language_model.layers.28.self_attn.o_proj": 72,
+    "model.language_model.layers.28.self_attn.q_proj": 72,
+    "model.language_model.layers.28.self_attn.v_proj": 72,
+    "model.language_model.layers.29.mlp.down_proj": 72,
+    "model.language_model.layers.29.mlp.gate_proj": 72,
+    "model.language_model.layers.29.mlp.up_proj": 72,
+    "model.language_model.layers.29.self_attn.k_proj": 72,
+    "model.language_model.layers.29.self_attn.o_proj": 72,
+    "model.language_model.layers.29.self_attn.q_proj": 72,
+    "model.language_model.layers.29.self_attn.v_proj": 72,
+    "model.language_model.layers.30.mlp.down_proj": 72,
+    "model.language_model.layers.30.mlp.gate_proj": 72,
+    "model.language_model.layers.30.mlp.up_proj": 72,
+    "model.language_model.layers.30.self_attn.k_proj": 72,
+    "model.language_model.layers.30.self_attn.o_proj": 72,
+    "model.language_model.layers.30.self_attn.q_proj": 72,
+    "model.language_model.layers.30.self_attn.v_proj": 72,
+    "model.language_model.layers.31.mlp.down_proj": 72,
+    "model.language_model.layers.31.mlp.gate_proj": 72,
+    "model.language_model.layers.31.mlp.up_proj": 72,
+    "model.language_model.layers.31.self_attn.k_proj": 72,
+    "model.language_model.layers.31.self_attn.o_proj": 72,
+    "model.language_model.layers.31.self_attn.q_proj": 72,
+    "model.language_model.layers.31.self_attn.v_proj": 72,
+    "model.language_model.layers.32.mlp.down_proj": 72,
+    "model.language_model.layers.32.mlp.gate_proj": 72,
+    "model.language_model.layers.32.mlp.up_proj": 72,
+    "model.language_model.layers.32.self_attn.k_proj": 72,
+    "model.language_model.layers.32.self_attn.o_proj": 72,
+    "model.language_model.layers.32.self_attn.q_proj": 72,
+    "model.language_model.layers.32.self_attn.v_proj": 72,
+    "model.language_model.layers.33.mlp.down_proj": 72,
+    "model.language_model.layers.33.mlp.gate_proj": 72,
+    "model.language_model.layers.33.mlp.up_proj": 72,
+    "model.language_model.layers.33.self_attn.k_proj": 72,
+    "model.language_model.layers.33.self_attn.o_proj": 72,
+    "model.language_model.layers.33.self_attn.q_proj": 72,
+    "model.language_model.layers.33.self_attn.v_proj": 72,
+    "model.language_model.layers.34.mlp.down_proj": 72,
+    "model.language_model.layers.34.mlp.gate_proj": 72,
+    "model.language_model.layers.34.mlp.up_proj": 72,
+    "model.language_model.layers.34.self_attn.k_proj": 72,
+    "model.language_model.layers.34.self_attn.o_proj": 72,
+    "model.language_model.layers.34.self_attn.q_proj": 72,
+    "model.language_model.layers.34.self_attn.v_proj": 72,
+    "model.language_model.layers.35.mlp.down_proj": 72,
+    "model.language_model.layers.35.mlp.gate_proj": 72,
+    "model.language_model.layers.35.mlp.up_proj": 72,
+    "model.language_model.layers.35.self_attn.k_proj": 72,
+    "model.language_model.layers.35.self_attn.o_proj": 72,
+    "model.language_model.layers.35.self_attn.q_proj": 72,
+    "model.language_model.layers.35.self_attn.v_proj": 72
+  },
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.32.mlp.gate_proj",
+    "model.language_model.layers.30.self_attn.q_proj",
+    "model.language_model.layers.35.self_attn.v_proj",
+    "model.language_model.layers.34.self_attn.v_proj",
+    "model.language_model.layers.29.mlp.gate_proj",
+    "model.language_model.layers.29.self_attn.o_proj",
+    "model.language_model.layers.30.self_attn.o_proj",
+    "model.language_model.layers.33.mlp.down_proj",
+    "model.language_model.layers.30.mlp.up_proj",
+    "model.language_model.layers.28.self_attn.o_proj",
+    "model.language_model.layers.31.mlp.gate_proj",
+    "model.language_model.layers.29.self_attn.k_proj",
+    "model.language_model.layers.29.self_attn.q_proj",
+    "model.language_model.layers.33.self_attn.v_proj",
+    "model.language_model.layers.30.self_attn.k_proj",
+    "model.language_model.layers.35.mlp.gate_proj",
+    "model.language_model.layers.35.self_attn.o_proj",
+    "model.language_model.layers.28.self_attn.q_proj",
+    "model.language_model.layers.34.self_attn.o_proj",
+    "model.language_model.layers.35.mlp.down_proj",
+    "model.language_model.layers.31.self_attn.o_proj",
+    "model.language_model.layers.32.mlp.down_proj",
+    "model.language_model.layers.34.self_attn.k_proj",
+    "model.language_model.layers.28.mlp.gate_proj",
+    "model.language_model.layers.28.mlp.up_proj",
+    "model.language_model.layers.33.mlp.up_proj",
+    "model.language_model.layers.32.self_attn.q_proj",
+    "model.language_model.layers.31.self_attn.k_proj",
+    "model.language_model.layers.32.self_attn.v_proj",
+    "model.language_model.layers.30.mlp.gate_proj",
+    "model.language_model.layers.30.mlp.down_proj",
+    "model.language_model.layers.33.self_attn.k_proj",
+    "model.language_model.layers.30.self_attn.v_proj",
+    "model.language_model.layers.33.mlp.gate_proj",
+    "model.language_model.layers.29.mlp.down_proj",
+    "model.language_model.layers.28.self_attn.k_proj",
+    "model.language_model.layers.35.mlp.up_proj",
+    "model.language_model.layers.33.self_attn.o_proj",
+    "model.language_model.layers.29.self_attn.v_proj",
+    "model.language_model.layers.34.mlp.gate_proj",
+    "model.language_model.layers.32.self_attn.o_proj",
+    "model.language_model.layers.34.mlp.up_proj",
+    "model.language_model.layers.29.mlp.up_proj",
+    "model.language_model.layers.28.self_attn.v_proj",
+    "model.language_model.layers.31.self_attn.q_proj",
+    "model.language_model.layers.32.self_attn.k_proj",
+    "model.language_model.layers.32.mlp.up_proj",
+    "model.language_model.layers.33.self_attn.q_proj",
+    "model.language_model.layers.28.mlp.down_proj",
+    "model.language_model.layers.34.self_attn.q_proj",
+    "model.language_model.layers.35.self_attn.k_proj",
+    "model.language_model.layers.31.mlp.down_proj",
+    "model.language_model.layers.34.mlp.down_proj",
+    "model.language_model.layers.31.mlp.up_proj",
+    "model.language_model.layers.35.self_attn.q_proj",
+    "model.language_model.layers.31.self_attn.v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/fixed_internvl/removal/Deep8_s42/config.json

@@ -0,0 +1,43 @@
+{
+  "behavior": "removal",
+  "seed": 42,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "world_size": 1,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "arm": {
+    "arm": "Deep8",
+    "layers": [
+      28,
+      29,
+      30,
+      31,
+      32,
+      33,
+      34,
+      35
+    ],
+    "ranks": {
+      "28": 72,
+      "29": 72,
+      "30": 72,
+      "31": 72,
+      "32": 72,
+      "33": 72,
+      "34": 72,
+      "35": 72
+    },
+    "parameters": 43646976,
+    "budget": 43646976
+  }
+}

runs/fixed_internvl/removal/Deep8_s42/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 3.5928497314453125, "loss_scale": 4.0, "retries": 0, "seconds": 4.6, "max_memory_gb": 16.83, "ce": 1.0494785578921437, "rank": 0.15784265100955963}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 2.62544584274292, "loss_scale": 4.0, "retries": 0, "seconds": 8.1, "max_memory_gb": 17.17, "ce": 0.8269532863961331, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 3.4363298416137695, "loss_scale": 4.0, "retries": 0, "seconds": 12.0, "max_memory_gb": 17.17, "ce": 0.8293758119398262, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 2.692655086517334, "loss_scale": 4.0, "retries": 0, "seconds": 16.4, "max_memory_gb": 17.17, "ce": 0.918099548893224, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 1.9858918190002441, "loss_scale": 4.0, "retries": 0, "seconds": 21.0, "max_memory_gb": 17.17, "ce": 0.7018670355319045, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 2.662168264389038, "loss_scale": 4.0, "retries": 0, "seconds": 25.5, "max_memory_gb": 17.17, "ce": 0.6125597203354118, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 3.309232711791992, "loss_scale": 4.0, "retries": 0, "seconds": 30.0, "max_memory_gb": 17.17, "ce": 0.9183178474195302, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 0.38363566994667053, "loss_scale": 4.0, "retries": 0, "seconds": 34.2, "max_memory_gb": 17.17, "ce": 0.23871546569216662, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 1.3605924844741821, "loss_scale": 4.0, "retries": 0, "seconds": 38.1, "max_memory_gb": 17.17, "ce": 0.6319025885313749, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 3.651216983795166, "loss_scale": 4.0, "retries": 0, "seconds": 41.9, "max_memory_gb": 17.17, "ce": 0.6380606597958831, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.8924686312675476, "loss_scale": 4.0, "retries": 0, "seconds": 45.7, "max_memory_gb": 17.17, "ce": 0.20003985371886301, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 1.6942071914672852, "loss_scale": 4.0, "retries": 0, "seconds": 49.6, "max_memory_gb": 17.17, "ce": 0.6944043247694935, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 0.9722230434417725, "loss_scale": 4.0, "retries": 0, "seconds": 53.5, "max_memory_gb": 17.17, "ce": 0.41953259077854455, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 1.0049196481704712, "loss_scale": 4.0, "retries": 0, "seconds": 58.0, "max_memory_gb": 17.17, "ce": 0.37492540158564225, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 2.9516804218292236, "loss_scale": 4.0, "retries": 0, "seconds": 62.0, "max_memory_gb": 17.17, "ce": 0.6406988694798201, "rank": 0.04516996443271637}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 2.852262496948242, "loss_scale": 4.0, "retries": 0, "seconds": 65.8, "max_memory_gb": 17.17, "ce": 0.48949069462560146, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.6384485960006714, "loss_scale": 4.0, "retries": 0, "seconds": 69.6, "max_memory_gb": 17.17, "ce": 0.2706412187544629, "rank": 0.10819727927446365}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 0.7079763412475586, "loss_scale": 4.0, "retries": 0, "seconds": 73.9, "max_memory_gb": 17.17, "ce": 0.280787966796197, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 0.39622023701667786, "loss_scale": 4.0, "retries": 0, "seconds": 77.8, "max_memory_gb": 17.17, "ce": 0.15229908575383888, "rank": 0.0}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.8910082578659058, "loss_scale": 4.0, "retries": 0, "seconds": 81.6, "max_memory_gb": 17.17, "ce": 0.18590117164421827, "rank": 0.050916463136672974}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.25435614585876465, "loss_scale": 4.0, "retries": 0, "seconds": 85.3, "max_memory_gb": 17.17, "ce": 0.04464612151059555, "rank": 0.0}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.7877507209777832, "loss_scale": 4.0, "retries": 0, "seconds": 89.2, "max_memory_gb": 17.17, "ce": 0.19423101109168783, "rank": 0.0}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 0.965488612651825, "loss_scale": 4.0, "retries": 0, "seconds": 93.0, "max_memory_gb": 17.17, "ce": 0.21759229103918187, "rank": 0.0}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 1.4902585744857788, "loss_scale": 4.0, "retries": 0, "seconds": 96.7, "max_memory_gb": 17.17, "ce": 0.19088635728985537, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 1.4552547931671143, "loss_scale": 4.0, "retries": 0, "seconds": 100.6, "max_memory_gb": 17.17, "ce": 0.11284319250262342, "rank": 0.04245065897703171}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 0.3463004529476166, "loss_scale": 4.0, "retries": 0, "seconds": 104.4, "max_memory_gb": 17.17, "ce": 0.04790731852699537, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 3.3034517765045166, "loss_scale": 4.0, "retries": 0, "seconds": 108.1, "max_memory_gb": 17.17, "ce": 0.41674875224998686, "rank": 0.0}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 7.560912609100342, "loss_scale": 4.0, "retries": 0, "seconds": 112.0, "max_memory_gb": 17.17, "ce": 0.3693556410908059, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 1.625077724456787, "loss_scale": 4.0, "retries": 0, "seconds": 115.9, "max_memory_gb": 17.17, "ce": 0.16273016677587293, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 11.574764251708984, "loss_scale": 4.0, "retries": 0, "seconds": 119.7, "max_memory_gb": 17.17, "ce": 0.6042625004629372, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 3.203517436981201, "loss_scale": 4.0, "retries": 0, "seconds": 123.5, "max_memory_gb": 17.17, "ce": 0.5673908626340562, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 2.1700823307037354, "loss_scale": 4.0, "retries": 0, "seconds": 127.4, "max_memory_gb": 17.17, "ce": 0.09395865241822321, "rank": 0.0}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.18552514910697937, "loss_scale": 4.0, "retries": 0, "seconds": 131.1, "max_memory_gb": 17.17, "ce": 0.019774639689785545, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.16945494711399078, "loss_scale": 4.0, "retries": 0, "seconds": 134.9, "max_memory_gb": 17.18, "ce": 0.011908096057595685, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.6088302135467529, "loss_scale": 4.0, "retries": 0, "seconds": 138.8, "max_memory_gb": 17.18, "ce": 0.07726091704898863, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 0.981393039226532, "loss_scale": 4.0, "retries": 0, "seconds": 142.6, "max_memory_gb": 17.18, "ce": 0.29746167336452345, "rank": 0.041596874594688416}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 2.2405879497528076, "loss_scale": 4.0, "retries": 0, "seconds": 146.3, "max_memory_gb": 17.18, "ce": 0.24442870391794713, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.3752487599849701, "loss_scale": 4.0, "retries": 0, "seconds": 150.2, "max_memory_gb": 17.18, "ce": 0.035444889013888314, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 2.9018712043762207, "loss_scale": 4.0, "retries": 0, "seconds": 153.9, "max_memory_gb": 17.18, "ce": 0.257311339082662, "rank": 0.0}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.2312551885843277, "loss_scale": 4.0, "retries": 0, "seconds": 157.6, "max_memory_gb": 17.18, "ce": 0.02521385262843978, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.1866355836391449, "loss_scale": 4.0, "retries": 0, "seconds": 161.5, "max_memory_gb": 17.18, "ce": 0.017809640936320648, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.9170281887054443, "loss_scale": 4.0, "retries": 0, "seconds": 165.3, "max_memory_gb": 17.18, "ce": 0.12992010898597073, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.7504344582557678, "loss_scale": 4.0, "retries": 0, "seconds": 169.0, "max_memory_gb": 17.18, "ce": 0.08016240593133261, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 1.959059476852417, "loss_scale": 4.0, "retries": 0, "seconds": 172.8, "max_memory_gb": 17.18, "ce": 0.37628533421229804, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 1.860151767730713, "loss_scale": 4.0, "retries": 0, "seconds": 176.7, "max_memory_gb": 17.18, "ce": 0.3171013137034606, "rank": 0.038560979068279266}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.9803328514099121, "loss_scale": 4.0, "retries": 0, "seconds": 180.4, "max_memory_gb": 17.18, "ce": 0.2609523044229718, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.15229669213294983, "loss_scale": 4.0, "retries": 0, "seconds": 184.2, "max_memory_gb": 17.18, "ce": 0.008494449110003188, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.8352135419845581, "loss_scale": 4.0, "retries": 0, "seconds": 188.0, "max_memory_gb": 17.18, "ce": 0.13901466892275494, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 2.098327875137329, "loss_scale": 4.0, "retries": 0, "seconds": 191.8, "max_memory_gb": 17.18, "ce": 0.1398436587624019, "rank": 0.12499351799488068}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.4389934241771698, "loss_scale": 4.0, "retries": 0, "seconds": 195.5, "max_memory_gb": 17.18, "ce": 0.037647759570972994, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 0.9645562171936035, "loss_scale": 4.0, "retries": 0, "seconds": 199.4, "max_memory_gb": 17.18, "ce": 0.09128142044937704, "rank": 0.03444526344537735}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.07511485368013382, "loss_scale": 4.0, "retries": 0, "seconds": 203.1, "max_memory_gb": 17.18, "ce": 0.007227269148643245, "rank": 0.12558028101921082}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 0.12144222855567932, "loss_scale": 4.0, "retries": 0, "seconds": 206.9, "max_memory_gb": 17.18, "ce": 0.015283969152733334, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 0.47800713777542114, "loss_scale": 4.0, "retries": 0, "seconds": 210.6, "max_memory_gb": 17.18, "ce": 0.0997800142467895, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 1.7509044408798218, "loss_scale": 4.0, "retries": 0, "seconds": 214.4, "max_memory_gb": 17.18, "ce": 0.2658736083603799, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.5735439658164978, "loss_scale": 4.0, "retries": 0, "seconds": 218.1, "max_memory_gb": 17.18, "ce": 0.039835190400481224, "rank": 0.0}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.5649451613426208, "loss_scale": 4.0, "retries": 0, "seconds": 221.8, "max_memory_gb": 17.18, "ce": 0.050096153501272056, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.7923935651779175, "loss_scale": 4.0, "retries": 0, "seconds": 225.7, "max_memory_gb": 17.18, "ce": 0.06719749004332698, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.08329164236783981, "loss_scale": 4.0, "retries": 0, "seconds": 229.4, "max_memory_gb": 17.18, "ce": 0.005805397366202669, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.567791223526001, "loss_scale": 4.0, "retries": 0, "seconds": 233.1, "max_memory_gb": 17.18, "ce": 0.11644622147650807, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.4051273465156555, "loss_scale": 4.0, "retries": 0, "seconds": 237.0, "max_memory_gb": 17.18, "ce": 0.02694380584944156, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.7459110021591187, "loss_scale": 4.0, "retries": 0, "seconds": 240.8, "max_memory_gb": 17.18, "ce": 0.04386004188563675, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.9797006249427795, "loss_scale": 4.0, "retries": 0, "seconds": 244.5, "max_memory_gb": 17.18, "ce": 0.19113128362391762, "rank": 0.0}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.31034398078918457, "loss_scale": 4.0, "retries": 0, "seconds": 248.4, "max_memory_gb": 17.18, "ce": 0.05194177335306449, "rank": 0.0}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 2.1234095096588135, "loss_scale": 4.0, "retries": 0, "seconds": 252.1, "max_memory_gb": 17.18, "ce": 0.4701519418558746, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.16896481812000275, "loss_scale": 4.0, "retries": 0, "seconds": 255.8, "max_memory_gb": 17.18, "ce": 0.02139597693712858, "rank": 0.0}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.4054163992404938, "loss_scale": 4.0, "retries": 0, "seconds": 259.6, "max_memory_gb": 17.18, "ce": 0.014732017458754854, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.166154146194458, "loss_scale": 4.0, "retries": 0, "seconds": 263.4, "max_memory_gb": 17.18, "ce": 0.017928207665818263, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 1.0606162548065186, "loss_scale": 4.0, "retries": 0, "seconds": 267.1, "max_memory_gb": 17.18, "ce": 0.04756512326093798, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 1.3303321599960327, "loss_scale": 4.0, "retries": 0, "seconds": 270.9, "max_memory_gb": 17.18, "ce": 0.21807992421270228, "rank": 0.0}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 1.7583674192428589, "loss_scale": 4.0, "retries": 0, "seconds": 274.6, "max_memory_gb": 17.18, "ce": 0.28236321845315615, "rank": 0.11508232355117798}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.0537932813167572, "loss_scale": 4.0, "retries": 0, "seconds": 278.4, "max_memory_gb": 17.18, "ce": 0.007246022684284981, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 1.46067476272583, "loss_scale": 4.0, "retries": 0, "seconds": 282.2, "max_memory_gb": 17.18, "ce": 0.358526711268496, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.3358597755432129, "loss_scale": 4.0, "retries": 0, "seconds": 286.0, "max_memory_gb": 17.18, "ce": 0.05351063640546272, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 2.39221453666687, "loss_scale": 4.0, "retries": 0, "seconds": 289.6, "max_memory_gb": 17.18, "ce": 0.22197217128041302, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 0.8915339112281799, "loss_scale": 4.0, "retries": 0, "seconds": 293.4, "max_memory_gb": 17.18, "ce": 0.04723127983743325, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 1.2604261636734009, "loss_scale": 4.0, "retries": 0, "seconds": 297.3, "max_memory_gb": 17.18, "ce": 0.06700504983791689, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 1.6029688119888306, "loss_scale": 4.0, "retries": 0, "seconds": 301.0, "max_memory_gb": 17.18, "ce": 0.09579684113850817, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.9459554553031921, "loss_scale": 4.0, "retries": 0, "seconds": 304.7, "max_memory_gb": 17.18, "ce": 0.11341913953538096, "rank": 0.0}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.6157288551330566, "loss_scale": 4.0, "retries": 0, "seconds": 308.5, "max_memory_gb": 17.18, "ce": 0.164030495802308, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 1.3598229885101318, "loss_scale": 4.0, "retries": 0, "seconds": 312.3, "max_memory_gb": 17.18, "ce": 0.4770253968445104, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.5718889236450195, "loss_scale": 4.0, "retries": 0, "seconds": 316.1, "max_memory_gb": 17.18, "ce": 0.02823082921167952, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 0.7808135747909546, "loss_scale": 4.0, "retries": 0, "seconds": 319.9, "max_memory_gb": 17.18, "ce": 0.044526891735586105, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 1.2097041606903076, "loss_scale": 4.0, "retries": 0, "seconds": 323.7, "max_memory_gb": 17.18, "ce": 0.11712957566396653, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.21337182819843292, "loss_scale": 4.0, "retries": 0, "seconds": 327.4, "max_memory_gb": 17.18, "ce": 0.04672417482288438, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.1755194514989853, "loss_scale": 4.0, "retries": 0, "seconds": 331.1, "max_memory_gb": 17.18, "ce": 0.022530486293362628, "rank": 0.12539909780025482}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.22254250943660736, "loss_scale": 4.0, "retries": 0, "seconds": 335.0, "max_memory_gb": 17.18, "ce": 0.013817627270327648, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.5001793503761292, "loss_scale": 4.0, "retries": 0, "seconds": 338.5, "max_memory_gb": 17.18, "ce": 0.05618754997703945, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.21580572426319122, "loss_scale": 4.0, "retries": 0, "seconds": 341.9, "max_memory_gb": 17.18, "ce": 0.008626032267784467, "rank": 0.0}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.11152593046426773, "loss_scale": 4.0, "retries": 0, "seconds": 345.5, "max_memory_gb": 17.18, "ce": 0.0013758194868387363, "rank": 0.08346407115459442}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 0.6287996172904968, "loss_scale": 4.0, "retries": 0, "seconds": 348.9, "max_memory_gb": 17.18, "ce": 0.051988197124046565, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 0.6223370432853699, "loss_scale": 4.0, "retries": 0, "seconds": 352.5, "max_memory_gb": 17.18, "ce": 0.04726877582106681, "rank": 0.13249561190605164}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 0.8662456274032593, "loss_scale": 4.0, "retries": 0, "seconds": 356.0, "max_memory_gb": 17.18, "ce": 0.11364567945020099, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 2.0518596172332764, "loss_scale": 4.0, "retries": 0, "seconds": 359.5, "max_memory_gb": 17.18, "ce": 0.4804794355168269, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 0.7436104416847229, "loss_scale": 4.0, "retries": 0, "seconds": 362.9, "max_memory_gb": 17.18, "ce": 0.09965132773413643, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 0.5884253978729248, "loss_scale": 4.0, "retries": 0, "seconds": 366.4, "max_memory_gb": 17.18, "ce": 0.05746497831933084, "rank": 0.0}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.18416735529899597, "loss_scale": 4.0, "retries": 0, "seconds": 369.8, "max_memory_gb": 17.18, "ce": 0.017868980930188627, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.09642462432384491, "loss_scale": 4.0, "retries": 0, "seconds": 373.2, "max_memory_gb": 17.18, "ce": 0.01269735800633498, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.5489397048950195, "loss_scale": 4.0, "retries": 0, "seconds": 376.6, "max_memory_gb": 17.18, "ce": 0.05852587317076541, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.07915239036083221, "loss_scale": 4.0, "retries": 0, "seconds": 380.3, "max_memory_gb": 17.18, "ce": 0.008300451598188374, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.8153777718544006, "loss_scale": 4.0, "retries": 0, "seconds": 383.8, "max_memory_gb": 17.18, "ce": 0.18904429121630528, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.1812126338481903, "loss_scale": 4.0, "retries": 0, "seconds": 387.5, "max_memory_gb": 17.18, "ce": 0.009380262043123366, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.7376787066459656, "loss_scale": 4.0, "retries": 0, "seconds": 391.3, "max_memory_gb": 17.18, "ce": 0.17076006210072592, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.7059599757194519, "loss_scale": 4.0, "retries": 0, "seconds": 395.0, "max_memory_gb": 17.18, "ce": 0.11843978271826927, "rank": 0.0}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.7350231409072876, "loss_scale": 4.0, "retries": 0, "seconds": 398.7, "max_memory_gb": 17.18, "ce": 0.10040359920094488, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 1.044793963432312, "loss_scale": 4.0, "retries": 0, "seconds": 402.4, "max_memory_gb": 17.18, "ce": 0.0853775034292994, "rank": 0.0}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 0.3836919367313385, "loss_scale": 4.0, "retries": 0, "seconds": 406.2, "max_memory_gb": 17.18, "ce": 0.08348419922049288, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 1.1895138025283813, "loss_scale": 4.0, "retries": 0, "seconds": 410.0, "max_memory_gb": 17.18, "ce": 0.22543542584435272, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 0.5011661648750305, "loss_scale": 4.0, "retries": 0, "seconds": 413.8, "max_memory_gb": 17.18, "ce": 0.1348953121937484, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.6978033185005188, "loss_scale": 4.0, "retries": 0, "seconds": 417.6, "max_memory_gb": 17.18, "ce": 0.0498115012251219, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.01856033131480217, "loss_scale": 4.0, "retries": 0, "seconds": 421.4, "max_memory_gb": 17.18, "ce": 0.0014099026102485368, "rank": 0.12489701062440872}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.3081075847148895, "loss_scale": 4.0, "retries": 0, "seconds": 425.1, "max_memory_gb": 17.18, "ce": 0.04518127163214558, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 1.3012590408325195, "loss_scale": 4.0, "retries": 0, "seconds": 428.9, "max_memory_gb": 17.18, "ce": 0.4818135764635372, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.2928352653980255, "loss_scale": 4.0, "retries": 0, "seconds": 432.6, "max_memory_gb": 17.18, "ce": 0.014665119449091435, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 1.973570466041565, "loss_scale": 4.0, "retries": 0, "seconds": 436.4, "max_memory_gb": 17.18, "ce": 0.27231086823894657, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.1011194959282875, "loss_scale": 4.0, "retries": 0, "seconds": 440.1, "max_memory_gb": 17.18, "ce": 0.004156780201356014, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 1.2565382719039917, "loss_scale": 4.0, "retries": 0, "seconds": 444.0, "max_memory_gb": 17.18, "ce": 0.2502185591702073, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 2.634794235229492, "loss_scale": 4.0, "retries": 0, "seconds": 447.7, "max_memory_gb": 17.18, "ce": 0.39220824993537917, "rank": 0.12504348158836365}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 2.466700792312622, "loss_scale": 4.0, "retries": 0, "seconds": 451.4, "max_memory_gb": 17.18, "ce": 0.12648221903305057, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 0.5927269458770752, "loss_scale": 4.0, "retries": 0, "seconds": 455.2, "max_memory_gb": 17.18, "ce": 0.041942035063584626, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.5389817953109741, "loss_scale": 4.0, "retries": 0, "seconds": 459.0, "max_memory_gb": 17.18, "ce": 0.058201957864184806, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.7588679790496826, "loss_scale": 4.0, "retries": 0, "seconds": 462.7, "max_memory_gb": 17.18, "ce": 0.03766426137144663, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.13969404995441437, "loss_scale": 4.0, "retries": 0, "seconds": 466.5, "max_memory_gb": 17.18, "ce": 0.0053319020673825435, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 0.47409293055534363, "loss_scale": 4.0, "retries": 0, "seconds": 470.3, "max_memory_gb": 17.18, "ce": 0.14308786720789612, "rank": 0.15967383980751038}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 1.2611932754516602, "loss_scale": 4.0, "retries": 0, "seconds": 474.0, "max_memory_gb": 17.18, "ce": 0.07983091511096063, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 1.264945149421692, "loss_scale": 4.0, "retries": 0, "seconds": 477.7, "max_memory_gb": 17.18, "ce": 0.46645638225436414, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.24445578455924988, "loss_scale": 4.0, "retries": 0, "seconds": 481.6, "max_memory_gb": 17.18, "ce": 0.04440260138767371, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 0.26036337018013, "loss_scale": 4.0, "retries": 0, "seconds": 485.2, "max_memory_gb": 17.18, "ce": 0.031060966107929744, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 1.1222892999649048, "loss_scale": 4.0, "retries": 0, "seconds": 488.9, "max_memory_gb": 17.18, "ce": 0.15313890064646785, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 0.2650059163570404, "loss_scale": 4.0, "retries": 0, "seconds": 492.7, "max_memory_gb": 17.18, "ce": 0.030401986113588464, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 0.49070265889167786, "loss_scale": 4.0, "retries": 0, "seconds": 496.4, "max_memory_gb": 17.18, "ce": 0.14714277823168231, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 0.01300299633294344, "loss_scale": 4.0, "retries": 0, "seconds": 500.1, "max_memory_gb": 17.18, "ce": 0.0011474745433588396, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 0.07949627935886383, "loss_scale": 4.0, "retries": 0, "seconds": 503.9, "max_memory_gb": 17.18, "ce": 0.00778538551139718, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.4132659435272217, "loss_scale": 4.0, "retries": 0, "seconds": 507.6, "max_memory_gb": 17.18, "ce": 0.02208552781689832, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 1.4426133632659912, "loss_scale": 4.0, "retries": 0, "seconds": 511.3, "max_memory_gb": 17.18, "ce": 0.44861822758252856, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 0.058664094656705856, "loss_scale": 4.0, "retries": 0, "seconds": 515.2, "max_memory_gb": 17.18, "ce": 0.005088876002446341, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 1.6627339124679565, "loss_scale": 4.0, "retries": 0, "seconds": 518.9, "max_memory_gb": 17.18, "ce": 0.2015609300367487, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 0.163107231259346, "loss_scale": 4.0, "retries": 0, "seconds": 522.6, "max_memory_gb": 17.18, "ce": 0.01463914701162139, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 1.5408812761306763, "loss_scale": 4.0, "retries": 0, "seconds": 526.4, "max_memory_gb": 17.18, "ce": 0.36733722853568906, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.10739997029304504, "loss_scale": 4.0, "retries": 0, "seconds": 530.1, "max_memory_gb": 17.18, "ce": 0.0049171879054483725, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 1.6307766437530518, "loss_scale": 4.0, "retries": 0, "seconds": 533.9, "max_memory_gb": 17.18, "ce": 0.09282316085568709, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.46306556463241577, "loss_scale": 4.0, "retries": 0, "seconds": 537.6, "max_memory_gb": 17.18, "ce": 0.02797609414847102, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 0.5442484021186829, "loss_scale": 4.0, "retries": 0, "seconds": 541.5, "max_memory_gb": 17.18, "ce": 0.14634005679965867, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.32155805826187134, "loss_scale": 4.0, "retries": 0, "seconds": 545.2, "max_memory_gb": 17.18, "ce": 0.014667158786323853, "rank": 0.12539148330688477}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.08932413160800934, "loss_scale": 4.0, "retries": 0, "seconds": 548.9, "max_memory_gb": 17.18, "ce": 0.009011229069074034, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.23547488451004028, "loss_scale": 4.0, "retries": 0, "seconds": 552.7, "max_memory_gb": 17.18, "ce": 0.025478974663656118, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 0.4938916862010956, "loss_scale": 4.0, "retries": 0, "seconds": 556.5, "max_memory_gb": 17.18, "ce": 0.06753379371536994, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 2.0756547451019287, "loss_scale": 4.0, "retries": 0, "seconds": 560.2, "max_memory_gb": 17.18, "ce": 0.2580169139841928, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 3.0221004486083984, "loss_scale": 4.0, "retries": 0, "seconds": 564.0, "max_memory_gb": 17.18, "ce": 0.3789631621039007, "rank": 0.0}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.9991071224212646, "loss_scale": 4.0, "retries": 0, "seconds": 567.7, "max_memory_gb": 17.18, "ce": 0.08219794721651397, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.11045601963996887, "loss_scale": 4.0, "retries": 0, "seconds": 571.4, "max_memory_gb": 17.18, "ce": 0.006771729040337959, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 0.5934526324272156, "loss_scale": 4.0, "retries": 0, "seconds": 575.3, "max_memory_gb": 17.18, "ce": 0.06421292845220705, "rank": 0.07749839127063751}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.18704532086849213, "loss_scale": 4.0, "retries": 0, "seconds": 579.0, "max_memory_gb": 17.18, "ce": 0.013966684527076723, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.062308043241500854, "loss_scale": 4.0, "retries": 0, "seconds": 582.7, "max_memory_gb": 17.18, "ce": 0.005984674103501675, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 0.5301313400268555, "loss_scale": 4.0, "retries": 0, "seconds": 586.4, "max_memory_gb": 17.18, "ce": 0.10956820168041759, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.5526496767997742, "loss_scale": 4.0, "retries": 0, "seconds": 590.3, "max_memory_gb": 17.18, "ce": 0.07158565233942227, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.5631216168403625, "loss_scale": 4.0, "retries": 0, "seconds": 594.0, "max_memory_gb": 17.18, "ce": 0.01622501531812759, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 0.9773927330970764, "loss_scale": 4.0, "retries": 0, "seconds": 597.8, "max_memory_gb": 17.18, "ce": 0.11874272612749337, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.094304159283638, "loss_scale": 4.0, "retries": 0, "seconds": 601.6, "max_memory_gb": 17.18, "ce": 0.011703506389153517, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 3.139007091522217, "loss_scale": 4.0, "retries": 0, "seconds": 605.3, "max_memory_gb": 17.18, "ce": 0.21255946678184046, "rank": 0.019204020500183105}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 0.4280242919921875, "loss_scale": 4.0, "retries": 0, "seconds": 609.1, "max_memory_gb": 17.18, "ce": 0.061810434409608206, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.49732157588005066, "loss_scale": 4.0, "retries": 0, "seconds": 613.0, "max_memory_gb": 17.18, "ce": 0.04943495702332257, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 0.3247259557247162, "loss_scale": 4.0, "retries": 0, "seconds": 616.7, "max_memory_gb": 17.18, "ce": 0.06063533541691868, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.6723558306694031, "loss_scale": 4.0, "retries": 0, "seconds": 620.4, "max_memory_gb": 17.18, "ce": 0.1498200534060743, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.0967007651925087, "loss_scale": 4.0, "retries": 0, "seconds": 624.2, "max_memory_gb": 17.18, "ce": 0.005448885868645448, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 0.13996286690235138, "loss_scale": 4.0, "retries": 0, "seconds": 627.9, "max_memory_gb": 17.18, "ce": 0.02253643353924417, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.035354916006326675, "loss_scale": 4.0, "retries": 0, "seconds": 631.6, "max_memory_gb": 17.18, "ce": 0.0027876282401848584, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.5157453417778015, "loss_scale": 4.0, "retries": 0, "seconds": 635.5, "max_memory_gb": 17.18, "ce": 0.02261819043781088, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 0.6391798257827759, "loss_scale": 4.0, "retries": 0, "seconds": 639.2, "max_memory_gb": 17.18, "ce": 0.09246423360116296, "rank": 0.24932565540075302}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.4305863082408905, "loss_scale": 4.0, "retries": 0, "seconds": 643.0, "max_memory_gb": 17.18, "ce": 0.052645153828962066, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.7130811810493469, "loss_scale": 4.0, "retries": 0, "seconds": 646.7, "max_memory_gb": 17.18, "ce": 0.11265065616635184, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.7406650185585022, "loss_scale": 4.0, "retries": 0, "seconds": 650.5, "max_memory_gb": 17.18, "ce": 0.12680703383500713, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.4406994581222534, "loss_scale": 4.0, "retries": 0, "seconds": 654.3, "max_memory_gb": 17.18, "ce": 0.09817342922769967, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 1.2475805282592773, "loss_scale": 4.0, "retries": 0, "seconds": 658.0, "max_memory_gb": 17.18, "ce": 0.15461192139559898, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 0.478618323802948, "loss_scale": 4.0, "retries": 0, "seconds": 661.9, "max_memory_gb": 17.18, "ce": 0.023416891592205502, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.3150842487812042, "loss_scale": 4.0, "retries": 0, "seconds": 665.6, "max_memory_gb": 17.18, "ce": 0.0723761475073843, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 1.8963462114334106, "loss_scale": 4.0, "retries": 0, "seconds": 669.3, "max_memory_gb": 17.18, "ce": 0.21918599698733487, "rank": 0.0}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 0.4588668644428253, "loss_scale": 4.0, "retries": 0, "seconds": 673.2, "max_memory_gb": 17.18, "ce": 0.0614189633291744, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 0.4687882661819458, "loss_scale": 4.0, "retries": 0, "seconds": 676.9, "max_memory_gb": 17.18, "ce": 0.08968845309931339, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.8027229309082031, "loss_scale": 4.0, "retries": 0, "seconds": 680.6, "max_memory_gb": 17.18, "ce": 0.24327668663249824, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 1.136387825012207, "loss_scale": 4.0, "retries": 0, "seconds": 684.4, "max_memory_gb": 17.18, "ce": 0.2740225227753399, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 1.7155296802520752, "loss_scale": 4.0, "retries": 0, "seconds": 688.3, "max_memory_gb": 17.18, "ce": 0.18108946106383428, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.02260848693549633, "loss_scale": 4.0, "retries": 0, "seconds": 692.0, "max_memory_gb": 17.18, "ce": 0.0014935544377294718, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.02049359679222107, "loss_scale": 4.0, "retries": 0, "seconds": 695.7, "max_memory_gb": 17.18, "ce": 0.0011399577081192547, "rank": 0.0}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.549531102180481, "loss_scale": 4.0, "retries": 0, "seconds": 699.6, "max_memory_gb": 17.18, "ce": 0.001144131992077746, "rank": 0.050012215971946716}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 0.23007197678089142, "loss_scale": 4.0, "retries": 0, "seconds": 703.3, "max_memory_gb": 17.18, "ce": 0.02804035532017224, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.7494022250175476, "loss_scale": 4.0, "retries": 0, "seconds": 707.1, "max_memory_gb": 17.18, "ce": 0.11641680495313267, "rank": 0.0}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 0.39449724555015564, "loss_scale": 4.0, "retries": 0, "seconds": 711.7, "max_memory_gb": 17.18, "ce": 0.08559042964884611, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 0.6247878074645996, "loss_scale": 4.0, "retries": 0, "seconds": 716.2, "max_memory_gb": 17.18, "ce": 0.09685593090011935, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.2555667459964752, "loss_scale": 4.0, "retries": 0, "seconds": 720.7, "max_memory_gb": 17.18, "ce": 0.025318063504528254, "rank": 0.12509571015834808}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 0.056507475674152374, "loss_scale": 4.0, "retries": 0, "seconds": 725.2, "max_memory_gb": 17.18, "ce": 0.0031504469815217817, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 3.5334110260009766, "loss_scale": 4.0, "retries": 0, "seconds": 729.5, "max_memory_gb": 17.18, "ce": 0.4942474876622782, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.5397329926490784, "loss_scale": 4.0, "retries": 0, "seconds": 733.3, "max_memory_gb": 17.18, "ce": 0.0489075592458903, "rank": 0.0}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.2765093743801117, "loss_scale": 4.0, "retries": 0, "seconds": 737.2, "max_memory_gb": 17.18, "ce": 0.013522184198109244, "rank": 0.12473482638597488}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.11209830641746521, "loss_scale": 4.0, "retries": 0, "seconds": 741.0, "max_memory_gb": 17.18, "ce": 0.008041268184115324, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.027103155851364136, "loss_scale": 4.0, "retries": 0, "seconds": 744.8, "max_memory_gb": 17.18, "ce": 0.0025347673038140783, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.12208854407072067, "loss_scale": 4.0, "retries": 0, "seconds": 748.6, "max_memory_gb": 17.18, "ce": 0.004627569365425188, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.6144877672195435, "loss_scale": 4.0, "retries": 0, "seconds": 752.1, "max_memory_gb": 17.18, "ce": 0.05179365279298054, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 0.7086641192436218, "loss_scale": 4.0, "retries": 0, "seconds": 755.5, "max_memory_gb": 17.18, "ce": 0.13226479532170288, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 0.4107907712459564, "loss_scale": 4.0, "retries": 0, "seconds": 759.0, "max_memory_gb": 17.18, "ce": 0.0779942145323389, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.20214694738388062, "loss_scale": 4.0, "retries": 0, "seconds": 762.5, "max_memory_gb": 17.18, "ce": 0.014515182260765869, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.9526816606521606, "loss_scale": 4.0, "retries": 0, "seconds": 765.9, "max_memory_gb": 17.18, "ce": 0.30756720467024934, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.08878341317176819, "loss_scale": 4.0, "retries": 0, "seconds": 769.3, "max_memory_gb": 17.18, "ce": 0.007239387029244426, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.40244972705841064, "loss_scale": 4.0, "retries": 0, "seconds": 772.9, "max_memory_gb": 17.18, "ce": 0.052120738020221324, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 0.13557709753513336, "loss_scale": 4.0, "retries": 0, "seconds": 776.3, "max_memory_gb": 17.18, "ce": 0.012416195161904398, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 1.0316472053527832, "loss_scale": 4.0, "retries": 0, "seconds": 779.7, "max_memory_gb": 17.18, "ce": 0.04255793216429993, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.2401445508003235, "loss_scale": 4.0, "retries": 0, "seconds": 783.3, "max_memory_gb": 17.18, "ce": 0.019149032830796386, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.3368481993675232, "loss_scale": 4.0, "retries": 0, "seconds": 786.7, "max_memory_gb": 17.18, "ce": 0.028820833616691743, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.44801583886146545, "loss_scale": 4.0, "retries": 0, "seconds": 790.2, "max_memory_gb": 17.18, "ce": 0.04473615733309089, "rank": 0.0}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 0.45373380184173584, "loss_scale": 4.0, "retries": 0, "seconds": 793.6, "max_memory_gb": 17.18, "ce": 0.06191593161986475, "rank": 0.0}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.1942744255065918, "loss_scale": 4.0, "retries": 0, "seconds": 797.2, "max_memory_gb": 17.18, "ce": 0.010046254604731075, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 0.8012603521347046, "loss_scale": 4.0, "retries": 0, "seconds": 800.6, "max_memory_gb": 17.18, "ce": 0.20398393484811095, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 1.3027962446212769, "loss_scale": 4.0, "retries": 0, "seconds": 804.0, "max_memory_gb": 17.18, "ce": 0.4778534263584788, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 0.11652832478284836, "loss_scale": 4.0, "retries": 0, "seconds": 807.6, "max_memory_gb": 17.18, "ce": 0.010045046728123452, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 0.5109735131263733, "loss_scale": 4.0, "retries": 0, "seconds": 810.9, "max_memory_gb": 17.18, "ce": 0.042976036473532986, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 1.3527319431304932, "loss_scale": 4.0, "retries": 0, "seconds": 814.4, "max_memory_gb": 17.18, "ce": 0.16665215226691998, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.6578958630561829, "loss_scale": 4.0, "retries": 0, "seconds": 817.9, "max_memory_gb": 17.18, "ce": 0.04411166038130432, "rank": 0.12499568611383438}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 2.316621780395508, "loss_scale": 4.0, "retries": 0, "seconds": 821.4, "max_memory_gb": 17.18, "ce": 0.38493672764036546, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.16379345953464508, "loss_scale": 4.0, "retries": 0, "seconds": 824.8, "max_memory_gb": 17.18, "ce": 0.019816875364199404, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.8319656848907471, "loss_scale": 4.0, "retries": 0, "seconds": 828.3, "max_memory_gb": 17.18, "ce": 0.1312419727764791, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 1.4812127351760864, "loss_scale": 4.0, "retries": 0, "seconds": 831.8, "max_memory_gb": 17.18, "ce": 0.2425119193064802, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.2917631268501282, "loss_scale": 4.0, "retries": 0, "seconds": 835.2, "max_memory_gb": 17.18, "ce": 0.03236759001555356, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 1.1701935529708862, "loss_scale": 4.0, "retries": 0, "seconds": 838.6, "max_memory_gb": 17.18, "ce": 0.11173970716185977, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.013523785397410393, "loss_scale": 4.0, "retries": 0, "seconds": 842.2, "max_memory_gb": 17.18, "ce": 0.0007154167669796152, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.7673935294151306, "loss_scale": 4.0, "retries": 0, "seconds": 845.6, "max_memory_gb": 17.18, "ce": 0.23009235491082336, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.7340469360351562, "loss_scale": 4.0, "retries": 0, "seconds": 849.1, "max_memory_gb": 17.18, "ce": 0.1465715090293429, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.14856091141700745, "loss_scale": 4.0, "retries": 0, "seconds": 853.0, "max_memory_gb": 17.18, "ce": 0.0113178012076105, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 1.6934349536895752, "loss_scale": 4.0, "retries": 0, "seconds": 856.8, "max_memory_gb": 17.18, "ce": 0.09727599650136654, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 1.1679470539093018, "loss_scale": 4.0, "retries": 0, "seconds": 860.6, "max_memory_gb": 17.18, "ce": 0.41374173176006934, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 0.0697023794054985, "loss_scale": 4.0, "retries": 0, "seconds": 864.5, "max_memory_gb": 17.18, "ce": 0.006025430794579734, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.45004719495773315, "loss_scale": 4.0, "retries": 0, "seconds": 868.2, "max_memory_gb": 17.18, "ce": 0.14239397570690926, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.3530881106853485, "loss_scale": 4.0, "retries": 0, "seconds": 872.0, "max_memory_gb": 17.18, "ce": 0.0057353923318714806, "rank": 0.05800805985927582}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.34402889013290405, "loss_scale": 4.0, "retries": 0, "seconds": 875.9, "max_memory_gb": 17.18, "ce": 0.0419023750271208, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.3801746070384979, "loss_scale": 4.0, "retries": 0, "seconds": 879.7, "max_memory_gb": 17.18, "ce": 0.03598484096505672, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 4.056292533874512, "loss_scale": 4.0, "retries": 0, "seconds": 883.5, "max_memory_gb": 17.18, "ce": 0.43698465246325213, "rank": 0.06120172142982483}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.5527850985527039, "loss_scale": 4.0, "retries": 0, "seconds": 887.4, "max_memory_gb": 17.18, "ce": 0.08366413917769933, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 1.0109304189682007, "loss_scale": 4.0, "retries": 0, "seconds": 891.2, "max_memory_gb": 17.18, "ce": 0.15895031965851558, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 1.8371132612228394, "loss_scale": 4.0, "retries": 0, "seconds": 895.0, "max_memory_gb": 17.18, "ce": 0.26423307067761925, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.15788744390010834, "loss_scale": 4.0, "retries": 0, "seconds": 898.9, "max_memory_gb": 17.18, "ce": 0.009887619278742932, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 1.367752194404602, "loss_scale": 4.0, "retries": 0, "seconds": 902.7, "max_memory_gb": 17.18, "ce": 0.31323447955594474, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.27237367630004883, "loss_scale": 4.0, "retries": 0, "seconds": 906.5, "max_memory_gb": 17.18, "ce": 0.02450303328305381, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 1.7235021591186523, "loss_scale": 4.0, "retries": 0, "seconds": 910.4, "max_memory_gb": 17.18, "ce": 0.2785520800106269, "rank": 0.12406541407108307}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.8899996280670166, "loss_scale": 4.0, "retries": 0, "seconds": 914.3, "max_memory_gb": 17.18, "ce": 0.1515671756469601, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.4597082734107971, "loss_scale": 4.0, "retries": 0, "seconds": 918.0, "max_memory_gb": 17.18, "ce": 0.05805831516681792, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 0.9515429139137268, "loss_scale": 4.0, "retries": 0, "seconds": 921.8, "max_memory_gb": 17.18, "ce": 0.16208465745648937, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.5775726437568665, "loss_scale": 4.0, "retries": 0, "seconds": 925.7, "max_memory_gb": 17.18, "ce": 0.06801899965887515, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.050120774656534195, "loss_scale": 4.0, "retries": 0, "seconds": 929.4, "max_memory_gb": 17.18, "ce": 0.003490841537313827, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.04444807395339012, "loss_scale": 4.0, "retries": 0, "seconds": 933.1, "max_memory_gb": 17.18, "ce": 0.003310046456135751, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.24584375321865082, "loss_scale": 4.0, "retries": 0, "seconds": 937.0, "max_memory_gb": 17.18, "ce": 0.03622322658475241, "rank": 0.10104799270629883}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.054422345012426376, "loss_scale": 4.0, "retries": 0, "seconds": 940.7, "max_memory_gb": 17.18, "ce": 0.0052713758301479174, "rank": 0.11760595440864563}

runs/fixed_internvl/removal/Deep8_s73/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/removal/Deep8_s73/adapter/adapter_config.json

@@ -0,0 +1,213 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.28.mlp.down_proj": 144,
+    "model.language_model.layers.28.mlp.gate_proj": 144,
+    "model.language_model.layers.28.mlp.up_proj": 144,
+    "model.language_model.layers.28.self_attn.k_proj": 144,
+    "model.language_model.layers.28.self_attn.o_proj": 144,
+    "model.language_model.layers.28.self_attn.q_proj": 144,
+    "model.language_model.layers.28.self_attn.v_proj": 144,
+    "model.language_model.layers.29.mlp.down_proj": 144,
+    "model.language_model.layers.29.mlp.gate_proj": 144,
+    "model.language_model.layers.29.mlp.up_proj": 144,
+    "model.language_model.layers.29.self_attn.k_proj": 144,
+    "model.language_model.layers.29.self_attn.o_proj": 144,
+    "model.language_model.layers.29.self_attn.q_proj": 144,
+    "model.language_model.layers.29.self_attn.v_proj": 144,
+    "model.language_model.layers.30.mlp.down_proj": 144,
+    "model.language_model.layers.30.mlp.gate_proj": 144,
+    "model.language_model.layers.30.mlp.up_proj": 144,
+    "model.language_model.layers.30.self_attn.k_proj": 144,
+    "model.language_model.layers.30.self_attn.o_proj": 144,
+    "model.language_model.layers.30.self_attn.q_proj": 144,
+    "model.language_model.layers.30.self_attn.v_proj": 144,
+    "model.language_model.layers.31.mlp.down_proj": 144,
+    "model.language_model.layers.31.mlp.gate_proj": 144,
+    "model.language_model.layers.31.mlp.up_proj": 144,
+    "model.language_model.layers.31.self_attn.k_proj": 144,
+    "model.language_model.layers.31.self_attn.o_proj": 144,
+    "model.language_model.layers.31.self_attn.q_proj": 144,
+    "model.language_model.layers.31.self_attn.v_proj": 144,
+    "model.language_model.layers.32.mlp.down_proj": 144,
+    "model.language_model.layers.32.mlp.gate_proj": 144,
+    "model.language_model.layers.32.mlp.up_proj": 144,
+    "model.language_model.layers.32.self_attn.k_proj": 144,
+    "model.language_model.layers.32.self_attn.o_proj": 144,
+    "model.language_model.layers.32.self_attn.q_proj": 144,
+    "model.language_model.layers.32.self_attn.v_proj": 144,
+    "model.language_model.layers.33.mlp.down_proj": 144,
+    "model.language_model.layers.33.mlp.gate_proj": 144,
+    "model.language_model.layers.33.mlp.up_proj": 144,
+    "model.language_model.layers.33.self_attn.k_proj": 144,
+    "model.language_model.layers.33.self_attn.o_proj": 144,
+    "model.language_model.layers.33.self_attn.q_proj": 144,
+    "model.language_model.layers.33.self_attn.v_proj": 144,
+    "model.language_model.layers.34.mlp.down_proj": 144,
+    "model.language_model.layers.34.mlp.gate_proj": 144,
+    "model.language_model.layers.34.mlp.up_proj": 144,
+    "model.language_model.layers.34.self_attn.k_proj": 144,
+    "model.language_model.layers.34.self_attn.o_proj": 144,
+    "model.language_model.layers.34.self_attn.q_proj": 144,
+    "model.language_model.layers.34.self_attn.v_proj": 144,
+    "model.language_model.layers.35.mlp.down_proj": 144,
+    "model.language_model.layers.35.mlp.gate_proj": 144,
+    "model.language_model.layers.35.mlp.up_proj": 144,
+    "model.language_model.layers.35.self_attn.k_proj": 144,
+    "model.language_model.layers.35.self_attn.o_proj": 144,
+    "model.language_model.layers.35.self_attn.q_proj": 144,
+    "model.language_model.layers.35.self_attn.v_proj": 144
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.28.mlp.down_proj": 72,
+    "model.language_model.layers.28.mlp.gate_proj": 72,
+    "model.language_model.layers.28.mlp.up_proj": 72,
+    "model.language_model.layers.28.self_attn.k_proj": 72,
+    "model.language_model.layers.28.self_attn.o_proj": 72,
+    "model.language_model.layers.28.self_attn.q_proj": 72,
+    "model.language_model.layers.28.self_attn.v_proj": 72,
+    "model.language_model.layers.29.mlp.down_proj": 72,
+    "model.language_model.layers.29.mlp.gate_proj": 72,
+    "model.language_model.layers.29.mlp.up_proj": 72,
+    "model.language_model.layers.29.self_attn.k_proj": 72,
+    "model.language_model.layers.29.self_attn.o_proj": 72,
+    "model.language_model.layers.29.self_attn.q_proj": 72,
+    "model.language_model.layers.29.self_attn.v_proj": 72,
+    "model.language_model.layers.30.mlp.down_proj": 72,
+    "model.language_model.layers.30.mlp.gate_proj": 72,
+    "model.language_model.layers.30.mlp.up_proj": 72,
+    "model.language_model.layers.30.self_attn.k_proj": 72,
+    "model.language_model.layers.30.self_attn.o_proj": 72,
+    "model.language_model.layers.30.self_attn.q_proj": 72,
+    "model.language_model.layers.30.self_attn.v_proj": 72,
+    "model.language_model.layers.31.mlp.down_proj": 72,
+    "model.language_model.layers.31.mlp.gate_proj": 72,
+    "model.language_model.layers.31.mlp.up_proj": 72,
+    "model.language_model.layers.31.self_attn.k_proj": 72,
+    "model.language_model.layers.31.self_attn.o_proj": 72,
+    "model.language_model.layers.31.self_attn.q_proj": 72,
+    "model.language_model.layers.31.self_attn.v_proj": 72,
+    "model.language_model.layers.32.mlp.down_proj": 72,
+    "model.language_model.layers.32.mlp.gate_proj": 72,
+    "model.language_model.layers.32.mlp.up_proj": 72,
+    "model.language_model.layers.32.self_attn.k_proj": 72,
+    "model.language_model.layers.32.self_attn.o_proj": 72,
+    "model.language_model.layers.32.self_attn.q_proj": 72,
+    "model.language_model.layers.32.self_attn.v_proj": 72,
+    "model.language_model.layers.33.mlp.down_proj": 72,
+    "model.language_model.layers.33.mlp.gate_proj": 72,
+    "model.language_model.layers.33.mlp.up_proj": 72,
+    "model.language_model.layers.33.self_attn.k_proj": 72,
+    "model.language_model.layers.33.self_attn.o_proj": 72,
+    "model.language_model.layers.33.self_attn.q_proj": 72,
+    "model.language_model.layers.33.self_attn.v_proj": 72,
+    "model.language_model.layers.34.mlp.down_proj": 72,
+    "model.language_model.layers.34.mlp.gate_proj": 72,
+    "model.language_model.layers.34.mlp.up_proj": 72,
+    "model.language_model.layers.34.self_attn.k_proj": 72,
+    "model.language_model.layers.34.self_attn.o_proj": 72,
+    "model.language_model.layers.34.self_attn.q_proj": 72,
+    "model.language_model.layers.34.self_attn.v_proj": 72,
+    "model.language_model.layers.35.mlp.down_proj": 72,
+    "model.language_model.layers.35.mlp.gate_proj": 72,
+    "model.language_model.layers.35.mlp.up_proj": 72,
+    "model.language_model.layers.35.self_attn.k_proj": 72,
+    "model.language_model.layers.35.self_attn.o_proj": 72,
+    "model.language_model.layers.35.self_attn.q_proj": 72,
+    "model.language_model.layers.35.self_attn.v_proj": 72
+  },
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.34.self_attn.q_proj",
+    "model.language_model.layers.31.mlp.down_proj",
+    "model.language_model.layers.31.self_attn.o_proj",
+    "model.language_model.layers.30.self_attn.v_proj",
+    "model.language_model.layers.34.mlp.up_proj",
+    "model.language_model.layers.33.mlp.up_proj",
+    "model.language_model.layers.35.mlp.up_proj",
+    "model.language_model.layers.33.self_attn.v_proj",
+    "model.language_model.layers.31.self_attn.v_proj",
+    "model.language_model.layers.35.self_attn.q_proj",
+    "model.language_model.layers.33.self_attn.q_proj",
+    "model.language_model.layers.32.self_attn.k_proj",
+    "model.language_model.layers.35.mlp.gate_proj",
+    "model.language_model.layers.28.self_attn.o_proj",
+    "model.language_model.layers.29.self_attn.q_proj",
+    "model.language_model.layers.32.mlp.down_proj",
+    "model.language_model.layers.32.self_attn.o_proj",
+    "model.language_model.layers.34.self_attn.o_proj",
+    "model.language_model.layers.31.self_attn.q_proj",
+    "model.language_model.layers.30.mlp.gate_proj",
+    "model.language_model.layers.32.self_attn.q_proj",
+    "model.language_model.layers.29.self_attn.o_proj",
+    "model.language_model.layers.34.self_attn.k_proj",
+    "model.language_model.layers.30.self_attn.q_proj",
+    "model.language_model.layers.35.mlp.down_proj",
+    "model.language_model.layers.29.mlp.up_proj",
+    "model.language_model.layers.28.self_attn.v_proj",
+    "model.language_model.layers.30.mlp.up_proj",
+    "model.language_model.layers.30.self_attn.k_proj",
+    "model.language_model.layers.31.mlp.up_proj",
+    "model.language_model.layers.35.self_attn.v_proj",
+    "model.language_model.layers.29.self_attn.v_proj",
+    "model.language_model.layers.34.mlp.down_proj",
+    "model.language_model.layers.33.self_attn.k_proj",
+    "model.language_model.layers.33.mlp.gate_proj",
+    "model.language_model.layers.28.mlp.down_proj",
+    "model.language_model.layers.30.mlp.down_proj",
+    "model.language_model.layers.32.mlp.up_proj",
+    "model.language_model.layers.35.self_attn.o_proj",
+    "model.language_model.layers.34.mlp.gate_proj",
+    "model.language_model.layers.30.self_attn.o_proj",
+    "model.language_model.layers.31.mlp.gate_proj",
+    "model.language_model.layers.29.mlp.gate_proj",
+    "model.language_model.layers.31.self_attn.k_proj",
+    "model.language_model.layers.35.self_attn.k_proj",
+    "model.language_model.layers.29.self_attn.k_proj",
+    "model.language_model.layers.29.mlp.down_proj",
+    "model.language_model.layers.32.mlp.gate_proj",
+    "model.language_model.layers.28.mlp.up_proj",
+    "model.language_model.layers.32.self_attn.v_proj",
+    "model.language_model.layers.33.mlp.down_proj",
+    "model.language_model.layers.28.self_attn.k_proj",
+    "model.language_model.layers.28.self_attn.q_proj",
+    "model.language_model.layers.34.self_attn.v_proj",
+    "model.language_model.layers.28.mlp.gate_proj",
+    "model.language_model.layers.33.self_attn.o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/fixed_internvl/removal/Deep8_s73/config.json

@@ -0,0 +1,43 @@
+{
+  "behavior": "removal",
+  "seed": 73,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "world_size": 1,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "arm": {
+    "arm": "Deep8",
+    "layers": [
+      28,
+      29,
+      30,
+      31,
+      32,
+      33,
+      34,
+      35
+    ],
+    "ranks": {
+      "28": 72,
+      "29": 72,
+      "30": 72,
+      "31": 72,
+      "32": 72,
+      "33": 72,
+      "34": 72,
+      "35": 72
+    },
+    "parameters": 43646976,
+    "budget": 43646976
+  }
+}

runs/fixed_internvl/removal/Deep8_s73/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 3.4532971382141113, "loss_scale": 4.0, "retries": 0, "seconds": 5.5, "max_memory_gb": 16.83, "ce": 0.8941777415893739, "rank": 0.0}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 5.019935131072998, "loss_scale": 4.0, "retries": 0, "seconds": 10.0, "max_memory_gb": 17.17, "ce": 1.6876674722880125, "rank": 0.018019914627075195}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 2.815467119216919, "loss_scale": 4.0, "retries": 0, "seconds": 14.4, "max_memory_gb": 17.17, "ce": 0.7547073739551706, "rank": 0.09108561277389526}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 2.9399614334106445, "loss_scale": 4.0, "retries": 0, "seconds": 18.8, "max_memory_gb": 17.17, "ce": 0.8587086296902271, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 2.331336736679077, "loss_scale": 4.0, "retries": 0, "seconds": 23.4, "max_memory_gb": 17.17, "ce": 0.6386007346372935, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 1.9865918159484863, "loss_scale": 4.0, "retries": 0, "seconds": 27.7, "max_memory_gb": 17.18, "ce": 0.6453824860509485, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 0.7268761396408081, "loss_scale": 4.0, "retries": 0, "seconds": 32.1, "max_memory_gb": 17.18, "ce": 0.18180312306867563, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 1.6131032705307007, "loss_scale": 4.0, "retries": 0, "seconds": 36.6, "max_memory_gb": 17.18, "ce": 0.690955174552073, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 3.1679205894470215, "loss_scale": 4.0, "retries": 0, "seconds": 41.0, "max_memory_gb": 17.18, "ce": 0.8069800860248506, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 1.4258242845535278, "loss_scale": 4.0, "retries": 0, "seconds": 45.4, "max_memory_gb": 17.18, "ce": 0.46358419776152004, "rank": 0.004084929823875427}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 3.1577463150024414, "loss_scale": 4.0, "retries": 0, "seconds": 49.8, "max_memory_gb": 17.18, "ce": 0.605385493952781, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 1.2810035943984985, "loss_scale": 4.0, "retries": 0, "seconds": 54.4, "max_memory_gb": 17.18, "ce": 0.2515988088866834, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 1.4295815229415894, "loss_scale": 4.0, "retries": 0, "seconds": 58.8, "max_memory_gb": 17.18, "ce": 0.3870170414884342, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 0.5678296089172363, "loss_scale": 4.0, "retries": 0, "seconds": 63.2, "max_memory_gb": 17.18, "ce": 0.28021068579391795, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 0.7264125347137451, "loss_scale": 4.0, "retries": 0, "seconds": 67.7, "max_memory_gb": 17.18, "ce": 0.1815431755821919, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 1.764896035194397, "loss_scale": 4.0, "retries": 0, "seconds": 72.1, "max_memory_gb": 17.18, "ce": 0.9415201395750046, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.681647777557373, "loss_scale": 4.0, "retries": 0, "seconds": 76.5, "max_memory_gb": 17.18, "ce": 0.2605918222980108, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 0.5217577815055847, "loss_scale": 4.0, "retries": 0, "seconds": 80.9, "max_memory_gb": 17.18, "ce": 0.16576789987448137, "rank": 0.12469053268432617}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 1.8710418939590454, "loss_scale": 4.0, "retries": 0, "seconds": 85.4, "max_memory_gb": 17.18, "ce": 0.44357636103086406, "rank": 0.0}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.905767560005188, "loss_scale": 4.0, "retries": 0, "seconds": 89.8, "max_memory_gb": 17.18, "ce": 0.28454819764738204, "rank": 0.0}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.5461301207542419, "loss_scale": 4.0, "retries": 0, "seconds": 94.2, "max_memory_gb": 17.18, "ce": 0.14513845590045094, "rank": 0.0}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.9050872921943665, "loss_scale": 4.0, "retries": 0, "seconds": 98.8, "max_memory_gb": 17.18, "ce": 0.27173716135439463, "rank": 0.0}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 0.8110631704330444, "loss_scale": 4.0, "retries": 0, "seconds": 103.2, "max_memory_gb": 17.18, "ce": 0.1579979475682194, "rank": 0.0}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 1.0041532516479492, "loss_scale": 4.0, "retries": 0, "seconds": 107.6, "max_memory_gb": 17.18, "ce": 0.19460048543987796, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 0.562853991985321, "loss_scale": 4.0, "retries": 0, "seconds": 111.8, "max_memory_gb": 17.18, "ce": 0.12220198410796002, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 2.3575546741485596, "loss_scale": 4.0, "retries": 0, "seconds": 115.9, "max_memory_gb": 17.18, "ce": 0.2584207752770453, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 1.489514708518982, "loss_scale": 4.0, "retries": 0, "seconds": 120.0, "max_memory_gb": 17.18, "ce": 0.16796920378692448, "rank": 0.0}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 0.6405346989631653, "loss_scale": 4.0, "retries": 0, "seconds": 124.1, "max_memory_gb": 17.18, "ce": 0.02470743010781007, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 3.377102851867676, "loss_scale": 4.0, "retries": 0, "seconds": 128.2, "max_memory_gb": 17.18, "ce": 0.09806000796379521, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 2.660224199295044, "loss_scale": 4.0, "retries": 0, "seconds": 132.3, "max_memory_gb": 17.18, "ce": 0.12462650539237075, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 1.7031893730163574, "loss_scale": 4.0, "retries": 0, "seconds": 136.4, "max_memory_gb": 17.18, "ce": 0.11515787022653967, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 2.322887897491455, "loss_scale": 4.0, "retries": 0, "seconds": 140.6, "max_memory_gb": 17.18, "ce": 0.24462700008007232, "rank": 0.12504231929779053}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.5596831440925598, "loss_scale": 4.0, "retries": 0, "seconds": 144.6, "max_memory_gb": 17.18, "ce": 0.028795230740797706, "rank": 0.12530437111854553}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.42263665795326233, "loss_scale": 4.0, "retries": 0, "seconds": 148.7, "max_memory_gb": 17.18, "ce": 0.08375837290805066, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.4223482012748718, "loss_scale": 4.0, "retries": 0, "seconds": 152.8, "max_memory_gb": 17.18, "ce": 0.051976941438624635, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 1.0052284002304077, "loss_scale": 4.0, "retries": 0, "seconds": 156.9, "max_memory_gb": 17.18, "ce": 0.16262173005088698, "rank": 0.1218961551785469}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.5209650993347168, "loss_scale": 4.0, "retries": 0, "seconds": 161.0, "max_memory_gb": 17.18, "ce": 0.04521148509957129, "rank": 0.09861954301595688}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.2127251774072647, "loss_scale": 4.0, "retries": 0, "seconds": 165.1, "max_memory_gb": 17.18, "ce": 0.030248009159549838, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 1.2154062986373901, "loss_scale": 4.0, "retries": 0, "seconds": 169.2, "max_memory_gb": 17.18, "ce": 0.19550285928562516, "rank": 0.0}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.2559123933315277, "loss_scale": 4.0, "retries": 0, "seconds": 173.2, "max_memory_gb": 17.18, "ce": 0.024628117691463558, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.32775941491127014, "loss_scale": 4.0, "retries": 0, "seconds": 177.4, "max_memory_gb": 17.18, "ce": 0.02325535831914749, "rank": 0.1169472262263298}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 1.8812830448150635, "loss_scale": 4.0, "retries": 0, "seconds": 181.5, "max_memory_gb": 17.18, "ce": 0.1983772822713945, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 3.1839025020599365, "loss_scale": 4.0, "retries": 0, "seconds": 185.5, "max_memory_gb": 17.18, "ce": 0.26040813743384206, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 0.19777846336364746, "loss_scale": 4.0, "retries": 0, "seconds": 189.7, "max_memory_gb": 17.18, "ce": 0.009733081198646687, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 0.5612711310386658, "loss_scale": 4.0, "retries": 0, "seconds": 193.5, "max_memory_gb": 17.18, "ce": 0.07393577148468466, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.5945975184440613, "loss_scale": 4.0, "retries": 0, "seconds": 196.9, "max_memory_gb": 17.18, "ce": 0.21770312988519436, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.06798316538333893, "loss_scale": 4.0, "retries": 0, "seconds": 200.4, "max_memory_gb": 17.18, "ce": 0.00781321515023592, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.2243049144744873, "loss_scale": 4.0, "retries": 0, "seconds": 203.7, "max_memory_gb": 17.18, "ce": 0.030598871502661495, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.364393413066864, "loss_scale": 4.0, "retries": 0, "seconds": 207.1, "max_memory_gb": 17.18, "ce": 0.02434379760597949, "rank": 0.1253773272037506}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.08652658015489578, "loss_scale": 4.0, "retries": 0, "seconds": 210.5, "max_memory_gb": 17.18, "ce": 0.005105982093482453, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 0.7153976559638977, "loss_scale": 4.0, "retries": 0, "seconds": 214.0, "max_memory_gb": 17.18, "ce": 0.07664047974321875, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 3.634326219558716, "loss_scale": 4.0, "retries": 0, "seconds": 217.3, "max_memory_gb": 17.18, "ce": 0.3514279210130553, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 1.8303114175796509, "loss_scale": 4.0, "retries": 0, "seconds": 220.7, "max_memory_gb": 17.18, "ce": 0.5798480767753063, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 1.01401686668396, "loss_scale": 4.0, "retries": 0, "seconds": 224.2, "max_memory_gb": 17.18, "ce": 0.3272007702416886, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 2.1002087593078613, "loss_scale": 4.0, "retries": 0, "seconds": 227.5, "max_memory_gb": 17.18, "ce": 0.1103508008782228, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.5820963382720947, "loss_scale": 4.0, "retries": 0, "seconds": 230.9, "max_memory_gb": 17.18, "ce": 0.04257003606653598, "rank": 0.0}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 1.3552095890045166, "loss_scale": 4.0, "retries": 0, "seconds": 234.4, "max_memory_gb": 17.18, "ce": 0.17302848481449473, "rank": 0.0075483620166778564}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.2547125518321991, "loss_scale": 4.0, "retries": 0, "seconds": 237.8, "max_memory_gb": 17.18, "ce": 0.011548275402674335, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.37744224071502686, "loss_scale": 4.0, "retries": 0, "seconds": 241.2, "max_memory_gb": 17.18, "ce": 0.03270252197398804, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.1338895559310913, "loss_scale": 4.0, "retries": 0, "seconds": 244.6, "max_memory_gb": 17.18, "ce": 0.011618654288213293, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 1.1227279901504517, "loss_scale": 4.0, "retries": 0, "seconds": 248.1, "max_memory_gb": 17.18, "ce": 0.1562224132430856, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 1.7524244785308838, "loss_scale": 4.0, "retries": 0, "seconds": 251.4, "max_memory_gb": 17.18, "ce": 0.3541928211452614, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.14719177782535553, "loss_scale": 4.0, "retries": 0, "seconds": 254.8, "max_memory_gb": 17.18, "ce": 0.01803094585193321, "rank": 0.12507955729961395}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 3.9815003871917725, "loss_scale": 4.0, "retries": 0, "seconds": 258.3, "max_memory_gb": 17.18, "ce": 0.2995218860669411, "rank": 0.007845990359783173}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 0.5912206172943115, "loss_scale": 4.0, "retries": 0, "seconds": 261.7, "max_memory_gb": 17.18, "ce": 0.0339980377439133, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.0848848894238472, "loss_scale": 4.0, "retries": 0, "seconds": 265.1, "max_memory_gb": 17.18, "ce": 0.008616350183729082, "rank": 0.0}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.9872722029685974, "loss_scale": 4.0, "retries": 0, "seconds": 268.5, "max_memory_gb": 17.18, "ce": 0.31400147630301944, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.2528180181980133, "loss_scale": 4.0, "retries": 0, "seconds": 271.9, "max_memory_gb": 17.18, "ce": 0.019728378437321226, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.38307636976242065, "loss_scale": 4.0, "retries": 0, "seconds": 275.3, "max_memory_gb": 17.18, "ce": 0.070634122458614, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.38098734617233276, "loss_scale": 4.0, "retries": 0, "seconds": 278.7, "max_memory_gb": 17.18, "ce": 0.03206003525883716, "rank": 0.0}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.673692524433136, "loss_scale": 4.0, "retries": 0, "seconds": 282.1, "max_memory_gb": 17.18, "ce": 0.03496257910819622, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.11300291866064072, "loss_scale": 4.0, "retries": 0, "seconds": 285.5, "max_memory_gb": 17.18, "ce": 0.0117156550868458, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 0.42358118295669556, "loss_scale": 4.0, "retries": 0, "seconds": 288.9, "max_memory_gb": 17.18, "ce": 0.08005395279133154, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 1.4283769130706787, "loss_scale": 4.0, "retries": 0, "seconds": 292.4, "max_memory_gb": 17.18, "ce": 0.0759788914198225, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.10881520062685013, "loss_scale": 4.0, "retries": 0, "seconds": 295.7, "max_memory_gb": 17.18, "ce": 0.014738319773641706, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 1.0440680980682373, "loss_scale": 4.0, "retries": 0, "seconds": 299.1, "max_memory_gb": 17.18, "ce": 0.3696428954544899, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.05509132519364357, "loss_scale": 4.0, "retries": 0, "seconds": 302.5, "max_memory_gb": 17.18, "ce": 0.004493387808906846, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 1.2101799249649048, "loss_scale": 4.0, "retries": 0, "seconds": 305.9, "max_memory_gb": 17.18, "ce": 0.2365198519810292, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.21676595509052277, "loss_scale": 4.0, "retries": 0, "seconds": 309.3, "max_memory_gb": 17.18, "ce": 0.03740725181819471, "rank": 0.0}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 1.0989506244659424, "loss_scale": 4.0, "retries": 0, "seconds": 312.7, "max_memory_gb": 17.18, "ce": 0.20864557310051168, "rank": 0.12494557350873947}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 3.4230563640594482, "loss_scale": 4.0, "retries": 0, "seconds": 316.3, "max_memory_gb": 17.18, "ce": 0.22057149895317707, "rank": 0.12484768778085709}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.24711260199546814, "loss_scale": 4.0, "retries": 0, "seconds": 320.2, "max_memory_gb": 17.18, "ce": 0.01776240999379297, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 0.42682328820228577, "loss_scale": 4.0, "retries": 0, "seconds": 324.0, "max_memory_gb": 17.18, "ce": 0.02525167698036057, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.5296120643615723, "loss_scale": 4.0, "retries": 0, "seconds": 327.4, "max_memory_gb": 17.18, "ce": 0.0385364540950377, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.11062880605459213, "loss_scale": 4.0, "retries": 0, "seconds": 330.8, "max_memory_gb": 17.18, "ce": 0.008473357958337147, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.8864479064941406, "loss_scale": 4.0, "retries": 0, "seconds": 334.2, "max_memory_gb": 17.18, "ce": 0.09259365706373046, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.03600281849503517, "loss_scale": 4.0, "retries": 0, "seconds": 337.6, "max_memory_gb": 17.18, "ce": 0.002724609815686563, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.24288228154182434, "loss_scale": 4.0, "retries": 0, "seconds": 341.3, "max_memory_gb": 17.18, "ce": 0.025106815101935354, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.7390896081924438, "loss_scale": 4.0, "retries": 0, "seconds": 345.0, "max_memory_gb": 17.18, "ce": 0.11554920201251662, "rank": 0.0}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.30702266097068787, "loss_scale": 4.0, "retries": 0, "seconds": 348.8, "max_memory_gb": 17.18, "ce": 0.043284177429313786, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 0.9889817237854004, "loss_scale": 4.0, "retries": 0, "seconds": 352.5, "max_memory_gb": 17.18, "ce": 0.1172043551898696, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 1.4096909761428833, "loss_scale": 4.0, "retries": 0, "seconds": 356.3, "max_memory_gb": 17.18, "ce": 0.1012159683837126, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 1.6945921182632446, "loss_scale": 4.0, "retries": 0, "seconds": 360.1, "max_memory_gb": 17.18, "ce": 0.3647765023508782, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 0.19977913796901703, "loss_scale": 4.0, "retries": 0, "seconds": 363.8, "max_memory_gb": 17.18, "ce": 0.0394008971538824, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 1.2298550605773926, "loss_scale": 4.0, "retries": 0, "seconds": 367.5, "max_memory_gb": 17.18, "ce": 0.3146759421792922, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 2.7618730068206787, "loss_scale": 4.0, "retries": 0, "seconds": 371.1, "max_memory_gb": 17.18, "ce": 0.2135189586792876, "rank": 0.0}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 1.484161376953125, "loss_scale": 4.0, "retries": 0, "seconds": 375.0, "max_memory_gb": 17.18, "ce": 0.07944410403069924, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.08333052694797516, "loss_scale": 4.0, "retries": 0, "seconds": 378.7, "max_memory_gb": 17.18, "ce": 0.00892403331999958, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.4771726727485657, "loss_scale": 4.0, "retries": 0, "seconds": 382.4, "max_memory_gb": 17.18, "ce": 0.084732116559735, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.4297899305820465, "loss_scale": 4.0, "retries": 0, "seconds": 386.2, "max_memory_gb": 17.18, "ce": 0.06834142057141435, "rank": 0.12509159743785858}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 1.2585692405700684, "loss_scale": 4.0, "retries": 0, "seconds": 389.9, "max_memory_gb": 17.18, "ce": 0.306406449592032, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.17584508657455444, "loss_scale": 4.0, "retries": 0, "seconds": 393.6, "max_memory_gb": 17.18, "ce": 0.012137018427893054, "rank": 0.17928502708673477}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.11944954842329025, "loss_scale": 4.0, "retries": 0, "seconds": 397.4, "max_memory_gb": 17.18, "ce": 0.010651017057170975, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 3.227524995803833, "loss_scale": 4.0, "retries": 0, "seconds": 401.1, "max_memory_gb": 17.18, "ce": 0.24488238092726533, "rank": 0.0}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.7845852375030518, "loss_scale": 4.0, "retries": 0, "seconds": 404.8, "max_memory_gb": 17.18, "ce": 0.1358345064054447, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.6105194091796875, "loss_scale": 4.0, "retries": 0, "seconds": 408.6, "max_memory_gb": 17.18, "ce": 0.05212419654344558, "rank": 0.0}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 0.08587448298931122, "loss_scale": 4.0, "retries": 0, "seconds": 412.4, "max_memory_gb": 17.18, "ce": 0.010415468196697475, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 0.2745843529701233, "loss_scale": 4.0, "retries": 0, "seconds": 416.1, "max_memory_gb": 17.18, "ce": 0.039532567800051766, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 0.04248062148690224, "loss_scale": 4.0, "retries": 0, "seconds": 420.0, "max_memory_gb": 17.18, "ce": 0.004701263645074505, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.038320209830999374, "loss_scale": 4.0, "retries": 0, "seconds": 423.6, "max_memory_gb": 17.18, "ce": 0.0035605222778940515, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.8599393963813782, "loss_scale": 4.0, "retries": 0, "seconds": 427.3, "max_memory_gb": 17.18, "ce": 0.1505351929133667, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.08057151734828949, "loss_scale": 4.0, "retries": 0, "seconds": 431.1, "max_memory_gb": 17.18, "ce": 0.010474470706526517, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 1.7440017461776733, "loss_scale": 4.0, "retries": 0, "seconds": 434.9, "max_memory_gb": 17.18, "ce": 0.4232871285485089, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.8320292830467224, "loss_scale": 4.0, "retries": 0, "seconds": 438.5, "max_memory_gb": 17.18, "ce": 0.25911406710815754, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.7964742183685303, "loss_scale": 4.0, "retries": 0, "seconds": 442.2, "max_memory_gb": 17.18, "ce": 0.02916766594580622, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 1.7417235374450684, "loss_scale": 4.0, "retries": 0, "seconds": 446.1, "max_memory_gb": 17.18, "ce": 0.37166202350363164, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.3704460859298706, "loss_scale": 4.0, "retries": 0, "seconds": 449.8, "max_memory_gb": 17.18, "ce": 0.04610824263068025, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 0.7958759665489197, "loss_scale": 4.0, "retries": 0, "seconds": 453.5, "max_memory_gb": 17.18, "ce": 0.07038135849234095, "rank": 0.12500205636024475}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.09141457080841064, "loss_scale": 4.0, "retries": 0, "seconds": 457.3, "max_memory_gb": 17.18, "ce": 0.008798569846476312, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 1.3641444444656372, "loss_scale": 4.0, "retries": 0, "seconds": 461.0, "max_memory_gb": 17.18, "ce": 0.21931867778221203, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.8246560096740723, "loss_scale": 4.0, "retries": 0, "seconds": 464.7, "max_memory_gb": 17.18, "ce": 0.13541516474253967, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.8394246697425842, "loss_scale": 4.0, "retries": 0, "seconds": 468.6, "max_memory_gb": 17.18, "ce": 0.1702746560781634, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 1.5120896100997925, "loss_scale": 4.0, "retries": 0, "seconds": 472.3, "max_memory_gb": 17.18, "ce": 0.3301331824729914, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 0.03162800148129463, "loss_scale": 4.0, "retries": 0, "seconds": 476.0, "max_memory_gb": 17.18, "ce": 0.0018374224511035209, "rank": 0.0}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 1.1094398498535156, "loss_scale": 4.0, "retries": 0, "seconds": 479.7, "max_memory_gb": 17.18, "ce": 0.284689154077455, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 0.19151094555854797, "loss_scale": 4.0, "retries": 0, "seconds": 483.6, "max_memory_gb": 17.18, "ce": 0.03699502860786197, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.0211639441549778, "loss_scale": 4.0, "retries": 0, "seconds": 487.3, "max_memory_gb": 17.18, "ce": 0.0016522097582765127, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 0.13900433480739594, "loss_scale": 4.0, "retries": 0, "seconds": 491.0, "max_memory_gb": 17.18, "ce": 0.013879656210974645, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 0.2747596502304077, "loss_scale": 4.0, "retries": 0, "seconds": 494.8, "max_memory_gb": 17.18, "ce": 0.031269303143744764, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 0.7277802228927612, "loss_scale": 4.0, "retries": 0, "seconds": 498.5, "max_memory_gb": 17.18, "ce": 0.08071857588970488, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 0.45948728919029236, "loss_scale": 4.0, "retries": 0, "seconds": 502.2, "max_memory_gb": 17.18, "ce": 0.11803218910699798, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 0.14073878526687622, "loss_scale": 4.0, "retries": 0, "seconds": 506.1, "max_memory_gb": 17.18, "ce": 0.01040066928453598, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 4.098428726196289, "loss_scale": 4.0, "retries": 0, "seconds": 509.8, "max_memory_gb": 17.18, "ce": 0.4498196909876242, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.051477789878845215, "loss_scale": 4.0, "retries": 0, "seconds": 513.5, "max_memory_gb": 17.18, "ce": 0.00771574415239229, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 1.2430064678192139, "loss_scale": 4.0, "retries": 0, "seconds": 517.2, "max_memory_gb": 17.18, "ce": 0.0703679495304641, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 1.6512924432754517, "loss_scale": 4.0, "retries": 0, "seconds": 521.0, "max_memory_gb": 17.18, "ce": 0.4987251290556287, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 1.6115669012069702, "loss_scale": 4.0, "retries": 0, "seconds": 524.7, "max_memory_gb": 17.18, "ce": 0.181775995028147, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 0.06501193344593048, "loss_scale": 4.0, "retries": 0, "seconds": 528.4, "max_memory_gb": 17.18, "ce": 0.006725464789496982, "rank": 0.08584269881248474}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.06464971601963043, "loss_scale": 4.0, "retries": 0, "seconds": 532.3, "max_memory_gb": 17.18, "ce": 0.0071100865252446965, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.7532241344451904, "loss_scale": 4.0, "retries": 0, "seconds": 536.0, "max_memory_gb": 17.18, "ce": 0.05416367318434823, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.03906503692269325, "loss_scale": 4.0, "retries": 0, "seconds": 539.7, "max_memory_gb": 17.18, "ce": 0.004863746895807708, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.43835213780403137, "loss_scale": 4.0, "retries": 0, "seconds": 543.5, "max_memory_gb": 17.18, "ce": 0.12370280993513916, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 0.946363091468811, "loss_scale": 4.0, "retries": 0, "seconds": 547.2, "max_memory_gb": 17.18, "ce": 0.21239052221756083, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.0813298299908638, "loss_scale": 4.0, "retries": 0, "seconds": 550.9, "max_memory_gb": 17.18, "ce": 0.006243841955893004, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.9647223949432373, "loss_scale": 4.0, "retries": 0, "seconds": 554.7, "max_memory_gb": 17.18, "ce": 0.07337022720548703, "rank": 0.009243115782737732}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.9847042560577393, "loss_scale": 4.0, "retries": 0, "seconds": 558.5, "max_memory_gb": 17.18, "ce": 0.1522928971262445, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 0.5289798974990845, "loss_scale": 4.0, "retries": 0, "seconds": 562.2, "max_memory_gb": 17.18, "ce": 0.05109498304932458, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.6651791930198669, "loss_scale": 4.0, "retries": 0, "seconds": 565.9, "max_memory_gb": 17.18, "ce": 0.0608458140741277, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 1.5328351259231567, "loss_scale": 4.0, "retries": 0, "seconds": 569.7, "max_memory_gb": 17.18, "ce": 0.5533306660565813, "rank": 0.1477024257183075}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.7157093286514282, "loss_scale": 4.0, "retries": 0, "seconds": 573.4, "max_memory_gb": 17.18, "ce": 0.05741406564675344, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.5114960670471191, "loss_scale": 4.0, "retries": 0, "seconds": 577.1, "max_memory_gb": 17.18, "ce": 0.13194556059988827, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 1.0365248918533325, "loss_scale": 4.0, "retries": 0, "seconds": 580.9, "max_memory_gb": 17.18, "ce": 0.3520741302552324, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.08510412275791168, "loss_scale": 4.0, "retries": 0, "seconds": 584.6, "max_memory_gb": 17.18, "ce": 0.005847229995197267, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 1.1060839891433716, "loss_scale": 4.0, "retries": 0, "seconds": 588.3, "max_memory_gb": 17.18, "ce": 0.11934176803401897, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 0.24421322345733643, "loss_scale": 4.0, "retries": 0, "seconds": 592.0, "max_memory_gb": 17.18, "ce": 0.028925697274644335, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.20513981580734253, "loss_scale": 4.0, "retries": 0, "seconds": 595.9, "max_memory_gb": 17.18, "ce": 0.021057689723875228, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 2.789202928543091, "loss_scale": 4.0, "retries": 0, "seconds": 599.6, "max_memory_gb": 17.18, "ce": 0.5075734440295037, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 0.055180665105581284, "loss_scale": 4.0, "retries": 0, "seconds": 603.4, "max_memory_gb": 17.18, "ce": 0.005555189053552567, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 1.1293691396713257, "loss_scale": 4.0, "retries": 0, "seconds": 607.2, "max_memory_gb": 17.18, "ce": 0.1014931012875877, "rank": 0.12474221736192703}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 0.5600832104682922, "loss_scale": 4.0, "retries": 0, "seconds": 610.9, "max_memory_gb": 17.18, "ce": 0.05874133181009711, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 0.33901461958885193, "loss_scale": 4.0, "retries": 0, "seconds": 614.6, "max_memory_gb": 17.18, "ce": 0.05602373543013073, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.24359726905822754, "loss_scale": 4.0, "retries": 0, "seconds": 618.3, "max_memory_gb": 17.18, "ce": 0.028615283581530093, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 0.40404704213142395, "loss_scale": 4.0, "retries": 0, "seconds": 622.1, "max_memory_gb": 17.18, "ce": 0.04661344359897157, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.42260321974754333, "loss_scale": 4.0, "retries": 0, "seconds": 625.9, "max_memory_gb": 17.18, "ce": 0.03432255711959442, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.0320989228785038, "loss_scale": 4.0, "retries": 0, "seconds": 630.3, "max_memory_gb": 17.18, "ce": 0.0026564035051706014, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 0.11496325582265854, "loss_scale": 4.0, "retries": 0, "seconds": 634.9, "max_memory_gb": 17.18, "ce": 0.009648771070260409, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 1.121115803718567, "loss_scale": 4.0, "retries": 0, "seconds": 639.3, "max_memory_gb": 17.18, "ce": 0.2922264874150642, "rank": 0.07103058695793152}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 1.7111800909042358, "loss_scale": 4.0, "retries": 0, "seconds": 643.7, "max_memory_gb": 17.18, "ce": 0.20431628965252457, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 0.06393357366323471, "loss_scale": 4.0, "retries": 0, "seconds": 648.3, "max_memory_gb": 17.18, "ce": 0.004213179038288217, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.6281237602233887, "loss_scale": 4.0, "retries": 0, "seconds": 652.7, "max_memory_gb": 17.18, "ce": 0.140590425406117, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.2622138261795044, "loss_scale": 4.0, "retries": 0, "seconds": 657.1, "max_memory_gb": 17.18, "ce": 0.05080422092964909, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.9767456650733948, "loss_scale": 4.0, "retries": 0, "seconds": 661.5, "max_memory_gb": 17.18, "ce": 0.17922612831785045, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.30578625202178955, "loss_scale": 4.0, "retries": 0, "seconds": 665.9, "max_memory_gb": 17.18, "ce": 0.047705959344966686, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 1.198890209197998, "loss_scale": 4.0, "retries": 0, "seconds": 670.3, "max_memory_gb": 17.18, "ce": 0.1637478170448503, "rank": 0.12486540526151657}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 0.9794846773147583, "loss_scale": 4.0, "retries": 0, "seconds": 674.9, "max_memory_gb": 17.18, "ce": 0.20062588172504547, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.8805906176567078, "loss_scale": 4.0, "retries": 0, "seconds": 679.2, "max_memory_gb": 17.18, "ce": 0.16277062353606198, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 0.5970230102539062, "loss_scale": 4.0, "retries": 0, "seconds": 683.3, "max_memory_gb": 17.18, "ce": 0.0913281998181219, "rank": 0.0}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 0.04200371354818344, "loss_scale": 4.0, "retries": 0, "seconds": 687.1, "max_memory_gb": 17.18, "ce": 0.003644035967909076, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 0.5066088438034058, "loss_scale": 4.0, "retries": 0, "seconds": 690.8, "max_memory_gb": 17.18, "ce": 0.11142308119315203, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.11075595766305923, "loss_scale": 4.0, "retries": 0, "seconds": 694.5, "max_memory_gb": 17.18, "ce": 0.012362820707039646, "rank": 0.1267930269241333}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.7068567276000977, "loss_scale": 4.0, "retries": 0, "seconds": 698.2, "max_memory_gb": 17.18, "ce": 0.13482437469599517, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 0.26804935932159424, "loss_scale": 4.0, "retries": 0, "seconds": 702.0, "max_memory_gb": 17.18, "ce": 0.01923728678923453, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.5956520438194275, "loss_scale": 4.0, "retries": 0, "seconds": 705.7, "max_memory_gb": 17.18, "ce": 0.045114649664355966, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.7774375081062317, "loss_scale": 4.0, "retries": 0, "seconds": 709.4, "max_memory_gb": 17.18, "ce": 0.22175545399136354, "rank": 0.0}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.6047135591506958, "loss_scale": 4.0, "retries": 0, "seconds": 713.2, "max_memory_gb": 17.18, "ce": 0.05347668907620573, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 0.4564187526702881, "loss_scale": 4.0, "retries": 0, "seconds": 716.8, "max_memory_gb": 17.18, "ce": 0.04369877686622203, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 1.1281324625015259, "loss_scale": 4.0, "retries": 0, "seconds": 720.6, "max_memory_gb": 17.18, "ce": 0.13492435660919, "rank": 0.0}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 0.42656180262565613, "loss_scale": 4.0, "retries": 0, "seconds": 724.3, "max_memory_gb": 17.18, "ce": 0.049346162041160824, "rank": 0.12490150332450867}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 0.3975173234939575, "loss_scale": 4.0, "retries": 0, "seconds": 728.0, "max_memory_gb": 17.18, "ce": 0.030818226757673983, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 1.325864553451538, "loss_scale": 4.0, "retries": 0, "seconds": 731.7, "max_memory_gb": 17.18, "ce": 0.17630282569189148, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 0.05196024477481842, "loss_scale": 4.0, "retries": 0, "seconds": 735.5, "max_memory_gb": 17.18, "ce": 0.005338418292012648, "rank": 0.12503118813037872}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 0.1518472135066986, "loss_scale": 4.0, "retries": 0, "seconds": 739.2, "max_memory_gb": 17.18, "ce": 0.03829017038560778, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.1283182054758072, "loss_scale": 4.0, "retries": 0, "seconds": 743.0, "max_memory_gb": 17.18, "ce": 0.013811975660701137, "rank": 0.0}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.4989543557167053, "loss_scale": 4.0, "retries": 0, "seconds": 746.8, "max_memory_gb": 17.18, "ce": 0.06532013215661436, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.32805752754211426, "loss_scale": 4.0, "retries": 0, "seconds": 750.5, "max_memory_gb": 17.18, "ce": 0.0659506795307152, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.054568223655223846, "loss_scale": 4.0, "retries": 0, "seconds": 754.2, "max_memory_gb": 17.18, "ce": 0.005018292367367394, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.009610531851649284, "loss_scale": 4.0, "retries": 0, "seconds": 757.9, "max_memory_gb": 17.18, "ce": 0.0007824438584975724, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.49788734316825867, "loss_scale": 4.0, "retries": 0, "seconds": 761.7, "max_memory_gb": 17.18, "ce": 0.12192177765587076, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 0.029738083481788635, "loss_scale": 4.0, "retries": 0, "seconds": 765.4, "max_memory_gb": 17.18, "ce": 0.002140758241466756, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 0.7678240537643433, "loss_scale": 4.0, "retries": 0, "seconds": 769.1, "max_memory_gb": 17.18, "ce": 0.16509993440013204, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.4652121067047119, "loss_scale": 4.0, "retries": 0, "seconds": 773.0, "max_memory_gb": 17.18, "ce": 0.032733936449176326, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.18240948021411896, "loss_scale": 4.0, "retries": 0, "seconds": 776.7, "max_memory_gb": 17.18, "ce": 0.023862404646365576, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.4097086787223816, "loss_scale": 4.0, "retries": 0, "seconds": 780.3, "max_memory_gb": 17.18, "ce": 0.05676348519023122, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.033426813781261444, "loss_scale": 4.0, "retries": 0, "seconds": 784.1, "max_memory_gb": 17.18, "ce": 0.0024056302161170606, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 1.2687195539474487, "loss_scale": 4.0, "retries": 0, "seconds": 787.8, "max_memory_gb": 17.18, "ce": 0.28876158508091976, "rank": 0.1248602345585823}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 2.6265981197357178, "loss_scale": 4.0, "retries": 0, "seconds": 791.5, "max_memory_gb": 17.18, "ce": 0.3938618801212215, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.6346415281295776, "loss_scale": 4.0, "retries": 0, "seconds": 795.2, "max_memory_gb": 17.18, "ce": 0.02668011030539219, "rank": 0.18119144439697266}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.1743876039981842, "loss_scale": 4.0, "retries": 0, "seconds": 799.0, "max_memory_gb": 17.18, "ce": 0.012606662886810227, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.11468200385570526, "loss_scale": 4.0, "retries": 0, "seconds": 802.7, "max_memory_gb": 17.18, "ce": 0.009281442823521502, "rank": 0.0}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 0.3361991047859192, "loss_scale": 4.0, "retries": 0, "seconds": 806.4, "max_memory_gb": 17.18, "ce": 0.030210452949631872, "rank": 0.0}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 2.0620102882385254, "loss_scale": 4.0, "retries": 0, "seconds": 810.2, "max_memory_gb": 17.18, "ce": 0.46461740343784186, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 0.7167537212371826, "loss_scale": 4.0, "retries": 0, "seconds": 813.9, "max_memory_gb": 17.18, "ce": 0.11654949087812838, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.505178689956665, "loss_scale": 4.0, "retries": 0, "seconds": 817.7, "max_memory_gb": 17.18, "ce": 0.031481694571539265, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 0.488546758890152, "loss_scale": 4.0, "retries": 0, "seconds": 821.4, "max_memory_gb": 17.18, "ce": 0.08828943785181309, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 0.1387643814086914, "loss_scale": 4.0, "retries": 0, "seconds": 825.1, "max_memory_gb": 17.18, "ce": 0.020934584049314253, "rank": 0.08801557123661041}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.5168823003768921, "loss_scale": 4.0, "retries": 0, "seconds": 828.8, "max_memory_gb": 17.18, "ce": 0.028774169854841603, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.19415728747844696, "loss_scale": 4.0, "retries": 0, "seconds": 832.5, "max_memory_gb": 17.18, "ce": 0.013685659841485176, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 0.5093706250190735, "loss_scale": 4.0, "retries": 0, "seconds": 836.3, "max_memory_gb": 17.18, "ce": 0.09789504201535237, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.10797566175460815, "loss_scale": 4.0, "retries": 0, "seconds": 840.0, "max_memory_gb": 17.18, "ce": 0.005800531362410766, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 1.1819978952407837, "loss_scale": 4.0, "retries": 0, "seconds": 843.7, "max_memory_gb": 17.18, "ce": 0.14468419304807867, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.5562521815299988, "loss_scale": 4.0, "retries": 0, "seconds": 847.5, "max_memory_gb": 17.18, "ce": 0.05709224508552779, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 2.234527826309204, "loss_scale": 4.0, "retries": 0, "seconds": 851.2, "max_memory_gb": 17.18, "ce": 0.13268977261327564, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 1.3588664531707764, "loss_scale": 4.0, "retries": 0, "seconds": 854.9, "max_memory_gb": 17.18, "ce": 0.13927837900484974, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.8966917991638184, "loss_scale": 4.0, "retries": 0, "seconds": 858.8, "max_memory_gb": 17.18, "ce": 0.043682605607727965, "rank": 0.1242193728685379}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.18259334564208984, "loss_scale": 4.0, "retries": 0, "seconds": 862.5, "max_memory_gb": 17.18, "ce": 0.008290596498682135, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.7386863827705383, "loss_scale": 4.0, "retries": 0, "seconds": 866.0, "max_memory_gb": 17.18, "ce": 0.044522278520616965, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.5001338720321655, "loss_scale": 4.0, "retries": 0, "seconds": 869.5, "max_memory_gb": 17.18, "ce": 0.09831232446572358, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.619777262210846, "loss_scale": 4.0, "retries": 0, "seconds": 872.9, "max_memory_gb": 17.18, "ce": 0.07874955317049626, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.04535279422998428, "loss_scale": 4.0, "retries": 0, "seconds": 876.3, "max_memory_gb": 17.18, "ce": 0.004020131610786848, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 0.0699869841337204, "loss_scale": 4.0, "retries": 0, "seconds": 879.7, "max_memory_gb": 17.18, "ce": 0.0066948313611874255, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.5174715518951416, "loss_scale": 4.0, "retries": 0, "seconds": 883.1, "max_memory_gb": 17.18, "ce": 0.11658144495356737, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.017512215301394463, "loss_scale": 4.0, "retries": 0, "seconds": 886.5, "max_memory_gb": 17.18, "ce": 0.0012282288654432705, "rank": 0.1249428316950798}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.5533246994018555, "loss_scale": 4.0, "retries": 0, "seconds": 889.8, "max_memory_gb": 17.18, "ce": 0.02975176169684346, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.9200445413589478, "loss_scale": 4.0, "retries": 0, "seconds": 893.3, "max_memory_gb": 17.18, "ce": 0.10579931145343835, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 1.8625285625457764, "loss_scale": 4.0, "retries": 0, "seconds": 896.7, "max_memory_gb": 17.18, "ce": 0.762012755099704, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.06577391922473907, "loss_scale": 4.0, "retries": 0, "seconds": 900.0, "max_memory_gb": 17.18, "ce": 0.00719644248943041, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.8735042214393616, "loss_scale": 4.0, "retries": 0, "seconds": 903.5, "max_memory_gb": 17.18, "ce": 0.051721382840867136, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.19111433625221252, "loss_scale": 4.0, "retries": 0, "seconds": 906.8, "max_memory_gb": 17.18, "ce": 0.021889510657615574, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.7098212838172913, "loss_scale": 4.0, "retries": 0, "seconds": 910.2, "max_memory_gb": 17.18, "ce": 0.10609634006357282, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 0.6147843599319458, "loss_scale": 4.0, "retries": 0, "seconds": 913.6, "max_memory_gb": 17.18, "ce": 0.06405602573477154, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.3651232123374939, "loss_scale": 4.0, "retries": 0, "seconds": 917.0, "max_memory_gb": 17.18, "ce": 0.09371657799079003, "rank": 0.11897692829370499}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.23419998586177826, "loss_scale": 4.0, "retries": 0, "seconds": 920.3, "max_memory_gb": 17.18, "ce": 0.032523968300438355, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.5444049835205078, "loss_scale": 4.0, "retries": 0, "seconds": 923.8, "max_memory_gb": 17.18, "ce": 0.024913973608818196, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.7145923376083374, "loss_scale": 4.0, "retries": 0, "seconds": 927.2, "max_memory_gb": 17.18, "ce": 0.142570839356722, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 0.22206488251686096, "loss_scale": 4.0, "retries": 0, "seconds": 930.6, "max_memory_gb": 17.18, "ce": 0.026848678652072522, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.6196074485778809, "loss_scale": 4.0, "retries": 0, "seconds": 933.9, "max_memory_gb": 17.18, "ce": 0.1399104668498694, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.7383043169975281, "loss_scale": 4.0, "retries": 0, "seconds": 937.4, "max_memory_gb": 17.18, "ce": 0.039539835431924075, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.026117227971553802, "loss_scale": 4.0, "retries": 0, "seconds": 940.8, "max_memory_gb": 17.18, "ce": 0.00119204846384946, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.10456501692533493, "loss_scale": 4.0, "retries": 0, "seconds": 944.2, "max_memory_gb": 17.18, "ce": 0.013100053025027592, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 2.1539127826690674, "loss_scale": 4.0, "retries": 0, "seconds": 947.5, "max_memory_gb": 17.18, "ce": 0.14013635346509545, "rank": 0.0}

runs/fixed_internvl/removal/Full_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/removal/Full_s13/adapter/adapter_config.json

@@ -0,0 +1,556 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.0.mlp.down_proj": 32,
+    "model.language_model.layers.0.mlp.gate_proj": 32,
+    "model.language_model.layers.0.mlp.up_proj": 32,
+    "model.language_model.layers.0.self_attn.k_proj": 32,
+    "model.language_model.layers.0.self_attn.o_proj": 32,
+    "model.language_model.layers.0.self_attn.q_proj": 32,
+    "model.language_model.layers.0.self_attn.v_proj": 32,
+    "model.language_model.layers.1.mlp.down_proj": 32,
+    "model.language_model.layers.1.mlp.gate_proj": 32,
+    "model.language_model.layers.1.mlp.up_proj": 32,
+    "model.language_model.layers.1.self_attn.k_proj": 32,
+    "model.language_model.layers.1.self_attn.o_proj": 32,
+    "model.language_model.layers.1.self_attn.q_proj": 32,
+    "model.language_model.layers.1.self_attn.v_proj": 32,
+    "model.language_model.layers.10.mlp.down_proj": 32,
+    "model.language_model.layers.10.mlp.gate_proj": 32,
+    "model.language_model.layers.10.mlp.up_proj": 32,
+    "model.language_model.layers.10.self_attn.k_proj": 32,
+    "model.language_model.layers.10.self_attn.o_proj": 32,
+    "model.language_model.layers.10.self_attn.q_proj": 32,
+    "model.language_model.layers.10.self_attn.v_proj": 32,
+    "model.language_model.layers.11.mlp.down_proj": 32,
+    "model.language_model.layers.11.mlp.gate_proj": 32,
+    "model.language_model.layers.11.mlp.up_proj": 32,
+    "model.language_model.layers.11.self_attn.k_proj": 32,
+    "model.language_model.layers.11.self_attn.o_proj": 32,
+    "model.language_model.layers.11.self_attn.q_proj": 32,
+    "model.language_model.layers.11.self_attn.v_proj": 32,
+    "model.language_model.layers.12.mlp.down_proj": 32,
+    "model.language_model.layers.12.mlp.gate_proj": 32,
+    "model.language_model.layers.12.mlp.up_proj": 32,
+    "model.language_model.layers.12.self_attn.k_proj": 32,
+    "model.language_model.layers.12.self_attn.o_proj": 32,
+    "model.language_model.layers.12.self_attn.q_proj": 32,
+    "model.language_model.layers.12.self_attn.v_proj": 32,
+    "model.language_model.layers.13.mlp.down_proj": 32,
+    "model.language_model.layers.13.mlp.gate_proj": 32,
+    "model.language_model.layers.13.mlp.up_proj": 32,
+    "model.language_model.layers.13.self_attn.k_proj": 32,
+    "model.language_model.layers.13.self_attn.o_proj": 32,
+    "model.language_model.layers.13.self_attn.q_proj": 32,
+    "model.language_model.layers.13.self_attn.v_proj": 32,
+    "model.language_model.layers.14.mlp.down_proj": 32,
+    "model.language_model.layers.14.mlp.gate_proj": 32,
+    "model.language_model.layers.14.mlp.up_proj": 32,
+    "model.language_model.layers.14.self_attn.k_proj": 32,
+    "model.language_model.layers.14.self_attn.o_proj": 32,
+    "model.language_model.layers.14.self_attn.q_proj": 32,
+    "model.language_model.layers.14.self_attn.v_proj": 32,
+    "model.language_model.layers.15.mlp.down_proj": 32,
+    "model.language_model.layers.15.mlp.gate_proj": 32,
+    "model.language_model.layers.15.mlp.up_proj": 32,
+    "model.language_model.layers.15.self_attn.k_proj": 32,
+    "model.language_model.layers.15.self_attn.o_proj": 32,
+    "model.language_model.layers.15.self_attn.q_proj": 32,
+    "model.language_model.layers.15.self_attn.v_proj": 32,
+    "model.language_model.layers.16.mlp.down_proj": 32,
+    "model.language_model.layers.16.mlp.gate_proj": 32,
+    "model.language_model.layers.16.mlp.up_proj": 32,
+    "model.language_model.layers.16.self_attn.k_proj": 32,
+    "model.language_model.layers.16.self_attn.o_proj": 32,
+    "model.language_model.layers.16.self_attn.q_proj": 32,
+    "model.language_model.layers.16.self_attn.v_proj": 32,
+    "model.language_model.layers.17.mlp.down_proj": 32,
+    "model.language_model.layers.17.mlp.gate_proj": 32,
+    "model.language_model.layers.17.mlp.up_proj": 32,
+    "model.language_model.layers.17.self_attn.k_proj": 32,
+    "model.language_model.layers.17.self_attn.o_proj": 32,
+    "model.language_model.layers.17.self_attn.q_proj": 32,
+    "model.language_model.layers.17.self_attn.v_proj": 32,
+    "model.language_model.layers.18.mlp.down_proj": 32,
+    "model.language_model.layers.18.mlp.gate_proj": 32,
+    "model.language_model.layers.18.mlp.up_proj": 32,
+    "model.language_model.layers.18.self_attn.k_proj": 32,
+    "model.language_model.layers.18.self_attn.o_proj": 32,
+    "model.language_model.layers.18.self_attn.q_proj": 32,
+    "model.language_model.layers.18.self_attn.v_proj": 32,
+    "model.language_model.layers.19.mlp.down_proj": 32,
+    "model.language_model.layers.19.mlp.gate_proj": 32,
+    "model.language_model.layers.19.mlp.up_proj": 32,
+    "model.language_model.layers.19.self_attn.k_proj": 32,
+    "model.language_model.layers.19.self_attn.o_proj": 32,
+    "model.language_model.layers.19.self_attn.q_proj": 32,
+    "model.language_model.layers.19.self_attn.v_proj": 32,
+    "model.language_model.layers.2.mlp.down_proj": 32,
+    "model.language_model.layers.2.mlp.gate_proj": 32,
+    "model.language_model.layers.2.mlp.up_proj": 32,
+    "model.language_model.layers.2.self_attn.k_proj": 32,
+    "model.language_model.layers.2.self_attn.o_proj": 32,
+    "model.language_model.layers.2.self_attn.q_proj": 32,
+    "model.language_model.layers.2.self_attn.v_proj": 32,
+    "model.language_model.layers.20.mlp.down_proj": 32,
+    "model.language_model.layers.20.mlp.gate_proj": 32,
+    "model.language_model.layers.20.mlp.up_proj": 32,
+    "model.language_model.layers.20.self_attn.k_proj": 32,
+    "model.language_model.layers.20.self_attn.o_proj": 32,
+    "model.language_model.layers.20.self_attn.q_proj": 32,
+    "model.language_model.layers.20.self_attn.v_proj": 32,
+    "model.language_model.layers.21.mlp.down_proj": 32,
+    "model.language_model.layers.21.mlp.gate_proj": 32,
+    "model.language_model.layers.21.mlp.up_proj": 32,
+    "model.language_model.layers.21.self_attn.k_proj": 32,
+    "model.language_model.layers.21.self_attn.o_proj": 32,
+    "model.language_model.layers.21.self_attn.q_proj": 32,
+    "model.language_model.layers.21.self_attn.v_proj": 32,
+    "model.language_model.layers.22.mlp.down_proj": 32,
+    "model.language_model.layers.22.mlp.gate_proj": 32,
+    "model.language_model.layers.22.mlp.up_proj": 32,
+    "model.language_model.layers.22.self_attn.k_proj": 32,
+    "model.language_model.layers.22.self_attn.o_proj": 32,
+    "model.language_model.layers.22.self_attn.q_proj": 32,
+    "model.language_model.layers.22.self_attn.v_proj": 32,
+    "model.language_model.layers.23.mlp.down_proj": 32,
+    "model.language_model.layers.23.mlp.gate_proj": 32,
+    "model.language_model.layers.23.mlp.up_proj": 32,
+    "model.language_model.layers.23.self_attn.k_proj": 32,
+    "model.language_model.layers.23.self_attn.o_proj": 32,
+    "model.language_model.layers.23.self_attn.q_proj": 32,
+    "model.language_model.layers.23.self_attn.v_proj": 32,
+    "model.language_model.layers.24.mlp.down_proj": 32,
+    "model.language_model.layers.24.mlp.gate_proj": 32,
+    "model.language_model.layers.24.mlp.up_proj": 32,
+    "model.language_model.layers.24.self_attn.k_proj": 32,
+    "model.language_model.layers.24.self_attn.o_proj": 32,
+    "model.language_model.layers.24.self_attn.q_proj": 32,
+    "model.language_model.layers.24.self_attn.v_proj": 32,
+    "model.language_model.layers.25.mlp.down_proj": 32,
+    "model.language_model.layers.25.mlp.gate_proj": 32,
+    "model.language_model.layers.25.mlp.up_proj": 32,
+    "model.language_model.layers.25.self_attn.k_proj": 32,
+    "model.language_model.layers.25.self_attn.o_proj": 32,
+    "model.language_model.layers.25.self_attn.q_proj": 32,
+    "model.language_model.layers.25.self_attn.v_proj": 32,
+    "model.language_model.layers.26.mlp.down_proj": 32,
+    "model.language_model.layers.26.mlp.gate_proj": 32,
+    "model.language_model.layers.26.mlp.up_proj": 32,
+    "model.language_model.layers.26.self_attn.k_proj": 32,
+    "model.language_model.layers.26.self_attn.o_proj": 32,
+    "model.language_model.layers.26.self_attn.q_proj": 32,
+    "model.language_model.layers.26.self_attn.v_proj": 32,
+    "model.language_model.layers.27.mlp.down_proj": 32,
+    "model.language_model.layers.27.mlp.gate_proj": 32,
+    "model.language_model.layers.27.mlp.up_proj": 32,
+    "model.language_model.layers.27.self_attn.k_proj": 32,
+    "model.language_model.layers.27.self_attn.o_proj": 32,
+    "model.language_model.layers.27.self_attn.q_proj": 32,
+    "model.language_model.layers.27.self_attn.v_proj": 32,
+    "model.language_model.layers.28.mlp.down_proj": 32,
+    "model.language_model.layers.28.mlp.gate_proj": 32,
+    "model.language_model.layers.28.mlp.up_proj": 32,
+    "model.language_model.layers.28.self_attn.k_proj": 32,
+    "model.language_model.layers.28.self_attn.o_proj": 32,
+    "model.language_model.layers.28.self_attn.q_proj": 32,
+    "model.language_model.layers.28.self_attn.v_proj": 32,
+    "model.language_model.layers.29.mlp.down_proj": 32,
+    "model.language_model.layers.29.mlp.gate_proj": 32,
+    "model.language_model.layers.29.mlp.up_proj": 32,
+    "model.language_model.layers.29.self_attn.k_proj": 32,
+    "model.language_model.layers.29.self_attn.o_proj": 32,
+    "model.language_model.layers.29.self_attn.q_proj": 32,
+    "model.language_model.layers.29.self_attn.v_proj": 32,
+    "model.language_model.layers.3.mlp.down_proj": 32,
+    "model.language_model.layers.3.mlp.gate_proj": 32,
+    "model.language_model.layers.3.mlp.up_proj": 32,
+    "model.language_model.layers.3.self_attn.k_proj": 32,
+    "model.language_model.layers.3.self_attn.o_proj": 32,
+    "model.language_model.layers.3.self_attn.q_proj": 32,
+    "model.language_model.layers.3.self_attn.v_proj": 32,
+    "model.language_model.layers.30.mlp.down_proj": 32,
+    "model.language_model.layers.30.mlp.gate_proj": 32,
+    "model.language_model.layers.30.mlp.up_proj": 32,
+    "model.language_model.layers.30.self_attn.k_proj": 32,
+    "model.language_model.layers.30.self_attn.o_proj": 32,
+    "model.language_model.layers.30.self_attn.q_proj": 32,
+    "model.language_model.layers.30.self_attn.v_proj": 32,
+    "model.language_model.layers.31.mlp.down_proj": 32,
+    "model.language_model.layers.31.mlp.gate_proj": 32,
+    "model.language_model.layers.31.mlp.up_proj": 32,
+    "model.language_model.layers.31.self_attn.k_proj": 32,
+    "model.language_model.layers.31.self_attn.o_proj": 32,
+    "model.language_model.layers.31.self_attn.q_proj": 32,
+    "model.language_model.layers.31.self_attn.v_proj": 32,
+    "model.language_model.layers.32.mlp.down_proj": 32,
+    "model.language_model.layers.32.mlp.gate_proj": 32,
+    "model.language_model.layers.32.mlp.up_proj": 32,
+    "model.language_model.layers.32.self_attn.k_proj": 32,
+    "model.language_model.layers.32.self_attn.o_proj": 32,
+    "model.language_model.layers.32.self_attn.q_proj": 32,
+    "model.language_model.layers.32.self_attn.v_proj": 32,
+    "model.language_model.layers.33.mlp.down_proj": 32,
+    "model.language_model.layers.33.mlp.gate_proj": 32,
+    "model.language_model.layers.33.mlp.up_proj": 32,
+    "model.language_model.layers.33.self_attn.k_proj": 32,
+    "model.language_model.layers.33.self_attn.o_proj": 32,
+    "model.language_model.layers.33.self_attn.q_proj": 32,
+    "model.language_model.layers.33.self_attn.v_proj": 32,
+    "model.language_model.layers.34.mlp.down_proj": 32,
+    "model.language_model.layers.34.mlp.gate_proj": 32,
+    "model.language_model.layers.34.mlp.up_proj": 32,
+    "model.language_model.layers.34.self_attn.k_proj": 32,
+    "model.language_model.layers.34.self_attn.o_proj": 32,
+    "model.language_model.layers.34.self_attn.q_proj": 32,
+    "model.language_model.layers.34.self_attn.v_proj": 32,
+    "model.language_model.layers.35.mlp.down_proj": 32,
+    "model.language_model.layers.35.mlp.gate_proj": 32,
+    "model.language_model.layers.35.mlp.up_proj": 32,
+    "model.language_model.layers.35.self_attn.k_proj": 32,
+    "model.language_model.layers.35.self_attn.o_proj": 32,
+    "model.language_model.layers.35.self_attn.q_proj": 32,
+    "model.language_model.layers.35.self_attn.v_proj": 32,
+    "model.language_model.layers.4.mlp.down_proj": 32,
+    "model.language_model.layers.4.mlp.gate_proj": 32,
+    "model.language_model.layers.4.mlp.up_proj": 32,
+    "model.language_model.layers.4.self_attn.k_proj": 32,
+    "model.language_model.layers.4.self_attn.o_proj": 32,
+    "model.language_model.layers.4.self_attn.q_proj": 32,
+    "model.language_model.layers.4.self_attn.v_proj": 32,
+    "model.language_model.layers.5.mlp.down_proj": 32,
+    "model.language_model.layers.5.mlp.gate_proj": 32,
+    "model.language_model.layers.5.mlp.up_proj": 32,
+    "model.language_model.layers.5.self_attn.k_proj": 32,
+    "model.language_model.layers.5.self_attn.o_proj": 32,
+    "model.language_model.layers.5.self_attn.q_proj": 32,
+    "model.language_model.layers.5.self_attn.v_proj": 32,
+    "model.language_model.layers.6.mlp.down_proj": 32,
+    "model.language_model.layers.6.mlp.gate_proj": 32,
+    "model.language_model.layers.6.mlp.up_proj": 32,
+    "model.language_model.layers.6.self_attn.k_proj": 32,
+    "model.language_model.layers.6.self_attn.o_proj": 32,
+    "model.language_model.layers.6.self_attn.q_proj": 32,
+    "model.language_model.layers.6.self_attn.v_proj": 32,
+    "model.language_model.layers.7.mlp.down_proj": 32,
+    "model.language_model.layers.7.mlp.gate_proj": 32,
+    "model.language_model.layers.7.mlp.up_proj": 32,
+    "model.language_model.layers.7.self_attn.k_proj": 32,
+    "model.language_model.layers.7.self_attn.o_proj": 32,
+    "model.language_model.layers.7.self_attn.q_proj": 32,
+    "model.language_model.layers.7.self_attn.v_proj": 32,
+    "model.language_model.layers.8.mlp.down_proj": 32,
+    "model.language_model.layers.8.mlp.gate_proj": 32,
+    "model.language_model.layers.8.mlp.up_proj": 32,
+    "model.language_model.layers.8.self_attn.k_proj": 32,
+    "model.language_model.layers.8.self_attn.o_proj": 32,
+    "model.language_model.layers.8.self_attn.q_proj": 32,
+    "model.language_model.layers.8.self_attn.v_proj": 32,
+    "model.language_model.layers.9.mlp.down_proj": 32,
+    "model.language_model.layers.9.mlp.gate_proj": 32,
+    "model.language_model.layers.9.mlp.up_proj": 32,
+    "model.language_model.layers.9.self_attn.k_proj": 32,
+    "model.language_model.layers.9.self_attn.o_proj": 32,
+    "model.language_model.layers.9.self_attn.q_proj": 32,
+    "model.language_model.layers.9.self_attn.v_proj": 32
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.0.mlp.down_proj": 16,
+    "model.language_model.layers.0.mlp.gate_proj": 16,
+    "model.language_model.layers.0.mlp.up_proj": 16,
+    "model.language_model.layers.0.self_attn.k_proj": 16,
+    "model.language_model.layers.0.self_attn.o_proj": 16,
+    "model.language_model.layers.0.self_attn.q_proj": 16,
+    "model.language_model.layers.0.self_attn.v_proj": 16,
+    "model.language_model.layers.1.mlp.down_proj": 16,
+    "model.language_model.layers.1.mlp.gate_proj": 16,
+    "model.language_model.layers.1.mlp.up_proj": 16,
+    "model.language_model.layers.1.self_attn.k_proj": 16,
+    "model.language_model.layers.1.self_attn.o_proj": 16,
+    "model.language_model.layers.1.self_attn.q_proj": 16,
+    "model.language_model.layers.1.self_attn.v_proj": 16,
+    "model.language_model.layers.10.mlp.down_proj": 16,
+    "model.language_model.layers.10.mlp.gate_proj": 16,
+    "model.language_model.layers.10.mlp.up_proj": 16,
+    "model.language_model.layers.10.self_attn.k_proj": 16,
+    "model.language_model.layers.10.self_attn.o_proj": 16,
+    "model.language_model.layers.10.self_attn.q_proj": 16,
+    "model.language_model.layers.10.self_attn.v_proj": 16,
+    "model.language_model.layers.11.mlp.down_proj": 16,
+    "model.language_model.layers.11.mlp.gate_proj": 16,
+    "model.language_model.layers.11.mlp.up_proj": 16,
+    "model.language_model.layers.11.self_attn.k_proj": 16,
+    "model.language_model.layers.11.self_attn.o_proj": 16,
+    "model.language_model.layers.11.self_attn.q_proj": 16,
+    "model.language_model.layers.11.self_attn.v_proj": 16,
+    "model.language_model.layers.12.mlp.down_proj": 16,
+    "model.language_model.layers.12.mlp.gate_proj": 16,
+    "model.language_model.layers.12.mlp.up_proj": 16,
+    "model.language_model.layers.12.self_attn.k_proj": 16,
+    "model.language_model.layers.12.self_attn.o_proj": 16,
+    "model.language_model.layers.12.self_attn.q_proj": 16,
+    "model.language_model.layers.12.self_attn.v_proj": 16,
+    "model.language_model.layers.13.mlp.down_proj": 16,
+    "model.language_model.layers.13.mlp.gate_proj": 16,
+    "model.language_model.layers.13.mlp.up_proj": 16,
+    "model.language_model.layers.13.self_attn.k_proj": 16,
+    "model.language_model.layers.13.self_attn.o_proj": 16,
+    "model.language_model.layers.13.self_attn.q_proj": 16,
+    "model.language_model.layers.13.self_attn.v_proj": 16,
+    "model.language_model.layers.14.mlp.down_proj": 16,
+    "model.language_model.layers.14.mlp.gate_proj": 16,
+    "model.language_model.layers.14.mlp.up_proj": 16,
+    "model.language_model.layers.14.self_attn.k_proj": 16,
+    "model.language_model.layers.14.self_attn.o_proj": 16,
+    "model.language_model.layers.14.self_attn.q_proj": 16,
+    "model.language_model.layers.14.self_attn.v_proj": 16,
+    "model.language_model.layers.15.mlp.down_proj": 16,
+    "model.language_model.layers.15.mlp.gate_proj": 16,
+    "model.language_model.layers.15.mlp.up_proj": 16,
+    "model.language_model.layers.15.self_attn.k_proj": 16,
+    "model.language_model.layers.15.self_attn.o_proj": 16,
+    "model.language_model.layers.15.self_attn.q_proj": 16,
+    "model.language_model.layers.15.self_attn.v_proj": 16,
+    "model.language_model.layers.16.mlp.down_proj": 16,
+    "model.language_model.layers.16.mlp.gate_proj": 16,
+    "model.language_model.layers.16.mlp.up_proj": 16,
+    "model.language_model.layers.16.self_attn.k_proj": 16,
+    "model.language_model.layers.16.self_attn.o_proj": 16,
+    "model.language_model.layers.16.self_attn.q_proj": 16,
+    "model.language_model.layers.16.self_attn.v_proj": 16,
+    "model.language_model.layers.17.mlp.down_proj": 16,
+    "model.language_model.layers.17.mlp.gate_proj": 16,
+    "model.language_model.layers.17.mlp.up_proj": 16,
+    "model.language_model.layers.17.self_attn.k_proj": 16,
+    "model.language_model.layers.17.self_attn.o_proj": 16,
+    "model.language_model.layers.17.self_attn.q_proj": 16,
+    "model.language_model.layers.17.self_attn.v_proj": 16,
+    "model.language_model.layers.18.mlp.down_proj": 16,
+    "model.language_model.layers.18.mlp.gate_proj": 16,
+    "model.language_model.layers.18.mlp.up_proj": 16,
+    "model.language_model.layers.18.self_attn.k_proj": 16,
+    "model.language_model.layers.18.self_attn.o_proj": 16,
+    "model.language_model.layers.18.self_attn.q_proj": 16,
+    "model.language_model.layers.18.self_attn.v_proj": 16,
+    "model.language_model.layers.19.mlp.down_proj": 16,
+    "model.language_model.layers.19.mlp.gate_proj": 16,
+    "model.language_model.layers.19.mlp.up_proj": 16,
+    "model.language_model.layers.19.self_attn.k_proj": 16,
+    "model.language_model.layers.19.self_attn.o_proj": 16,
+    "model.language_model.layers.19.self_attn.q_proj": 16,
+    "model.language_model.layers.19.self_attn.v_proj": 16,
+    "model.language_model.layers.2.mlp.down_proj": 16,
+    "model.language_model.layers.2.mlp.gate_proj": 16,
+    "model.language_model.layers.2.mlp.up_proj": 16,
+    "model.language_model.layers.2.self_attn.k_proj": 16,
+    "model.language_model.layers.2.self_attn.o_proj": 16,
+    "model.language_model.layers.2.self_attn.q_proj": 16,
+    "model.language_model.layers.2.self_attn.v_proj": 16,
+    "model.language_model.layers.20.mlp.down_proj": 16,
+    "model.language_model.layers.20.mlp.gate_proj": 16,
+    "model.language_model.layers.20.mlp.up_proj": 16,
+    "model.language_model.layers.20.self_attn.k_proj": 16,
+    "model.language_model.layers.20.self_attn.o_proj": 16,
+    "model.language_model.layers.20.self_attn.q_proj": 16,
+    "model.language_model.layers.20.self_attn.v_proj": 16,
+    "model.language_model.layers.21.mlp.down_proj": 16,
+    "model.language_model.layers.21.mlp.gate_proj": 16,
+    "model.language_model.layers.21.mlp.up_proj": 16,
+    "model.language_model.layers.21.self_attn.k_proj": 16,
+    "model.language_model.layers.21.self_attn.o_proj": 16,
+    "model.language_model.layers.21.self_attn.q_proj": 16,
+    "model.language_model.layers.21.self_attn.v_proj": 16,
+    "model.language_model.layers.22.mlp.down_proj": 16,
+    "model.language_model.layers.22.mlp.gate_proj": 16,
+    "model.language_model.layers.22.mlp.up_proj": 16,
+    "model.language_model.layers.22.self_attn.k_proj": 16,
+    "model.language_model.layers.22.self_attn.o_proj": 16,
+    "model.language_model.layers.22.self_attn.q_proj": 16,
+    "model.language_model.layers.22.self_attn.v_proj": 16,
+    "model.language_model.layers.23.mlp.down_proj": 16,
+    "model.language_model.layers.23.mlp.gate_proj": 16,
+    "model.language_model.layers.23.mlp.up_proj": 16,
+    "model.language_model.layers.23.self_attn.k_proj": 16,
+    "model.language_model.layers.23.self_attn.o_proj": 16,
+    "model.language_model.layers.23.self_attn.q_proj": 16,
+    "model.language_model.layers.23.self_attn.v_proj": 16,
+    "model.language_model.layers.24.mlp.down_proj": 16,
+    "model.language_model.layers.24.mlp.gate_proj": 16,
+    "model.language_model.layers.24.mlp.up_proj": 16,
+    "model.language_model.layers.24.self_attn.k_proj": 16,
+    "model.language_model.layers.24.self_attn.o_proj": 16,
+    "model.language_model.layers.24.self_attn.q_proj": 16,
+    "model.language_model.layers.24.self_attn.v_proj": 16,
+    "model.language_model.layers.25.mlp.down_proj": 16,
+    "model.language_model.layers.25.mlp.gate_proj": 16,
+    "model.language_model.layers.25.mlp.up_proj": 16,
+    "model.language_model.layers.25.self_attn.k_proj": 16,
+    "model.language_model.layers.25.self_attn.o_proj": 16,
+    "model.language_model.layers.25.self_attn.q_proj": 16,
+    "model.language_model.layers.25.self_attn.v_proj": 16,
+    "model.language_model.layers.26.mlp.down_proj": 16,
+    "model.language_model.layers.26.mlp.gate_proj": 16,
+    "model.language_model.layers.26.mlp.up_proj": 16,
+    "model.language_model.layers.26.self_attn.k_proj": 16,
+    "model.language_model.layers.26.self_attn.o_proj": 16,
+    "model.language_model.layers.26.self_attn.q_proj": 16,
+    "model.language_model.layers.26.self_attn.v_proj": 16,
+    "model.language_model.layers.27.mlp.down_proj": 16,
+    "model.language_model.layers.27.mlp.gate_proj": 16,
+    "model.language_model.layers.27.mlp.up_proj": 16,
+    "model.language_model.layers.27.self_attn.k_proj": 16,
+    "model.language_model.layers.27.self_attn.o_proj": 16,
+    "model.language_model.layers.27.self_attn.q_proj": 16,
+    "model.language_model.layers.27.self_attn.v_proj": 16,
+    "model.language_model.layers.28.mlp.down_proj": 16,
+    "model.language_model.layers.28.mlp.gate_proj": 16,
+    "model.language_model.layers.28.mlp.up_proj": 16,
+    "model.language_model.layers.28.self_attn.k_proj": 16,
+    "model.language_model.layers.28.self_attn.o_proj": 16,
+    "model.language_model.layers.28.self_attn.q_proj": 16,
+    "model.language_model.layers.28.self_attn.v_proj": 16,
+    "model.language_model.layers.29.mlp.down_proj": 16,
+    "model.language_model.layers.29.mlp.gate_proj": 16,
+    "model.language_model.layers.29.mlp.up_proj": 16,
+    "model.language_model.layers.29.self_attn.k_proj": 16,
+    "model.language_model.layers.29.self_attn.o_proj": 16,
+    "model.language_model.layers.29.self_attn.q_proj": 16,
+    "model.language_model.layers.29.self_attn.v_proj": 16,
+    "model.language_model.layers.3.mlp.down_proj": 16,
+    "model.language_model.layers.3.mlp.gate_proj": 16,
+    "model.language_model.layers.3.mlp.up_proj": 16,
+    "model.language_model.layers.3.self_attn.k_proj": 16,
+    "model.language_model.layers.3.self_attn.o_proj": 16,
+    "model.language_model.layers.3.self_attn.q_proj": 16,
+    "model.language_model.layers.3.self_attn.v_proj": 16,
+    "model.language_model.layers.30.mlp.down_proj": 16,
+    "model.language_model.layers.30.mlp.gate_proj": 16,
+    "model.language_model.layers.30.mlp.up_proj": 16,
+    "model.language_model.layers.30.self_attn.k_proj": 16,
+    "model.language_model.layers.30.self_attn.o_proj": 16,
+    "model.language_model.layers.30.self_attn.q_proj": 16,
+    "model.language_model.layers.30.self_attn.v_proj": 16,
+    "model.language_model.layers.31.mlp.down_proj": 16,
+    "model.language_model.layers.31.mlp.gate_proj": 16,
+    "model.language_model.layers.31.mlp.up_proj": 16,
+    "model.language_model.layers.31.self_attn.k_proj": 16,
+    "model.language_model.layers.31.self_attn.o_proj": 16,
+    "model.language_model.layers.31.self_attn.q_proj": 16,
+    "model.language_model.layers.31.self_attn.v_proj": 16,
+    "model.language_model.layers.32.mlp.down_proj": 16,
+    "model.language_model.layers.32.mlp.gate_proj": 16,
+    "model.language_model.layers.32.mlp.up_proj": 16,
+    "model.language_model.layers.32.self_attn.k_proj": 16,
+    "model.language_model.layers.32.self_attn.o_proj": 16,
+    "model.language_model.layers.32.self_attn.q_proj": 16,
+    "model.language_model.layers.32.self_attn.v_proj": 16,
+    "model.language_model.layers.33.mlp.down_proj": 16,
+    "model.language_model.layers.33.mlp.gate_proj": 16,
+    "model.language_model.layers.33.mlp.up_proj": 16,
+    "model.language_model.layers.33.self_attn.k_proj": 16,
+    "model.language_model.layers.33.self_attn.o_proj": 16,
+    "model.language_model.layers.33.self_attn.q_proj": 16,
+    "model.language_model.layers.33.self_attn.v_proj": 16,
+    "model.language_model.layers.34.mlp.down_proj": 16,
+    "model.language_model.layers.34.mlp.gate_proj": 16,
+    "model.language_model.layers.34.mlp.up_proj": 16,
+    "model.language_model.layers.34.self_attn.k_proj": 16,
+    "model.language_model.layers.34.self_attn.o_proj": 16,
+    "model.language_model.layers.34.self_attn.q_proj": 16,
+    "model.language_model.layers.34.self_attn.v_proj": 16,
+    "model.language_model.layers.35.mlp.down_proj": 16,
+    "model.language_model.layers.35.mlp.gate_proj": 16,
+    "model.language_model.layers.35.mlp.up_proj": 16,
+    "model.language_model.layers.35.self_attn.k_proj": 16,
+    "model.language_model.layers.35.self_attn.o_proj": 16,
+    "model.language_model.layers.35.self_attn.q_proj": 16,
+    "model.language_model.layers.35.self_attn.v_proj": 16,
+    "model.language_model.layers.4.mlp.down_proj": 16,
+    "model.language_model.layers.4.mlp.gate_proj": 16,
+    "model.language_model.layers.4.mlp.up_proj": 16,
+    "model.language_model.layers.4.self_attn.k_proj": 16,
+    "model.language_model.layers.4.self_attn.o_proj": 16,
+    "model.language_model.layers.4.self_attn.q_proj": 16,
+    "model.language_model.layers.4.self_attn.v_proj": 16,
+    "model.language_model.layers.5.mlp.down_proj": 16,
+    "model.language_model.layers.5.mlp.gate_proj": 16,
+    "model.language_model.layers.5.mlp.up_proj": 16,
+    "model.language_model.layers.5.self_attn.k_proj": 16,
+    "model.language_model.layers.5.self_attn.o_proj": 16,
+    "model.language_model.layers.5.self_attn.q_proj": 16,
+    "model.language_model.layers.5.self_attn.v_proj": 16,
+    "model.language_model.layers.6.mlp.down_proj": 16,
+    "model.language_model.layers.6.mlp.gate_proj": 16,
+    "model.language_model.layers.6.mlp.up_proj": 16,
+    "model.language_model.layers.6.self_attn.k_proj": 16,
+    "model.language_model.layers.6.self_attn.o_proj": 16,
+    "model.language_model.layers.6.self_attn.q_proj": 16,
+    "model.language_model.layers.6.self_attn.v_proj": 16,
+    "model.language_model.layers.7.mlp.down_proj": 16,
+    "model.language_model.layers.7.mlp.gate_proj": 16,
+    "model.language_model.layers.7.mlp.up_proj": 16,
+    "model.language_model.layers.7.self_attn.k_proj": 16,
+    "model.language_model.layers.7.self_attn.o_proj": 16,
+    "model.language_model.layers.7.self_attn.q_proj": 16,
+    "model.language_model.layers.7.self_attn.v_proj": 16,
+    "model.language_model.layers.8.mlp.down_proj": 16,
+    "model.language_model.layers.8.mlp.gate_proj": 16,
+    "model.language_model.layers.8.mlp.up_proj": 16,
+    "model.language_model.layers.8.self_attn.k_proj": 16,
+    "model.language_model.layers.8.self_attn.o_proj": 16,
+    "model.language_model.layers.8.self_attn.q_proj": 16,
+    "model.language_model.layers.8.self_attn.v_proj": 16,
+    "model.language_model.layers.9.mlp.down_proj": 16,
+    "model.language_model.layers.9.mlp.gate_proj": 16,
+    "model.language_model.layers.9.mlp.up_proj": 16,
+    "model.language_model.layers.9.self_attn.k_proj": 16,
+    "model.language_model.layers.9.self_attn.o_proj": 16,
+    "model.language_model.layers.9.self_attn.q_proj": 16,
+    "model.language_model.layers.9.self_attn.v_proj": 16
+  },
+  "revision": null,
+  "target_modules": [
+    "up_proj",
+    "gate_proj",
+    "o_proj",
+    "self_attn.v_proj",
+    "down_proj",
+    "self_attn.q_proj",
+    "self_attn.k_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/fixed_internvl/removal/Full_s13/config.json

@@ -0,0 +1,99 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "world_size": 1,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "arm": {
+    "arm": "Full",
+    "layers": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      10,
+      11,
+      12,
+      13,
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21,
+      22,
+      23,
+      24,
+      25,
+      26,
+      27,
+      28,
+      29,
+      30,
+      31,
+      32,
+      33,
+      34,
+      35
+    ],
+    "ranks": {
+      "0": 16,
+      "1": 16,
+      "2": 16,
+      "3": 16,
+      "4": 16,
+      "5": 16,
+      "6": 16,
+      "7": 16,
+      "8": 16,
+      "9": 16,
+      "10": 16,
+      "11": 16,
+      "12": 16,
+      "13": 16,
+      "14": 16,
+      "15": 16,
+      "16": 16,
+      "17": 16,
+      "18": 16,
+      "19": 16,
+      "20": 16,
+      "21": 16,
+      "22": 16,
+      "23": 16,
+      "24": 16,
+      "25": 16,
+      "26": 16,
+      "27": 16,
+      "28": 16,
+      "29": 16,
+      "30": 16,
+      "31": 16,
+      "32": 16,
+      "33": 16,
+      "34": 16,
+      "35": 16
+    },
+    "parameters": 43646976,
+    "budget": 43646976
+  }
+}

runs/fixed_internvl/removal/Full_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 4.981058120727539, "loss_scale": 4.0, "retries": 0, "seconds": 8.6, "max_memory_gb": 16.8, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 4.590186595916748, "loss_scale": 4.0, "retries": 0, "seconds": 16.5, "max_memory_gb": 17.13, "ce": 0.77901219269188, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 17.192995071411133, "loss_scale": 4.0, "retries": 0, "seconds": 24.1, "max_memory_gb": 17.13, "ce": 1.3247385900467634, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 8.059958457946777, "loss_scale": 4.0, "retries": 0, "seconds": 32.7, "max_memory_gb": 17.13, "ce": 0.6640835713060369, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.1461889147758484, "loss_scale": 4.0, "retries": 0, "seconds": 39.7, "max_memory_gb": 17.13, "ce": 0.03981915004897019, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 3.0121402740478516, "loss_scale": 4.0, "retries": 0, "seconds": 46.3, "max_memory_gb": 17.13, "ce": 0.33055196083296323, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 3.778456449508667, "loss_scale": 4.0, "retries": 0, "seconds": 54.2, "max_memory_gb": 17.13, "ce": 0.471326766493803, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 1.7083464860916138, "loss_scale": 4.0, "retries": 0, "seconds": 61.0, "max_memory_gb": 17.13, "ce": 0.38763631135225296, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 2.698458194732666, "loss_scale": 4.0, "retries": 0, "seconds": 68.7, "max_memory_gb": 17.13, "ce": 0.3207473971415311, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.36560770869255066, "loss_scale": 4.0, "retries": 0, "seconds": 77.4, "max_memory_gb": 17.13, "ce": 0.13272225500259083, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.4582597017288208, "loss_scale": 4.0, "retries": 0, "seconds": 84.3, "max_memory_gb": 17.13, "ce": 0.2137529277933936, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 0.29146531224250793, "loss_scale": 4.0, "retries": 0, "seconds": 90.7, "max_memory_gb": 17.13, "ce": 0.11582513250323245, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 0.46937090158462524, "loss_scale": 4.0, "retries": 0, "seconds": 97.2, "max_memory_gb": 17.13, "ce": 0.23944027335892315, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 2.05010986328125, "loss_scale": 4.0, "retries": 0, "seconds": 103.9, "max_memory_gb": 17.13, "ce": 0.3398498755414039, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 1.1708117723464966, "loss_scale": 4.0, "retries": 0, "seconds": 111.8, "max_memory_gb": 17.13, "ce": 0.2308383713643707, "rank": 0.0356190949678421}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.3247017562389374, "loss_scale": 4.0, "retries": 0, "seconds": 118.3, "max_memory_gb": 17.13, "ce": 0.09229540186061058, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 1.1593444347381592, "loss_scale": 4.0, "retries": 0, "seconds": 125.0, "max_memory_gb": 17.13, "ce": 0.22031208920816425, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 0.8287562131881714, "loss_scale": 4.0, "retries": 0, "seconds": 131.7, "max_memory_gb": 17.14, "ce": 0.1971027128456626, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 0.966474175453186, "loss_scale": 4.0, "retries": 0, "seconds": 138.5, "max_memory_gb": 17.14, "ce": 0.17650844737363514, "rank": 0.09930314123630524}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.2562708854675293, "loss_scale": 4.0, "retries": 0, "seconds": 144.9, "max_memory_gb": 17.14, "ce": 0.0249739358023362, "rank": 0.1241072565317154}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.8507742285728455, "loss_scale": 4.0, "retries": 0, "seconds": 150.7, "max_memory_gb": 17.14, "ce": 0.1014767421875149, "rank": 0.021632522344589233}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.9605963230133057, "loss_scale": 4.0, "retries": 0, "seconds": 157.5, "max_memory_gb": 17.14, "ce": 0.14598382560507162, "rank": 0.09291243553161621}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 0.8952455520629883, "loss_scale": 4.0, "retries": 0, "seconds": 164.3, "max_memory_gb": 17.14, "ce": 0.11940209922249778, "rank": 0.0}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 0.4498852491378784, "loss_scale": 4.0, "retries": 0, "seconds": 171.2, "max_memory_gb": 17.14, "ce": 0.06873440637718886, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 2.44499135017395, "loss_scale": 4.0, "retries": 0, "seconds": 177.3, "max_memory_gb": 17.14, "ce": 0.20595756609691307, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 3.634030342102051, "loss_scale": 4.0, "retries": 0, "seconds": 183.9, "max_memory_gb": 17.14, "ce": 0.23013461322989315, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 4.195781230926514, "loss_scale": 4.0, "retries": 0, "seconds": 190.8, "max_memory_gb": 17.14, "ce": 0.22832797567389207, "rank": 0.07279010117053986}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 1.591339349746704, "loss_scale": 4.0, "retries": 0, "seconds": 197.5, "max_memory_gb": 17.14, "ce": 0.19856128099854686, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 3.58278489112854, "loss_scale": 4.0, "retries": 0, "seconds": 204.5, "max_memory_gb": 17.14, "ce": 0.6328671218143427, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 0.2727772295475006, "loss_scale": 4.0, "retries": 0, "seconds": 211.6, "max_memory_gb": 17.14, "ce": 0.01416842665025797, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 5.126420974731445, "loss_scale": 4.0, "retries": 0, "seconds": 218.3, "max_memory_gb": 17.14, "ce": 0.7550525569313322, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 0.3808686137199402, "loss_scale": 4.0, "retries": 0, "seconds": 225.2, "max_memory_gb": 17.14, "ce": 0.02771706932981033, "rank": 0.0}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 2.9616668224334717, "loss_scale": 4.0, "retries": 0, "seconds": 232.0, "max_memory_gb": 17.14, "ce": 0.2656576212575601, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.6132935881614685, "loss_scale": 4.0, "retries": 0, "seconds": 238.8, "max_memory_gb": 17.14, "ce": 0.08313469367931248, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.8110634684562683, "loss_scale": 4.0, "retries": 0, "seconds": 245.6, "max_memory_gb": 17.14, "ce": 0.02177028713413165, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 2.2276477813720703, "loss_scale": 4.0, "retries": 0, "seconds": 252.4, "max_memory_gb": 17.14, "ce": 0.22941515435559268, "rank": 0.0}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.7343551516532898, "loss_scale": 4.0, "retries": 0, "seconds": 259.2, "max_memory_gb": 17.14, "ce": 0.19083878158744483, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.9181537628173828, "loss_scale": 4.0, "retries": 0, "seconds": 266.0, "max_memory_gb": 17.14, "ce": 0.14270393656624947, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 1.399296522140503, "loss_scale": 4.0, "retries": 0, "seconds": 272.7, "max_memory_gb": 17.14, "ce": 0.14522496786048578, "rank": 0.1251312792301178}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.12491526454687119, "loss_scale": 4.0, "retries": 0, "seconds": 279.6, "max_memory_gb": 17.14, "ce": 0.010564136091488763, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 2.7249817848205566, "loss_scale": 4.0, "retries": 0, "seconds": 285.9, "max_memory_gb": 17.14, "ce": 0.4069232242381986, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.0754738450050354, "loss_scale": 4.0, "retries": 0, "seconds": 292.8, "max_memory_gb": 17.14, "ce": 0.006583641601537238, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.12513042986392975, "loss_scale": 4.0, "retries": 0, "seconds": 299.7, "max_memory_gb": 17.14, "ce": 0.009028487651448813, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 6.089638710021973, "loss_scale": 4.0, "retries": 0, "seconds": 306.6, "max_memory_gb": 17.14, "ce": 0.27397518010866406, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 3.0162525177001953, "loss_scale": 4.0, "retries": 0, "seconds": 313.5, "max_memory_gb": 17.14, "ce": 0.3827268466957321, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.7768010497093201, "loss_scale": 4.0, "retries": 0, "seconds": 320.5, "max_memory_gb": 17.14, "ce": 0.0893762925734336, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.37273237109184265, "loss_scale": 4.0, "retries": 0, "seconds": 327.4, "max_memory_gb": 17.14, "ce": 0.02726660364533018, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.045193158090114594, "loss_scale": 4.0, "retries": 0, "seconds": 334.3, "max_memory_gb": 17.14, "ce": 0.0026887765006904374, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.10536487400531769, "loss_scale": 4.0, "retries": 0, "seconds": 341.0, "max_memory_gb": 17.14, "ce": 0.010579120700640487, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 1.145180583000183, "loss_scale": 4.0, "retries": 0, "seconds": 347.8, "max_memory_gb": 17.14, "ce": 0.09840793042349105, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 3.2653167247772217, "loss_scale": 4.0, "retries": 0, "seconds": 354.7, "max_memory_gb": 17.14, "ce": 0.12346262922437745, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.3584343492984772, "loss_scale": 4.0, "retries": 0, "seconds": 364.0, "max_memory_gb": 17.14, "ce": 0.03236424609349342, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 1.4198286533355713, "loss_scale": 4.0, "retries": 0, "seconds": 371.8, "max_memory_gb": 17.14, "ce": 0.08927485830190562, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 1.144562840461731, "loss_scale": 4.0, "retries": 0, "seconds": 378.6, "max_memory_gb": 17.14, "ce": 0.0708089342333551, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.2128383219242096, "loss_scale": 4.0, "retries": 0, "seconds": 385.4, "max_memory_gb": 17.14, "ce": 0.016327369149621518, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.3666914999485016, "loss_scale": 4.0, "retries": 0, "seconds": 393.3, "max_memory_gb": 17.14, "ce": 0.029340732586206286, "rank": 0.0}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.13826020061969757, "loss_scale": 4.0, "retries": 0, "seconds": 401.7, "max_memory_gb": 17.14, "ce": 0.010516630365600577, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.046275168657302856, "loss_scale": 4.0, "retries": 0, "seconds": 409.9, "max_memory_gb": 17.14, "ce": 0.0022858624831769703, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 1.8404186964035034, "loss_scale": 4.0, "retries": 0, "seconds": 416.6, "max_memory_gb": 17.14, "ce": 0.18094945408938656, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.8642301559448242, "loss_scale": 4.0, "retries": 0, "seconds": 423.3, "max_memory_gb": 17.14, "ce": 0.07560737729636458, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 1.5061815977096558, "loss_scale": 4.0, "retries": 0, "seconds": 429.9, "max_memory_gb": 17.14, "ce": 0.2351179960915033, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.2786043584346771, "loss_scale": 4.0, "retries": 0, "seconds": 436.7, "max_memory_gb": 17.14, "ce": 0.02801107016057358, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.446357399225235, "loss_scale": 4.0, "retries": 0, "seconds": 443.5, "max_memory_gb": 17.14, "ce": 0.03310869666256622, "rank": 0.0}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.16465970873832703, "loss_scale": 4.0, "retries": 0, "seconds": 450.4, "max_memory_gb": 17.14, "ce": 0.010375284984547761, "rank": 0.0}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 0.9646717309951782, "loss_scale": 4.0, "retries": 0, "seconds": 457.1, "max_memory_gb": 17.14, "ce": 0.14322117145343327, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 1.825376033782959, "loss_scale": 4.0, "retries": 0, "seconds": 463.8, "max_memory_gb": 17.14, "ce": 0.22240298272117798, "rank": 0.11792880296707153}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.03251269459724426, "loss_scale": 4.0, "retries": 0, "seconds": 470.8, "max_memory_gb": 17.14, "ce": 0.0026010929452695564, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.1597859263420105, "loss_scale": 4.0, "retries": 0, "seconds": 477.6, "max_memory_gb": 17.14, "ce": 0.007979706668493236, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 3.712266683578491, "loss_scale": 4.0, "retries": 0, "seconds": 484.5, "max_memory_gb": 17.14, "ce": 0.69574435333152, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.36574670672416687, "loss_scale": 4.0, "retries": 0, "seconds": 491.3, "max_memory_gb": 17.14, "ce": 0.0060790032716795395, "rank": 0.0}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.05566347390413284, "loss_scale": 4.0, "retries": 0, "seconds": 497.3, "max_memory_gb": 17.14, "ce": 0.003356255165044786, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.8049778342247009, "loss_scale": 4.0, "retries": 0, "seconds": 503.2, "max_memory_gb": 17.14, "ce": 0.08379264425411748, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 1.4829051494598389, "loss_scale": 4.0, "retries": 0, "seconds": 509.9, "max_memory_gb": 17.14, "ce": 0.12840258110691138, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 1.2429851293563843, "loss_scale": 4.0, "retries": 0, "seconds": 517.1, "max_memory_gb": 17.14, "ce": 0.14733686914718191, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 4.485665321350098, "loss_scale": 4.0, "retries": 0, "seconds": 524.0, "max_memory_gb": 17.14, "ce": 0.7491490049478671, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 1.389663577079773, "loss_scale": 4.0, "retries": 0, "seconds": 530.9, "max_memory_gb": 17.14, "ce": 0.3340991425877746, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.8785117864608765, "loss_scale": 4.0, "retries": 0, "seconds": 537.7, "max_memory_gb": 17.14, "ce": 0.03950157989515901, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.2176039069890976, "loss_scale": 4.0, "retries": 0, "seconds": 545.0, "max_memory_gb": 17.14, "ce": 0.02254590142001689, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.18168248236179352, "loss_scale": 4.0, "retries": 0, "seconds": 551.8, "max_memory_gb": 17.14, "ce": 0.007235241690523253, "rank": 0.0}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.23251041769981384, "loss_scale": 4.0, "retries": 0, "seconds": 558.6, "max_memory_gb": 17.14, "ce": 0.01554220369871473, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.4621719717979431, "loss_scale": 4.0, "retries": 0, "seconds": 565.6, "max_memory_gb": 17.14, "ce": 0.047645656589338614, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 1.3757529258728027, "loss_scale": 4.0, "retries": 0, "seconds": 572.6, "max_memory_gb": 17.14, "ce": 0.12736054266997598, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 1.173776626586914, "loss_scale": 4.0, "retries": 0, "seconds": 579.6, "max_memory_gb": 17.14, "ce": 0.1880453054905047, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.5603624582290649, "loss_scale": 4.0, "retries": 0, "seconds": 586.7, "max_memory_gb": 17.14, "ce": 0.033408132534077595, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.04628722742199898, "loss_scale": 4.0, "retries": 0, "seconds": 594.9, "max_memory_gb": 17.14, "ce": 0.001405254400197009, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.5787186026573181, "loss_scale": 4.0, "retries": 0, "seconds": 601.6, "max_memory_gb": 17.14, "ce": 0.04983163460110518, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.9823869466781616, "loss_scale": 4.0, "retries": 0, "seconds": 608.0, "max_memory_gb": 17.14, "ce": 0.1499347729627516, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.44909295439720154, "loss_scale": 4.0, "retries": 0, "seconds": 614.4, "max_memory_gb": 17.14, "ce": 0.046516264741512714, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.1876159906387329, "loss_scale": 4.0, "retries": 0, "seconds": 620.6, "max_memory_gb": 17.14, "ce": 0.019502355230997637, "rank": 0.12499324977397919}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.5924435257911682, "loss_scale": 4.0, "retries": 0, "seconds": 627.1, "max_memory_gb": 17.14, "ce": 0.06553721719137684, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 3.5198636054992676, "loss_scale": 4.0, "retries": 0, "seconds": 634.0, "max_memory_gb": 17.14, "ce": 0.4968470806757068, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 2.84112286567688, "loss_scale": 4.0, "retries": 0, "seconds": 640.7, "max_memory_gb": 17.14, "ce": 0.1774820447753882, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 1.417137622833252, "loss_scale": 4.0, "retries": 0, "seconds": 647.5, "max_memory_gb": 17.14, "ce": 0.08911150695848846, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 1.1359502077102661, "loss_scale": 4.0, "retries": 0, "seconds": 654.3, "max_memory_gb": 17.14, "ce": 0.04630063385866379, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 3.6986284255981445, "loss_scale": 4.0, "retries": 0, "seconds": 661.1, "max_memory_gb": 17.14, "ce": 0.49307101642307316, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 3.137284755706787, "loss_scale": 4.0, "retries": 0, "seconds": 667.9, "max_memory_gb": 17.14, "ce": 0.3836221985420707, "rank": 0.3178386092185974}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.3193855583667755, "loss_scale": 4.0, "retries": 0, "seconds": 674.7, "max_memory_gb": 17.14, "ce": 0.02141864057807652, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.8402315974235535, "loss_scale": 4.0, "retries": 0, "seconds": 681.4, "max_memory_gb": 17.14, "ce": 0.049789212534960825, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.08373928815126419, "loss_scale": 4.0, "retries": 0, "seconds": 688.2, "max_memory_gb": 17.14, "ce": 0.00950998088956112, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.5509420037269592, "loss_scale": 4.0, "retries": 0, "seconds": 695.0, "max_memory_gb": 17.14, "ce": 0.0849513304683569, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.1772456169128418, "loss_scale": 4.0, "retries": 0, "seconds": 701.8, "max_memory_gb": 17.14, "ce": 0.01164435634336769, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.16960132122039795, "loss_scale": 4.0, "retries": 0, "seconds": 708.6, "max_memory_gb": 17.14, "ce": 0.012393402250154395, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.07598864287137985, "loss_scale": 4.0, "retries": 0, "seconds": 715.4, "max_memory_gb": 17.14, "ce": 0.005619256046429655, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.2212269902229309, "loss_scale": 4.0, "retries": 0, "seconds": 722.3, "max_memory_gb": 17.14, "ce": 0.013793837137200171, "rank": 0.12493916600942612}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.7268126010894775, "loss_scale": 4.0, "retries": 0, "seconds": 729.2, "max_memory_gb": 17.14, "ce": 0.06441286583572037, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.06226871535181999, "loss_scale": 4.0, "retries": 0, "seconds": 735.9, "max_memory_gb": 17.14, "ce": 0.0018616363663568336, "rank": 0.1110164150595665}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 1.3154194355010986, "loss_scale": 4.0, "retries": 0, "seconds": 742.5, "max_memory_gb": 17.14, "ce": 0.11164300925020143, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 2.3937108516693115, "loss_scale": 4.0, "retries": 0, "seconds": 749.4, "max_memory_gb": 17.14, "ce": 0.2819630079338822, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 1.6174430847167969, "loss_scale": 4.0, "retries": 0, "seconds": 756.4, "max_memory_gb": 17.14, "ce": 0.15041023060803127, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.5203225612640381, "loss_scale": 4.0, "retries": 0, "seconds": 763.1, "max_memory_gb": 17.14, "ce": 0.02854466362941821, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.290547251701355, "loss_scale": 4.0, "retries": 0, "seconds": 769.7, "max_memory_gb": 17.14, "ce": 0.019737456663506237, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.05291268602013588, "loss_scale": 4.0, "retries": 0, "seconds": 776.5, "max_memory_gb": 17.14, "ce": 0.0032574272577221564, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.17322584986686707, "loss_scale": 4.0, "retries": 0, "seconds": 783.4, "max_memory_gb": 17.14, "ce": 0.015655120815154078, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.8282269239425659, "loss_scale": 4.0, "retries": 0, "seconds": 790.2, "max_memory_gb": 17.14, "ce": 0.1131381451971265, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 1.1517448425292969, "loss_scale": 4.0, "retries": 0, "seconds": 797.2, "max_memory_gb": 17.14, "ce": 0.030849096100155293, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.9043034911155701, "loss_scale": 4.0, "retries": 0, "seconds": 803.7, "max_memory_gb": 17.14, "ce": 0.17842944758831436, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.07117130607366562, "loss_scale": 4.0, "retries": 0, "seconds": 810.5, "max_memory_gb": 17.14, "ce": 0.003122930354265918, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 1.4687933921813965, "loss_scale": 4.0, "retries": 0, "seconds": 817.3, "max_memory_gb": 17.14, "ce": 0.22120414085406992, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.058476004749536514, "loss_scale": 4.0, "retries": 0, "seconds": 823.9, "max_memory_gb": 17.14, "ce": 0.0031476317515171104, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 3.4342849254608154, "loss_scale": 4.0, "retries": 0, "seconds": 830.6, "max_memory_gb": 17.14, "ce": 0.16619124883186487, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.5815711617469788, "loss_scale": 4.0, "retries": 0, "seconds": 837.4, "max_memory_gb": 17.14, "ce": 0.05387097834704946, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 1.108630657196045, "loss_scale": 4.0, "retries": 0, "seconds": 844.3, "max_memory_gb": 17.14, "ce": 0.11231769495498156, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.3586333990097046, "loss_scale": 4.0, "retries": 0, "seconds": 851.4, "max_memory_gb": 17.14, "ce": 0.031390062458967805, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.663276195526123, "loss_scale": 4.0, "retries": 0, "seconds": 858.3, "max_memory_gb": 17.14, "ce": 0.5332318641283109, "rank": 0.0}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.7916858792304993, "loss_scale": 4.0, "retries": 0, "seconds": 865.2, "max_memory_gb": 17.14, "ce": 0.13991356932297094, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 2.634786605834961, "loss_scale": 4.0, "retries": 0, "seconds": 872.2, "max_memory_gb": 17.14, "ce": 0.48009803203672163, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.4404490292072296, "loss_scale": 4.0, "retries": 0, "seconds": 879.3, "max_memory_gb": 17.14, "ce": 0.014795341648778049, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 1.5120353698730469, "loss_scale": 4.0, "retries": 0, "seconds": 886.3, "max_memory_gb": 17.14, "ce": 0.1579814441020062, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 1.3294143676757812, "loss_scale": 4.0, "retries": 0, "seconds": 893.0, "max_memory_gb": 17.14, "ce": 0.09791502773884986, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 0.7679275870323181, "loss_scale": 4.0, "retries": 0, "seconds": 899.9, "max_memory_gb": 17.14, "ce": 0.1425912306031023, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 3.073422431945801, "loss_scale": 4.0, "retries": 0, "seconds": 907.0, "max_memory_gb": 17.14, "ce": 0.20464139069872544, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 1.9338104724884033, "loss_scale": 4.0, "retries": 0, "seconds": 914.1, "max_memory_gb": 17.14, "ce": 0.23453717718598455, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 1.250792145729065, "loss_scale": 4.0, "retries": 0, "seconds": 920.9, "max_memory_gb": 17.14, "ce": 0.3026461194561989, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 1.1700389385223389, "loss_scale": 4.0, "retries": 0, "seconds": 928.4, "max_memory_gb": 17.14, "ce": 0.20475966559979497, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 0.3031292259693146, "loss_scale": 4.0, "retries": 0, "seconds": 936.0, "max_memory_gb": 17.14, "ce": 0.018874939435136184, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 1.30765962600708, "loss_scale": 4.0, "retries": 0, "seconds": 943.0, "max_memory_gb": 17.14, "ce": 0.060418235501401796, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.19816091656684875, "loss_scale": 4.0, "retries": 0, "seconds": 950.2, "max_memory_gb": 17.14, "ce": 0.010978703326372852, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 0.8954368829727173, "loss_scale": 4.0, "retries": 0, "seconds": 957.2, "max_memory_gb": 17.14, "ce": 0.07193676500514812, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.7334327101707458, "loss_scale": 4.0, "retries": 0, "seconds": 964.3, "max_memory_gb": 17.14, "ce": 0.03797438777633033, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.03598671406507492, "loss_scale": 4.0, "retries": 0, "seconds": 971.4, "max_memory_gb": 17.14, "ce": 0.0027859568856456463, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.04751298949122429, "loss_scale": 4.0, "retries": 0, "seconds": 978.5, "max_memory_gb": 17.14, "ce": 0.00427543995169799, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 1.567734956741333, "loss_scale": 4.0, "retries": 0, "seconds": 985.6, "max_memory_gb": 17.14, "ce": 0.0856563587533401, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 2.494232654571533, "loss_scale": 4.0, "retries": 0, "seconds": 991.7, "max_memory_gb": 17.14, "ce": 0.2417696760835497, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.778961718082428, "loss_scale": 4.0, "retries": 0, "seconds": 998.4, "max_memory_gb": 17.14, "ce": 0.04333211550806482, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.03425243869423866, "loss_scale": 4.0, "retries": 0, "seconds": 1005.3, "max_memory_gb": 17.14, "ce": 0.0022759190380838845, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.11072809994220734, "loss_scale": 4.0, "retries": 0, "seconds": 1012.3, "max_memory_gb": 17.14, "ce": 0.01177781139324452, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 1.5783413648605347, "loss_scale": 4.0, "retries": 0, "seconds": 1018.7, "max_memory_gb": 17.14, "ce": 0.4701987240161998, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.3392716348171234, "loss_scale": 4.0, "retries": 0, "seconds": 1025.7, "max_memory_gb": 17.14, "ce": 0.008548360544182287, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.1279367208480835, "loss_scale": 4.0, "retries": 0, "seconds": 1032.6, "max_memory_gb": 17.14, "ce": 0.008361042933074714, "rank": 0.11834049969911575}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.5564550161361694, "loss_scale": 4.0, "retries": 0, "seconds": 1039.7, "max_memory_gb": 17.14, "ce": 0.08301258523260913, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 1.8611319065093994, "loss_scale": 4.0, "retries": 0, "seconds": 1046.7, "max_memory_gb": 17.14, "ce": 0.0429118818774441, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 0.27688494324684143, "loss_scale": 4.0, "retries": 0, "seconds": 1053.6, "max_memory_gb": 17.14, "ce": 0.02055707812132823, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.603584885597229, "loss_scale": 4.0, "retries": 0, "seconds": 1060.6, "max_memory_gb": 17.14, "ce": 0.10413880152646016, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.3202536106109619, "loss_scale": 4.0, "retries": 0, "seconds": 1067.9, "max_memory_gb": 17.14, "ce": 0.02561813060788154, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 1.7937997579574585, "loss_scale": 4.0, "retries": 0, "seconds": 1074.9, "max_memory_gb": 17.14, "ce": 0.26801050733388365, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.10217585414648056, "loss_scale": 4.0, "retries": 0, "seconds": 1081.8, "max_memory_gb": 17.14, "ce": 0.004032834515555805, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.17012758553028107, "loss_scale": 4.0, "retries": 0, "seconds": 1088.2, "max_memory_gb": 17.14, "ce": 0.014882644293265912, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 4.977742671966553, "loss_scale": 4.0, "retries": 0, "seconds": 1095.1, "max_memory_gb": 17.14, "ce": 0.6729495923614195, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.1953451931476593, "loss_scale": 4.0, "retries": 0, "seconds": 1101.7, "max_memory_gb": 17.14, "ce": 0.013199081746279262, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 1.0755492448806763, "loss_scale": 4.0, "retries": 0, "seconds": 1109.1, "max_memory_gb": 17.14, "ce": 0.1672481274022175, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 1.4565426111221313, "loss_scale": 4.0, "retries": 0, "seconds": 1116.1, "max_memory_gb": 17.14, "ce": 0.14234552786933818, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.7031934857368469, "loss_scale": 4.0, "retries": 0, "seconds": 1123.0, "max_memory_gb": 17.14, "ce": 0.057144069607147685, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 3.5934667587280273, "loss_scale": 4.0, "retries": 0, "seconds": 1130.2, "max_memory_gb": 17.14, "ce": 0.3101637553600085, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.8006026744842529, "loss_scale": 4.0, "retries": 0, "seconds": 1137.2, "max_memory_gb": 17.14, "ce": 0.06831622198001241, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.10417565703392029, "loss_scale": 4.0, "retries": 0, "seconds": 1143.7, "max_memory_gb": 17.14, "ce": 0.009602078286661708, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 2.115365505218506, "loss_scale": 4.0, "retries": 0, "seconds": 1152.1, "max_memory_gb": 17.14, "ce": 0.2767337850000331, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.15076696872711182, "loss_scale": 4.0, "retries": 0, "seconds": 1159.4, "max_memory_gb": 17.14, "ce": 0.008945373947199187, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.06845108419656754, "loss_scale": 4.0, "retries": 0, "seconds": 1166.2, "max_memory_gb": 17.14, "ce": 0.004891135774869326, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 1.1048998832702637, "loss_scale": 4.0, "retries": 0, "seconds": 1173.3, "max_memory_gb": 17.14, "ce": 0.30809782030155475, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.02439854107797146, "loss_scale": 4.0, "retries": 0, "seconds": 1180.2, "max_memory_gb": 17.14, "ce": 0.0008035803552957077, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.4365694522857666, "loss_scale": 4.0, "retries": 0, "seconds": 1187.3, "max_memory_gb": 17.14, "ce": 0.0578781238054944, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.07481403648853302, "loss_scale": 4.0, "retries": 0, "seconds": 1195.4, "max_memory_gb": 17.14, "ce": 0.006515672039313358, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.14468581974506378, "loss_scale": 4.0, "retries": 0, "seconds": 1202.4, "max_memory_gb": 17.14, "ce": 0.009781547247712297, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.5337491631507874, "loss_scale": 4.0, "retries": 0, "seconds": 1209.2, "max_memory_gb": 17.14, "ce": 0.04074297420265793, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 1.1545064449310303, "loss_scale": 4.0, "retries": 0, "seconds": 1216.0, "max_memory_gb": 17.14, "ce": 0.11479999200219027, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.01878025196492672, "loss_scale": 4.0, "retries": 0, "seconds": 1223.0, "max_memory_gb": 17.14, "ce": 0.0012387098199724278, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 0.8573433756828308, "loss_scale": 4.0, "retries": 0, "seconds": 1229.8, "max_memory_gb": 17.14, "ce": 0.2543159844922229, "rank": 0.12497342377901077}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 1.9712728261947632, "loss_scale": 4.0, "retries": 0, "seconds": 1236.8, "max_memory_gb": 17.14, "ce": 0.11950566288851405, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 1.111816644668579, "loss_scale": 4.0, "retries": 0, "seconds": 1243.4, "max_memory_gb": 17.14, "ce": 0.37381641977117397, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.1886102706193924, "loss_scale": 4.0, "retries": 0, "seconds": 1249.7, "max_memory_gb": 17.14, "ce": 0.01927218523042029, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.5832303166389465, "loss_scale": 4.0, "retries": 0, "seconds": 1256.6, "max_memory_gb": 17.14, "ce": 0.010590258499178162, "rank": 0.04943598061800003}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 0.931955873966217, "loss_scale": 4.0, "retries": 0, "seconds": 1263.5, "max_memory_gb": 17.14, "ce": 0.12373776516506041, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.06905553489923477, "loss_scale": 4.0, "retries": 0, "seconds": 1270.5, "max_memory_gb": 17.14, "ce": 0.00299558873530259, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.35323673486709595, "loss_scale": 4.0, "retries": 0, "seconds": 1277.9, "max_memory_gb": 17.14, "ce": 0.017293327687866622, "rank": 0.15766426175832748}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.4041571319103241, "loss_scale": 4.0, "retries": 0, "seconds": 1285.6, "max_memory_gb": 17.14, "ce": 0.039402126636787216, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 2.77119517326355, "loss_scale": 4.0, "retries": 0, "seconds": 1292.3, "max_memory_gb": 17.14, "ce": 0.11989684065747497, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.8377702236175537, "loss_scale": 4.0, "retries": 0, "seconds": 1298.4, "max_memory_gb": 17.14, "ce": 0.13035425161729108, "rank": 0.13058294355869293}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.4468610286712646, "loss_scale": 4.0, "retries": 0, "seconds": 1304.8, "max_memory_gb": 17.14, "ce": 0.17602917242538751, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 2.329923629760742, "loss_scale": 4.0, "retries": 0, "seconds": 1311.7, "max_memory_gb": 17.14, "ce": 0.2904727556551734, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.14988189935684204, "loss_scale": 4.0, "retries": 0, "seconds": 1319.7, "max_memory_gb": 17.14, "ce": 0.0048953358880226006, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 2.6235857009887695, "loss_scale": 4.0, "retries": 0, "seconds": 1327.6, "max_memory_gb": 17.14, "ce": 0.17340246756907618, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.1003515720367432, "loss_scale": 4.0, "retries": 0, "seconds": 1335.9, "max_memory_gb": 17.14, "ce": 0.33974043734815496, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.013987804763019085, "loss_scale": 4.0, "retries": 0, "seconds": 1345.2, "max_memory_gb": 17.14, "ce": 0.0010593506040095235, "rank": 0.12502628564834595}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.3502209186553955, "loss_scale": 4.0, "retries": 0, "seconds": 1354.6, "max_memory_gb": 17.14, "ce": 0.040448626645229524, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.35626620054244995, "loss_scale": 4.0, "retries": 0, "seconds": 1364.2, "max_memory_gb": 17.14, "ce": 0.06881316072599475, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.13939619064331055, "loss_scale": 4.0, "retries": 0, "seconds": 1372.8, "max_memory_gb": 17.14, "ce": 0.008031285358583773, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 1.407476782798767, "loss_scale": 4.0, "retries": 0, "seconds": 1381.8, "max_memory_gb": 17.14, "ce": 0.1485972982234216, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.42337197065353394, "loss_scale": 4.0, "retries": 0, "seconds": 1390.8, "max_memory_gb": 17.14, "ce": 0.05837500641905535, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 1.1646409034729004, "loss_scale": 4.0, "retries": 0, "seconds": 1398.7, "max_memory_gb": 17.14, "ce": 0.0942599154684558, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 1.0165479183197021, "loss_scale": 4.0, "retries": 0, "seconds": 1406.1, "max_memory_gb": 17.14, "ce": 0.21027624175189885, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.6369781494140625, "loss_scale": 4.0, "retries": 0, "seconds": 1415.5, "max_memory_gb": 17.14, "ce": 0.06131456198750129, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.03324758633971214, "loss_scale": 4.0, "retries": 0, "seconds": 1424.6, "max_memory_gb": 17.14, "ce": 0.002550078861304428, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.283395916223526, "loss_scale": 4.0, "retries": 0, "seconds": 1433.6, "max_memory_gb": 17.14, "ce": 0.016040481981235644, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.3053356409072876, "loss_scale": 4.0, "retries": 0, "seconds": 1440.6, "max_memory_gb": 17.14, "ce": 0.02223592643599659, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 0.807225227355957, "loss_scale": 4.0, "retries": 0, "seconds": 1447.5, "max_memory_gb": 17.14, "ce": 0.09082538714210386, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.24262776970863342, "loss_scale": 4.0, "retries": 0, "seconds": 1454.4, "max_memory_gb": 17.14, "ce": 0.018538012404633264, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.2326953262090683, "loss_scale": 4.0, "retries": 0, "seconds": 1461.2, "max_memory_gb": 17.14, "ce": 0.05518517908649301, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.043461620807647705, "loss_scale": 4.0, "retries": 0, "seconds": 1467.4, "max_memory_gb": 17.14, "ce": 0.0023213434358240193, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.6519845724105835, "loss_scale": 4.0, "retries": 0, "seconds": 1474.4, "max_memory_gb": 17.14, "ce": 0.1044022688286077, "rank": 0.12510399520397186}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 1.3677740097045898, "loss_scale": 4.0, "retries": 0, "seconds": 1481.9, "max_memory_gb": 17.14, "ce": 0.36681831398823306, "rank": 0.0}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 2.279374361038208, "loss_scale": 4.0, "retries": 0, "seconds": 1489.6, "max_memory_gb": 17.14, "ce": 0.11614983303729787, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 0.9552568197250366, "loss_scale": 4.0, "retries": 0, "seconds": 1496.6, "max_memory_gb": 17.14, "ce": 0.1820225484916591, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.5634607672691345, "loss_scale": 4.0, "retries": 0, "seconds": 1503.5, "max_memory_gb": 17.14, "ce": 0.14549040724978113, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 1.1612675189971924, "loss_scale": 4.0, "retries": 0, "seconds": 1510.4, "max_memory_gb": 17.14, "ce": 0.21887059113896612, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 2.225637674331665, "loss_scale": 4.0, "retries": 0, "seconds": 1517.3, "max_memory_gb": 17.14, "ce": 0.4056066800399094, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.32960349321365356, "loss_scale": 4.0, "retries": 0, "seconds": 1524.4, "max_memory_gb": 17.14, "ce": 0.026990287067519603, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.06267890334129333, "loss_scale": 4.0, "retries": 0, "seconds": 1531.4, "max_memory_gb": 17.14, "ce": 0.0038588064560372004, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 0.8899222612380981, "loss_scale": 4.0, "retries": 0, "seconds": 1537.5, "max_memory_gb": 17.14, "ce": 0.1707739490827862, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.32477623224258423, "loss_scale": 4.0, "retries": 0, "seconds": 1544.5, "max_memory_gb": 17.14, "ce": 0.03709057204105193, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.32608070969581604, "loss_scale": 4.0, "retries": 0, "seconds": 1551.5, "max_memory_gb": 17.14, "ce": 0.022736953333605925, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.42651668190956116, "loss_scale": 4.0, "retries": 0, "seconds": 1558.6, "max_memory_gb": 17.14, "ce": 0.027462062665108533, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.3488953113555908, "loss_scale": 4.0, "retries": 0, "seconds": 1565.5, "max_memory_gb": 17.14, "ce": 0.02559644683356055, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 1.577423095703125, "loss_scale": 4.0, "retries": 0, "seconds": 1572.3, "max_memory_gb": 17.14, "ce": 0.06787815560528543, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.4868641793727875, "loss_scale": 4.0, "retries": 0, "seconds": 1579.3, "max_memory_gb": 17.14, "ce": 0.02979529212802845, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.13283143937587738, "loss_scale": 4.0, "retries": 0, "seconds": 1586.5, "max_memory_gb": 17.14, "ce": 0.010276647120917914, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.42161306738853455, "loss_scale": 4.0, "retries": 0, "seconds": 1593.5, "max_memory_gb": 17.14, "ce": 0.02052526962347656, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.6519005298614502, "loss_scale": 4.0, "retries": 0, "seconds": 1601.2, "max_memory_gb": 17.14, "ce": 0.029041324531590362, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.09761982411146164, "loss_scale": 4.0, "retries": 0, "seconds": 1608.1, "max_memory_gb": 17.14, "ce": 0.008141435861034552, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.4447712004184723, "loss_scale": 4.0, "retries": 0, "seconds": 1615.0, "max_memory_gb": 17.14, "ce": 0.05177694473172778, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 2.453523635864258, "loss_scale": 4.0, "retries": 0, "seconds": 1621.8, "max_memory_gb": 17.14, "ce": 0.37569968418256394, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.3427349925041199, "loss_scale": 4.0, "retries": 0, "seconds": 1628.5, "max_memory_gb": 17.14, "ce": 0.04053521906030255, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.2879558801651001, "loss_scale": 4.0, "retries": 0, "seconds": 1635.2, "max_memory_gb": 17.14, "ce": 0.01866549206511081, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.3200804889202118, "loss_scale": 4.0, "retries": 0, "seconds": 1642.4, "max_memory_gb": 17.14, "ce": 0.02284522429226854, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.6948325634002686, "loss_scale": 4.0, "retries": 0, "seconds": 1649.5, "max_memory_gb": 17.14, "ce": 0.04483842444642505, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 1.9831271171569824, "loss_scale": 4.0, "retries": 0, "seconds": 1656.4, "max_memory_gb": 17.14, "ce": 0.22375257347607658, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.17115634679794312, "loss_scale": 4.0, "retries": 0, "seconds": 1663.2, "max_memory_gb": 17.14, "ce": 0.0165755502164302, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.6872214674949646, "loss_scale": 4.0, "retries": 0, "seconds": 1670.0, "max_memory_gb": 17.14, "ce": 0.053733328892349164, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.40967056155204773, "loss_scale": 4.0, "retries": 0, "seconds": 1676.8, "max_memory_gb": 17.14, "ce": 0.03335415237665984, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.3542289435863495, "loss_scale": 4.0, "retries": 0, "seconds": 1684.3, "max_memory_gb": 17.14, "ce": 0.08726233560878427, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 0.3260124623775482, "loss_scale": 4.0, "retries": 0, "seconds": 1691.1, "max_memory_gb": 17.14, "ce": 0.023221631078058635, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.527292013168335, "loss_scale": 4.0, "retries": 0, "seconds": 1698.3, "max_memory_gb": 17.14, "ce": 0.06789099220213757, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.15018512308597565, "loss_scale": 4.0, "retries": 0, "seconds": 1705.4, "max_memory_gb": 17.14, "ce": 0.010385115248027432, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.016925768926739693, "loss_scale": 4.0, "retries": 0, "seconds": 1712.5, "max_memory_gb": 17.14, "ce": 0.0012117911942368664, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.41859710216522217, "loss_scale": 4.0, "retries": 0, "seconds": 1719.7, "max_memory_gb": 17.14, "ce": 0.04006745654055521, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 1.5981661081314087, "loss_scale": 4.0, "retries": 0, "seconds": 1726.8, "max_memory_gb": 17.14, "ce": 0.2438361587419422, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.022397061809897423, "loss_scale": 4.0, "retries": 0, "seconds": 1733.6, "max_memory_gb": 17.14, "ce": 0.0012137550334045955, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.5816811919212341, "loss_scale": 4.0, "retries": 0, "seconds": 1740.5, "max_memory_gb": 17.14, "ce": 0.04847267815057421, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.10019528120756149, "loss_scale": 4.0, "retries": 0, "seconds": 1747.3, "max_memory_gb": 17.14, "ce": 0.007344565624180177, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.18660199642181396, "loss_scale": 4.0, "retries": 0, "seconds": 1754.2, "max_memory_gb": 17.14, "ce": 0.013494065026407043, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.2884538769721985, "loss_scale": 4.0, "retries": 0, "seconds": 1761.8, "max_memory_gb": 17.14, "ce": 0.017133379073698052, "rank": 0.0}

runs/fixed_internvl/removal/Full_s42/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/removal/Full_s42/adapter/adapter_config.json

@@ -0,0 +1,556 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.0.mlp.down_proj": 32,
+    "model.language_model.layers.0.mlp.gate_proj": 32,
+    "model.language_model.layers.0.mlp.up_proj": 32,
+    "model.language_model.layers.0.self_attn.k_proj": 32,
+    "model.language_model.layers.0.self_attn.o_proj": 32,
+    "model.language_model.layers.0.self_attn.q_proj": 32,
+    "model.language_model.layers.0.self_attn.v_proj": 32,
+    "model.language_model.layers.1.mlp.down_proj": 32,
+    "model.language_model.layers.1.mlp.gate_proj": 32,
+    "model.language_model.layers.1.mlp.up_proj": 32,
+    "model.language_model.layers.1.self_attn.k_proj": 32,
+    "model.language_model.layers.1.self_attn.o_proj": 32,
+    "model.language_model.layers.1.self_attn.q_proj": 32,
+    "model.language_model.layers.1.self_attn.v_proj": 32,
+    "model.language_model.layers.10.mlp.down_proj": 32,
+    "model.language_model.layers.10.mlp.gate_proj": 32,
+    "model.language_model.layers.10.mlp.up_proj": 32,
+    "model.language_model.layers.10.self_attn.k_proj": 32,
+    "model.language_model.layers.10.self_attn.o_proj": 32,
+    "model.language_model.layers.10.self_attn.q_proj": 32,
+    "model.language_model.layers.10.self_attn.v_proj": 32,
+    "model.language_model.layers.11.mlp.down_proj": 32,
+    "model.language_model.layers.11.mlp.gate_proj": 32,
+    "model.language_model.layers.11.mlp.up_proj": 32,
+    "model.language_model.layers.11.self_attn.k_proj": 32,
+    "model.language_model.layers.11.self_attn.o_proj": 32,
+    "model.language_model.layers.11.self_attn.q_proj": 32,
+    "model.language_model.layers.11.self_attn.v_proj": 32,
+    "model.language_model.layers.12.mlp.down_proj": 32,
+    "model.language_model.layers.12.mlp.gate_proj": 32,
+    "model.language_model.layers.12.mlp.up_proj": 32,
+    "model.language_model.layers.12.self_attn.k_proj": 32,
+    "model.language_model.layers.12.self_attn.o_proj": 32,
+    "model.language_model.layers.12.self_attn.q_proj": 32,
+    "model.language_model.layers.12.self_attn.v_proj": 32,
+    "model.language_model.layers.13.mlp.down_proj": 32,
+    "model.language_model.layers.13.mlp.gate_proj": 32,
+    "model.language_model.layers.13.mlp.up_proj": 32,
+    "model.language_model.layers.13.self_attn.k_proj": 32,
+    "model.language_model.layers.13.self_attn.o_proj": 32,
+    "model.language_model.layers.13.self_attn.q_proj": 32,
+    "model.language_model.layers.13.self_attn.v_proj": 32,
+    "model.language_model.layers.14.mlp.down_proj": 32,
+    "model.language_model.layers.14.mlp.gate_proj": 32,
+    "model.language_model.layers.14.mlp.up_proj": 32,
+    "model.language_model.layers.14.self_attn.k_proj": 32,
+    "model.language_model.layers.14.self_attn.o_proj": 32,
+    "model.language_model.layers.14.self_attn.q_proj": 32,
+    "model.language_model.layers.14.self_attn.v_proj": 32,
+    "model.language_model.layers.15.mlp.down_proj": 32,
+    "model.language_model.layers.15.mlp.gate_proj": 32,
+    "model.language_model.layers.15.mlp.up_proj": 32,
+    "model.language_model.layers.15.self_attn.k_proj": 32,
+    "model.language_model.layers.15.self_attn.o_proj": 32,
+    "model.language_model.layers.15.self_attn.q_proj": 32,
+    "model.language_model.layers.15.self_attn.v_proj": 32,
+    "model.language_model.layers.16.mlp.down_proj": 32,
+    "model.language_model.layers.16.mlp.gate_proj": 32,
+    "model.language_model.layers.16.mlp.up_proj": 32,
+    "model.language_model.layers.16.self_attn.k_proj": 32,
+    "model.language_model.layers.16.self_attn.o_proj": 32,
+    "model.language_model.layers.16.self_attn.q_proj": 32,
+    "model.language_model.layers.16.self_attn.v_proj": 32,
+    "model.language_model.layers.17.mlp.down_proj": 32,
+    "model.language_model.layers.17.mlp.gate_proj": 32,
+    "model.language_model.layers.17.mlp.up_proj": 32,
+    "model.language_model.layers.17.self_attn.k_proj": 32,
+    "model.language_model.layers.17.self_attn.o_proj": 32,
+    "model.language_model.layers.17.self_attn.q_proj": 32,
+    "model.language_model.layers.17.self_attn.v_proj": 32,
+    "model.language_model.layers.18.mlp.down_proj": 32,
+    "model.language_model.layers.18.mlp.gate_proj": 32,
+    "model.language_model.layers.18.mlp.up_proj": 32,
+    "model.language_model.layers.18.self_attn.k_proj": 32,
+    "model.language_model.layers.18.self_attn.o_proj": 32,
+    "model.language_model.layers.18.self_attn.q_proj": 32,
+    "model.language_model.layers.18.self_attn.v_proj": 32,
+    "model.language_model.layers.19.mlp.down_proj": 32,
+    "model.language_model.layers.19.mlp.gate_proj": 32,
+    "model.language_model.layers.19.mlp.up_proj": 32,
+    "model.language_model.layers.19.self_attn.k_proj": 32,
+    "model.language_model.layers.19.self_attn.o_proj": 32,
+    "model.language_model.layers.19.self_attn.q_proj": 32,
+    "model.language_model.layers.19.self_attn.v_proj": 32,
+    "model.language_model.layers.2.mlp.down_proj": 32,
+    "model.language_model.layers.2.mlp.gate_proj": 32,
+    "model.language_model.layers.2.mlp.up_proj": 32,
+    "model.language_model.layers.2.self_attn.k_proj": 32,
+    "model.language_model.layers.2.self_attn.o_proj": 32,
+    "model.language_model.layers.2.self_attn.q_proj": 32,
+    "model.language_model.layers.2.self_attn.v_proj": 32,
+    "model.language_model.layers.20.mlp.down_proj": 32,
+    "model.language_model.layers.20.mlp.gate_proj": 32,
+    "model.language_model.layers.20.mlp.up_proj": 32,
+    "model.language_model.layers.20.self_attn.k_proj": 32,
+    "model.language_model.layers.20.self_attn.o_proj": 32,
+    "model.language_model.layers.20.self_attn.q_proj": 32,
+    "model.language_model.layers.20.self_attn.v_proj": 32,
+    "model.language_model.layers.21.mlp.down_proj": 32,
+    "model.language_model.layers.21.mlp.gate_proj": 32,
+    "model.language_model.layers.21.mlp.up_proj": 32,
+    "model.language_model.layers.21.self_attn.k_proj": 32,
+    "model.language_model.layers.21.self_attn.o_proj": 32,
+    "model.language_model.layers.21.self_attn.q_proj": 32,
+    "model.language_model.layers.21.self_attn.v_proj": 32,
+    "model.language_model.layers.22.mlp.down_proj": 32,
+    "model.language_model.layers.22.mlp.gate_proj": 32,
+    "model.language_model.layers.22.mlp.up_proj": 32,
+    "model.language_model.layers.22.self_attn.k_proj": 32,
+    "model.language_model.layers.22.self_attn.o_proj": 32,
+    "model.language_model.layers.22.self_attn.q_proj": 32,
+    "model.language_model.layers.22.self_attn.v_proj": 32,
+    "model.language_model.layers.23.mlp.down_proj": 32,
+    "model.language_model.layers.23.mlp.gate_proj": 32,
+    "model.language_model.layers.23.mlp.up_proj": 32,
+    "model.language_model.layers.23.self_attn.k_proj": 32,
+    "model.language_model.layers.23.self_attn.o_proj": 32,
+    "model.language_model.layers.23.self_attn.q_proj": 32,
+    "model.language_model.layers.23.self_attn.v_proj": 32,
+    "model.language_model.layers.24.mlp.down_proj": 32,
+    "model.language_model.layers.24.mlp.gate_proj": 32,
+    "model.language_model.layers.24.mlp.up_proj": 32,
+    "model.language_model.layers.24.self_attn.k_proj": 32,
+    "model.language_model.layers.24.self_attn.o_proj": 32,
+    "model.language_model.layers.24.self_attn.q_proj": 32,
+    "model.language_model.layers.24.self_attn.v_proj": 32,
+    "model.language_model.layers.25.mlp.down_proj": 32,
+    "model.language_model.layers.25.mlp.gate_proj": 32,
+    "model.language_model.layers.25.mlp.up_proj": 32,
+    "model.language_model.layers.25.self_attn.k_proj": 32,
+    "model.language_model.layers.25.self_attn.o_proj": 32,
+    "model.language_model.layers.25.self_attn.q_proj": 32,
+    "model.language_model.layers.25.self_attn.v_proj": 32,
+    "model.language_model.layers.26.mlp.down_proj": 32,
+    "model.language_model.layers.26.mlp.gate_proj": 32,
+    "model.language_model.layers.26.mlp.up_proj": 32,
+    "model.language_model.layers.26.self_attn.k_proj": 32,
+    "model.language_model.layers.26.self_attn.o_proj": 32,
+    "model.language_model.layers.26.self_attn.q_proj": 32,
+    "model.language_model.layers.26.self_attn.v_proj": 32,
+    "model.language_model.layers.27.mlp.down_proj": 32,
+    "model.language_model.layers.27.mlp.gate_proj": 32,
+    "model.language_model.layers.27.mlp.up_proj": 32,
+    "model.language_model.layers.27.self_attn.k_proj": 32,
+    "model.language_model.layers.27.self_attn.o_proj": 32,
+    "model.language_model.layers.27.self_attn.q_proj": 32,
+    "model.language_model.layers.27.self_attn.v_proj": 32,
+    "model.language_model.layers.28.mlp.down_proj": 32,
+    "model.language_model.layers.28.mlp.gate_proj": 32,
+    "model.language_model.layers.28.mlp.up_proj": 32,
+    "model.language_model.layers.28.self_attn.k_proj": 32,
+    "model.language_model.layers.28.self_attn.o_proj": 32,
+    "model.language_model.layers.28.self_attn.q_proj": 32,
+    "model.language_model.layers.28.self_attn.v_proj": 32,
+    "model.language_model.layers.29.mlp.down_proj": 32,
+    "model.language_model.layers.29.mlp.gate_proj": 32,
+    "model.language_model.layers.29.mlp.up_proj": 32,
+    "model.language_model.layers.29.self_attn.k_proj": 32,
+    "model.language_model.layers.29.self_attn.o_proj": 32,
+    "model.language_model.layers.29.self_attn.q_proj": 32,
+    "model.language_model.layers.29.self_attn.v_proj": 32,
+    "model.language_model.layers.3.mlp.down_proj": 32,
+    "model.language_model.layers.3.mlp.gate_proj": 32,
+    "model.language_model.layers.3.mlp.up_proj": 32,
+    "model.language_model.layers.3.self_attn.k_proj": 32,
+    "model.language_model.layers.3.self_attn.o_proj": 32,
+    "model.language_model.layers.3.self_attn.q_proj": 32,
+    "model.language_model.layers.3.self_attn.v_proj": 32,
+    "model.language_model.layers.30.mlp.down_proj": 32,
+    "model.language_model.layers.30.mlp.gate_proj": 32,
+    "model.language_model.layers.30.mlp.up_proj": 32,
+    "model.language_model.layers.30.self_attn.k_proj": 32,
+    "model.language_model.layers.30.self_attn.o_proj": 32,
+    "model.language_model.layers.30.self_attn.q_proj": 32,
+    "model.language_model.layers.30.self_attn.v_proj": 32,
+    "model.language_model.layers.31.mlp.down_proj": 32,
+    "model.language_model.layers.31.mlp.gate_proj": 32,
+    "model.language_model.layers.31.mlp.up_proj": 32,
+    "model.language_model.layers.31.self_attn.k_proj": 32,
+    "model.language_model.layers.31.self_attn.o_proj": 32,
+    "model.language_model.layers.31.self_attn.q_proj": 32,
+    "model.language_model.layers.31.self_attn.v_proj": 32,
+    "model.language_model.layers.32.mlp.down_proj": 32,
+    "model.language_model.layers.32.mlp.gate_proj": 32,
+    "model.language_model.layers.32.mlp.up_proj": 32,
+    "model.language_model.layers.32.self_attn.k_proj": 32,
+    "model.language_model.layers.32.self_attn.o_proj": 32,
+    "model.language_model.layers.32.self_attn.q_proj": 32,
+    "model.language_model.layers.32.self_attn.v_proj": 32,
+    "model.language_model.layers.33.mlp.down_proj": 32,
+    "model.language_model.layers.33.mlp.gate_proj": 32,
+    "model.language_model.layers.33.mlp.up_proj": 32,
+    "model.language_model.layers.33.self_attn.k_proj": 32,
+    "model.language_model.layers.33.self_attn.o_proj": 32,
+    "model.language_model.layers.33.self_attn.q_proj": 32,
+    "model.language_model.layers.33.self_attn.v_proj": 32,
+    "model.language_model.layers.34.mlp.down_proj": 32,
+    "model.language_model.layers.34.mlp.gate_proj": 32,
+    "model.language_model.layers.34.mlp.up_proj": 32,
+    "model.language_model.layers.34.self_attn.k_proj": 32,
+    "model.language_model.layers.34.self_attn.o_proj": 32,
+    "model.language_model.layers.34.self_attn.q_proj": 32,
+    "model.language_model.layers.34.self_attn.v_proj": 32,
+    "model.language_model.layers.35.mlp.down_proj": 32,
+    "model.language_model.layers.35.mlp.gate_proj": 32,
+    "model.language_model.layers.35.mlp.up_proj": 32,
+    "model.language_model.layers.35.self_attn.k_proj": 32,
+    "model.language_model.layers.35.self_attn.o_proj": 32,
+    "model.language_model.layers.35.self_attn.q_proj": 32,
+    "model.language_model.layers.35.self_attn.v_proj": 32,
+    "model.language_model.layers.4.mlp.down_proj": 32,
+    "model.language_model.layers.4.mlp.gate_proj": 32,
+    "model.language_model.layers.4.mlp.up_proj": 32,
+    "model.language_model.layers.4.self_attn.k_proj": 32,
+    "model.language_model.layers.4.self_attn.o_proj": 32,
+    "model.language_model.layers.4.self_attn.q_proj": 32,
+    "model.language_model.layers.4.self_attn.v_proj": 32,
+    "model.language_model.layers.5.mlp.down_proj": 32,
+    "model.language_model.layers.5.mlp.gate_proj": 32,
+    "model.language_model.layers.5.mlp.up_proj": 32,
+    "model.language_model.layers.5.self_attn.k_proj": 32,
+    "model.language_model.layers.5.self_attn.o_proj": 32,
+    "model.language_model.layers.5.self_attn.q_proj": 32,
+    "model.language_model.layers.5.self_attn.v_proj": 32,
+    "model.language_model.layers.6.mlp.down_proj": 32,
+    "model.language_model.layers.6.mlp.gate_proj": 32,
+    "model.language_model.layers.6.mlp.up_proj": 32,
+    "model.language_model.layers.6.self_attn.k_proj": 32,
+    "model.language_model.layers.6.self_attn.o_proj": 32,
+    "model.language_model.layers.6.self_attn.q_proj": 32,
+    "model.language_model.layers.6.self_attn.v_proj": 32,
+    "model.language_model.layers.7.mlp.down_proj": 32,
+    "model.language_model.layers.7.mlp.gate_proj": 32,
+    "model.language_model.layers.7.mlp.up_proj": 32,
+    "model.language_model.layers.7.self_attn.k_proj": 32,
+    "model.language_model.layers.7.self_attn.o_proj": 32,
+    "model.language_model.layers.7.self_attn.q_proj": 32,
+    "model.language_model.layers.7.self_attn.v_proj": 32,
+    "model.language_model.layers.8.mlp.down_proj": 32,
+    "model.language_model.layers.8.mlp.gate_proj": 32,
+    "model.language_model.layers.8.mlp.up_proj": 32,
+    "model.language_model.layers.8.self_attn.k_proj": 32,
+    "model.language_model.layers.8.self_attn.o_proj": 32,
+    "model.language_model.layers.8.self_attn.q_proj": 32,
+    "model.language_model.layers.8.self_attn.v_proj": 32,
+    "model.language_model.layers.9.mlp.down_proj": 32,
+    "model.language_model.layers.9.mlp.gate_proj": 32,
+    "model.language_model.layers.9.mlp.up_proj": 32,
+    "model.language_model.layers.9.self_attn.k_proj": 32,
+    "model.language_model.layers.9.self_attn.o_proj": 32,
+    "model.language_model.layers.9.self_attn.q_proj": 32,
+    "model.language_model.layers.9.self_attn.v_proj": 32
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.0.mlp.down_proj": 16,
+    "model.language_model.layers.0.mlp.gate_proj": 16,
+    "model.language_model.layers.0.mlp.up_proj": 16,
+    "model.language_model.layers.0.self_attn.k_proj": 16,
+    "model.language_model.layers.0.self_attn.o_proj": 16,
+    "model.language_model.layers.0.self_attn.q_proj": 16,
+    "model.language_model.layers.0.self_attn.v_proj": 16,
+    "model.language_model.layers.1.mlp.down_proj": 16,
+    "model.language_model.layers.1.mlp.gate_proj": 16,
+    "model.language_model.layers.1.mlp.up_proj": 16,
+    "model.language_model.layers.1.self_attn.k_proj": 16,
+    "model.language_model.layers.1.self_attn.o_proj": 16,
+    "model.language_model.layers.1.self_attn.q_proj": 16,
+    "model.language_model.layers.1.self_attn.v_proj": 16,
+    "model.language_model.layers.10.mlp.down_proj": 16,
+    "model.language_model.layers.10.mlp.gate_proj": 16,
+    "model.language_model.layers.10.mlp.up_proj": 16,
+    "model.language_model.layers.10.self_attn.k_proj": 16,
+    "model.language_model.layers.10.self_attn.o_proj": 16,
+    "model.language_model.layers.10.self_attn.q_proj": 16,
+    "model.language_model.layers.10.self_attn.v_proj": 16,
+    "model.language_model.layers.11.mlp.down_proj": 16,
+    "model.language_model.layers.11.mlp.gate_proj": 16,
+    "model.language_model.layers.11.mlp.up_proj": 16,
+    "model.language_model.layers.11.self_attn.k_proj": 16,
+    "model.language_model.layers.11.self_attn.o_proj": 16,
+    "model.language_model.layers.11.self_attn.q_proj": 16,
+    "model.language_model.layers.11.self_attn.v_proj": 16,
+    "model.language_model.layers.12.mlp.down_proj": 16,
+    "model.language_model.layers.12.mlp.gate_proj": 16,
+    "model.language_model.layers.12.mlp.up_proj": 16,
+    "model.language_model.layers.12.self_attn.k_proj": 16,
+    "model.language_model.layers.12.self_attn.o_proj": 16,
+    "model.language_model.layers.12.self_attn.q_proj": 16,
+    "model.language_model.layers.12.self_attn.v_proj": 16,
+    "model.language_model.layers.13.mlp.down_proj": 16,
+    "model.language_model.layers.13.mlp.gate_proj": 16,
+    "model.language_model.layers.13.mlp.up_proj": 16,
+    "model.language_model.layers.13.self_attn.k_proj": 16,
+    "model.language_model.layers.13.self_attn.o_proj": 16,
+    "model.language_model.layers.13.self_attn.q_proj": 16,
+    "model.language_model.layers.13.self_attn.v_proj": 16,
+    "model.language_model.layers.14.mlp.down_proj": 16,
+    "model.language_model.layers.14.mlp.gate_proj": 16,
+    "model.language_model.layers.14.mlp.up_proj": 16,
+    "model.language_model.layers.14.self_attn.k_proj": 16,
+    "model.language_model.layers.14.self_attn.o_proj": 16,
+    "model.language_model.layers.14.self_attn.q_proj": 16,
+    "model.language_model.layers.14.self_attn.v_proj": 16,
+    "model.language_model.layers.15.mlp.down_proj": 16,
+    "model.language_model.layers.15.mlp.gate_proj": 16,
+    "model.language_model.layers.15.mlp.up_proj": 16,
+    "model.language_model.layers.15.self_attn.k_proj": 16,
+    "model.language_model.layers.15.self_attn.o_proj": 16,
+    "model.language_model.layers.15.self_attn.q_proj": 16,
+    "model.language_model.layers.15.self_attn.v_proj": 16,
+    "model.language_model.layers.16.mlp.down_proj": 16,
+    "model.language_model.layers.16.mlp.gate_proj": 16,
+    "model.language_model.layers.16.mlp.up_proj": 16,
+    "model.language_model.layers.16.self_attn.k_proj": 16,
+    "model.language_model.layers.16.self_attn.o_proj": 16,
+    "model.language_model.layers.16.self_attn.q_proj": 16,
+    "model.language_model.layers.16.self_attn.v_proj": 16,
+    "model.language_model.layers.17.mlp.down_proj": 16,
+    "model.language_model.layers.17.mlp.gate_proj": 16,
+    "model.language_model.layers.17.mlp.up_proj": 16,
+    "model.language_model.layers.17.self_attn.k_proj": 16,
+    "model.language_model.layers.17.self_attn.o_proj": 16,
+    "model.language_model.layers.17.self_attn.q_proj": 16,
+    "model.language_model.layers.17.self_attn.v_proj": 16,
+    "model.language_model.layers.18.mlp.down_proj": 16,
+    "model.language_model.layers.18.mlp.gate_proj": 16,
+    "model.language_model.layers.18.mlp.up_proj": 16,
+    "model.language_model.layers.18.self_attn.k_proj": 16,
+    "model.language_model.layers.18.self_attn.o_proj": 16,
+    "model.language_model.layers.18.self_attn.q_proj": 16,
+    "model.language_model.layers.18.self_attn.v_proj": 16,
+    "model.language_model.layers.19.mlp.down_proj": 16,
+    "model.language_model.layers.19.mlp.gate_proj": 16,
+    "model.language_model.layers.19.mlp.up_proj": 16,
+    "model.language_model.layers.19.self_attn.k_proj": 16,
+    "model.language_model.layers.19.self_attn.o_proj": 16,
+    "model.language_model.layers.19.self_attn.q_proj": 16,
+    "model.language_model.layers.19.self_attn.v_proj": 16,
+    "model.language_model.layers.2.mlp.down_proj": 16,
+    "model.language_model.layers.2.mlp.gate_proj": 16,
+    "model.language_model.layers.2.mlp.up_proj": 16,
+    "model.language_model.layers.2.self_attn.k_proj": 16,
+    "model.language_model.layers.2.self_attn.o_proj": 16,
+    "model.language_model.layers.2.self_attn.q_proj": 16,
+    "model.language_model.layers.2.self_attn.v_proj": 16,
+    "model.language_model.layers.20.mlp.down_proj": 16,
+    "model.language_model.layers.20.mlp.gate_proj": 16,
+    "model.language_model.layers.20.mlp.up_proj": 16,
+    "model.language_model.layers.20.self_attn.k_proj": 16,
+    "model.language_model.layers.20.self_attn.o_proj": 16,
+    "model.language_model.layers.20.self_attn.q_proj": 16,
+    "model.language_model.layers.20.self_attn.v_proj": 16,
+    "model.language_model.layers.21.mlp.down_proj": 16,
+    "model.language_model.layers.21.mlp.gate_proj": 16,
+    "model.language_model.layers.21.mlp.up_proj": 16,
+    "model.language_model.layers.21.self_attn.k_proj": 16,
+    "model.language_model.layers.21.self_attn.o_proj": 16,
+    "model.language_model.layers.21.self_attn.q_proj": 16,
+    "model.language_model.layers.21.self_attn.v_proj": 16,
+    "model.language_model.layers.22.mlp.down_proj": 16,
+    "model.language_model.layers.22.mlp.gate_proj": 16,
+    "model.language_model.layers.22.mlp.up_proj": 16,
+    "model.language_model.layers.22.self_attn.k_proj": 16,
+    "model.language_model.layers.22.self_attn.o_proj": 16,
+    "model.language_model.layers.22.self_attn.q_proj": 16,
+    "model.language_model.layers.22.self_attn.v_proj": 16,
+    "model.language_model.layers.23.mlp.down_proj": 16,
+    "model.language_model.layers.23.mlp.gate_proj": 16,
+    "model.language_model.layers.23.mlp.up_proj": 16,
+    "model.language_model.layers.23.self_attn.k_proj": 16,
+    "model.language_model.layers.23.self_attn.o_proj": 16,
+    "model.language_model.layers.23.self_attn.q_proj": 16,
+    "model.language_model.layers.23.self_attn.v_proj": 16,
+    "model.language_model.layers.24.mlp.down_proj": 16,
+    "model.language_model.layers.24.mlp.gate_proj": 16,
+    "model.language_model.layers.24.mlp.up_proj": 16,
+    "model.language_model.layers.24.self_attn.k_proj": 16,
+    "model.language_model.layers.24.self_attn.o_proj": 16,
+    "model.language_model.layers.24.self_attn.q_proj": 16,
+    "model.language_model.layers.24.self_attn.v_proj": 16,
+    "model.language_model.layers.25.mlp.down_proj": 16,
+    "model.language_model.layers.25.mlp.gate_proj": 16,
+    "model.language_model.layers.25.mlp.up_proj": 16,
+    "model.language_model.layers.25.self_attn.k_proj": 16,
+    "model.language_model.layers.25.self_attn.o_proj": 16,
+    "model.language_model.layers.25.self_attn.q_proj": 16,
+    "model.language_model.layers.25.self_attn.v_proj": 16,
+    "model.language_model.layers.26.mlp.down_proj": 16,
+    "model.language_model.layers.26.mlp.gate_proj": 16,
+    "model.language_model.layers.26.mlp.up_proj": 16,
+    "model.language_model.layers.26.self_attn.k_proj": 16,
+    "model.language_model.layers.26.self_attn.o_proj": 16,
+    "model.language_model.layers.26.self_attn.q_proj": 16,
+    "model.language_model.layers.26.self_attn.v_proj": 16,
+    "model.language_model.layers.27.mlp.down_proj": 16,
+    "model.language_model.layers.27.mlp.gate_proj": 16,
+    "model.language_model.layers.27.mlp.up_proj": 16,
+    "model.language_model.layers.27.self_attn.k_proj": 16,
+    "model.language_model.layers.27.self_attn.o_proj": 16,
+    "model.language_model.layers.27.self_attn.q_proj": 16,
+    "model.language_model.layers.27.self_attn.v_proj": 16,
+    "model.language_model.layers.28.mlp.down_proj": 16,
+    "model.language_model.layers.28.mlp.gate_proj": 16,
+    "model.language_model.layers.28.mlp.up_proj": 16,
+    "model.language_model.layers.28.self_attn.k_proj": 16,
+    "model.language_model.layers.28.self_attn.o_proj": 16,
+    "model.language_model.layers.28.self_attn.q_proj": 16,
+    "model.language_model.layers.28.self_attn.v_proj": 16,
+    "model.language_model.layers.29.mlp.down_proj": 16,
+    "model.language_model.layers.29.mlp.gate_proj": 16,
+    "model.language_model.layers.29.mlp.up_proj": 16,
+    "model.language_model.layers.29.self_attn.k_proj": 16,
+    "model.language_model.layers.29.self_attn.o_proj": 16,
+    "model.language_model.layers.29.self_attn.q_proj": 16,
+    "model.language_model.layers.29.self_attn.v_proj": 16,
+    "model.language_model.layers.3.mlp.down_proj": 16,
+    "model.language_model.layers.3.mlp.gate_proj": 16,
+    "model.language_model.layers.3.mlp.up_proj": 16,
+    "model.language_model.layers.3.self_attn.k_proj": 16,
+    "model.language_model.layers.3.self_attn.o_proj": 16,
+    "model.language_model.layers.3.self_attn.q_proj": 16,
+    "model.language_model.layers.3.self_attn.v_proj": 16,
+    "model.language_model.layers.30.mlp.down_proj": 16,
+    "model.language_model.layers.30.mlp.gate_proj": 16,
+    "model.language_model.layers.30.mlp.up_proj": 16,
+    "model.language_model.layers.30.self_attn.k_proj": 16,
+    "model.language_model.layers.30.self_attn.o_proj": 16,
+    "model.language_model.layers.30.self_attn.q_proj": 16,
+    "model.language_model.layers.30.self_attn.v_proj": 16,
+    "model.language_model.layers.31.mlp.down_proj": 16,
+    "model.language_model.layers.31.mlp.gate_proj": 16,
+    "model.language_model.layers.31.mlp.up_proj": 16,
+    "model.language_model.layers.31.self_attn.k_proj": 16,
+    "model.language_model.layers.31.self_attn.o_proj": 16,
+    "model.language_model.layers.31.self_attn.q_proj": 16,
+    "model.language_model.layers.31.self_attn.v_proj": 16,
+    "model.language_model.layers.32.mlp.down_proj": 16,
+    "model.language_model.layers.32.mlp.gate_proj": 16,
+    "model.language_model.layers.32.mlp.up_proj": 16,
+    "model.language_model.layers.32.self_attn.k_proj": 16,
+    "model.language_model.layers.32.self_attn.o_proj": 16,
+    "model.language_model.layers.32.self_attn.q_proj": 16,
+    "model.language_model.layers.32.self_attn.v_proj": 16,
+    "model.language_model.layers.33.mlp.down_proj": 16,
+    "model.language_model.layers.33.mlp.gate_proj": 16,
+    "model.language_model.layers.33.mlp.up_proj": 16,
+    "model.language_model.layers.33.self_attn.k_proj": 16,
+    "model.language_model.layers.33.self_attn.o_proj": 16,
+    "model.language_model.layers.33.self_attn.q_proj": 16,
+    "model.language_model.layers.33.self_attn.v_proj": 16,
+    "model.language_model.layers.34.mlp.down_proj": 16,
+    "model.language_model.layers.34.mlp.gate_proj": 16,
+    "model.language_model.layers.34.mlp.up_proj": 16,
+    "model.language_model.layers.34.self_attn.k_proj": 16,
+    "model.language_model.layers.34.self_attn.o_proj": 16,
+    "model.language_model.layers.34.self_attn.q_proj": 16,
+    "model.language_model.layers.34.self_attn.v_proj": 16,
+    "model.language_model.layers.35.mlp.down_proj": 16,
+    "model.language_model.layers.35.mlp.gate_proj": 16,
+    "model.language_model.layers.35.mlp.up_proj": 16,
+    "model.language_model.layers.35.self_attn.k_proj": 16,
+    "model.language_model.layers.35.self_attn.o_proj": 16,
+    "model.language_model.layers.35.self_attn.q_proj": 16,
+    "model.language_model.layers.35.self_attn.v_proj": 16,
+    "model.language_model.layers.4.mlp.down_proj": 16,
+    "model.language_model.layers.4.mlp.gate_proj": 16,
+    "model.language_model.layers.4.mlp.up_proj": 16,
+    "model.language_model.layers.4.self_attn.k_proj": 16,
+    "model.language_model.layers.4.self_attn.o_proj": 16,
+    "model.language_model.layers.4.self_attn.q_proj": 16,
+    "model.language_model.layers.4.self_attn.v_proj": 16,
+    "model.language_model.layers.5.mlp.down_proj": 16,
+    "model.language_model.layers.5.mlp.gate_proj": 16,
+    "model.language_model.layers.5.mlp.up_proj": 16,
+    "model.language_model.layers.5.self_attn.k_proj": 16,
+    "model.language_model.layers.5.self_attn.o_proj": 16,
+    "model.language_model.layers.5.self_attn.q_proj": 16,
+    "model.language_model.layers.5.self_attn.v_proj": 16,
+    "model.language_model.layers.6.mlp.down_proj": 16,
+    "model.language_model.layers.6.mlp.gate_proj": 16,
+    "model.language_model.layers.6.mlp.up_proj": 16,
+    "model.language_model.layers.6.self_attn.k_proj": 16,
+    "model.language_model.layers.6.self_attn.o_proj": 16,
+    "model.language_model.layers.6.self_attn.q_proj": 16,
+    "model.language_model.layers.6.self_attn.v_proj": 16,
+    "model.language_model.layers.7.mlp.down_proj": 16,
+    "model.language_model.layers.7.mlp.gate_proj": 16,
+    "model.language_model.layers.7.mlp.up_proj": 16,
+    "model.language_model.layers.7.self_attn.k_proj": 16,
+    "model.language_model.layers.7.self_attn.o_proj": 16,
+    "model.language_model.layers.7.self_attn.q_proj": 16,
+    "model.language_model.layers.7.self_attn.v_proj": 16,
+    "model.language_model.layers.8.mlp.down_proj": 16,
+    "model.language_model.layers.8.mlp.gate_proj": 16,
+    "model.language_model.layers.8.mlp.up_proj": 16,
+    "model.language_model.layers.8.self_attn.k_proj": 16,
+    "model.language_model.layers.8.self_attn.o_proj": 16,
+    "model.language_model.layers.8.self_attn.q_proj": 16,
+    "model.language_model.layers.8.self_attn.v_proj": 16,
+    "model.language_model.layers.9.mlp.down_proj": 16,
+    "model.language_model.layers.9.mlp.gate_proj": 16,
+    "model.language_model.layers.9.mlp.up_proj": 16,
+    "model.language_model.layers.9.self_attn.k_proj": 16,
+    "model.language_model.layers.9.self_attn.o_proj": 16,
+    "model.language_model.layers.9.self_attn.q_proj": 16,
+    "model.language_model.layers.9.self_attn.v_proj": 16
+  },
+  "revision": null,
+  "target_modules": [
+    "self_attn.q_proj",
+    "self_attn.k_proj",
+    "self_attn.v_proj",
+    "down_proj",
+    "up_proj",
+    "o_proj",
+    "gate_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/fixed_internvl/removal/Full_s42/config.json

@@ -0,0 +1,99 @@
+{
+  "behavior": "removal",
+  "seed": 42,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "world_size": 1,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "arm": {
+    "arm": "Full",
+    "layers": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      10,
+      11,
+      12,
+      13,
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21,
+      22,
+      23,
+      24,
+      25,
+      26,
+      27,
+      28,
+      29,
+      30,
+      31,
+      32,
+      33,
+      34,
+      35
+    ],
+    "ranks": {
+      "0": 16,
+      "1": 16,
+      "2": 16,
+      "3": 16,
+      "4": 16,
+      "5": 16,
+      "6": 16,
+      "7": 16,
+      "8": 16,
+      "9": 16,
+      "10": 16,
+      "11": 16,
+      "12": 16,
+      "13": 16,
+      "14": 16,
+      "15": 16,
+      "16": 16,
+      "17": 16,
+      "18": 16,
+      "19": 16,
+      "20": 16,
+      "21": 16,
+      "22": 16,
+      "23": 16,
+      "24": 16,
+      "25": 16,
+      "26": 16,
+      "27": 16,
+      "28": 16,
+      "29": 16,
+      "30": 16,
+      "31": 16,
+      "32": 16,
+      "33": 16,
+      "34": 16,
+      "35": 16
+    },
+    "parameters": 43646976,
+    "budget": 43646976
+  }
+}

runs/fixed_internvl/removal/Full_s42/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 10.6151704788208, "loss_scale": 4.0, "retries": 0, "seconds": 6.1, "max_memory_gb": 16.81, "ce": 1.0494785578921437, "rank": 0.15784265100955963}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 6.945900917053223, "loss_scale": 4.0, "retries": 0, "seconds": 11.4, "max_memory_gb": 17.13, "ce": 0.8236040879639859, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 12.252994537353516, "loss_scale": 4.0, "retries": 0, "seconds": 16.8, "max_memory_gb": 17.13, "ce": 0.8158213595452253, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 10.192559242248535, "loss_scale": 4.0, "retries": 0, "seconds": 22.2, "max_memory_gb": 17.13, "ce": 0.8382595653893077, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 4.007576942443848, "loss_scale": 4.0, "retries": 0, "seconds": 27.5, "max_memory_gb": 17.13, "ce": 0.6193590731127188, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 4.061712741851807, "loss_scale": 4.0, "retries": 0, "seconds": 32.9, "max_memory_gb": 17.13, "ce": 0.3885403181484435, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 2.8685574531555176, "loss_scale": 4.0, "retries": 0, "seconds": 38.3, "max_memory_gb": 17.13, "ce": 0.550345950294286, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 1.4529266357421875, "loss_scale": 4.0, "retries": 0, "seconds": 43.6, "max_memory_gb": 17.13, "ce": 0.23961634641432283, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 2.6838157176971436, "loss_scale": 4.0, "retries": 0, "seconds": 48.9, "max_memory_gb": 17.13, "ce": 0.600857689161785, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 2.719325065612793, "loss_scale": 4.0, "retries": 0, "seconds": 54.3, "max_memory_gb": 17.13, "ce": 0.2562550976726925, "rank": 0.20723474025726318}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.3553110957145691, "loss_scale": 4.0, "retries": 0, "seconds": 59.7, "max_memory_gb": 17.13, "ce": 0.14331114825472469, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 4.017502784729004, "loss_scale": 4.0, "retries": 0, "seconds": 65.1, "max_memory_gb": 17.13, "ce": 0.6065822996570205, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 0.8914949297904968, "loss_scale": 4.0, "retries": 0, "seconds": 70.5, "max_memory_gb": 17.13, "ce": 0.373904571053572, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 0.47786572575569153, "loss_scale": 4.0, "retries": 0, "seconds": 75.8, "max_memory_gb": 17.13, "ce": 0.3207811414904427, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 4.697871685028076, "loss_scale": 4.0, "retries": 0, "seconds": 81.1, "max_memory_gb": 17.13, "ce": 0.5159571440890431, "rank": 0.1958904266357422}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 1.2991162538528442, "loss_scale": 4.0, "retries": 0, "seconds": 86.5, "max_memory_gb": 17.13, "ce": 0.34418409356112534, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.7050463557243347, "loss_scale": 4.0, "retries": 0, "seconds": 91.9, "max_memory_gb": 17.13, "ce": 0.24818349638371728, "rank": 0.10988393425941467}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 1.5272469520568848, "loss_scale": 4.0, "retries": 0, "seconds": 97.4, "max_memory_gb": 17.13, "ce": 0.28959724912419915, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 0.5157034993171692, "loss_scale": 4.0, "retries": 0, "seconds": 102.7, "max_memory_gb": 17.13, "ce": 0.1311718609649688, "rank": 0.0}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 1.0561237335205078, "loss_scale": 4.0, "retries": 0, "seconds": 108.1, "max_memory_gb": 17.13, "ce": 0.15225983370328322, "rank": 0.0}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.3612208366394043, "loss_scale": 4.0, "retries": 0, "seconds": 113.5, "max_memory_gb": 17.13, "ce": 0.03267141105607152, "rank": 0.0}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 1.0211119651794434, "loss_scale": 4.0, "retries": 0, "seconds": 118.9, "max_memory_gb": 17.13, "ce": 0.15439729309582617, "rank": 0.0}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.062707543373108, "loss_scale": 4.0, "retries": 0, "seconds": 124.3, "max_memory_gb": 17.13, "ce": 0.11961013348627603, "rank": 0.0}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 1.1819528341293335, "loss_scale": 4.0, "retries": 0, "seconds": 129.6, "max_memory_gb": 17.13, "ce": 0.1832938530624233, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 1.3984650373458862, "loss_scale": 4.0, "retries": 0, "seconds": 135.0, "max_memory_gb": 17.13, "ce": 0.056724515081441496, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 3.5375516414642334, "loss_scale": 4.0, "retries": 0, "seconds": 140.4, "max_memory_gb": 17.13, "ce": 0.11222052813741357, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 12.411982536315918, "loss_scale": 4.0, "retries": 0, "seconds": 145.8, "max_memory_gb": 17.13, "ce": 0.8889353150518673, "rank": 0.0}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 2.837730646133423, "loss_scale": 4.0, "retries": 0, "seconds": 151.1, "max_memory_gb": 17.13, "ce": 0.17555827878823038, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 0.9875444173812866, "loss_scale": 4.0, "retries": 0, "seconds": 156.4, "max_memory_gb": 17.13, "ce": 0.0797605759871658, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 3.770798444747925, "loss_scale": 4.0, "retries": 0, "seconds": 161.9, "max_memory_gb": 17.13, "ce": 0.18369213794358075, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 5.577774524688721, "loss_scale": 4.0, "retries": 0, "seconds": 167.2, "max_memory_gb": 17.13, "ce": 0.73538101081067, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 0.4344407021999359, "loss_scale": 4.0, "retries": 0, "seconds": 172.5, "max_memory_gb": 17.13, "ce": 0.01756783516611904, "rank": 0.0}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.221170574426651, "loss_scale": 4.0, "retries": 0, "seconds": 177.9, "max_memory_gb": 17.13, "ce": 0.00536154678411549, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.19024786353111267, "loss_scale": 4.0, "retries": 0, "seconds": 183.3, "max_memory_gb": 17.14, "ce": 0.006538763593425756, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.7928850054740906, "loss_scale": 4.0, "retries": 0, "seconds": 188.7, "max_memory_gb": 17.14, "ce": 0.05731032436597161, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 2.6933720111846924, "loss_scale": 4.0, "retries": 0, "seconds": 193.9, "max_memory_gb": 17.14, "ce": 0.31374669635340524, "rank": 0.07161256670951843}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 1.8441320657730103, "loss_scale": 4.0, "retries": 0, "seconds": 199.3, "max_memory_gb": 17.14, "ce": 0.2092086788998131, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.22211842238903046, "loss_scale": 4.0, "retries": 0, "seconds": 204.7, "max_memory_gb": 17.14, "ce": 0.008456760089757154, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 5.698313236236572, "loss_scale": 4.0, "retries": 0, "seconds": 210.1, "max_memory_gb": 17.14, "ce": 0.38441482194775745, "rank": 0.0}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.5021747946739197, "loss_scale": 4.0, "retries": 0, "seconds": 215.5, "max_memory_gb": 17.14, "ce": 0.024273592196564664, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.33905917406082153, "loss_scale": 4.0, "retries": 0, "seconds": 220.8, "max_memory_gb": 17.14, "ce": 0.011174373582434782, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 1.888137936592102, "loss_scale": 4.0, "retries": 0, "seconds": 226.2, "max_memory_gb": 17.14, "ce": 0.17004811153037735, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 2.8468339443206787, "loss_scale": 4.0, "retries": 0, "seconds": 231.6, "max_memory_gb": 17.14, "ce": 0.10787333984990255, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 3.4755470752716064, "loss_scale": 4.0, "retries": 0, "seconds": 237.0, "max_memory_gb": 17.14, "ce": 0.2798678646568078, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 4.1775407791137695, "loss_scale": 4.0, "retries": 0, "seconds": 242.4, "max_memory_gb": 17.14, "ce": 0.24462937535281526, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 1.2819387912750244, "loss_scale": 4.0, "retries": 0, "seconds": 247.7, "max_memory_gb": 17.14, "ce": 0.19071294579043752, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.12703323364257812, "loss_scale": 4.0, "retries": 0, "seconds": 253.1, "max_memory_gb": 17.14, "ce": 0.003689463748742128, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.5543484091758728, "loss_scale": 4.0, "retries": 0, "seconds": 258.5, "max_memory_gb": 17.14, "ce": 0.07004885396418103, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 1.1622474193572998, "loss_scale": 4.0, "retries": 0, "seconds": 264.0, "max_memory_gb": 17.14, "ce": 0.06109652039071989, "rank": 0.12500378489494324}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.37678614258766174, "loss_scale": 4.0, "retries": 0, "seconds": 269.4, "max_memory_gb": 17.14, "ce": 0.01931131640685635, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 2.117887020111084, "loss_scale": 4.0, "retries": 0, "seconds": 274.7, "max_memory_gb": 17.14, "ce": 0.0885596234370496, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.6973091959953308, "loss_scale": 4.0, "retries": 0, "seconds": 280.1, "max_memory_gb": 17.14, "ce": 0.011938880218565373, "rank": 0.12561187148094177}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 0.5454883575439453, "loss_scale": 4.0, "retries": 0, "seconds": 285.5, "max_memory_gb": 17.14, "ce": 0.027815131841407492, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 1.7271220684051514, "loss_scale": 4.0, "retries": 0, "seconds": 290.9, "max_memory_gb": 17.14, "ce": 0.1209528666145161, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 7.770111560821533, "loss_scale": 4.0, "retries": 0, "seconds": 296.1, "max_memory_gb": 17.14, "ce": 0.5561055475945977, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 1.04017174243927, "loss_scale": 4.0, "retries": 0, "seconds": 301.5, "max_memory_gb": 17.14, "ce": 0.02351537787876623, "rank": 0.0}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.4051321744918823, "loss_scale": 4.0, "retries": 0, "seconds": 306.9, "max_memory_gb": 17.14, "ce": 0.026099358894498437, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 1.605074405670166, "loss_scale": 4.0, "retries": 0, "seconds": 312.3, "max_memory_gb": 17.14, "ce": 0.11773355063178315, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.17286454141139984, "loss_scale": 4.0, "retries": 0, "seconds": 317.8, "max_memory_gb": 17.14, "ce": 0.005214700513079151, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.9993451237678528, "loss_scale": 4.0, "retries": 0, "seconds": 323.0, "max_memory_gb": 17.14, "ce": 0.10528555257243966, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.5985634326934814, "loss_scale": 4.0, "retries": 0, "seconds": 328.4, "max_memory_gb": 17.14, "ce": 0.03631105687327363, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.7535181045532227, "loss_scale": 4.0, "retries": 0, "seconds": 333.9, "max_memory_gb": 17.14, "ce": 0.05348911108012544, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 2.41853404045105, "loss_scale": 4.0, "retries": 0, "seconds": 339.3, "max_memory_gb": 17.14, "ce": 0.21388321082531547, "rank": 0.0027889013290405273}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.8887636065483093, "loss_scale": 4.0, "retries": 0, "seconds": 344.7, "max_memory_gb": 17.14, "ce": 0.0716786056740375, "rank": 0.0}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 5.187752723693848, "loss_scale": 4.0, "retries": 0, "seconds": 350.0, "max_memory_gb": 17.14, "ce": 0.43061310021585086, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 5.278092861175537, "loss_scale": 4.0, "retries": 0, "seconds": 355.4, "max_memory_gb": 17.14, "ce": 0.06941668012427726, "rank": 0.0}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.1869095414876938, "loss_scale": 4.0, "retries": 0, "seconds": 360.8, "max_memory_gb": 17.14, "ce": 0.007187664690263773, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.1773175448179245, "loss_scale": 4.0, "retries": 0, "seconds": 366.2, "max_memory_gb": 17.14, "ce": 0.010126505050038759, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 1.433310627937317, "loss_scale": 4.0, "retries": 0, "seconds": 371.6, "max_memory_gb": 17.14, "ce": 0.07262256781177712, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 2.7784249782562256, "loss_scale": 4.0, "retries": 0, "seconds": 376.9, "max_memory_gb": 17.14, "ce": 0.2078415378821319, "rank": 0.0}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 4.395998477935791, "loss_scale": 4.0, "retries": 0, "seconds": 382.3, "max_memory_gb": 17.14, "ce": 0.6316136891809947, "rank": 0.12137866765260696}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.055826298892498016, "loss_scale": 4.0, "retries": 0, "seconds": 387.7, "max_memory_gb": 17.14, "ce": 0.002147298806562503, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 3.6243674755096436, "loss_scale": 4.0, "retries": 0, "seconds": 393.2, "max_memory_gb": 17.14, "ce": 0.4463406340501024, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.8768568634986877, "loss_scale": 4.0, "retries": 0, "seconds": 398.6, "max_memory_gb": 17.14, "ce": 0.03914720724787912, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.7007854580879211, "loss_scale": 4.0, "retries": 0, "seconds": 403.8, "max_memory_gb": 17.14, "ce": 0.03780129791266518, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 0.9054008722305298, "loss_scale": 4.0, "retries": 0, "seconds": 409.3, "max_memory_gb": 17.14, "ce": 0.04063136596460026, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.7914982438087463, "loss_scale": 4.0, "retries": 0, "seconds": 414.7, "max_memory_gb": 17.14, "ce": 0.039783188082765264, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.3119487762451172, "loss_scale": 4.0, "retries": 0, "seconds": 420.1, "max_memory_gb": 17.14, "ce": 0.024476099469211476, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.8510826230049133, "loss_scale": 4.0, "retries": 0, "seconds": 425.4, "max_memory_gb": 17.14, "ce": 0.10199003157003972, "rank": 0.0}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 1.0344016551971436, "loss_scale": 4.0, "retries": 0, "seconds": 430.8, "max_memory_gb": 17.14, "ce": 0.19313757861618797, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 4.013585090637207, "loss_scale": 4.0, "retries": 0, "seconds": 436.2, "max_memory_gb": 17.14, "ce": 0.6259372276867907, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.5308628082275391, "loss_scale": 4.0, "retries": 0, "seconds": 441.7, "max_memory_gb": 17.14, "ce": 0.029814303625698813, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 0.8595882058143616, "loss_scale": 4.0, "retries": 0, "seconds": 447.1, "max_memory_gb": 17.14, "ce": 0.046352697710062785, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 1.7349971532821655, "loss_scale": 4.0, "retries": 0, "seconds": 452.4, "max_memory_gb": 17.14, "ce": 0.14789288142583246, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.3358794152736664, "loss_scale": 4.0, "retries": 0, "seconds": 457.8, "max_memory_gb": 17.14, "ce": 0.03183551633651405, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.38429713249206543, "loss_scale": 4.0, "retries": 0, "seconds": 463.2, "max_memory_gb": 17.14, "ce": 0.016667362709767986, "rank": 0.12560532987117767}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.3393920361995697, "loss_scale": 4.0, "retries": 0, "seconds": 468.6, "max_memory_gb": 17.14, "ce": 0.015869703841417504, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.9870346188545227, "loss_scale": 4.0, "retries": 0, "seconds": 473.9, "max_memory_gb": 17.14, "ce": 0.042841701831093815, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.07968872785568237, "loss_scale": 4.0, "retries": 0, "seconds": 479.3, "max_memory_gb": 17.14, "ce": 0.003319330866020209, "rank": 0.0}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.6089972257614136, "loss_scale": 4.0, "retries": 0, "seconds": 484.8, "max_memory_gb": 17.14, "ce": 0.00024252070170405204, "rank": 0.06566829979419708}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 0.375345379114151, "loss_scale": 4.0, "retries": 0, "seconds": 490.2, "max_memory_gb": 17.14, "ce": 0.01870869746835524, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 0.7989307641983032, "loss_scale": 4.0, "retries": 0, "seconds": 495.5, "max_memory_gb": 17.14, "ce": 0.04247581103891207, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 2.519761323928833, "loss_scale": 4.0, "retries": 0, "seconds": 500.9, "max_memory_gb": 17.14, "ce": 0.1628929017133487, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 2.999180555343628, "loss_scale": 4.0, "retries": 0, "seconds": 506.3, "max_memory_gb": 17.14, "ce": 0.23039669994955148, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 0.5737359523773193, "loss_scale": 4.0, "retries": 0, "seconds": 511.7, "max_memory_gb": 17.14, "ce": 0.036783246420782234, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 0.5555291175842285, "loss_scale": 4.0, "retries": 0, "seconds": 517.2, "max_memory_gb": 17.14, "ce": 0.0490541549165755, "rank": 0.0}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.34500521421432495, "loss_scale": 4.0, "retries": 0, "seconds": 522.4, "max_memory_gb": 17.14, "ce": 0.016489628154886304, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.0974913164973259, "loss_scale": 4.0, "retries": 0, "seconds": 527.8, "max_memory_gb": 17.14, "ce": 0.007233618402537445, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.8314237594604492, "loss_scale": 4.0, "retries": 0, "seconds": 533.3, "max_memory_gb": 17.14, "ce": 0.07228075954617452, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.09575086832046509, "loss_scale": 4.0, "retries": 0, "seconds": 538.7, "max_memory_gb": 17.14, "ce": 0.004945444512941322, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 2.04138445854187, "loss_scale": 4.0, "retries": 0, "seconds": 544.1, "max_memory_gb": 17.14, "ce": 0.2205761944958624, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.07108656316995621, "loss_scale": 4.0, "retries": 0, "seconds": 549.4, "max_memory_gb": 17.14, "ce": 0.0021630404274475268, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 2.154996633529663, "loss_scale": 4.0, "retries": 0, "seconds": 554.7, "max_memory_gb": 17.14, "ce": 0.22775210229656295, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 1.352699875831604, "loss_scale": 4.0, "retries": 0, "seconds": 560.1, "max_memory_gb": 17.14, "ce": 0.1332011520192964, "rank": 0.0}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 2.475691318511963, "loss_scale": 4.0, "retries": 0, "seconds": 565.5, "max_memory_gb": 17.14, "ce": 0.13924264665143937, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 3.465393543243408, "loss_scale": 4.0, "retries": 0, "seconds": 570.9, "max_memory_gb": 17.14, "ce": 0.18382663861075343, "rank": 0.0}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 0.8590087890625, "loss_scale": 4.0, "retries": 0, "seconds": 576.2, "max_memory_gb": 17.14, "ce": 0.09684213252876361, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 2.438518524169922, "loss_scale": 4.0, "retries": 0, "seconds": 581.6, "max_memory_gb": 17.14, "ce": 0.273420975906447, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 0.7031298279762268, "loss_scale": 4.0, "retries": 0, "seconds": 587.0, "max_memory_gb": 17.14, "ce": 0.12381832125822712, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 1.0912127494812012, "loss_scale": 4.0, "retries": 0, "seconds": 592.4, "max_memory_gb": 17.14, "ce": 0.039435230673461774, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.04730210453271866, "loss_scale": 4.0, "retries": 0, "seconds": 597.8, "max_memory_gb": 17.14, "ce": 0.0012642090480312618, "rank": 0.1250070184469223}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 1.31795072555542, "loss_scale": 4.0, "retries": 0, "seconds": 603.1, "max_memory_gb": 17.14, "ce": 0.08048950957334, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 1.633182406425476, "loss_scale": 4.0, "retries": 0, "seconds": 608.4, "max_memory_gb": 17.14, "ce": 0.4613135800857435, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.09628598392009735, "loss_scale": 4.0, "retries": 0, "seconds": 613.9, "max_memory_gb": 17.14, "ce": 0.005422090304477933, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 5.1567511558532715, "loss_scale": 4.0, "retries": 0, "seconds": 619.3, "max_memory_gb": 17.14, "ce": 0.32855879044529956, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.1867193877696991, "loss_scale": 4.0, "retries": 0, "seconds": 624.7, "max_memory_gb": 17.14, "ce": 0.004959856517871231, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 2.501558542251587, "loss_scale": 4.0, "retries": 0, "seconds": 630.0, "max_memory_gb": 17.14, "ce": 0.30659487584580347, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 2.8172216415405273, "loss_scale": 4.0, "retries": 0, "seconds": 635.8, "max_memory_gb": 17.14, "ce": 0.3723077416150318, "rank": 0.125019833445549}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.8382743000984192, "loss_scale": 4.0, "retries": 0, "seconds": 642.1, "max_memory_gb": 17.14, "ce": 0.03217396507960757, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 1.304898738861084, "loss_scale": 4.0, "retries": 0, "seconds": 647.6, "max_memory_gb": 17.14, "ce": 0.04553298307291698, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 1.1521819829940796, "loss_scale": 4.0, "retries": 0, "seconds": 653.0, "max_memory_gb": 17.14, "ce": 0.0746723854016409, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.6239356398582458, "loss_scale": 4.0, "retries": 0, "seconds": 658.3, "max_memory_gb": 17.14, "ce": 0.03294895812712184, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.021558227017521858, "loss_scale": 4.0, "retries": 0, "seconds": 663.8, "max_memory_gb": 17.14, "ce": 0.0009913814469655335, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 0.6938427686691284, "loss_scale": 4.0, "retries": 0, "seconds": 669.2, "max_memory_gb": 17.14, "ce": 0.13060354456820278, "rank": 0.15594542026519775}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 4.118208408355713, "loss_scale": 4.0, "retries": 0, "seconds": 674.5, "max_memory_gb": 17.14, "ce": 0.08484445972726462, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 1.625993251800537, "loss_scale": 4.0, "retries": 0, "seconds": 679.9, "max_memory_gb": 17.14, "ce": 0.44750313604026815, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.40162745118141174, "loss_scale": 4.0, "retries": 0, "seconds": 685.4, "max_memory_gb": 17.14, "ce": 0.045202096480807086, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 0.3887918293476105, "loss_scale": 4.0, "retries": 0, "seconds": 690.8, "max_memory_gb": 17.14, "ce": 0.03217640424418278, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 2.2688465118408203, "loss_scale": 4.0, "retries": 0, "seconds": 696.2, "max_memory_gb": 17.14, "ce": 0.1740074096405806, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 0.5389565825462341, "loss_scale": 4.0, "retries": 0, "seconds": 701.4, "max_memory_gb": 17.14, "ce": 0.031052187216346283, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 0.46445536613464355, "loss_scale": 4.0, "retries": 0, "seconds": 706.9, "max_memory_gb": 17.14, "ce": 0.12085286418778196, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 0.011988352052867413, "loss_scale": 4.0, "retries": 0, "seconds": 712.3, "max_memory_gb": 17.14, "ce": 0.0007493223972687701, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 0.11787918955087662, "loss_scale": 4.0, "retries": 0, "seconds": 717.7, "max_memory_gb": 17.14, "ce": 0.007252443099105221, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 1.6730678081512451, "loss_scale": 4.0, "retries": 0, "seconds": 723.1, "max_memory_gb": 17.14, "ce": 0.04650604778930756, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 1.5481626987457275, "loss_scale": 4.0, "retries": 0, "seconds": 728.4, "max_memory_gb": 17.14, "ce": 0.409410810870213, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 0.11727361381053925, "loss_scale": 4.0, "retries": 0, "seconds": 734.0, "max_memory_gb": 17.14, "ce": 0.004944168590100162, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 2.9550864696502686, "loss_scale": 4.0, "retries": 0, "seconds": 739.5, "max_memory_gb": 17.14, "ce": 0.3337569761215491, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 0.37668630480766296, "loss_scale": 4.0, "retries": 0, "seconds": 744.9, "max_memory_gb": 17.14, "ce": 0.020475899216535254, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 1.7564458847045898, "loss_scale": 4.0, "retries": 0, "seconds": 750.3, "max_memory_gb": 17.14, "ce": 0.256996418198014, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.08773884177207947, "loss_scale": 4.0, "retries": 0, "seconds": 755.8, "max_memory_gb": 17.14, "ce": 0.00412481908438167, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 1.276082158088684, "loss_scale": 4.0, "retries": 0, "seconds": 761.3, "max_memory_gb": 17.14, "ce": 0.09028744865872795, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.1863229125738144, "loss_scale": 4.0, "retries": 0, "seconds": 766.8, "max_memory_gb": 17.14, "ce": 0.013045911718108982, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 1.00006103515625, "loss_scale": 4.0, "retries": 0, "seconds": 772.2, "max_memory_gb": 17.14, "ce": 0.1459330774512182, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.40828949213027954, "loss_scale": 4.0, "retries": 0, "seconds": 777.6, "max_memory_gb": 17.14, "ce": 0.024227554841672827, "rank": 0.1253863126039505}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.10563201457262039, "loss_scale": 4.0, "retries": 0, "seconds": 783.0, "max_memory_gb": 17.14, "ce": 0.008012495021603172, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.30998942255973816, "loss_scale": 4.0, "retries": 0, "seconds": 788.5, "max_memory_gb": 17.14, "ce": 0.023795692555154346, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 0.6836053133010864, "loss_scale": 4.0, "retries": 0, "seconds": 794.0, "max_memory_gb": 17.14, "ce": 0.05084874333942935, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 1.9945063591003418, "loss_scale": 4.0, "retries": 0, "seconds": 799.4, "max_memory_gb": 17.14, "ce": 0.28070282524765844, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 2.3009121417999268, "loss_scale": 4.0, "retries": 0, "seconds": 804.7, "max_memory_gb": 17.14, "ce": 0.28262556002755446, "rank": 0.0}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.6442495584487915, "loss_scale": 4.0, "retries": 0, "seconds": 810.1, "max_memory_gb": 17.14, "ce": 0.04343151966503456, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.032222356647253036, "loss_scale": 4.0, "retries": 0, "seconds": 815.0, "max_memory_gb": 17.14, "ce": 0.001940471425314172, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 2.5299482345581055, "loss_scale": 4.0, "retries": 0, "seconds": 819.9, "max_memory_gb": 17.14, "ce": 0.13002835389261236, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.2981431782245636, "loss_scale": 4.0, "retries": 0, "seconds": 824.7, "max_memory_gb": 17.14, "ce": 0.012151501348171223, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.09573808312416077, "loss_scale": 4.0, "retries": 0, "seconds": 829.6, "max_memory_gb": 17.14, "ce": 0.006604339928344416, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 0.419095516204834, "loss_scale": 4.0, "retries": 0, "seconds": 834.5, "max_memory_gb": 17.14, "ce": 0.10581685615647984, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.46352076530456543, "loss_scale": 4.0, "retries": 0, "seconds": 839.4, "max_memory_gb": 17.14, "ce": 0.05576528697361027, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.3320927023887634, "loss_scale": 4.0, "retries": 0, "seconds": 844.3, "max_memory_gb": 17.14, "ce": 0.007625814881862425, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 0.6443742513656616, "loss_scale": 4.0, "retries": 0, "seconds": 849.1, "max_memory_gb": 17.14, "ce": 0.07225643170340845, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.11231677979230881, "loss_scale": 4.0, "retries": 0, "seconds": 853.9, "max_memory_gb": 17.14, "ce": 0.006760539880360739, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 4.689795017242432, "loss_scale": 4.0, "retries": 0, "seconds": 858.8, "max_memory_gb": 17.14, "ce": 0.24831867214777503, "rank": 0.10890889167785645}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 1.6848342418670654, "loss_scale": 4.0, "retries": 0, "seconds": 863.7, "max_memory_gb": 17.14, "ce": 0.10901366027792392, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 1.4634584188461304, "loss_scale": 4.0, "retries": 0, "seconds": 868.5, "max_memory_gb": 17.14, "ce": 0.0461169028067161, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 0.3192816972732544, "loss_scale": 4.0, "retries": 0, "seconds": 873.4, "max_memory_gb": 17.14, "ce": 0.057642157995928756, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 1.1261268854141235, "loss_scale": 4.0, "retries": 0, "seconds": 878.3, "max_memory_gb": 17.14, "ce": 0.15688396106202163, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.09170681238174438, "loss_scale": 4.0, "retries": 0, "seconds": 883.1, "max_memory_gb": 17.14, "ce": 0.005414501378709247, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 0.2873830795288086, "loss_scale": 4.0, "retries": 0, "seconds": 888.0, "max_memory_gb": 17.14, "ce": 0.018033265992585257, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.028778590261936188, "loss_scale": 4.0, "retries": 0, "seconds": 892.9, "max_memory_gb": 17.14, "ce": 0.0015909367757558357, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.433971107006073, "loss_scale": 4.0, "retries": 0, "seconds": 897.7, "max_memory_gb": 17.14, "ce": 0.02741175629580539, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 1.4120616912841797, "loss_scale": 4.0, "retries": 0, "seconds": 902.6, "max_memory_gb": 17.14, "ce": 0.0941864384319615, "rank": 0.24934814125299454}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.4822242259979248, "loss_scale": 4.0, "retries": 0, "seconds": 907.4, "max_memory_gb": 17.14, "ce": 0.05907113982812007, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 1.1876294612884521, "loss_scale": 4.0, "retries": 0, "seconds": 912.3, "max_memory_gb": 17.14, "ce": 0.06403262703383916, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 1.56269371509552, "loss_scale": 4.0, "retries": 0, "seconds": 917.2, "max_memory_gb": 17.14, "ce": 0.15114240311936555, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.6444684267044067, "loss_scale": 4.0, "retries": 0, "seconds": 922.0, "max_memory_gb": 17.14, "ce": 0.08414648578502693, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 2.4695677757263184, "loss_scale": 4.0, "retries": 0, "seconds": 926.9, "max_memory_gb": 17.14, "ce": 0.09034853252319408, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 0.2583989202976227, "loss_scale": 4.0, "retries": 0, "seconds": 931.7, "max_memory_gb": 17.14, "ce": 0.014090146950820781, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.45434945821762085, "loss_scale": 4.0, "retries": 0, "seconds": 936.6, "max_memory_gb": 17.14, "ce": 0.07124038653705611, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 2.7819371223449707, "loss_scale": 4.0, "retries": 0, "seconds": 941.5, "max_memory_gb": 17.14, "ce": 0.1673228907675366, "rank": 0.022364318370819092}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 0.7162186503410339, "loss_scale": 4.0, "retries": 0, "seconds": 946.3, "max_memory_gb": 17.14, "ce": 0.03979621040616621, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 0.6393552422523499, "loss_scale": 4.0, "retries": 0, "seconds": 951.1, "max_memory_gb": 17.14, "ce": 0.07658758723118808, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 1.1979602575302124, "loss_scale": 4.0, "retries": 0, "seconds": 956.6, "max_memory_gb": 17.14, "ce": 0.20900187507845658, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 2.5396108627319336, "loss_scale": 4.0, "retries": 0, "seconds": 961.4, "max_memory_gb": 17.14, "ce": 0.26333469577548385, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 1.5592507123947144, "loss_scale": 4.0, "retries": 0, "seconds": 966.3, "max_memory_gb": 17.14, "ce": 0.1566411592973509, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.03136974945664406, "loss_scale": 4.0, "retries": 0, "seconds": 971.0, "max_memory_gb": 17.14, "ce": 0.0013302231077432225, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.0057536237873137, "loss_scale": 4.0, "retries": 0, "seconds": 975.9, "max_memory_gb": 17.14, "ce": 0.00032205014315422886, "rank": 0.0}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.0072507429867982864, "loss_scale": 4.0, "retries": 0, "seconds": 980.8, "max_memory_gb": 17.14, "ce": 0.00046254492065145314, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 0.31263965368270874, "loss_scale": 4.0, "retries": 0, "seconds": 985.6, "max_memory_gb": 17.14, "ce": 0.013567620034791616, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 1.5001789331436157, "loss_scale": 4.0, "retries": 0, "seconds": 990.5, "max_memory_gb": 17.14, "ce": 0.14944791387006262, "rank": 0.0}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 0.9044526219367981, "loss_scale": 4.0, "retries": 0, "seconds": 995.2, "max_memory_gb": 17.14, "ce": 0.08053139656658459, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 0.9988430142402649, "loss_scale": 4.0, "retries": 0, "seconds": 1000.1, "max_memory_gb": 17.14, "ce": 0.08516403949772666, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.2994988262653351, "loss_scale": 4.0, "retries": 0, "seconds": 1004.9, "max_memory_gb": 17.14, "ce": 0.019681533780385507, "rank": 0.12498103827238083}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 0.01624327152967453, "loss_scale": 4.0, "retries": 0, "seconds": 1009.8, "max_memory_gb": 17.14, "ce": 0.001092905373980102, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.9296603202819824, "loss_scale": 4.0, "retries": 0, "seconds": 1014.5, "max_memory_gb": 17.14, "ce": 0.3489091747881048, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.27929380536079407, "loss_scale": 4.0, "retries": 0, "seconds": 1019.4, "max_memory_gb": 17.14, "ce": 0.015173389025449069, "rank": 0.0}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.11225489526987076, "loss_scale": 4.0, "retries": 0, "seconds": 1024.2, "max_memory_gb": 17.14, "ce": 0.004899247324829048, "rank": 0.12497728317975998}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.1019732803106308, "loss_scale": 4.0, "retries": 0, "seconds": 1029.1, "max_memory_gb": 17.14, "ce": 0.003032154982975044, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.008294079452753067, "loss_scale": 4.0, "retries": 0, "seconds": 1034.0, "max_memory_gb": 17.14, "ce": 0.0004201962819934124, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.24141459167003632, "loss_scale": 4.0, "retries": 0, "seconds": 1038.7, "max_memory_gb": 17.14, "ce": 0.008252836116753315, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.3696516752243042, "loss_scale": 4.0, "retries": 0, "seconds": 1043.6, "max_memory_gb": 17.14, "ce": 0.029111553766824727, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 1.1844156980514526, "loss_scale": 4.0, "retries": 0, "seconds": 1048.4, "max_memory_gb": 17.14, "ce": 0.1532747352290471, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 0.668267011642456, "loss_scale": 4.0, "retries": 0, "seconds": 1053.3, "max_memory_gb": 17.14, "ce": 0.09999682663487874, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.3237141966819763, "loss_scale": 4.0, "retries": 0, "seconds": 1058.2, "max_memory_gb": 17.14, "ce": 0.0125570410268665, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 1.111505150794983, "loss_scale": 4.0, "retries": 0, "seconds": 1062.9, "max_memory_gb": 17.14, "ce": 0.30427329224062305, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.04415366053581238, "loss_scale": 4.0, "retries": 0, "seconds": 1067.8, "max_memory_gb": 17.14, "ce": 0.002512835553716286, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.5460375547409058, "loss_scale": 4.0, "retries": 0, "seconds": 1072.6, "max_memory_gb": 17.14, "ce": 0.05894227780208894, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 0.11338159441947937, "loss_scale": 4.0, "retries": 0, "seconds": 1077.4, "max_memory_gb": 17.14, "ce": 0.0055738960802500515, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.4756585955619812, "loss_scale": 4.0, "retries": 0, "seconds": 1082.2, "max_memory_gb": 17.14, "ce": 0.02076823505876746, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.1371569186449051, "loss_scale": 4.0, "retries": 0, "seconds": 1087.1, "max_memory_gb": 17.14, "ce": 0.005357298858825743, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.33092910051345825, "loss_scale": 4.0, "retries": 0, "seconds": 1092.0, "max_memory_gb": 17.14, "ce": 0.021578828455858456, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.5477721095085144, "loss_scale": 4.0, "retries": 0, "seconds": 1096.8, "max_memory_gb": 17.14, "ce": 0.024450943461943098, "rank": 0.0}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 0.5111315846443176, "loss_scale": 4.0, "retries": 0, "seconds": 1101.7, "max_memory_gb": 17.14, "ce": 0.039904197741861935, "rank": 0.0}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.2572185695171356, "loss_scale": 4.0, "retries": 0, "seconds": 1106.5, "max_memory_gb": 17.14, "ce": 0.012100386326665102, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 1.1237878799438477, "loss_scale": 4.0, "retries": 0, "seconds": 1111.3, "max_memory_gb": 17.14, "ce": 0.21092248719537565, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 1.3263263702392578, "loss_scale": 4.0, "retries": 0, "seconds": 1116.2, "max_memory_gb": 17.14, "ce": 0.49868763931249305, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 0.12506094574928284, "loss_scale": 4.0, "retries": 0, "seconds": 1121.0, "max_memory_gb": 17.14, "ce": 0.009873044435835254, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 0.3187040090560913, "loss_scale": 4.0, "retries": 0, "seconds": 1125.8, "max_memory_gb": 17.14, "ce": 0.010846009332226458, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 1.101452350616455, "loss_scale": 4.0, "retries": 0, "seconds": 1130.6, "max_memory_gb": 17.14, "ce": 0.046318766423155466, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.8381028771400452, "loss_scale": 4.0, "retries": 0, "seconds": 1135.5, "max_memory_gb": 17.14, "ce": 0.06947145829803958, "rank": 0.12451799213886261}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 2.226257085800171, "loss_scale": 4.0, "retries": 0, "seconds": 1140.3, "max_memory_gb": 17.14, "ce": 0.3058172401383672, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.23737499117851257, "loss_scale": 4.0, "retries": 0, "seconds": 1145.2, "max_memory_gb": 17.14, "ce": 0.01399123087389853, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.9210057258605957, "loss_scale": 4.0, "retries": 0, "seconds": 1150.1, "max_memory_gb": 17.14, "ce": 0.10248041169666067, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 2.5246777534484863, "loss_scale": 4.0, "retries": 0, "seconds": 1154.9, "max_memory_gb": 17.14, "ce": 0.3027439430808272, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.595944881439209, "loss_scale": 4.0, "retries": 0, "seconds": 1159.6, "max_memory_gb": 17.14, "ce": 0.02433308165655035, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 1.8808822631835938, "loss_scale": 4.0, "retries": 0, "seconds": 1164.5, "max_memory_gb": 17.14, "ce": 0.08200227803217786, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.008916834369301796, "loss_scale": 4.0, "retries": 0, "seconds": 1169.3, "max_memory_gb": 17.14, "ce": 0.00039020876329232124, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 1.2541391849517822, "loss_scale": 4.0, "retries": 0, "seconds": 1174.2, "max_memory_gb": 17.14, "ce": 0.26646489678580565, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 1.7149924039840698, "loss_scale": 4.0, "retries": 0, "seconds": 1179.0, "max_memory_gb": 17.14, "ce": 0.15204541178707132, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.11756545305252075, "loss_scale": 4.0, "retries": 0, "seconds": 1183.8, "max_memory_gb": 17.14, "ce": 0.007956748937658631, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 1.082209587097168, "loss_scale": 4.0, "retries": 0, "seconds": 1188.6, "max_memory_gb": 17.14, "ce": 0.07698289096583721, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 1.310976505279541, "loss_scale": 4.0, "retries": 0, "seconds": 1193.4, "max_memory_gb": 17.14, "ce": 0.371190938898053, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 0.17697812616825104, "loss_scale": 4.0, "retries": 0, "seconds": 1198.3, "max_memory_gb": 17.14, "ce": 0.011774138112741639, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 1.839921236038208, "loss_scale": 4.0, "retries": 0, "seconds": 1203.1, "max_memory_gb": 17.14, "ce": 0.1273026013495837, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.36906614899635315, "loss_scale": 4.0, "retries": 0, "seconds": 1207.8, "max_memory_gb": 17.14, "ce": 0.007059409862904431, "rank": 0.0651615709066391}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.2704868018627167, "loss_scale": 4.0, "retries": 0, "seconds": 1212.7, "max_memory_gb": 17.14, "ce": 0.027910804728662697, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.7295812964439392, "loss_scale": 4.0, "retries": 0, "seconds": 1217.5, "max_memory_gb": 17.14, "ce": 0.031174562987871468, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 4.2051472663879395, "loss_scale": 4.0, "retries": 0, "seconds": 1222.4, "max_memory_gb": 17.14, "ce": 0.42039032362367834, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.49727484583854675, "loss_scale": 4.0, "retries": 0, "seconds": 1227.2, "max_memory_gb": 17.14, "ce": 0.06918558849542933, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.8947786688804626, "loss_scale": 4.0, "retries": 0, "seconds": 1231.9, "max_memory_gb": 17.14, "ce": 0.1604073237404009, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 3.813605308532715, "loss_scale": 4.0, "retries": 0, "seconds": 1236.8, "max_memory_gb": 17.14, "ce": 0.27345942641272813, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.3747487962245941, "loss_scale": 4.0, "retries": 0, "seconds": 1241.6, "max_memory_gb": 17.14, "ce": 0.017698786804885458, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 2.5974044799804688, "loss_scale": 4.0, "retries": 0, "seconds": 1246.4, "max_memory_gb": 17.14, "ce": 0.36859964815141666, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.2591002881526947, "loss_scale": 4.0, "retries": 0, "seconds": 1251.3, "max_memory_gb": 17.14, "ce": 0.02053975424132659, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 1.699869990348816, "loss_scale": 4.0, "retries": 0, "seconds": 1256.0, "max_memory_gb": 17.14, "ce": 0.2624414300473745, "rank": 0.12473256886005402}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 1.6885302066802979, "loss_scale": 4.0, "retries": 0, "seconds": 1260.9, "max_memory_gb": 17.14, "ce": 0.1717947721051587, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.7719675302505493, "loss_scale": 4.0, "retries": 0, "seconds": 1265.7, "max_memory_gb": 17.14, "ce": 0.057991446229493704, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 0.8824498057365417, "loss_scale": 4.0, "retries": 0, "seconds": 1270.4, "max_memory_gb": 17.14, "ce": 0.15597226682507426, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.6455137133598328, "loss_scale": 4.0, "retries": 0, "seconds": 1275.3, "max_memory_gb": 17.14, "ce": 0.06562039962352628, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.08337759971618652, "loss_scale": 4.0, "retries": 0, "seconds": 1280.1, "max_memory_gb": 17.14, "ce": 0.003566333958588075, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.09501033276319504, "loss_scale": 4.0, "retries": 0, "seconds": 1285.0, "max_memory_gb": 17.14, "ce": 0.003069127528988247, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.4675388038158417, "loss_scale": 4.0, "retries": 0, "seconds": 1289.8, "max_memory_gb": 17.14, "ce": 0.044160388502632486, "rank": 0.11788623034954071}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.16017687320709229, "loss_scale": 4.0, "retries": 0, "seconds": 1294.6, "max_memory_gb": 17.14, "ce": 0.006239204798021092, "rank": 0.11098869889974594}

runs/fixed_internvl/removal/Full_s73/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/removal/Full_s73/adapter/adapter_config.json

@@ -0,0 +1,556 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.0.mlp.down_proj": 32,
+    "model.language_model.layers.0.mlp.gate_proj": 32,
+    "model.language_model.layers.0.mlp.up_proj": 32,
+    "model.language_model.layers.0.self_attn.k_proj": 32,
+    "model.language_model.layers.0.self_attn.o_proj": 32,
+    "model.language_model.layers.0.self_attn.q_proj": 32,
+    "model.language_model.layers.0.self_attn.v_proj": 32,
+    "model.language_model.layers.1.mlp.down_proj": 32,
+    "model.language_model.layers.1.mlp.gate_proj": 32,
+    "model.language_model.layers.1.mlp.up_proj": 32,
+    "model.language_model.layers.1.self_attn.k_proj": 32,
+    "model.language_model.layers.1.self_attn.o_proj": 32,
+    "model.language_model.layers.1.self_attn.q_proj": 32,
+    "model.language_model.layers.1.self_attn.v_proj": 32,
+    "model.language_model.layers.10.mlp.down_proj": 32,
+    "model.language_model.layers.10.mlp.gate_proj": 32,
+    "model.language_model.layers.10.mlp.up_proj": 32,
+    "model.language_model.layers.10.self_attn.k_proj": 32,
+    "model.language_model.layers.10.self_attn.o_proj": 32,
+    "model.language_model.layers.10.self_attn.q_proj": 32,
+    "model.language_model.layers.10.self_attn.v_proj": 32,
+    "model.language_model.layers.11.mlp.down_proj": 32,
+    "model.language_model.layers.11.mlp.gate_proj": 32,
+    "model.language_model.layers.11.mlp.up_proj": 32,
+    "model.language_model.layers.11.self_attn.k_proj": 32,
+    "model.language_model.layers.11.self_attn.o_proj": 32,
+    "model.language_model.layers.11.self_attn.q_proj": 32,
+    "model.language_model.layers.11.self_attn.v_proj": 32,
+    "model.language_model.layers.12.mlp.down_proj": 32,
+    "model.language_model.layers.12.mlp.gate_proj": 32,
+    "model.language_model.layers.12.mlp.up_proj": 32,
+    "model.language_model.layers.12.self_attn.k_proj": 32,
+    "model.language_model.layers.12.self_attn.o_proj": 32,
+    "model.language_model.layers.12.self_attn.q_proj": 32,
+    "model.language_model.layers.12.self_attn.v_proj": 32,
+    "model.language_model.layers.13.mlp.down_proj": 32,
+    "model.language_model.layers.13.mlp.gate_proj": 32,
+    "model.language_model.layers.13.mlp.up_proj": 32,
+    "model.language_model.layers.13.self_attn.k_proj": 32,
+    "model.language_model.layers.13.self_attn.o_proj": 32,
+    "model.language_model.layers.13.self_attn.q_proj": 32,
+    "model.language_model.layers.13.self_attn.v_proj": 32,
+    "model.language_model.layers.14.mlp.down_proj": 32,
+    "model.language_model.layers.14.mlp.gate_proj": 32,
+    "model.language_model.layers.14.mlp.up_proj": 32,
+    "model.language_model.layers.14.self_attn.k_proj": 32,
+    "model.language_model.layers.14.self_attn.o_proj": 32,
+    "model.language_model.layers.14.self_attn.q_proj": 32,
+    "model.language_model.layers.14.self_attn.v_proj": 32,
+    "model.language_model.layers.15.mlp.down_proj": 32,
+    "model.language_model.layers.15.mlp.gate_proj": 32,
+    "model.language_model.layers.15.mlp.up_proj": 32,
+    "model.language_model.layers.15.self_attn.k_proj": 32,
+    "model.language_model.layers.15.self_attn.o_proj": 32,
+    "model.language_model.layers.15.self_attn.q_proj": 32,
+    "model.language_model.layers.15.self_attn.v_proj": 32,
+    "model.language_model.layers.16.mlp.down_proj": 32,
+    "model.language_model.layers.16.mlp.gate_proj": 32,
+    "model.language_model.layers.16.mlp.up_proj": 32,
+    "model.language_model.layers.16.self_attn.k_proj": 32,
+    "model.language_model.layers.16.self_attn.o_proj": 32,
+    "model.language_model.layers.16.self_attn.q_proj": 32,
+    "model.language_model.layers.16.self_attn.v_proj": 32,
+    "model.language_model.layers.17.mlp.down_proj": 32,
+    "model.language_model.layers.17.mlp.gate_proj": 32,
+    "model.language_model.layers.17.mlp.up_proj": 32,
+    "model.language_model.layers.17.self_attn.k_proj": 32,
+    "model.language_model.layers.17.self_attn.o_proj": 32,
+    "model.language_model.layers.17.self_attn.q_proj": 32,
+    "model.language_model.layers.17.self_attn.v_proj": 32,
+    "model.language_model.layers.18.mlp.down_proj": 32,
+    "model.language_model.layers.18.mlp.gate_proj": 32,
+    "model.language_model.layers.18.mlp.up_proj": 32,
+    "model.language_model.layers.18.self_attn.k_proj": 32,
+    "model.language_model.layers.18.self_attn.o_proj": 32,
+    "model.language_model.layers.18.self_attn.q_proj": 32,
+    "model.language_model.layers.18.self_attn.v_proj": 32,
+    "model.language_model.layers.19.mlp.down_proj": 32,
+    "model.language_model.layers.19.mlp.gate_proj": 32,
+    "model.language_model.layers.19.mlp.up_proj": 32,
+    "model.language_model.layers.19.self_attn.k_proj": 32,
+    "model.language_model.layers.19.self_attn.o_proj": 32,
+    "model.language_model.layers.19.self_attn.q_proj": 32,
+    "model.language_model.layers.19.self_attn.v_proj": 32,
+    "model.language_model.layers.2.mlp.down_proj": 32,
+    "model.language_model.layers.2.mlp.gate_proj": 32,
+    "model.language_model.layers.2.mlp.up_proj": 32,
+    "model.language_model.layers.2.self_attn.k_proj": 32,
+    "model.language_model.layers.2.self_attn.o_proj": 32,
+    "model.language_model.layers.2.self_attn.q_proj": 32,
+    "model.language_model.layers.2.self_attn.v_proj": 32,
+    "model.language_model.layers.20.mlp.down_proj": 32,
+    "model.language_model.layers.20.mlp.gate_proj": 32,
+    "model.language_model.layers.20.mlp.up_proj": 32,
+    "model.language_model.layers.20.self_attn.k_proj": 32,
+    "model.language_model.layers.20.self_attn.o_proj": 32,
+    "model.language_model.layers.20.self_attn.q_proj": 32,
+    "model.language_model.layers.20.self_attn.v_proj": 32,
+    "model.language_model.layers.21.mlp.down_proj": 32,
+    "model.language_model.layers.21.mlp.gate_proj": 32,
+    "model.language_model.layers.21.mlp.up_proj": 32,
+    "model.language_model.layers.21.self_attn.k_proj": 32,
+    "model.language_model.layers.21.self_attn.o_proj": 32,
+    "model.language_model.layers.21.self_attn.q_proj": 32,
+    "model.language_model.layers.21.self_attn.v_proj": 32,
+    "model.language_model.layers.22.mlp.down_proj": 32,
+    "model.language_model.layers.22.mlp.gate_proj": 32,
+    "model.language_model.layers.22.mlp.up_proj": 32,
+    "model.language_model.layers.22.self_attn.k_proj": 32,
+    "model.language_model.layers.22.self_attn.o_proj": 32,
+    "model.language_model.layers.22.self_attn.q_proj": 32,
+    "model.language_model.layers.22.self_attn.v_proj": 32,
+    "model.language_model.layers.23.mlp.down_proj": 32,
+    "model.language_model.layers.23.mlp.gate_proj": 32,
+    "model.language_model.layers.23.mlp.up_proj": 32,
+    "model.language_model.layers.23.self_attn.k_proj": 32,
+    "model.language_model.layers.23.self_attn.o_proj": 32,
+    "model.language_model.layers.23.self_attn.q_proj": 32,
+    "model.language_model.layers.23.self_attn.v_proj": 32,
+    "model.language_model.layers.24.mlp.down_proj": 32,
+    "model.language_model.layers.24.mlp.gate_proj": 32,
+    "model.language_model.layers.24.mlp.up_proj": 32,
+    "model.language_model.layers.24.self_attn.k_proj": 32,
+    "model.language_model.layers.24.self_attn.o_proj": 32,
+    "model.language_model.layers.24.self_attn.q_proj": 32,
+    "model.language_model.layers.24.self_attn.v_proj": 32,
+    "model.language_model.layers.25.mlp.down_proj": 32,
+    "model.language_model.layers.25.mlp.gate_proj": 32,
+    "model.language_model.layers.25.mlp.up_proj": 32,
+    "model.language_model.layers.25.self_attn.k_proj": 32,
+    "model.language_model.layers.25.self_attn.o_proj": 32,
+    "model.language_model.layers.25.self_attn.q_proj": 32,
+    "model.language_model.layers.25.self_attn.v_proj": 32,
+    "model.language_model.layers.26.mlp.down_proj": 32,
+    "model.language_model.layers.26.mlp.gate_proj": 32,
+    "model.language_model.layers.26.mlp.up_proj": 32,
+    "model.language_model.layers.26.self_attn.k_proj": 32,
+    "model.language_model.layers.26.self_attn.o_proj": 32,
+    "model.language_model.layers.26.self_attn.q_proj": 32,
+    "model.language_model.layers.26.self_attn.v_proj": 32,
+    "model.language_model.layers.27.mlp.down_proj": 32,
+    "model.language_model.layers.27.mlp.gate_proj": 32,
+    "model.language_model.layers.27.mlp.up_proj": 32,
+    "model.language_model.layers.27.self_attn.k_proj": 32,
+    "model.language_model.layers.27.self_attn.o_proj": 32,
+    "model.language_model.layers.27.self_attn.q_proj": 32,
+    "model.language_model.layers.27.self_attn.v_proj": 32,
+    "model.language_model.layers.28.mlp.down_proj": 32,
+    "model.language_model.layers.28.mlp.gate_proj": 32,
+    "model.language_model.layers.28.mlp.up_proj": 32,
+    "model.language_model.layers.28.self_attn.k_proj": 32,
+    "model.language_model.layers.28.self_attn.o_proj": 32,
+    "model.language_model.layers.28.self_attn.q_proj": 32,
+    "model.language_model.layers.28.self_attn.v_proj": 32,
+    "model.language_model.layers.29.mlp.down_proj": 32,
+    "model.language_model.layers.29.mlp.gate_proj": 32,
+    "model.language_model.layers.29.mlp.up_proj": 32,
+    "model.language_model.layers.29.self_attn.k_proj": 32,
+    "model.language_model.layers.29.self_attn.o_proj": 32,
+    "model.language_model.layers.29.self_attn.q_proj": 32,
+    "model.language_model.layers.29.self_attn.v_proj": 32,
+    "model.language_model.layers.3.mlp.down_proj": 32,
+    "model.language_model.layers.3.mlp.gate_proj": 32,
+    "model.language_model.layers.3.mlp.up_proj": 32,
+    "model.language_model.layers.3.self_attn.k_proj": 32,
+    "model.language_model.layers.3.self_attn.o_proj": 32,
+    "model.language_model.layers.3.self_attn.q_proj": 32,
+    "model.language_model.layers.3.self_attn.v_proj": 32,
+    "model.language_model.layers.30.mlp.down_proj": 32,
+    "model.language_model.layers.30.mlp.gate_proj": 32,
+    "model.language_model.layers.30.mlp.up_proj": 32,
+    "model.language_model.layers.30.self_attn.k_proj": 32,
+    "model.language_model.layers.30.self_attn.o_proj": 32,
+    "model.language_model.layers.30.self_attn.q_proj": 32,
+    "model.language_model.layers.30.self_attn.v_proj": 32,
+    "model.language_model.layers.31.mlp.down_proj": 32,
+    "model.language_model.layers.31.mlp.gate_proj": 32,
+    "model.language_model.layers.31.mlp.up_proj": 32,
+    "model.language_model.layers.31.self_attn.k_proj": 32,
+    "model.language_model.layers.31.self_attn.o_proj": 32,
+    "model.language_model.layers.31.self_attn.q_proj": 32,
+    "model.language_model.layers.31.self_attn.v_proj": 32,
+    "model.language_model.layers.32.mlp.down_proj": 32,
+    "model.language_model.layers.32.mlp.gate_proj": 32,
+    "model.language_model.layers.32.mlp.up_proj": 32,
+    "model.language_model.layers.32.self_attn.k_proj": 32,
+    "model.language_model.layers.32.self_attn.o_proj": 32,
+    "model.language_model.layers.32.self_attn.q_proj": 32,
+    "model.language_model.layers.32.self_attn.v_proj": 32,
+    "model.language_model.layers.33.mlp.down_proj": 32,
+    "model.language_model.layers.33.mlp.gate_proj": 32,
+    "model.language_model.layers.33.mlp.up_proj": 32,
+    "model.language_model.layers.33.self_attn.k_proj": 32,
+    "model.language_model.layers.33.self_attn.o_proj": 32,
+    "model.language_model.layers.33.self_attn.q_proj": 32,
+    "model.language_model.layers.33.self_attn.v_proj": 32,
+    "model.language_model.layers.34.mlp.down_proj": 32,
+    "model.language_model.layers.34.mlp.gate_proj": 32,
+    "model.language_model.layers.34.mlp.up_proj": 32,
+    "model.language_model.layers.34.self_attn.k_proj": 32,
+    "model.language_model.layers.34.self_attn.o_proj": 32,
+    "model.language_model.layers.34.self_attn.q_proj": 32,
+    "model.language_model.layers.34.self_attn.v_proj": 32,
+    "model.language_model.layers.35.mlp.down_proj": 32,
+    "model.language_model.layers.35.mlp.gate_proj": 32,
+    "model.language_model.layers.35.mlp.up_proj": 32,
+    "model.language_model.layers.35.self_attn.k_proj": 32,
+    "model.language_model.layers.35.self_attn.o_proj": 32,
+    "model.language_model.layers.35.self_attn.q_proj": 32,
+    "model.language_model.layers.35.self_attn.v_proj": 32,
+    "model.language_model.layers.4.mlp.down_proj": 32,
+    "model.language_model.layers.4.mlp.gate_proj": 32,
+    "model.language_model.layers.4.mlp.up_proj": 32,
+    "model.language_model.layers.4.self_attn.k_proj": 32,
+    "model.language_model.layers.4.self_attn.o_proj": 32,
+    "model.language_model.layers.4.self_attn.q_proj": 32,
+    "model.language_model.layers.4.self_attn.v_proj": 32,
+    "model.language_model.layers.5.mlp.down_proj": 32,
+    "model.language_model.layers.5.mlp.gate_proj": 32,
+    "model.language_model.layers.5.mlp.up_proj": 32,
+    "model.language_model.layers.5.self_attn.k_proj": 32,
+    "model.language_model.layers.5.self_attn.o_proj": 32,
+    "model.language_model.layers.5.self_attn.q_proj": 32,
+    "model.language_model.layers.5.self_attn.v_proj": 32,
+    "model.language_model.layers.6.mlp.down_proj": 32,
+    "model.language_model.layers.6.mlp.gate_proj": 32,
+    "model.language_model.layers.6.mlp.up_proj": 32,
+    "model.language_model.layers.6.self_attn.k_proj": 32,
+    "model.language_model.layers.6.self_attn.o_proj": 32,
+    "model.language_model.layers.6.self_attn.q_proj": 32,
+    "model.language_model.layers.6.self_attn.v_proj": 32,
+    "model.language_model.layers.7.mlp.down_proj": 32,
+    "model.language_model.layers.7.mlp.gate_proj": 32,
+    "model.language_model.layers.7.mlp.up_proj": 32,
+    "model.language_model.layers.7.self_attn.k_proj": 32,
+    "model.language_model.layers.7.self_attn.o_proj": 32,
+    "model.language_model.layers.7.self_attn.q_proj": 32,
+    "model.language_model.layers.7.self_attn.v_proj": 32,
+    "model.language_model.layers.8.mlp.down_proj": 32,
+    "model.language_model.layers.8.mlp.gate_proj": 32,
+    "model.language_model.layers.8.mlp.up_proj": 32,
+    "model.language_model.layers.8.self_attn.k_proj": 32,
+    "model.language_model.layers.8.self_attn.o_proj": 32,
+    "model.language_model.layers.8.self_attn.q_proj": 32,
+    "model.language_model.layers.8.self_attn.v_proj": 32,
+    "model.language_model.layers.9.mlp.down_proj": 32,
+    "model.language_model.layers.9.mlp.gate_proj": 32,
+    "model.language_model.layers.9.mlp.up_proj": 32,
+    "model.language_model.layers.9.self_attn.k_proj": 32,
+    "model.language_model.layers.9.self_attn.o_proj": 32,
+    "model.language_model.layers.9.self_attn.q_proj": 32,
+    "model.language_model.layers.9.self_attn.v_proj": 32
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.0.mlp.down_proj": 16,
+    "model.language_model.layers.0.mlp.gate_proj": 16,
+    "model.language_model.layers.0.mlp.up_proj": 16,
+    "model.language_model.layers.0.self_attn.k_proj": 16,
+    "model.language_model.layers.0.self_attn.o_proj": 16,
+    "model.language_model.layers.0.self_attn.q_proj": 16,
+    "model.language_model.layers.0.self_attn.v_proj": 16,
+    "model.language_model.layers.1.mlp.down_proj": 16,
+    "model.language_model.layers.1.mlp.gate_proj": 16,
+    "model.language_model.layers.1.mlp.up_proj": 16,
+    "model.language_model.layers.1.self_attn.k_proj": 16,
+    "model.language_model.layers.1.self_attn.o_proj": 16,
+    "model.language_model.layers.1.self_attn.q_proj": 16,
+    "model.language_model.layers.1.self_attn.v_proj": 16,
+    "model.language_model.layers.10.mlp.down_proj": 16,
+    "model.language_model.layers.10.mlp.gate_proj": 16,
+    "model.language_model.layers.10.mlp.up_proj": 16,
+    "model.language_model.layers.10.self_attn.k_proj": 16,
+    "model.language_model.layers.10.self_attn.o_proj": 16,
+    "model.language_model.layers.10.self_attn.q_proj": 16,
+    "model.language_model.layers.10.self_attn.v_proj": 16,
+    "model.language_model.layers.11.mlp.down_proj": 16,
+    "model.language_model.layers.11.mlp.gate_proj": 16,
+    "model.language_model.layers.11.mlp.up_proj": 16,
+    "model.language_model.layers.11.self_attn.k_proj": 16,
+    "model.language_model.layers.11.self_attn.o_proj": 16,
+    "model.language_model.layers.11.self_attn.q_proj": 16,
+    "model.language_model.layers.11.self_attn.v_proj": 16,
+    "model.language_model.layers.12.mlp.down_proj": 16,
+    "model.language_model.layers.12.mlp.gate_proj": 16,
+    "model.language_model.layers.12.mlp.up_proj": 16,
+    "model.language_model.layers.12.self_attn.k_proj": 16,
+    "model.language_model.layers.12.self_attn.o_proj": 16,
+    "model.language_model.layers.12.self_attn.q_proj": 16,
+    "model.language_model.layers.12.self_attn.v_proj": 16,
+    "model.language_model.layers.13.mlp.down_proj": 16,
+    "model.language_model.layers.13.mlp.gate_proj": 16,
+    "model.language_model.layers.13.mlp.up_proj": 16,
+    "model.language_model.layers.13.self_attn.k_proj": 16,
+    "model.language_model.layers.13.self_attn.o_proj": 16,
+    "model.language_model.layers.13.self_attn.q_proj": 16,
+    "model.language_model.layers.13.self_attn.v_proj": 16,
+    "model.language_model.layers.14.mlp.down_proj": 16,
+    "model.language_model.layers.14.mlp.gate_proj": 16,
+    "model.language_model.layers.14.mlp.up_proj": 16,
+    "model.language_model.layers.14.self_attn.k_proj": 16,
+    "model.language_model.layers.14.self_attn.o_proj": 16,
+    "model.language_model.layers.14.self_attn.q_proj": 16,
+    "model.language_model.layers.14.self_attn.v_proj": 16,
+    "model.language_model.layers.15.mlp.down_proj": 16,
+    "model.language_model.layers.15.mlp.gate_proj": 16,
+    "model.language_model.layers.15.mlp.up_proj": 16,
+    "model.language_model.layers.15.self_attn.k_proj": 16,
+    "model.language_model.layers.15.self_attn.o_proj": 16,
+    "model.language_model.layers.15.self_attn.q_proj": 16,
+    "model.language_model.layers.15.self_attn.v_proj": 16,
+    "model.language_model.layers.16.mlp.down_proj": 16,
+    "model.language_model.layers.16.mlp.gate_proj": 16,
+    "model.language_model.layers.16.mlp.up_proj": 16,
+    "model.language_model.layers.16.self_attn.k_proj": 16,
+    "model.language_model.layers.16.self_attn.o_proj": 16,
+    "model.language_model.layers.16.self_attn.q_proj": 16,
+    "model.language_model.layers.16.self_attn.v_proj": 16,
+    "model.language_model.layers.17.mlp.down_proj": 16,
+    "model.language_model.layers.17.mlp.gate_proj": 16,
+    "model.language_model.layers.17.mlp.up_proj": 16,
+    "model.language_model.layers.17.self_attn.k_proj": 16,
+    "model.language_model.layers.17.self_attn.o_proj": 16,
+    "model.language_model.layers.17.self_attn.q_proj": 16,
+    "model.language_model.layers.17.self_attn.v_proj": 16,
+    "model.language_model.layers.18.mlp.down_proj": 16,
+    "model.language_model.layers.18.mlp.gate_proj": 16,
+    "model.language_model.layers.18.mlp.up_proj": 16,
+    "model.language_model.layers.18.self_attn.k_proj": 16,
+    "model.language_model.layers.18.self_attn.o_proj": 16,
+    "model.language_model.layers.18.self_attn.q_proj": 16,
+    "model.language_model.layers.18.self_attn.v_proj": 16,
+    "model.language_model.layers.19.mlp.down_proj": 16,
+    "model.language_model.layers.19.mlp.gate_proj": 16,
+    "model.language_model.layers.19.mlp.up_proj": 16,
+    "model.language_model.layers.19.self_attn.k_proj": 16,
+    "model.language_model.layers.19.self_attn.o_proj": 16,
+    "model.language_model.layers.19.self_attn.q_proj": 16,
+    "model.language_model.layers.19.self_attn.v_proj": 16,
+    "model.language_model.layers.2.mlp.down_proj": 16,
+    "model.language_model.layers.2.mlp.gate_proj": 16,
+    "model.language_model.layers.2.mlp.up_proj": 16,
+    "model.language_model.layers.2.self_attn.k_proj": 16,
+    "model.language_model.layers.2.self_attn.o_proj": 16,
+    "model.language_model.layers.2.self_attn.q_proj": 16,
+    "model.language_model.layers.2.self_attn.v_proj": 16,
+    "model.language_model.layers.20.mlp.down_proj": 16,
+    "model.language_model.layers.20.mlp.gate_proj": 16,
+    "model.language_model.layers.20.mlp.up_proj": 16,
+    "model.language_model.layers.20.self_attn.k_proj": 16,
+    "model.language_model.layers.20.self_attn.o_proj": 16,
+    "model.language_model.layers.20.self_attn.q_proj": 16,
+    "model.language_model.layers.20.self_attn.v_proj": 16,
+    "model.language_model.layers.21.mlp.down_proj": 16,
+    "model.language_model.layers.21.mlp.gate_proj": 16,
+    "model.language_model.layers.21.mlp.up_proj": 16,
+    "model.language_model.layers.21.self_attn.k_proj": 16,
+    "model.language_model.layers.21.self_attn.o_proj": 16,
+    "model.language_model.layers.21.self_attn.q_proj": 16,
+    "model.language_model.layers.21.self_attn.v_proj": 16,
+    "model.language_model.layers.22.mlp.down_proj": 16,
+    "model.language_model.layers.22.mlp.gate_proj": 16,
+    "model.language_model.layers.22.mlp.up_proj": 16,
+    "model.language_model.layers.22.self_attn.k_proj": 16,
+    "model.language_model.layers.22.self_attn.o_proj": 16,
+    "model.language_model.layers.22.self_attn.q_proj": 16,
+    "model.language_model.layers.22.self_attn.v_proj": 16,
+    "model.language_model.layers.23.mlp.down_proj": 16,
+    "model.language_model.layers.23.mlp.gate_proj": 16,
+    "model.language_model.layers.23.mlp.up_proj": 16,
+    "model.language_model.layers.23.self_attn.k_proj": 16,
+    "model.language_model.layers.23.self_attn.o_proj": 16,
+    "model.language_model.layers.23.self_attn.q_proj": 16,
+    "model.language_model.layers.23.self_attn.v_proj": 16,
+    "model.language_model.layers.24.mlp.down_proj": 16,
+    "model.language_model.layers.24.mlp.gate_proj": 16,
+    "model.language_model.layers.24.mlp.up_proj": 16,
+    "model.language_model.layers.24.self_attn.k_proj": 16,
+    "model.language_model.layers.24.self_attn.o_proj": 16,
+    "model.language_model.layers.24.self_attn.q_proj": 16,
+    "model.language_model.layers.24.self_attn.v_proj": 16,
+    "model.language_model.layers.25.mlp.down_proj": 16,
+    "model.language_model.layers.25.mlp.gate_proj": 16,
+    "model.language_model.layers.25.mlp.up_proj": 16,
+    "model.language_model.layers.25.self_attn.k_proj": 16,
+    "model.language_model.layers.25.self_attn.o_proj": 16,
+    "model.language_model.layers.25.self_attn.q_proj": 16,
+    "model.language_model.layers.25.self_attn.v_proj": 16,
+    "model.language_model.layers.26.mlp.down_proj": 16,
+    "model.language_model.layers.26.mlp.gate_proj": 16,
+    "model.language_model.layers.26.mlp.up_proj": 16,
+    "model.language_model.layers.26.self_attn.k_proj": 16,
+    "model.language_model.layers.26.self_attn.o_proj": 16,
+    "model.language_model.layers.26.self_attn.q_proj": 16,
+    "model.language_model.layers.26.self_attn.v_proj": 16,
+    "model.language_model.layers.27.mlp.down_proj": 16,
+    "model.language_model.layers.27.mlp.gate_proj": 16,
+    "model.language_model.layers.27.mlp.up_proj": 16,
+    "model.language_model.layers.27.self_attn.k_proj": 16,
+    "model.language_model.layers.27.self_attn.o_proj": 16,
+    "model.language_model.layers.27.self_attn.q_proj": 16,
+    "model.language_model.layers.27.self_attn.v_proj": 16,
+    "model.language_model.layers.28.mlp.down_proj": 16,
+    "model.language_model.layers.28.mlp.gate_proj": 16,
+    "model.language_model.layers.28.mlp.up_proj": 16,
+    "model.language_model.layers.28.self_attn.k_proj": 16,
+    "model.language_model.layers.28.self_attn.o_proj": 16,
+    "model.language_model.layers.28.self_attn.q_proj": 16,
+    "model.language_model.layers.28.self_attn.v_proj": 16,
+    "model.language_model.layers.29.mlp.down_proj": 16,
+    "model.language_model.layers.29.mlp.gate_proj": 16,
+    "model.language_model.layers.29.mlp.up_proj": 16,
+    "model.language_model.layers.29.self_attn.k_proj": 16,
+    "model.language_model.layers.29.self_attn.o_proj": 16,
+    "model.language_model.layers.29.self_attn.q_proj": 16,
+    "model.language_model.layers.29.self_attn.v_proj": 16,
+    "model.language_model.layers.3.mlp.down_proj": 16,
+    "model.language_model.layers.3.mlp.gate_proj": 16,
+    "model.language_model.layers.3.mlp.up_proj": 16,
+    "model.language_model.layers.3.self_attn.k_proj": 16,
+    "model.language_model.layers.3.self_attn.o_proj": 16,
+    "model.language_model.layers.3.self_attn.q_proj": 16,
+    "model.language_model.layers.3.self_attn.v_proj": 16,
+    "model.language_model.layers.30.mlp.down_proj": 16,
+    "model.language_model.layers.30.mlp.gate_proj": 16,
+    "model.language_model.layers.30.mlp.up_proj": 16,
+    "model.language_model.layers.30.self_attn.k_proj": 16,
+    "model.language_model.layers.30.self_attn.o_proj": 16,
+    "model.language_model.layers.30.self_attn.q_proj": 16,
+    "model.language_model.layers.30.self_attn.v_proj": 16,
+    "model.language_model.layers.31.mlp.down_proj": 16,
+    "model.language_model.layers.31.mlp.gate_proj": 16,
+    "model.language_model.layers.31.mlp.up_proj": 16,
+    "model.language_model.layers.31.self_attn.k_proj": 16,
+    "model.language_model.layers.31.self_attn.o_proj": 16,
+    "model.language_model.layers.31.self_attn.q_proj": 16,
+    "model.language_model.layers.31.self_attn.v_proj": 16,
+    "model.language_model.layers.32.mlp.down_proj": 16,
+    "model.language_model.layers.32.mlp.gate_proj": 16,
+    "model.language_model.layers.32.mlp.up_proj": 16,
+    "model.language_model.layers.32.self_attn.k_proj": 16,
+    "model.language_model.layers.32.self_attn.o_proj": 16,
+    "model.language_model.layers.32.self_attn.q_proj": 16,
+    "model.language_model.layers.32.self_attn.v_proj": 16,
+    "model.language_model.layers.33.mlp.down_proj": 16,
+    "model.language_model.layers.33.mlp.gate_proj": 16,
+    "model.language_model.layers.33.mlp.up_proj": 16,
+    "model.language_model.layers.33.self_attn.k_proj": 16,
+    "model.language_model.layers.33.self_attn.o_proj": 16,
+    "model.language_model.layers.33.self_attn.q_proj": 16,
+    "model.language_model.layers.33.self_attn.v_proj": 16,
+    "model.language_model.layers.34.mlp.down_proj": 16,
+    "model.language_model.layers.34.mlp.gate_proj": 16,
+    "model.language_model.layers.34.mlp.up_proj": 16,
+    "model.language_model.layers.34.self_attn.k_proj": 16,
+    "model.language_model.layers.34.self_attn.o_proj": 16,
+    "model.language_model.layers.34.self_attn.q_proj": 16,
+    "model.language_model.layers.34.self_attn.v_proj": 16,
+    "model.language_model.layers.35.mlp.down_proj": 16,
+    "model.language_model.layers.35.mlp.gate_proj": 16,
+    "model.language_model.layers.35.mlp.up_proj": 16,
+    "model.language_model.layers.35.self_attn.k_proj": 16,
+    "model.language_model.layers.35.self_attn.o_proj": 16,
+    "model.language_model.layers.35.self_attn.q_proj": 16,
+    "model.language_model.layers.35.self_attn.v_proj": 16,
+    "model.language_model.layers.4.mlp.down_proj": 16,
+    "model.language_model.layers.4.mlp.gate_proj": 16,
+    "model.language_model.layers.4.mlp.up_proj": 16,
+    "model.language_model.layers.4.self_attn.k_proj": 16,
+    "model.language_model.layers.4.self_attn.o_proj": 16,
+    "model.language_model.layers.4.self_attn.q_proj": 16,
+    "model.language_model.layers.4.self_attn.v_proj": 16,
+    "model.language_model.layers.5.mlp.down_proj": 16,
+    "model.language_model.layers.5.mlp.gate_proj": 16,
+    "model.language_model.layers.5.mlp.up_proj": 16,
+    "model.language_model.layers.5.self_attn.k_proj": 16,
+    "model.language_model.layers.5.self_attn.o_proj": 16,
+    "model.language_model.layers.5.self_attn.q_proj": 16,
+    "model.language_model.layers.5.self_attn.v_proj": 16,
+    "model.language_model.layers.6.mlp.down_proj": 16,
+    "model.language_model.layers.6.mlp.gate_proj": 16,
+    "model.language_model.layers.6.mlp.up_proj": 16,
+    "model.language_model.layers.6.self_attn.k_proj": 16,
+    "model.language_model.layers.6.self_attn.o_proj": 16,
+    "model.language_model.layers.6.self_attn.q_proj": 16,
+    "model.language_model.layers.6.self_attn.v_proj": 16,
+    "model.language_model.layers.7.mlp.down_proj": 16,
+    "model.language_model.layers.7.mlp.gate_proj": 16,
+    "model.language_model.layers.7.mlp.up_proj": 16,
+    "model.language_model.layers.7.self_attn.k_proj": 16,
+    "model.language_model.layers.7.self_attn.o_proj": 16,
+    "model.language_model.layers.7.self_attn.q_proj": 16,
+    "model.language_model.layers.7.self_attn.v_proj": 16,
+    "model.language_model.layers.8.mlp.down_proj": 16,
+    "model.language_model.layers.8.mlp.gate_proj": 16,
+    "model.language_model.layers.8.mlp.up_proj": 16,
+    "model.language_model.layers.8.self_attn.k_proj": 16,
+    "model.language_model.layers.8.self_attn.o_proj": 16,
+    "model.language_model.layers.8.self_attn.q_proj": 16,
+    "model.language_model.layers.8.self_attn.v_proj": 16,
+    "model.language_model.layers.9.mlp.down_proj": 16,
+    "model.language_model.layers.9.mlp.gate_proj": 16,
+    "model.language_model.layers.9.mlp.up_proj": 16,
+    "model.language_model.layers.9.self_attn.k_proj": 16,
+    "model.language_model.layers.9.self_attn.o_proj": 16,
+    "model.language_model.layers.9.self_attn.q_proj": 16,
+    "model.language_model.layers.9.self_attn.v_proj": 16
+  },
+  "revision": null,
+  "target_modules": [
+    "self_attn.q_proj",
+    "o_proj",
+    "self_attn.v_proj",
+    "self_attn.k_proj",
+    "gate_proj",
+    "down_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/fixed_internvl/removal/Full_s73/config.json

@@ -0,0 +1,99 @@
+{
+  "behavior": "removal",
+  "seed": 73,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "world_size": 1,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "arm": {
+    "arm": "Full",
+    "layers": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      10,
+      11,
+      12,
+      13,
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21,
+      22,
+      23,
+      24,
+      25,
+      26,
+      27,
+      28,
+      29,
+      30,
+      31,
+      32,
+      33,
+      34,
+      35
+    ],
+    "ranks": {
+      "0": 16,
+      "1": 16,
+      "2": 16,
+      "3": 16,
+      "4": 16,
+      "5": 16,
+      "6": 16,
+      "7": 16,
+      "8": 16,
+      "9": 16,
+      "10": 16,
+      "11": 16,
+      "12": 16,
+      "13": 16,
+      "14": 16,
+      "15": 16,
+      "16": 16,
+      "17": 16,
+      "18": 16,
+      "19": 16,
+      "20": 16,
+      "21": 16,
+      "22": 16,
+      "23": 16,
+      "24": 16,
+      "25": 16,
+      "26": 16,
+      "27": 16,
+      "28": 16,
+      "29": 16,
+      "30": 16,
+      "31": 16,
+      "32": 16,
+      "33": 16,
+      "34": 16,
+      "35": 16
+    },
+    "parameters": 43646976,
+    "budget": 43646976
+  }
+}

runs/fixed_internvl/removal/Full_s73/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 10.391112327575684, "loss_scale": 4.0, "retries": 0, "seconds": 6.8, "max_memory_gb": 16.81, "ce": 0.8941777415893739, "rank": 0.0}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 19.926008224487305, "loss_scale": 4.0, "retries": 0, "seconds": 11.8, "max_memory_gb": 17.13, "ce": 1.676835773512721, "rank": 0.013918399810791016}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 9.654364585876465, "loss_scale": 4.0, "retries": 0, "seconds": 16.9, "max_memory_gb": 17.13, "ce": 0.7270631677529309, "rank": 0.0917009562253952}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 10.18923282623291, "loss_scale": 4.0, "retries": 0, "seconds": 22.4, "max_memory_gb": 17.13, "ce": 0.7430673324415693, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 4.285770893096924, "loss_scale": 4.0, "retries": 0, "seconds": 27.8, "max_memory_gb": 17.13, "ce": 0.5431220506434329, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 2.5611155033111572, "loss_scale": 4.0, "retries": 0, "seconds": 33.3, "max_memory_gb": 17.13, "ce": 0.5015375977382064, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 0.2934437394142151, "loss_scale": 4.0, "retries": 0, "seconds": 38.7, "max_memory_gb": 17.13, "ce": 0.13600707601290196, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 1.6007730960845947, "loss_scale": 4.0, "retries": 0, "seconds": 44.1, "max_memory_gb": 17.13, "ce": 0.6199841365050816, "rank": 0.11033773422241211}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 1.623672366142273, "loss_scale": 4.0, "retries": 0, "seconds": 49.6, "max_memory_gb": 17.13, "ce": 0.3790773244109005, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 1.4346492290496826, "loss_scale": 4.0, "retries": 0, "seconds": 54.9, "max_memory_gb": 17.13, "ce": 0.3336561380747298, "rank": 0.0073717087507247925}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.592932403087616, "loss_scale": 4.0, "retries": 0, "seconds": 60.5, "max_memory_gb": 17.13, "ce": 0.30825941276270896, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 1.441508412361145, "loss_scale": 4.0, "retries": 0, "seconds": 66.0, "max_memory_gb": 17.13, "ce": 0.17842040719460783, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 1.8090556859970093, "loss_scale": 4.0, "retries": 0, "seconds": 71.4, "max_memory_gb": 17.13, "ce": 0.35074416103452677, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 1.1565991640090942, "loss_scale": 4.0, "retries": 0, "seconds": 76.9, "max_memory_gb": 17.13, "ce": 0.2593700561674268, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 3.3766531944274902, "loss_scale": 4.0, "retries": 0, "seconds": 82.2, "max_memory_gb": 17.13, "ce": 0.18497625380405225, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 3.8132920265197754, "loss_scale": 4.0, "retries": 0, "seconds": 87.7, "max_memory_gb": 17.13, "ce": 0.9304846587474458, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 1.9411778450012207, "loss_scale": 4.0, "retries": 0, "seconds": 93.2, "max_memory_gb": 17.13, "ce": 0.25784209225093946, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 0.7029021978378296, "loss_scale": 4.0, "retries": 0, "seconds": 98.9, "max_memory_gb": 17.13, "ce": 0.1610467783320928, "rank": 0.1245153471827507}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 3.7276570796966553, "loss_scale": 4.0, "retries": 0, "seconds": 104.3, "max_memory_gb": 17.13, "ce": 0.3807283078931505, "rank": 0.0}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.8722597360610962, "loss_scale": 4.0, "retries": 0, "seconds": 109.8, "max_memory_gb": 17.13, "ce": 0.286794607018237, "rank": 0.0}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.7735145688056946, "loss_scale": 4.0, "retries": 0, "seconds": 115.3, "max_memory_gb": 17.13, "ce": 0.09904459494282492, "rank": 0.0}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.9340561032295227, "loss_scale": 4.0, "retries": 0, "seconds": 120.8, "max_memory_gb": 17.13, "ce": 0.2615545258158818, "rank": 0.0}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.1987276077270508, "loss_scale": 4.0, "retries": 0, "seconds": 126.3, "max_memory_gb": 17.13, "ce": 0.1369329979061149, "rank": 0.0}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 0.8588661551475525, "loss_scale": 4.0, "retries": 0, "seconds": 131.7, "max_memory_gb": 17.13, "ce": 0.16209361201617867, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 0.8520823121070862, "loss_scale": 4.0, "retries": 0, "seconds": 137.2, "max_memory_gb": 17.13, "ce": 0.07903683405311313, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 2.0063581466674805, "loss_scale": 4.0, "retries": 0, "seconds": 142.7, "max_memory_gb": 17.13, "ce": 0.2437014665320021, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 4.501798629760742, "loss_scale": 4.0, "retries": 0, "seconds": 148.2, "max_memory_gb": 17.13, "ce": 0.15679297261522152, "rank": 0.0}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 0.37804779410362244, "loss_scale": 4.0, "retries": 0, "seconds": 153.6, "max_memory_gb": 17.13, "ce": 0.020657026892877184, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 2.5001509189605713, "loss_scale": 4.0, "retries": 0, "seconds": 159.0, "max_memory_gb": 17.13, "ce": 0.06475102774129482, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 2.9621849060058594, "loss_scale": 4.0, "retries": 0, "seconds": 164.4, "max_memory_gb": 17.13, "ce": 0.11248124852863839, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 0.904564619064331, "loss_scale": 4.0, "retries": 0, "seconds": 169.9, "max_memory_gb": 17.13, "ce": 0.08791177747480106, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 2.4147098064422607, "loss_scale": 4.0, "retries": 0, "seconds": 175.4, "max_memory_gb": 17.13, "ce": 0.18777998254518025, "rank": 0.12497752904891968}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.5538399815559387, "loss_scale": 4.0, "retries": 0, "seconds": 180.9, "max_memory_gb": 17.13, "ce": 0.02976758710155991, "rank": 0.1250648945569992}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 1.232145071029663, "loss_scale": 4.0, "retries": 0, "seconds": 186.2, "max_memory_gb": 17.13, "ce": 0.12085448769903451, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.8350602388381958, "loss_scale": 4.0, "retries": 0, "seconds": 191.7, "max_memory_gb": 17.13, "ce": 0.031259041988960234, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 1.2111707925796509, "loss_scale": 4.0, "retries": 0, "seconds": 197.2, "max_memory_gb": 17.14, "ce": 0.10627137402480002, "rank": 0.11835412681102753}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.4922720789909363, "loss_scale": 4.0, "retries": 0, "seconds": 202.7, "max_memory_gb": 17.14, "ce": 0.04422270337090595, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.17139998078346252, "loss_scale": 4.0, "retries": 0, "seconds": 208.1, "max_memory_gb": 17.14, "ce": 0.014086650170156645, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 2.7085304260253906, "loss_scale": 4.0, "retries": 0, "seconds": 213.5, "max_memory_gb": 17.14, "ce": 0.32569585929968525, "rank": 0.0}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.28019458055496216, "loss_scale": 4.0, "retries": 0, "seconds": 218.9, "max_memory_gb": 17.14, "ce": 0.016452973448735975, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.38112643361091614, "loss_scale": 4.0, "retries": 0, "seconds": 224.4, "max_memory_gb": 17.14, "ce": 0.010505238971745712, "rank": 0.10561634600162506}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 6.228343963623047, "loss_scale": 4.0, "retries": 0, "seconds": 229.9, "max_memory_gb": 17.14, "ce": 0.21753185676152498, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 3.321584939956665, "loss_scale": 4.0, "retries": 0, "seconds": 235.3, "max_memory_gb": 17.14, "ce": 0.2833238873954542, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 0.18513742089271545, "loss_scale": 4.0, "retries": 0, "seconds": 240.9, "max_memory_gb": 17.14, "ce": 0.005183828749750319, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 1.0590862035751343, "loss_scale": 4.0, "retries": 0, "seconds": 246.3, "max_memory_gb": 17.14, "ce": 0.07077210214083607, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 1.7096519470214844, "loss_scale": 4.0, "retries": 0, "seconds": 251.7, "max_memory_gb": 17.14, "ce": 0.2655944443995395, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.056533750146627426, "loss_scale": 4.0, "retries": 0, "seconds": 257.0, "max_memory_gb": 17.14, "ce": 0.0019590839769989543, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.4602523744106293, "loss_scale": 4.0, "retries": 0, "seconds": 262.4, "max_memory_gb": 17.14, "ce": 0.02023502785709752, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.04057519510388374, "loss_scale": 4.0, "retries": 0, "seconds": 267.8, "max_memory_gb": 17.14, "ce": 0.0024090358238026965, "rank": 0.12516599893569946}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.03395649790763855, "loss_scale": 4.0, "retries": 0, "seconds": 273.3, "max_memory_gb": 17.14, "ce": 0.0010452379956404911, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 0.9662983417510986, "loss_scale": 4.0, "retries": 0, "seconds": 278.7, "max_memory_gb": 17.14, "ce": 0.033404710465674725, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 5.165358543395996, "loss_scale": 4.0, "retries": 0, "seconds": 284.0, "max_memory_gb": 17.14, "ce": 0.33484570369910216, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 2.923396587371826, "loss_scale": 4.0, "retries": 0, "seconds": 289.4, "max_memory_gb": 17.14, "ce": 0.3730494087465104, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 1.7368097305297852, "loss_scale": 4.0, "retries": 0, "seconds": 294.8, "max_memory_gb": 17.14, "ce": 0.3016150931434822, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 1.101285696029663, "loss_scale": 4.0, "retries": 0, "seconds": 300.3, "max_memory_gb": 17.14, "ce": 0.02402084692766948, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.5570195913314819, "loss_scale": 4.0, "retries": 0, "seconds": 305.7, "max_memory_gb": 17.14, "ce": 0.03387808571505957, "rank": 0.0}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 1.3858287334442139, "loss_scale": 4.0, "retries": 0, "seconds": 311.1, "max_memory_gb": 17.14, "ce": 0.1402413873538535, "rank": 0.05189911276102066}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.6247104406356812, "loss_scale": 4.0, "retries": 0, "seconds": 316.5, "max_memory_gb": 17.14, "ce": 0.024773960744369106, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.30167192220687866, "loss_scale": 4.0, "retries": 0, "seconds": 322.0, "max_memory_gb": 17.14, "ce": 0.008959718097230507, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.20846153795719147, "loss_scale": 4.0, "retries": 0, "seconds": 327.5, "max_memory_gb": 17.14, "ce": 0.006019975240633357, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 3.0673954486846924, "loss_scale": 4.0, "retries": 0, "seconds": 332.3, "max_memory_gb": 17.14, "ce": 0.10981140522380883, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 3.0091004371643066, "loss_scale": 4.0, "retries": 0, "seconds": 337.2, "max_memory_gb": 17.14, "ce": 0.3334167602938578, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.12236333638429642, "loss_scale": 4.0, "retries": 0, "seconds": 342.2, "max_memory_gb": 17.14, "ce": 0.010091316110447224, "rank": 0.1250346302986145}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 2.503911018371582, "loss_scale": 4.0, "retries": 0, "seconds": 347.2, "max_memory_gb": 17.14, "ce": 0.22749070618647238, "rank": 0.0}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 0.3398677110671997, "loss_scale": 4.0, "retries": 0, "seconds": 352.1, "max_memory_gb": 17.14, "ce": 0.010853199822577153, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.06810019165277481, "loss_scale": 4.0, "retries": 0, "seconds": 357.0, "max_memory_gb": 17.14, "ce": 0.005054086947211545, "rank": 0.0}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 2.828094244003296, "loss_scale": 4.0, "retries": 0, "seconds": 361.9, "max_memory_gb": 17.14, "ce": 0.37348877300337335, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.302779883146286, "loss_scale": 4.0, "retries": 0, "seconds": 366.9, "max_memory_gb": 17.14, "ce": 0.011240698350547973, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.4726927578449249, "loss_scale": 4.0, "retries": 0, "seconds": 371.8, "max_memory_gb": 17.14, "ce": 0.053396792971170726, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.28200817108154297, "loss_scale": 4.0, "retries": 0, "seconds": 376.6, "max_memory_gb": 17.14, "ce": 0.014038664074632834, "rank": 0.0}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.3020313084125519, "loss_scale": 4.0, "retries": 0, "seconds": 381.6, "max_memory_gb": 17.14, "ce": 0.013048438188889122, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.1344814896583557, "loss_scale": 4.0, "retries": 0, "seconds": 386.5, "max_memory_gb": 17.14, "ce": 0.006820434568226119, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 1.5303210020065308, "loss_scale": 4.0, "retries": 0, "seconds": 391.4, "max_memory_gb": 17.14, "ce": 0.18187379363507716, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.6091917157173157, "loss_scale": 4.0, "retries": 0, "seconds": 396.4, "max_memory_gb": 17.14, "ce": 0.03212062219654399, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.14653556048870087, "loss_scale": 4.0, "retries": 0, "seconds": 401.2, "max_memory_gb": 17.14, "ce": 0.009731605188790127, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 1.3949095010757446, "loss_scale": 4.0, "retries": 0, "seconds": 406.3, "max_memory_gb": 17.14, "ce": 0.4260299841109827, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.05108525604009628, "loss_scale": 4.0, "retries": 0, "seconds": 411.8, "max_memory_gb": 17.14, "ce": 0.002469674901817598, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 1.0948630571365356, "loss_scale": 4.0, "retries": 0, "seconds": 417.2, "max_memory_gb": 17.14, "ce": 0.24522445024035733, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.4195149540901184, "loss_scale": 4.0, "retries": 0, "seconds": 422.5, "max_memory_gb": 17.14, "ce": 0.02108143368650417, "rank": 0.0}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 1.7863523960113525, "loss_scale": 4.0, "retries": 0, "seconds": 428.0, "max_memory_gb": 17.14, "ce": 0.22326217187423936, "rank": 0.12493602186441422}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 2.0361013412475586, "loss_scale": 4.0, "retries": 0, "seconds": 433.5, "max_memory_gb": 17.14, "ce": 0.130062302202532, "rank": 0.12475918233394623}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.11156850308179855, "loss_scale": 4.0, "retries": 0, "seconds": 438.8, "max_memory_gb": 17.14, "ce": 0.006126583113655215, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 0.24754907190799713, "loss_scale": 4.0, "retries": 0, "seconds": 444.3, "max_memory_gb": 17.14, "ce": 0.014453551122642239, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.5384380221366882, "loss_scale": 4.0, "retries": 0, "seconds": 449.7, "max_memory_gb": 17.14, "ce": 0.044084874514510375, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.3340654671192169, "loss_scale": 4.0, "retries": 0, "seconds": 455.0, "max_memory_gb": 17.14, "ce": 0.009921896571313482, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.8576056361198425, "loss_scale": 4.0, "retries": 0, "seconds": 460.4, "max_memory_gb": 17.14, "ce": 0.10139584590865525, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.05252159386873245, "loss_scale": 4.0, "retries": 0, "seconds": 465.8, "max_memory_gb": 17.14, "ce": 0.0030322227685246617, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.15868358314037323, "loss_scale": 4.0, "retries": 0, "seconds": 471.3, "max_memory_gb": 17.14, "ce": 0.008260549775513937, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 2.6921095848083496, "loss_scale": 4.0, "retries": 0, "seconds": 476.8, "max_memory_gb": 17.14, "ce": 0.15123564680993695, "rank": 0.0}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.2734247148036957, "loss_scale": 4.0, "retries": 0, "seconds": 482.0, "max_memory_gb": 17.14, "ce": 0.03763086206231492, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 2.144266366958618, "loss_scale": 4.0, "retries": 0, "seconds": 487.4, "max_memory_gb": 17.14, "ce": 0.06776766752022922, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 0.9093470573425293, "loss_scale": 4.0, "retries": 0, "seconds": 492.9, "max_memory_gb": 17.14, "ce": 0.0574523703739942, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 3.810645580291748, "loss_scale": 4.0, "retries": 0, "seconds": 498.3, "max_memory_gb": 17.14, "ce": 0.5468590347172722, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 0.2182520627975464, "loss_scale": 4.0, "retries": 0, "seconds": 503.7, "max_memory_gb": 17.14, "ce": 0.026713448771033654, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 1.597951054573059, "loss_scale": 4.0, "retries": 0, "seconds": 509.1, "max_memory_gb": 17.14, "ce": 0.3199287383290539, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 1.6579166650772095, "loss_scale": 4.0, "retries": 0, "seconds": 514.5, "max_memory_gb": 17.14, "ce": 0.11223676188501486, "rank": 0.0}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.46819329261779785, "loss_scale": 4.0, "retries": 0, "seconds": 520.0, "max_memory_gb": 17.14, "ce": 0.030003924087395717, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.11704594641923904, "loss_scale": 4.0, "retries": 0, "seconds": 525.4, "max_memory_gb": 17.14, "ce": 0.009275196465296176, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 1.2118451595306396, "loss_scale": 4.0, "retries": 0, "seconds": 530.8, "max_memory_gb": 17.14, "ce": 0.06465644290892669, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.3878817558288574, "loss_scale": 4.0, "retries": 0, "seconds": 536.1, "max_memory_gb": 17.14, "ce": 0.05522729910990165, "rank": 0.12504151463508606}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 1.9487590789794922, "loss_scale": 4.0, "retries": 0, "seconds": 541.5, "max_memory_gb": 17.14, "ce": 0.2177693139819894, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.24088338017463684, "loss_scale": 4.0, "retries": 0, "seconds": 547.0, "max_memory_gb": 17.14, "ce": 0.012008432595848717, "rank": 0.21385730803012848}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.05341389775276184, "loss_scale": 4.0, "retries": 0, "seconds": 552.4, "max_memory_gb": 17.14, "ce": 0.003943835571590171, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 1.9668335914611816, "loss_scale": 4.0, "retries": 0, "seconds": 557.8, "max_memory_gb": 17.14, "ce": 0.1827131361130796, "rank": 0.0}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 1.1696109771728516, "loss_scale": 4.0, "retries": 0, "seconds": 563.1, "max_memory_gb": 17.14, "ce": 0.13925139114780904, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.334175705909729, "loss_scale": 4.0, "retries": 0, "seconds": 568.5, "max_memory_gb": 17.14, "ce": 0.02223731317644706, "rank": 0.0}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 0.07437499612569809, "loss_scale": 4.0, "retries": 0, "seconds": 574.1, "max_memory_gb": 17.14, "ce": 0.007659364277515124, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 0.15728294849395752, "loss_scale": 4.0, "retries": 0, "seconds": 579.5, "max_memory_gb": 17.14, "ce": 0.011533399670952349, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 0.039949122816324234, "loss_scale": 4.0, "retries": 0, "seconds": 585.0, "max_memory_gb": 17.14, "ce": 0.0032300378152285703, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.052718643099069595, "loss_scale": 4.0, "retries": 0, "seconds": 590.3, "max_memory_gb": 17.14, "ce": 0.0023524335583715583, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 1.3632738590240479, "loss_scale": 4.0, "retries": 0, "seconds": 595.8, "max_memory_gb": 17.14, "ce": 0.08723388639941732, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.19167965650558472, "loss_scale": 4.0, "retries": 0, "seconds": 601.2, "max_memory_gb": 17.14, "ce": 0.016047721830091177, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 2.827630043029785, "loss_scale": 4.0, "retries": 0, "seconds": 606.6, "max_memory_gb": 17.14, "ce": 0.5383827787204609, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 1.4348196983337402, "loss_scale": 4.0, "retries": 0, "seconds": 612.0, "max_memory_gb": 17.14, "ce": 0.2556356271588811, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.16065208613872528, "loss_scale": 4.0, "retries": 0, "seconds": 617.3, "max_memory_gb": 17.14, "ce": 0.009396749919687863, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 2.8706140518188477, "loss_scale": 4.0, "retries": 0, "seconds": 622.7, "max_memory_gb": 17.14, "ce": 0.4085771110917449, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.3560119569301605, "loss_scale": 4.0, "retries": 0, "seconds": 628.1, "max_memory_gb": 17.14, "ce": 0.025112099036505242, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 0.7925133109092712, "loss_scale": 4.0, "retries": 0, "seconds": 633.4, "max_memory_gb": 17.14, "ce": 0.02179509015536496, "rank": 0.12548218667507172}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.17610332369804382, "loss_scale": 4.0, "retries": 0, "seconds": 638.8, "max_memory_gb": 17.14, "ce": 0.00510464015837897, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 1.182103157043457, "loss_scale": 4.0, "retries": 0, "seconds": 644.2, "max_memory_gb": 17.14, "ce": 0.1901613196323524, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 1.2037782669067383, "loss_scale": 4.0, "retries": 0, "seconds": 649.8, "max_memory_gb": 17.14, "ce": 0.12671956510712334, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 1.3255770206451416, "loss_scale": 4.0, "retries": 0, "seconds": 655.2, "max_memory_gb": 17.14, "ce": 0.09280054088435463, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 3.790130615234375, "loss_scale": 4.0, "retries": 0, "seconds": 660.5, "max_memory_gb": 17.14, "ce": 0.38625632207254057, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 0.025528736412525177, "loss_scale": 4.0, "retries": 0, "seconds": 666.0, "max_memory_gb": 17.14, "ce": 0.0018370543107266712, "rank": 0.0}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 1.3074184656143188, "loss_scale": 4.0, "retries": 0, "seconds": 671.5, "max_memory_gb": 17.14, "ce": 0.3007087348287314, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 0.16724646091461182, "loss_scale": 4.0, "retries": 0, "seconds": 676.8, "max_memory_gb": 17.14, "ce": 0.01766439295965938, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.026956286281347275, "loss_scale": 4.0, "retries": 0, "seconds": 681.7, "max_memory_gb": 17.14, "ce": 0.0012286291546388384, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 0.1635567843914032, "loss_scale": 4.0, "retries": 0, "seconds": 686.5, "max_memory_gb": 17.14, "ce": 0.012365054526526364, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 0.860812246799469, "loss_scale": 4.0, "retries": 0, "seconds": 691.4, "max_memory_gb": 17.14, "ce": 0.038624829215223144, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 1.298282265663147, "loss_scale": 4.0, "retries": 0, "seconds": 696.3, "max_memory_gb": 17.14, "ce": 0.06985747807902953, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 0.9996907114982605, "loss_scale": 4.0, "retries": 0, "seconds": 701.0, "max_memory_gb": 17.14, "ce": 0.11710087682331505, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 0.42742881178855896, "loss_scale": 4.0, "retries": 0, "seconds": 706.0, "max_memory_gb": 17.14, "ce": 0.021458318474060434, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 4.824329853057861, "loss_scale": 4.0, "retries": 0, "seconds": 710.9, "max_memory_gb": 17.14, "ce": 0.40950948108161356, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.047206465154886246, "loss_scale": 4.0, "retries": 0, "seconds": 715.8, "max_memory_gb": 17.14, "ce": 0.005065484067699799, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 5.209353923797607, "loss_scale": 4.0, "retries": 0, "seconds": 720.7, "max_memory_gb": 17.14, "ce": 0.27770853120910033, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 3.604588508605957, "loss_scale": 4.0, "retries": 0, "seconds": 725.5, "max_memory_gb": 17.14, "ce": 0.6622670264819135, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 5.605648517608643, "loss_scale": 4.0, "retries": 0, "seconds": 730.4, "max_memory_gb": 17.14, "ce": 0.2681472126942026, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 0.0764448493719101, "loss_scale": 4.0, "retries": 0, "seconds": 735.3, "max_memory_gb": 17.14, "ce": 0.0031301312160394446, "rank": 0.08751403540372849}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.05734318867325783, "loss_scale": 4.0, "retries": 0, "seconds": 740.2, "max_memory_gb": 17.14, "ce": 0.004705596401436196, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.711392343044281, "loss_scale": 4.0, "retries": 0, "seconds": 745.1, "max_memory_gb": 17.14, "ce": 0.06308968359940081, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.05392371490597725, "loss_scale": 4.0, "retries": 0, "seconds": 750.0, "max_memory_gb": 17.14, "ce": 0.004673021855069237, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 1.0221328735351562, "loss_scale": 4.0, "retries": 0, "seconds": 754.9, "max_memory_gb": 17.14, "ce": 0.1761573585995393, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 1.9437018632888794, "loss_scale": 4.0, "retries": 0, "seconds": 759.8, "max_memory_gb": 17.14, "ce": 0.15632318148300328, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.6655849814414978, "loss_scale": 4.0, "retries": 0, "seconds": 764.7, "max_memory_gb": 17.14, "ce": 0.011729156111186967, "rank": 0.038669779896736145}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.677093505859375, "loss_scale": 4.0, "retries": 0, "seconds": 769.5, "max_memory_gb": 17.14, "ce": 0.07486626898821669, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 3.864980697631836, "loss_scale": 4.0, "retries": 0, "seconds": 774.4, "max_memory_gb": 17.14, "ce": 0.1818102349768651, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 1.7339894771575928, "loss_scale": 4.0, "retries": 0, "seconds": 779.3, "max_memory_gb": 17.14, "ce": 0.07052283933853687, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.2821095585823059, "loss_scale": 4.0, "retries": 0, "seconds": 784.3, "max_memory_gb": 17.14, "ce": 0.022831339206277335, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 3.520822286605835, "loss_scale": 4.0, "retries": 0, "seconds": 789.2, "max_memory_gb": 17.14, "ce": 0.6780990562137958, "rank": 0.19742298126220703}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.46745437383651733, "loss_scale": 4.0, "retries": 0, "seconds": 794.0, "max_memory_gb": 17.14, "ce": 0.040554051399794844, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.7427651882171631, "loss_scale": 4.0, "retries": 0, "seconds": 798.9, "max_memory_gb": 17.14, "ce": 0.15177638542604655, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 1.5039869546890259, "loss_scale": 4.0, "retries": 0, "seconds": 803.8, "max_memory_gb": 17.14, "ce": 0.32347221792565506, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.09217789024114609, "loss_scale": 4.0, "retries": 0, "seconds": 808.8, "max_memory_gb": 17.14, "ce": 0.004412805465108249, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 1.616355061531067, "loss_scale": 4.0, "retries": 0, "seconds": 813.7, "max_memory_gb": 17.14, "ce": 0.065205915601382, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 0.23055143654346466, "loss_scale": 4.0, "retries": 0, "seconds": 818.5, "max_memory_gb": 17.14, "ce": 0.013932103326851575, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.3715192675590515, "loss_scale": 4.0, "retries": 0, "seconds": 823.4, "max_memory_gb": 17.14, "ce": 0.02201530499269211, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 1.9055252075195312, "loss_scale": 4.0, "retries": 0, "seconds": 828.4, "max_memory_gb": 17.14, "ce": 0.4053576063131459, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 0.0736076682806015, "loss_scale": 4.0, "retries": 0, "seconds": 833.3, "max_memory_gb": 17.14, "ce": 0.006298884962006923, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.3937457799911499, "loss_scale": 4.0, "retries": 0, "seconds": 838.1, "max_memory_gb": 17.14, "ce": 0.011411355352947794, "rank": 0.12477051466703415}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 0.30389732122421265, "loss_scale": 4.0, "retries": 0, "seconds": 843.1, "max_memory_gb": 17.14, "ce": 0.03296166204017936, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 0.46929094195365906, "loss_scale": 4.0, "retries": 0, "seconds": 848.0, "max_memory_gb": 17.14, "ce": 0.0601398946801055, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.35595187544822693, "loss_scale": 4.0, "retries": 0, "seconds": 852.9, "max_memory_gb": 17.14, "ce": 0.0234314451154205, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 0.7326541543006897, "loss_scale": 4.0, "retries": 0, "seconds": 857.9, "max_memory_gb": 17.14, "ce": 0.04863317540389289, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.37080419063568115, "loss_scale": 4.0, "retries": 0, "seconds": 862.7, "max_memory_gb": 17.14, "ce": 0.030962162688865646, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.027436479926109314, "loss_scale": 4.0, "retries": 0, "seconds": 867.6, "max_memory_gb": 17.14, "ce": 0.0016841279143591237, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 0.12022608518600464, "loss_scale": 4.0, "retries": 0, "seconds": 872.6, "max_memory_gb": 17.14, "ce": 0.004488656867579266, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 1.5044901371002197, "loss_scale": 4.0, "retries": 0, "seconds": 877.5, "max_memory_gb": 17.14, "ce": 0.30593658045359007, "rank": 0.03233739733695984}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 1.0698481798171997, "loss_scale": 4.0, "retries": 0, "seconds": 882.4, "max_memory_gb": 17.14, "ce": 0.1423716501576564, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 0.05628940835595131, "loss_scale": 4.0, "retries": 0, "seconds": 887.3, "max_memory_gb": 17.14, "ce": 0.0035429089632543764, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 3.4318737983703613, "loss_scale": 4.0, "retries": 0, "seconds": 892.2, "max_memory_gb": 17.14, "ce": 0.11350193246607887, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.807723879814148, "loss_scale": 4.0, "retries": 0, "seconds": 897.1, "max_memory_gb": 17.14, "ce": 0.07108803806590913, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 1.650346040725708, "loss_scale": 4.0, "retries": 0, "seconds": 902.0, "max_memory_gb": 17.14, "ce": 0.1628149533441956, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.3565777540206909, "loss_scale": 4.0, "retries": 0, "seconds": 906.8, "max_memory_gb": 17.14, "ce": 0.031957830001601906, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 3.051513671875, "loss_scale": 4.0, "retries": 0, "seconds": 911.7, "max_memory_gb": 17.14, "ce": 0.2834306215768265, "rank": 0.12493531405925751}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 1.0594648122787476, "loss_scale": 4.0, "retries": 0, "seconds": 916.6, "max_memory_gb": 17.14, "ce": 0.15621968881964676, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 1.5309127569198608, "loss_scale": 4.0, "retries": 0, "seconds": 921.5, "max_memory_gb": 17.14, "ce": 0.20053357458550636, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 0.9591386318206787, "loss_scale": 4.0, "retries": 0, "seconds": 926.4, "max_memory_gb": 17.14, "ce": 0.08124973002759361, "rank": 0.0}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 0.026146143674850464, "loss_scale": 4.0, "retries": 0, "seconds": 931.3, "max_memory_gb": 17.14, "ce": 0.0016916589984248276, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 0.4633875787258148, "loss_scale": 4.0, "retries": 0, "seconds": 936.2, "max_memory_gb": 17.14, "ce": 0.06614085140972747, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 1.4148614406585693, "loss_scale": 4.0, "retries": 0, "seconds": 941.1, "max_memory_gb": 17.14, "ce": 0.030638842497410224, "rank": 0.126815065741539}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 1.2877452373504639, "loss_scale": 4.0, "retries": 0, "seconds": 946.0, "max_memory_gb": 17.14, "ce": 0.11815367627355045, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 0.16966405510902405, "loss_scale": 4.0, "retries": 0, "seconds": 950.8, "max_memory_gb": 17.14, "ce": 0.010078558867462561, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 1.2753770351409912, "loss_scale": 4.0, "retries": 0, "seconds": 955.8, "max_memory_gb": 17.14, "ce": 0.11879576248793455, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 1.4354320764541626, "loss_scale": 4.0, "retries": 0, "seconds": 960.7, "max_memory_gb": 17.14, "ce": 0.20116343451581997, "rank": 0.0}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 2.3649144172668457, "loss_scale": 4.0, "retries": 0, "seconds": 965.6, "max_memory_gb": 17.14, "ce": 0.11671976272646134, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 1.4451013803482056, "loss_scale": 4.0, "retries": 0, "seconds": 970.5, "max_memory_gb": 17.14, "ce": 0.07253632837318946, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.7293073534965515, "loss_scale": 4.0, "retries": 0, "seconds": 975.4, "max_memory_gb": 17.14, "ce": 0.09504920351241708, "rank": 0.0}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 0.25487568974494934, "loss_scale": 4.0, "retries": 0, "seconds": 980.2, "max_memory_gb": 17.14, "ce": 0.026475899073830078, "rank": 0.12492610514163971}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 0.23038220405578613, "loss_scale": 4.0, "retries": 0, "seconds": 985.2, "max_memory_gb": 17.14, "ce": 0.02008112943121887, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 2.8128130435943604, "loss_scale": 4.0, "retries": 0, "seconds": 990.1, "max_memory_gb": 17.14, "ce": 0.07638707242836063, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 0.043997183442115784, "loss_scale": 4.0, "retries": 0, "seconds": 995.0, "max_memory_gb": 17.14, "ce": 0.0036927653895872936, "rank": 0.12503230571746826}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 0.37656182050704956, "loss_scale": 4.0, "retries": 0, "seconds": 999.8, "max_memory_gb": 17.14, "ce": 0.046480264606088895, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.21646647155284882, "loss_scale": 4.0, "retries": 0, "seconds": 1004.8, "max_memory_gb": 17.14, "ce": 0.012101052997650186, "rank": 0.0}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 1.4319473505020142, "loss_scale": 4.0, "retries": 0, "seconds": 1009.7, "max_memory_gb": 17.14, "ce": 0.06056194009147475, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.5277372598648071, "loss_scale": 4.0, "retries": 0, "seconds": 1014.6, "max_memory_gb": 17.14, "ce": 0.0579180312047356, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.03504440188407898, "loss_scale": 4.0, "retries": 0, "seconds": 1019.6, "max_memory_gb": 17.14, "ce": 0.002282512158217287, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.004166353028267622, "loss_scale": 4.0, "retries": 0, "seconds": 1024.5, "max_memory_gb": 17.14, "ce": 0.00018314003000341472, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.5067665576934814, "loss_scale": 4.0, "retries": 0, "seconds": 1029.3, "max_memory_gb": 17.14, "ce": 0.09820201192383138, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 0.026653682813048363, "loss_scale": 4.0, "retries": 0, "seconds": 1034.2, "max_memory_gb": 17.14, "ce": 0.0012566637190047913, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 0.7349677681922913, "loss_scale": 4.0, "retries": 0, "seconds": 1039.2, "max_memory_gb": 17.14, "ce": 0.17248374852488269, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.22401057183742523, "loss_scale": 4.0, "retries": 0, "seconds": 1044.1, "max_memory_gb": 17.14, "ce": 0.014507601190189234, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.11955351382493973, "loss_scale": 4.0, "retries": 0, "seconds": 1049.1, "max_memory_gb": 17.14, "ce": 0.011422343082813313, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.9533209800720215, "loss_scale": 4.0, "retries": 0, "seconds": 1053.9, "max_memory_gb": 17.14, "ce": 0.0530119462255243, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.022727031260728836, "loss_scale": 4.0, "retries": 0, "seconds": 1058.8, "max_memory_gb": 17.14, "ce": 0.0009649549035657401, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 2.2455217838287354, "loss_scale": 4.0, "retries": 0, "seconds": 1063.7, "max_memory_gb": 17.14, "ce": 0.31982619103189336, "rank": 0.12487230449914932}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 2.1432056427001953, "loss_scale": 4.0, "retries": 0, "seconds": 1068.6, "max_memory_gb": 17.14, "ce": 0.39849315640094574, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.1237468421459198, "loss_scale": 4.0, "retries": 0, "seconds": 1073.6, "max_memory_gb": 17.14, "ce": 0.006436611992739927, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.17989160120487213, "loss_scale": 4.0, "retries": 0, "seconds": 1078.4, "max_memory_gb": 17.14, "ce": 0.012735228785231811, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.16757330298423767, "loss_scale": 4.0, "retries": 0, "seconds": 1083.3, "max_memory_gb": 17.14, "ce": 0.008355431963082083, "rank": 0.0}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 1.2108477354049683, "loss_scale": 4.0, "retries": 0, "seconds": 1088.2, "max_memory_gb": 17.14, "ce": 0.03885113460705725, "rank": 0.0}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 1.762557864189148, "loss_scale": 4.0, "retries": 0, "seconds": 1093.2, "max_memory_gb": 17.14, "ce": 0.46759353294282846, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 0.49508118629455566, "loss_scale": 4.0, "retries": 0, "seconds": 1098.1, "max_memory_gb": 17.14, "ce": 0.07990304854547503, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.47403833270072937, "loss_scale": 4.0, "retries": 0, "seconds": 1103.0, "max_memory_gb": 17.14, "ce": 0.03334642968275148, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 0.8589410781860352, "loss_scale": 4.0, "retries": 0, "seconds": 1107.9, "max_memory_gb": 17.14, "ce": 0.09263440313748106, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 0.16157495975494385, "loss_scale": 4.0, "retries": 0, "seconds": 1112.8, "max_memory_gb": 17.14, "ce": 0.010736555063203923, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.3332303464412689, "loss_scale": 4.0, "retries": 0, "seconds": 1117.7, "max_memory_gb": 17.14, "ce": 0.02678333673384259, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.1906294822692871, "loss_scale": 4.0, "retries": 0, "seconds": 1122.6, "max_memory_gb": 17.14, "ce": 0.011421512320112015, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 0.9096120595932007, "loss_scale": 4.0, "retries": 0, "seconds": 1127.5, "max_memory_gb": 17.14, "ce": 0.11745756698837795, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.303097128868103, "loss_scale": 4.0, "retries": 0, "seconds": 1132.4, "max_memory_gb": 17.14, "ce": 0.010788884483304173, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 1.2048767805099487, "loss_scale": 4.0, "retries": 0, "seconds": 1137.4, "max_memory_gb": 17.14, "ce": 0.11136283096175248, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.5720533132553101, "loss_scale": 4.0, "retries": 0, "seconds": 1142.2, "max_memory_gb": 17.14, "ce": 0.024783976411072217, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 1.8322465419769287, "loss_scale": 4.0, "retries": 0, "seconds": 1147.1, "max_memory_gb": 17.14, "ce": 0.15619075452070774, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 0.6445693373680115, "loss_scale": 4.0, "retries": 0, "seconds": 1152.0, "max_memory_gb": 17.14, "ce": 0.067461839018506, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 1.252183437347412, "loss_scale": 4.0, "retries": 0, "seconds": 1157.0, "max_memory_gb": 17.14, "ce": 0.06585031275562869, "rank": 0.12476453185081482}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.3318134844303131, "loss_scale": 4.0, "retries": 0, "seconds": 1162.0, "max_memory_gb": 17.14, "ce": 0.016093487359512437, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.4779915511608124, "loss_scale": 4.0, "retries": 0, "seconds": 1166.7, "max_memory_gb": 17.14, "ce": 0.03118686490370237, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.7451112866401672, "loss_scale": 4.0, "retries": 0, "seconds": 1171.7, "max_memory_gb": 17.14, "ce": 0.0914734944221891, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.5335241556167603, "loss_scale": 4.0, "retries": 0, "seconds": 1176.6, "max_memory_gb": 17.14, "ce": 0.05956900701903578, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.03533582016825676, "loss_scale": 4.0, "retries": 0, "seconds": 1181.5, "max_memory_gb": 17.14, "ce": 0.002714699260877751, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 0.15809066593647003, "loss_scale": 4.0, "retries": 0, "seconds": 1186.5, "max_memory_gb": 17.14, "ce": 0.009038995279524897, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.9842574596405029, "loss_scale": 4.0, "retries": 0, "seconds": 1191.3, "max_memory_gb": 17.14, "ce": 0.128416366473175, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.008947260677814484, "loss_scale": 4.0, "retries": 0, "seconds": 1196.5, "max_memory_gb": 17.14, "ce": 0.0005093768600090698, "rank": 0.1250055581331253}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.5172214508056641, "loss_scale": 4.0, "retries": 0, "seconds": 1201.5, "max_memory_gb": 17.14, "ce": 0.03211998327674337, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 1.3428263664245605, "loss_scale": 4.0, "retries": 0, "seconds": 1206.4, "max_memory_gb": 17.14, "ce": 0.09178843662675717, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 2.6664113998413086, "loss_scale": 4.0, "retries": 0, "seconds": 1211.2, "max_memory_gb": 17.14, "ce": 0.9016642621425035, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.05853632837533951, "loss_scale": 4.0, "retries": 0, "seconds": 1216.1, "max_memory_gb": 17.14, "ce": 0.005163849380323882, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.4111171364784241, "loss_scale": 4.0, "retries": 0, "seconds": 1221.0, "max_memory_gb": 17.14, "ce": 0.00967020273651542, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.12869016826152802, "loss_scale": 4.0, "retries": 0, "seconds": 1226.0, "max_memory_gb": 17.14, "ce": 0.009929307420861733, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.6857593059539795, "loss_scale": 4.0, "retries": 0, "seconds": 1230.9, "max_memory_gb": 17.14, "ce": 0.11111623919646263, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 0.8674184083938599, "loss_scale": 4.0, "retries": 0, "seconds": 1235.7, "max_memory_gb": 17.14, "ce": 0.03843006082593092, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.49737778306007385, "loss_scale": 4.0, "retries": 0, "seconds": 1240.6, "max_memory_gb": 17.14, "ce": 0.08740564985964738, "rank": 0.12167138606309891}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.278738796710968, "loss_scale": 4.0, "retries": 0, "seconds": 1245.6, "max_memory_gb": 17.14, "ce": 0.03003835121467091, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.7357633113861084, "loss_scale": 4.0, "retries": 0, "seconds": 1250.5, "max_memory_gb": 17.14, "ce": 0.043894903224099835, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.6332590579986572, "loss_scale": 4.0, "retries": 0, "seconds": 1255.4, "max_memory_gb": 17.14, "ce": 0.0802058207286791, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 0.16026346385478973, "loss_scale": 4.0, "retries": 0, "seconds": 1260.2, "max_memory_gb": 17.14, "ce": 0.01827562720950482, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.6843631267547607, "loss_scale": 4.0, "retries": 0, "seconds": 1265.1, "max_memory_gb": 17.14, "ce": 0.13441806332411943, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.8856080174446106, "loss_scale": 4.0, "retries": 0, "seconds": 1270.1, "max_memory_gb": 17.14, "ce": 0.03424264052500803, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.006400703452527523, "loss_scale": 4.0, "retries": 0, "seconds": 1275.0, "max_memory_gb": 17.14, "ce": 0.0004428077934335306, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.10884229093790054, "loss_scale": 4.0, "retries": 0, "seconds": 1279.8, "max_memory_gb": 17.14, "ce": 0.00942451635114594, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 2.513030767440796, "loss_scale": 4.0, "retries": 0, "seconds": 1284.7, "max_memory_gb": 17.14, "ce": 0.1401472454224404, "rank": 0.0}

runs/fixed_internvl/removal/Middle8_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/removal/Middle8_s13/adapter/adapter_config.json

@@ -0,0 +1,213 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.14.mlp.down_proj": 144,
+    "model.language_model.layers.14.mlp.gate_proj": 144,
+    "model.language_model.layers.14.mlp.up_proj": 144,
+    "model.language_model.layers.14.self_attn.k_proj": 144,
+    "model.language_model.layers.14.self_attn.o_proj": 144,
+    "model.language_model.layers.14.self_attn.q_proj": 144,
+    "model.language_model.layers.14.self_attn.v_proj": 144,
+    "model.language_model.layers.15.mlp.down_proj": 144,
+    "model.language_model.layers.15.mlp.gate_proj": 144,
+    "model.language_model.layers.15.mlp.up_proj": 144,
+    "model.language_model.layers.15.self_attn.k_proj": 144,
+    "model.language_model.layers.15.self_attn.o_proj": 144,
+    "model.language_model.layers.15.self_attn.q_proj": 144,
+    "model.language_model.layers.15.self_attn.v_proj": 144,
+    "model.language_model.layers.16.mlp.down_proj": 144,
+    "model.language_model.layers.16.mlp.gate_proj": 144,
+    "model.language_model.layers.16.mlp.up_proj": 144,
+    "model.language_model.layers.16.self_attn.k_proj": 144,
+    "model.language_model.layers.16.self_attn.o_proj": 144,
+    "model.language_model.layers.16.self_attn.q_proj": 144,
+    "model.language_model.layers.16.self_attn.v_proj": 144,
+    "model.language_model.layers.17.mlp.down_proj": 144,
+    "model.language_model.layers.17.mlp.gate_proj": 144,
+    "model.language_model.layers.17.mlp.up_proj": 144,
+    "model.language_model.layers.17.self_attn.k_proj": 144,
+    "model.language_model.layers.17.self_attn.o_proj": 144,
+    "model.language_model.layers.17.self_attn.q_proj": 144,
+    "model.language_model.layers.17.self_attn.v_proj": 144,
+    "model.language_model.layers.18.mlp.down_proj": 144,
+    "model.language_model.layers.18.mlp.gate_proj": 144,
+    "model.language_model.layers.18.mlp.up_proj": 144,
+    "model.language_model.layers.18.self_attn.k_proj": 144,
+    "model.language_model.layers.18.self_attn.o_proj": 144,
+    "model.language_model.layers.18.self_attn.q_proj": 144,
+    "model.language_model.layers.18.self_attn.v_proj": 144,
+    "model.language_model.layers.19.mlp.down_proj": 144,
+    "model.language_model.layers.19.mlp.gate_proj": 144,
+    "model.language_model.layers.19.mlp.up_proj": 144,
+    "model.language_model.layers.19.self_attn.k_proj": 144,
+    "model.language_model.layers.19.self_attn.o_proj": 144,
+    "model.language_model.layers.19.self_attn.q_proj": 144,
+    "model.language_model.layers.19.self_attn.v_proj": 144,
+    "model.language_model.layers.20.mlp.down_proj": 144,
+    "model.language_model.layers.20.mlp.gate_proj": 144,
+    "model.language_model.layers.20.mlp.up_proj": 144,
+    "model.language_model.layers.20.self_attn.k_proj": 144,
+    "model.language_model.layers.20.self_attn.o_proj": 144,
+    "model.language_model.layers.20.self_attn.q_proj": 144,
+    "model.language_model.layers.20.self_attn.v_proj": 144,
+    "model.language_model.layers.21.mlp.down_proj": 144,
+    "model.language_model.layers.21.mlp.gate_proj": 144,
+    "model.language_model.layers.21.mlp.up_proj": 144,
+    "model.language_model.layers.21.self_attn.k_proj": 144,
+    "model.language_model.layers.21.self_attn.o_proj": 144,
+    "model.language_model.layers.21.self_attn.q_proj": 144,
+    "model.language_model.layers.21.self_attn.v_proj": 144
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.14.mlp.down_proj": 72,
+    "model.language_model.layers.14.mlp.gate_proj": 72,
+    "model.language_model.layers.14.mlp.up_proj": 72,
+    "model.language_model.layers.14.self_attn.k_proj": 72,
+    "model.language_model.layers.14.self_attn.o_proj": 72,
+    "model.language_model.layers.14.self_attn.q_proj": 72,
+    "model.language_model.layers.14.self_attn.v_proj": 72,
+    "model.language_model.layers.15.mlp.down_proj": 72,
+    "model.language_model.layers.15.mlp.gate_proj": 72,
+    "model.language_model.layers.15.mlp.up_proj": 72,
+    "model.language_model.layers.15.self_attn.k_proj": 72,
+    "model.language_model.layers.15.self_attn.o_proj": 72,
+    "model.language_model.layers.15.self_attn.q_proj": 72,
+    "model.language_model.layers.15.self_attn.v_proj": 72,
+    "model.language_model.layers.16.mlp.down_proj": 72,
+    "model.language_model.layers.16.mlp.gate_proj": 72,
+    "model.language_model.layers.16.mlp.up_proj": 72,
+    "model.language_model.layers.16.self_attn.k_proj": 72,
+    "model.language_model.layers.16.self_attn.o_proj": 72,
+    "model.language_model.layers.16.self_attn.q_proj": 72,
+    "model.language_model.layers.16.self_attn.v_proj": 72,
+    "model.language_model.layers.17.mlp.down_proj": 72,
+    "model.language_model.layers.17.mlp.gate_proj": 72,
+    "model.language_model.layers.17.mlp.up_proj": 72,
+    "model.language_model.layers.17.self_attn.k_proj": 72,
+    "model.language_model.layers.17.self_attn.o_proj": 72,
+    "model.language_model.layers.17.self_attn.q_proj": 72,
+    "model.language_model.layers.17.self_attn.v_proj": 72,
+    "model.language_model.layers.18.mlp.down_proj": 72,
+    "model.language_model.layers.18.mlp.gate_proj": 72,
+    "model.language_model.layers.18.mlp.up_proj": 72,
+    "model.language_model.layers.18.self_attn.k_proj": 72,
+    "model.language_model.layers.18.self_attn.o_proj": 72,
+    "model.language_model.layers.18.self_attn.q_proj": 72,
+    "model.language_model.layers.18.self_attn.v_proj": 72,
+    "model.language_model.layers.19.mlp.down_proj": 72,
+    "model.language_model.layers.19.mlp.gate_proj": 72,
+    "model.language_model.layers.19.mlp.up_proj": 72,
+    "model.language_model.layers.19.self_attn.k_proj": 72,
+    "model.language_model.layers.19.self_attn.o_proj": 72,
+    "model.language_model.layers.19.self_attn.q_proj": 72,
+    "model.language_model.layers.19.self_attn.v_proj": 72,
+    "model.language_model.layers.20.mlp.down_proj": 72,
+    "model.language_model.layers.20.mlp.gate_proj": 72,
+    "model.language_model.layers.20.mlp.up_proj": 72,
+    "model.language_model.layers.20.self_attn.k_proj": 72,
+    "model.language_model.layers.20.self_attn.o_proj": 72,
+    "model.language_model.layers.20.self_attn.q_proj": 72,
+    "model.language_model.layers.20.self_attn.v_proj": 72,
+    "model.language_model.layers.21.mlp.down_proj": 72,
+    "model.language_model.layers.21.mlp.gate_proj": 72,
+    "model.language_model.layers.21.mlp.up_proj": 72,
+    "model.language_model.layers.21.self_attn.k_proj": 72,
+    "model.language_model.layers.21.self_attn.o_proj": 72,
+    "model.language_model.layers.21.self_attn.q_proj": 72,
+    "model.language_model.layers.21.self_attn.v_proj": 72
+  },
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.16.mlp.down_proj",
+    "model.language_model.layers.17.self_attn.k_proj",
+    "model.language_model.layers.16.self_attn.o_proj",
+    "model.language_model.layers.17.mlp.gate_proj",
+    "model.language_model.layers.19.mlp.gate_proj",
+    "model.language_model.layers.21.mlp.down_proj",
+    "model.language_model.layers.14.self_attn.k_proj",
+    "model.language_model.layers.16.mlp.up_proj",
+    "model.language_model.layers.21.mlp.up_proj",
+    "model.language_model.layers.19.self_attn.k_proj",
+    "model.language_model.layers.20.self_attn.k_proj",
+    "model.language_model.layers.14.mlp.up_proj",
+    "model.language_model.layers.20.self_attn.q_proj",
+    "model.language_model.layers.21.self_attn.q_proj",
+    "model.language_model.layers.19.mlp.down_proj",
+    "model.language_model.layers.18.mlp.down_proj",
+    "model.language_model.layers.18.mlp.up_proj",
+    "model.language_model.layers.14.self_attn.o_proj",
+    "model.language_model.layers.19.self_attn.o_proj",
+    "model.language_model.layers.14.self_attn.q_proj",
+    "model.language_model.layers.21.mlp.gate_proj",
+    "model.language_model.layers.21.self_attn.k_proj",
+    "model.language_model.layers.21.self_attn.o_proj",
+    "model.language_model.layers.20.self_attn.v_proj",
+    "model.language_model.layers.16.self_attn.v_proj",
+    "model.language_model.layers.15.mlp.down_proj",
+    "model.language_model.layers.17.mlp.up_proj",
+    "model.language_model.layers.17.mlp.down_proj",
+    "model.language_model.layers.15.self_attn.v_proj",
+    "model.language_model.layers.17.self_attn.v_proj",
+    "model.language_model.layers.15.mlp.up_proj",
+    "model.language_model.layers.21.self_attn.v_proj",
+    "model.language_model.layers.14.self_attn.v_proj",
+    "model.language_model.layers.15.self_attn.q_proj",
+    "model.language_model.layers.17.self_attn.o_proj",
+    "model.language_model.layers.15.mlp.gate_proj",
+    "model.language_model.layers.16.self_attn.k_proj",
+    "model.language_model.layers.19.mlp.up_proj",
+    "model.language_model.layers.20.mlp.up_proj",
+    "model.language_model.layers.19.self_attn.v_proj",
+    "model.language_model.layers.15.self_attn.k_proj",
+    "model.language_model.layers.14.mlp.down_proj",
+    "model.language_model.layers.18.self_attn.v_proj",
+    "model.language_model.layers.20.mlp.gate_proj",
+    "model.language_model.layers.15.self_attn.o_proj",
+    "model.language_model.layers.18.mlp.gate_proj",
+    "model.language_model.layers.19.self_attn.q_proj",
+    "model.language_model.layers.16.mlp.gate_proj",
+    "model.language_model.layers.16.self_attn.q_proj",
+    "model.language_model.layers.18.self_attn.k_proj",
+    "model.language_model.layers.14.mlp.gate_proj",
+    "model.language_model.layers.20.self_attn.o_proj",
+    "model.language_model.layers.18.self_attn.q_proj",
+    "model.language_model.layers.18.self_attn.o_proj",
+    "model.language_model.layers.17.self_attn.q_proj",
+    "model.language_model.layers.20.mlp.down_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/fixed_internvl/removal/Middle8_s13/config.json

@@ -0,0 +1,43 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "world_size": 1,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "arm": {
+    "arm": "Middle8",
+    "layers": [
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21
+    ],
+    "ranks": {
+      "14": 72,
+      "15": 72,
+      "16": 72,
+      "17": 72,
+      "18": 72,
+      "19": 72,
+      "20": 72,
+      "21": 72
+    },
+    "parameters": 43646976,
+    "budget": 43646976
+  }
+}

runs/fixed_internvl/removal/Middle8_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 6.598066329956055, "loss_scale": 4.0, "retries": 0, "seconds": 5.6, "max_memory_gb": 16.79, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 6.3826398849487305, "loss_scale": 4.0, "retries": 0, "seconds": 10.5, "max_memory_gb": 17.11, "ce": 0.7750466839788714, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 19.67627716064453, "loss_scale": 4.0, "retries": 0, "seconds": 15.1, "max_memory_gb": 17.11, "ce": 1.1678408375009894, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 5.073079586029053, "loss_scale": 4.0, "retries": 0, "seconds": 19.4, "max_memory_gb": 17.11, "ce": 0.49718301206485194, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.17749643325805664, "loss_scale": 4.0, "retries": 0, "seconds": 24.0, "max_memory_gb": 17.11, "ce": 0.039426019050551986, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 4.058267593383789, "loss_scale": 4.0, "retries": 0, "seconds": 28.3, "max_memory_gb": 17.11, "ce": 0.28679325202392647, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 1.0740123987197876, "loss_scale": 4.0, "retries": 0, "seconds": 32.8, "max_memory_gb": 17.11, "ce": 0.3734158815223054, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 0.7579994797706604, "loss_scale": 4.0, "retries": 0, "seconds": 37.5, "max_memory_gb": 17.11, "ce": 0.3433576838579029, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 6.59818696975708, "loss_scale": 4.0, "retries": 0, "seconds": 42.2, "max_memory_gb": 17.11, "ce": 0.3427119280677289, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.3694741129875183, "loss_scale": 4.0, "retries": 0, "seconds": 46.9, "max_memory_gb": 17.11, "ce": 0.12434440776269184, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.5638008117675781, "loss_scale": 4.0, "retries": 0, "seconds": 51.7, "max_memory_gb": 17.11, "ce": 0.20025790796717047, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 0.2850085198879242, "loss_scale": 4.0, "retries": 0, "seconds": 56.3, "max_memory_gb": 17.11, "ce": 0.10578646038266015, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 0.5198869705200195, "loss_scale": 4.0, "retries": 0, "seconds": 60.8, "max_memory_gb": 17.11, "ce": 0.22648852357360738, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 1.029803991317749, "loss_scale": 4.0, "retries": 0, "seconds": 65.6, "max_memory_gb": 17.11, "ce": 0.28177842660807073, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 0.8700073957443237, "loss_scale": 4.0, "retries": 0, "seconds": 70.5, "max_memory_gb": 17.11, "ce": 0.19100808236180455, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.411034494638443, "loss_scale": 4.0, "retries": 0, "seconds": 75.2, "max_memory_gb": 17.11, "ce": 0.0705857476204983, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 1.2125587463378906, "loss_scale": 4.0, "retries": 0, "seconds": 80.0, "max_memory_gb": 17.11, "ce": 0.16415997081639944, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 2.0547685623168945, "loss_scale": 4.0, "retries": 0, "seconds": 84.9, "max_memory_gb": 17.12, "ce": 0.1884987503872253, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 1.0034780502319336, "loss_scale": 4.0, "retries": 0, "seconds": 89.6, "max_memory_gb": 17.12, "ce": 0.09122996879159473, "rank": 0.09213253855705261}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.14214573800563812, "loss_scale": 4.0, "retries": 0, "seconds": 94.3, "max_memory_gb": 17.12, "ce": 0.010704057346401896, "rank": 0.12496363371610641}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.9571057558059692, "loss_scale": 4.0, "retries": 0, "seconds": 99.3, "max_memory_gb": 17.12, "ce": 0.0716275303520888, "rank": 0.03467627614736557}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 4.4030327796936035, "loss_scale": 4.0, "retries": 0, "seconds": 104.1, "max_memory_gb": 17.12, "ce": 0.16969054362107272, "rank": 0.0}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.408992886543274, "loss_scale": 4.0, "retries": 0, "seconds": 108.9, "max_memory_gb": 17.12, "ce": 0.17254889193725376, "rank": 0.0}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 8.957677841186523, "loss_scale": 4.0, "retries": 0, "seconds": 113.6, "max_memory_gb": 17.12, "ce": 0.20275761973607587, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 3.8762524127960205, "loss_scale": 4.0, "retries": 0, "seconds": 118.5, "max_memory_gb": 17.12, "ce": 0.20177579280061764, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 4.292171955108643, "loss_scale": 4.0, "retries": 0, "seconds": 123.3, "max_memory_gb": 17.12, "ce": 0.18961571979525615, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 6.195641994476318, "loss_scale": 4.0, "retries": 0, "seconds": 128.0, "max_memory_gb": 17.12, "ce": 0.37321196822085767, "rank": 0.024211935698986053}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 2.454153537750244, "loss_scale": 4.0, "retries": 0, "seconds": 132.8, "max_memory_gb": 17.12, "ce": 0.20988660617149435, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 6.321992874145508, "loss_scale": 4.0, "retries": 0, "seconds": 137.7, "max_memory_gb": 17.12, "ce": 0.6695810800119943, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 0.1515321433544159, "loss_scale": 4.0, "retries": 0, "seconds": 142.3, "max_memory_gb": 17.12, "ce": 0.00407798730225295, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 6.618696212768555, "loss_scale": 4.0, "retries": 0, "seconds": 147.1, "max_memory_gb": 17.12, "ce": 0.6209080419503152, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 0.42099952697753906, "loss_scale": 4.0, "retries": 0, "seconds": 151.9, "max_memory_gb": 17.12, "ce": 0.016740163289796328, "rank": 0.0}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 3.748732566833496, "loss_scale": 4.0, "retries": 0, "seconds": 156.6, "max_memory_gb": 17.12, "ce": 0.2346042126773682, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.8426681756973267, "loss_scale": 4.0, "retries": 0, "seconds": 161.2, "max_memory_gb": 17.12, "ce": 0.09421060817840043, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 3.4155020713806152, "loss_scale": 4.0, "retries": 0, "seconds": 166.2, "max_memory_gb": 17.12, "ce": 0.040355077959247865, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 1.479185938835144, "loss_scale": 4.0, "retries": 0, "seconds": 171.0, "max_memory_gb": 17.12, "ce": 0.1280105489786365, "rank": 0.0}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 1.1716946363449097, "loss_scale": 4.0, "retries": 0, "seconds": 175.6, "max_memory_gb": 17.12, "ce": 0.17556215591321234, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.8541341423988342, "loss_scale": 4.0, "retries": 0, "seconds": 180.3, "max_memory_gb": 17.12, "ce": 0.10730055649764836, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 1.3031580448150635, "loss_scale": 4.0, "retries": 0, "seconds": 184.7, "max_memory_gb": 17.12, "ce": 0.09262039774330333, "rank": 0.12485933303833008}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.3421081304550171, "loss_scale": 4.0, "retries": 0, "seconds": 189.4, "max_memory_gb": 17.12, "ce": 0.014528023482853314, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 3.434464454650879, "loss_scale": 4.0, "retries": 0, "seconds": 194.2, "max_memory_gb": 17.12, "ce": 0.37315857151406817, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.2857740819454193, "loss_scale": 4.0, "retries": 0, "seconds": 198.9, "max_memory_gb": 17.12, "ce": 0.016538988129468635, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.1444348245859146, "loss_scale": 4.0, "retries": 0, "seconds": 203.5, "max_memory_gb": 17.12, "ce": 0.006803233962273225, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 7.395954608917236, "loss_scale": 4.0, "retries": 0, "seconds": 208.0, "max_memory_gb": 17.12, "ce": 0.2723563513981162, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 3.6248860359191895, "loss_scale": 4.0, "retries": 0, "seconds": 212.7, "max_memory_gb": 17.12, "ce": 0.4196672100079013, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 1.1258935928344727, "loss_scale": 4.0, "retries": 0, "seconds": 217.3, "max_memory_gb": 17.12, "ce": 0.10241915180085925, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.6109721660614014, "loss_scale": 4.0, "retries": 0, "seconds": 222.1, "max_memory_gb": 17.12, "ce": 0.0338591985482708, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.09946216642856598, "loss_scale": 4.0, "retries": 0, "seconds": 226.8, "max_memory_gb": 17.12, "ce": 0.0034585196167427057, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.16306255757808685, "loss_scale": 4.0, "retries": 0, "seconds": 231.6, "max_memory_gb": 17.12, "ce": 0.010840152221135213, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 1.2032861709594727, "loss_scale": 4.0, "retries": 0, "seconds": 236.2, "max_memory_gb": 17.12, "ce": 0.09368656364767958, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 5.434545040130615, "loss_scale": 4.0, "retries": 0, "seconds": 240.8, "max_memory_gb": 17.12, "ce": 0.17239610676824668, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.37075379490852356, "loss_scale": 4.0, "retries": 0, "seconds": 245.5, "max_memory_gb": 17.12, "ce": 0.029599888115626527, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 1.944976568222046, "loss_scale": 4.0, "retries": 0, "seconds": 250.0, "max_memory_gb": 17.12, "ce": 0.09054694652695616, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 1.164765477180481, "loss_scale": 4.0, "retries": 0, "seconds": 254.6, "max_memory_gb": 17.12, "ce": 0.08003289225962362, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.14764390885829926, "loss_scale": 4.0, "retries": 0, "seconds": 259.4, "max_memory_gb": 17.12, "ce": 0.008075302363636183, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.1773679256439209, "loss_scale": 4.0, "retries": 0, "seconds": 264.0, "max_memory_gb": 17.12, "ce": 0.01344966624128574, "rank": 0.0}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.12284113466739655, "loss_scale": 4.0, "retries": 0, "seconds": 268.6, "max_memory_gb": 17.12, "ce": 0.006024058094681095, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.05710173398256302, "loss_scale": 4.0, "retries": 0, "seconds": 273.3, "max_memory_gb": 17.12, "ce": 0.001437012622844236, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 4.686634063720703, "loss_scale": 4.0, "retries": 0, "seconds": 278.1, "max_memory_gb": 17.12, "ce": 0.4244370101765753, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.7147172093391418, "loss_scale": 4.0, "retries": 0, "seconds": 282.7, "max_memory_gb": 17.12, "ce": 0.06429499425712493, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 4.172966003417969, "loss_scale": 4.0, "retries": 0, "seconds": 287.3, "max_memory_gb": 17.12, "ce": 0.35485367513319943, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.43244150280952454, "loss_scale": 4.0, "retries": 0, "seconds": 291.9, "max_memory_gb": 17.12, "ce": 0.025478194096422158, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.5626148581504822, "loss_scale": 4.0, "retries": 0, "seconds": 296.6, "max_memory_gb": 17.12, "ce": 0.037496156815905124, "rank": 0.0}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.12207791209220886, "loss_scale": 4.0, "retries": 0, "seconds": 301.2, "max_memory_gb": 17.12, "ce": 0.004585706902162201, "rank": 0.0}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 1.0686596632003784, "loss_scale": 4.0, "retries": 0, "seconds": 305.8, "max_memory_gb": 17.12, "ce": 0.13899114668856782, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 1.5840075016021729, "loss_scale": 4.0, "retries": 0, "seconds": 310.6, "max_memory_gb": 17.12, "ce": 0.21393927964084725, "rank": 0.12432586401700974}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.022203106433153152, "loss_scale": 4.0, "retries": 0, "seconds": 315.2, "max_memory_gb": 17.12, "ce": 0.0011423514998512019, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 3.106335163116455, "loss_scale": 4.0, "retries": 0, "seconds": 319.8, "max_memory_gb": 17.12, "ce": 0.037454161052764334, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 5.255605697631836, "loss_scale": 4.0, "retries": 0, "seconds": 324.8, "max_memory_gb": 17.12, "ce": 0.8243549255989819, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.2532045841217041, "loss_scale": 4.0, "retries": 0, "seconds": 329.4, "max_memory_gb": 17.12, "ce": 0.002572024297535336, "rank": 0.0}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.03916134685277939, "loss_scale": 4.0, "retries": 0, "seconds": 334.0, "max_memory_gb": 17.12, "ce": 0.0016418345869055884, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.6155794858932495, "loss_scale": 4.0, "retries": 0, "seconds": 338.8, "max_memory_gb": 17.12, "ce": 0.07545313691298361, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 3.2039592266082764, "loss_scale": 4.0, "retries": 0, "seconds": 343.3, "max_memory_gb": 17.12, "ce": 0.1869796390508327, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 1.193235158920288, "loss_scale": 4.0, "retries": 0, "seconds": 347.8, "max_memory_gb": 17.12, "ce": 0.10523624654919317, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 2.5607872009277344, "loss_scale": 4.0, "retries": 0, "seconds": 352.7, "max_memory_gb": 17.12, "ce": 0.7010341068671551, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 1.4673333168029785, "loss_scale": 4.0, "retries": 0, "seconds": 357.1, "max_memory_gb": 17.12, "ce": 0.38346636774258513, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.47586819529533386, "loss_scale": 4.0, "retries": 0, "seconds": 361.7, "max_memory_gb": 17.12, "ce": 0.027186445109691704, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.2665303647518158, "loss_scale": 4.0, "retries": 0, "seconds": 366.2, "max_memory_gb": 17.12, "ce": 0.01947251748151757, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.5344730615615845, "loss_scale": 4.0, "retries": 0, "seconds": 370.9, "max_memory_gb": 17.12, "ce": 0.012922749287099577, "rank": 0.0}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.2943722605705261, "loss_scale": 4.0, "retries": 0, "seconds": 375.5, "max_memory_gb": 17.12, "ce": 0.02195617384882098, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.8738061189651489, "loss_scale": 4.0, "retries": 0, "seconds": 380.2, "max_memory_gb": 17.12, "ce": 0.057998546997623635, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.4823867678642273, "loss_scale": 4.0, "retries": 0, "seconds": 385.1, "max_memory_gb": 17.12, "ce": 0.058023238510941155, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 1.1300069093704224, "loss_scale": 4.0, "retries": 0, "seconds": 389.7, "max_memory_gb": 17.12, "ce": 0.20861851077802385, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.5322508811950684, "loss_scale": 4.0, "retries": 0, "seconds": 394.3, "max_memory_gb": 17.12, "ce": 0.027417215517061777, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 1.2931289672851562, "loss_scale": 4.0, "retries": 0, "seconds": 399.0, "max_memory_gb": 17.12, "ce": 0.015372879610254131, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 1.9159115552902222, "loss_scale": 4.0, "retries": 0, "seconds": 403.8, "max_memory_gb": 17.12, "ce": 0.06694891789993562, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.8989990949630737, "loss_scale": 4.0, "retries": 0, "seconds": 408.5, "max_memory_gb": 17.12, "ce": 0.09130096225817397, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.5084582567214966, "loss_scale": 4.0, "retries": 0, "seconds": 413.3, "max_memory_gb": 17.12, "ce": 0.021800515194627224, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.2097194641828537, "loss_scale": 4.0, "retries": 0, "seconds": 417.9, "max_memory_gb": 17.12, "ce": 0.015939376974984043, "rank": 0.1249980479478836}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.9967199563980103, "loss_scale": 4.0, "retries": 0, "seconds": 422.2, "max_memory_gb": 17.12, "ce": 0.07034587865018693, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 3.9940505027770996, "loss_scale": 4.0, "retries": 0, "seconds": 426.9, "max_memory_gb": 17.12, "ce": 0.5441641529187109, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 2.071902275085449, "loss_scale": 4.0, "retries": 0, "seconds": 431.6, "max_memory_gb": 17.12, "ce": 0.05158369196806234, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 2.167703151702881, "loss_scale": 4.0, "retries": 0, "seconds": 436.2, "max_memory_gb": 17.12, "ce": 0.06877861691464204, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 0.23723559081554413, "loss_scale": 4.0, "retries": 0, "seconds": 440.6, "max_memory_gb": 17.12, "ce": 0.012410963954607723, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 5.843066215515137, "loss_scale": 4.0, "retries": 0, "seconds": 445.3, "max_memory_gb": 17.12, "ce": 0.55385350952929, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 5.829464912414551, "loss_scale": 4.0, "retries": 0, "seconds": 449.9, "max_memory_gb": 17.12, "ce": 0.3846247872143067, "rank": 0.2036549150943756}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.06130506098270416, "loss_scale": 4.0, "retries": 0, "seconds": 454.5, "max_memory_gb": 17.12, "ce": 0.004276986484910594, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 1.964557409286499, "loss_scale": 4.0, "retries": 0, "seconds": 459.1, "max_memory_gb": 17.12, "ce": 0.10390072699919983, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.5613493323326111, "loss_scale": 4.0, "retries": 0, "seconds": 463.7, "max_memory_gb": 17.12, "ce": 0.017050164304237114, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.48001572489738464, "loss_scale": 4.0, "retries": 0, "seconds": 468.3, "max_memory_gb": 17.12, "ce": 0.08571043700067094, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.29195332527160645, "loss_scale": 4.0, "retries": 0, "seconds": 472.9, "max_memory_gb": 17.12, "ce": 0.01073919903501519, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.2608169615268707, "loss_scale": 4.0, "retries": 0, "seconds": 477.9, "max_memory_gb": 17.12, "ce": 0.01875240184745053, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.09713775664567947, "loss_scale": 4.0, "retries": 0, "seconds": 482.6, "max_memory_gb": 17.12, "ce": 0.0061322529527387815, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 1.0765600204467773, "loss_scale": 4.0, "retries": 0, "seconds": 487.0, "max_memory_gb": 17.12, "ce": 0.027039438937208615, "rank": 0.12474949657917023}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.5309938788414001, "loss_scale": 4.0, "retries": 0, "seconds": 491.6, "max_memory_gb": 17.12, "ce": 0.046526548900146736, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.18876369297504425, "loss_scale": 4.0, "retries": 0, "seconds": 496.3, "max_memory_gb": 17.12, "ce": 0.0028114287542848615, "rank": 0.09071093797683716}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 1.3088449239730835, "loss_scale": 4.0, "retries": 0, "seconds": 500.9, "max_memory_gb": 17.12, "ce": 0.1002839090388079, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 2.8959286212921143, "loss_scale": 4.0, "retries": 0, "seconds": 505.3, "max_memory_gb": 17.12, "ce": 0.4223950971245358, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 3.2705464363098145, "loss_scale": 4.0, "retries": 0, "seconds": 510.1, "max_memory_gb": 17.12, "ce": 0.2905675871479616, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.16363964974880219, "loss_scale": 4.0, "retries": 0, "seconds": 514.9, "max_memory_gb": 17.12, "ce": 0.010917718263954157, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.39590173959732056, "loss_scale": 4.0, "retries": 0, "seconds": 519.5, "max_memory_gb": 17.12, "ce": 0.01697402639865686, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.051056910306215286, "loss_scale": 4.0, "retries": 0, "seconds": 524.1, "max_memory_gb": 17.12, "ce": 0.0030021602360648103, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.19553415477275848, "loss_scale": 4.0, "retries": 0, "seconds": 528.7, "max_memory_gb": 17.12, "ce": 0.01750983869897027, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 1.0207414627075195, "loss_scale": 4.0, "retries": 0, "seconds": 533.3, "max_memory_gb": 17.12, "ce": 0.12267291716125328, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 2.1432414054870605, "loss_scale": 4.0, "retries": 0, "seconds": 538.0, "max_memory_gb": 17.12, "ce": 0.0720583074617025, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.8030210137367249, "loss_scale": 4.0, "retries": 0, "seconds": 542.5, "max_memory_gb": 17.12, "ce": 0.14488294570401195, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.13043604791164398, "loss_scale": 4.0, "retries": 0, "seconds": 547.1, "max_memory_gb": 17.12, "ce": 0.0046765283832428395, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 2.112762212753296, "loss_scale": 4.0, "retries": 0, "seconds": 551.7, "max_memory_gb": 17.12, "ce": 0.20077250914437172, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.0665544793009758, "loss_scale": 4.0, "retries": 0, "seconds": 556.3, "max_memory_gb": 17.12, "ce": 0.003404123480322596, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 7.262181758880615, "loss_scale": 4.0, "retries": 0, "seconds": 560.9, "max_memory_gb": 17.12, "ce": 0.3110910465939014, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.7243046164512634, "loss_scale": 4.0, "retries": 0, "seconds": 565.5, "max_memory_gb": 17.12, "ce": 0.05796994353568152, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.9291557669639587, "loss_scale": 4.0, "retries": 0, "seconds": 569.9, "max_memory_gb": 17.12, "ce": 0.087561667414775, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.578356921672821, "loss_scale": 4.0, "retries": 0, "seconds": 574.6, "max_memory_gb": 17.12, "ce": 0.033856654857117974, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.9850081205368042, "loss_scale": 4.0, "retries": 0, "seconds": 579.1, "max_memory_gb": 17.12, "ce": 0.5120908092485479, "rank": 0.0}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.7154951691627502, "loss_scale": 4.0, "retries": 0, "seconds": 583.6, "max_memory_gb": 17.12, "ce": 0.11247116871527396, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 2.4928081035614014, "loss_scale": 4.0, "retries": 0, "seconds": 588.3, "max_memory_gb": 17.12, "ce": 0.38247936500647484, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.2682717442512512, "loss_scale": 4.0, "retries": 0, "seconds": 592.8, "max_memory_gb": 17.12, "ce": 0.007430748007209331, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 2.515709400177002, "loss_scale": 4.0, "retries": 0, "seconds": 597.4, "max_memory_gb": 17.12, "ce": 0.27609911161925993, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 1.0635453462600708, "loss_scale": 4.0, "retries": 0, "seconds": 602.1, "max_memory_gb": 17.12, "ce": 0.09093820977614087, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 1.0362324714660645, "loss_scale": 4.0, "retries": 0, "seconds": 606.7, "max_memory_gb": 17.12, "ce": 0.14176840411255398, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 2.4250144958496094, "loss_scale": 4.0, "retries": 0, "seconds": 610.9, "max_memory_gb": 17.12, "ce": 0.07913143674431922, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 2.2252631187438965, "loss_scale": 4.0, "retries": 0, "seconds": 615.5, "max_memory_gb": 17.12, "ce": 0.25582638827017945, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 1.4574503898620605, "loss_scale": 4.0, "retries": 0, "seconds": 620.1, "max_memory_gb": 17.12, "ce": 0.23491668570477486, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 1.8701059818267822, "loss_scale": 4.0, "retries": 0, "seconds": 624.7, "max_memory_gb": 17.12, "ce": 0.27976836002153505, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 1.7881841659545898, "loss_scale": 4.0, "retries": 0, "seconds": 629.1, "max_memory_gb": 17.12, "ce": 0.059660757948222454, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 3.947270631790161, "loss_scale": 4.0, "retries": 0, "seconds": 633.7, "max_memory_gb": 17.12, "ce": 0.29367034773440537, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.10674870014190674, "loss_scale": 4.0, "retries": 0, "seconds": 638.3, "max_memory_gb": 17.12, "ce": 0.007999846170605451, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 4.658782958984375, "loss_scale": 4.0, "retries": 0, "seconds": 642.9, "max_memory_gb": 17.12, "ce": 0.15669765523489332, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 1.331538438796997, "loss_scale": 4.0, "retries": 0, "seconds": 647.5, "max_memory_gb": 17.12, "ce": 0.06254201256706438, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.05983223393559456, "loss_scale": 4.0, "retries": 0, "seconds": 652.3, "max_memory_gb": 17.12, "ce": 0.0037256895566315507, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.13019777834415436, "loss_scale": 4.0, "retries": 0, "seconds": 656.5, "max_memory_gb": 17.12, "ce": 0.008850918616190029, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 2.572288751602173, "loss_scale": 4.0, "retries": 0, "seconds": 661.3, "max_memory_gb": 17.12, "ce": 0.11359752879434382, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 4.183173179626465, "loss_scale": 4.0, "retries": 0, "seconds": 666.0, "max_memory_gb": 17.12, "ce": 0.32452498948441644, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.36914360523223877, "loss_scale": 4.0, "retries": 0, "seconds": 670.5, "max_memory_gb": 17.12, "ce": 0.02979950289318367, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.043113354593515396, "loss_scale": 4.0, "retries": 0, "seconds": 676.8, "max_memory_gb": 17.12, "ce": 0.0020162234013696434, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.15694499015808105, "loss_scale": 4.0, "retries": 0, "seconds": 681.9, "max_memory_gb": 17.12, "ce": 0.013924166477409017, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 1.6420713663101196, "loss_scale": 4.0, "retries": 0, "seconds": 687.1, "max_memory_gb": 17.12, "ce": 0.4158444633903855, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.1425618678331375, "loss_scale": 4.0, "retries": 0, "seconds": 692.0, "max_memory_gb": 17.12, "ce": 0.007031220862700138, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.5401785969734192, "loss_scale": 4.0, "retries": 0, "seconds": 698.0, "max_memory_gb": 17.12, "ce": 0.02091255340474163, "rank": 0.1100495457649231}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.6679081916809082, "loss_scale": 4.0, "retries": 0, "seconds": 702.8, "max_memory_gb": 17.12, "ce": 0.08408629457971983, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 4.446317672729492, "loss_scale": 4.0, "retries": 0, "seconds": 707.3, "max_memory_gb": 17.12, "ce": 0.1229386136692483, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 0.25044524669647217, "loss_scale": 4.0, "retries": 0, "seconds": 711.9, "max_memory_gb": 17.12, "ce": 0.01948076491953543, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.5332352519035339, "loss_scale": 4.0, "retries": 0, "seconds": 716.9, "max_memory_gb": 17.12, "ce": 0.09984058100872062, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.3768148422241211, "loss_scale": 4.0, "retries": 0, "seconds": 722.0, "max_memory_gb": 17.12, "ce": 0.03999000711610279, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 1.7545065879821777, "loss_scale": 4.0, "retries": 0, "seconds": 726.7, "max_memory_gb": 17.12, "ce": 0.21035979432053864, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.3723757266998291, "loss_scale": 4.0, "retries": 0, "seconds": 731.6, "max_memory_gb": 17.12, "ce": 0.007914566360341269, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.23000025749206543, "loss_scale": 4.0, "retries": 0, "seconds": 736.9, "max_memory_gb": 17.12, "ce": 0.018812592406902695, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 6.017022609710693, "loss_scale": 4.0, "retries": 0, "seconds": 742.9, "max_memory_gb": 17.12, "ce": 0.5973434596107836, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.39601966738700867, "loss_scale": 4.0, "retries": 0, "seconds": 748.0, "max_memory_gb": 17.12, "ce": 0.024773787650701706, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 1.170515537261963, "loss_scale": 4.0, "retries": 0, "seconds": 752.7, "max_memory_gb": 17.12, "ce": 0.1685128777553473, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 1.2861632108688354, "loss_scale": 4.0, "retries": 0, "seconds": 757.2, "max_memory_gb": 17.12, "ce": 0.1482666540741775, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.9552218317985535, "loss_scale": 4.0, "retries": 0, "seconds": 761.6, "max_memory_gb": 17.12, "ce": 0.05873192675426253, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 2.56320858001709, "loss_scale": 4.0, "retries": 0, "seconds": 766.3, "max_memory_gb": 17.12, "ce": 0.21411733271997946, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.8241234421730042, "loss_scale": 4.0, "retries": 0, "seconds": 770.9, "max_memory_gb": 17.12, "ce": 0.06539489516399044, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.1534876674413681, "loss_scale": 4.0, "retries": 0, "seconds": 776.6, "max_memory_gb": 17.12, "ce": 0.012309670999457012, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 2.9446356296539307, "loss_scale": 4.0, "retries": 0, "seconds": 781.7, "max_memory_gb": 17.12, "ce": 0.2537460562580236, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.23581278324127197, "loss_scale": 4.0, "retries": 0, "seconds": 786.3, "max_memory_gb": 17.12, "ce": 0.013992966025398346, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.08285266906023026, "loss_scale": 4.0, "retries": 0, "seconds": 791.0, "max_memory_gb": 17.12, "ce": 0.006803352327551693, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 1.2663018703460693, "loss_scale": 4.0, "retries": 0, "seconds": 795.7, "max_memory_gb": 17.12, "ce": 0.28487821559065196, "rank": 0.05032210052013397}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.07127250730991364, "loss_scale": 4.0, "retries": 0, "seconds": 800.2, "max_memory_gb": 17.12, "ce": 0.002191494175349362, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.5198721885681152, "loss_scale": 4.0, "retries": 0, "seconds": 804.8, "max_memory_gb": 17.12, "ce": 0.05904815638132277, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.1283380389213562, "loss_scale": 4.0, "retries": 0, "seconds": 809.4, "max_memory_gb": 17.12, "ce": 0.012000191754850675, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.21928933262825012, "loss_scale": 4.0, "retries": 0, "seconds": 814.0, "max_memory_gb": 17.12, "ce": 0.016765575415774947, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.2713588774204254, "loss_scale": 4.0, "retries": 0, "seconds": 818.3, "max_memory_gb": 17.12, "ce": 0.022144731945445528, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 2.2941665649414062, "loss_scale": 4.0, "retries": 0, "seconds": 822.8, "max_memory_gb": 17.12, "ce": 0.1022451396293036, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.046064820140600204, "loss_scale": 4.0, "retries": 0, "seconds": 827.6, "max_memory_gb": 17.12, "ce": 0.0027869910691151745, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 0.9422528147697449, "loss_scale": 4.0, "retries": 0, "seconds": 832.6, "max_memory_gb": 17.12, "ce": 0.2633738179356442, "rank": 0.1249089241027832}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 1.7531416416168213, "loss_scale": 4.0, "retries": 0, "seconds": 837.0, "max_memory_gb": 17.12, "ce": 0.10821881935407873, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 1.476615071296692, "loss_scale": 4.0, "retries": 0, "seconds": 841.2, "max_memory_gb": 17.12, "ce": 0.3597028391659478, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.3766648471355438, "loss_scale": 4.0, "retries": 0, "seconds": 846.0, "max_memory_gb": 17.12, "ce": 0.03442368183823419, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.2975331246852875, "loss_scale": 4.0, "retries": 0, "seconds": 850.7, "max_memory_gb": 17.12, "ce": 0.01835720300914545, "rank": 0.10199758410453796}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 1.1005377769470215, "loss_scale": 4.0, "retries": 0, "seconds": 855.2, "max_memory_gb": 17.12, "ce": 0.10859826169871667, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.038405437022447586, "loss_scale": 4.0, "retries": 0, "seconds": 860.0, "max_memory_gb": 17.12, "ce": 0.0023128497387006064, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.4015806317329407, "loss_scale": 4.0, "retries": 0, "seconds": 864.7, "max_memory_gb": 17.12, "ce": 0.015195938409306109, "rank": 0.12677280604839325}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.3529452085494995, "loss_scale": 4.0, "retries": 0, "seconds": 869.3, "max_memory_gb": 17.12, "ce": 0.042427931855854695, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 3.288980484008789, "loss_scale": 4.0, "retries": 0, "seconds": 874.5, "max_memory_gb": 17.12, "ce": 0.14733813603925228, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.5706399083137512, "loss_scale": 4.0, "retries": 0, "seconds": 879.2, "max_memory_gb": 17.12, "ce": 0.1339216844717157, "rank": 0.1352209448814392}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.7275830507278442, "loss_scale": 4.0, "retries": 0, "seconds": 883.7, "max_memory_gb": 17.12, "ce": 0.21641473462796057, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 3.6974143981933594, "loss_scale": 4.0, "retries": 0, "seconds": 888.4, "max_memory_gb": 17.12, "ce": 0.33084514396341547, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.18633092939853668, "loss_scale": 4.0, "retries": 0, "seconds": 893.1, "max_memory_gb": 17.12, "ce": 0.007621231023222208, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 2.4280683994293213, "loss_scale": 4.0, "retries": 0, "seconds": 897.6, "max_memory_gb": 17.12, "ce": 0.1405635388009614, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.3807693719863892, "loss_scale": 4.0, "retries": 0, "seconds": 902.1, "max_memory_gb": 17.12, "ce": 0.3190351658486179, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.01833043433725834, "loss_scale": 4.0, "retries": 0, "seconds": 906.9, "max_memory_gb": 17.12, "ce": 0.0012605274419001944, "rank": 0.12497183680534363}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.4967119097709656, "loss_scale": 4.0, "retries": 0, "seconds": 911.7, "max_memory_gb": 17.12, "ce": 0.04629750930780574, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.6357827186584473, "loss_scale": 4.0, "retries": 0, "seconds": 916.4, "max_memory_gb": 17.12, "ce": 0.054852857156220125, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.19369003176689148, "loss_scale": 4.0, "retries": 0, "seconds": 920.8, "max_memory_gb": 17.12, "ce": 0.01213630987876968, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 1.724447250366211, "loss_scale": 4.0, "retries": 0, "seconds": 925.4, "max_memory_gb": 17.12, "ce": 0.15906715331311716, "rank": 0.016070812940597534}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.6497434973716736, "loss_scale": 4.0, "retries": 0, "seconds": 930.1, "max_memory_gb": 17.12, "ce": 0.07523464049336326, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 1.176532506942749, "loss_scale": 4.0, "retries": 0, "seconds": 934.7, "max_memory_gb": 17.12, "ce": 0.11362451293643971, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 1.794536828994751, "loss_scale": 4.0, "retries": 0, "seconds": 939.4, "max_memory_gb": 17.12, "ce": 0.2602916902433208, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.8610201478004456, "loss_scale": 4.0, "retries": 0, "seconds": 944.1, "max_memory_gb": 17.12, "ce": 0.06485152796813054, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.042376961559057236, "loss_scale": 4.0, "retries": 0, "seconds": 948.6, "max_memory_gb": 17.12, "ce": 0.0027398597840146977, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.6338059306144714, "loss_scale": 4.0, "retries": 0, "seconds": 953.3, "max_memory_gb": 17.12, "ce": 0.018740473454272433, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.26835188269615173, "loss_scale": 4.0, "retries": 0, "seconds": 957.9, "max_memory_gb": 17.12, "ce": 0.020114681746235874, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 2.1289165019989014, "loss_scale": 4.0, "retries": 0, "seconds": 962.5, "max_memory_gb": 17.12, "ce": 0.13193021173083253, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.2324903905391693, "loss_scale": 4.0, "retries": 0, "seconds": 967.3, "max_memory_gb": 17.12, "ce": 0.016594705931311182, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.2752910554409027, "loss_scale": 4.0, "retries": 0, "seconds": 971.9, "max_memory_gb": 17.12, "ce": 0.056458752835169435, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.16262835264205933, "loss_scale": 4.0, "retries": 0, "seconds": 976.4, "max_memory_gb": 17.12, "ce": 0.006285107256189804, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.765999436378479, "loss_scale": 4.0, "retries": 0, "seconds": 980.9, "max_memory_gb": 17.12, "ce": 0.11687581093883637, "rank": 0.12520499527454376}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 2.278709650039673, "loss_scale": 4.0, "retries": 0, "seconds": 985.7, "max_memory_gb": 17.12, "ce": 0.49563141089038254, "rank": 0.0}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.665377676486969, "loss_scale": 4.0, "retries": 0, "seconds": 990.3, "max_memory_gb": 17.12, "ce": 0.11222350358730182, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 1.279664158821106, "loss_scale": 4.0, "retries": 0, "seconds": 994.9, "max_memory_gb": 17.12, "ce": 0.18776625012105796, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.7108903527259827, "loss_scale": 4.0, "retries": 0, "seconds": 999.4, "max_memory_gb": 17.12, "ce": 0.1476507587776723, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 1.0046626329421997, "loss_scale": 4.0, "retries": 0, "seconds": 1004.1, "max_memory_gb": 17.12, "ce": 0.21829014861759788, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 2.6395065784454346, "loss_scale": 4.0, "retries": 0, "seconds": 1008.6, "max_memory_gb": 17.12, "ce": 0.39740413308572897, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.4943912625312805, "loss_scale": 4.0, "retries": 0, "seconds": 1013.3, "max_memory_gb": 17.12, "ce": 0.032398596559687576, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.07067061960697174, "loss_scale": 4.0, "retries": 0, "seconds": 1018.0, "max_memory_gb": 17.12, "ce": 0.0030428149075305555, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 1.4451329708099365, "loss_scale": 4.0, "retries": 0, "seconds": 1023.2, "max_memory_gb": 17.12, "ce": 0.18343939871010662, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.3605383336544037, "loss_scale": 4.0, "retries": 0, "seconds": 1029.3, "max_memory_gb": 17.12, "ce": 0.034510930579926935, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.30121225118637085, "loss_scale": 4.0, "retries": 0, "seconds": 1035.0, "max_memory_gb": 17.12, "ce": 0.0211797994288645, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.39165011048316956, "loss_scale": 4.0, "retries": 0, "seconds": 1039.6, "max_memory_gb": 17.12, "ce": 0.03747304492071635, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.36361193656921387, "loss_scale": 4.0, "retries": 0, "seconds": 1044.2, "max_memory_gb": 17.12, "ce": 0.02932023587072763, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 1.3441956043243408, "loss_scale": 4.0, "retries": 0, "seconds": 1048.9, "max_memory_gb": 17.12, "ce": 0.047300122115302656, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.3438606262207031, "loss_scale": 4.0, "retries": 0, "seconds": 1054.0, "max_memory_gb": 17.12, "ce": 0.02792982170194591, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.16021573543548584, "loss_scale": 4.0, "retries": 0, "seconds": 1059.2, "max_memory_gb": 17.12, "ce": 0.01332091722451878, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.7525970339775085, "loss_scale": 4.0, "retries": 0, "seconds": 1064.4, "max_memory_gb": 17.12, "ce": 0.02984377264056093, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.8767707347869873, "loss_scale": 4.0, "retries": 0, "seconds": 1070.5, "max_memory_gb": 17.12, "ce": 0.02795846372100641, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.10184191912412643, "loss_scale": 4.0, "retries": 0, "seconds": 1075.6, "max_memory_gb": 17.12, "ce": 0.00872929378056142, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.5585289597511292, "loss_scale": 4.0, "retries": 0, "seconds": 1080.2, "max_memory_gb": 17.12, "ce": 0.04791138206837786, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 3.359096050262451, "loss_scale": 4.0, "retries": 0, "seconds": 1084.7, "max_memory_gb": 17.12, "ce": 0.3337946823303355, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.4934890568256378, "loss_scale": 4.0, "retries": 0, "seconds": 1089.1, "max_memory_gb": 17.12, "ce": 0.04636655112790322, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.3027906119823456, "loss_scale": 4.0, "retries": 0, "seconds": 1093.6, "max_memory_gb": 17.12, "ce": 0.02088684834734522, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.5573132038116455, "loss_scale": 4.0, "retries": 0, "seconds": 1098.2, "max_memory_gb": 17.12, "ce": 0.03121734000797005, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 1.6758642196655273, "loss_scale": 4.0, "retries": 0, "seconds": 1102.9, "max_memory_gb": 17.12, "ce": 0.07279039537752396, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 2.4523239135742188, "loss_scale": 4.0, "retries": 0, "seconds": 1107.5, "max_memory_gb": 17.12, "ce": 0.16875105348117359, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.20013116300106049, "loss_scale": 4.0, "retries": 0, "seconds": 1112.0, "max_memory_gb": 17.12, "ce": 0.013253085397082032, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 1.4112472534179688, "loss_scale": 4.0, "retries": 0, "seconds": 1116.6, "max_memory_gb": 17.12, "ce": 0.06320495449381269, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.5575615167617798, "loss_scale": 4.0, "retries": 0, "seconds": 1120.9, "max_memory_gb": 17.12, "ce": 0.03033530153788888, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.5899892449378967, "loss_scale": 4.0, "retries": 0, "seconds": 1125.1, "max_memory_gb": 17.12, "ce": 0.08444168698815702, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 0.4647599160671234, "loss_scale": 4.0, "retries": 0, "seconds": 1129.8, "max_memory_gb": 17.12, "ce": 0.03849682993586612, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.5364479422569275, "loss_scale": 4.0, "retries": 0, "seconds": 1134.4, "max_memory_gb": 17.12, "ce": 0.05401310291381378, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.13667933642864227, "loss_scale": 4.0, "retries": 0, "seconds": 1138.9, "max_memory_gb": 17.12, "ce": 0.010621792197525792, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.030448708683252335, "loss_scale": 4.0, "retries": 0, "seconds": 1143.7, "max_memory_gb": 17.12, "ce": 0.001313860587288218, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.6540173292160034, "loss_scale": 4.0, "retries": 0, "seconds": 1148.3, "max_memory_gb": 17.12, "ce": 0.03537506119937461, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 1.928240180015564, "loss_scale": 4.0, "retries": 0, "seconds": 1152.6, "max_memory_gb": 17.12, "ce": 0.25705014152572403, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.05382009223103523, "loss_scale": 4.0, "retries": 0, "seconds": 1157.1, "max_memory_gb": 17.12, "ce": 0.0015331801505453768, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.8356745839118958, "loss_scale": 4.0, "retries": 0, "seconds": 1161.4, "max_memory_gb": 17.12, "ce": 0.07636781763176259, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.12332363426685333, "loss_scale": 4.0, "retries": 0, "seconds": 1165.9, "max_memory_gb": 17.12, "ce": 0.008454265758700785, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.20013713836669922, "loss_scale": 4.0, "retries": 0, "seconds": 1170.3, "max_memory_gb": 17.12, "ce": 0.014056235692805785, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.1589430868625641, "loss_scale": 4.0, "retries": 0, "seconds": 1174.7, "max_memory_gb": 17.12, "ce": 0.0072529676244812435, "rank": 0.0}

runs/fixed_internvl/removal/Middle8_s42/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/removal/Middle8_s42/adapter/adapter_config.json

@@ -0,0 +1,213 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.14.mlp.down_proj": 144,
+    "model.language_model.layers.14.mlp.gate_proj": 144,
+    "model.language_model.layers.14.mlp.up_proj": 144,
+    "model.language_model.layers.14.self_attn.k_proj": 144,
+    "model.language_model.layers.14.self_attn.o_proj": 144,
+    "model.language_model.layers.14.self_attn.q_proj": 144,
+    "model.language_model.layers.14.self_attn.v_proj": 144,
+    "model.language_model.layers.15.mlp.down_proj": 144,
+    "model.language_model.layers.15.mlp.gate_proj": 144,
+    "model.language_model.layers.15.mlp.up_proj": 144,
+    "model.language_model.layers.15.self_attn.k_proj": 144,
+    "model.language_model.layers.15.self_attn.o_proj": 144,
+    "model.language_model.layers.15.self_attn.q_proj": 144,
+    "model.language_model.layers.15.self_attn.v_proj": 144,
+    "model.language_model.layers.16.mlp.down_proj": 144,
+    "model.language_model.layers.16.mlp.gate_proj": 144,
+    "model.language_model.layers.16.mlp.up_proj": 144,
+    "model.language_model.layers.16.self_attn.k_proj": 144,
+    "model.language_model.layers.16.self_attn.o_proj": 144,
+    "model.language_model.layers.16.self_attn.q_proj": 144,
+    "model.language_model.layers.16.self_attn.v_proj": 144,
+    "model.language_model.layers.17.mlp.down_proj": 144,
+    "model.language_model.layers.17.mlp.gate_proj": 144,
+    "model.language_model.layers.17.mlp.up_proj": 144,
+    "model.language_model.layers.17.self_attn.k_proj": 144,
+    "model.language_model.layers.17.self_attn.o_proj": 144,
+    "model.language_model.layers.17.self_attn.q_proj": 144,
+    "model.language_model.layers.17.self_attn.v_proj": 144,
+    "model.language_model.layers.18.mlp.down_proj": 144,
+    "model.language_model.layers.18.mlp.gate_proj": 144,
+    "model.language_model.layers.18.mlp.up_proj": 144,
+    "model.language_model.layers.18.self_attn.k_proj": 144,
+    "model.language_model.layers.18.self_attn.o_proj": 144,
+    "model.language_model.layers.18.self_attn.q_proj": 144,
+    "model.language_model.layers.18.self_attn.v_proj": 144,
+    "model.language_model.layers.19.mlp.down_proj": 144,
+    "model.language_model.layers.19.mlp.gate_proj": 144,
+    "model.language_model.layers.19.mlp.up_proj": 144,
+    "model.language_model.layers.19.self_attn.k_proj": 144,
+    "model.language_model.layers.19.self_attn.o_proj": 144,
+    "model.language_model.layers.19.self_attn.q_proj": 144,
+    "model.language_model.layers.19.self_attn.v_proj": 144,
+    "model.language_model.layers.20.mlp.down_proj": 144,
+    "model.language_model.layers.20.mlp.gate_proj": 144,
+    "model.language_model.layers.20.mlp.up_proj": 144,
+    "model.language_model.layers.20.self_attn.k_proj": 144,
+    "model.language_model.layers.20.self_attn.o_proj": 144,
+    "model.language_model.layers.20.self_attn.q_proj": 144,
+    "model.language_model.layers.20.self_attn.v_proj": 144,
+    "model.language_model.layers.21.mlp.down_proj": 144,
+    "model.language_model.layers.21.mlp.gate_proj": 144,
+    "model.language_model.layers.21.mlp.up_proj": 144,
+    "model.language_model.layers.21.self_attn.k_proj": 144,
+    "model.language_model.layers.21.self_attn.o_proj": 144,
+    "model.language_model.layers.21.self_attn.q_proj": 144,
+    "model.language_model.layers.21.self_attn.v_proj": 144
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.14.mlp.down_proj": 72,
+    "model.language_model.layers.14.mlp.gate_proj": 72,
+    "model.language_model.layers.14.mlp.up_proj": 72,
+    "model.language_model.layers.14.self_attn.k_proj": 72,
+    "model.language_model.layers.14.self_attn.o_proj": 72,
+    "model.language_model.layers.14.self_attn.q_proj": 72,
+    "model.language_model.layers.14.self_attn.v_proj": 72,
+    "model.language_model.layers.15.mlp.down_proj": 72,
+    "model.language_model.layers.15.mlp.gate_proj": 72,
+    "model.language_model.layers.15.mlp.up_proj": 72,
+    "model.language_model.layers.15.self_attn.k_proj": 72,
+    "model.language_model.layers.15.self_attn.o_proj": 72,
+    "model.language_model.layers.15.self_attn.q_proj": 72,
+    "model.language_model.layers.15.self_attn.v_proj": 72,
+    "model.language_model.layers.16.mlp.down_proj": 72,
+    "model.language_model.layers.16.mlp.gate_proj": 72,
+    "model.language_model.layers.16.mlp.up_proj": 72,
+    "model.language_model.layers.16.self_attn.k_proj": 72,
+    "model.language_model.layers.16.self_attn.o_proj": 72,
+    "model.language_model.layers.16.self_attn.q_proj": 72,
+    "model.language_model.layers.16.self_attn.v_proj": 72,
+    "model.language_model.layers.17.mlp.down_proj": 72,
+    "model.language_model.layers.17.mlp.gate_proj": 72,
+    "model.language_model.layers.17.mlp.up_proj": 72,
+    "model.language_model.layers.17.self_attn.k_proj": 72,
+    "model.language_model.layers.17.self_attn.o_proj": 72,
+    "model.language_model.layers.17.self_attn.q_proj": 72,
+    "model.language_model.layers.17.self_attn.v_proj": 72,
+    "model.language_model.layers.18.mlp.down_proj": 72,
+    "model.language_model.layers.18.mlp.gate_proj": 72,
+    "model.language_model.layers.18.mlp.up_proj": 72,
+    "model.language_model.layers.18.self_attn.k_proj": 72,
+    "model.language_model.layers.18.self_attn.o_proj": 72,
+    "model.language_model.layers.18.self_attn.q_proj": 72,
+    "model.language_model.layers.18.self_attn.v_proj": 72,
+    "model.language_model.layers.19.mlp.down_proj": 72,
+    "model.language_model.layers.19.mlp.gate_proj": 72,
+    "model.language_model.layers.19.mlp.up_proj": 72,
+    "model.language_model.layers.19.self_attn.k_proj": 72,
+    "model.language_model.layers.19.self_attn.o_proj": 72,
+    "model.language_model.layers.19.self_attn.q_proj": 72,
+    "model.language_model.layers.19.self_attn.v_proj": 72,
+    "model.language_model.layers.20.mlp.down_proj": 72,
+    "model.language_model.layers.20.mlp.gate_proj": 72,
+    "model.language_model.layers.20.mlp.up_proj": 72,
+    "model.language_model.layers.20.self_attn.k_proj": 72,
+    "model.language_model.layers.20.self_attn.o_proj": 72,
+    "model.language_model.layers.20.self_attn.q_proj": 72,
+    "model.language_model.layers.20.self_attn.v_proj": 72,
+    "model.language_model.layers.21.mlp.down_proj": 72,
+    "model.language_model.layers.21.mlp.gate_proj": 72,
+    "model.language_model.layers.21.mlp.up_proj": 72,
+    "model.language_model.layers.21.self_attn.k_proj": 72,
+    "model.language_model.layers.21.self_attn.o_proj": 72,
+    "model.language_model.layers.21.self_attn.q_proj": 72,
+    "model.language_model.layers.21.self_attn.v_proj": 72
+  },
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.15.self_attn.q_proj",
+    "model.language_model.layers.21.self_attn.o_proj",
+    "model.language_model.layers.17.mlp.up_proj",
+    "model.language_model.layers.18.self_attn.v_proj",
+    "model.language_model.layers.14.self_attn.q_proj",
+    "model.language_model.layers.19.self_attn.o_proj",
+    "model.language_model.layers.20.self_attn.q_proj",
+    "model.language_model.layers.16.mlp.up_proj",
+    "model.language_model.layers.17.self_attn.v_proj",
+    "model.language_model.layers.17.mlp.down_proj",
+    "model.language_model.layers.21.self_attn.k_proj",
+    "model.language_model.layers.19.self_attn.v_proj",
+    "model.language_model.layers.16.mlp.gate_proj",
+    "model.language_model.layers.14.self_attn.v_proj",
+    "model.language_model.layers.19.mlp.gate_proj",
+    "model.language_model.layers.16.self_attn.k_proj",
+    "model.language_model.layers.19.self_attn.k_proj",
+    "model.language_model.layers.15.self_attn.v_proj",
+    "model.language_model.layers.18.mlp.up_proj",
+    "model.language_model.layers.15.mlp.gate_proj",
+    "model.language_model.layers.18.self_attn.q_proj",
+    "model.language_model.layers.21.self_attn.q_proj",
+    "model.language_model.layers.16.self_attn.q_proj",
+    "model.language_model.layers.17.mlp.gate_proj",
+    "model.language_model.layers.20.mlp.down_proj",
+    "model.language_model.layers.16.self_attn.o_proj",
+    "model.language_model.layers.15.mlp.down_proj",
+    "model.language_model.layers.14.self_attn.o_proj",
+    "model.language_model.layers.20.self_attn.o_proj",
+    "model.language_model.layers.18.mlp.down_proj",
+    "model.language_model.layers.19.mlp.up_proj",
+    "model.language_model.layers.14.mlp.down_proj",
+    "model.language_model.layers.21.self_attn.v_proj",
+    "model.language_model.layers.14.mlp.up_proj",
+    "model.language_model.layers.20.mlp.up_proj",
+    "model.language_model.layers.17.self_attn.k_proj",
+    "model.language_model.layers.18.mlp.gate_proj",
+    "model.language_model.layers.15.self_attn.o_proj",
+    "model.language_model.layers.21.mlp.gate_proj",
+    "model.language_model.layers.14.mlp.gate_proj",
+    "model.language_model.layers.19.mlp.down_proj",
+    "model.language_model.layers.18.self_attn.k_proj",
+    "model.language_model.layers.20.self_attn.v_proj",
+    "model.language_model.layers.15.mlp.up_proj",
+    "model.language_model.layers.19.self_attn.q_proj",
+    "model.language_model.layers.15.self_attn.k_proj",
+    "model.language_model.layers.21.mlp.down_proj",
+    "model.language_model.layers.16.self_attn.v_proj",
+    "model.language_model.layers.14.self_attn.k_proj",
+    "model.language_model.layers.17.self_attn.q_proj",
+    "model.language_model.layers.17.self_attn.o_proj",
+    "model.language_model.layers.20.self_attn.k_proj",
+    "model.language_model.layers.16.mlp.down_proj",
+    "model.language_model.layers.18.self_attn.o_proj",
+    "model.language_model.layers.20.mlp.gate_proj",
+    "model.language_model.layers.21.mlp.up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/fixed_internvl/removal/Middle8_s42/config.json

@@ -0,0 +1,43 @@
+{
+  "behavior": "removal",
+  "seed": 42,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "world_size": 1,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "arm": {
+    "arm": "Middle8",
+    "layers": [
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21
+    ],
+    "ranks": {
+      "14": 72,
+      "15": 72,
+      "16": 72,
+      "17": 72,
+      "18": 72,
+      "19": 72,
+      "20": 72,
+      "21": 72
+    },
+    "parameters": 43646976,
+    "budget": 43646976
+  }
+}

runs/fixed_internvl/removal/Middle8_s42/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 14.541537284851074, "loss_scale": 4.0, "retries": 0, "seconds": 4.2, "max_memory_gb": 16.79, "ce": 1.0494785578921437, "rank": 0.15784265100955963}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 8.96788501739502, "loss_scale": 4.0, "retries": 0, "seconds": 7.8, "max_memory_gb": 17.11, "ce": 0.8133026539934463, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 20.02604866027832, "loss_scale": 4.0, "retries": 0, "seconds": 11.2, "max_memory_gb": 17.11, "ce": 0.7390169245772995, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 9.645132064819336, "loss_scale": 4.0, "retries": 0, "seconds": 14.6, "max_memory_gb": 17.11, "ce": 0.5868777836731169, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 2.31530499458313, "loss_scale": 4.0, "retries": 0, "seconds": 18.5, "max_memory_gb": 17.11, "ce": 0.5174583586049266, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 2.984994649887085, "loss_scale": 4.0, "retries": 0, "seconds": 22.3, "max_memory_gb": 17.11, "ce": 0.29493023874238133, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 2.9234676361083984, "loss_scale": 4.0, "retries": 0, "seconds": 26.0, "max_memory_gb": 17.11, "ce": 0.49851335072889924, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 2.171297550201416, "loss_scale": 4.0, "retries": 0, "seconds": 29.9, "max_memory_gb": 17.11, "ce": 0.24360183298745142, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 3.7415268421173096, "loss_scale": 4.0, "retries": 0, "seconds": 33.6, "max_memory_gb": 17.11, "ce": 0.5802769715082832, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 1.7819887399673462, "loss_scale": 4.0, "retries": 0, "seconds": 37.4, "max_memory_gb": 17.11, "ce": 0.23427223625185434, "rank": 0.20456336438655853}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.3776566684246063, "loss_scale": 4.0, "retries": 0, "seconds": 41.1, "max_memory_gb": 17.11, "ce": 0.14012424186603312, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 5.733058929443359, "loss_scale": 4.0, "retries": 0, "seconds": 45.0, "max_memory_gb": 17.11, "ce": 0.584752653092437, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 1.127847671508789, "loss_scale": 4.0, "retries": 0, "seconds": 48.8, "max_memory_gb": 17.11, "ce": 0.36304191139061004, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 0.5526084899902344, "loss_scale": 4.0, "retries": 0, "seconds": 52.3, "max_memory_gb": 17.11, "ce": 0.3122620893700514, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 7.160830497741699, "loss_scale": 4.0, "retries": 0, "seconds": 55.7, "max_memory_gb": 17.11, "ce": 0.4936718854587525, "rank": 0.18133649230003357}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 1.1743067502975464, "loss_scale": 4.0, "retries": 0, "seconds": 59.1, "max_memory_gb": 17.11, "ce": 0.30679665748175466, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 0.8415310382843018, "loss_scale": 4.0, "retries": 0, "seconds": 62.5, "max_memory_gb": 17.11, "ce": 0.21328609209740534, "rank": 0.09352219104766846}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 2.1498196125030518, "loss_scale": 4.0, "retries": 0, "seconds": 66.1, "max_memory_gb": 17.11, "ce": 0.2635482484474778, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 0.8312537670135498, "loss_scale": 4.0, "retries": 0, "seconds": 69.4, "max_memory_gb": 17.11, "ce": 0.09258844379655784, "rank": 0.0}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 2.205714225769043, "loss_scale": 4.0, "retries": 0, "seconds": 72.9, "max_memory_gb": 17.11, "ce": 0.14080483512952924, "rank": 0.0}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.31541988253593445, "loss_scale": 4.0, "retries": 0, "seconds": 76.2, "max_memory_gb": 17.11, "ce": 0.019463941556750797, "rank": 0.0}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 2.6443772315979004, "loss_scale": 4.0, "retries": 0, "seconds": 79.7, "max_memory_gb": 17.12, "ce": 0.1507583865518427, "rank": 0.0}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.4927879571914673, "loss_scale": 4.0, "retries": 0, "seconds": 83.1, "max_memory_gb": 17.12, "ce": 0.11900094597149291, "rank": 0.0}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 1.9802536964416504, "loss_scale": 4.0, "retries": 0, "seconds": 86.5, "max_memory_gb": 17.12, "ce": 0.19917862407965004, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 0.6121885776519775, "loss_scale": 4.0, "retries": 0, "seconds": 90.0, "max_memory_gb": 17.12, "ce": 0.027092723043097067, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 2.4765946865081787, "loss_scale": 4.0, "retries": 0, "seconds": 93.4, "max_memory_gb": 17.12, "ce": 0.04895234927130332, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 12.856739044189453, "loss_scale": 4.0, "retries": 0, "seconds": 96.8, "max_memory_gb": 17.12, "ce": 0.4895435199653093, "rank": 0.0}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 2.2313573360443115, "loss_scale": 4.0, "retries": 0, "seconds": 100.2, "max_memory_gb": 17.12, "ce": 0.18052522777725244, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 4.938350677490234, "loss_scale": 4.0, "retries": 0, "seconds": 103.7, "max_memory_gb": 17.12, "ce": 0.11324438370138523, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 6.133568286895752, "loss_scale": 4.0, "retries": 0, "seconds": 107.2, "max_memory_gb": 17.12, "ce": 0.14285584549725172, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 9.230165481567383, "loss_scale": 4.0, "retries": 0, "seconds": 110.6, "max_memory_gb": 17.12, "ce": 0.6423275995784934, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 2.822807550430298, "loss_scale": 4.0, "retries": 0, "seconds": 114.1, "max_memory_gb": 17.12, "ce": 0.04534477332344977, "rank": 0.0}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.24899141490459442, "loss_scale": 4.0, "retries": 0, "seconds": 117.4, "max_memory_gb": 17.12, "ce": 0.004800589715614478, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.7313058376312256, "loss_scale": 4.0, "retries": 0, "seconds": 120.8, "max_memory_gb": 17.12, "ce": 0.014170943285989779, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 2.0181357860565186, "loss_scale": 4.0, "retries": 0, "seconds": 124.4, "max_memory_gb": 17.12, "ce": 0.10679222552789724, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 2.216264486312866, "loss_scale": 4.0, "retries": 0, "seconds": 127.8, "max_memory_gb": 17.12, "ce": 0.27487532524992275, "rank": 0.027888208627700806}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 1.3946465253829956, "loss_scale": 4.0, "retries": 0, "seconds": 131.2, "max_memory_gb": 17.12, "ce": 0.1401438775301358, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.23155851662158966, "loss_scale": 4.0, "retries": 0, "seconds": 134.7, "max_memory_gb": 17.12, "ce": 0.007492425553209614, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 4.149398326873779, "loss_scale": 4.0, "retries": 0, "seconds": 138.4, "max_memory_gb": 17.12, "ce": 0.2890280952469766, "rank": 0.0}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.25956371426582336, "loss_scale": 4.0, "retries": 0, "seconds": 142.2, "max_memory_gb": 17.12, "ce": 0.00567680763538192, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.19365662336349487, "loss_scale": 4.0, "retries": 0, "seconds": 145.9, "max_memory_gb": 17.12, "ce": 0.004838030873997923, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 2.333667278289795, "loss_scale": 4.0, "retries": 0, "seconds": 149.8, "max_memory_gb": 17.12, "ce": 0.22029744354040304, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 1.7096751928329468, "loss_scale": 4.0, "retries": 0, "seconds": 153.5, "max_memory_gb": 17.12, "ce": 0.049995414694421925, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 4.911208152770996, "loss_scale": 4.0, "retries": 0, "seconds": 157.2, "max_memory_gb": 17.12, "ce": 0.30202876671592094, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 4.968907356262207, "loss_scale": 4.0, "retries": 0, "seconds": 161.1, "max_memory_gb": 17.12, "ce": 0.4281510868768237, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 2.243736982345581, "loss_scale": 4.0, "retries": 0, "seconds": 164.8, "max_memory_gb": 17.12, "ce": 0.2536872363434668, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.2903687059879303, "loss_scale": 4.0, "retries": 0, "seconds": 168.6, "max_memory_gb": 17.12, "ce": 0.006058402381313499, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 1.1232746839523315, "loss_scale": 4.0, "retries": 0, "seconds": 172.3, "max_memory_gb": 17.12, "ce": 0.12978568230028031, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 6.718685626983643, "loss_scale": 4.0, "retries": 0, "seconds": 176.3, "max_memory_gb": 17.12, "ce": 0.16500543351867236, "rank": 0.12496959418058395}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.8654673099517822, "loss_scale": 4.0, "retries": 0, "seconds": 180.0, "max_memory_gb": 17.12, "ce": 0.02855017172259977, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 1.9109145402908325, "loss_scale": 4.0, "retries": 0, "seconds": 183.8, "max_memory_gb": 17.12, "ce": 0.07909437039961631, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.20244036614894867, "loss_scale": 4.0, "retries": 0, "seconds": 187.7, "max_memory_gb": 17.12, "ce": 0.005722835407141247, "rank": 0.1254684180021286}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 0.4079347550868988, "loss_scale": 4.0, "retries": 0, "seconds": 191.5, "max_memory_gb": 17.12, "ce": 0.022068533764468157, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 2.867194414138794, "loss_scale": 4.0, "retries": 0, "seconds": 195.2, "max_memory_gb": 17.12, "ce": 0.1560627202343312, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 13.277656555175781, "loss_scale": 4.0, "retries": 0, "seconds": 199.1, "max_memory_gb": 17.12, "ce": 0.3354476106032962, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.29240286350250244, "loss_scale": 4.0, "retries": 0, "seconds": 202.9, "max_memory_gb": 17.12, "ce": 0.00947932746043989, "rank": 0.0}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.591690182685852, "loss_scale": 4.0, "retries": 0, "seconds": 206.6, "max_memory_gb": 17.12, "ce": 0.039080410917449626, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 2.015712261199951, "loss_scale": 4.0, "retries": 0, "seconds": 210.5, "max_memory_gb": 17.12, "ce": 0.15971108456324146, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.38670673966407776, "loss_scale": 4.0, "retries": 0, "seconds": 214.4, "max_memory_gb": 17.12, "ce": 0.020472049345698906, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 1.6341631412506104, "loss_scale": 4.0, "retries": 0, "seconds": 218.1, "max_memory_gb": 17.12, "ce": 0.20097212938344455, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 0.3748232424259186, "loss_scale": 4.0, "retries": 0, "seconds": 221.9, "max_memory_gb": 17.12, "ce": 0.028797668961487943, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.4355652332305908, "loss_scale": 4.0, "retries": 0, "seconds": 225.9, "max_memory_gb": 17.12, "ce": 0.04517901404597069, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 4.1201252937316895, "loss_scale": 4.0, "retries": 0, "seconds": 229.7, "max_memory_gb": 17.12, "ce": 0.2957993225172686, "rank": 0.0}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.7569916248321533, "loss_scale": 4.0, "retries": 0, "seconds": 233.5, "max_memory_gb": 17.12, "ce": 0.057569689302908955, "rank": 0.0}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 5.2015790939331055, "loss_scale": 4.0, "retries": 0, "seconds": 237.0, "max_memory_gb": 17.12, "ce": 0.12332788948333473, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 6.800431728363037, "loss_scale": 4.0, "retries": 0, "seconds": 240.6, "max_memory_gb": 17.12, "ce": 0.09092592473371042, "rank": 0.0}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.26591092348098755, "loss_scale": 4.0, "retries": 0, "seconds": 244.0, "max_memory_gb": 17.12, "ce": 0.016109110723846243, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.2740042209625244, "loss_scale": 4.0, "retries": 0, "seconds": 247.4, "max_memory_gb": 17.12, "ce": 0.015096242786967196, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.7181020379066467, "loss_scale": 4.0, "retries": 0, "seconds": 251.0, "max_memory_gb": 17.12, "ce": 0.04990944235760253, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 2.8510658740997314, "loss_scale": 4.0, "retries": 0, "seconds": 254.4, "max_memory_gb": 17.12, "ce": 0.26992870531466906, "rank": 0.0}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 5.713971138000488, "loss_scale": 4.0, "retries": 0, "seconds": 257.8, "max_memory_gb": 17.12, "ce": 0.5542462930388865, "rank": 0.12340590357780457}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.040803924202919006, "loss_scale": 4.0, "retries": 0, "seconds": 261.4, "max_memory_gb": 17.12, "ce": 0.0023449512855222565, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 4.8157124519348145, "loss_scale": 4.0, "retries": 0, "seconds": 264.8, "max_memory_gb": 17.12, "ce": 0.498045371596163, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.46395188570022583, "loss_scale": 4.0, "retries": 0, "seconds": 268.3, "max_memory_gb": 17.12, "ce": 0.02185292174181086, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.472908616065979, "loss_scale": 4.0, "retries": 0, "seconds": 271.8, "max_memory_gb": 17.12, "ce": 0.024248096440715017, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 0.9698245525360107, "loss_scale": 4.0, "retries": 0, "seconds": 275.3, "max_memory_gb": 17.12, "ce": 0.03198768215952441, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.5124938488006592, "loss_scale": 4.0, "retries": 0, "seconds": 278.7, "max_memory_gb": 17.12, "ce": 0.03963797273218006, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.2350018471479416, "loss_scale": 4.0, "retries": 0, "seconds": 282.2, "max_memory_gb": 17.12, "ce": 0.021178163467993727, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.8547150492668152, "loss_scale": 4.0, "retries": 0, "seconds": 285.7, "max_memory_gb": 17.12, "ce": 0.08934835627565008, "rank": 0.0}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 1.1975147724151611, "loss_scale": 4.0, "retries": 0, "seconds": 289.1, "max_memory_gb": 17.12, "ce": 0.15943603675805207, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 4.763069152832031, "loss_scale": 4.0, "retries": 0, "seconds": 292.6, "max_memory_gb": 17.12, "ce": 0.6538134568963869, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.41289931535720825, "loss_scale": 4.0, "retries": 0, "seconds": 296.1, "max_memory_gb": 17.12, "ce": 0.029496874965843745, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 1.0521435737609863, "loss_scale": 4.0, "retries": 0, "seconds": 299.6, "max_memory_gb": 17.12, "ce": 0.08596458953797992, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 1.3792709112167358, "loss_scale": 4.0, "retries": 0, "seconds": 303.0, "max_memory_gb": 17.12, "ce": 0.15143259840078827, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.27058151364326477, "loss_scale": 4.0, "retries": 0, "seconds": 306.5, "max_memory_gb": 17.12, "ce": 0.027843990876363023, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 2.9621386528015137, "loss_scale": 4.0, "retries": 0, "seconds": 310.0, "max_memory_gb": 17.12, "ce": 0.10030020839076315, "rank": 0.12493292987346649}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.45052430033683777, "loss_scale": 4.0, "retries": 0, "seconds": 313.4, "max_memory_gb": 17.12, "ce": 0.030032581288651272, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 3.4056777954101562, "loss_scale": 4.0, "retries": 0, "seconds": 316.9, "max_memory_gb": 17.12, "ce": 0.12424224735582357, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.07691792398691177, "loss_scale": 4.0, "retries": 0, "seconds": 320.5, "max_memory_gb": 17.12, "ce": 0.004409285937072127, "rank": 0.0}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.009955656714737415, "loss_scale": 4.0, "retries": 0, "seconds": 324.3, "max_memory_gb": 17.12, "ce": 0.0006204262749633926, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 0.7597697973251343, "loss_scale": 4.0, "retries": 0, "seconds": 328.1, "max_memory_gb": 17.12, "ce": 0.02205229552782839, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 0.3531816005706787, "loss_scale": 4.0, "retries": 0, "seconds": 332.0, "max_memory_gb": 17.12, "ce": 0.02394968009502918, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 2.498377561569214, "loss_scale": 4.0, "retries": 0, "seconds": 335.9, "max_memory_gb": 17.12, "ce": 0.12437466926348861, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 1.533186435699463, "loss_scale": 4.0, "retries": 0, "seconds": 339.6, "max_memory_gb": 17.12, "ce": 0.23271170066618652, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 1.2815380096435547, "loss_scale": 4.0, "retries": 0, "seconds": 343.4, "max_memory_gb": 17.12, "ce": 0.04125910512084374, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 0.5794828534126282, "loss_scale": 4.0, "retries": 0, "seconds": 347.3, "max_memory_gb": 17.12, "ce": 0.08381546729015099, "rank": 0.0}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.5001173615455627, "loss_scale": 4.0, "retries": 0, "seconds": 351.1, "max_memory_gb": 17.12, "ce": 0.02210674229718279, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.179878368973732, "loss_scale": 4.0, "retries": 0, "seconds": 354.9, "max_memory_gb": 17.12, "ce": 0.007706421482282622, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 4.916553497314453, "loss_scale": 4.0, "retries": 0, "seconds": 358.8, "max_memory_gb": 17.12, "ce": 0.17386809079494014, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.05304969102144241, "loss_scale": 4.0, "retries": 0, "seconds": 362.6, "max_memory_gb": 17.12, "ce": 0.0028449725291466166, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 2.7099974155426025, "loss_scale": 4.0, "retries": 0, "seconds": 366.4, "max_memory_gb": 17.12, "ce": 0.18473561429661345, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.04916379973292351, "loss_scale": 4.0, "retries": 0, "seconds": 370.2, "max_memory_gb": 17.12, "ce": 0.001832991419732366, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 3.785292863845825, "loss_scale": 4.0, "retries": 0, "seconds": 374.0, "max_memory_gb": 17.12, "ce": 0.3987406776050193, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 1.1480904817581177, "loss_scale": 4.0, "retries": 0, "seconds": 377.8, "max_memory_gb": 17.12, "ce": 0.08910957085481641, "rank": 0.0}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 3.521522045135498, "loss_scale": 4.0, "retries": 0, "seconds": 381.5, "max_memory_gb": 17.12, "ce": 0.12548035755344245, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 4.489286422729492, "loss_scale": 4.0, "retries": 0, "seconds": 385.4, "max_memory_gb": 17.12, "ce": 0.3547712325107568, "rank": 0.0}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 1.5518380403518677, "loss_scale": 4.0, "retries": 0, "seconds": 389.1, "max_memory_gb": 17.12, "ce": 0.1611893923338812, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 2.268834114074707, "loss_scale": 4.0, "retries": 0, "seconds": 392.9, "max_memory_gb": 17.12, "ce": 0.3277679214570526, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 0.7800321578979492, "loss_scale": 4.0, "retries": 0, "seconds": 396.8, "max_memory_gb": 17.12, "ce": 0.13695543330322835, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.4732320308685303, "loss_scale": 4.0, "retries": 0, "seconds": 400.5, "max_memory_gb": 17.12, "ce": 0.013175944253816851, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.015825588256120682, "loss_scale": 4.0, "retries": 0, "seconds": 404.2, "max_memory_gb": 17.12, "ce": 0.000802815945007751, "rank": 0.12502026557922363}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 1.6720088720321655, "loss_scale": 4.0, "retries": 0, "seconds": 408.1, "max_memory_gb": 17.12, "ce": 0.08434325046664526, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 2.0539984703063965, "loss_scale": 4.0, "retries": 0, "seconds": 411.8, "max_memory_gb": 17.12, "ce": 0.4847859986457479, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.0909927636384964, "loss_scale": 4.0, "retries": 0, "seconds": 415.6, "max_memory_gb": 17.12, "ce": 0.004560198018225492, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 7.868557929992676, "loss_scale": 4.0, "retries": 0, "seconds": 419.3, "max_memory_gb": 17.12, "ce": 0.40439860018796026, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.5828046202659607, "loss_scale": 4.0, "retries": 0, "seconds": 423.2, "max_memory_gb": 17.12, "ce": 0.017542152738315053, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 3.041454553604126, "loss_scale": 4.0, "retries": 0, "seconds": 427.0, "max_memory_gb": 17.12, "ce": 0.29021052372672784, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 1.6794503927230835, "loss_scale": 4.0, "retries": 0, "seconds": 430.8, "max_memory_gb": 17.12, "ce": 0.27677460570566836, "rank": 0.12501397728919983}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.2571784257888794, "loss_scale": 4.0, "retries": 0, "seconds": 434.6, "max_memory_gb": 17.12, "ce": 0.016421160383970346, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 2.1841073036193848, "loss_scale": 4.0, "retries": 0, "seconds": 438.4, "max_memory_gb": 17.12, "ce": 0.0960537663265768, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.9100566506385803, "loss_scale": 4.0, "retries": 0, "seconds": 442.1, "max_memory_gb": 17.12, "ce": 0.06183164560343357, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.3592011332511902, "loss_scale": 4.0, "retries": 0, "seconds": 445.9, "max_memory_gb": 17.12, "ce": 0.025952834558665927, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.04829680174589157, "loss_scale": 4.0, "retries": 0, "seconds": 449.7, "max_memory_gb": 17.12, "ce": 0.0028724458661599783, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.8323827981948853, "loss_scale": 4.0, "retries": 0, "seconds": 453.5, "max_memory_gb": 17.12, "ce": 0.15088534693586553, "rank": 0.23424921929836273}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.2642148733139038, "loss_scale": 4.0, "retries": 0, "seconds": 457.2, "max_memory_gb": 17.12, "ce": 0.007575976661428285, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 1.6155385971069336, "loss_scale": 4.0, "retries": 0, "seconds": 461.1, "max_memory_gb": 17.12, "ce": 0.33743029417564685, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.39862605929374695, "loss_scale": 4.0, "retries": 0, "seconds": 464.8, "max_memory_gb": 17.12, "ce": 0.04238350663490564, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 0.6129640340805054, "loss_scale": 4.0, "retries": 0, "seconds": 468.5, "max_memory_gb": 17.12, "ce": 0.04773070863302564, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 3.167983293533325, "loss_scale": 4.0, "retries": 0, "seconds": 472.3, "max_memory_gb": 17.12, "ce": 0.16331867713779502, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 0.509170413017273, "loss_scale": 4.0, "retries": 0, "seconds": 476.1, "max_memory_gb": 17.12, "ce": 0.0376794184821847, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 0.4503350555896759, "loss_scale": 4.0, "retries": 0, "seconds": 479.8, "max_memory_gb": 17.12, "ce": 0.12076325954512868, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 0.026098258793354034, "loss_scale": 4.0, "retries": 0, "seconds": 483.5, "max_memory_gb": 17.12, "ce": 0.001554713820496545, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 0.22518105804920197, "loss_scale": 4.0, "retries": 0, "seconds": 487.3, "max_memory_gb": 17.12, "ce": 0.012455858340217674, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 1.8216705322265625, "loss_scale": 4.0, "retries": 0, "seconds": 491.1, "max_memory_gb": 17.12, "ce": 0.0975262659458167, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 1.4964570999145508, "loss_scale": 4.0, "retries": 0, "seconds": 494.8, "max_memory_gb": 17.12, "ce": 0.31045624232137925, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 0.1890372484922409, "loss_scale": 4.0, "retries": 0, "seconds": 498.7, "max_memory_gb": 17.12, "ce": 0.007713438434620912, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 2.9306604862213135, "loss_scale": 4.0, "retries": 0, "seconds": 502.4, "max_memory_gb": 17.12, "ce": 0.3803698268775406, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 0.6924198269844055, "loss_scale": 4.0, "retries": 0, "seconds": 506.2, "max_memory_gb": 17.12, "ce": 0.024495906644006027, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 2.6191318035125732, "loss_scale": 4.0, "retries": 0, "seconds": 510.6, "max_memory_gb": 17.12, "ce": 0.20662156806520215, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.07476934790611267, "loss_scale": 4.0, "retries": 0, "seconds": 515.2, "max_memory_gb": 17.12, "ce": 0.004474112348361814, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.7300732731819153, "loss_scale": 4.0, "retries": 0, "seconds": 519.6, "max_memory_gb": 17.12, "ce": 0.05591565087070194, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 0.19454945623874664, "loss_scale": 4.0, "retries": 0, "seconds": 524.0, "max_memory_gb": 17.12, "ce": 0.015589083430313622, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 1.4917014837265015, "loss_scale": 4.0, "retries": 0, "seconds": 528.6, "max_memory_gb": 17.12, "ce": 0.17478872568153747, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.5313669443130493, "loss_scale": 4.0, "retries": 0, "seconds": 532.3, "max_memory_gb": 17.12, "ce": 0.05087837052542454, "rank": 0.1254391223192215}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.07154972106218338, "loss_scale": 4.0, "retries": 0, "seconds": 536.0, "max_memory_gb": 17.12, "ce": 0.005750526132487721, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.21453028917312622, "loss_scale": 4.0, "retries": 0, "seconds": 539.9, "max_memory_gb": 17.12, "ce": 0.008178615434189851, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 1.0416004657745361, "loss_scale": 4.0, "retries": 0, "seconds": 543.6, "max_memory_gb": 17.12, "ce": 0.06097328925898182, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 1.9121555089950562, "loss_scale": 4.0, "retries": 0, "seconds": 547.3, "max_memory_gb": 17.12, "ce": 0.23299736361695977, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 2.5962412357330322, "loss_scale": 4.0, "retries": 0, "seconds": 551.2, "max_memory_gb": 17.12, "ce": 0.303363752174846, "rank": 0.0}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.7505083084106445, "loss_scale": 4.0, "retries": 0, "seconds": 554.9, "max_memory_gb": 17.12, "ce": 0.05267626056399877, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.039222270250320435, "loss_scale": 4.0, "retries": 0, "seconds": 558.7, "max_memory_gb": 17.12, "ce": 0.002328943473912659, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 2.841959238052368, "loss_scale": 4.0, "retries": 0, "seconds": 562.4, "max_memory_gb": 17.12, "ce": 0.08401554352803942, "rank": 0.0797567218542099}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.10904309898614883, "loss_scale": 4.0, "retries": 0, "seconds": 566.3, "max_memory_gb": 17.12, "ce": 0.007654869633370254, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.07614351063966751, "loss_scale": 4.0, "retries": 0, "seconds": 570.0, "max_memory_gb": 17.12, "ce": 0.005211457310906553, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 0.3949647843837738, "loss_scale": 4.0, "retries": 0, "seconds": 573.7, "max_memory_gb": 17.12, "ce": 0.10592243042265181, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 1.2997181415557861, "loss_scale": 4.0, "retries": 0, "seconds": 578.0, "max_memory_gb": 17.12, "ce": 0.10340794496823946, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.204217791557312, "loss_scale": 4.0, "retries": 0, "seconds": 582.5, "max_memory_gb": 17.12, "ce": 0.0060425823294281145, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 0.575056254863739, "loss_scale": 4.0, "retries": 0, "seconds": 586.6, "max_memory_gb": 17.12, "ce": 0.0710965514599593, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.11737531423568726, "loss_scale": 4.0, "retries": 0, "seconds": 590.4, "max_memory_gb": 17.12, "ce": 0.007592912640575378, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 3.091353178024292, "loss_scale": 4.0, "retries": 0, "seconds": 594.3, "max_memory_gb": 17.12, "ce": 0.14829067626305914, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 2.6969823837280273, "loss_scale": 4.0, "retries": 0, "seconds": 598.0, "max_memory_gb": 17.12, "ce": 0.12734174412071297, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.8149659037590027, "loss_scale": 4.0, "retries": 0, "seconds": 601.8, "max_memory_gb": 17.12, "ce": 0.014912018945324235, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 0.390199214220047, "loss_scale": 4.0, "retries": 0, "seconds": 605.6, "max_memory_gb": 17.12, "ce": 0.05954391159411898, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 1.2757987976074219, "loss_scale": 4.0, "retries": 0, "seconds": 609.4, "max_memory_gb": 17.12, "ce": 0.15415272496829857, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.16065733134746552, "loss_scale": 4.0, "retries": 0, "seconds": 613.1, "max_memory_gb": 17.12, "ce": 0.00860152448694862, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 0.20097963511943817, "loss_scale": 4.0, "retries": 0, "seconds": 616.9, "max_memory_gb": 17.12, "ce": 0.011359702761183144, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.07530168443918228, "loss_scale": 4.0, "retries": 0, "seconds": 620.7, "max_memory_gb": 17.12, "ce": 0.002878594582398364, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.4801188111305237, "loss_scale": 4.0, "retries": 0, "seconds": 624.4, "max_memory_gb": 17.12, "ce": 0.04723741550606064, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 1.6490117311477661, "loss_scale": 4.0, "retries": 0, "seconds": 628.1, "max_memory_gb": 17.12, "ce": 0.04513493593822204, "rank": 0.24928081780672073}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.6254961490631104, "loss_scale": 4.0, "retries": 0, "seconds": 632.0, "max_memory_gb": 17.12, "ce": 0.06612034455429239, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 1.4499891996383667, "loss_scale": 4.0, "retries": 0, "seconds": 635.7, "max_memory_gb": 17.12, "ce": 0.06764573712371202, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 2.00616717338562, "loss_scale": 4.0, "retries": 0, "seconds": 639.4, "max_memory_gb": 17.12, "ce": 0.15314812895667274, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.48720046877861023, "loss_scale": 4.0, "retries": 0, "seconds": 643.3, "max_memory_gb": 17.12, "ce": 0.09011177470893017, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.7991514205932617, "loss_scale": 4.0, "retries": 0, "seconds": 647.0, "max_memory_gb": 17.12, "ce": 0.012170804862762452, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 0.18780644237995148, "loss_scale": 4.0, "retries": 0, "seconds": 650.8, "max_memory_gb": 17.12, "ce": 0.012243216391652822, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.5764698386192322, "loss_scale": 4.0, "retries": 0, "seconds": 654.6, "max_memory_gb": 17.12, "ce": 0.0630453711582959, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 2.4825446605682373, "loss_scale": 4.0, "retries": 0, "seconds": 658.4, "max_memory_gb": 17.12, "ce": 0.1358184712680668, "rank": 0.04741291701793671}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 0.41813409328460693, "loss_scale": 4.0, "retries": 0, "seconds": 662.2, "max_memory_gb": 17.12, "ce": 0.02399159678111573, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 0.6025328040122986, "loss_scale": 4.0, "retries": 0, "seconds": 666.0, "max_memory_gb": 17.12, "ce": 0.06774748098268901, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 1.1979132890701294, "loss_scale": 4.0, "retries": 0, "seconds": 669.8, "max_memory_gb": 17.12, "ce": 0.20361310100997798, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 2.8293251991271973, "loss_scale": 4.0, "retries": 0, "seconds": 673.6, "max_memory_gb": 17.12, "ce": 0.3606318278116305, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 2.0303306579589844, "loss_scale": 4.0, "retries": 0, "seconds": 677.3, "max_memory_gb": 17.12, "ce": 0.18766127002527355, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.05745557323098183, "loss_scale": 4.0, "retries": 0, "seconds": 681.2, "max_memory_gb": 17.12, "ce": 0.0024788638093014015, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.008337416686117649, "loss_scale": 4.0, "retries": 0, "seconds": 685.0, "max_memory_gb": 17.12, "ce": 0.0004713154339697212, "rank": 0.0}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.5867779850959778, "loss_scale": 4.0, "retries": 0, "seconds": 688.8, "max_memory_gb": 17.12, "ce": 0.0005353992528398521, "rank": 0.05521132051944733}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 0.23601126670837402, "loss_scale": 4.0, "retries": 0, "seconds": 692.6, "max_memory_gb": 17.12, "ce": 0.011710992129337683, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 2.296304941177368, "loss_scale": 4.0, "retries": 0, "seconds": 696.4, "max_memory_gb": 17.12, "ce": 0.19705040886628922, "rank": 0.0}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.3942252397537231, "loss_scale": 4.0, "retries": 0, "seconds": 700.1, "max_memory_gb": 17.12, "ce": 0.08097513265693124, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 1.0574572086334229, "loss_scale": 4.0, "retries": 0, "seconds": 703.9, "max_memory_gb": 17.12, "ce": 0.08207585728177946, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.3803788423538208, "loss_scale": 4.0, "retries": 0, "seconds": 707.8, "max_memory_gb": 17.12, "ce": 0.029725013860115723, "rank": 0.12497111409902573}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 0.022841259837150574, "loss_scale": 4.0, "retries": 0, "seconds": 711.6, "max_memory_gb": 17.12, "ce": 0.0011531227871728333, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.9176431894302368, "loss_scale": 4.0, "retries": 0, "seconds": 715.3, "max_memory_gb": 17.12, "ce": 0.3605690823560508, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.3393828868865967, "loss_scale": 4.0, "retries": 0, "seconds": 719.2, "max_memory_gb": 17.12, "ce": 0.02092794488044092, "rank": 0.0}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.19274507462978363, "loss_scale": 4.0, "retries": 0, "seconds": 723.0, "max_memory_gb": 17.12, "ce": 0.008824693868518807, "rank": 0.12497655302286148}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.011588752269744873, "loss_scale": 4.0, "retries": 0, "seconds": 726.8, "max_memory_gb": 17.12, "ce": 0.000669754756927432, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.008079533465206623, "loss_scale": 4.0, "retries": 0, "seconds": 730.6, "max_memory_gb": 17.12, "ce": 0.0005006754745409125, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.4562554955482483, "loss_scale": 4.0, "retries": 0, "seconds": 734.5, "max_memory_gb": 17.12, "ce": 0.024471269486753044, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.4631969630718231, "loss_scale": 4.0, "retries": 0, "seconds": 738.2, "max_memory_gb": 17.12, "ce": 0.042103392658646044, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 1.3358155488967896, "loss_scale": 4.0, "retries": 0, "seconds": 742.0, "max_memory_gb": 17.12, "ce": 0.15270308123308496, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 0.5422853231430054, "loss_scale": 4.0, "retries": 0, "seconds": 745.9, "max_memory_gb": 17.12, "ce": 0.09022093558860433, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.3936530649662018, "loss_scale": 4.0, "retries": 0, "seconds": 749.7, "max_memory_gb": 17.12, "ce": 0.01157164935466426, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.9876220226287842, "loss_scale": 4.0, "retries": 0, "seconds": 753.4, "max_memory_gb": 17.12, "ce": 0.3073575521088969, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.04834958165884018, "loss_scale": 4.0, "retries": 0, "seconds": 757.3, "max_memory_gb": 17.12, "ce": 0.0028454543735279003, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.5648462176322937, "loss_scale": 4.0, "retries": 0, "seconds": 761.1, "max_memory_gb": 17.12, "ce": 0.057529797782081005, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 0.09730590134859085, "loss_scale": 4.0, "retries": 0, "seconds": 764.8, "max_memory_gb": 17.12, "ce": 0.004857681794874225, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.8559697270393372, "loss_scale": 4.0, "retries": 0, "seconds": 768.6, "max_memory_gb": 17.12, "ce": 0.045623245311162464, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.21184232831001282, "loss_scale": 4.0, "retries": 0, "seconds": 772.5, "max_memory_gb": 17.12, "ce": 0.005543676037632395, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.40453800559043884, "loss_scale": 4.0, "retries": 0, "seconds": 776.2, "max_memory_gb": 17.12, "ce": 0.028284568399612908, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.28319117426872253, "loss_scale": 4.0, "retries": 0, "seconds": 779.9, "max_memory_gb": 17.12, "ce": 0.01101502385790809, "rank": 0.0}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 0.6812562346458435, "loss_scale": 4.0, "retries": 0, "seconds": 783.8, "max_memory_gb": 17.12, "ce": 0.04123318167921752, "rank": 0.0}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.28941696882247925, "loss_scale": 4.0, "retries": 0, "seconds": 787.6, "max_memory_gb": 17.12, "ce": 0.010088501321433796, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 1.3868566751480103, "loss_scale": 4.0, "retries": 0, "seconds": 791.4, "max_memory_gb": 17.12, "ce": 0.20833896496242232, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 1.5293970108032227, "loss_scale": 4.0, "retries": 0, "seconds": 795.2, "max_memory_gb": 17.12, "ce": 0.49206952333042864, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 0.13563068211078644, "loss_scale": 4.0, "retries": 0, "seconds": 799.0, "max_memory_gb": 17.12, "ce": 0.010346680657676188, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 0.6905328631401062, "loss_scale": 4.0, "retries": 0, "seconds": 802.8, "max_memory_gb": 17.12, "ce": 0.020782823168701725, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.84656822681427, "loss_scale": 4.0, "retries": 0, "seconds": 806.6, "max_memory_gb": 17.12, "ce": 0.026779630441524205, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.7083597779273987, "loss_scale": 4.0, "retries": 0, "seconds": 810.5, "max_memory_gb": 17.12, "ce": 0.05336597039058688, "rank": 0.12480466812849045}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 2.3448479175567627, "loss_scale": 4.0, "retries": 0, "seconds": 814.2, "max_memory_gb": 17.12, "ce": 0.31315973679056697, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.3472302258014679, "loss_scale": 4.0, "retries": 0, "seconds": 818.0, "max_memory_gb": 17.12, "ce": 0.01636409626553359, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 1.2726141214370728, "loss_scale": 4.0, "retries": 0, "seconds": 821.9, "max_memory_gb": 17.12, "ce": 0.10631861926049169, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 2.731895685195923, "loss_scale": 4.0, "retries": 0, "seconds": 825.6, "max_memory_gb": 17.12, "ce": 0.2765367923962003, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.9183510541915894, "loss_scale": 4.0, "retries": 0, "seconds": 829.4, "max_memory_gb": 17.12, "ce": 0.03165525567783334, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 2.613924980163574, "loss_scale": 4.0, "retries": 0, "seconds": 833.3, "max_memory_gb": 17.12, "ce": 0.07444843873963691, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.009507580660283566, "loss_scale": 4.0, "retries": 0, "seconds": 837.0, "max_memory_gb": 17.12, "ce": 0.0005071591084515603, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 1.3828771114349365, "loss_scale": 4.0, "retries": 0, "seconds": 840.8, "max_memory_gb": 17.12, "ce": 0.22401390511822683, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 1.9975751638412476, "loss_scale": 4.0, "retries": 0, "seconds": 844.5, "max_memory_gb": 17.12, "ce": 0.14785522964666598, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.09107360243797302, "loss_scale": 4.0, "retries": 0, "seconds": 848.4, "max_memory_gb": 17.12, "ce": 0.006955413142350153, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.8395522832870483, "loss_scale": 4.0, "retries": 0, "seconds": 852.2, "max_memory_gb": 17.12, "ce": 0.06903569124642672, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 1.5421667098999023, "loss_scale": 4.0, "retries": 0, "seconds": 855.9, "max_memory_gb": 17.12, "ce": 0.3574168964491946, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 0.15590159595012665, "loss_scale": 4.0, "retries": 0, "seconds": 859.8, "max_memory_gb": 17.12, "ce": 0.011140024342694232, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 3.3145816326141357, "loss_scale": 4.0, "retries": 0, "seconds": 863.5, "max_memory_gb": 17.12, "ce": 0.12705380240640807, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.5120692253112793, "loss_scale": 4.0, "retries": 0, "seconds": 867.2, "max_memory_gb": 17.12, "ce": 0.00503998538988526, "rank": 0.03860075771808624}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.42307591438293457, "loss_scale": 4.0, "retries": 0, "seconds": 870.9, "max_memory_gb": 17.12, "ce": 0.03236115669278661, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 1.9553067684173584, "loss_scale": 4.0, "retries": 0, "seconds": 874.8, "max_memory_gb": 17.12, "ce": 0.0698819234153234, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 6.476656913757324, "loss_scale": 4.0, "retries": 0, "seconds": 878.5, "max_memory_gb": 17.12, "ce": 0.5485986235880773, "rank": 0.05969130992889404}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.5519070029258728, "loss_scale": 4.0, "retries": 0, "seconds": 882.2, "max_memory_gb": 17.12, "ce": 0.06242305818523164, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.7802249789237976, "loss_scale": 4.0, "retries": 0, "seconds": 886.1, "max_memory_gb": 17.12, "ce": 0.14412285512662493, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 4.343350887298584, "loss_scale": 4.0, "retries": 0, "seconds": 889.8, "max_memory_gb": 17.12, "ce": 0.30243177492138784, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.44560495018959045, "loss_scale": 4.0, "retries": 0, "seconds": 893.6, "max_memory_gb": 17.12, "ce": 0.01558136401581578, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 3.347395658493042, "loss_scale": 4.0, "retries": 0, "seconds": 897.4, "max_memory_gb": 17.12, "ce": 0.41440071447141236, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.27110907435417175, "loss_scale": 4.0, "retries": 0, "seconds": 901.1, "max_memory_gb": 17.12, "ce": 0.017291899725933035, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 1.5743564367294312, "loss_scale": 4.0, "retries": 0, "seconds": 904.9, "max_memory_gb": 17.12, "ce": 0.20075406496243886, "rank": 0.12447119504213333}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 2.0994250774383545, "loss_scale": 4.0, "retries": 0, "seconds": 908.6, "max_memory_gb": 17.12, "ce": 0.1557876885171936, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.8433253765106201, "loss_scale": 4.0, "retries": 0, "seconds": 912.4, "max_memory_gb": 17.12, "ce": 0.04646491120274732, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 0.7900804281234741, "loss_scale": 4.0, "retries": 0, "seconds": 916.2, "max_memory_gb": 17.12, "ce": 0.14485339904058492, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.572478175163269, "loss_scale": 4.0, "retries": 0, "seconds": 919.9, "max_memory_gb": 17.12, "ce": 0.05229274054272537, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.051012493669986725, "loss_scale": 4.0, "retries": 0, "seconds": 923.7, "max_memory_gb": 17.12, "ce": 0.0028143792364971887, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.025630515068769455, "loss_scale": 4.0, "retries": 0, "seconds": 927.4, "max_memory_gb": 17.12, "ce": 0.0011714080974343233, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.3344130516052246, "loss_scale": 4.0, "retries": 0, "seconds": 931.2, "max_memory_gb": 17.12, "ce": 0.033108263832218654, "rank": 0.10890927910804749}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.11208243668079376, "loss_scale": 4.0, "retries": 0, "seconds": 934.9, "max_memory_gb": 17.12, "ce": 0.004372019995571463, "rank": 0.11445732414722443}

runs/fixed_internvl/removal/Middle8_s73/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/removal/Middle8_s73/adapter/adapter_config.json

@@ -0,0 +1,213 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.14.mlp.down_proj": 144,
+    "model.language_model.layers.14.mlp.gate_proj": 144,
+    "model.language_model.layers.14.mlp.up_proj": 144,
+    "model.language_model.layers.14.self_attn.k_proj": 144,
+    "model.language_model.layers.14.self_attn.o_proj": 144,
+    "model.language_model.layers.14.self_attn.q_proj": 144,
+    "model.language_model.layers.14.self_attn.v_proj": 144,
+    "model.language_model.layers.15.mlp.down_proj": 144,
+    "model.language_model.layers.15.mlp.gate_proj": 144,
+    "model.language_model.layers.15.mlp.up_proj": 144,
+    "model.language_model.layers.15.self_attn.k_proj": 144,
+    "model.language_model.layers.15.self_attn.o_proj": 144,
+    "model.language_model.layers.15.self_attn.q_proj": 144,
+    "model.language_model.layers.15.self_attn.v_proj": 144,
+    "model.language_model.layers.16.mlp.down_proj": 144,
+    "model.language_model.layers.16.mlp.gate_proj": 144,
+    "model.language_model.layers.16.mlp.up_proj": 144,
+    "model.language_model.layers.16.self_attn.k_proj": 144,
+    "model.language_model.layers.16.self_attn.o_proj": 144,
+    "model.language_model.layers.16.self_attn.q_proj": 144,
+    "model.language_model.layers.16.self_attn.v_proj": 144,
+    "model.language_model.layers.17.mlp.down_proj": 144,
+    "model.language_model.layers.17.mlp.gate_proj": 144,
+    "model.language_model.layers.17.mlp.up_proj": 144,
+    "model.language_model.layers.17.self_attn.k_proj": 144,
+    "model.language_model.layers.17.self_attn.o_proj": 144,
+    "model.language_model.layers.17.self_attn.q_proj": 144,
+    "model.language_model.layers.17.self_attn.v_proj": 144,
+    "model.language_model.layers.18.mlp.down_proj": 144,
+    "model.language_model.layers.18.mlp.gate_proj": 144,
+    "model.language_model.layers.18.mlp.up_proj": 144,
+    "model.language_model.layers.18.self_attn.k_proj": 144,
+    "model.language_model.layers.18.self_attn.o_proj": 144,
+    "model.language_model.layers.18.self_attn.q_proj": 144,
+    "model.language_model.layers.18.self_attn.v_proj": 144,
+    "model.language_model.layers.19.mlp.down_proj": 144,
+    "model.language_model.layers.19.mlp.gate_proj": 144,
+    "model.language_model.layers.19.mlp.up_proj": 144,
+    "model.language_model.layers.19.self_attn.k_proj": 144,
+    "model.language_model.layers.19.self_attn.o_proj": 144,
+    "model.language_model.layers.19.self_attn.q_proj": 144,
+    "model.language_model.layers.19.self_attn.v_proj": 144,
+    "model.language_model.layers.20.mlp.down_proj": 144,
+    "model.language_model.layers.20.mlp.gate_proj": 144,
+    "model.language_model.layers.20.mlp.up_proj": 144,
+    "model.language_model.layers.20.self_attn.k_proj": 144,
+    "model.language_model.layers.20.self_attn.o_proj": 144,
+    "model.language_model.layers.20.self_attn.q_proj": 144,
+    "model.language_model.layers.20.self_attn.v_proj": 144,
+    "model.language_model.layers.21.mlp.down_proj": 144,
+    "model.language_model.layers.21.mlp.gate_proj": 144,
+    "model.language_model.layers.21.mlp.up_proj": 144,
+    "model.language_model.layers.21.self_attn.k_proj": 144,
+    "model.language_model.layers.21.self_attn.o_proj": 144,
+    "model.language_model.layers.21.self_attn.q_proj": 144,
+    "model.language_model.layers.21.self_attn.v_proj": 144
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.14.mlp.down_proj": 72,
+    "model.language_model.layers.14.mlp.gate_proj": 72,
+    "model.language_model.layers.14.mlp.up_proj": 72,
+    "model.language_model.layers.14.self_attn.k_proj": 72,
+    "model.language_model.layers.14.self_attn.o_proj": 72,
+    "model.language_model.layers.14.self_attn.q_proj": 72,
+    "model.language_model.layers.14.self_attn.v_proj": 72,
+    "model.language_model.layers.15.mlp.down_proj": 72,
+    "model.language_model.layers.15.mlp.gate_proj": 72,
+    "model.language_model.layers.15.mlp.up_proj": 72,
+    "model.language_model.layers.15.self_attn.k_proj": 72,
+    "model.language_model.layers.15.self_attn.o_proj": 72,
+    "model.language_model.layers.15.self_attn.q_proj": 72,
+    "model.language_model.layers.15.self_attn.v_proj": 72,
+    "model.language_model.layers.16.mlp.down_proj": 72,
+    "model.language_model.layers.16.mlp.gate_proj": 72,
+    "model.language_model.layers.16.mlp.up_proj": 72,
+    "model.language_model.layers.16.self_attn.k_proj": 72,
+    "model.language_model.layers.16.self_attn.o_proj": 72,
+    "model.language_model.layers.16.self_attn.q_proj": 72,
+    "model.language_model.layers.16.self_attn.v_proj": 72,
+    "model.language_model.layers.17.mlp.down_proj": 72,
+    "model.language_model.layers.17.mlp.gate_proj": 72,
+    "model.language_model.layers.17.mlp.up_proj": 72,
+    "model.language_model.layers.17.self_attn.k_proj": 72,
+    "model.language_model.layers.17.self_attn.o_proj": 72,
+    "model.language_model.layers.17.self_attn.q_proj": 72,
+    "model.language_model.layers.17.self_attn.v_proj": 72,
+    "model.language_model.layers.18.mlp.down_proj": 72,
+    "model.language_model.layers.18.mlp.gate_proj": 72,
+    "model.language_model.layers.18.mlp.up_proj": 72,
+    "model.language_model.layers.18.self_attn.k_proj": 72,
+    "model.language_model.layers.18.self_attn.o_proj": 72,
+    "model.language_model.layers.18.self_attn.q_proj": 72,
+    "model.language_model.layers.18.self_attn.v_proj": 72,
+    "model.language_model.layers.19.mlp.down_proj": 72,
+    "model.language_model.layers.19.mlp.gate_proj": 72,
+    "model.language_model.layers.19.mlp.up_proj": 72,
+    "model.language_model.layers.19.self_attn.k_proj": 72,
+    "model.language_model.layers.19.self_attn.o_proj": 72,
+    "model.language_model.layers.19.self_attn.q_proj": 72,
+    "model.language_model.layers.19.self_attn.v_proj": 72,
+    "model.language_model.layers.20.mlp.down_proj": 72,
+    "model.language_model.layers.20.mlp.gate_proj": 72,
+    "model.language_model.layers.20.mlp.up_proj": 72,
+    "model.language_model.layers.20.self_attn.k_proj": 72,
+    "model.language_model.layers.20.self_attn.o_proj": 72,
+    "model.language_model.layers.20.self_attn.q_proj": 72,
+    "model.language_model.layers.20.self_attn.v_proj": 72,
+    "model.language_model.layers.21.mlp.down_proj": 72,
+    "model.language_model.layers.21.mlp.gate_proj": 72,
+    "model.language_model.layers.21.mlp.up_proj": 72,
+    "model.language_model.layers.21.self_attn.k_proj": 72,
+    "model.language_model.layers.21.self_attn.o_proj": 72,
+    "model.language_model.layers.21.self_attn.q_proj": 72,
+    "model.language_model.layers.21.self_attn.v_proj": 72
+  },
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.21.self_attn.q_proj",
+    "model.language_model.layers.21.mlp.gate_proj",
+    "model.language_model.layers.16.self_attn.v_proj",
+    "model.language_model.layers.19.self_attn.o_proj",
+    "model.language_model.layers.18.mlp.gate_proj",
+    "model.language_model.layers.15.mlp.down_proj",
+    "model.language_model.layers.19.self_attn.k_proj",
+    "model.language_model.layers.18.self_attn.o_proj",
+    "model.language_model.layers.19.mlp.gate_proj",
+    "model.language_model.layers.17.mlp.up_proj",
+    "model.language_model.layers.14.mlp.up_proj",
+    "model.language_model.layers.15.self_attn.o_proj",
+    "model.language_model.layers.20.mlp.up_proj",
+    "model.language_model.layers.20.self_attn.q_proj",
+    "model.language_model.layers.16.self_attn.k_proj",
+    "model.language_model.layers.14.self_attn.k_proj",
+    "model.language_model.layers.20.mlp.down_proj",
+    "model.language_model.layers.20.self_attn.k_proj",
+    "model.language_model.layers.16.mlp.up_proj",
+    "model.language_model.layers.17.self_attn.q_proj",
+    "model.language_model.layers.14.self_attn.v_proj",
+    "model.language_model.layers.19.mlp.down_proj",
+    "model.language_model.layers.16.mlp.gate_proj",
+    "model.language_model.layers.17.self_attn.o_proj",
+    "model.language_model.layers.21.self_attn.o_proj",
+    "model.language_model.layers.15.mlp.gate_proj",
+    "model.language_model.layers.19.mlp.up_proj",
+    "model.language_model.layers.14.mlp.down_proj",
+    "model.language_model.layers.14.self_attn.q_proj",
+    "model.language_model.layers.20.mlp.gate_proj",
+    "model.language_model.layers.15.self_attn.q_proj",
+    "model.language_model.layers.17.self_attn.k_proj",
+    "model.language_model.layers.16.self_attn.o_proj",
+    "model.language_model.layers.19.self_attn.q_proj",
+    "model.language_model.layers.20.self_attn.o_proj",
+    "model.language_model.layers.15.mlp.up_proj",
+    "model.language_model.layers.20.self_attn.v_proj",
+    "model.language_model.layers.21.mlp.down_proj",
+    "model.language_model.layers.17.self_attn.v_proj",
+    "model.language_model.layers.18.self_attn.q_proj",
+    "model.language_model.layers.17.mlp.down_proj",
+    "model.language_model.layers.21.mlp.up_proj",
+    "model.language_model.layers.15.self_attn.k_proj",
+    "model.language_model.layers.18.self_attn.v_proj",
+    "model.language_model.layers.18.mlp.up_proj",
+    "model.language_model.layers.16.self_attn.q_proj",
+    "model.language_model.layers.14.mlp.gate_proj",
+    "model.language_model.layers.16.mlp.down_proj",
+    "model.language_model.layers.21.self_attn.k_proj",
+    "model.language_model.layers.15.self_attn.v_proj",
+    "model.language_model.layers.17.mlp.gate_proj",
+    "model.language_model.layers.19.self_attn.v_proj",
+    "model.language_model.layers.18.mlp.down_proj",
+    "model.language_model.layers.14.self_attn.o_proj",
+    "model.language_model.layers.18.self_attn.k_proj",
+    "model.language_model.layers.21.self_attn.v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/fixed_internvl/removal/Middle8_s73/config.json

@@ -0,0 +1,43 @@
+{
+  "behavior": "removal",
+  "seed": 73,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "world_size": 1,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "arm": {
+    "arm": "Middle8",
+    "layers": [
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21
+    ],
+    "ranks": {
+      "14": 72,
+      "15": 72,
+      "16": 72,
+      "17": 72,
+      "18": 72,
+      "19": 72,
+      "20": 72,
+      "21": 72
+    },
+    "parameters": 43646976,
+    "budget": 43646976
+  }
+}

runs/fixed_internvl/removal/Middle8_s73/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 13.436259269714355, "loss_scale": 4.0, "retries": 0, "seconds": 5.7, "max_memory_gb": 16.79, "ce": 0.8941777415893739, "rank": 0.0}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 26.672250747680664, "loss_scale": 4.0, "retries": 0, "seconds": 10.5, "max_memory_gb": 17.11, "ce": 1.6419285386800766, "rank": 0.003568887710571289}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 10.458600997924805, "loss_scale": 4.0, "retries": 0, "seconds": 15.0, "max_memory_gb": 17.11, "ce": 0.6132747025694698, "rank": 0.08514091372489929}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 6.506412029266357, "loss_scale": 4.0, "retries": 0, "seconds": 19.4, "max_memory_gb": 17.11, "ce": 0.4907797488485812, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 2.447234869003296, "loss_scale": 4.0, "retries": 0, "seconds": 24.1, "max_memory_gb": 17.11, "ce": 0.4216889661111054, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 2.4550230503082275, "loss_scale": 4.0, "retries": 0, "seconds": 29.0, "max_memory_gb": 17.11, "ce": 0.4549932007212192, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 0.25650233030319214, "loss_scale": 4.0, "retries": 0, "seconds": 33.8, "max_memory_gb": 17.11, "ce": 0.1323393858028794, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 1.7011147737503052, "loss_scale": 4.0, "retries": 0, "seconds": 38.7, "max_memory_gb": 17.11, "ce": 0.6035347363504115, "rank": 0.13481245934963226}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 2.2520806789398193, "loss_scale": 4.0, "retries": 0, "seconds": 43.5, "max_memory_gb": 17.11, "ce": 0.3745700386352837, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 1.8382548093795776, "loss_scale": 4.0, "retries": 0, "seconds": 48.2, "max_memory_gb": 17.11, "ce": 0.31691970812971704, "rank": 0.009565278887748718}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.632993221282959, "loss_scale": 4.0, "retries": 0, "seconds": 53.0, "max_memory_gb": 17.11, "ce": 0.3030546316294931, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 2.6581172943115234, "loss_scale": 4.0, "retries": 0, "seconds": 57.7, "max_memory_gb": 17.11, "ce": 0.1780335916569129, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 1.2893325090408325, "loss_scale": 4.0, "retries": 0, "seconds": 62.5, "max_memory_gb": 17.11, "ce": 0.35784779640016495, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 1.6307822465896606, "loss_scale": 4.0, "retries": 0, "seconds": 67.1, "max_memory_gb": 17.11, "ce": 0.25391735082666855, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 3.066166400909424, "loss_scale": 4.0, "retries": 0, "seconds": 71.9, "max_memory_gb": 17.11, "ce": 0.17114662833046168, "rank": 0.0}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 5.514317035675049, "loss_scale": 4.0, "retries": 0, "seconds": 76.6, "max_memory_gb": 17.11, "ce": 0.938036427978659, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 2.9506216049194336, "loss_scale": 4.0, "retries": 0, "seconds": 81.5, "max_memory_gb": 17.11, "ce": 0.26621702866395935, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 1.0158041715621948, "loss_scale": 4.0, "retries": 0, "seconds": 86.4, "max_memory_gb": 17.11, "ce": 0.15409540315158665, "rank": 0.12386918812990189}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 5.432159900665283, "loss_scale": 4.0, "retries": 0, "seconds": 91.1, "max_memory_gb": 17.11, "ce": 0.38797543347027386, "rank": 0.0}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 1.1096538305282593, "loss_scale": 4.0, "retries": 0, "seconds": 95.9, "max_memory_gb": 17.11, "ce": 0.25818545745278243, "rank": 0.0}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.7121283411979675, "loss_scale": 4.0, "retries": 0, "seconds": 100.7, "max_memory_gb": 17.11, "ce": 0.06725746660958976, "rank": 0.0}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 1.0730153322219849, "loss_scale": 4.0, "retries": 0, "seconds": 105.4, "max_memory_gb": 17.11, "ce": 0.25114768208004534, "rank": 0.0}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 2.178257465362549, "loss_scale": 4.0, "retries": 0, "seconds": 109.9, "max_memory_gb": 17.11, "ce": 0.15699606679118006, "rank": 0.0}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 1.1890534162521362, "loss_scale": 4.0, "retries": 0, "seconds": 114.7, "max_memory_gb": 17.11, "ce": 0.15755468094721437, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 1.3675308227539062, "loss_scale": 4.0, "retries": 0, "seconds": 119.7, "max_memory_gb": 17.11, "ce": 0.08990749806980602, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 2.297800064086914, "loss_scale": 4.0, "retries": 0, "seconds": 124.6, "max_memory_gb": 17.11, "ce": 0.2513436555673252, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 6.217153072357178, "loss_scale": 4.0, "retries": 0, "seconds": 129.5, "max_memory_gb": 17.11, "ce": 0.1224615094542969, "rank": 0.0}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 0.18650837242603302, "loss_scale": 4.0, "retries": 0, "seconds": 134.3, "max_memory_gb": 17.11, "ce": 0.00743257386420737, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 5.004401206970215, "loss_scale": 4.0, "retries": 0, "seconds": 139.1, "max_memory_gb": 17.11, "ce": 0.04606162249183399, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 1.5851845741271973, "loss_scale": 4.0, "retries": 0, "seconds": 143.7, "max_memory_gb": 17.11, "ce": 0.054980734956188826, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 1.1546977758407593, "loss_scale": 4.0, "retries": 0, "seconds": 148.4, "max_memory_gb": 17.11, "ce": 0.10894547691714251, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 2.865694999694824, "loss_scale": 4.0, "retries": 0, "seconds": 153.3, "max_memory_gb": 17.11, "ce": 0.16963003069901106, "rank": 0.12498049437999725}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 0.23752309381961823, "loss_scale": 4.0, "retries": 0, "seconds": 157.8, "max_memory_gb": 17.11, "ce": 0.01879752028457915, "rank": 0.12502992153167725}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 2.1024351119995117, "loss_scale": 4.0, "retries": 0, "seconds": 161.9, "max_memory_gb": 17.11, "ce": 0.19564727772012702, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 1.8734874725341797, "loss_scale": 4.0, "retries": 0, "seconds": 166.5, "max_memory_gb": 17.11, "ce": 0.05544855356856715, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 1.5444568395614624, "loss_scale": 4.0, "retries": 0, "seconds": 170.7, "max_memory_gb": 17.11, "ce": 0.10132466462164302, "rank": 0.10043014585971832}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.4650845229625702, "loss_scale": 4.0, "retries": 0, "seconds": 175.3, "max_memory_gb": 17.11, "ce": 0.030090044328972, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.39125168323516846, "loss_scale": 4.0, "retries": 0, "seconds": 179.8, "max_memory_gb": 17.12, "ce": 0.017344479973189664, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 3.4542131423950195, "loss_scale": 4.0, "retries": 0, "seconds": 184.1, "max_memory_gb": 17.12, "ce": 0.4455449634569959, "rank": 0.0}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.26523497700691223, "loss_scale": 4.0, "retries": 0, "seconds": 189.1, "max_memory_gb": 17.12, "ce": 0.009110149479965912, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 0.751754879951477, "loss_scale": 4.0, "retries": 0, "seconds": 193.8, "max_memory_gb": 17.12, "ce": 0.012589963069331134, "rank": 0.08525872230529785}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 5.391781330108643, "loss_scale": 4.0, "retries": 0, "seconds": 198.6, "max_memory_gb": 17.12, "ce": 0.1950008433959738, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 4.675686836242676, "loss_scale": 4.0, "retries": 0, "seconds": 203.4, "max_memory_gb": 17.12, "ce": 0.31340645279124146, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 0.039416052401065826, "loss_scale": 4.0, "retries": 0, "seconds": 208.3, "max_memory_gb": 17.12, "ce": 0.0013082629448035732, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 0.9910180568695068, "loss_scale": 4.0, "retries": 0, "seconds": 213.1, "max_memory_gb": 17.12, "ce": 0.07462087098610937, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 1.5787137746810913, "loss_scale": 4.0, "retries": 0, "seconds": 217.6, "max_memory_gb": 17.12, "ce": 0.230427686134135, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.06419845670461655, "loss_scale": 4.0, "retries": 0, "seconds": 222.1, "max_memory_gb": 17.12, "ce": 0.0025247544162994018, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.5584640502929688, "loss_scale": 4.0, "retries": 0, "seconds": 226.6, "max_memory_gb": 17.12, "ce": 0.017046625122020487, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.08976782113313675, "loss_scale": 4.0, "retries": 0, "seconds": 231.1, "max_memory_gb": 17.12, "ce": 0.004897560411336599, "rank": 0.12433164566755295}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 0.047581225633621216, "loss_scale": 4.0, "retries": 0, "seconds": 235.7, "max_memory_gb": 17.12, "ce": 0.0017636048123677028, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 1.0312809944152832, "loss_scale": 4.0, "retries": 0, "seconds": 240.3, "max_memory_gb": 17.12, "ce": 0.03786903394939145, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 4.850098609924316, "loss_scale": 4.0, "retries": 0, "seconds": 245.2, "max_memory_gb": 17.12, "ce": 0.21372896519642381, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 2.716017007827759, "loss_scale": 4.0, "retries": 0, "seconds": 249.8, "max_memory_gb": 17.12, "ce": 0.41099271507255253, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 1.8040498495101929, "loss_scale": 4.0, "retries": 0, "seconds": 254.5, "max_memory_gb": 17.12, "ce": 0.2738776506703289, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.886081874370575, "loss_scale": 4.0, "retries": 0, "seconds": 259.3, "max_memory_gb": 17.12, "ce": 0.014571729629096808, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.37936118245124817, "loss_scale": 4.0, "retries": 0, "seconds": 263.9, "max_memory_gb": 17.12, "ce": 0.03407898623481742, "rank": 0.0}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 1.0701767206192017, "loss_scale": 4.0, "retries": 0, "seconds": 268.7, "max_memory_gb": 17.12, "ce": 0.15557097082273685, "rank": 0.12138243019580841}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.5348837375640869, "loss_scale": 4.0, "retries": 0, "seconds": 273.5, "max_memory_gb": 17.12, "ce": 0.0391536720608201, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 0.1104636937379837, "loss_scale": 4.0, "retries": 0, "seconds": 278.2, "max_memory_gb": 17.12, "ce": 0.006286751534389623, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.25075894594192505, "loss_scale": 4.0, "retries": 0, "seconds": 282.8, "max_memory_gb": 17.12, "ce": 0.00794297010088485, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 3.4767086505889893, "loss_scale": 4.0, "retries": 0, "seconds": 287.3, "max_memory_gb": 17.12, "ce": 0.15758986697619548, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 3.547419309616089, "loss_scale": 4.0, "retries": 0, "seconds": 292.0, "max_memory_gb": 17.12, "ce": 0.31531768795139214, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.24036753177642822, "loss_scale": 4.0, "retries": 0, "seconds": 296.7, "max_memory_gb": 17.12, "ce": 0.014943035905162105, "rank": 0.12509365379810333}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 1.1184386014938354, "loss_scale": 4.0, "retries": 0, "seconds": 301.5, "max_memory_gb": 17.12, "ce": 0.15999420312618895, "rank": 0.0}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 0.11121529340744019, "loss_scale": 4.0, "retries": 0, "seconds": 305.9, "max_memory_gb": 17.12, "ce": 0.004833744798816042, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 0.07223325967788696, "loss_scale": 4.0, "retries": 0, "seconds": 310.4, "max_memory_gb": 17.12, "ce": 0.004700104649600689, "rank": 0.0}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 3.503278970718384, "loss_scale": 4.0, "retries": 0, "seconds": 314.3, "max_memory_gb": 17.12, "ce": 0.3999174161144765, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.15905869007110596, "loss_scale": 4.0, "retries": 0, "seconds": 318.8, "max_memory_gb": 17.12, "ce": 0.009373634450639656, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 0.4902346134185791, "loss_scale": 4.0, "retries": 0, "seconds": 323.4, "max_memory_gb": 17.12, "ce": 0.05390199230714643, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.1898934245109558, "loss_scale": 4.0, "retries": 0, "seconds": 328.2, "max_memory_gb": 17.12, "ce": 0.009698270054286695, "rank": 0.0}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.20957855880260468, "loss_scale": 4.0, "retries": 0, "seconds": 332.7, "max_memory_gb": 17.12, "ce": 0.012146045120061899, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.2143690139055252, "loss_scale": 4.0, "retries": 0, "seconds": 337.5, "max_memory_gb": 17.12, "ce": 0.010175656111869102, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 1.852254033088684, "loss_scale": 4.0, "retries": 0, "seconds": 342.0, "max_memory_gb": 17.12, "ce": 0.2047840589129919, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 0.5870364308357239, "loss_scale": 4.0, "retries": 0, "seconds": 347.9, "max_memory_gb": 17.12, "ce": 0.03983686904939532, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 0.3298374116420746, "loss_scale": 4.0, "retries": 0, "seconds": 353.1, "max_memory_gb": 17.12, "ce": 0.01683716591560369, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 1.7570234537124634, "loss_scale": 4.0, "retries": 0, "seconds": 358.3, "max_memory_gb": 17.12, "ce": 0.4311948867753017, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 0.04858742654323578, "loss_scale": 4.0, "retries": 0, "seconds": 363.0, "max_memory_gb": 17.12, "ce": 0.0024334150766662788, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.9127107262611389, "loss_scale": 4.0, "retries": 0, "seconds": 369.0, "max_memory_gb": 17.12, "ce": 0.19418020318335039, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.3677281439304352, "loss_scale": 4.0, "retries": 0, "seconds": 374.1, "max_memory_gb": 17.12, "ce": 0.01520908694328682, "rank": 0.0}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 2.2366487979888916, "loss_scale": 4.0, "retries": 0, "seconds": 378.2, "max_memory_gb": 17.12, "ce": 0.2378938229776395, "rank": 0.12488237023353577}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 2.0164453983306885, "loss_scale": 4.0, "retries": 0, "seconds": 382.7, "max_memory_gb": 17.12, "ce": 0.15947008271177765, "rank": 0.12449143081903458}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 0.1585961878299713, "loss_scale": 4.0, "retries": 0, "seconds": 387.5, "max_memory_gb": 17.12, "ce": 0.008838417087645212, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 0.2891503572463989, "loss_scale": 4.0, "retries": 0, "seconds": 393.2, "max_memory_gb": 17.12, "ce": 0.020008654117191327, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.48873773217201233, "loss_scale": 4.0, "retries": 0, "seconds": 398.0, "max_memory_gb": 17.12, "ce": 0.039975298966965056, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.47490859031677246, "loss_scale": 4.0, "retries": 0, "seconds": 402.7, "max_memory_gb": 17.12, "ce": 0.01132911689910543, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.6014585494995117, "loss_scale": 4.0, "retries": 0, "seconds": 407.9, "max_memory_gb": 17.12, "ce": 0.08352411314444907, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 0.10450608283281326, "loss_scale": 4.0, "retries": 0, "seconds": 413.9, "max_memory_gb": 17.12, "ce": 0.006642316906436463, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.1365700215101242, "loss_scale": 4.0, "retries": 0, "seconds": 419.3, "max_memory_gb": 17.12, "ce": 0.008494505256749108, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 2.4311225414276123, "loss_scale": 4.0, "retries": 0, "seconds": 423.4, "max_memory_gb": 17.12, "ce": 0.17199993306894612, "rank": 0.0}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.35109755396842957, "loss_scale": 4.0, "retries": 0, "seconds": 427.2, "max_memory_gb": 17.12, "ce": 0.041951600131142186, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 2.3282394409179688, "loss_scale": 4.0, "retries": 0, "seconds": 431.8, "max_memory_gb": 17.12, "ce": 0.09672825300731347, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 1.091375708580017, "loss_scale": 4.0, "retries": 0, "seconds": 436.5, "max_memory_gb": 17.12, "ce": 0.0723481132527013, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 6.173498630523682, "loss_scale": 4.0, "retries": 0, "seconds": 441.0, "max_memory_gb": 17.12, "ce": 0.6577855839286713, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 0.2209208905696869, "loss_scale": 4.0, "retries": 0, "seconds": 446.2, "max_memory_gb": 17.12, "ce": 0.019570600050428766, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 2.4718403816223145, "loss_scale": 4.0, "retries": 0, "seconds": 451.5, "max_memory_gb": 17.12, "ce": 0.3896505129887373, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 1.1229033470153809, "loss_scale": 4.0, "retries": 0, "seconds": 456.1, "max_memory_gb": 17.12, "ce": 0.09130778010603535, "rank": 0.0}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.24462397396564484, "loss_scale": 4.0, "retries": 0, "seconds": 460.7, "max_memory_gb": 17.12, "ce": 0.016895315837246017, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.15989361703395844, "loss_scale": 4.0, "retries": 0, "seconds": 465.2, "max_memory_gb": 17.12, "ce": 0.011672536542391754, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 1.0611454248428345, "loss_scale": 4.0, "retries": 0, "seconds": 469.8, "max_memory_gb": 17.12, "ce": 0.05672030573623488, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.6496474742889404, "loss_scale": 4.0, "retries": 0, "seconds": 474.3, "max_memory_gb": 17.12, "ce": 0.05485334797322139, "rank": 0.12495198845863342}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 1.822195291519165, "loss_scale": 4.0, "retries": 0, "seconds": 478.4, "max_memory_gb": 17.12, "ce": 0.16695615412481857, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.36441725492477417, "loss_scale": 4.0, "retries": 0, "seconds": 482.6, "max_memory_gb": 17.12, "ce": 0.017002241660520667, "rank": 0.22075384855270386}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.16290949285030365, "loss_scale": 4.0, "retries": 0, "seconds": 487.1, "max_memory_gb": 17.12, "ce": 0.005721288378936151, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 1.0390348434448242, "loss_scale": 4.0, "retries": 0, "seconds": 491.5, "max_memory_gb": 17.12, "ce": 0.140262103823261, "rank": 0.0}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 1.8946293592453003, "loss_scale": 4.0, "retries": 0, "seconds": 495.9, "max_memory_gb": 17.12, "ce": 0.18022649290742265, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.4594515562057495, "loss_scale": 4.0, "retries": 0, "seconds": 500.5, "max_memory_gb": 17.12, "ce": 0.04001897686794109, "rank": 0.0}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 0.1220848336815834, "loss_scale": 4.0, "retries": 0, "seconds": 505.1, "max_memory_gb": 17.12, "ce": 0.011234167045586219, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 0.2984865605831146, "loss_scale": 4.0, "retries": 0, "seconds": 509.8, "max_memory_gb": 17.12, "ce": 0.012995205741390237, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 0.047450561076402664, "loss_scale": 4.0, "retries": 0, "seconds": 514.5, "max_memory_gb": 17.12, "ce": 0.0035301158950460376, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.047002725303173065, "loss_scale": 4.0, "retries": 0, "seconds": 519.1, "max_memory_gb": 17.12, "ce": 0.0017999422570937895, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.8484663367271423, "loss_scale": 4.0, "retries": 0, "seconds": 523.6, "max_memory_gb": 17.12, "ce": 0.049068553383222024, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.2117183804512024, "loss_scale": 4.0, "retries": 0, "seconds": 528.5, "max_memory_gb": 17.12, "ce": 0.01705819287326449, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 3.2414443492889404, "loss_scale": 4.0, "retries": 0, "seconds": 533.1, "max_memory_gb": 17.12, "ce": 0.5991854473322746, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 1.9244704246520996, "loss_scale": 4.0, "retries": 0, "seconds": 538.0, "max_memory_gb": 17.12, "ce": 0.1992466353844975, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.13394862413406372, "loss_scale": 4.0, "retries": 0, "seconds": 542.8, "max_memory_gb": 17.12, "ce": 0.00805112061152613, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 3.487236738204956, "loss_scale": 4.0, "retries": 0, "seconds": 548.1, "max_memory_gb": 17.12, "ce": 0.37410381955760386, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.719628632068634, "loss_scale": 4.0, "retries": 0, "seconds": 552.7, "max_memory_gb": 17.12, "ce": 0.024527845826014527, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 2.450289249420166, "loss_scale": 4.0, "retries": 0, "seconds": 557.2, "max_memory_gb": 17.12, "ce": 0.05968911724448844, "rank": 0.1261221170425415}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.22089450061321259, "loss_scale": 4.0, "retries": 0, "seconds": 561.8, "max_memory_gb": 17.12, "ce": 0.007707816977926996, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 2.5552570819854736, "loss_scale": 4.0, "retries": 0, "seconds": 566.4, "max_memory_gb": 17.12, "ce": 0.1886818364841929, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 1.2010242938995361, "loss_scale": 4.0, "retries": 0, "seconds": 571.0, "max_memory_gb": 17.12, "ce": 0.08896532972175919, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 1.122506022453308, "loss_scale": 4.0, "retries": 0, "seconds": 575.8, "max_memory_gb": 17.12, "ce": 0.05437654394427227, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 4.769791126251221, "loss_scale": 4.0, "retries": 0, "seconds": 580.8, "max_memory_gb": 17.12, "ce": 0.3910314580925842, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 0.019627340137958527, "loss_scale": 4.0, "retries": 0, "seconds": 585.6, "max_memory_gb": 17.12, "ce": 0.0014374549250533164, "rank": 0.0}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 3.7325994968414307, "loss_scale": 4.0, "retries": 0, "seconds": 590.2, "max_memory_gb": 17.12, "ce": 0.39645135937917075, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 0.25194239616394043, "loss_scale": 4.0, "retries": 0, "seconds": 594.9, "max_memory_gb": 17.12, "ce": 0.016269537512926036, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.09581532329320908, "loss_scale": 4.0, "retries": 0, "seconds": 599.5, "max_memory_gb": 17.12, "ce": 0.002204875324423483, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 0.23272110521793365, "loss_scale": 4.0, "retries": 0, "seconds": 604.1, "max_memory_gb": 17.12, "ce": 0.014417649583265302, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 2.2645456790924072, "loss_scale": 4.0, "retries": 0, "seconds": 608.8, "max_memory_gb": 17.12, "ce": 0.06385512670112803, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 2.2991135120391846, "loss_scale": 4.0, "retries": 0, "seconds": 613.4, "max_memory_gb": 17.12, "ce": 0.08866256780083859, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 1.6869333982467651, "loss_scale": 4.0, "retries": 0, "seconds": 617.9, "max_memory_gb": 17.12, "ce": 0.15716658606322653, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 0.41186127066612244, "loss_scale": 4.0, "retries": 0, "seconds": 622.6, "max_memory_gb": 17.12, "ce": 0.02005889215706702, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 8.954947471618652, "loss_scale": 4.0, "retries": 0, "seconds": 627.5, "max_memory_gb": 17.12, "ce": 0.6498515129619591, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 0.05733467638492584, "loss_scale": 4.0, "retries": 0, "seconds": 632.0, "max_memory_gb": 17.12, "ce": 0.0044385458295437275, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 6.531551361083984, "loss_scale": 4.0, "retries": 0, "seconds": 636.5, "max_memory_gb": 17.12, "ce": 0.4916419701871746, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 5.154825210571289, "loss_scale": 4.0, "retries": 0, "seconds": 641.1, "max_memory_gb": 17.12, "ce": 0.6944143468003858, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 7.137136936187744, "loss_scale": 4.0, "retries": 0, "seconds": 645.6, "max_memory_gb": 17.12, "ce": 0.32477721167242635, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 0.059468235820531845, "loss_scale": 4.0, "retries": 0, "seconds": 650.3, "max_memory_gb": 17.12, "ce": 0.002455330432894698, "rank": 0.0940154418349266}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.047387246042490005, "loss_scale": 4.0, "retries": 0, "seconds": 655.1, "max_memory_gb": 17.12, "ce": 0.004086017307599832, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.5571552515029907, "loss_scale": 4.0, "retries": 0, "seconds": 660.1, "max_memory_gb": 17.12, "ce": 0.05118219928931467, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.046506937593221664, "loss_scale": 4.0, "retries": 0, "seconds": 664.8, "max_memory_gb": 17.12, "ce": 0.0035931474508288375, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 1.2880898714065552, "loss_scale": 4.0, "retries": 0, "seconds": 669.5, "max_memory_gb": 17.12, "ce": 0.18643965358023706, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 1.6367974281311035, "loss_scale": 4.0, "retries": 0, "seconds": 674.3, "max_memory_gb": 17.12, "ce": 0.1309057383664367, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.26515787839889526, "loss_scale": 4.0, "retries": 0, "seconds": 678.8, "max_memory_gb": 17.12, "ce": 0.012363098759578861, "rank": 0.11823219060897827}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.5695147514343262, "loss_scale": 4.0, "retries": 0, "seconds": 683.4, "max_memory_gb": 17.12, "ce": 0.07846090158636798, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 2.862149953842163, "loss_scale": 4.0, "retries": 0, "seconds": 688.1, "max_memory_gb": 17.12, "ce": 0.20647986814628894, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 3.706865072250366, "loss_scale": 4.0, "retries": 0, "seconds": 692.6, "max_memory_gb": 17.12, "ce": 0.1448132243249347, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.25514841079711914, "loss_scale": 4.0, "retries": 0, "seconds": 698.5, "max_memory_gb": 17.12, "ce": 0.01827840849182394, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 3.978609085083008, "loss_scale": 4.0, "retries": 0, "seconds": 704.6, "max_memory_gb": 17.12, "ce": 0.6411743228063642, "rank": 0.1660962700843811}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.4132387936115265, "loss_scale": 4.0, "retries": 0, "seconds": 709.2, "max_memory_gb": 17.12, "ce": 0.04309547836146521, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.7947603464126587, "loss_scale": 4.0, "retries": 0, "seconds": 713.6, "max_memory_gb": 17.12, "ce": 0.13925993529846892, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 1.4978610277175903, "loss_scale": 4.0, "retries": 0, "seconds": 718.1, "max_memory_gb": 17.12, "ce": 0.2894127802433104, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.161650151014328, "loss_scale": 4.0, "retries": 0, "seconds": 723.0, "max_memory_gb": 17.12, "ce": 0.006847802354968735, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 2.207153558731079, "loss_scale": 4.0, "retries": 0, "seconds": 728.0, "max_memory_gb": 17.12, "ce": 0.07179000746873498, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 0.325554758310318, "loss_scale": 4.0, "retries": 0, "seconds": 733.2, "max_memory_gb": 17.12, "ce": 0.016646211614897766, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.374505877494812, "loss_scale": 4.0, "retries": 0, "seconds": 738.8, "max_memory_gb": 17.12, "ce": 0.03169056265596737, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 2.545008897781372, "loss_scale": 4.0, "retries": 0, "seconds": 744.8, "max_memory_gb": 17.12, "ce": 0.36164299084202867, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 0.09109768271446228, "loss_scale": 4.0, "retries": 0, "seconds": 749.7, "max_memory_gb": 17.12, "ce": 0.006293744307640736, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.05652584508061409, "loss_scale": 4.0, "retries": 0, "seconds": 754.1, "max_memory_gb": 17.12, "ce": 0.0020335231752142136, "rank": 0.12490641325712204}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 0.22076217830181122, "loss_scale": 4.0, "retries": 0, "seconds": 758.6, "max_memory_gb": 17.12, "ce": 0.027232466282839596, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 0.810545802116394, "loss_scale": 4.0, "retries": 0, "seconds": 762.8, "max_memory_gb": 17.12, "ce": 0.07538945676151343, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.4507915675640106, "loss_scale": 4.0, "retries": 0, "seconds": 767.4, "max_memory_gb": 17.12, "ce": 0.02744794487125546, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 1.2235915660858154, "loss_scale": 4.0, "retries": 0, "seconds": 772.3, "max_memory_gb": 17.12, "ce": 0.06432915795130612, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.31609296798706055, "loss_scale": 4.0, "retries": 0, "seconds": 776.8, "max_memory_gb": 17.12, "ce": 0.02732163782548014, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.014507317915558815, "loss_scale": 4.0, "retries": 0, "seconds": 781.4, "max_memory_gb": 17.12, "ce": 0.0010295793108525686, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 0.07470735907554626, "loss_scale": 4.0, "retries": 0, "seconds": 786.4, "max_memory_gb": 17.12, "ce": 0.0031297615641960874, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 1.451315999031067, "loss_scale": 4.0, "retries": 0, "seconds": 791.0, "max_memory_gb": 17.12, "ce": 0.2757520259397097, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 1.08879554271698, "loss_scale": 4.0, "retries": 0, "seconds": 795.6, "max_memory_gb": 17.12, "ce": 0.11438888126212987, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 0.027840739116072655, "loss_scale": 4.0, "retries": 0, "seconds": 800.2, "max_memory_gb": 17.12, "ce": 0.0017024767844304733, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 6.400229454040527, "loss_scale": 4.0, "retries": 0, "seconds": 805.0, "max_memory_gb": 17.12, "ce": 0.13785429184781606, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.8273285627365112, "loss_scale": 4.0, "retries": 0, "seconds": 809.5, "max_memory_gb": 17.12, "ce": 0.06655793357822404, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 2.728282928466797, "loss_scale": 4.0, "retries": 0, "seconds": 814.0, "max_memory_gb": 17.12, "ce": 0.2186724376847451, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.23600512742996216, "loss_scale": 4.0, "retries": 0, "seconds": 818.6, "max_memory_gb": 17.12, "ce": 0.02636928285119211, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 4.4738545417785645, "loss_scale": 4.0, "retries": 0, "seconds": 823.3, "max_memory_gb": 17.12, "ce": 0.3893431256021813, "rank": 0.12494180351495743}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 1.2780017852783203, "loss_scale": 4.0, "retries": 0, "seconds": 827.8, "max_memory_gb": 17.12, "ce": 0.19014817246215898, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 2.7100327014923096, "loss_scale": 4.0, "retries": 0, "seconds": 832.4, "max_memory_gb": 17.12, "ce": 0.19347935903442703, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 0.8388937711715698, "loss_scale": 4.0, "retries": 0, "seconds": 837.1, "max_memory_gb": 17.12, "ce": 0.0735607460637766, "rank": 0.0}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 0.030882878229022026, "loss_scale": 4.0, "retries": 0, "seconds": 841.6, "max_memory_gb": 17.12, "ce": 0.0017719019051583018, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 0.3720935881137848, "loss_scale": 4.0, "retries": 0, "seconds": 846.1, "max_memory_gb": 17.12, "ce": 0.056069892036020974, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 1.7710355520248413, "loss_scale": 4.0, "retries": 0, "seconds": 850.8, "max_memory_gb": 17.12, "ce": 0.07254816244858375, "rank": 0.12888850271701813}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 1.7360374927520752, "loss_scale": 4.0, "retries": 0, "seconds": 855.3, "max_memory_gb": 17.12, "ce": 0.12202983229190068, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 0.08076432347297668, "loss_scale": 4.0, "retries": 0, "seconds": 859.9, "max_memory_gb": 17.12, "ce": 0.004518914690834208, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 1.0937741994857788, "loss_scale": 4.0, "retries": 0, "seconds": 864.7, "max_memory_gb": 17.12, "ce": 0.06440101930866149, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 1.513821005821228, "loss_scale": 4.0, "retries": 0, "seconds": 869.4, "max_memory_gb": 17.12, "ce": 0.1837194498361896, "rank": 0.0}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 3.1524341106414795, "loss_scale": 4.0, "retries": 0, "seconds": 874.2, "max_memory_gb": 17.12, "ce": 0.14483227940354482, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 1.1127163171768188, "loss_scale": 4.0, "retries": 0, "seconds": 879.1, "max_memory_gb": 17.12, "ce": 0.035052444525717874, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.4954606890678406, "loss_scale": 4.0, "retries": 0, "seconds": 884.0, "max_memory_gb": 17.12, "ce": 0.07424335814584992, "rank": 0.0}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 0.31836891174316406, "loss_scale": 4.0, "retries": 0, "seconds": 888.6, "max_memory_gb": 17.12, "ce": 0.025246092174938894, "rank": 0.124973826110363}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 0.11412812024354935, "loss_scale": 4.0, "retries": 0, "seconds": 893.2, "max_memory_gb": 17.12, "ce": 0.010099083211116522, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 2.170858144760132, "loss_scale": 4.0, "retries": 0, "seconds": 897.9, "max_memory_gb": 17.12, "ce": 0.04948342942725503, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 0.056109145283699036, "loss_scale": 4.0, "retries": 0, "seconds": 902.5, "max_memory_gb": 17.12, "ce": 0.0026408546168568137, "rank": 0.12495836615562439}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 0.428865909576416, "loss_scale": 4.0, "retries": 0, "seconds": 907.0, "max_memory_gb": 17.12, "ce": 0.04186084850107363, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.6184023022651672, "loss_scale": 4.0, "retries": 0, "seconds": 911.8, "max_memory_gb": 17.12, "ce": 0.025524405885789747, "rank": 0.0}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 1.5874279737472534, "loss_scale": 4.0, "retries": 0, "seconds": 916.3, "max_memory_gb": 17.12, "ce": 0.09062744188895522, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.6899311542510986, "loss_scale": 4.0, "retries": 0, "seconds": 920.7, "max_memory_gb": 17.12, "ce": 0.05429384321860198, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.04788210242986679, "loss_scale": 4.0, "retries": 0, "seconds": 926.0, "max_memory_gb": 17.12, "ce": 0.0018069527989155176, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 0.002886631526052952, "loss_scale": 4.0, "retries": 0, "seconds": 931.2, "max_memory_gb": 17.12, "ce": 0.00021140584544809826, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.7066804766654968, "loss_scale": 4.0, "retries": 0, "seconds": 936.3, "max_memory_gb": 17.12, "ce": 0.09813464992703302, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 0.0693710669875145, "loss_scale": 4.0, "retries": 0, "seconds": 940.8, "max_memory_gb": 17.12, "ce": 0.0019739125962132675, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 0.8153383731842041, "loss_scale": 4.0, "retries": 0, "seconds": 945.4, "max_memory_gb": 17.12, "ce": 0.20441520331860374, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.2938569486141205, "loss_scale": 4.0, "retries": 0, "seconds": 949.3, "max_memory_gb": 17.12, "ce": 0.01960891464841552, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.12780919671058655, "loss_scale": 4.0, "retries": 0, "seconds": 953.8, "max_memory_gb": 17.12, "ce": 0.009008417850964179, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 1.043459415435791, "loss_scale": 4.0, "retries": 0, "seconds": 958.4, "max_memory_gb": 17.12, "ce": 0.04704742486410396, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.025361863896250725, "loss_scale": 4.0, "retries": 0, "seconds": 963.0, "max_memory_gb": 17.12, "ce": 0.0010526610694796545, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 2.400402069091797, "loss_scale": 4.0, "retries": 0, "seconds": 967.6, "max_memory_gb": 17.12, "ce": 0.30061837855828344, "rank": 0.12481676042079926}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 1.4563928842544556, "loss_scale": 4.0, "retries": 0, "seconds": 972.2, "max_memory_gb": 17.12, "ce": 0.31158860029290736, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.13662061095237732, "loss_scale": 4.0, "retries": 0, "seconds": 977.1, "max_memory_gb": 17.12, "ce": 0.00569069328525984, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.1102566048502922, "loss_scale": 4.0, "retries": 0, "seconds": 981.5, "max_memory_gb": 17.12, "ce": 0.0073588006782756565, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.060054704546928406, "loss_scale": 4.0, "retries": 0, "seconds": 986.0, "max_memory_gb": 17.12, "ce": 0.00243538911036012, "rank": 0.0}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 2.0344736576080322, "loss_scale": 4.0, "retries": 0, "seconds": 990.5, "max_memory_gb": 17.12, "ce": 0.04348743273635591, "rank": 0.0}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 2.613408088684082, "loss_scale": 4.0, "retries": 0, "seconds": 994.8, "max_memory_gb": 17.12, "ce": 0.5105032926235253, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 0.4126608669757843, "loss_scale": 4.0, "retries": 0, "seconds": 998.6, "max_memory_gb": 17.12, "ce": 0.04451528289655471, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.5181417465209961, "loss_scale": 4.0, "retries": 0, "seconds": 1003.2, "max_memory_gb": 17.12, "ce": 0.02983936590317171, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 1.1645530462265015, "loss_scale": 4.0, "retries": 0, "seconds": 1007.8, "max_memory_gb": 17.12, "ce": 0.10743865102767813, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 0.14527051150798798, "loss_scale": 4.0, "retries": 0, "seconds": 1012.3, "max_memory_gb": 17.12, "ce": 0.006469555646617664, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.3060683012008667, "loss_scale": 4.0, "retries": 0, "seconds": 1016.8, "max_memory_gb": 17.12, "ce": 0.026465952514172386, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.10853501409292221, "loss_scale": 4.0, "retries": 0, "seconds": 1021.4, "max_memory_gb": 17.12, "ce": 0.0059432234397718275, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 1.4997590780258179, "loss_scale": 4.0, "retries": 0, "seconds": 1025.9, "max_memory_gb": 17.12, "ce": 0.1458703729044828, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.369209349155426, "loss_scale": 4.0, "retries": 0, "seconds": 1030.5, "max_memory_gb": 17.12, "ce": 0.011995409058727091, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.9760810732841492, "loss_scale": 4.0, "retries": 0, "seconds": 1035.2, "max_memory_gb": 17.12, "ce": 0.09684057538788693, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.5368170738220215, "loss_scale": 4.0, "retries": 0, "seconds": 1039.7, "max_memory_gb": 17.12, "ce": 0.0196033865154277, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 1.6090660095214844, "loss_scale": 4.0, "retries": 0, "seconds": 1044.3, "max_memory_gb": 17.12, "ce": 0.1295995717587175, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 0.6828924417495728, "loss_scale": 4.0, "retries": 0, "seconds": 1049.2, "max_memory_gb": 17.12, "ce": 0.08884288234048654, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 1.1093385219573975, "loss_scale": 4.0, "retries": 0, "seconds": 1054.0, "max_memory_gb": 17.12, "ce": 0.04939437467373864, "rank": 0.12490932643413544}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.47930875420570374, "loss_scale": 4.0, "retries": 0, "seconds": 1058.8, "max_memory_gb": 17.12, "ce": 0.018839088055642605, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.5416098833084106, "loss_scale": 4.0, "retries": 0, "seconds": 1063.6, "max_memory_gb": 17.12, "ce": 0.03090536849936143, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 1.1008332967758179, "loss_scale": 4.0, "retries": 0, "seconds": 1068.2, "max_memory_gb": 17.12, "ce": 0.12276776295448144, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.7066801190376282, "loss_scale": 4.0, "retries": 0, "seconds": 1072.9, "max_memory_gb": 17.12, "ce": 0.06610991949264644, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.022953707724809647, "loss_scale": 4.0, "retries": 0, "seconds": 1077.7, "max_memory_gb": 17.12, "ce": 0.0013192488781896827, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 0.24255123734474182, "loss_scale": 4.0, "retries": 0, "seconds": 1082.5, "max_memory_gb": 17.12, "ce": 0.009090880034818838, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 1.3515615463256836, "loss_scale": 4.0, "retries": 0, "seconds": 1086.9, "max_memory_gb": 17.12, "ce": 0.1339718219983297, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.003959357272833586, "loss_scale": 4.0, "retries": 0, "seconds": 1091.5, "max_memory_gb": 17.12, "ce": 0.0002409630033071153, "rank": 0.12500284612178802}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.8342549800872803, "loss_scale": 4.0, "retries": 0, "seconds": 1096.1, "max_memory_gb": 17.12, "ce": 0.0475806614317662, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 1.002671241760254, "loss_scale": 4.0, "retries": 0, "seconds": 1100.8, "max_memory_gb": 17.12, "ce": 0.073995579899929, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 3.1351070404052734, "loss_scale": 4.0, "retries": 0, "seconds": 1105.4, "max_memory_gb": 17.12, "ce": 0.8974514583176187, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.05871051549911499, "loss_scale": 4.0, "retries": 0, "seconds": 1109.9, "max_memory_gb": 17.12, "ce": 0.004372172829334886, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.5646336674690247, "loss_scale": 4.0, "retries": 0, "seconds": 1114.5, "max_memory_gb": 17.12, "ce": 0.009555765524964954, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.07783433794975281, "loss_scale": 4.0, "retries": 0, "seconds": 1119.1, "max_memory_gb": 17.12, "ce": 0.0047651106838202395, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.6599056720733643, "loss_scale": 4.0, "retries": 0, "seconds": 1123.5, "max_memory_gb": 17.12, "ce": 0.09648711769750662, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 0.6202747821807861, "loss_scale": 4.0, "retries": 0, "seconds": 1128.0, "max_memory_gb": 17.12, "ce": 0.02163600801713983, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.44232067465782166, "loss_scale": 4.0, "retries": 0, "seconds": 1132.5, "max_memory_gb": 17.12, "ce": 0.08524972400391562, "rank": 0.122459277510643}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.3680242896080017, "loss_scale": 4.0, "retries": 0, "seconds": 1137.0, "max_memory_gb": 17.12, "ce": 0.025892435465721064, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.8073717355728149, "loss_scale": 4.0, "retries": 0, "seconds": 1141.5, "max_memory_gb": 17.12, "ce": 0.0569058078181115, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 1.197458028793335, "loss_scale": 4.0, "retries": 0, "seconds": 1146.2, "max_memory_gb": 17.12, "ce": 0.1067789451299177, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 0.1929658055305481, "loss_scale": 4.0, "retries": 0, "seconds": 1150.8, "max_memory_gb": 17.12, "ce": 0.015203543289771915, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.9290750622749329, "loss_scale": 4.0, "retries": 0, "seconds": 1155.5, "max_memory_gb": 17.12, "ce": 0.14037493984278626, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 2.4639804363250732, "loss_scale": 4.0, "retries": 0, "seconds": 1160.4, "max_memory_gb": 17.12, "ce": 0.05883979047121102, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.005891185253858566, "loss_scale": 4.0, "retries": 0, "seconds": 1165.2, "max_memory_gb": 17.12, "ce": 0.00035293696555527276, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.10602609813213348, "loss_scale": 4.0, "retries": 0, "seconds": 1169.8, "max_memory_gb": 17.12, "ce": 0.007726887557055306, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 5.727860450744629, "loss_scale": 4.0, "retries": 0, "seconds": 1174.3, "max_memory_gb": 17.12, "ce": 0.18414130634937464, "rank": 0.0}

runs/fixed_internvl/removal/Random8_8101_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/removal/Random8_8101_s13/adapter/adapter_config.json

@@ -0,0 +1,213 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.1.mlp.down_proj": 144,
+    "model.language_model.layers.1.mlp.gate_proj": 144,
+    "model.language_model.layers.1.mlp.up_proj": 144,
+    "model.language_model.layers.1.self_attn.k_proj": 144,
+    "model.language_model.layers.1.self_attn.o_proj": 144,
+    "model.language_model.layers.1.self_attn.q_proj": 144,
+    "model.language_model.layers.1.self_attn.v_proj": 144,
+    "model.language_model.layers.10.mlp.down_proj": 144,
+    "model.language_model.layers.10.mlp.gate_proj": 144,
+    "model.language_model.layers.10.mlp.up_proj": 144,
+    "model.language_model.layers.10.self_attn.k_proj": 144,
+    "model.language_model.layers.10.self_attn.o_proj": 144,
+    "model.language_model.layers.10.self_attn.q_proj": 144,
+    "model.language_model.layers.10.self_attn.v_proj": 144,
+    "model.language_model.layers.21.mlp.down_proj": 144,
+    "model.language_model.layers.21.mlp.gate_proj": 144,
+    "model.language_model.layers.21.mlp.up_proj": 144,
+    "model.language_model.layers.21.self_attn.k_proj": 144,
+    "model.language_model.layers.21.self_attn.o_proj": 144,
+    "model.language_model.layers.21.self_attn.q_proj": 144,
+    "model.language_model.layers.21.self_attn.v_proj": 144,
+    "model.language_model.layers.28.mlp.down_proj": 144,
+    "model.language_model.layers.28.mlp.gate_proj": 144,
+    "model.language_model.layers.28.mlp.up_proj": 144,
+    "model.language_model.layers.28.self_attn.k_proj": 144,
+    "model.language_model.layers.28.self_attn.o_proj": 144,
+    "model.language_model.layers.28.self_attn.q_proj": 144,
+    "model.language_model.layers.28.self_attn.v_proj": 144,
+    "model.language_model.layers.31.mlp.down_proj": 144,
+    "model.language_model.layers.31.mlp.gate_proj": 144,
+    "model.language_model.layers.31.mlp.up_proj": 144,
+    "model.language_model.layers.31.self_attn.k_proj": 144,
+    "model.language_model.layers.31.self_attn.o_proj": 144,
+    "model.language_model.layers.31.self_attn.q_proj": 144,
+    "model.language_model.layers.31.self_attn.v_proj": 144,
+    "model.language_model.layers.35.mlp.down_proj": 144,
+    "model.language_model.layers.35.mlp.gate_proj": 144,
+    "model.language_model.layers.35.mlp.up_proj": 144,
+    "model.language_model.layers.35.self_attn.k_proj": 144,
+    "model.language_model.layers.35.self_attn.o_proj": 144,
+    "model.language_model.layers.35.self_attn.q_proj": 144,
+    "model.language_model.layers.35.self_attn.v_proj": 144,
+    "model.language_model.layers.4.mlp.down_proj": 144,
+    "model.language_model.layers.4.mlp.gate_proj": 144,
+    "model.language_model.layers.4.mlp.up_proj": 144,
+    "model.language_model.layers.4.self_attn.k_proj": 144,
+    "model.language_model.layers.4.self_attn.o_proj": 144,
+    "model.language_model.layers.4.self_attn.q_proj": 144,
+    "model.language_model.layers.4.self_attn.v_proj": 144,
+    "model.language_model.layers.9.mlp.down_proj": 144,
+    "model.language_model.layers.9.mlp.gate_proj": 144,
+    "model.language_model.layers.9.mlp.up_proj": 144,
+    "model.language_model.layers.9.self_attn.k_proj": 144,
+    "model.language_model.layers.9.self_attn.o_proj": 144,
+    "model.language_model.layers.9.self_attn.q_proj": 144,
+    "model.language_model.layers.9.self_attn.v_proj": 144
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.1.mlp.down_proj": 72,
+    "model.language_model.layers.1.mlp.gate_proj": 72,
+    "model.language_model.layers.1.mlp.up_proj": 72,
+    "model.language_model.layers.1.self_attn.k_proj": 72,
+    "model.language_model.layers.1.self_attn.o_proj": 72,
+    "model.language_model.layers.1.self_attn.q_proj": 72,
+    "model.language_model.layers.1.self_attn.v_proj": 72,
+    "model.language_model.layers.10.mlp.down_proj": 72,
+    "model.language_model.layers.10.mlp.gate_proj": 72,
+    "model.language_model.layers.10.mlp.up_proj": 72,
+    "model.language_model.layers.10.self_attn.k_proj": 72,
+    "model.language_model.layers.10.self_attn.o_proj": 72,
+    "model.language_model.layers.10.self_attn.q_proj": 72,
+    "model.language_model.layers.10.self_attn.v_proj": 72,
+    "model.language_model.layers.21.mlp.down_proj": 72,
+    "model.language_model.layers.21.mlp.gate_proj": 72,
+    "model.language_model.layers.21.mlp.up_proj": 72,
+    "model.language_model.layers.21.self_attn.k_proj": 72,
+    "model.language_model.layers.21.self_attn.o_proj": 72,
+    "model.language_model.layers.21.self_attn.q_proj": 72,
+    "model.language_model.layers.21.self_attn.v_proj": 72,
+    "model.language_model.layers.28.mlp.down_proj": 72,
+    "model.language_model.layers.28.mlp.gate_proj": 72,
+    "model.language_model.layers.28.mlp.up_proj": 72,
+    "model.language_model.layers.28.self_attn.k_proj": 72,
+    "model.language_model.layers.28.self_attn.o_proj": 72,
+    "model.language_model.layers.28.self_attn.q_proj": 72,
+    "model.language_model.layers.28.self_attn.v_proj": 72,
+    "model.language_model.layers.31.mlp.down_proj": 72,
+    "model.language_model.layers.31.mlp.gate_proj": 72,
+    "model.language_model.layers.31.mlp.up_proj": 72,
+    "model.language_model.layers.31.self_attn.k_proj": 72,
+    "model.language_model.layers.31.self_attn.o_proj": 72,
+    "model.language_model.layers.31.self_attn.q_proj": 72,
+    "model.language_model.layers.31.self_attn.v_proj": 72,
+    "model.language_model.layers.35.mlp.down_proj": 72,
+    "model.language_model.layers.35.mlp.gate_proj": 72,
+    "model.language_model.layers.35.mlp.up_proj": 72,
+    "model.language_model.layers.35.self_attn.k_proj": 72,
+    "model.language_model.layers.35.self_attn.o_proj": 72,
+    "model.language_model.layers.35.self_attn.q_proj": 72,
+    "model.language_model.layers.35.self_attn.v_proj": 72,
+    "model.language_model.layers.4.mlp.down_proj": 72,
+    "model.language_model.layers.4.mlp.gate_proj": 72,
+    "model.language_model.layers.4.mlp.up_proj": 72,
+    "model.language_model.layers.4.self_attn.k_proj": 72,
+    "model.language_model.layers.4.self_attn.o_proj": 72,
+    "model.language_model.layers.4.self_attn.q_proj": 72,
+    "model.language_model.layers.4.self_attn.v_proj": 72,
+    "model.language_model.layers.9.mlp.down_proj": 72,
+    "model.language_model.layers.9.mlp.gate_proj": 72,
+    "model.language_model.layers.9.mlp.up_proj": 72,
+    "model.language_model.layers.9.self_attn.k_proj": 72,
+    "model.language_model.layers.9.self_attn.o_proj": 72,
+    "model.language_model.layers.9.self_attn.q_proj": 72,
+    "model.language_model.layers.9.self_attn.v_proj": 72
+  },
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.4.self_attn.k_proj",
+    "model.language_model.layers.1.mlp.gate_proj",
+    "model.language_model.layers.10.self_attn.k_proj",
+    "model.language_model.layers.21.mlp.down_proj",
+    "model.language_model.layers.10.mlp.down_proj",
+    "model.language_model.layers.31.self_attn.o_proj",
+    "model.language_model.layers.35.mlp.up_proj",
+    "model.language_model.layers.1.self_attn.v_proj",
+    "model.language_model.layers.1.mlp.down_proj",
+    "model.language_model.layers.9.mlp.up_proj",
+    "model.language_model.layers.28.mlp.gate_proj",
+    "model.language_model.layers.21.mlp.gate_proj",
+    "model.language_model.layers.31.self_attn.v_proj",
+    "model.language_model.layers.31.self_attn.k_proj",
+    "model.language_model.layers.4.self_attn.o_proj",
+    "model.language_model.layers.9.mlp.gate_proj",
+    "model.language_model.layers.21.self_attn.v_proj",
+    "model.language_model.layers.35.self_attn.v_proj",
+    "model.language_model.layers.35.mlp.down_proj",
+    "model.language_model.layers.21.self_attn.q_proj",
+    "model.language_model.layers.28.self_attn.v_proj",
+    "model.language_model.layers.9.self_attn.q_proj",
+    "model.language_model.layers.21.mlp.up_proj",
+    "model.language_model.layers.9.mlp.down_proj",
+    "model.language_model.layers.28.self_attn.q_proj",
+    "model.language_model.layers.31.self_attn.q_proj",
+    "model.language_model.layers.10.self_attn.o_proj",
+    "model.language_model.layers.1.self_attn.q_proj",
+    "model.language_model.layers.35.self_attn.k_proj",
+    "model.language_model.layers.4.mlp.up_proj",
+    "model.language_model.layers.1.self_attn.k_proj",
+    "model.language_model.layers.35.self_attn.o_proj",
+    "model.language_model.layers.31.mlp.down_proj",
+    "model.language_model.layers.10.self_attn.q_proj",
+    "model.language_model.layers.1.mlp.up_proj",
+    "model.language_model.layers.28.mlp.up_proj",
+    "model.language_model.layers.9.self_attn.v_proj",
+    "model.language_model.layers.10.self_attn.v_proj",
+    "model.language_model.layers.28.self_attn.o_proj",
+    "model.language_model.layers.10.mlp.gate_proj",
+    "model.language_model.layers.4.mlp.down_proj",
+    "model.language_model.layers.4.mlp.gate_proj",
+    "model.language_model.layers.9.self_attn.k_proj",
+    "model.language_model.layers.31.mlp.up_proj",
+    "model.language_model.layers.31.mlp.gate_proj",
+    "model.language_model.layers.21.self_attn.k_proj",
+    "model.language_model.layers.28.self_attn.k_proj",
+    "model.language_model.layers.4.self_attn.q_proj",
+    "model.language_model.layers.28.mlp.down_proj",
+    "model.language_model.layers.1.self_attn.o_proj",
+    "model.language_model.layers.9.self_attn.o_proj",
+    "model.language_model.layers.10.mlp.up_proj",
+    "model.language_model.layers.35.mlp.gate_proj",
+    "model.language_model.layers.4.self_attn.v_proj",
+    "model.language_model.layers.35.self_attn.q_proj",
+    "model.language_model.layers.21.self_attn.o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/fixed_internvl/removal/Random8_8101_s13/config.json

@@ -0,0 +1,43 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "world_size": 1,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "arm": {
+    "arm": "Random8_8101",
+    "layers": [
+      1,
+      4,
+      9,
+      10,
+      21,
+      28,
+      31,
+      35
+    ],
+    "ranks": {
+      "1": 72,
+      "4": 72,
+      "9": 72,
+      "10": 72,
+      "21": 72,
+      "28": 72,
+      "31": 72,
+      "35": 72
+    },
+    "parameters": 43646976,
+    "budget": 43646976
+  }
+}

runs/fixed_internvl/removal/Random8_8101_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 3.9568498134613037, "loss_scale": 4.0, "retries": 0, "seconds": 5.6, "max_memory_gb": 16.83, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 3.7762367725372314, "loss_scale": 4.0, "retries": 0, "seconds": 10.3, "max_memory_gb": 17.17, "ce": 0.7790627728209074, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 13.604935646057129, "loss_scale": 4.0, "retries": 0, "seconds": 14.6, "max_memory_gb": 17.18, "ce": 1.2965743411332369, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 5.797018527984619, "loss_scale": 4.0, "retries": 0, "seconds": 19.2, "max_memory_gb": 17.18, "ce": 0.6225569332145824, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.12452664226293564, "loss_scale": 4.0, "retries": 0, "seconds": 24.0, "max_memory_gb": 17.18, "ce": 0.03959712369578483, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 2.6275978088378906, "loss_scale": 4.0, "retries": 0, "seconds": 28.6, "max_memory_gb": 17.18, "ce": 0.32501460958155803, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 3.3463478088378906, "loss_scale": 4.0, "retries": 0, "seconds": 33.1, "max_memory_gb": 17.18, "ce": 0.4815762882317358, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 1.8358675241470337, "loss_scale": 4.0, "retries": 0, "seconds": 37.9, "max_memory_gb": 17.18, "ce": 0.40443418454378843, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 1.9803768396377563, "loss_scale": 4.0, "retries": 0, "seconds": 42.5, "max_memory_gb": 17.18, "ce": 0.32655010162852705, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.3339636027812958, "loss_scale": 4.0, "retries": 0, "seconds": 47.2, "max_memory_gb": 17.18, "ce": 0.13376795437943656, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.4194285571575165, "loss_scale": 4.0, "retries": 0, "seconds": 51.8, "max_memory_gb": 17.18, "ce": 0.21605679792992305, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 0.27397018671035767, "loss_scale": 4.0, "retries": 0, "seconds": 56.3, "max_memory_gb": 17.18, "ce": 0.11791098889443674, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 0.4256654977798462, "loss_scale": 4.0, "retries": 0, "seconds": 60.9, "max_memory_gb": 17.18, "ce": 0.2410823161535518, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 2.253268241882324, "loss_scale": 4.0, "retries": 0, "seconds": 66.8, "max_memory_gb": 17.18, "ce": 0.3631359206046909, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 0.7756187319755554, "loss_scale": 4.0, "retries": 0, "seconds": 71.5, "max_memory_gb": 17.18, "ce": 0.237314545192703, "rank": 0.07412397861480713}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.3022322952747345, "loss_scale": 4.0, "retries": 0, "seconds": 76.1, "max_memory_gb": 17.18, "ce": 0.09796632788493298, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 1.14309823513031, "loss_scale": 4.0, "retries": 0, "seconds": 80.8, "max_memory_gb": 17.18, "ce": 0.24741197963885497, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 0.8501092791557312, "loss_scale": 4.0, "retries": 0, "seconds": 84.7, "max_memory_gb": 17.18, "ce": 0.21134046063525602, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 1.1345562934875488, "loss_scale": 4.0, "retries": 0, "seconds": 89.4, "max_memory_gb": 17.18, "ce": 0.20312918168929173, "rank": 0.1252088025212288}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.24344049394130707, "loss_scale": 4.0, "retries": 0, "seconds": 93.9, "max_memory_gb": 17.18, "ce": 0.028700710186967626, "rank": 0.12307664006948471}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.6764103174209595, "loss_scale": 4.0, "retries": 0, "seconds": 98.3, "max_memory_gb": 17.18, "ce": 0.1295334679307416, "rank": 0.03339735418558121}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.8175079226493835, "loss_scale": 4.0, "retries": 0, "seconds": 102.9, "max_memory_gb": 17.18, "ce": 0.16863948020909447, "rank": 0.10701080411672592}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 1.0852044820785522, "loss_scale": 4.0, "retries": 0, "seconds": 107.6, "max_memory_gb": 17.18, "ce": 0.12386723174859071, "rank": 0.0}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 0.6146025657653809, "loss_scale": 4.0, "retries": 0, "seconds": 112.2, "max_memory_gb": 17.18, "ce": 0.07236738700885326, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 2.92268705368042, "loss_scale": 4.0, "retries": 0, "seconds": 116.9, "max_memory_gb": 17.18, "ce": 0.19893384390161373, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 3.69590163230896, "loss_scale": 4.0, "retries": 0, "seconds": 121.6, "max_memory_gb": 17.18, "ce": 0.21478138442034833, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 4.084661483764648, "loss_scale": 4.0, "retries": 0, "seconds": 126.1, "max_memory_gb": 17.18, "ce": 0.19729050542446203, "rank": 0.01240219920873642}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 1.5493613481521606, "loss_scale": 4.0, "retries": 0, "seconds": 130.8, "max_memory_gb": 17.18, "ce": 0.1932385705604247, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 3.20501708984375, "loss_scale": 4.0, "retries": 0, "seconds": 135.3, "max_memory_gb": 17.18, "ce": 0.5819581701871357, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 0.175001323223114, "loss_scale": 4.0, "retries": 0, "seconds": 139.9, "max_memory_gb": 17.18, "ce": 0.012267685134247586, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 4.401132106781006, "loss_scale": 4.0, "retries": 0, "seconds": 144.4, "max_memory_gb": 17.18, "ce": 0.6924290051056232, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 0.48486965894699097, "loss_scale": 4.0, "retries": 0, "seconds": 149.1, "max_memory_gb": 17.18, "ce": 0.03489955856275628, "rank": 0.0}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 2.6466047763824463, "loss_scale": 4.0, "retries": 0, "seconds": 153.6, "max_memory_gb": 17.18, "ce": 0.2651484800862818, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.7438403367996216, "loss_scale": 4.0, "retries": 0, "seconds": 158.1, "max_memory_gb": 17.18, "ce": 0.07979918423370691, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.9231069087982178, "loss_scale": 4.0, "retries": 0, "seconds": 162.8, "max_memory_gb": 17.18, "ce": 0.01925926461990457, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 2.716668128967285, "loss_scale": 4.0, "retries": 0, "seconds": 167.3, "max_memory_gb": 17.18, "ce": 0.23487227603436622, "rank": 0.0}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.9733133316040039, "loss_scale": 4.0, "retries": 0, "seconds": 171.9, "max_memory_gb": 17.18, "ce": 0.17810123944218503, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 1.1833362579345703, "loss_scale": 4.0, "retries": 0, "seconds": 176.5, "max_memory_gb": 17.18, "ce": 0.1366238938717288, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 1.3708266019821167, "loss_scale": 4.0, "retries": 0, "seconds": 181.3, "max_memory_gb": 17.18, "ce": 0.17116698129029828, "rank": 0.12522007524967194}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.16982834041118622, "loss_scale": 4.0, "retries": 0, "seconds": 185.9, "max_memory_gb": 17.18, "ce": 0.015148623402637895, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 2.5789620876312256, "loss_scale": 4.0, "retries": 0, "seconds": 190.4, "max_memory_gb": 17.18, "ce": 0.3907042252458268, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.08515830338001251, "loss_scale": 4.0, "retries": 0, "seconds": 195.2, "max_memory_gb": 17.18, "ce": 0.007229226520848897, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.22624832391738892, "loss_scale": 4.0, "retries": 0, "seconds": 199.7, "max_memory_gb": 17.18, "ce": 0.01257861324802434, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 3.9323668479919434, "loss_scale": 4.0, "retries": 0, "seconds": 204.3, "max_memory_gb": 17.18, "ce": 0.22022492271571537, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 2.7694342136383057, "loss_scale": 4.0, "retries": 0, "seconds": 209.1, "max_memory_gb": 17.18, "ce": 0.3357751646253746, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.7770522236824036, "loss_scale": 4.0, "retries": 0, "seconds": 213.7, "max_memory_gb": 17.18, "ce": 0.09310372351501428, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.7346490025520325, "loss_scale": 4.0, "retries": 0, "seconds": 218.4, "max_memory_gb": 17.18, "ce": 0.04596217098787747, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.05146433413028717, "loss_scale": 4.0, "retries": 0, "seconds": 223.1, "max_memory_gb": 17.18, "ce": 0.0030324748054226802, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.11017880588769913, "loss_scale": 4.0, "retries": 0, "seconds": 227.8, "max_memory_gb": 17.18, "ce": 0.010650482106029813, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 1.8457382917404175, "loss_scale": 4.0, "retries": 0, "seconds": 232.3, "max_memory_gb": 17.18, "ce": 0.12482464800814341, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 2.8317928314208984, "loss_scale": 4.0, "retries": 0, "seconds": 236.7, "max_memory_gb": 17.18, "ce": 0.09079118695353827, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.3814280331134796, "loss_scale": 4.0, "retries": 0, "seconds": 241.3, "max_memory_gb": 17.18, "ce": 0.029797104253702855, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 1.1276423931121826, "loss_scale": 4.0, "retries": 0, "seconds": 245.9, "max_memory_gb": 17.18, "ce": 0.08445728291326304, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 2.1444573402404785, "loss_scale": 4.0, "retries": 0, "seconds": 250.5, "max_memory_gb": 17.18, "ce": 0.12161015740184666, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.31696027517318726, "loss_scale": 4.0, "retries": 0, "seconds": 255.1, "max_memory_gb": 17.18, "ce": 0.019657737565012212, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.424528032541275, "loss_scale": 4.0, "retries": 0, "seconds": 259.7, "max_memory_gb": 17.18, "ce": 0.027529806004622515, "rank": 0.0}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.14362183213233948, "loss_scale": 4.0, "retries": 0, "seconds": 264.2, "max_memory_gb": 17.18, "ce": 0.01121448264711944, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.07130568474531174, "loss_scale": 4.0, "retries": 0, "seconds": 268.9, "max_memory_gb": 17.18, "ce": 0.0052339950507303, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 1.8764368295669556, "loss_scale": 4.0, "retries": 0, "seconds": 273.6, "max_memory_gb": 17.18, "ce": 0.16769291010132292, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 1.127175211906433, "loss_scale": 4.0, "retries": 0, "seconds": 278.1, "max_memory_gb": 17.18, "ce": 0.08228716103303668, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 1.5058358907699585, "loss_scale": 4.0, "retries": 0, "seconds": 282.7, "max_memory_gb": 17.18, "ce": 0.21712581249175855, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.3999904692173004, "loss_scale": 4.0, "retries": 0, "seconds": 287.2, "max_memory_gb": 17.18, "ce": 0.031165862488705898, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.520976185798645, "loss_scale": 4.0, "retries": 0, "seconds": 291.8, "max_memory_gb": 17.18, "ce": 0.03452941853811353, "rank": 0.0}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.25319671630859375, "loss_scale": 4.0, "retries": 0, "seconds": 296.3, "max_memory_gb": 17.18, "ce": 0.011834219763841247, "rank": 0.0}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 1.1691862344741821, "loss_scale": 4.0, "retries": 0, "seconds": 300.8, "max_memory_gb": 17.18, "ce": 0.15198847756619216, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 2.15779185295105, "loss_scale": 4.0, "retries": 0, "seconds": 305.3, "max_memory_gb": 17.18, "ce": 0.24370302590591564, "rank": 0.11981849372386932}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.03614472597837448, "loss_scale": 4.0, "retries": 0, "seconds": 309.8, "max_memory_gb": 17.18, "ce": 0.00295404838561808, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.12341621518135071, "loss_scale": 4.0, "retries": 0, "seconds": 314.3, "max_memory_gb": 17.18, "ce": 0.00803229820485285, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 3.340817928314209, "loss_scale": 4.0, "retries": 0, "seconds": 319.1, "max_memory_gb": 17.18, "ce": 0.6334924872617194, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.2318660467863083, "loss_scale": 4.0, "retries": 0, "seconds": 323.8, "max_memory_gb": 17.18, "ce": 0.0051322051249371725, "rank": 0.0}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.07498499006032944, "loss_scale": 4.0, "retries": 0, "seconds": 328.5, "max_memory_gb": 17.18, "ce": 0.0048852702657313785, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 1.0341066122055054, "loss_scale": 4.0, "retries": 0, "seconds": 333.3, "max_memory_gb": 17.18, "ce": 0.08735296015095173, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 1.0885772705078125, "loss_scale": 4.0, "retries": 0, "seconds": 338.0, "max_memory_gb": 17.18, "ce": 0.11432382163025068, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 1.1170188188552856, "loss_scale": 4.0, "retries": 0, "seconds": 342.8, "max_memory_gb": 17.18, "ce": 0.14449791378604004, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 2.9300477504730225, "loss_scale": 4.0, "retries": 0, "seconds": 347.5, "max_memory_gb": 17.18, "ce": 0.7530341952765411, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 1.463902473449707, "loss_scale": 4.0, "retries": 0, "seconds": 352.1, "max_memory_gb": 17.18, "ce": 0.31604537957002776, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 1.7753396034240723, "loss_scale": 4.0, "retries": 0, "seconds": 356.7, "max_memory_gb": 17.18, "ce": 0.07258384150031816, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.27083447575569153, "loss_scale": 4.0, "retries": 0, "seconds": 361.1, "max_memory_gb": 17.18, "ce": 0.024872354817489395, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.1270778477191925, "loss_scale": 4.0, "retries": 0, "seconds": 365.8, "max_memory_gb": 17.18, "ce": 0.006639418289068999, "rank": 0.0}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.21298369765281677, "loss_scale": 4.0, "retries": 0, "seconds": 370.4, "max_memory_gb": 17.18, "ce": 0.014110190934843558, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.4824538826942444, "loss_scale": 4.0, "retries": 0, "seconds": 375.1, "max_memory_gb": 17.18, "ce": 0.04345392128948333, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 1.752191185951233, "loss_scale": 4.0, "retries": 0, "seconds": 379.8, "max_memory_gb": 17.18, "ce": 0.17202648043689805, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 1.3572489023208618, "loss_scale": 4.0, "retries": 0, "seconds": 384.3, "max_memory_gb": 17.18, "ce": 0.19682317048591358, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.5748764276504517, "loss_scale": 4.0, "retries": 0, "seconds": 389.0, "max_memory_gb": 17.18, "ce": 0.0377574400197318, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.055044397711753845, "loss_scale": 4.0, "retries": 0, "seconds": 393.7, "max_memory_gb": 17.18, "ce": 0.0019518403837537335, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.3363531827926636, "loss_scale": 4.0, "retries": 0, "seconds": 398.5, "max_memory_gb": 17.18, "ce": 0.036944232339919836, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 1.0048577785491943, "loss_scale": 4.0, "retries": 0, "seconds": 403.4, "max_memory_gb": 17.18, "ce": 0.15251427253315342, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.4393872320652008, "loss_scale": 4.0, "retries": 0, "seconds": 408.7, "max_memory_gb": 17.18, "ce": 0.05302448389920755, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.200414776802063, "loss_scale": 4.0, "retries": 0, "seconds": 413.3, "max_memory_gb": 17.18, "ce": 0.020218208296228113, "rank": 0.12498898804187775}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.7043676972389221, "loss_scale": 4.0, "retries": 0, "seconds": 417.8, "max_memory_gb": 17.18, "ce": 0.0740542433823066, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 2.987546682357788, "loss_scale": 4.0, "retries": 0, "seconds": 422.4, "max_memory_gb": 17.18, "ce": 0.5503916982638657, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 2.1499342918395996, "loss_scale": 4.0, "retries": 0, "seconds": 427.1, "max_memory_gb": 17.18, "ce": 0.12739420952311775, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 1.1000418663024902, "loss_scale": 4.0, "retries": 0, "seconds": 431.9, "max_memory_gb": 17.18, "ce": 0.08809661977784344, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 0.8885397911071777, "loss_scale": 4.0, "retries": 0, "seconds": 436.5, "max_memory_gb": 17.18, "ce": 0.04099342273821094, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 3.0849361419677734, "loss_scale": 4.0, "retries": 0, "seconds": 441.3, "max_memory_gb": 17.18, "ce": 0.49637497201729275, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 2.8003487586975098, "loss_scale": 4.0, "retries": 0, "seconds": 446.0, "max_memory_gb": 17.18, "ce": 0.37501536394620416, "rank": 0.31643542647361755}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.4157310128211975, "loss_scale": 4.0, "retries": 0, "seconds": 450.5, "max_memory_gb": 17.18, "ce": 0.0322153510426233, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.8591516613960266, "loss_scale": 4.0, "retries": 0, "seconds": 455.2, "max_memory_gb": 17.18, "ce": 0.04930701281864458, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.10659565031528473, "loss_scale": 4.0, "retries": 0, "seconds": 459.8, "max_memory_gb": 17.18, "ce": 0.011366640965661645, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.5689082741737366, "loss_scale": 4.0, "retries": 0, "seconds": 464.5, "max_memory_gb": 17.18, "ce": 0.08083082923758411, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.2076612114906311, "loss_scale": 4.0, "retries": 0, "seconds": 469.1, "max_memory_gb": 17.18, "ce": 0.015128993528378487, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.1644233763217926, "loss_scale": 4.0, "retries": 0, "seconds": 473.8, "max_memory_gb": 17.18, "ce": 0.01207973696909903, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.10670573264360428, "loss_scale": 4.0, "retries": 0, "seconds": 478.4, "max_memory_gb": 17.18, "ce": 0.006678233687125612, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.27410009503364563, "loss_scale": 4.0, "retries": 0, "seconds": 483.1, "max_memory_gb": 17.18, "ce": 0.017963296395464567, "rank": 0.1249653622508049}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.6618877053260803, "loss_scale": 4.0, "retries": 0, "seconds": 487.7, "max_memory_gb": 17.18, "ce": 0.08263613116969282, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.0954851284623146, "loss_scale": 4.0, "retries": 0, "seconds": 492.6, "max_memory_gb": 17.18, "ce": 0.0035268257038296724, "rank": 0.10698921978473663}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 1.071466088294983, "loss_scale": 4.0, "retries": 0, "seconds": 497.3, "max_memory_gb": 17.18, "ce": 0.09821686947907438, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 2.2316529750823975, "loss_scale": 4.0, "retries": 0, "seconds": 501.5, "max_memory_gb": 17.18, "ce": 0.26417944766103574, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 1.4722273349761963, "loss_scale": 4.0, "retries": 0, "seconds": 506.3, "max_memory_gb": 17.18, "ce": 0.16307062886608037, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.7023147344589233, "loss_scale": 4.0, "retries": 0, "seconds": 510.9, "max_memory_gb": 17.18, "ce": 0.0348267021488482, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.31423717737197876, "loss_scale": 4.0, "retries": 0, "seconds": 515.6, "max_memory_gb": 17.18, "ce": 0.024243445322099433, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.08522994071245193, "loss_scale": 4.0, "retries": 0, "seconds": 520.4, "max_memory_gb": 17.18, "ce": 0.005266609726731986, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.18255198001861572, "loss_scale": 4.0, "retries": 0, "seconds": 525.0, "max_memory_gb": 17.18, "ce": 0.016274595017421234, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.7330578565597534, "loss_scale": 4.0, "retries": 0, "seconds": 529.5, "max_memory_gb": 17.18, "ce": 0.11580841003387832, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 0.9386060833930969, "loss_scale": 4.0, "retries": 0, "seconds": 534.1, "max_memory_gb": 17.18, "ce": 0.032155599101542975, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.8451766967773438, "loss_scale": 4.0, "retries": 0, "seconds": 539.0, "max_memory_gb": 17.18, "ce": 0.16988177348366662, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.10762186348438263, "loss_scale": 4.0, "retries": 0, "seconds": 543.3, "max_memory_gb": 17.18, "ce": 0.005093204746799529, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 1.2181159257888794, "loss_scale": 4.0, "retries": 0, "seconds": 547.2, "max_memory_gb": 17.18, "ce": 0.23075612130185164, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.04571960121393204, "loss_scale": 4.0, "retries": 0, "seconds": 551.8, "max_memory_gb": 17.18, "ce": 0.002910886981908334, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 2.0632548332214355, "loss_scale": 4.0, "retries": 0, "seconds": 556.4, "max_memory_gb": 17.18, "ce": 0.11971682531338956, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.5222517251968384, "loss_scale": 4.0, "retries": 0, "seconds": 560.9, "max_memory_gb": 17.18, "ce": 0.05616734852026184, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 1.044014573097229, "loss_scale": 4.0, "retries": 0, "seconds": 565.5, "max_memory_gb": 17.18, "ce": 0.0983351083729076, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.44243714213371277, "loss_scale": 4.0, "retries": 0, "seconds": 570.2, "max_memory_gb": 17.18, "ce": 0.039527845683664964, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.6932283639907837, "loss_scale": 4.0, "retries": 0, "seconds": 574.8, "max_memory_gb": 17.18, "ce": 0.5158506545801629, "rank": 0.0}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.7976396679878235, "loss_scale": 4.0, "retries": 0, "seconds": 579.5, "max_memory_gb": 17.18, "ce": 0.15128254688511333, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 2.3467917442321777, "loss_scale": 4.0, "retries": 0, "seconds": 584.7, "max_memory_gb": 17.18, "ce": 0.4779499157821192, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.6716574430465698, "loss_scale": 4.0, "retries": 0, "seconds": 589.3, "max_memory_gb": 17.18, "ce": 0.02061717347373815, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 1.3934205770492554, "loss_scale": 4.0, "retries": 0, "seconds": 594.1, "max_memory_gb": 17.18, "ce": 0.16817230714832476, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 1.5029860734939575, "loss_scale": 4.0, "retries": 0, "seconds": 598.8, "max_memory_gb": 17.18, "ce": 0.100746081249099, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 0.8899848461151123, "loss_scale": 4.0, "retries": 0, "seconds": 603.4, "max_memory_gb": 17.18, "ce": 0.17250455468979453, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 3.028799533843994, "loss_scale": 4.0, "retries": 0, "seconds": 608.0, "max_memory_gb": 17.18, "ce": 0.30584878390936865, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 2.0518136024475098, "loss_scale": 4.0, "retries": 0, "seconds": 613.1, "max_memory_gb": 17.18, "ce": 0.23316927306609614, "rank": 0.0}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 1.3417061567306519, "loss_scale": 4.0, "retries": 0, "seconds": 617.6, "max_memory_gb": 17.18, "ce": 0.2926818527686237, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 1.1374199390411377, "loss_scale": 4.0, "retries": 0, "seconds": 622.5, "max_memory_gb": 17.18, "ce": 0.203420956811442, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 0.21438609063625336, "loss_scale": 4.0, "retries": 0, "seconds": 628.4, "max_memory_gb": 17.18, "ce": 0.012559760554267996, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 0.6544608473777771, "loss_scale": 4.0, "retries": 0, "seconds": 633.2, "max_memory_gb": 17.18, "ce": 0.037943729998801246, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.2770752012729645, "loss_scale": 4.0, "retries": 0, "seconds": 638.0, "max_memory_gb": 17.18, "ce": 0.015416570399565899, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 1.2804617881774902, "loss_scale": 4.0, "retries": 0, "seconds": 642.6, "max_memory_gb": 17.18, "ce": 0.09163595124982749, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.6609867215156555, "loss_scale": 4.0, "retries": 0, "seconds": 647.4, "max_memory_gb": 17.18, "ce": 0.037790050903026895, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.03288111835718155, "loss_scale": 4.0, "retries": 0, "seconds": 652.1, "max_memory_gb": 17.18, "ce": 0.0024486041216960075, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.04741397127509117, "loss_scale": 4.0, "retries": 0, "seconds": 656.7, "max_memory_gb": 17.18, "ce": 0.004103964054820608, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 1.1549409627914429, "loss_scale": 4.0, "retries": 0, "seconds": 661.3, "max_memory_gb": 17.18, "ce": 0.0744369787909136, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 2.334022283554077, "loss_scale": 4.0, "retries": 0, "seconds": 667.2, "max_memory_gb": 17.18, "ce": 0.2527537009386833, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.6276100873947144, "loss_scale": 4.0, "retries": 0, "seconds": 671.9, "max_memory_gb": 17.18, "ce": 0.0462488166484718, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.025549711659550667, "loss_scale": 4.0, "retries": 0, "seconds": 676.5, "max_memory_gb": 17.18, "ce": 0.0019211674737107387, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.13905265927314758, "loss_scale": 4.0, "retries": 0, "seconds": 681.3, "max_memory_gb": 17.18, "ce": 0.01392630332088629, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 1.649082899093628, "loss_scale": 4.0, "retries": 0, "seconds": 685.9, "max_memory_gb": 17.18, "ce": 0.4885216203155096, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.12120780348777771, "loss_scale": 4.0, "retries": 0, "seconds": 690.6, "max_memory_gb": 17.18, "ce": 0.008894561280840207, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.08137239515781403, "loss_scale": 4.0, "retries": 0, "seconds": 695.3, "max_memory_gb": 17.18, "ce": 0.006873757340144948, "rank": 0.12087884545326233}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.607549250125885, "loss_scale": 4.0, "retries": 0, "seconds": 699.9, "max_memory_gb": 17.18, "ce": 0.08600385946374445, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 1.2403247356414795, "loss_scale": 4.0, "retries": 0, "seconds": 704.5, "max_memory_gb": 17.18, "ce": 0.033346870249260974, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 0.43607985973358154, "loss_scale": 4.0, "retries": 0, "seconds": 709.2, "max_memory_gb": 17.18, "ce": 0.032927884354307935, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.6411385536193848, "loss_scale": 4.0, "retries": 0, "seconds": 713.8, "max_memory_gb": 17.18, "ce": 0.10252399812611657, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.3361033797264099, "loss_scale": 4.0, "retries": 0, "seconds": 718.4, "max_memory_gb": 17.18, "ce": 0.024009302371723606, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 2.0128695964813232, "loss_scale": 4.0, "retries": 0, "seconds": 722.8, "max_memory_gb": 17.18, "ce": 0.26040328316912564, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.05567608401179314, "loss_scale": 4.0, "retries": 0, "seconds": 727.5, "max_memory_gb": 17.18, "ce": 0.0032736642040163133, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.18668854236602783, "loss_scale": 4.0, "retries": 0, "seconds": 731.9, "max_memory_gb": 17.18, "ce": 0.016342022538083256, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 4.911190032958984, "loss_scale": 4.0, "retries": 0, "seconds": 736.5, "max_memory_gb": 17.18, "ce": 0.6002878136137042, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.20320682227611542, "loss_scale": 4.0, "retries": 0, "seconds": 741.3, "max_memory_gb": 17.18, "ce": 0.013355658455338926, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 1.0556538105010986, "loss_scale": 4.0, "retries": 0, "seconds": 745.9, "max_memory_gb": 17.18, "ce": 0.16817147035408198, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 1.2312995195388794, "loss_scale": 4.0, "retries": 0, "seconds": 750.5, "max_memory_gb": 17.18, "ce": 0.13279697312418648, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.6614229083061218, "loss_scale": 4.0, "retries": 0, "seconds": 755.9, "max_memory_gb": 17.18, "ce": 0.05421261424885415, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 2.942824602127075, "loss_scale": 4.0, "retries": 0, "seconds": 760.7, "max_memory_gb": 17.18, "ce": 0.3722640794353538, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 0.6629612445831299, "loss_scale": 4.0, "retries": 0, "seconds": 765.2, "max_memory_gb": 17.18, "ce": 0.0654453733891387, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.10749766230583191, "loss_scale": 4.0, "retries": 0, "seconds": 769.9, "max_memory_gb": 17.18, "ce": 0.010296032378619202, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 1.9086300134658813, "loss_scale": 4.0, "retries": 0, "seconds": 774.6, "max_memory_gb": 17.18, "ce": 0.2681336576101785, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.14173220098018646, "loss_scale": 4.0, "retries": 0, "seconds": 779.3, "max_memory_gb": 17.18, "ce": 0.009096045913338457, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.07164929807186127, "loss_scale": 4.0, "retries": 0, "seconds": 784.0, "max_memory_gb": 17.18, "ce": 0.005355962948442539, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 1.0323519706726074, "loss_scale": 4.0, "retries": 0, "seconds": 788.8, "max_memory_gb": 17.18, "ce": 0.3080095838804482, "rank": 0.0}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.015090575441718102, "loss_scale": 4.0, "retries": 0, "seconds": 794.7, "max_memory_gb": 17.18, "ce": 0.000566623870327021, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.4857264459133148, "loss_scale": 4.0, "retries": 0, "seconds": 800.3, "max_memory_gb": 17.18, "ce": 0.06658718696689903, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.0825604498386383, "loss_scale": 4.0, "retries": 0, "seconds": 806.4, "max_memory_gb": 17.18, "ce": 0.006728388106694183, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.2109023630619049, "loss_scale": 4.0, "retries": 0, "seconds": 812.4, "max_memory_gb": 17.18, "ce": 0.012594459753927367, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.6365481615066528, "loss_scale": 4.0, "retries": 0, "seconds": 819.2, "max_memory_gb": 17.18, "ce": 0.04553798204688064, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 1.568887710571289, "loss_scale": 4.0, "retries": 0, "seconds": 825.6, "max_memory_gb": 17.18, "ce": 0.13391346828075257, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.02176824025809765, "loss_scale": 4.0, "retries": 0, "seconds": 832.1, "max_memory_gb": 17.18, "ce": 0.001419511822405184, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 0.8999837040901184, "loss_scale": 4.0, "retries": 0, "seconds": 838.4, "max_memory_gb": 17.18, "ce": 0.27754175575682893, "rank": 0.12496555596590042}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 1.4207935333251953, "loss_scale": 4.0, "retries": 0, "seconds": 844.3, "max_memory_gb": 17.18, "ce": 0.10177148958791804, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 1.0421143770217896, "loss_scale": 4.0, "retries": 0, "seconds": 850.2, "max_memory_gb": 17.18, "ce": 0.3597520897733375, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.2057550996541977, "loss_scale": 4.0, "retries": 0, "seconds": 856.2, "max_memory_gb": 17.18, "ce": 0.01996334789919274, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.18892298638820648, "loss_scale": 4.0, "retries": 0, "seconds": 862.3, "max_memory_gb": 17.18, "ce": 0.010940265532099147, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 1.0318549871444702, "loss_scale": 4.0, "retries": 0, "seconds": 868.3, "max_memory_gb": 17.18, "ce": 0.11620934498932911, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.06544139981269836, "loss_scale": 4.0, "retries": 0, "seconds": 874.2, "max_memory_gb": 17.18, "ce": 0.0025792259348236257, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.39786723256111145, "loss_scale": 4.0, "retries": 0, "seconds": 879.2, "max_memory_gb": 17.18, "ce": 0.018869073160203698, "rank": 0.1591736599802971}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.5786656141281128, "loss_scale": 4.0, "retries": 0, "seconds": 885.2, "max_memory_gb": 17.18, "ce": 0.04818815086309769, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 2.0653693675994873, "loss_scale": 4.0, "retries": 0, "seconds": 891.5, "max_memory_gb": 17.18, "ce": 0.09301795537976432, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.6422439813613892, "loss_scale": 4.0, "retries": 0, "seconds": 897.4, "max_memory_gb": 17.18, "ce": 0.12615382918420437, "rank": 0.13099250197410583}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.2793688774108887, "loss_scale": 4.0, "retries": 0, "seconds": 903.3, "max_memory_gb": 17.18, "ce": 0.18542780762231814, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 1.9125328063964844, "loss_scale": 4.0, "retries": 0, "seconds": 909.2, "max_memory_gb": 17.18, "ce": 0.2747898209838695, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.18298007547855377, "loss_scale": 4.0, "retries": 0, "seconds": 913.9, "max_memory_gb": 17.18, "ce": 0.005575560658144241, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 1.917968511581421, "loss_scale": 4.0, "retries": 0, "seconds": 918.5, "max_memory_gb": 17.18, "ce": 0.15428993396039914, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.027644157409668, "loss_scale": 4.0, "retries": 0, "seconds": 923.1, "max_memory_gb": 17.18, "ce": 0.33925543055966045, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.014771933667361736, "loss_scale": 4.0, "retries": 0, "seconds": 927.6, "max_memory_gb": 17.18, "ce": 0.0010821279427091213, "rank": 0.12502481043338776}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.41603246331214905, "loss_scale": 4.0, "retries": 0, "seconds": 931.5, "max_memory_gb": 17.18, "ce": 0.04630678411444933, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.3672660291194916, "loss_scale": 4.0, "retries": 0, "seconds": 936.3, "max_memory_gb": 17.18, "ce": 0.07376269838664484, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.18515273928642273, "loss_scale": 4.0, "retries": 0, "seconds": 941.1, "max_memory_gb": 17.18, "ce": 0.009209623744482087, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 1.3754547834396362, "loss_scale": 4.0, "retries": 0, "seconds": 945.7, "max_memory_gb": 17.18, "ce": 0.13773489616096413, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.587562620639801, "loss_scale": 4.0, "retries": 0, "seconds": 950.3, "max_memory_gb": 17.18, "ce": 0.05322990139382, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 1.6799248456954956, "loss_scale": 4.0, "retries": 0, "seconds": 955.3, "max_memory_gb": 17.18, "ce": 0.11166287086052762, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 1.061173677444458, "loss_scale": 4.0, "retries": 0, "seconds": 959.9, "max_memory_gb": 17.18, "ce": 0.20864768728392846, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.6160846948623657, "loss_scale": 4.0, "retries": 0, "seconds": 965.0, "max_memory_gb": 17.18, "ce": 0.06184125104459781, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.02479529194533825, "loss_scale": 4.0, "retries": 0, "seconds": 969.5, "max_memory_gb": 17.18, "ce": 0.002155203998199795, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.23198525607585907, "loss_scale": 4.0, "retries": 0, "seconds": 974.1, "max_memory_gb": 17.18, "ce": 0.014955682772551881, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.3396233022212982, "loss_scale": 4.0, "retries": 0, "seconds": 978.7, "max_memory_gb": 17.18, "ce": 0.02623056818583791, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 0.6302710175514221, "loss_scale": 4.0, "retries": 0, "seconds": 983.0, "max_memory_gb": 17.18, "ce": 0.0855121318268175, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.29377880692481995, "loss_scale": 4.0, "retries": 0, "seconds": 987.8, "max_memory_gb": 17.18, "ce": 0.02035446086756565, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.3108270466327667, "loss_scale": 4.0, "retries": 0, "seconds": 992.3, "max_memory_gb": 17.18, "ce": 0.05933248035694305, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.06855019181966782, "loss_scale": 4.0, "retries": 0, "seconds": 996.9, "max_memory_gb": 17.18, "ce": 0.0026638936053586804, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.7180216312408447, "loss_scale": 4.0, "retries": 0, "seconds": 1001.6, "max_memory_gb": 17.18, "ce": 0.10034475438806112, "rank": 0.12509629130363464}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 1.2123417854309082, "loss_scale": 4.0, "retries": 0, "seconds": 1006.5, "max_memory_gb": 17.18, "ce": 0.3614841803279205, "rank": 0.0}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 1.269762635231018, "loss_scale": 4.0, "retries": 0, "seconds": 1011.1, "max_memory_gb": 17.18, "ce": 0.1239110970980164, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 1.044919729232788, "loss_scale": 4.0, "retries": 0, "seconds": 1015.7, "max_memory_gb": 17.18, "ce": 0.20506044909620869, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.6793874502182007, "loss_scale": 4.0, "retries": 0, "seconds": 1020.2, "max_memory_gb": 17.18, "ce": 0.14272373295125362, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 1.4572138786315918, "loss_scale": 4.0, "retries": 0, "seconds": 1024.7, "max_memory_gb": 17.18, "ce": 0.2314076632344495, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 2.0430381298065186, "loss_scale": 4.0, "retries": 0, "seconds": 1029.3, "max_memory_gb": 17.18, "ce": 0.41932565496404095, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.3664964437484741, "loss_scale": 4.0, "retries": 0, "seconds": 1034.1, "max_memory_gb": 17.18, "ce": 0.03177882240549934, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.05402015149593353, "loss_scale": 4.0, "retries": 0, "seconds": 1038.5, "max_memory_gb": 17.18, "ce": 0.0034090743495198694, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 0.9489364624023438, "loss_scale": 4.0, "retries": 0, "seconds": 1043.1, "max_memory_gb": 17.18, "ce": 0.17653950954399988, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.30876344442367554, "loss_scale": 4.0, "retries": 0, "seconds": 1047.7, "max_memory_gb": 17.18, "ce": 0.0402269325237512, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.35394421219825745, "loss_scale": 4.0, "retries": 0, "seconds": 1052.5, "max_memory_gb": 17.18, "ce": 0.025338642003816858, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.3662009537220001, "loss_scale": 4.0, "retries": 0, "seconds": 1057.5, "max_memory_gb": 17.18, "ce": 0.01960608511490136, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.35466983914375305, "loss_scale": 4.0, "retries": 0, "seconds": 1062.3, "max_memory_gb": 17.18, "ce": 0.024799567320201277, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 1.3424705266952515, "loss_scale": 4.0, "retries": 0, "seconds": 1066.9, "max_memory_gb": 17.18, "ce": 0.06323434559385532, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.40412065386772156, "loss_scale": 4.0, "retries": 0, "seconds": 1071.6, "max_memory_gb": 17.18, "ce": 0.02748280974356021, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.12658677995204926, "loss_scale": 4.0, "retries": 0, "seconds": 1076.9, "max_memory_gb": 17.18, "ce": 0.00899046039762652, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.48121994733810425, "loss_scale": 4.0, "retries": 0, "seconds": 1081.7, "max_memory_gb": 17.18, "ce": 0.026434079665136778, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.4732903242111206, "loss_scale": 4.0, "retries": 0, "seconds": 1086.4, "max_memory_gb": 17.18, "ce": 0.027921853268026098, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.10917413234710693, "loss_scale": 4.0, "retries": 0, "seconds": 1091.0, "max_memory_gb": 17.18, "ce": 0.009101145897375318, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.6078220009803772, "loss_scale": 4.0, "retries": 0, "seconds": 1095.6, "max_memory_gb": 17.18, "ce": 0.05474770057662681, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 2.2587943077087402, "loss_scale": 4.0, "retries": 0, "seconds": 1100.2, "max_memory_gb": 17.18, "ce": 0.3471306488411301, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.39379027485847473, "loss_scale": 4.0, "retries": 0, "seconds": 1104.6, "max_memory_gb": 17.18, "ce": 0.04416617441700055, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.26117536425590515, "loss_scale": 4.0, "retries": 0, "seconds": 1109.2, "max_memory_gb": 17.18, "ce": 0.015999980211063303, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.2767586410045624, "loss_scale": 4.0, "retries": 0, "seconds": 1114.0, "max_memory_gb": 17.18, "ce": 0.022583866033528466, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 0.6311593055725098, "loss_scale": 4.0, "retries": 0, "seconds": 1118.7, "max_memory_gb": 17.18, "ce": 0.04492552206977507, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 1.8138759136199951, "loss_scale": 4.0, "retries": 0, "seconds": 1123.3, "max_memory_gb": 17.18, "ce": 0.24057420815472597, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.15618480741977692, "loss_scale": 4.0, "retries": 0, "seconds": 1127.9, "max_memory_gb": 17.18, "ce": 0.015619832643437803, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 1.0374866724014282, "loss_scale": 4.0, "retries": 0, "seconds": 1132.6, "max_memory_gb": 17.18, "ce": 0.05445904547320879, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.41391804814338684, "loss_scale": 4.0, "retries": 0, "seconds": 1137.4, "max_memory_gb": 17.18, "ce": 0.03302727806817529, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.48727601766586304, "loss_scale": 4.0, "retries": 0, "seconds": 1142.2, "max_memory_gb": 17.18, "ce": 0.08568423442568474, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 0.3405400812625885, "loss_scale": 4.0, "retries": 0, "seconds": 1146.7, "max_memory_gb": 17.18, "ce": 0.022325589386355205, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.574748694896698, "loss_scale": 4.0, "retries": 0, "seconds": 1151.5, "max_memory_gb": 17.18, "ce": 0.07629982208959518, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.1269477754831314, "loss_scale": 4.0, "retries": 0, "seconds": 1157.0, "max_memory_gb": 17.18, "ce": 0.010382386647393105, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.020716868340969086, "loss_scale": 4.0, "retries": 0, "seconds": 1161.9, "max_memory_gb": 17.18, "ce": 0.0012908364092254487, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.47489699721336365, "loss_scale": 4.0, "retries": 0, "seconds": 1166.6, "max_memory_gb": 17.18, "ce": 0.04883110400442092, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 1.4521918296813965, "loss_scale": 4.0, "retries": 0, "seconds": 1171.2, "max_memory_gb": 17.18, "ce": 0.24223584710159685, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.024722497910261154, "loss_scale": 4.0, "retries": 0, "seconds": 1175.8, "max_memory_gb": 17.18, "ce": 0.0013984114870027042, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.6657779216766357, "loss_scale": 4.0, "retries": 0, "seconds": 1180.1, "max_memory_gb": 17.18, "ce": 0.045765583508114105, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.09809114784002304, "loss_scale": 4.0, "retries": 0, "seconds": 1184.8, "max_memory_gb": 17.18, "ce": 0.007020831533964156, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.18033266067504883, "loss_scale": 4.0, "retries": 0, "seconds": 1189.5, "max_memory_gb": 17.18, "ce": 0.013056184523179581, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.46283280849456787, "loss_scale": 4.0, "retries": 0, "seconds": 1194.0, "max_memory_gb": 17.18, "ce": 0.020502615789041556, "rank": 0.0}

runs/fixed_internvl/removal/Random8_8102_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/removal/Random8_8102_s13/adapter/adapter_config.json

@@ -0,0 +1,213 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.10.mlp.down_proj": 144,
+    "model.language_model.layers.10.mlp.gate_proj": 144,
+    "model.language_model.layers.10.mlp.up_proj": 144,
+    "model.language_model.layers.10.self_attn.k_proj": 144,
+    "model.language_model.layers.10.self_attn.o_proj": 144,
+    "model.language_model.layers.10.self_attn.q_proj": 144,
+    "model.language_model.layers.10.self_attn.v_proj": 144,
+    "model.language_model.layers.11.mlp.down_proj": 144,
+    "model.language_model.layers.11.mlp.gate_proj": 144,
+    "model.language_model.layers.11.mlp.up_proj": 144,
+    "model.language_model.layers.11.self_attn.k_proj": 144,
+    "model.language_model.layers.11.self_attn.o_proj": 144,
+    "model.language_model.layers.11.self_attn.q_proj": 144,
+    "model.language_model.layers.11.self_attn.v_proj": 144,
+    "model.language_model.layers.15.mlp.down_proj": 144,
+    "model.language_model.layers.15.mlp.gate_proj": 144,
+    "model.language_model.layers.15.mlp.up_proj": 144,
+    "model.language_model.layers.15.self_attn.k_proj": 144,
+    "model.language_model.layers.15.self_attn.o_proj": 144,
+    "model.language_model.layers.15.self_attn.q_proj": 144,
+    "model.language_model.layers.15.self_attn.v_proj": 144,
+    "model.language_model.layers.18.mlp.down_proj": 144,
+    "model.language_model.layers.18.mlp.gate_proj": 144,
+    "model.language_model.layers.18.mlp.up_proj": 144,
+    "model.language_model.layers.18.self_attn.k_proj": 144,
+    "model.language_model.layers.18.self_attn.o_proj": 144,
+    "model.language_model.layers.18.self_attn.q_proj": 144,
+    "model.language_model.layers.18.self_attn.v_proj": 144,
+    "model.language_model.layers.32.mlp.down_proj": 144,
+    "model.language_model.layers.32.mlp.gate_proj": 144,
+    "model.language_model.layers.32.mlp.up_proj": 144,
+    "model.language_model.layers.32.self_attn.k_proj": 144,
+    "model.language_model.layers.32.self_attn.o_proj": 144,
+    "model.language_model.layers.32.self_attn.q_proj": 144,
+    "model.language_model.layers.32.self_attn.v_proj": 144,
+    "model.language_model.layers.34.mlp.down_proj": 144,
+    "model.language_model.layers.34.mlp.gate_proj": 144,
+    "model.language_model.layers.34.mlp.up_proj": 144,
+    "model.language_model.layers.34.self_attn.k_proj": 144,
+    "model.language_model.layers.34.self_attn.o_proj": 144,
+    "model.language_model.layers.34.self_attn.q_proj": 144,
+    "model.language_model.layers.34.self_attn.v_proj": 144,
+    "model.language_model.layers.4.mlp.down_proj": 144,
+    "model.language_model.layers.4.mlp.gate_proj": 144,
+    "model.language_model.layers.4.mlp.up_proj": 144,
+    "model.language_model.layers.4.self_attn.k_proj": 144,
+    "model.language_model.layers.4.self_attn.o_proj": 144,
+    "model.language_model.layers.4.self_attn.q_proj": 144,
+    "model.language_model.layers.4.self_attn.v_proj": 144,
+    "model.language_model.layers.9.mlp.down_proj": 144,
+    "model.language_model.layers.9.mlp.gate_proj": 144,
+    "model.language_model.layers.9.mlp.up_proj": 144,
+    "model.language_model.layers.9.self_attn.k_proj": 144,
+    "model.language_model.layers.9.self_attn.o_proj": 144,
+    "model.language_model.layers.9.self_attn.q_proj": 144,
+    "model.language_model.layers.9.self_attn.v_proj": 144
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.10.mlp.down_proj": 72,
+    "model.language_model.layers.10.mlp.gate_proj": 72,
+    "model.language_model.layers.10.mlp.up_proj": 72,
+    "model.language_model.layers.10.self_attn.k_proj": 72,
+    "model.language_model.layers.10.self_attn.o_proj": 72,
+    "model.language_model.layers.10.self_attn.q_proj": 72,
+    "model.language_model.layers.10.self_attn.v_proj": 72,
+    "model.language_model.layers.11.mlp.down_proj": 72,
+    "model.language_model.layers.11.mlp.gate_proj": 72,
+    "model.language_model.layers.11.mlp.up_proj": 72,
+    "model.language_model.layers.11.self_attn.k_proj": 72,
+    "model.language_model.layers.11.self_attn.o_proj": 72,
+    "model.language_model.layers.11.self_attn.q_proj": 72,
+    "model.language_model.layers.11.self_attn.v_proj": 72,
+    "model.language_model.layers.15.mlp.down_proj": 72,
+    "model.language_model.layers.15.mlp.gate_proj": 72,
+    "model.language_model.layers.15.mlp.up_proj": 72,
+    "model.language_model.layers.15.self_attn.k_proj": 72,
+    "model.language_model.layers.15.self_attn.o_proj": 72,
+    "model.language_model.layers.15.self_attn.q_proj": 72,
+    "model.language_model.layers.15.self_attn.v_proj": 72,
+    "model.language_model.layers.18.mlp.down_proj": 72,
+    "model.language_model.layers.18.mlp.gate_proj": 72,
+    "model.language_model.layers.18.mlp.up_proj": 72,
+    "model.language_model.layers.18.self_attn.k_proj": 72,
+    "model.language_model.layers.18.self_attn.o_proj": 72,
+    "model.language_model.layers.18.self_attn.q_proj": 72,
+    "model.language_model.layers.18.self_attn.v_proj": 72,
+    "model.language_model.layers.32.mlp.down_proj": 72,
+    "model.language_model.layers.32.mlp.gate_proj": 72,
+    "model.language_model.layers.32.mlp.up_proj": 72,
+    "model.language_model.layers.32.self_attn.k_proj": 72,
+    "model.language_model.layers.32.self_attn.o_proj": 72,
+    "model.language_model.layers.32.self_attn.q_proj": 72,
+    "model.language_model.layers.32.self_attn.v_proj": 72,
+    "model.language_model.layers.34.mlp.down_proj": 72,
+    "model.language_model.layers.34.mlp.gate_proj": 72,
+    "model.language_model.layers.34.mlp.up_proj": 72,
+    "model.language_model.layers.34.self_attn.k_proj": 72,
+    "model.language_model.layers.34.self_attn.o_proj": 72,
+    "model.language_model.layers.34.self_attn.q_proj": 72,
+    "model.language_model.layers.34.self_attn.v_proj": 72,
+    "model.language_model.layers.4.mlp.down_proj": 72,
+    "model.language_model.layers.4.mlp.gate_proj": 72,
+    "model.language_model.layers.4.mlp.up_proj": 72,
+    "model.language_model.layers.4.self_attn.k_proj": 72,
+    "model.language_model.layers.4.self_attn.o_proj": 72,
+    "model.language_model.layers.4.self_attn.q_proj": 72,
+    "model.language_model.layers.4.self_attn.v_proj": 72,
+    "model.language_model.layers.9.mlp.down_proj": 72,
+    "model.language_model.layers.9.mlp.gate_proj": 72,
+    "model.language_model.layers.9.mlp.up_proj": 72,
+    "model.language_model.layers.9.self_attn.k_proj": 72,
+    "model.language_model.layers.9.self_attn.o_proj": 72,
+    "model.language_model.layers.9.self_attn.q_proj": 72,
+    "model.language_model.layers.9.self_attn.v_proj": 72
+  },
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.15.mlp.gate_proj",
+    "model.language_model.layers.34.self_attn.o_proj",
+    "model.language_model.layers.11.self_attn.v_proj",
+    "model.language_model.layers.10.self_attn.q_proj",
+    "model.language_model.layers.11.mlp.down_proj",
+    "model.language_model.layers.4.mlp.down_proj",
+    "model.language_model.layers.15.self_attn.o_proj",
+    "model.language_model.layers.15.mlp.up_proj",
+    "model.language_model.layers.4.self_attn.q_proj",
+    "model.language_model.layers.15.self_attn.v_proj",
+    "model.language_model.layers.15.self_attn.k_proj",
+    "model.language_model.layers.32.self_attn.v_proj",
+    "model.language_model.layers.34.mlp.down_proj",
+    "model.language_model.layers.10.mlp.down_proj",
+    "model.language_model.layers.10.self_attn.k_proj",
+    "model.language_model.layers.34.mlp.gate_proj",
+    "model.language_model.layers.4.mlp.up_proj",
+    "model.language_model.layers.10.mlp.up_proj",
+    "model.language_model.layers.34.self_attn.k_proj",
+    "model.language_model.layers.32.self_attn.q_proj",
+    "model.language_model.layers.32.mlp.up_proj",
+    "model.language_model.layers.10.self_attn.o_proj",
+    "model.language_model.layers.4.self_attn.v_proj",
+    "model.language_model.layers.11.mlp.gate_proj",
+    "model.language_model.layers.9.mlp.up_proj",
+    "model.language_model.layers.15.mlp.down_proj",
+    "model.language_model.layers.10.mlp.gate_proj",
+    "model.language_model.layers.15.self_attn.q_proj",
+    "model.language_model.layers.18.mlp.gate_proj",
+    "model.language_model.layers.4.mlp.gate_proj",
+    "model.language_model.layers.18.self_attn.q_proj",
+    "model.language_model.layers.32.self_attn.o_proj",
+    "model.language_model.layers.32.mlp.down_proj",
+    "model.language_model.layers.18.mlp.up_proj",
+    "model.language_model.layers.34.self_attn.q_proj",
+    "model.language_model.layers.34.mlp.up_proj",
+    "model.language_model.layers.18.self_attn.k_proj",
+    "model.language_model.layers.9.mlp.gate_proj",
+    "model.language_model.layers.9.self_attn.o_proj",
+    "model.language_model.layers.32.mlp.gate_proj",
+    "model.language_model.layers.9.mlp.down_proj",
+    "model.language_model.layers.34.self_attn.v_proj",
+    "model.language_model.layers.4.self_attn.o_proj",
+    "model.language_model.layers.9.self_attn.q_proj",
+    "model.language_model.layers.11.mlp.up_proj",
+    "model.language_model.layers.18.self_attn.v_proj",
+    "model.language_model.layers.11.self_attn.q_proj",
+    "model.language_model.layers.11.self_attn.o_proj",
+    "model.language_model.layers.18.self_attn.o_proj",
+    "model.language_model.layers.18.mlp.down_proj",
+    "model.language_model.layers.4.self_attn.k_proj",
+    "model.language_model.layers.11.self_attn.k_proj",
+    "model.language_model.layers.10.self_attn.v_proj",
+    "model.language_model.layers.9.self_attn.k_proj",
+    "model.language_model.layers.32.self_attn.k_proj",
+    "model.language_model.layers.9.self_attn.v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

runs/fixed_internvl/removal/Random8_8102_s13/config.json

@@ -0,0 +1,43 @@
+{
+  "behavior": "removal",
+  "seed": 13,
+  "model": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "backbone": "internvl",
+  "train": "/data/OPSD/assets/v3/removal.jsonl",
+  "items": 1000,
+  "steps": 250,
+  "epochs": 1,
+  "steps_per_epoch": 250,
+  "world_size": 1,
+  "lr": 5e-05,
+  "scheduler": "cosine",
+  "warmup_steps": 16,
+  "final_fraction": 0.1,
+  "clip": 1.0,
+  "initial_loss_scale": 4.0,
+  "arm": {
+    "arm": "Random8_8102",
+    "layers": [
+      4,
+      9,
+      10,
+      11,
+      15,
+      18,
+      32,
+      34
+    ],
+    "ranks": {
+      "4": 72,
+      "9": 72,
+      "10": 72,
+      "11": 72,
+      "15": 72,
+      "18": 72,
+      "32": 72,
+      "34": 72
+    },
+    "parameters": 43646976,
+    "budget": 43646976
+  }
+}

runs/fixed_internvl/removal/Random8_8102_s13/steps.jsonl

@@ -0,0 +1,250 @@
+{"step": 1, "steps": 250, "lr": 3.125e-06, "grad_norm": 4.844179630279541, "loss_scale": 4.0, "retries": 0, "seconds": 5.1, "max_memory_gb": 16.82, "ce": 0.7595979786710814, "rank": 0.06138449162244797}
+{"step": 2, "steps": 250, "lr": 6.25e-06, "grad_norm": 4.3511433601379395, "loss_scale": 4.0, "retries": 0, "seconds": 9.2, "max_memory_gb": 17.17, "ce": 0.7786771657156351, "rank": 0.0}
+{"step": 3, "steps": 250, "lr": 9.375000000000001e-06, "grad_norm": 14.94561767578125, "loss_scale": 4.0, "retries": 0, "seconds": 12.7, "max_memory_gb": 17.17, "ce": 1.2514219284057617, "rank": 0.0}
+{"step": 4, "steps": 250, "lr": 1.25e-05, "grad_norm": 5.328332424163818, "loss_scale": 4.0, "retries": 0, "seconds": 16.2, "max_memory_gb": 17.17, "ce": 0.5714567624527263, "rank": 0.0}
+{"step": 5, "steps": 250, "lr": 1.5625e-05, "grad_norm": 0.14669226109981537, "loss_scale": 4.0, "retries": 0, "seconds": 20.0, "max_memory_gb": 17.17, "ce": 0.03963614213716937, "rank": 0.0}
+{"step": 6, "steps": 250, "lr": 1.8750000000000002e-05, "grad_norm": 2.6883718967437744, "loss_scale": 4.0, "retries": 0, "seconds": 23.7, "max_memory_gb": 17.17, "ce": 0.308474262143136, "rank": 0.0}
+{"step": 7, "steps": 250, "lr": 2.1875e-05, "grad_norm": 1.9393088817596436, "loss_scale": 4.0, "retries": 0, "seconds": 27.5, "max_memory_gb": 17.17, "ce": 0.4151864342402405, "rank": 0.0}
+{"step": 8, "steps": 250, "lr": 2.5e-05, "grad_norm": 1.2061587572097778, "loss_scale": 4.0, "retries": 0, "seconds": 31.2, "max_memory_gb": 17.17, "ce": 0.3685875553637743, "rank": 0.0}
+{"step": 9, "steps": 250, "lr": 2.8125000000000003e-05, "grad_norm": 3.333446502685547, "loss_scale": 4.0, "retries": 0, "seconds": 35.0, "max_memory_gb": 17.17, "ce": 0.32168777426704764, "rank": 0.0}
+{"step": 10, "steps": 250, "lr": 3.125e-05, "grad_norm": 0.35413238406181335, "loss_scale": 4.0, "retries": 0, "seconds": 38.8, "max_memory_gb": 17.17, "ce": 0.13017215420404682, "rank": 0.0}
+{"step": 11, "steps": 250, "lr": 3.4375e-05, "grad_norm": 0.42844894528388977, "loss_scale": 4.0, "retries": 0, "seconds": 42.5, "max_memory_gb": 17.17, "ce": 0.20928040890066768, "rank": 0.0}
+{"step": 12, "steps": 250, "lr": 3.7500000000000003e-05, "grad_norm": 0.27436837553977966, "loss_scale": 4.0, "retries": 0, "seconds": 46.3, "max_memory_gb": 17.17, "ce": 0.11397152859808557, "rank": 0.0}
+{"step": 13, "steps": 250, "lr": 4.0625000000000005e-05, "grad_norm": 0.43469753861427307, "loss_scale": 4.0, "retries": 0, "seconds": 50.2, "max_memory_gb": 17.17, "ce": 0.23526306218082027, "rank": 0.0}
+{"step": 14, "steps": 250, "lr": 4.375e-05, "grad_norm": 2.0822701454162598, "loss_scale": 4.0, "retries": 0, "seconds": 53.9, "max_memory_gb": 17.17, "ce": 0.343903909670189, "rank": 0.0}
+{"step": 15, "steps": 250, "lr": 4.6875e-05, "grad_norm": 1.3025949001312256, "loss_scale": 4.0, "retries": 0, "seconds": 57.8, "max_memory_gb": 17.17, "ce": 0.22829187802744855, "rank": 0.013109564781188965}
+{"step": 16, "steps": 250, "lr": 5e-05, "grad_norm": 0.3236083984375, "loss_scale": 4.0, "retries": 0, "seconds": 61.5, "max_memory_gb": 17.17, "ce": 0.0903001449914882, "rank": 0.0}
+{"step": 17, "steps": 250, "lr": 5e-05, "grad_norm": 1.137690544128418, "loss_scale": 4.0, "retries": 0, "seconds": 65.2, "max_memory_gb": 17.17, "ce": 0.2213946279080119, "rank": 0.0}
+{"step": 18, "steps": 250, "lr": 4.999797225177623e-05, "grad_norm": 0.73683100938797, "loss_scale": 4.0, "retries": 0, "seconds": 69.1, "max_memory_gb": 17.17, "ce": 0.21727499799453653, "rank": 0.0}
+{"step": 19, "steps": 250, "lr": 4.999188937259492e-05, "grad_norm": 0.8399162888526917, "loss_scale": 4.0, "retries": 0, "seconds": 72.9, "max_memory_gb": 17.17, "ce": 0.21020351076731458, "rank": 0.10380304604768753}
+{"step": 20, "steps": 250, "lr": 4.998175245886032e-05, "grad_norm": 0.2905007004737854, "loss_scale": 4.0, "retries": 0, "seconds": 76.6, "max_memory_gb": 17.17, "ce": 0.027466858060506638, "rank": 0.12524062395095825}
+{"step": 21, "steps": 250, "lr": 4.996756333769319e-05, "grad_norm": 0.6000041365623474, "loss_scale": 4.0, "retries": 0, "seconds": 80.3, "max_memory_gb": 17.17, "ce": 0.12689167266944423, "rank": 0.04700116068124771}
+{"step": 22, "steps": 250, "lr": 4.994932456660157e-05, "grad_norm": 0.8932632803916931, "loss_scale": 4.0, "retries": 0, "seconds": 84.2, "max_memory_gb": 17.17, "ce": 0.16064449722034624, "rank": 0.11578066647052765}
+{"step": 23, "steps": 250, "lr": 4.992703943301973e-05, "grad_norm": 0.8265670537948608, "loss_scale": 4.0, "retries": 0, "seconds": 88.0, "max_memory_gb": 17.17, "ce": 0.13513414390035905, "rank": 0.0}
+{"step": 24, "steps": 250, "lr": 4.9900711953715675e-05, "grad_norm": 0.5909660458564758, "loss_scale": 4.0, "retries": 0, "seconds": 91.7, "max_memory_gb": 17.17, "ce": 0.0874569135485217, "rank": 0.0}
+{"step": 25, "steps": 250, "lr": 4.987034687406713e-05, "grad_norm": 2.0445191860198975, "loss_scale": 4.0, "retries": 0, "seconds": 95.6, "max_memory_gb": 17.17, "ce": 0.19671757415198954, "rank": 0.0}
+{"step": 26, "steps": 250, "lr": 4.9835949667206215e-05, "grad_norm": 2.146022081375122, "loss_scale": 4.0, "retries": 0, "seconds": 99.4, "max_memory_gb": 17.17, "ce": 0.17728892661398277, "rank": 0.0}
+{"step": 27, "steps": 250, "lr": 4.9797526533032945e-05, "grad_norm": 3.515455722808838, "loss_scale": 4.0, "retries": 0, "seconds": 103.1, "max_memory_gb": 17.17, "ce": 0.20325405194307677, "rank": 0.08668495714664459}
+{"step": 28, "steps": 250, "lr": 4.9755084397097716e-05, "grad_norm": 1.3828215599060059, "loss_scale": 4.0, "retries": 0, "seconds": 106.9, "max_memory_gb": 17.17, "ce": 0.18381694089475786, "rank": 0.0}
+{"step": 29, "steps": 250, "lr": 4.970863090935304e-05, "grad_norm": 4.315817832946777, "loss_scale": 4.0, "retries": 0, "seconds": 110.7, "max_memory_gb": 17.17, "ce": 0.5747861217023456, "rank": 0.0}
+{"step": 30, "steps": 250, "lr": 4.965817444277468e-05, "grad_norm": 0.1437193900346756, "loss_scale": 4.0, "retries": 0, "seconds": 114.4, "max_memory_gb": 17.17, "ce": 0.00821324112257571, "rank": 0.0}
+{"step": 31, "steps": 250, "lr": 4.9603724091852456e-05, "grad_norm": 5.48463773727417, "loss_scale": 4.0, "retries": 0, "seconds": 118.2, "max_memory_gb": 17.17, "ce": 0.7734333815697028, "rank": 0.0}
+{"step": 32, "steps": 250, "lr": 4.9545289670951025e-05, "grad_norm": 0.28492501378059387, "loss_scale": 4.0, "retries": 0, "seconds": 122.1, "max_memory_gb": 17.17, "ce": 0.015958218387822853, "rank": 0.0}
+{"step": 33, "steps": 250, "lr": 4.948288171254089e-05, "grad_norm": 3.2092437744140625, "loss_scale": 4.0, "retries": 0, "seconds": 125.8, "max_memory_gb": 17.17, "ce": 0.21382187397466623, "rank": 0.0}
+{"step": 34, "steps": 250, "lr": 4.941651146529998e-05, "grad_norm": 0.7258099317550659, "loss_scale": 4.0, "retries": 0, "seconds": 129.6, "max_memory_gb": 17.17, "ce": 0.08874697198370995, "rank": 0.0}
+{"step": 35, "steps": 250, "lr": 4.9346190892086174e-05, "grad_norm": 0.73626309633255, "loss_scale": 4.0, "retries": 0, "seconds": 133.4, "max_memory_gb": 17.17, "ce": 0.009492292321738205, "rank": 0.0}
+{"step": 36, "steps": 250, "lr": 4.927193266778101e-05, "grad_norm": 2.3545548915863037, "loss_scale": 4.0, "retries": 0, "seconds": 137.2, "max_memory_gb": 17.17, "ce": 0.2266004098146368, "rank": 0.0}
+{"step": 37, "steps": 250, "lr": 4.919375017700515e-05, "grad_norm": 0.7838371992111206, "loss_scale": 4.0, "retries": 0, "seconds": 140.9, "max_memory_gb": 17.17, "ce": 0.1884981338025682, "rank": 0.0}
+{"step": 38, "steps": 250, "lr": 4.911165751170588e-05, "grad_norm": 0.9446011185646057, "loss_scale": 4.0, "retries": 0, "seconds": 144.7, "max_memory_gb": 17.17, "ce": 0.12583313461800572, "rank": 0.0}
+{"step": 39, "steps": 250, "lr": 4.902566946861708e-05, "grad_norm": 1.6749180555343628, "loss_scale": 4.0, "retries": 0, "seconds": 148.6, "max_memory_gb": 17.17, "ce": 0.14749493186900509, "rank": 0.12515996396541595}
+{"step": 40, "steps": 250, "lr": 4.89358015465922e-05, "grad_norm": 0.09490246325731277, "loss_scale": 4.0, "retries": 0, "seconds": 152.3, "max_memory_gb": 17.17, "ce": 0.005509052652087121, "rank": 0.0}
+{"step": 41, "steps": 250, "lr": 4.884206994381078e-05, "grad_norm": 2.8007349967956543, "loss_scale": 4.0, "retries": 0, "seconds": 156.0, "max_memory_gb": 17.17, "ce": 0.4634824843442402, "rank": 0.0}
+{"step": 42, "steps": 250, "lr": 4.874449155485866e-05, "grad_norm": 0.07535050809383392, "loss_scale": 4.0, "retries": 0, "seconds": 159.9, "max_memory_gb": 17.17, "ce": 0.004697512999428, "rank": 0.0}
+{"step": 43, "steps": 250, "lr": 4.864308396768294e-05, "grad_norm": 0.1045573353767395, "loss_scale": 4.0, "retries": 0, "seconds": 163.7, "max_memory_gb": 17.17, "ce": 0.006950762759515783, "rank": 0.0}
+{"step": 44, "steps": 250, "lr": 4.853786546042184e-05, "grad_norm": 6.875782489776611, "loss_scale": 4.0, "retries": 0, "seconds": 167.4, "max_memory_gb": 17.17, "ce": 0.2588995107327037, "rank": 0.0}
+{"step": 45, "steps": 250, "lr": 4.842885499811012e-05, "grad_norm": 4.183457851409912, "loss_scale": 4.0, "retries": 0, "seconds": 171.2, "max_memory_gb": 17.17, "ce": 0.43380303498270223, "rank": 0.0}
+{"step": 46, "steps": 250, "lr": 4.831607222926078e-05, "grad_norm": 0.8127029538154602, "loss_scale": 4.0, "retries": 0, "seconds": 175.1, "max_memory_gb": 17.17, "ce": 0.08935657917663775, "rank": 0.0}
+{"step": 47, "steps": 250, "lr": 4.8199537482323545e-05, "grad_norm": 0.631737470626831, "loss_scale": 4.0, "retries": 0, "seconds": 178.8, "max_memory_gb": 17.17, "ce": 0.04014495799128781, "rank": 0.0}
+{"step": 48, "steps": 250, "lr": 4.807927176202073e-05, "grad_norm": 0.023993581533432007, "loss_scale": 4.0, "retries": 0, "seconds": 182.6, "max_memory_gb": 17.17, "ce": 0.0012207948875584407, "rank": 0.0}
+{"step": 49, "steps": 250, "lr": 4.795529674556129e-05, "grad_norm": 0.13346020877361298, "loss_scale": 4.0, "retries": 0, "seconds": 186.5, "max_memory_gb": 17.17, "ce": 0.011402878644730663, "rank": 0.0}
+{"step": 50, "steps": 250, "lr": 4.782763477873361e-05, "grad_norm": 1.4926725625991821, "loss_scale": 4.0, "retries": 0, "seconds": 190.3, "max_memory_gb": 17.17, "ce": 0.11948713013543966, "rank": 0.0}
+{"step": 51, "steps": 250, "lr": 4.769630887187782e-05, "grad_norm": 49.19395065307617, "loss_scale": 4.0, "retries": 0, "seconds": 194.0, "max_memory_gb": 17.17, "ce": 0.16739702912354915, "rank": 0.0}
+{"step": 52, "steps": 250, "lr": 4.756134269573832e-05, "grad_norm": 0.2868931293487549, "loss_scale": 4.0, "retries": 0, "seconds": 197.9, "max_memory_gb": 17.17, "ce": 0.0253499682180518, "rank": 0.0}
+{"step": 53, "steps": 250, "lr": 4.7422760577197226e-05, "grad_norm": 1.558651089668274, "loss_scale": 4.0, "retries": 0, "seconds": 201.7, "max_memory_gb": 17.17, "ce": 0.10159115286614906, "rank": 0.0}
+{"step": 54, "steps": 250, "lr": 4.728058749488963e-05, "grad_norm": 1.4151690006256104, "loss_scale": 4.0, "retries": 0, "seconds": 205.5, "max_memory_gb": 17.17, "ce": 0.0633008400768631, "rank": 0.0}
+{"step": 55, "steps": 250, "lr": 4.713484907470133e-05, "grad_norm": 0.20245365798473358, "loss_scale": 4.0, "retries": 0, "seconds": 209.3, "max_memory_gb": 17.17, "ce": 0.014729244443060452, "rank": 0.0}
+{"step": 56, "steps": 250, "lr": 4.6985571585149876e-05, "grad_norm": 0.29406148195266724, "loss_scale": 4.0, "retries": 0, "seconds": 213.1, "max_memory_gb": 17.17, "ce": 0.02533724489649103, "rank": 0.0}
+{"step": 57, "steps": 250, "lr": 4.6832781932649884e-05, "grad_norm": 0.1673152893781662, "loss_scale": 4.0, "retries": 0, "seconds": 216.8, "max_memory_gb": 17.17, "ce": 0.011629879326847004, "rank": 0.0}
+{"step": 58, "steps": 250, "lr": 4.667650765666325e-05, "grad_norm": 0.016266755759716034, "loss_scale": 4.0, "retries": 0, "seconds": 220.6, "max_memory_gb": 17.18, "ce": 0.0005470185199669686, "rank": 0.0}
+{"step": 59, "steps": 250, "lr": 4.651677692473538e-05, "grad_norm": 1.321953296661377, "loss_scale": 4.0, "retries": 0, "seconds": 224.5, "max_memory_gb": 17.18, "ce": 0.13234839379697405, "rank": 0.0}
+{"step": 60, "steps": 250, "lr": 4.635361852741808e-05, "grad_norm": 0.9031835794448853, "loss_scale": 4.0, "retries": 0, "seconds": 228.2, "max_memory_gb": 17.18, "ce": 0.07214992105764395, "rank": 0.0}
+{"step": 61, "steps": 250, "lr": 4.6187061873080264e-05, "grad_norm": 1.4647942781448364, "loss_scale": 4.0, "retries": 0, "seconds": 232.0, "max_memory_gb": 17.18, "ce": 0.2217925501499849, "rank": 0.0}
+{"step": 62, "steps": 250, "lr": 4.601713698260727e-05, "grad_norm": 0.3795619308948517, "loss_scale": 4.0, "retries": 0, "seconds": 235.8, "max_memory_gb": 17.18, "ce": 0.030378405994724744, "rank": 0.0}
+{"step": 63, "steps": 250, "lr": 4.5843874483989746e-05, "grad_norm": 0.35963568091392517, "loss_scale": 4.0, "retries": 0, "seconds": 239.6, "max_memory_gb": 17.18, "ce": 0.024272675134398014, "rank": 0.0}
+{"step": 64, "steps": 250, "lr": 4.566730560680316e-05, "grad_norm": 0.15242238342761993, "loss_scale": 4.0, "retries": 0, "seconds": 243.4, "max_memory_gb": 17.18, "ce": 0.006973288387371213, "rank": 0.0}
+{"step": 65, "steps": 250, "lr": 4.548746217657878e-05, "grad_norm": 0.9385842084884644, "loss_scale": 4.0, "retries": 0, "seconds": 247.2, "max_memory_gb": 17.18, "ce": 0.12172466283072936, "rank": 0.0}
+{"step": 66, "steps": 250, "lr": 4.5304376609067396e-05, "grad_norm": 2.196089267730713, "loss_scale": 4.0, "retries": 0, "seconds": 251.1, "max_memory_gb": 17.18, "ce": 0.2623702435291193, "rank": 0.12096622586250305}
+{"step": 67, "steps": 250, "lr": 4.5118081904396504e-05, "grad_norm": 0.026227276772260666, "loss_scale": 4.0, "retries": 0, "seconds": 254.9, "max_memory_gb": 17.18, "ce": 0.001877193088603235, "rank": 0.0}
+{"step": 68, "steps": 250, "lr": 4.4928611641122225e-05, "grad_norm": 0.2951924502849579, "loss_scale": 4.0, "retries": 0, "seconds": 258.7, "max_memory_gb": 17.18, "ce": 0.008107844448829837, "rank": 0.0}
+{"step": 69, "steps": 250, "lr": 4.473599997017701e-05, "grad_norm": 3.8525028228759766, "loss_scale": 4.0, "retries": 0, "seconds": 262.6, "max_memory_gb": 17.18, "ce": 0.6876039672902152, "rank": 0.0}
+{"step": 70, "steps": 250, "lr": 4.4540281608714053e-05, "grad_norm": 0.10002949833869934, "loss_scale": 4.0, "retries": 0, "seconds": 266.4, "max_memory_gb": 17.18, "ce": 0.0019861610933276097, "rank": 0.0}
+{"step": 71, "steps": 250, "lr": 4.434149183384977e-05, "grad_norm": 0.04415198788046837, "loss_scale": 4.0, "retries": 0, "seconds": 270.2, "max_memory_gb": 17.18, "ce": 0.002387052796620992, "rank": 0.0}
+{"step": 72, "steps": 250, "lr": 4.413966647630534e-05, "grad_norm": 0.6534761190414429, "loss_scale": 4.0, "retries": 0, "seconds": 274.1, "max_memory_gb": 17.18, "ce": 0.06882057460938995, "rank": 0.0}
+{"step": 73, "steps": 250, "lr": 4.3934841913948323e-05, "grad_norm": 1.8193423748016357, "loss_scale": 4.0, "retries": 0, "seconds": 277.8, "max_memory_gb": 17.18, "ce": 0.14682153871649462, "rank": 0.0}
+{"step": 74, "steps": 250, "lr": 4.372705506523583e-05, "grad_norm": 1.30190908908844, "loss_scale": 4.0, "retries": 0, "seconds": 281.6, "max_memory_gb": 17.18, "ce": 0.13523011075996294, "rank": 0.0}
+{"step": 75, "steps": 250, "lr": 4.351634338256017e-05, "grad_norm": 2.7486977577209473, "loss_scale": 4.0, "retries": 0, "seconds": 285.3, "max_memory_gb": 17.18, "ce": 0.7458954380574596, "rank": 0.0}
+{"step": 76, "steps": 250, "lr": 4.330274484549825e-05, "grad_norm": 1.4918301105499268, "loss_scale": 4.0, "retries": 0, "seconds": 289.2, "max_memory_gb": 17.18, "ce": 0.3422910034387314, "rank": 0.0}
+{"step": 77, "steps": 250, "lr": 4.308629795396599e-05, "grad_norm": 1.4241575002670288, "loss_scale": 4.0, "retries": 0, "seconds": 293.0, "max_memory_gb": 17.18, "ce": 0.06251760961276887, "rank": 0.0}
+{"step": 78, "steps": 250, "lr": 4.286704172127894e-05, "grad_norm": 0.2531434893608093, "loss_scale": 4.0, "retries": 0, "seconds": 296.7, "max_memory_gb": 17.18, "ce": 0.024884825168555835, "rank": 0.0}
+{"step": 79, "steps": 250, "lr": 4.264501566712033e-05, "grad_norm": 0.1475549340248108, "loss_scale": 4.0, "retries": 0, "seconds": 300.6, "max_memory_gb": 17.18, "ce": 0.005289791132440769, "rank": 0.0}
+{"step": 80, "steps": 250, "lr": 4.2420259810417894e-05, "grad_norm": 0.23124472796916962, "loss_scale": 4.0, "retries": 0, "seconds": 304.3, "max_memory_gb": 17.18, "ce": 0.013894542194066162, "rank": 0.0}
+{"step": 81, "steps": 250, "lr": 4.219281466213066e-05, "grad_norm": 0.43361300230026245, "loss_scale": 4.0, "retries": 0, "seconds": 308.1, "max_memory_gb": 17.18, "ce": 0.03965195363116436, "rank": 0.0}
+{"step": 82, "steps": 250, "lr": 4.1962721217947135e-05, "grad_norm": 1.7404146194458008, "loss_scale": 4.0, "retries": 0, "seconds": 311.9, "max_memory_gb": 17.18, "ce": 0.1325498499618334, "rank": 0.0}
+{"step": 83, "steps": 250, "lr": 4.173002095089599e-05, "grad_norm": 1.2253247499465942, "loss_scale": 4.0, "retries": 0, "seconds": 315.8, "max_memory_gb": 17.18, "ce": 0.19320178250347908, "rank": 0.0}
+{"step": 84, "steps": 250, "lr": 4.149475580387084e-05, "grad_norm": 0.7034221887588501, "loss_scale": 4.0, "retries": 0, "seconds": 319.6, "max_memory_gb": 17.18, "ce": 0.03955816669054002, "rank": 0.0}
+{"step": 85, "steps": 250, "lr": 4.1256968182070275e-05, "grad_norm": 0.03841950744390488, "loss_scale": 4.0, "retries": 0, "seconds": 323.3, "max_memory_gb": 17.18, "ce": 0.0010452790929775801, "rank": 0.0}
+{"step": 86, "steps": 250, "lr": 4.101670094535453e-05, "grad_norm": 0.5871836543083191, "loss_scale": 4.0, "retries": 0, "seconds": 327.2, "max_memory_gb": 17.18, "ce": 0.05967830207100633, "rank": 0.0}
+{"step": 87, "steps": 250, "lr": 4.077399740052027e-05, "grad_norm": 1.1325300931930542, "loss_scale": 4.0, "retries": 0, "seconds": 331.0, "max_memory_gb": 17.18, "ce": 0.15616081462758302, "rank": 0.0}
+{"step": 88, "steps": 250, "lr": 4.0528901293494775e-05, "grad_norm": 0.5086588263511658, "loss_scale": 4.0, "retries": 0, "seconds": 334.8, "max_memory_gb": 17.18, "ce": 0.04443240111368141, "rank": 0.0}
+{"step": 89, "steps": 250, "lr": 4.0281456801451004e-05, "grad_norm": 0.1601952314376831, "loss_scale": 4.0, "retries": 0, "seconds": 338.6, "max_memory_gb": 17.18, "ce": 0.01770243116857273, "rank": 0.12499688565731049}
+{"step": 90, "steps": 250, "lr": 4.0031708524844884e-05, "grad_norm": 0.8793088793754578, "loss_scale": 4.0, "retries": 0, "seconds": 342.3, "max_memory_gb": 17.18, "ce": 0.09917591052726493, "rank": 0.0}
+{"step": 91, "steps": 250, "lr": 3.977970147937635e-05, "grad_norm": 2.700533628463745, "loss_scale": 4.0, "retries": 0, "seconds": 346.1, "max_memory_gb": 17.18, "ce": 0.4511317676508497, "rank": 0.0}
+{"step": 92, "steps": 250, "lr": 3.952548108787553e-05, "grad_norm": 2.8656203746795654, "loss_scale": 4.0, "retries": 0, "seconds": 349.9, "max_memory_gb": 17.18, "ce": 0.20974411966426487, "rank": 0.0}
+{"step": 93, "steps": 250, "lr": 3.9269093172115496e-05, "grad_norm": 2.4210500717163086, "loss_scale": 4.0, "retries": 0, "seconds": 353.6, "max_memory_gb": 17.18, "ce": 0.10000467408644909, "rank": 0.0}
+{"step": 94, "steps": 250, "lr": 3.901058394455323e-05, "grad_norm": 1.0028198957443237, "loss_scale": 4.0, "retries": 0, "seconds": 357.4, "max_memory_gb": 17.18, "ce": 0.04404124662096365, "rank": 0.0}
+{"step": 95, "steps": 250, "lr": 3.875e-05, "grad_norm": 3.857961654663086, "loss_scale": 4.0, "retries": 0, "seconds": 361.2, "max_memory_gb": 17.18, "ce": 0.4799502629298331, "rank": 0.0}
+{"step": 96, "steps": 250, "lr": 3.8487388307223005e-05, "grad_norm": 3.454028606414795, "loss_scale": 4.0, "retries": 0, "seconds": 365.1, "max_memory_gb": 17.18, "ce": 0.35605922162267234, "rank": 0.29493123292922974}
+{"step": 97, "steps": 250, "lr": 3.822279620047943e-05, "grad_norm": 0.2488049417734146, "loss_scale": 4.0, "retries": 0, "seconds": 368.8, "max_memory_gb": 17.18, "ce": 0.015771134300393896, "rank": 0.0}
+{"step": 98, "steps": 250, "lr": 3.795627137098479e-05, "grad_norm": 0.8383006453514099, "loss_scale": 4.0, "retries": 0, "seconds": 372.6, "max_memory_gb": 17.18, "ce": 0.04371650328789656, "rank": 0.0}
+{"step": 99, "steps": 250, "lr": 3.7687861858316844e-05, "grad_norm": 0.09764763712882996, "loss_scale": 4.0, "retries": 0, "seconds": 376.4, "max_memory_gb": 17.18, "ce": 0.01059283137260536, "rank": 0.0}
+{"step": 100, "steps": 250, "lr": 3.741761604175668e-05, "grad_norm": 0.5293636918067932, "loss_scale": 4.0, "retries": 0, "seconds": 380.2, "max_memory_gb": 17.18, "ce": 0.07953367675736445, "rank": 0.0}
+{"step": 101, "steps": 250, "lr": 3.714558263156872e-05, "grad_norm": 0.12436012178659439, "loss_scale": 4.0, "retries": 0, "seconds": 383.9, "max_memory_gb": 17.18, "ce": 0.009720967154862592, "rank": 0.0}
+{"step": 102, "steps": 250, "lr": 3.6871810660220904e-05, "grad_norm": 0.23825384676456451, "loss_scale": 4.0, "retries": 0, "seconds": 387.7, "max_memory_gb": 17.18, "ce": 0.015293785564608697, "rank": 0.0}
+{"step": 103, "steps": 250, "lr": 3.659634947354686e-05, "grad_norm": 0.08514232933521271, "loss_scale": 4.0, "retries": 0, "seconds": 391.5, "max_memory_gb": 17.18, "ce": 0.006690844170861965, "rank": 0.0}
+{"step": 104, "steps": 250, "lr": 3.631924872185169e-05, "grad_norm": 0.1267702281475067, "loss_scale": 4.0, "retries": 0, "seconds": 395.2, "max_memory_gb": 17.18, "ce": 0.010219092163424648, "rank": 0.12492857873439789}
+{"step": 105, "steps": 250, "lr": 3.6040558350962674e-05, "grad_norm": 0.6855973601341248, "loss_scale": 4.0, "retries": 0, "seconds": 399.0, "max_memory_gb": 17.18, "ce": 0.06015902753847513, "rank": 0.0}
+{"step": 106, "steps": 250, "lr": 3.576032859322687e-05, "grad_norm": 0.07049433141946793, "loss_scale": 4.0, "retries": 0, "seconds": 402.9, "max_memory_gb": 17.18, "ce": 0.0030043735687286244, "rank": 0.10971830785274506}
+{"step": 107, "steps": 250, "lr": 3.5478609958457056e-05, "grad_norm": 1.1604353189468384, "loss_scale": 4.0, "retries": 0, "seconds": 406.6, "max_memory_gb": 17.18, "ce": 0.10745059135979318, "rank": 0.0}
+{"step": 108, "steps": 250, "lr": 3.519545322482755e-05, "grad_norm": 2.366562604904175, "loss_scale": 4.0, "retries": 0, "seconds": 410.4, "max_memory_gb": 17.18, "ce": 0.25732528687171907, "rank": 0.0}
+{"step": 109, "steps": 250, "lr": 3.491090942972179e-05, "grad_norm": 1.5032366514205933, "loss_scale": 4.0, "retries": 0, "seconds": 414.2, "max_memory_gb": 17.18, "ce": 0.16047093803535972, "rank": 0.0}
+{"step": 110, "steps": 250, "lr": 3.4625029860533135e-05, "grad_norm": 0.6452511548995972, "loss_scale": 4.0, "retries": 0, "seconds": 418.0, "max_memory_gb": 17.18, "ce": 0.033901772504009386, "rank": 0.0}
+{"step": 111, "steps": 250, "lr": 3.433786604542056e-05, "grad_norm": 0.34032541513442993, "loss_scale": 4.0, "retries": 0, "seconds": 421.7, "max_memory_gb": 17.18, "ce": 0.02216405546050737, "rank": 0.0}
+{"step": 112, "steps": 250, "lr": 3.404946974402104e-05, "grad_norm": 0.10128169506788254, "loss_scale": 4.0, "retries": 0, "seconds": 425.5, "max_memory_gb": 17.18, "ce": 0.0058542714655231975, "rank": 0.0}
+{"step": 113, "steps": 250, "lr": 3.375989293812019e-05, "grad_norm": 0.5122324228286743, "loss_scale": 4.0, "retries": 0, "seconds": 429.4, "max_memory_gb": 17.18, "ce": 0.025056643358993824, "rank": 0.0}
+{"step": 114, "steps": 250, "lr": 3.3469187822282766e-05, "grad_norm": 0.8725573420524597, "loss_scale": 4.0, "retries": 0, "seconds": 433.2, "max_memory_gb": 17.18, "ce": 0.11165329797427148, "rank": 0.0}
+{"step": 115, "steps": 250, "lr": 3.317740679444499e-05, "grad_norm": 1.166893720626831, "loss_scale": 4.0, "retries": 0, "seconds": 436.9, "max_memory_gb": 17.18, "ce": 0.03592306389850819, "rank": 0.0}
+{"step": 116, "steps": 250, "lr": 3.2884602446470056e-05, "grad_norm": 0.7226559519767761, "loss_scale": 4.0, "retries": 0, "seconds": 440.7, "max_memory_gb": 17.18, "ce": 0.16494127850410223, "rank": 0.0}
+{"step": 117, "steps": 250, "lr": 3.259082755466876e-05, "grad_norm": 0.1338038593530655, "loss_scale": 4.0, "retries": 0, "seconds": 444.4, "max_memory_gb": 17.18, "ce": 0.00509007691334773, "rank": 0.0}
+{"step": 118, "steps": 250, "lr": 3.229613507028686e-05, "grad_norm": 1.371659278869629, "loss_scale": 4.0, "retries": 0, "seconds": 448.2, "max_memory_gb": 17.18, "ce": 0.21522522439738623, "rank": 0.0}
+{"step": 119, "steps": 250, "lr": 3.2000578109961004e-05, "grad_norm": 0.05768913775682449, "loss_scale": 4.0, "retries": 0, "seconds": 451.9, "max_memory_gb": 17.18, "ce": 0.003073839236776621, "rank": 0.0}
+{"step": 120, "steps": 250, "lr": 3.1704209946144616e-05, "grad_norm": 2.2049195766448975, "loss_scale": 4.0, "retries": 0, "seconds": 455.8, "max_memory_gb": 17.18, "ce": 0.10000767663939314, "rank": 0.0}
+{"step": 121, "steps": 250, "lr": 3.140708399750594e-05, "grad_norm": 0.49410173296928406, "loss_scale": 4.0, "retries": 0, "seconds": 459.6, "max_memory_gb": 17.18, "ce": 0.049888341086898436, "rank": 0.0}
+{"step": 122, "steps": 250, "lr": 3.1109253819299606e-05, "grad_norm": 0.9339712858200073, "loss_scale": 4.0, "retries": 0, "seconds": 463.3, "max_memory_gb": 17.18, "ce": 0.10111064322472885, "rank": 0.0}
+{"step": 123, "steps": 250, "lr": 3.081077309371361e-05, "grad_norm": 0.4240740239620209, "loss_scale": 4.0, "retries": 0, "seconds": 467.2, "max_memory_gb": 17.18, "ce": 0.03317867703663069, "rank": 0.0}
+{"step": 124, "steps": 250, "lr": 3.051169562019339e-05, "grad_norm": 1.8490060567855835, "loss_scale": 4.0, "retries": 0, "seconds": 470.9, "max_memory_gb": 17.18, "ce": 0.5379407063699091, "rank": 0.0}
+{"step": 125, "steps": 250, "lr": 3.021207530574477e-05, "grad_norm": 0.7946641445159912, "loss_scale": 4.0, "retries": 0, "seconds": 474.6, "max_memory_gb": 17.18, "ce": 0.13940476028562898, "rank": 0.0}
+{"step": 126, "steps": 250, "lr": 2.991196615521758e-05, "grad_norm": 3.426243305206299, "loss_scale": 4.0, "retries": 0, "seconds": 478.5, "max_memory_gb": 17.18, "ce": 0.5532796201832753, "rank": 0.0}
+{"step": 127, "steps": 250, "lr": 2.9611422261571536e-05, "grad_norm": 0.3899013102054596, "loss_scale": 4.0, "retries": 0, "seconds": 482.2, "max_memory_gb": 17.18, "ce": 0.013481769086524764, "rank": 0.0}
+{"step": 128, "steps": 250, "lr": 2.931049779612634e-05, "grad_norm": 1.3287353515625, "loss_scale": 4.0, "retries": 0, "seconds": 485.9, "max_memory_gb": 17.18, "ce": 0.1339584078638154, "rank": 0.0}
+{"step": 129, "steps": 250, "lr": 2.9009246998797557e-05, "grad_norm": 1.188004493713379, "loss_scale": 4.0, "retries": 0, "seconds": 489.8, "max_memory_gb": 17.18, "ce": 0.08508403886446558, "rank": 0.0}
+{"step": 130, "steps": 250, "lr": 2.8707724168320238e-05, "grad_norm": 0.8651183247566223, "loss_scale": 4.0, "retries": 0, "seconds": 493.6, "max_memory_gb": 17.18, "ce": 0.16112000689986417, "rank": 0.0}
+{"step": 131, "steps": 250, "lr": 2.840598365246184e-05, "grad_norm": 3.453805446624756, "loss_scale": 4.0, "retries": 0, "seconds": 497.3, "max_memory_gb": 17.18, "ce": 0.21577859223555151, "rank": 0.0}
+{"step": 132, "steps": 250, "lr": 2.8104079838226383e-05, "grad_norm": 2.105423927307129, "loss_scale": 4.0, "retries": 0, "seconds": 501.1, "max_memory_gb": 17.18, "ce": 0.2645968209718603, "rank": 0.019232727587223053}
+{"step": 133, "steps": 250, "lr": 2.7802067142051446e-05, "grad_norm": 1.2501418590545654, "loss_scale": 4.0, "retries": 0, "seconds": 504.9, "max_memory_gb": 17.18, "ce": 0.3120499819842735, "rank": 0.0}
+{"step": 134, "steps": 250, "lr": 2.7500000000000004e-05, "grad_norm": 1.1696155071258545, "loss_scale": 4.0, "retries": 0, "seconds": 508.7, "max_memory_gb": 17.18, "ce": 0.20286206462492373, "rank": 0.0}
+{"step": 135, "steps": 250, "lr": 2.7197932857948566e-05, "grad_norm": 0.19736653566360474, "loss_scale": 4.0, "retries": 0, "seconds": 512.4, "max_memory_gb": 17.18, "ce": 0.013036060649483261, "rank": 0.0}
+{"step": 136, "steps": 250, "lr": 2.689592016177363e-05, "grad_norm": 1.3194749355316162, "loss_scale": 4.0, "retries": 0, "seconds": 516.2, "max_memory_gb": 17.18, "ce": 0.05927401643339181, "rank": 0.0}
+{"step": 137, "steps": 250, "lr": 2.659401634753817e-05, "grad_norm": 0.29545649886131287, "loss_scale": 4.0, "retries": 0, "seconds": 520.0, "max_memory_gb": 17.18, "ce": 0.014311607938452653, "rank": 0.0}
+{"step": 138, "steps": 250, "lr": 2.6292275831679764e-05, "grad_norm": 1.147539496421814, "loss_scale": 4.0, "retries": 0, "seconds": 523.7, "max_memory_gb": 17.18, "ce": 0.0821282182014329, "rank": 0.0}
+{"step": 139, "steps": 250, "lr": 2.5990753001202445e-05, "grad_norm": 0.5142342448234558, "loss_scale": 4.0, "retries": 0, "seconds": 527.5, "max_memory_gb": 17.18, "ce": 0.025808557589471093, "rank": 0.0}
+{"step": 140, "steps": 250, "lr": 2.5689502203873672e-05, "grad_norm": 0.02779260277748108, "loss_scale": 4.0, "retries": 0, "seconds": 531.3, "max_memory_gb": 17.18, "ce": 0.0021579414790267037, "rank": 0.0}
+{"step": 141, "steps": 250, "lr": 2.538857773842847e-05, "grad_norm": 0.05062991753220558, "loss_scale": 4.0, "retries": 0, "seconds": 535.1, "max_memory_gb": 17.18, "ce": 0.0038267910515514814, "rank": 0.0}
+{"step": 142, "steps": 250, "lr": 2.5088033844782426e-05, "grad_norm": 2.0193212032318115, "loss_scale": 4.0, "retries": 0, "seconds": 538.8, "max_memory_gb": 17.18, "ce": 0.09754045784552545, "rank": 0.0}
+{"step": 143, "steps": 250, "lr": 2.4787924694255237e-05, "grad_norm": 3.1141879558563232, "loss_scale": 4.0, "retries": 0, "seconds": 542.6, "max_memory_gb": 17.18, "ce": 0.2166620752673225, "rank": 0.0}
+{"step": 144, "steps": 250, "lr": 2.4488304379806627e-05, "grad_norm": 0.5145749449729919, "loss_scale": 4.0, "retries": 0, "seconds": 546.4, "max_memory_gb": 17.18, "ce": 0.030936785765675268, "rank": 0.0}
+{"step": 145, "steps": 250, "lr": 2.4189226906286388e-05, "grad_norm": 0.02006860077381134, "loss_scale": 4.0, "retries": 0, "seconds": 550.1, "max_memory_gb": 17.18, "ce": 0.0013876453381271858, "rank": 0.0}
+{"step": 146, "steps": 250, "lr": 2.3890746180700395e-05, "grad_norm": 0.1105627492070198, "loss_scale": 4.0, "retries": 0, "seconds": 554.0, "max_memory_gb": 17.18, "ce": 0.01097357864443893, "rank": 0.0}
+{"step": 147, "steps": 250, "lr": 2.359291600249407e-05, "grad_norm": 1.6976550817489624, "loss_scale": 4.0, "retries": 0, "seconds": 557.8, "max_memory_gb": 17.18, "ce": 0.4698275368805298, "rank": 0.0}
+{"step": 148, "steps": 250, "lr": 2.3295790053855392e-05, "grad_norm": 0.11827452480792999, "loss_scale": 4.0, "retries": 0, "seconds": 561.5, "max_memory_gb": 17.18, "ce": 0.006990148356408099, "rank": 0.0}
+{"step": 149, "steps": 250, "lr": 2.299942189003901e-05, "grad_norm": 0.14813300967216492, "loss_scale": 4.0, "retries": 0, "seconds": 565.2, "max_memory_gb": 17.18, "ce": 0.00840225933200145, "rank": 0.12226150929927826}
+{"step": 150, "steps": 250, "lr": 2.2703864929713144e-05, "grad_norm": 0.5493444800376892, "loss_scale": 4.0, "retries": 0, "seconds": 569.0, "max_memory_gb": 17.18, "ce": 0.07819241663457888, "rank": 0.0}
+{"step": 151, "steps": 250, "lr": 2.2409172445331257e-05, "grad_norm": 0.7409102916717529, "loss_scale": 4.0, "retries": 0, "seconds": 572.8, "max_memory_gb": 17.18, "ce": 0.01812813842752803, "rank": 0.0}
+{"step": 152, "steps": 250, "lr": 2.211539755352996e-05, "grad_norm": 0.3035765588283539, "loss_scale": 4.0, "retries": 0, "seconds": 576.5, "max_memory_gb": 17.18, "ce": 0.024689826484745936, "rank": 0.0}
+{"step": 153, "steps": 250, "lr": 2.1822593205555016e-05, "grad_norm": 0.6221047043800354, "loss_scale": 4.0, "retries": 0, "seconds": 580.3, "max_memory_gb": 17.18, "ce": 0.1001168589776853, "rank": 0.0}
+{"step": 154, "steps": 250, "lr": 2.1530812177717236e-05, "grad_norm": 0.3531210422515869, "loss_scale": 4.0, "retries": 0, "seconds": 584.1, "max_memory_gb": 17.18, "ce": 0.029597805690173118, "rank": 0.0}
+{"step": 155, "steps": 250, "lr": 2.1240107061879815e-05, "grad_norm": 1.9973658323287964, "loss_scale": 4.0, "retries": 0, "seconds": 587.8, "max_memory_gb": 17.18, "ce": 0.26292775530987456, "rank": 0.0}
+{"step": 156, "steps": 250, "lr": 2.095053025597896e-05, "grad_norm": 0.07254761457443237, "loss_scale": 4.0, "retries": 0, "seconds": 591.6, "max_memory_gb": 17.18, "ce": 0.0027026198890780506, "rank": 0.0}
+{"step": 157, "steps": 250, "lr": 2.066213395457945e-05, "grad_norm": 0.16427314281463623, "loss_scale": 4.0, "retries": 0, "seconds": 595.4, "max_memory_gb": 17.18, "ce": 0.012523725117944196, "rank": 0.0}
+{"step": 158, "steps": 250, "lr": 2.0374970139466877e-05, "grad_norm": 3.7625041007995605, "loss_scale": 4.0, "retries": 0, "seconds": 599.1, "max_memory_gb": 17.18, "ce": 0.5574398205634452, "rank": 0.0}
+{"step": 159, "steps": 250, "lr": 2.0089090570278213e-05, "grad_norm": 0.22528238594532013, "loss_scale": 4.0, "retries": 0, "seconds": 602.9, "max_memory_gb": 17.18, "ce": 0.013080861406024269, "rank": 0.0}
+{"step": 160, "steps": 250, "lr": 1.9804546775172457e-05, "grad_norm": 1.4808956384658813, "loss_scale": 4.0, "retries": 0, "seconds": 606.7, "max_memory_gb": 17.18, "ce": 0.15226369775041348, "rank": 0.0}
+{"step": 161, "steps": 250, "lr": 1.9521390041542946e-05, "grad_norm": 1.1318385601043701, "loss_scale": 4.0, "retries": 0, "seconds": 610.4, "max_memory_gb": 17.18, "ce": 0.12627086005477395, "rank": 0.0}
+{"step": 162, "steps": 250, "lr": 1.923967140677313e-05, "grad_norm": 0.8011034727096558, "loss_scale": 4.0, "retries": 0, "seconds": 614.1, "max_memory_gb": 17.18, "ce": 0.06398793136213499, "rank": 0.0}
+{"step": 163, "steps": 250, "lr": 1.8959441649037335e-05, "grad_norm": 3.778258800506592, "loss_scale": 4.0, "retries": 0, "seconds": 618.0, "max_memory_gb": 17.18, "ce": 0.32121356373045273, "rank": 0.0}
+{"step": 164, "steps": 250, "lr": 1.868075127814832e-05, "grad_norm": 4.519586086273193, "loss_scale": 4.0, "retries": 0, "seconds": 621.7, "max_memory_gb": 17.18, "ce": 0.11012040618197716, "rank": 0.0}
+{"step": 165, "steps": 250, "lr": 1.8403650526453143e-05, "grad_norm": 0.12183576077222824, "loss_scale": 4.0, "retries": 0, "seconds": 625.4, "max_memory_gb": 17.18, "ce": 0.00971579732504324, "rank": 0.0}
+{"step": 166, "steps": 250, "lr": 1.812818933977912e-05, "grad_norm": 2.4199893474578857, "loss_scale": 4.0, "retries": 0, "seconds": 629.3, "max_memory_gb": 17.18, "ce": 0.276112469778127, "rank": 0.0}
+{"step": 167, "steps": 250, "lr": 1.785441736843128e-05, "grad_norm": 0.2990489900112152, "loss_scale": 4.0, "retries": 0, "seconds": 633.1, "max_memory_gb": 17.18, "ce": 0.01218956051297937, "rank": 0.0}
+{"step": 168, "steps": 250, "lr": 1.7582383958243314e-05, "grad_norm": 0.06361427903175354, "loss_scale": 4.0, "retries": 0, "seconds": 636.8, "max_memory_gb": 17.18, "ce": 0.0037936738879125187, "rank": 0.0}
+{"step": 169, "steps": 250, "lr": 1.7312138141683164e-05, "grad_norm": 1.237835168838501, "loss_scale": 4.0, "retries": 0, "seconds": 640.6, "max_memory_gb": 17.18, "ce": 0.2996345187375482, "rank": 0.03183973580598831}
+{"step": 170, "steps": 250, "lr": 1.704372862901521e-05, "grad_norm": 0.015495923347771168, "loss_scale": 4.0, "retries": 0, "seconds": 644.5, "max_memory_gb": 17.18, "ce": 0.0005479106303027947, "rank": 0.0}
+{"step": 171, "steps": 250, "lr": 1.6777203799520573e-05, "grad_norm": 0.5931796431541443, "loss_scale": 4.0, "retries": 0, "seconds": 648.2, "max_memory_gb": 17.18, "ce": 0.05592095182828416, "rank": 0.0}
+{"step": 172, "steps": 250, "lr": 1.6512611692777008e-05, "grad_norm": 0.09469091147184372, "loss_scale": 4.0, "retries": 0, "seconds": 652.0, "max_memory_gb": 17.18, "ce": 0.0074926484353454725, "rank": 0.0}
+{"step": 173, "steps": 250, "lr": 1.6250000000000005e-05, "grad_norm": 0.16140979528427124, "loss_scale": 4.0, "retries": 0, "seconds": 655.8, "max_memory_gb": 17.18, "ce": 0.011120527380171552, "rank": 0.0}
+{"step": 174, "steps": 250, "lr": 1.598941605544678e-05, "grad_norm": 0.5142970681190491, "loss_scale": 4.0, "retries": 0, "seconds": 659.6, "max_memory_gb": 17.18, "ce": 0.03819917408145557, "rank": 0.0}
+{"step": 175, "steps": 250, "lr": 1.5730906827884506e-05, "grad_norm": 0.8822360634803772, "loss_scale": 4.0, "retries": 0, "seconds": 663.3, "max_memory_gb": 17.18, "ce": 0.10331248297507045, "rank": 0.0}
+{"step": 176, "steps": 250, "lr": 1.5474518912124476e-05, "grad_norm": 0.014285567216575146, "loss_scale": 4.0, "retries": 0, "seconds": 667.1, "max_memory_gb": 17.18, "ce": 0.0008039307756462222, "rank": 0.0}
+{"step": 177, "steps": 250, "lr": 1.5220298520623646e-05, "grad_norm": 0.9872564673423767, "loss_scale": 4.0, "retries": 0, "seconds": 670.9, "max_memory_gb": 17.18, "ce": 0.27100581930062617, "rank": 0.12498511373996735}
+{"step": 178, "steps": 250, "lr": 1.4968291475155121e-05, "grad_norm": 1.8895055055618286, "loss_scale": 4.0, "retries": 0, "seconds": 674.7, "max_memory_gb": 17.18, "ce": 0.11668298536551447, "rank": 0.0}
+{"step": 179, "steps": 250, "lr": 1.4718543198548998e-05, "grad_norm": 1.1172008514404297, "loss_scale": 4.0, "retries": 0, "seconds": 678.4, "max_memory_gb": 17.18, "ce": 0.3582586061663733, "rank": 0.0}
+{"step": 180, "steps": 250, "lr": 1.4471098706505227e-05, "grad_norm": 0.24095511436462402, "loss_scale": 4.0, "retries": 0, "seconds": 682.2, "max_memory_gb": 17.18, "ce": 0.021841303222345232, "rank": 0.0}
+{"step": 181, "steps": 250, "lr": 1.4226002599479743e-05, "grad_norm": 0.11787819862365723, "loss_scale": 4.0, "retries": 0, "seconds": 686.0, "max_memory_gb": 17.18, "ce": 0.008316482980944784, "rank": 0.0}
+{"step": 182, "steps": 250, "lr": 1.3983299054645471e-05, "grad_norm": 1.6221401691436768, "loss_scale": 4.0, "retries": 0, "seconds": 689.7, "max_memory_gb": 17.18, "ce": 0.10907009811580792, "rank": 0.0}
+{"step": 183, "steps": 250, "lr": 1.3743031817929724e-05, "grad_norm": 0.06946616619825363, "loss_scale": 4.0, "retries": 0, "seconds": 693.6, "max_memory_gb": 17.18, "ce": 0.002567781016068693, "rank": 0.0}
+{"step": 184, "steps": 250, "lr": 1.3505244196129164e-05, "grad_norm": 0.5170499086380005, "loss_scale": 4.0, "retries": 0, "seconds": 697.3, "max_memory_gb": 17.18, "ce": 0.026748124299274423, "rank": 0.14158732444047928}
+{"step": 185, "steps": 250, "lr": 1.3269979049104014e-05, "grad_norm": 0.3943300247192383, "loss_scale": 4.0, "retries": 0, "seconds": 701.1, "max_memory_gb": 17.18, "ce": 0.03525510916483654, "rank": 0.0}
+{"step": 186, "steps": 250, "lr": 1.3037278782052863e-05, "grad_norm": 2.7819254398345947, "loss_scale": 4.0, "retries": 0, "seconds": 704.8, "max_memory_gb": 17.18, "ce": 0.1142041046130089, "rank": 0.0}
+{"step": 187, "steps": 250, "lr": 1.2807185337869343e-05, "grad_norm": 0.5529460906982422, "loss_scale": 4.0, "retries": 0, "seconds": 708.7, "max_memory_gb": 17.18, "ce": 0.11818518331006089, "rank": 0.1293942630290985}
+{"step": 188, "steps": 250, "lr": 1.2579740189582115e-05, "grad_norm": 1.4214451313018799, "loss_scale": 4.0, "retries": 0, "seconds": 712.4, "max_memory_gb": 17.18, "ce": 0.1863521980446876, "rank": 0.0}
+{"step": 189, "steps": 250, "lr": 1.2354984332879668e-05, "grad_norm": 2.753243923187256, "loss_scale": 4.0, "retries": 0, "seconds": 716.0, "max_memory_gb": 17.18, "ce": 0.3045139886671677, "rank": 0.0}
+{"step": 190, "steps": 250, "lr": 1.2132958278721064e-05, "grad_norm": 0.10885030031204224, "loss_scale": 4.0, "retries": 0, "seconds": 719.9, "max_memory_gb": 17.18, "ce": 0.003310898170184373, "rank": 0.0}
+{"step": 191, "steps": 250, "lr": 1.1913702046034016e-05, "grad_norm": 2.7462196350097656, "loss_scale": 4.0, "retries": 0, "seconds": 723.5, "max_memory_gb": 17.18, "ce": 0.15921576233699852, "rank": 0.0}
+{"step": 192, "steps": 250, "lr": 1.1697255154501756e-05, "grad_norm": 1.1273689270019531, "loss_scale": 4.0, "retries": 0, "seconds": 727.2, "max_memory_gb": 17.18, "ce": 0.3303896564398201, "rank": 0.0}
+{"step": 193, "steps": 250, "lr": 1.1483656617439842e-05, "grad_norm": 0.013958720490336418, "loss_scale": 4.0, "retries": 0, "seconds": 731.0, "max_memory_gb": 17.18, "ce": 0.0009039877066925328, "rank": 0.12502388656139374}
+{"step": 194, "steps": 250, "lr": 1.1272944934764177e-05, "grad_norm": 0.6165243983268738, "loss_scale": 4.0, "retries": 0, "seconds": 734.8, "max_memory_gb": 17.18, "ce": 0.03993248156120899, "rank": 0.0}
+{"step": 195, "steps": 250, "lr": 1.1065158086051684e-05, "grad_norm": 0.38963475823402405, "loss_scale": 4.0, "retries": 0, "seconds": 738.5, "max_memory_gb": 17.18, "ce": 0.08138401686153429, "rank": 0.0}
+{"step": 196, "steps": 250, "lr": 1.086033352369467e-05, "grad_norm": 0.13668780028820038, "loss_scale": 4.0, "retries": 0, "seconds": 742.3, "max_memory_gb": 17.18, "ce": 0.00825187292116425, "rank": 0.0}
+{"step": 197, "steps": 250, "lr": 1.0658508166150224e-05, "grad_norm": 1.9193662405014038, "loss_scale": 4.0, "retries": 0, "seconds": 746.1, "max_memory_gb": 17.18, "ce": 0.1530696043378157, "rank": 0.0}
+{"step": 198, "steps": 250, "lr": 1.0459718391285954e-05, "grad_norm": 0.5088796019554138, "loss_scale": 4.0, "retries": 0, "seconds": 749.8, "max_memory_gb": 17.18, "ce": 0.06704476663230707, "rank": 0.0}
+{"step": 199, "steps": 250, "lr": 1.0264000029822999e-05, "grad_norm": 1.2266583442687988, "loss_scale": 4.0, "retries": 0, "seconds": 753.6, "max_memory_gb": 17.18, "ce": 0.09473990415040134, "rank": 0.0}
+{"step": 200, "steps": 250, "lr": 1.0071388358877775e-05, "grad_norm": 1.4750323295593262, "loss_scale": 4.0, "retries": 0, "seconds": 757.4, "max_memory_gb": 17.18, "ce": 0.2283015530558714, "rank": 0.0}
+{"step": 201, "steps": 250, "lr": 9.881918095603501e-06, "grad_norm": 0.9261612892150879, "loss_scale": 4.0, "retries": 0, "seconds": 761.1, "max_memory_gb": 17.18, "ce": 0.073620861032623, "rank": 0.0}
+{"step": 202, "steps": 250, "lr": 9.695623390932612e-06, "grad_norm": 0.036846887320280075, "loss_scale": 4.0, "retries": 0, "seconds": 764.8, "max_memory_gb": 17.18, "ce": 0.0027008811209725536, "rank": 0.0}
+{"step": 203, "steps": 250, "lr": 9.51253782342123e-06, "grad_norm": 0.6617287993431091, "loss_scale": 4.0, "retries": 0, "seconds": 768.6, "max_memory_gb": 17.18, "ce": 0.026786089555344006, "rank": 0.0}
+{"step": 204, "steps": 250, "lr": 9.332694393196846e-06, "grad_norm": 0.2853492796421051, "loss_scale": 4.0, "retries": 0, "seconds": 772.4, "max_memory_gb": 17.18, "ce": 0.024849348648331215, "rank": 0.0}
+{"step": 205, "steps": 250, "lr": 9.156125516010256e-06, "grad_norm": 1.1629290580749512, "loss_scale": 4.0, "retries": 0, "seconds": 776.1, "max_memory_gb": 17.18, "ce": 0.09697938146109664, "rank": 0.0}
+{"step": 206, "steps": 250, "lr": 8.982863017392733e-06, "grad_norm": 0.23901709914207458, "loss_scale": 4.0, "retries": 0, "seconds": 779.8, "max_memory_gb": 17.18, "ce": 0.01326044192956033, "rank": 0.0}
+{"step": 207, "steps": 250, "lr": 8.81293812691974e-06, "grad_norm": 0.2243930995464325, "loss_scale": 4.0, "retries": 0, "seconds": 783.6, "max_memory_gb": 17.18, "ce": 0.05995897610773682, "rank": 0.0}
+{"step": 208, "steps": 250, "lr": 8.646381472581927e-06, "grad_norm": 0.05863875150680542, "loss_scale": 4.0, "retries": 0, "seconds": 787.4, "max_memory_gb": 17.18, "ce": 0.0025013935605784354, "rank": 0.0}
+{"step": 209, "steps": 250, "lr": 8.483223075264622e-06, "grad_norm": 0.7006769776344299, "loss_scale": 4.0, "retries": 0, "seconds": 791.1, "max_memory_gb": 17.18, "ce": 0.10863958289110087, "rank": 0.1250288337469101}
+{"step": 210, "steps": 250, "lr": 8.323492343336747e-06, "grad_norm": 2.6804239749908447, "loss_scale": 4.0, "retries": 0, "seconds": 794.9, "max_memory_gb": 17.18, "ce": 0.3614551834232316, "rank": 0.0}
+{"step": 211, "steps": 250, "lr": 8.167218067350122e-06, "grad_norm": 0.7149457931518555, "loss_scale": 4.0, "retries": 0, "seconds": 798.7, "max_memory_gb": 17.18, "ce": 0.10593070072889077, "rank": 0.0}
+{"step": 212, "steps": 250, "lr": 8.01442841485013e-06, "grad_norm": 1.1558820009231567, "loss_scale": 4.0, "retries": 0, "seconds": 802.4, "max_memory_gb": 17.18, "ce": 0.19628709417838763, "rank": 0.0}
+{"step": 213, "steps": 250, "lr": 7.865150925298678e-06, "grad_norm": 0.4886331558227539, "loss_scale": 4.0, "retries": 0, "seconds": 806.1, "max_memory_gb": 17.18, "ce": 0.13765285239173863, "rank": 0.0}
+{"step": 214, "steps": 250, "lr": 7.719412505110375e-06, "grad_norm": 1.2286702394485474, "loss_scale": 4.0, "retries": 0, "seconds": 810.0, "max_memory_gb": 17.18, "ce": 0.21736948534976364, "rank": 0.0}
+{"step": 215, "steps": 250, "lr": 7.57723942280278e-06, "grad_norm": 2.637843370437622, "loss_scale": 4.0, "retries": 0, "seconds": 813.7, "max_memory_gb": 17.18, "ce": 0.43233907241392444, "rank": 0.0}
+{"step": 216, "steps": 250, "lr": 7.438657304261687e-06, "grad_norm": 0.5897781848907471, "loss_scale": 4.0, "retries": 0, "seconds": 817.4, "max_memory_gb": 17.18, "ce": 0.04052973020725403, "rank": 0.0}
+{"step": 217, "steps": 250, "lr": 7.30369112812219e-06, "grad_norm": 0.030596476048231125, "loss_scale": 4.0, "retries": 0, "seconds": 821.2, "max_memory_gb": 17.18, "ce": 0.0021132683975793043, "rank": 0.0}
+{"step": 218, "steps": 250, "lr": 7.172365221266403e-06, "grad_norm": 1.0040878057479858, "loss_scale": 4.0, "retries": 0, "seconds": 824.9, "max_memory_gb": 17.18, "ce": 0.16573279937858842, "rank": 0.0}
+{"step": 219, "steps": 250, "lr": 7.0447032544387154e-06, "grad_norm": 0.4031512439250946, "loss_scale": 4.0, "retries": 0, "seconds": 828.6, "max_memory_gb": 17.18, "ce": 0.03754197947546345, "rank": 0.0}
+{"step": 220, "steps": 250, "lr": 6.920728237979274e-06, "grad_norm": 0.39769235253334045, "loss_scale": 4.0, "retries": 0, "seconds": 832.4, "max_memory_gb": 17.18, "ce": 0.03101239705029002, "rank": 0.0}
+{"step": 221, "steps": 250, "lr": 6.800462517676456e-06, "grad_norm": 0.35474908351898193, "loss_scale": 4.0, "retries": 0, "seconds": 836.2, "max_memory_gb": 17.18, "ce": 0.02209920013001465, "rank": 0.0}
+{"step": 222, "steps": 250, "lr": 6.683927770739222e-06, "grad_norm": 0.3236624598503113, "loss_scale": 4.0, "retries": 0, "seconds": 840.0, "max_memory_gb": 17.18, "ce": 0.023484134753630315, "rank": 0.0}
+{"step": 223, "steps": 250, "lr": 6.571145001889892e-06, "grad_norm": 1.1914570331573486, "loss_scale": 4.0, "retries": 0, "seconds": 843.7, "max_memory_gb": 17.18, "ce": 0.04638790451463137, "rank": 0.0}
+{"step": 224, "steps": 250, "lr": 6.46213453957817e-06, "grad_norm": 0.43149134516716003, "loss_scale": 4.0, "retries": 0, "seconds": 847.5, "max_memory_gb": 17.18, "ce": 0.026247330743672137, "rank": 0.0}
+{"step": 225, "steps": 250, "lr": 6.356916032317063e-06, "grad_norm": 0.10974714159965515, "loss_scale": 4.0, "retries": 0, "seconds": 851.2, "max_memory_gb": 17.18, "ce": 0.00825589835426399, "rank": 0.0}
+{"step": 226, "steps": 250, "lr": 6.255508445141349e-06, "grad_norm": 0.4430094361305237, "loss_scale": 4.0, "retries": 0, "seconds": 854.9, "max_memory_gb": 17.18, "ce": 0.01649886752443308, "rank": 0.0}
+{"step": 227, "steps": 250, "lr": 6.157930056189228e-06, "grad_norm": 0.7099319100379944, "loss_scale": 4.0, "retries": 0, "seconds": 858.8, "max_memory_gb": 17.18, "ce": 0.026679094419478133, "rank": 0.0}
+{"step": 228, "steps": 250, "lr": 6.064198453407796e-06, "grad_norm": 0.13542690873146057, "loss_scale": 4.0, "retries": 0, "seconds": 862.4, "max_memory_gb": 17.18, "ce": 0.007635586573655928, "rank": 0.0}
+{"step": 229, "steps": 250, "lr": 5.974330531382935e-06, "grad_norm": 0.44289666414260864, "loss_scale": 4.0, "retries": 0, "seconds": 866.1, "max_memory_gb": 17.18, "ce": 0.05059320861687411, "rank": 0.0}
+{"step": 230, "steps": 250, "lr": 5.888342488294125e-06, "grad_norm": 2.710984706878662, "loss_scale": 4.0, "retries": 0, "seconds": 869.8, "max_memory_gb": 17.18, "ce": 0.40652962220030986, "rank": 0.0}
+{"step": 231, "steps": 250, "lr": 5.806249822994849e-06, "grad_norm": 0.3262815475463867, "loss_scale": 4.0, "retries": 0, "seconds": 873.6, "max_memory_gb": 17.18, "ce": 0.038088351536998744, "rank": 0.0}
+{"step": 232, "steps": 250, "lr": 5.728067332218994e-06, "grad_norm": 0.34009379148483276, "loss_scale": 4.0, "retries": 0, "seconds": 877.3, "max_memory_gb": 17.18, "ce": 0.021945583948195235, "rank": 0.0}
+{"step": 233, "steps": 250, "lr": 5.65380910791383e-06, "grad_norm": 0.5393926501274109, "loss_scale": 4.0, "retries": 0, "seconds": 881.1, "max_memory_gb": 17.18, "ce": 0.024362810928323597, "rank": 0.0}
+{"step": 234, "steps": 250, "lr": 5.583488534700022e-06, "grad_norm": 1.1467746496200562, "loss_scale": 4.0, "retries": 0, "seconds": 884.9, "max_memory_gb": 17.18, "ce": 0.05948891801062928, "rank": 0.0}
+{"step": 235, "steps": 250, "lr": 5.517118287459118e-06, "grad_norm": 2.204561948776245, "loss_scale": 4.0, "retries": 0, "seconds": 888.6, "max_memory_gb": 17.18, "ce": 0.20665558143355156, "rank": 0.0}
+{"step": 236, "steps": 250, "lr": 5.454710329048978e-06, "grad_norm": 0.17737054824829102, "loss_scale": 4.0, "retries": 0, "seconds": 892.3, "max_memory_gb": 17.18, "ce": 0.016109394225509277, "rank": 0.0}
+{"step": 237, "steps": 250, "lr": 5.3962759081475436e-06, "grad_norm": 0.7555365562438965, "loss_scale": 4.0, "retries": 0, "seconds": 896.1, "max_memory_gb": 17.18, "ce": 0.05324910079730216, "rank": 0.0}
+{"step": 238, "steps": 250, "lr": 5.341825557225321e-06, "grad_norm": 0.5347591042518616, "loss_scale": 4.0, "retries": 0, "seconds": 899.8, "max_memory_gb": 17.18, "ce": 0.038858643786056746, "rank": 0.0}
+{"step": 239, "steps": 250, "lr": 5.291369090646961e-06, "grad_norm": 0.3276074230670929, "loss_scale": 4.0, "retries": 0, "seconds": 903.5, "max_memory_gb": 17.18, "ce": 0.08927673325763408, "rank": 0.0}
+{"step": 240, "steps": 250, "lr": 5.244915602902289e-06, "grad_norm": 0.25620338320732117, "loss_scale": 4.0, "retries": 0, "seconds": 907.4, "max_memory_gb": 17.18, "ce": 0.01808375915538818, "rank": 0.0}
+{"step": 241, "steps": 250, "lr": 5.20247346696706e-06, "grad_norm": 0.4335956275463104, "loss_scale": 4.0, "retries": 0, "seconds": 911.1, "max_memory_gb": 17.18, "ce": 0.05950355594086432, "rank": 0.0}
+{"step": 242, "steps": 250, "lr": 5.164050332793787e-06, "grad_norm": 0.1556134670972824, "loss_scale": 4.0, "retries": 0, "seconds": 914.8, "max_memory_gb": 17.18, "ce": 0.010722666317860785, "rank": 0.0}
+{"step": 243, "steps": 250, "lr": 5.129653125932873e-06, "grad_norm": 0.016481464728713036, "loss_scale": 4.0, "retries": 0, "seconds": 918.6, "max_memory_gb": 17.18, "ce": 0.001080853951748395, "rank": 0.0}
+{"step": 244, "steps": 250, "lr": 5.099288046284329e-06, "grad_norm": 0.5006457567214966, "loss_scale": 4.0, "retries": 0, "seconds": 922.4, "max_memory_gb": 17.18, "ce": 0.039173902021957474, "rank": 0.0}
+{"step": 245, "steps": 250, "lr": 5.072960566980277e-06, "grad_norm": 1.5676207542419434, "loss_scale": 4.0, "retries": 0, "seconds": 926.1, "max_memory_gb": 17.18, "ce": 0.22707322750704861, "rank": 0.0}
+{"step": 246, "steps": 250, "lr": 5.050675433398435e-06, "grad_norm": 0.015553131699562073, "loss_scale": 4.0, "retries": 0, "seconds": 929.6, "max_memory_gb": 17.18, "ce": 0.0008900313987396657, "rank": 0.0}
+{"step": 247, "steps": 250, "lr": 5.032436662306811e-06, "grad_norm": 0.5202685594558716, "loss_scale": 4.0, "retries": 0, "seconds": 933.0, "max_memory_gb": 17.18, "ce": 0.04747743179223107, "rank": 0.0}
+{"step": 248, "steps": 250, "lr": 5.018247541139684e-06, "grad_norm": 0.08102989196777344, "loss_scale": 4.0, "retries": 0, "seconds": 936.4, "max_memory_gb": 17.18, "ce": 0.004970824025804177, "rank": 0.0}
+{"step": 249, "steps": 250, "lr": 5.00811062740508e-06, "grad_norm": 0.20482872426509857, "loss_scale": 4.0, "retries": 0, "seconds": 939.7, "max_memory_gb": 17.18, "ce": 0.01353223422688643, "rank": 0.0}
+{"step": 250, "steps": 250, "lr": 5.002027748223778e-06, "grad_norm": 0.5391566753387451, "loss_scale": 4.0, "retries": 0, "seconds": 943.1, "max_memory_gb": 17.18, "ce": 0.02431224195345294, "rank": 0.0}

runs/fixed_internvl/removal/Random8_8103_s13/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /data/OPSD/assets/models/InternVL3_5-8B-HF
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/data/OPSD/assets/models/InternVL3_5-8B-HF
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.20.0

runs/fixed_internvl/removal/Random8_8103_s13/adapter/adapter_config.json

@@ -0,0 +1,213 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {
+    "model.language_model.layers.15.mlp.down_proj": 144,
+    "model.language_model.layers.15.mlp.gate_proj": 144,
+    "model.language_model.layers.15.mlp.up_proj": 144,
+    "model.language_model.layers.15.self_attn.k_proj": 144,
+    "model.language_model.layers.15.self_attn.o_proj": 144,
+    "model.language_model.layers.15.self_attn.q_proj": 144,
+    "model.language_model.layers.15.self_attn.v_proj": 144,
+    "model.language_model.layers.17.mlp.down_proj": 144,
+    "model.language_model.layers.17.mlp.gate_proj": 144,
+    "model.language_model.layers.17.mlp.up_proj": 144,
+    "model.language_model.layers.17.self_attn.k_proj": 144,
+    "model.language_model.layers.17.self_attn.o_proj": 144,
+    "model.language_model.layers.17.self_attn.q_proj": 144,
+    "model.language_model.layers.17.self_attn.v_proj": 144,
+    "model.language_model.layers.29.mlp.down_proj": 144,
+    "model.language_model.layers.29.mlp.gate_proj": 144,
+    "model.language_model.layers.29.mlp.up_proj": 144,
+    "model.language_model.layers.29.self_attn.k_proj": 144,
+    "model.language_model.layers.29.self_attn.o_proj": 144,
+    "model.language_model.layers.29.self_attn.q_proj": 144,
+    "model.language_model.layers.29.self_attn.v_proj": 144,
+    "model.language_model.layers.30.mlp.down_proj": 144,
+    "model.language_model.layers.30.mlp.gate_proj": 144,
+    "model.language_model.layers.30.mlp.up_proj": 144,
+    "model.language_model.layers.30.self_attn.k_proj": 144,
+    "model.language_model.layers.30.self_attn.o_proj": 144,
+    "model.language_model.layers.30.self_attn.q_proj": 144,
+    "model.language_model.layers.30.self_attn.v_proj": 144,
+    "model.language_model.layers.31.mlp.down_proj": 144,
+    "model.language_model.layers.31.mlp.gate_proj": 144,
+    "model.language_model.layers.31.mlp.up_proj": 144,
+    "model.language_model.layers.31.self_attn.k_proj": 144,
+    "model.language_model.layers.31.self_attn.o_proj": 144,
+    "model.language_model.layers.31.self_attn.q_proj": 144,
+    "model.language_model.layers.31.self_attn.v_proj": 144,
+    "model.language_model.layers.32.mlp.down_proj": 144,
+    "model.language_model.layers.32.mlp.gate_proj": 144,
+    "model.language_model.layers.32.mlp.up_proj": 144,
+    "model.language_model.layers.32.self_attn.k_proj": 144,
+    "model.language_model.layers.32.self_attn.o_proj": 144,
+    "model.language_model.layers.32.self_attn.q_proj": 144,
+    "model.language_model.layers.32.self_attn.v_proj": 144,
+    "model.language_model.layers.35.mlp.down_proj": 144,
+    "model.language_model.layers.35.mlp.gate_proj": 144,
+    "model.language_model.layers.35.mlp.up_proj": 144,
+    "model.language_model.layers.35.self_attn.k_proj": 144,
+    "model.language_model.layers.35.self_attn.o_proj": 144,
+    "model.language_model.layers.35.self_attn.q_proj": 144,
+    "model.language_model.layers.35.self_attn.v_proj": 144,
+    "model.language_model.layers.6.mlp.down_proj": 144,
+    "model.language_model.layers.6.mlp.gate_proj": 144,
+    "model.language_model.layers.6.mlp.up_proj": 144,
+    "model.language_model.layers.6.self_attn.k_proj": 144,
+    "model.language_model.layers.6.self_attn.o_proj": 144,
+    "model.language_model.layers.6.self_attn.q_proj": 144,
+    "model.language_model.layers.6.self_attn.v_proj": 144
+  },
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/data/OPSD/assets/models/InternVL3_5-8B-HF",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {
+    "model.language_model.layers.15.mlp.down_proj": 72,
+    "model.language_model.layers.15.mlp.gate_proj": 72,
+    "model.language_model.layers.15.mlp.up_proj": 72,
+    "model.language_model.layers.15.self_attn.k_proj": 72,
+    "model.language_model.layers.15.self_attn.o_proj": 72,
+    "model.language_model.layers.15.self_attn.q_proj": 72,
+    "model.language_model.layers.15.self_attn.v_proj": 72,
+    "model.language_model.layers.17.mlp.down_proj": 72,
+    "model.language_model.layers.17.mlp.gate_proj": 72,
+    "model.language_model.layers.17.mlp.up_proj": 72,
+    "model.language_model.layers.17.self_attn.k_proj": 72,
+    "model.language_model.layers.17.self_attn.o_proj": 72,
+    "model.language_model.layers.17.self_attn.q_proj": 72,
+    "model.language_model.layers.17.self_attn.v_proj": 72,
+    "model.language_model.layers.29.mlp.down_proj": 72,
+    "model.language_model.layers.29.mlp.gate_proj": 72,
+    "model.language_model.layers.29.mlp.up_proj": 72,
+    "model.language_model.layers.29.self_attn.k_proj": 72,
+    "model.language_model.layers.29.self_attn.o_proj": 72,
+    "model.language_model.layers.29.self_attn.q_proj": 72,
+    "model.language_model.layers.29.self_attn.v_proj": 72,
+    "model.language_model.layers.30.mlp.down_proj": 72,
+    "model.language_model.layers.30.mlp.gate_proj": 72,
+    "model.language_model.layers.30.mlp.up_proj": 72,
+    "model.language_model.layers.30.self_attn.k_proj": 72,
+    "model.language_model.layers.30.self_attn.o_proj": 72,
+    "model.language_model.layers.30.self_attn.q_proj": 72,
+    "model.language_model.layers.30.self_attn.v_proj": 72,
+    "model.language_model.layers.31.mlp.down_proj": 72,
+    "model.language_model.layers.31.mlp.gate_proj": 72,
+    "model.language_model.layers.31.mlp.up_proj": 72,
+    "model.language_model.layers.31.self_attn.k_proj": 72,
+    "model.language_model.layers.31.self_attn.o_proj": 72,
+    "model.language_model.layers.31.self_attn.q_proj": 72,
+    "model.language_model.layers.31.self_attn.v_proj": 72,
+    "model.language_model.layers.32.mlp.down_proj": 72,
+    "model.language_model.layers.32.mlp.gate_proj": 72,
+    "model.language_model.layers.32.mlp.up_proj": 72,
+    "model.language_model.layers.32.self_attn.k_proj": 72,
+    "model.language_model.layers.32.self_attn.o_proj": 72,
+    "model.language_model.layers.32.self_attn.q_proj": 72,
+    "model.language_model.layers.32.self_attn.v_proj": 72,
+    "model.language_model.layers.35.mlp.down_proj": 72,
+    "model.language_model.layers.35.mlp.gate_proj": 72,
+    "model.language_model.layers.35.mlp.up_proj": 72,
+    "model.language_model.layers.35.self_attn.k_proj": 72,
+    "model.language_model.layers.35.self_attn.o_proj": 72,
+    "model.language_model.layers.35.self_attn.q_proj": 72,
+    "model.language_model.layers.35.self_attn.v_proj": 72,
+    "model.language_model.layers.6.mlp.down_proj": 72,
+    "model.language_model.layers.6.mlp.gate_proj": 72,
+    "model.language_model.layers.6.mlp.up_proj": 72,
+    "model.language_model.layers.6.self_attn.k_proj": 72,
+    "model.language_model.layers.6.self_attn.o_proj": 72,
+    "model.language_model.layers.6.self_attn.q_proj": 72,
+    "model.language_model.layers.6.self_attn.v_proj": 72
+  },
+  "revision": null,
+  "target_modules": [
+    "model.language_model.layers.32.mlp.gate_proj",
+    "model.language_model.layers.30.self_attn.k_proj",
+    "model.language_model.layers.35.mlp.down_proj",
+    "model.language_model.layers.17.self_attn.v_proj",
+    "model.language_model.layers.30.self_attn.q_proj",
+    "model.language_model.layers.32.self_attn.k_proj",
+    "model.language_model.layers.6.self_attn.q_proj",
+    "model.language_model.layers.6.self_attn.v_proj",
+    "model.language_model.layers.17.mlp.up_proj",
+    "model.language_model.layers.29.mlp.down_proj",
+    "model.language_model.layers.30.mlp.gate_proj",
+    "model.language_model.layers.6.mlp.down_proj",
+    "model.language_model.layers.17.self_attn.o_proj",
+    "model.language_model.layers.29.self_attn.k_proj",
+    "model.language_model.layers.15.mlp.gate_proj",
+    "model.language_model.layers.15.self_attn.o_proj",
+    "model.language_model.layers.15.mlp.down_proj",
+    "model.language_model.layers.35.self_attn.k_proj",
+    "model.language_model.layers.31.mlp.down_proj",
+    "model.language_model.layers.31.self_attn.k_proj",
+    "model.language_model.layers.35.self_attn.v_proj",
+    "model.language_model.layers.17.self_attn.q_proj",
+    "model.language_model.layers.32.mlp.down_proj",
+    "model.language_model.layers.32.mlp.up_proj",
+    "model.language_model.layers.35.mlp.gate_proj",
+    "model.language_model.layers.31.self_attn.v_proj",
+    "model.language_model.layers.32.self_attn.v_proj",
+    "model.language_model.layers.29.self_attn.v_proj",
+    "model.language_model.layers.35.self_attn.q_proj",
+    "model.language_model.layers.15.mlp.up_proj",
+    "model.language_model.layers.29.mlp.gate_proj",
+    "model.language_model.layers.6.self_attn.o_proj",
+    "model.language_model.layers.15.self_attn.v_proj",
+    "model.language_model.layers.29.mlp.up_proj",
+    "model.language_model.layers.30.self_attn.o_proj",
+    "model.language_model.layers.29.self_attn.o_proj",
+    "model.language_model.layers.32.self_attn.o_proj",
+    "model.language_model.layers.17.mlp.gate_proj",
+    "model.language_model.layers.31.self_attn.q_proj",
+    "model.language_model.layers.17.self_attn.k_proj",
+    "model.language_model.layers.17.mlp.down_proj",
+    "model.language_model.layers.32.self_attn.q_proj",
+    "model.language_model.layers.6.mlp.gate_proj",
+    "model.language_model.layers.31.mlp.gate_proj",
+    "model.language_model.layers.31.mlp.up_proj",
+    "model.language_model.layers.31.self_attn.o_proj",
+    "model.language_model.layers.6.self_attn.k_proj",
+    "model.language_model.layers.15.self_attn.k_proj",
+    "model.language_model.layers.15.self_attn.q_proj",
+    "model.language_model.layers.35.self_attn.o_proj",
+    "model.language_model.layers.30.mlp.up_proj",
+    "model.language_model.layers.35.mlp.up_proj",
+    "model.language_model.layers.29.self_attn.q_proj",
+    "model.language_model.layers.6.mlp.up_proj",
+    "model.language_model.layers.30.mlp.down_proj",
+    "model.language_model.layers.30.self_attn.v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}