Upload folder using huggingface_hub

best_eval_1/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: ''
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- 'base_model:adapter:'
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

best_eval_1/adapter/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 512,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 256,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "gate",
+    "q_proj",
+    "up",
+    "v_proj",
+    "down",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

best_eval_1/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c362ce14355cae2dd56af8221693c7b6e8194a6338a41a21bf22b09aa62f6705
+size 155198248

best_eval_1/cuda_rng.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfa47d762301992efbdc1b1bd617b22937e18fd1d502354481860eea9ba3be63
+size 1584

best_eval_1/numpy_rng.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bd2bef57f4304c9f2ecf89022498801b55e0cf339fd3b7ec48666907b8ee80a
+size 2675

best_eval_1/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d942a9764f743e33e411c73334bbbc71be69f2048eba9e436a571521fdf3d7c4
+size 310433291

best_eval_1/python_rng.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8648e962b57e6a2136a6fba8892cfc7d625d7a3b4ed99f5561934fc7725f0ab6
+size 3762

best_eval_1/torch_rng.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ef73e8425e75b9b9ed7f1ddc0e8f1658494f12bbec61c3ee76bac7dfaae5c05
+size 6647

best_eval_1/train_state.json

@@ -0,0 +1,24 @@
+{
+  "step": 1,
+  "losses": [
+    2.1343601644039154
+  ],
+  "dataset_offset": 0,
+  "config": {
+    "king_dir": "/root/king_165_hf",
+    "optimizer": "adamw",
+    "lr": 5e-05,
+    "muon_lr": 0.02,
+    "lora_r": 256,
+    "steps": 5000,
+    "grad_accum": 4,
+    "seq_len": 2048,
+    "unfreeze_head": false,
+    "source_dirs": [
+      "/root/datasets/finewebedu"
+    ],
+    "source_weights": [
+      1.0
+    ]
+  }
+}

best_eval_10/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: ''
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- 'base_model:adapter:'
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

best_eval_10/adapter/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 512,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 256,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "gate",
+    "q_proj",
+    "up",
+    "v_proj",
+    "down",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

best_eval_10/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:93f527ed8b98f61b7e258c1c6f7d693fcfa9895ecd7bd7862eab34e72102f61b
+size 155198248

best_eval_10/cuda_rng.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfa47d762301992efbdc1b1bd617b22937e18fd1d502354481860eea9ba3be63
+size 1584

best_eval_10/numpy_rng.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bd2bef57f4304c9f2ecf89022498801b55e0cf339fd3b7ec48666907b8ee80a
+size 2675

best_eval_10/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b12b43ee78722156879cce3f567196f2120f0f8edaf4136363b278c7e7924ea
+size 310433291

best_eval_10/python_rng.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8648e962b57e6a2136a6fba8892cfc7d625d7a3b4ed99f5561934fc7725f0ab6
+size 3762

best_eval_10/torch_rng.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ef73e8425e75b9b9ed7f1ddc0e8f1658494f12bbec61c3ee76bac7dfaae5c05
+size 6647

best_eval_10/train_state.json

@@ -0,0 +1,924 @@
+{
+  "step": 901,
+  "losses": [
+    2.1343601644039154,
+    2.072067230939865,
+    1.8817282915115356,
+    1.5482478141784668,
+    2.0044714510440826,
+    1.9346624314785004,
+    2.0394379794597626,
+    2.0922091901302338,
+    1.9468883574008942,
+    1.7588119804859161,
+    1.874794840812683,
+    1.911599040031433,
+    1.6168376505374908,
+    1.7635800540447235,
+    2.070902556180954,
+    1.7358599305152893,
+    2.070700913667679,
+    1.9784598052501678,
+    1.9782940447330475,
+    1.8401454091072083,
+    1.6551836282014847,
+    1.9741482436656952,
+    1.990296185016632,
+    1.8002875596284866,
+    2.04819917678833,
+    1.8555984199047089,
+    2.1058743596076965,
+    1.9181069433689117,
+    2.1770740747451782,
+    2.1200993955135345,
+    1.5497342869639397,
+    2.041239619255066,
+    1.7999589145183563,
+    1.8640441000461578,
+    1.8068368434906006,
+    1.958029717206955,
+    2.3048327565193176,
+    2.0624635815620422,
+    1.7916592359542847,
+    2.197334885597229,
+    1.8518704771995544,
+    2.014060288667679,
+    2.3126999735832214,
+    2.0366491973400116,
+    2.1218522787094116,
+    1.9050887823104858,
+    2.140252709388733,
+    2.016432911157608,
+    2.0771953761577606,
+    1.99711474776268,
+    2.2010857462882996,
+    1.906881034374237,
+    1.9355229437351227,
+    1.568751234561205,
+    2.218832314014435,
+    2.282701849937439,
+    2.1470184922218323,
+    1.8618396520614624,
+    2.0435074865818024,
+    2.2046265304088593,
+    2.036127895116806,
+    1.9144421517848969,
+    1.7361707091331482,
+    1.8860499560832977,
+    1.9349249005317688,
+    2.091381222009659,
+    2.2041552662849426,
+    2.0161964893341064,
+    1.6289199590682983,
+    1.7889264225959778,
+    2.021867960691452,
+    2.1548608541488647,
+    1.7794556617736816,
+    1.9493914544582367,
+    1.8160816133022308,
+    1.8409814834594727,
+    2.2246702015399933,
+    2.080227106809616,
+    1.6188245043158531,
+    1.8832258582115173,
+    1.818663239479065,
+    2.0883366763591766,
+    2.00100439786911,
+    2.1166227757930756,
+    2.0359020829200745,
+    2.2903840243816376,
+    2.1877321600914,
+    1.8351092636585236,
+    2.1081435680389404,
+    1.876007080078125,
+    1.6549921333789825,
+    2.0958774387836456,
+    1.93923220038414,
+    1.8541749119758606,
+    2.41263547539711,
+    2.037635773420334,
+    2.0588943660259247,
+    1.969245046377182,
+    1.7285726368427277,
+    1.9336746335029602,
+    2.1948340237140656,
+    1.4296438321471214,
+    1.8817992806434631,
+    1.8943344950675964,
+    1.779104083776474,
+    1.9108014702796936,
+    1.805834949016571,
+    1.834848165512085,
+    1.9906627535820007,
+    2.1963523626327515,
+    2.1539515554904938,
+    1.9684762954711914,
+    1.910222202539444,
+    2.019984722137451,
+    1.9977181553840637,
+    1.6560729146003723,
+    1.9717278629541397,
+    1.8783657252788544,
+    1.8279190361499786,
+    2.0361652076244354,
+    1.8888441324234009,
+    2.0143093168735504,
+    1.7985135614871979,
+    2.2679185271263123,
+    1.9247187972068787,
+    1.7891883850097656,
+    2.112279176712036,
+    2.047613203525543,
+    2.1257645189762115,
+    2.0965375006198883,
+    1.8606078326702118,
+    1.986148178577423,
+    2.235414505004883,
+    2.0115409195423126,
+    1.8105634152889252,
+    2.055979162454605,
+    1.8827663362026215,
+    2.228433668613434,
+    2.168084293603897,
+    1.9350906312465668,
+    2.1963738203048706,
+    1.9545983374118805,
+    2.0042108595371246,
+    2.076897233724594,
+    2.164249151945114,
+    2.2511336505413055,
+    2.147927701473236,
+    1.666139394044876,
+    1.9886944591999054,
+    1.8915810883045197,
+    1.9184699952602386,
+    1.9324964582920074,
+    2.2565895915031433,
+    1.8644168674945831,
+    2.024057060480118,
+    2.1561132073402405,
+    1.831732153892517,
+    2.0300653278827667,
+    1.8685192167758942,
+    1.9508120715618134,
+    2.217629134654999,
+    1.7884540557861328,
+    1.9629022479057312,
+    2.014945089817047,
+    1.773240178823471,
+    1.8480553328990936,
+    2.174328625202179,
+    2.0528362691402435,
+    1.9184516966342926,
+    1.9994118809700012,
+    2.1629433631896973,
+    1.956313133239746,
+    1.4914195835590363,
+    2.0376154482364655,
+    1.9383164942264557,
+    1.8460872173309326,
+    2.1424327194690704,
+    2.047647774219513,
+    2.094540774822235,
+    1.3891073316335678,
+    1.9750277996063232,
+    2.0804302990436554,
+    1.9623984098434448,
+    1.7242400497198105,
+    1.6598125100135803,
+    2.072585940361023,
+    2.003816694021225,
+    1.7694512009620667,
+    2.2525570392608643,
+    1.8560697436332703,
+    2.7015225291252136,
+    2.017684131860733,
+    1.930792212486267,
+    1.9603004455566406,
+    2.167401760816574,
+    2.0757872462272644,
+    2.1265901923179626,
+    2.067963659763336,
+    1.9692361950874329,
+    2.0600834786891937,
+    2.0879141092300415,
+    1.8167440295219421,
+    1.657739669084549,
+    1.9096021354198456,
+    1.9149316847324371,
+    1.9598571062088013,
+    2.0564489662647247,
+    2.0975446701049805,
+    2.1563239097595215,
+    1.96071657538414,
+    1.478960782289505,
+    1.9158387184143066,
+    1.7507797181606293,
+    2.0135932564735413,
+    2.245123326778412,
+    2.182836413383484,
+    1.9111057817935944,
+    1.8482527136802673,
+    1.805978387594223,
+    2.0817373394966125,
+    1.896893858909607,
+    1.9579625725746155,
+    2.200235515832901,
+    2.0788964927196503,
+    1.858733743429184,
+    1.91632080078125,
+    1.9030437469482422,
+    1.8899632692337036,
+    1.7974070012569427,
+    1.7762640118598938,
+    1.4867072105407715,
+    1.8656535148620605,
+    2.1225537061691284,
+    1.9701803922653198,
+    1.8501644432544708,
+    1.847507894039154,
+    1.992827296257019,
+    1.8754442632198334,
+    1.9105960130691528,
+    1.7681310772895813,
+    1.8711368143558502,
+    1.9765931069850922,
+    1.879693478345871,
+    1.5103979259729385,
+    2.032119333744049,
+    2.084160417318344,
+    1.6835585832595825,
+    1.6743774712085724,
+    2.1584636867046356,
+    1.8118220269680023,
+    1.9816903471946716,
+    2.029897838830948,
+    2.163084864616394,
+    1.906882494688034,
+    2.3502153158187866,
+    1.755303293466568,
+    2.171054571866989,
+    1.9884036481380463,
+    1.8938909471035004,
+    2.2752005457878113,
+    1.8866967856884003,
+    1.8928779065608978,
+    2.0702185928821564,
+    1.9415528178215027,
+    2.025204062461853,
+    2.156549721956253,
+    1.763477623462677,
+    2.2139832973480225,
+    1.6408961713314056,
+    1.7618666142225266,
+    1.8543719351291656,
+    1.7970535457134247,
+    2.226055681705475,
+    1.944146305322647,
+    1.9856219589710236,
+    2.003736823797226,
+    1.9503918588161469,
+    1.7600468397140503,
+    2.275264799594879,
+    2.0795653760433197,
+    1.490852490067482,
+    1.8964970707893372,
+    1.9750999510288239,
+    2.043429046869278,
+    2.2627260088920593,
+    1.9151235222816467,
+    2.0287132263183594,
+    1.90212082862854,
+    1.7960082292556763,
+    1.94166761636734,
+    1.9386018514633179,
+    2.134494125843048,
+    2.0087448060512543,
+    2.0281916856765747,
+    1.9535616338253021,
+    1.9679738581180573,
+    1.596398025751114,
+    2.191830277442932,
+    2.0763327181339264,
+    2.1266538500785828,
+    2.274190306663513,
+    1.8964255154132843,
+    2.1241783499717712,
+    1.9198937714099884,
+    2.3034598231315613,
+    1.9052272140979767,
+    1.672550529241562,
+    2.046636402606964,
+    1.9329883009195328,
+    2.1311089992523193,
+    2.2328023314476013,
+    1.9483379125595093,
+    1.8120945990085602,
+    1.8161226511001587,
+    1.7844904065132141,
+    1.8877747356891632,
+    2.010182023048401,
+    1.9179834425449371,
+    2.0235649049282074,
+    2.120028495788574,
+    2.414934992790222,
+    1.8385069966316223,
+    1.8532257974147797,
+    2.0549443662166595,
+    1.9257167875766754,
+    1.7838368117809296,
+    1.9770499169826508,
+    2.02278470993042,
+    2.018839567899704,
+    1.8134825825691223,
+    1.912839412689209,
+    2.1395110189914703,
+    2.238103538751602,
+    1.6793973743915558,
+    2.275532752275467,
+    2.087612211704254,
+    1.8759168982505798,
+    1.6412090212106705,
+    1.9292035102844238,
+    2.205808609724045,
+    1.998661369085312,
+    2.052595764398575,
+    2.0764198303222656,
+    2.2456181049346924,
+    2.0933930575847626,
+    1.874241203069687,
+    1.7711814045906067,
+    1.5416613817214966,
+    1.7534068077802658,
+    2.0805560648441315,
+    1.6377001479268074,
+    1.884499341249466,
+    2.143370270729065,
+    2.110937684774399,
+    1.977731168270111,
+    1.97747004032135,
+    1.7653805166482925,
+    1.843018800020218,
+    2.2338719964027405,
+    2.2774742245674133,
+    2.079852908849716,
+    2.0538104474544525,
+    2.063538581132889,
+    1.7676877677440643,
+    2.0208137929439545,
+    1.9883665442466736,
+    2.0645232796669006,
+    1.9443430304527283,
+    1.7410898208618164,
+    2.0080275535583496,
+    1.9868277311325073,
+    2.078660637140274,
+    2.2050068378448486,
+    1.9919089078903198,
+    2.252542018890381,
+    1.9415084719657898,
+    1.8200271725654602,
+    1.9846361577510834,
+    1.9353410005569458,
+    1.912458896636963,
+    2.0796336233615875,
+    2.1193440556526184,
+    1.8158704936504364,
+    2.223644942045212,
+    2.0725834667682648,
+    2.0095846951007843,
+    2.1535970866680145,
+    2.0234865248203278,
+    1.796433001756668,
+    1.9197930991649628,
+    2.022257298231125,
+    1.6413149237632751,
+    1.9616879522800446,
+    1.9451987147331238,
+    2.2428542971611023,
+    2.030157655477524,
+    1.5762348473072052,
+    1.9567807912826538,
+    2.0858742594718933,
+    2.04095658659935,
+    2.1477834284305573,
+    1.857110321521759,
+    1.9249985218048096,
+    2.067128211259842,
+    1.8338475227355957,
+    1.8183660209178925,
+    1.7897652685642242,
+    2.016318142414093,
+    1.9572072327136993,
+    1.972920686006546,
+    2.010816305875778,
+    1.5412680506706238,
+    1.9066489338874817,
+    2.0087280571460724,
+    1.7885839641094208,
+    2.036726325750351,
+    1.8658419251441956,
+    1.9655573070049286,
+    2.1818976402282715,
+    1.7045586109161377,
+    2.3182644844055176,
+    1.8516510128974915,
+    1.766547679901123,
+    1.942388653755188,
+    2.11874783039093,
+    1.8681564033031464,
+    1.6302369683980942,
+    1.8748173117637634,
+    2.116931974887848,
+    2.1928129494190216,
+    1.8579590022563934,
+    2.000575751066208,
+    2.3248833417892456,
+    2.0099377036094666,
+    1.9941376149654388,
+    1.9139507114887238,
+    1.798469752073288,
+    2.045513093471527,
+    2.139172524213791,
+    2.0657175183296204,
+    2.5929186940193176,
+    1.887778788805008,
+    2.0180120766162872,
+    1.923815667629242,
+    2.1513286232948303,
+    2.0689505338668823,
+    2.0100492537021637,
+    2.10652357339859,
+    1.8175769448280334,
+    1.932394802570343,
+    1.8030853569507599,
+    2.1482163667678833,
+    1.8957428932189941,
+    2.1001981496810913,
+    1.8274339735507965,
+    1.755200743675232,
+    1.9862683713436127,
+    2.270208418369293,
+    1.8101052641868591,
+    1.9821595251560211,
+    1.5272532850503922,
+    2.014067441225052,
+    1.9742996096611023,
+    1.9847474098205566,
+    2.2911656200885773,
+    2.149272084236145,
+    2.224188506603241,
+    2.1615083813667297,
+    1.7423981428146362,
+    1.7683758437633514,
+    2.028497099876404,
+    1.7579623758792877,
+    1.9390694499015808,
+    2.1545024514198303,
+    2.048369884490967,
+    1.9410778284072876,
+    2.0469002723693848,
+    1.939433366060257,
+    1.8914868533611298,
+    2.118733376264572,
+    2.0376307666301727,
+    1.7661086916923523,
+    2.0115101635456085,
+    2.035685360431671,
+    1.9506032168865204,
+    1.6531559526920319,
+    2.022988259792328,
+    2.2393264770507812,
+    2.0491443276405334,
+    1.9842473566532135,
+    1.9727666676044464,
+    2.22902449965477,
+    1.8579447865486145,
+    2.0388655066490173,
+    2.01955908536911,
+    1.9977469444274902,
+    2.1867045164108276,
+    1.6018355786800385,
+    1.900251418352127,
+    2.2395838499069214,
+    1.797933965921402,
+    2.1339435279369354,
+    1.9796839654445648,
+    2.0070094764232635,
+    2.068116545677185,
+    2.143452674150467,
+    2.089024990797043,
+    2.096229612827301,
+    2.013333708047867,
+    1.9395509362220764,
+    1.7489879727363586,
+    2.155642122030258,
+    1.687512069940567,
+    2.1562404334545135,
+    2.119892954826355,
+    2.0649279952049255,
+    1.9891228377819061,
+    1.8696045875549316,
+    1.8906341791152954,
+    1.3779398649930954,
+    1.930278092622757,
+    1.9531156122684479,
+    1.7971335351467133,
+    1.9451295733451843,
+    1.9390936195850372,
+    2.0392781794071198,
+    2.332154870033264,
+    1.9894782602787018,
+    2.0120878517627716,
+    1.7841753661632538,
+    2.145608127117157,
+    1.8821957111358643,
+    1.9437867999076843,
+    1.8886327743530273,
+    1.6309072077274323,
+    1.7295852303504944,
+    2.2243327498435974,
+    1.8572883307933807,
+    2.025457978248596,
+    1.7401179671287537,
+    1.2589047253131866,
+    2.216827839612961,
+    1.7800424695014954,
+    2.0404772460460663,
+    1.883945882320404,
+    2.0013100802898407,
+    2.1914494931697845,
+    2.097546637058258,
+    2.110943377017975,
+    2.328992247581482,
+    2.1373319923877716,
+    2.1446226835250854,
+    1.8427353352308273,
+    1.8370108008384705,
+    1.9873742759227753,
+    1.9717270731925964,
+    1.6525997519493103,
+    1.8598004281520844,
+    1.9432488083839417,
+    2.1452393233776093,
+    1.902628868818283,
+    1.6516613364219666,
+    1.9785824716091156,
+    1.9252119362354279,
+    1.7771514356136322,
+    2.1968206465244293,
+    1.9527567625045776,
+    2.0789993703365326,
+    1.841865748167038,
+    1.8205523192882538,
+    1.8681734204292297,
+    1.7501431107521057,
+    2.1056064069271088,
+    1.8949854373931885,
+    1.9272009134292603,
+    1.8338072001934052,
+    1.8234700560569763,
+    2.0019575655460358,
+    1.5818779952824116,
+    2.024990826845169,
+    2.207423210144043,
+    1.7634362280368805,
+    1.8431354463100433,
+    1.984557956457138,
+    1.8526888191699982,
+    1.794599860906601,
+    1.6078735142946243,
+    1.8850816488265991,
+    1.9796228408813477,
+    2.2156145572662354,
+    2.195041835308075,
+    2.1863420605659485,
+    1.8231058716773987,
+    1.9898098707199097,
+    2.0260429978370667,
+    2.074339270591736,
+    1.288858488202095,
+    1.9821405112743378,
+    2.034531831741333,
+    2.0912082493305206,
+    2.1728965044021606,
+    1.7105523645877838,
+    2.0361987948417664,
+    2.019710510969162,
+    1.9856821298599243,
+    1.8924728333950043,
+    1.8184559047222137,
+    2.0388779640197754,
+    2.0392623245716095,
+    2.161120504140854,
+    1.7053113281726837,
+    1.9326909482479095,
+    2.131219744682312,
+    2.0349932610988617,
+    1.9700875282287598,
+    2.198672741651535,
+    1.7098436057567596,
+    1.4670239239931107,
+    1.8805603981018066,
+    2.08471417427063,
+    2.0385683178901672,
+    2.1507520973682404,
+    2.1592154502868652,
+    1.758884608745575,
+    1.8553593754768372,
+    2.205213576555252,
+    1.8334356546401978,
+    2.0568824410438538,
+    1.9245964884757996,
+    1.812854528427124,
+    2.0937885642051697,
+    2.075963944196701,
+    2.0015837252140045,
+    1.9177868962287903,
+    2.0384775698184967,
+    2.113261580467224,
+    1.8370648920536041,
+    1.9324750304222107,
+    1.7321276366710663,
+    2.1536856293678284,
+    2.1578134894371033,
+    1.9289006888866425,
+    1.844500482082367,
+    2.3947205543518066,
+    1.9309192895889282,
+    1.958142876625061,
+    1.963348239660263,
+    2.1394614577293396,
+    2.2124475836753845,
+    1.8503296673297882,
+    2.187890976667404,
+    1.9677619636058807,
+    2.2120757699012756,
+    2.331635594367981,
+    1.904624581336975,
+    2.202490597963333,
+    1.8061237633228302,
+    1.919066458940506,
+    2.0077410340309143,
+    1.8513532876968384,
+    2.071906268596649,
+    1.8770314753055573,
+    1.9955376088619232,
+    2.107666552066803,
+    2.1590643525123596,
+    2.2253186106681824,
+    2.4541269540786743,
+    2.217774510383606,
+    1.3695485442876816,
+    2.1457070410251617,
+    1.7335527837276459,
+    1.5995305925607681,
+    2.0256753265857697,
+    1.9565629065036774,
+    1.791105717420578,
+    2.0784226655960083,
+    2.1166191399097443,
+    1.9152996242046356,
+    1.8362950682640076,
+    1.7731340825557709,
+    2.155468314886093,
+    2.1131235361099243,
+    2.3282033801078796,
+    1.9125248193740845,
+    1.8689576089382172,
+    1.9485906958580017,
+    1.932999312877655,
+    2.4048546850681305,
+    1.9275791645050049,
+    1.9732040166854858,
+    1.8517394661903381,
+    1.9014380425214767,
+    1.882369488477707,
+    2.1833301186561584,
+    2.1289570331573486,
+    2.183875024318695,
+    2.171056389808655,
+    2.002423048019409,
+    1.9248660802841187,
+    2.0061028003692627,
+    2.028146803379059,
+    2.421733796596527,
+    1.8837913572788239,
+    1.9675268530845642,
+    2.374457895755768,
+    1.7807093858718872,
+    2.172850728034973,
+    2.153034210205078,
+    2.0617380142211914,
+    1.8835320174694061,
+    2.0785669684410095,
+    2.3138357996940613,
+    2.1923963725566864,
+    1.869694173336029,
+    2.0251356065273285,
+    2.1220491230487823,
+    2.0145900547504425,
+    2.1503432393074036,
+    2.190626412630081,
+    2.2802276611328125,
+    2.236900269985199,
+    1.9811552166938782,
+    2.092166304588318,
+    2.07303449511528,
+    1.6356765925884247,
+    2.077802836894989,
+    1.9610523879528046,
+    1.8983043730258942,
+    1.7379110455513,
+    2.0595612823963165,
+    1.7658316493034363,
+    2.0359201431274414,
+    2.078512877225876,
+    2.1333895325660706,
+    1.9176843166351318,
+    2.070345878601074,
+    2.202768564224243,
+    1.945077806711197,
+    1.6647659242153168,
+    1.449508160352707,
+    2.1967802345752716,
+    2.186197817325592,
+    1.8318515717983246,
+    1.9661785066127777,
+    2.1331526339054108,
+    1.9662226438522339,
+    1.9168857634067535,
+    1.9672299921512604,
+    2.0793337225914,
+    1.742641642689705,
+    1.8899154961109161,
+    1.9683457612991333,
+    2.1284988820552826,
+    1.8823730945587158,
+    1.9671491086483002,
+    1.7891699075698853,
+    2.0366631746292114,
+    2.170773684978485,
+    2.157770574092865,
+    1.531184360384941,
+    1.7942973375320435,
+    1.9538165926933289,
+    1.7740932703018188,
+    1.8444747924804688,
+    2.1593389213085175,
+    2.0222666561603546,
+    2.153877913951874,
+    2.1281114518642426,
+    1.8055661916732788,
+    1.8112592101097107,
+    2.1732959151268005,
+    2.2776984870433807,
+    2.01517316699028,
+    2.048113375902176,
+    2.3082817792892456,
+    1.9476797580718994,
+    1.9273951947689056,
+    2.175132006406784,
+    2.0655637979507446,
+    2.0974556505680084,
+    1.983421117067337,
+    2.0001393258571625,
+    1.8526580929756165,
+    1.875769704580307,
+    1.919754534959793,
+    2.061443716287613,
+    2.3129413425922394,
+    1.9128734767436981,
+    2.015645980834961,
+    2.237905263900757,
+    2.1461544036865234,
+    1.9235659837722778,
+    2.173466682434082,
+    1.6294988542795181,
+    1.895463079214096,
+    1.7323124408721924,
+    1.5954849123954773,
+    1.7995491325855255,
+    1.6643447279930115,
+    2.0132358074188232,
+    1.9786936044692993,
+    1.9528062641620636,
+    2.0898461639881134,
+    2.0195729434490204,
+    2.1435398161411285,
+    1.8278231024742126,
+    1.982870191335678,
+    1.7309416234493256,
+    2.242147445678711,
+    2.108400136232376,
+    1.7300675511360168,
+    1.8113693594932556,
+    2.2310619354248047,
+    2.0328956842422485,
+    2.18557870388031,
+    2.3422034978866577,
+    2.169844090938568,
+    1.8169803321361542,
+    2.132762908935547,
+    1.7648725509643555,
+    1.9677415788173676,
+    1.8632167875766754,
+    1.8988753855228424,
+    1.8592016994953156,
+    1.8096055090427399,
+    1.8690961003303528,
+    1.8140289187431335,
+    1.876509428024292,
+    1.9361322224140167,
+    1.9273012280464172,
+    1.6636610329151154,
+    2.111586958169937,
+    2.10248139500618,
+    2.076209932565689,
+    1.8651195466518402,
+    1.7480785846710205,
+    1.7089017778635025,
+    1.8724442422389984,
+    2.272181123495102,
+    1.7920746207237244,
+    1.9623207449913025,
+    2.0105388164520264,
+    1.8501586616039276,
+    1.9171726405620575,
+    2.1572029292583466,
+    1.8143680393695831,
+    2.1485328674316406,
+    2.0317324697971344,
+    2.1904807686805725,
+    2.2118465304374695,
+    1.621018499135971,
+    2.0450506508350372,
+    2.0084542334079742,
+    1.6542732417583466,
+    1.7350412607192993,
+    2.124550372362137,
+    1.9821723401546478,
+    1.8534682095050812,
+    2.0167691111564636,
+    2.191506564617157,
+    1.853075534105301,
+    2.064965844154358,
+    2.133788377046585,
+    1.9932806193828583,
+    2.0305879712104797,
+    2.002735823392868,
+    2.17588073015213,
+    1.9466276466846466,
+    1.9742610156536102,
+    2.0453275442123413,
+    2.1021065413951874,
+    1.926622450351715,
+    1.9230711162090302,
+    1.974651575088501,
+    2.1180694103240967,
+    2.0189504623413086,
+    1.9734396636486053,
+    2.0832147002220154,
+    2.1344321370124817,
+    1.9121268093585968,
+    1.7806911766529083,
+    1.894917517900467,
+    2.442289859056473,
+    2.151290535926819,
+    1.8238812685012817,
+    2.1155946850776672,
+    1.9933439195156097,
+    1.9308890998363495,
+    1.8547934293746948,
+    1.7293584942817688,
+    1.8168570399284363,
+    1.9901467859745026,
+    2.0452483892440796,
+    1.9110192954540253,
+    2.010503828525543,
+    2.047707736492157,
+    1.8187103271484375,
+    1.774341732263565,
+    2.115667313337326,
+    2.1247178316116333,
+    2.0494262874126434
+  ],
+  "dataset_offset": 0,
+  "config": {
+    "king_dir": "/root/king_165_hf",
+    "optimizer": "adamw",
+    "lr": 5e-05,
+    "muon_lr": 0.02,
+    "lora_r": 256,
+    "steps": 5000,
+    "grad_accum": 4,
+    "seq_len": 2048,
+    "unfreeze_head": false,
+    "source_dirs": [
+      "/root/datasets/finewebedu"
+    ],
+    "source_weights": [
+      1.0
+    ]
+  }
+}

best_eval_11/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: ''
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- 'base_model:adapter:'
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

best_eval_11/adapter/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 512,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 256,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "gate",
+    "q_proj",
+    "up",
+    "v_proj",
+    "down",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

best_eval_11/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6933928fe5a5ea2c3285bf6bd683249c5a8ff8a5e36381f3dbd9c824e05e637f
+size 155198248

best_eval_11/cuda_rng.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfa47d762301992efbdc1b1bd617b22937e18fd1d502354481860eea9ba3be63
+size 1584

best_eval_11/numpy_rng.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bd2bef57f4304c9f2ecf89022498801b55e0cf339fd3b7ec48666907b8ee80a
+size 2675

best_eval_11/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d42259663fa0c90997594a8602339a193e227d803c05cfbaa832466a8cf1df63
+size 310433291

best_eval_11/python_rng.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8648e962b57e6a2136a6fba8892cfc7d625d7a3b4ed99f5561934fc7725f0ab6
+size 3762

best_eval_11/torch_rng.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ef73e8425e75b9b9ed7f1ddc0e8f1658494f12bbec61c3ee76bac7dfaae5c05
+size 6647

best_eval_11/train_state.json

@@ -0,0 +1,1324 @@
+{
+  "step": 1301,
+  "losses": [
+    2.1343601644039154,
+    2.072067230939865,
+    1.8817282915115356,
+    1.5482478141784668,
+    2.0044714510440826,
+    1.9346624314785004,
+    2.0394379794597626,
+    2.0922091901302338,
+    1.9468883574008942,
+    1.7588119804859161,
+    1.874794840812683,
+    1.911599040031433,
+    1.6168376505374908,
+    1.7635800540447235,
+    2.070902556180954,
+    1.7358599305152893,
+    2.070700913667679,
+    1.9784598052501678,
+    1.9782940447330475,
+    1.8401454091072083,
+    1.6551836282014847,
+    1.9741482436656952,
+    1.990296185016632,
+    1.8002875596284866,
+    2.04819917678833,
+    1.8555984199047089,
+    2.1058743596076965,
+    1.9181069433689117,
+    2.1770740747451782,
+    2.1200993955135345,
+    1.5497342869639397,
+    2.041239619255066,
+    1.7999589145183563,
+    1.8640441000461578,
+    1.8068368434906006,
+    1.958029717206955,
+    2.3048327565193176,
+    2.0624635815620422,
+    1.7916592359542847,
+    2.197334885597229,
+    1.8518704771995544,
+    2.014060288667679,
+    2.3126999735832214,
+    2.0366491973400116,
+    2.1218522787094116,
+    1.9050887823104858,
+    2.140252709388733,
+    2.016432911157608,
+    2.0771953761577606,
+    1.99711474776268,
+    2.2010857462882996,
+    1.906881034374237,
+    1.9355229437351227,
+    1.568751234561205,
+    2.218832314014435,
+    2.282701849937439,
+    2.1470184922218323,
+    1.8618396520614624,
+    2.0435074865818024,
+    2.2046265304088593,
+    2.036127895116806,
+    1.9144421517848969,
+    1.7361707091331482,
+    1.8860499560832977,
+    1.9349249005317688,
+    2.091381222009659,
+    2.2041552662849426,
+    2.0161964893341064,
+    1.6289199590682983,
+    1.7889264225959778,
+    2.021867960691452,
+    2.1548608541488647,
+    1.7794556617736816,
+    1.9493914544582367,
+    1.8160816133022308,
+    1.8409814834594727,
+    2.2246702015399933,
+    2.080227106809616,
+    1.6188245043158531,
+    1.8832258582115173,
+    1.818663239479065,
+    2.0883366763591766,
+    2.00100439786911,
+    2.1166227757930756,
+    2.0359020829200745,
+    2.2903840243816376,
+    2.1877321600914,
+    1.8351092636585236,
+    2.1081435680389404,
+    1.876007080078125,
+    1.6549921333789825,
+    2.0958774387836456,
+    1.93923220038414,
+    1.8541749119758606,
+    2.41263547539711,
+    2.037635773420334,
+    2.0588943660259247,
+    1.969245046377182,
+    1.7285726368427277,
+    1.9336746335029602,
+    2.1948340237140656,
+    1.4296438321471214,
+    1.8817992806434631,
+    1.8943344950675964,
+    1.779104083776474,
+    1.9108014702796936,
+    1.805834949016571,
+    1.834848165512085,
+    1.9906627535820007,
+    2.1963523626327515,
+    2.1539515554904938,
+    1.9684762954711914,
+    1.910222202539444,
+    2.019984722137451,
+    1.9977181553840637,
+    1.6560729146003723,
+    1.9717278629541397,
+    1.8783657252788544,
+    1.8279190361499786,
+    2.0361652076244354,
+    1.8888441324234009,
+    2.0143093168735504,
+    1.7985135614871979,
+    2.2679185271263123,
+    1.9247187972068787,
+    1.7891883850097656,
+    2.112279176712036,
+    2.047613203525543,
+    2.1257645189762115,
+    2.0965375006198883,
+    1.8606078326702118,
+    1.986148178577423,
+    2.235414505004883,
+    2.0115409195423126,
+    1.8105634152889252,
+    2.055979162454605,
+    1.8827663362026215,
+    2.228433668613434,
+    2.168084293603897,
+    1.9350906312465668,
+    2.1963738203048706,
+    1.9545983374118805,
+    2.0042108595371246,
+    2.076897233724594,
+    2.164249151945114,
+    2.2511336505413055,
+    2.147927701473236,
+    1.666139394044876,
+    1.9886944591999054,
+    1.8915810883045197,
+    1.9184699952602386,
+    1.9324964582920074,
+    2.2565895915031433,
+    1.8644168674945831,
+    2.024057060480118,
+    2.1561132073402405,
+    1.831732153892517,
+    2.0300653278827667,
+    1.8685192167758942,
+    1.9508120715618134,
+    2.217629134654999,
+    1.7884540557861328,
+    1.9629022479057312,
+    2.014945089817047,
+    1.773240178823471,
+    1.8480553328990936,
+    2.174328625202179,
+    2.0528362691402435,
+    1.9184516966342926,
+    1.9994118809700012,
+    2.1629433631896973,
+    1.956313133239746,
+    1.4914195835590363,
+    2.0376154482364655,
+    1.9383164942264557,
+    1.8460872173309326,
+    2.1424327194690704,
+    2.047647774219513,
+    2.094540774822235,
+    1.3891073316335678,
+    1.9750277996063232,
+    2.0804302990436554,
+    1.9623984098434448,
+    1.7242400497198105,
+    1.6598125100135803,
+    2.072585940361023,
+    2.003816694021225,
+    1.7694512009620667,
+    2.2525570392608643,
+    1.8560697436332703,
+    2.7015225291252136,
+    2.017684131860733,
+    1.930792212486267,
+    1.9603004455566406,
+    2.167401760816574,
+    2.0757872462272644,
+    2.1265901923179626,
+    2.067963659763336,
+    1.9692361950874329,
+    2.0600834786891937,
+    2.0879141092300415,
+    1.8167440295219421,
+    1.657739669084549,
+    1.9096021354198456,
+    1.9149316847324371,
+    1.9598571062088013,
+    2.0564489662647247,
+    2.0975446701049805,
+    2.1563239097595215,
+    1.96071657538414,
+    1.478960782289505,
+    1.9158387184143066,
+    1.7507797181606293,
+    2.0135932564735413,
+    2.245123326778412,
+    2.182836413383484,
+    1.9111057817935944,
+    1.8482527136802673,
+    1.805978387594223,
+    2.0817373394966125,
+    1.896893858909607,
+    1.9579625725746155,
+    2.200235515832901,
+    2.0788964927196503,
+    1.858733743429184,
+    1.91632080078125,
+    1.9030437469482422,
+    1.8899632692337036,
+    1.7974070012569427,
+    1.7762640118598938,
+    1.4867072105407715,
+    1.8656535148620605,
+    2.1225537061691284,
+    1.9701803922653198,
+    1.8501644432544708,
+    1.847507894039154,
+    1.992827296257019,
+    1.8754442632198334,
+    1.9105960130691528,
+    1.7681310772895813,
+    1.8711368143558502,
+    1.9765931069850922,
+    1.879693478345871,
+    1.5103979259729385,
+    2.032119333744049,
+    2.084160417318344,
+    1.6835585832595825,
+    1.6743774712085724,
+    2.1584636867046356,
+    1.8118220269680023,
+    1.9816903471946716,
+    2.029897838830948,
+    2.163084864616394,
+    1.906882494688034,
+    2.3502153158187866,
+    1.755303293466568,
+    2.171054571866989,
+    1.9884036481380463,
+    1.8938909471035004,
+    2.2752005457878113,
+    1.8866967856884003,
+    1.8928779065608978,
+    2.0702185928821564,
+    1.9415528178215027,
+    2.025204062461853,
+    2.156549721956253,
+    1.763477623462677,
+    2.2139832973480225,
+    1.6408961713314056,
+    1.7618666142225266,
+    1.8543719351291656,
+    1.7970535457134247,
+    2.226055681705475,
+    1.944146305322647,
+    1.9856219589710236,
+    2.003736823797226,
+    1.9503918588161469,
+    1.7600468397140503,
+    2.275264799594879,
+    2.0795653760433197,
+    1.490852490067482,
+    1.8964970707893372,
+    1.9750999510288239,
+    2.043429046869278,
+    2.2627260088920593,
+    1.9151235222816467,
+    2.0287132263183594,
+    1.90212082862854,
+    1.7960082292556763,
+    1.94166761636734,
+    1.9386018514633179,
+    2.134494125843048,
+    2.0087448060512543,
+    2.0281916856765747,
+    1.9535616338253021,
+    1.9679738581180573,
+    1.596398025751114,
+    2.191830277442932,
+    2.0763327181339264,
+    2.1266538500785828,
+    2.274190306663513,
+    1.8964255154132843,
+    2.1241783499717712,
+    1.9198937714099884,
+    2.3034598231315613,
+    1.9052272140979767,
+    1.672550529241562,
+    2.046636402606964,
+    1.9329883009195328,
+    2.1311089992523193,
+    2.2328023314476013,
+    1.9483379125595093,
+    1.8120945990085602,
+    1.8161226511001587,
+    1.7844904065132141,
+    1.8877747356891632,
+    2.010182023048401,
+    1.9179834425449371,
+    2.0235649049282074,
+    2.120028495788574,
+    2.414934992790222,
+    1.8385069966316223,
+    1.8532257974147797,
+    2.0549443662166595,
+    1.9257167875766754,
+    1.7838368117809296,
+    1.9770499169826508,
+    2.02278470993042,
+    2.018839567899704,
+    1.8134825825691223,
+    1.912839412689209,
+    2.1395110189914703,
+    2.238103538751602,
+    1.6793973743915558,
+    2.275532752275467,
+    2.087612211704254,
+    1.8759168982505798,
+    1.6412090212106705,
+    1.9292035102844238,
+    2.205808609724045,
+    1.998661369085312,
+    2.052595764398575,
+    2.0764198303222656,
+    2.2456181049346924,
+    2.0933930575847626,
+    1.874241203069687,
+    1.7711814045906067,
+    1.5416613817214966,
+    1.7534068077802658,
+    2.0805560648441315,
+    1.6377001479268074,
+    1.884499341249466,
+    2.143370270729065,
+    2.110937684774399,
+    1.977731168270111,
+    1.97747004032135,
+    1.7653805166482925,
+    1.843018800020218,
+    2.2338719964027405,
+    2.2774742245674133,
+    2.079852908849716,
+    2.0538104474544525,
+    2.063538581132889,
+    1.7676877677440643,
+    2.0208137929439545,
+    1.9883665442466736,
+    2.0645232796669006,
+    1.9443430304527283,
+    1.7410898208618164,
+    2.0080275535583496,
+    1.9868277311325073,
+    2.078660637140274,
+    2.2050068378448486,
+    1.9919089078903198,
+    2.252542018890381,
+    1.9415084719657898,
+    1.8200271725654602,
+    1.9846361577510834,
+    1.9353410005569458,
+    1.912458896636963,
+    2.0796336233615875,
+    2.1193440556526184,
+    1.8158704936504364,
+    2.223644942045212,
+    2.0725834667682648,
+    2.0095846951007843,
+    2.1535970866680145,
+    2.0234865248203278,
+    1.796433001756668,
+    1.9197930991649628,
+    2.022257298231125,
+    1.6413149237632751,
+    1.9616879522800446,
+    1.9451987147331238,
+    2.2428542971611023,
+    2.030157655477524,
+    1.5762348473072052,
+    1.9567807912826538,
+    2.0858742594718933,
+    2.04095658659935,
+    2.1477834284305573,
+    1.857110321521759,
+    1.9249985218048096,
+    2.067128211259842,
+    1.8338475227355957,
+    1.8183660209178925,
+    1.7897652685642242,
+    2.016318142414093,
+    1.9572072327136993,
+    1.972920686006546,
+    2.010816305875778,
+    1.5412680506706238,
+    1.9066489338874817,
+    2.0087280571460724,
+    1.7885839641094208,
+    2.036726325750351,
+    1.8658419251441956,
+    1.9655573070049286,
+    2.1818976402282715,
+    1.7045586109161377,
+    2.3182644844055176,
+    1.8516510128974915,
+    1.766547679901123,
+    1.942388653755188,
+    2.11874783039093,
+    1.8681564033031464,
+    1.6302369683980942,
+    1.8748173117637634,
+    2.116931974887848,
+    2.1928129494190216,
+    1.8579590022563934,
+    2.000575751066208,
+    2.3248833417892456,
+    2.0099377036094666,
+    1.9941376149654388,
+    1.9139507114887238,
+    1.798469752073288,
+    2.045513093471527,
+    2.139172524213791,
+    2.0657175183296204,
+    2.5929186940193176,
+    1.887778788805008,
+    2.0180120766162872,
+    1.923815667629242,
+    2.1513286232948303,
+    2.0689505338668823,
+    2.0100492537021637,
+    2.10652357339859,
+    1.8175769448280334,
+    1.932394802570343,
+    1.8030853569507599,
+    2.1482163667678833,
+    1.8957428932189941,
+    2.1001981496810913,
+    1.8274339735507965,
+    1.755200743675232,
+    1.9862683713436127,
+    2.270208418369293,
+    1.8101052641868591,
+    1.9821595251560211,
+    1.5272532850503922,
+    2.014067441225052,
+    1.9742996096611023,
+    1.9847474098205566,
+    2.2911656200885773,
+    2.149272084236145,
+    2.224188506603241,
+    2.1615083813667297,
+    1.7423981428146362,
+    1.7683758437633514,
+    2.028497099876404,
+    1.7579623758792877,
+    1.9390694499015808,
+    2.1545024514198303,
+    2.048369884490967,
+    1.9410778284072876,
+    2.0469002723693848,
+    1.939433366060257,
+    1.8914868533611298,
+    2.118733376264572,
+    2.0376307666301727,
+    1.7661086916923523,
+    2.0115101635456085,
+    2.035685360431671,
+    1.9506032168865204,
+    1.6531559526920319,
+    2.022988259792328,
+    2.2393264770507812,
+    2.0491443276405334,
+    1.9842473566532135,
+    1.9727666676044464,
+    2.22902449965477,
+    1.8579447865486145,
+    2.0388655066490173,
+    2.01955908536911,
+    1.9977469444274902,
+    2.1867045164108276,
+    1.6018355786800385,
+    1.900251418352127,
+    2.2395838499069214,
+    1.797933965921402,
+    2.1339435279369354,
+    1.9796839654445648,
+    2.0070094764232635,
+    2.068116545677185,
+    2.143452674150467,
+    2.089024990797043,
+    2.096229612827301,
+    2.013333708047867,
+    1.9395509362220764,
+    1.7489879727363586,
+    2.155642122030258,
+    1.687512069940567,
+    2.1562404334545135,
+    2.119892954826355,
+    2.0649279952049255,
+    1.9891228377819061,
+    1.8696045875549316,
+    1.8906341791152954,
+    1.3779398649930954,
+    1.930278092622757,
+    1.9531156122684479,
+    1.7971335351467133,
+    1.9451295733451843,
+    1.9390936195850372,
+    2.0392781794071198,
+    2.332154870033264,
+    1.9894782602787018,
+    2.0120878517627716,
+    1.7841753661632538,
+    2.145608127117157,
+    1.8821957111358643,
+    1.9437867999076843,
+    1.8886327743530273,
+    1.6309072077274323,
+    1.7295852303504944,
+    2.2243327498435974,
+    1.8572883307933807,
+    2.025457978248596,
+    1.7401179671287537,
+    1.2589047253131866,
+    2.216827839612961,
+    1.7800424695014954,
+    2.0404772460460663,
+    1.883945882320404,
+    2.0013100802898407,
+    2.1914494931697845,
+    2.097546637058258,
+    2.110943377017975,
+    2.328992247581482,
+    2.1373319923877716,
+    2.1446226835250854,
+    1.8427353352308273,
+    1.8370108008384705,
+    1.9873742759227753,
+    1.9717270731925964,
+    1.6525997519493103,
+    1.8598004281520844,
+    1.9432488083839417,
+    2.1452393233776093,
+    1.902628868818283,
+    1.6516613364219666,
+    1.9785824716091156,
+    1.9252119362354279,
+    1.7771514356136322,
+    2.1968206465244293,
+    1.9527567625045776,
+    2.0789993703365326,
+    1.841865748167038,
+    1.8205523192882538,
+    1.8681734204292297,
+    1.7501431107521057,
+    2.1056064069271088,
+    1.8949854373931885,
+    1.9272009134292603,
+    1.8338072001934052,
+    1.8234700560569763,
+    2.0019575655460358,
+    1.5818779952824116,
+    2.024990826845169,
+    2.207423210144043,
+    1.7634362280368805,
+    1.8431354463100433,
+    1.984557956457138,
+    1.8526888191699982,
+    1.794599860906601,
+    1.6078735142946243,
+    1.8850816488265991,
+    1.9796228408813477,
+    2.2156145572662354,
+    2.195041835308075,
+    2.1863420605659485,
+    1.8231058716773987,
+    1.9898098707199097,
+    2.0260429978370667,
+    2.074339270591736,
+    1.288858488202095,
+    1.9821405112743378,
+    2.034531831741333,
+    2.0912082493305206,
+    2.1728965044021606,
+    1.7105523645877838,
+    2.0361987948417664,
+    2.019710510969162,
+    1.9856821298599243,
+    1.8924728333950043,
+    1.8184559047222137,
+    2.0388779640197754,
+    2.0392623245716095,
+    2.161120504140854,
+    1.7053113281726837,
+    1.9326909482479095,
+    2.131219744682312,
+    2.0349932610988617,
+    1.9700875282287598,
+    2.198672741651535,
+    1.7098436057567596,
+    1.4670239239931107,
+    1.8805603981018066,
+    2.08471417427063,
+    2.0385683178901672,
+    2.1507520973682404,
+    2.1592154502868652,
+    1.758884608745575,
+    1.8553593754768372,
+    2.205213576555252,
+    1.8334356546401978,
+    2.0568824410438538,
+    1.9245964884757996,
+    1.812854528427124,
+    2.0937885642051697,
+    2.075963944196701,
+    2.0015837252140045,
+    1.9177868962287903,
+    2.0384775698184967,
+    2.113261580467224,
+    1.8370648920536041,
+    1.9324750304222107,
+    1.7321276366710663,
+    2.1536856293678284,
+    2.1578134894371033,
+    1.9289006888866425,
+    1.844500482082367,
+    2.3947205543518066,
+    1.9309192895889282,
+    1.958142876625061,
+    1.963348239660263,
+    2.1394614577293396,
+    2.2124475836753845,
+    1.8503296673297882,
+    2.187890976667404,
+    1.9677619636058807,
+    2.2120757699012756,
+    2.331635594367981,
+    1.904624581336975,
+    2.202490597963333,
+    1.8061237633228302,
+    1.919066458940506,
+    2.0077410340309143,
+    1.8513532876968384,
+    2.071906268596649,
+    1.8770314753055573,
+    1.9955376088619232,
+    2.107666552066803,
+    2.1590643525123596,
+    2.2253186106681824,
+    2.4541269540786743,
+    2.217774510383606,
+    1.3695485442876816,
+    2.1457070410251617,
+    1.7335527837276459,
+    1.5995305925607681,
+    2.0256753265857697,
+    1.9565629065036774,
+    1.791105717420578,
+    2.0784226655960083,
+    2.1166191399097443,
+    1.9152996242046356,
+    1.8362950682640076,
+    1.7731340825557709,
+    2.155468314886093,
+    2.1131235361099243,
+    2.3282033801078796,
+    1.9125248193740845,
+    1.8689576089382172,
+    1.9485906958580017,
+    1.932999312877655,
+    2.4048546850681305,
+    1.9275791645050049,
+    1.9732040166854858,
+    1.8517394661903381,
+    1.9014380425214767,
+    1.882369488477707,
+    2.1833301186561584,
+    2.1289570331573486,
+    2.183875024318695,
+    2.171056389808655,
+    2.002423048019409,
+    1.9248660802841187,
+    2.0061028003692627,
+    2.028146803379059,
+    2.421733796596527,
+    1.8837913572788239,
+    1.9675268530845642,
+    2.374457895755768,
+    1.7807093858718872,
+    2.172850728034973,
+    2.153034210205078,
+    2.0617380142211914,
+    1.8835320174694061,
+    2.0785669684410095,
+    2.3138357996940613,
+    2.1923963725566864,
+    1.869694173336029,
+    2.0251356065273285,
+    2.1220491230487823,
+    2.0145900547504425,
+    2.1503432393074036,
+    2.190626412630081,
+    2.2802276611328125,
+    2.236900269985199,
+    1.9811552166938782,
+    2.092166304588318,
+    2.07303449511528,
+    1.6356765925884247,
+    2.077802836894989,
+    1.9610523879528046,
+    1.8983043730258942,
+    1.7379110455513,
+    2.0595612823963165,
+    1.7658316493034363,
+    2.0359201431274414,
+    2.078512877225876,
+    2.1333895325660706,
+    1.9176843166351318,
+    2.070345878601074,
+    2.202768564224243,
+    1.945077806711197,
+    1.6647659242153168,
+    1.449508160352707,
+    2.1967802345752716,
+    2.186197817325592,
+    1.8318515717983246,
+    1.9661785066127777,
+    2.1331526339054108,
+    1.9662226438522339,
+    1.9168857634067535,
+    1.9672299921512604,
+    2.0793337225914,
+    1.742641642689705,
+    1.8899154961109161,
+    1.9683457612991333,
+    2.1284988820552826,
+    1.8823730945587158,
+    1.9671491086483002,
+    1.7891699075698853,
+    2.0366631746292114,
+    2.170773684978485,
+    2.157770574092865,
+    1.531184360384941,
+    1.7942973375320435,
+    1.9538165926933289,
+    1.7740932703018188,
+    1.8444747924804688,
+    2.1593389213085175,
+    2.0222666561603546,
+    2.153877913951874,
+    2.1281114518642426,
+    1.8055661916732788,
+    1.8112592101097107,
+    2.1732959151268005,
+    2.2776984870433807,
+    2.01517316699028,
+    2.048113375902176,
+    2.3082817792892456,
+    1.9476797580718994,
+    1.9273951947689056,
+    2.175132006406784,
+    2.0655637979507446,
+    2.0974556505680084,
+    1.983421117067337,
+    2.0001393258571625,
+    1.8526580929756165,
+    1.875769704580307,
+    1.919754534959793,
+    2.061443716287613,
+    2.3129413425922394,
+    1.9128734767436981,
+    2.015645980834961,
+    2.237905263900757,
+    2.1461544036865234,
+    1.9235659837722778,
+    2.173466682434082,
+    1.6294988542795181,
+    1.895463079214096,
+    1.7323124408721924,
+    1.5954849123954773,
+    1.7995491325855255,
+    1.6643447279930115,
+    2.0132358074188232,
+    1.9786936044692993,
+    1.9528062641620636,
+    2.0898461639881134,
+    2.0195729434490204,
+    2.1435398161411285,
+    1.8278231024742126,
+    1.982870191335678,
+    1.7309416234493256,
+    2.242147445678711,
+    2.108400136232376,
+    1.7300675511360168,
+    1.8113693594932556,
+    2.2310619354248047,
+    2.0328956842422485,
+    2.18557870388031,
+    2.3422034978866577,
+    2.169844090938568,
+    1.8169803321361542,
+    2.132762908935547,
+    1.7648725509643555,
+    1.9677415788173676,
+    1.8632167875766754,
+    1.8988753855228424,
+    1.8592016994953156,
+    1.8096055090427399,
+    1.8690961003303528,
+    1.8140289187431335,
+    1.876509428024292,
+    1.9361322224140167,
+    1.9273012280464172,
+    1.6636610329151154,
+    2.111586958169937,
+    2.10248139500618,
+    2.076209932565689,
+    1.8651195466518402,
+    1.7480785846710205,
+    1.7089017778635025,
+    1.8724442422389984,
+    2.272181123495102,
+    1.7920746207237244,
+    1.9623207449913025,
+    2.0105388164520264,
+    1.8501586616039276,
+    1.9171726405620575,
+    2.1572029292583466,
+    1.8143680393695831,
+    2.1485328674316406,
+    2.0317324697971344,
+    2.1904807686805725,
+    2.2118465304374695,
+    1.621018499135971,
+    2.0450506508350372,
+    2.0084542334079742,
+    1.6542732417583466,
+    1.7350412607192993,
+    2.124550372362137,
+    1.9821723401546478,
+    1.8534682095050812,
+    2.0167691111564636,
+    2.191506564617157,
+    1.853075534105301,
+    2.064965844154358,
+    2.133788377046585,
+    1.9932806193828583,
+    2.0305879712104797,
+    2.002735823392868,
+    2.17588073015213,
+    1.9466276466846466,
+    1.9742610156536102,
+    2.0453275442123413,
+    2.1021065413951874,
+    1.926622450351715,
+    1.9230711162090302,
+    1.974651575088501,
+    2.1180694103240967,
+    2.0189504623413086,
+    1.9734396636486053,
+    2.0832147002220154,
+    2.1344321370124817,
+    1.9121268093585968,
+    1.7806911766529083,
+    1.894917517900467,
+    2.442289859056473,
+    2.151290535926819,
+    1.8238812685012817,
+    2.1155946850776672,
+    1.9933439195156097,
+    1.9308890998363495,
+    1.8547934293746948,
+    1.7293584942817688,
+    1.8168570399284363,
+    1.9901467859745026,
+    2.0452483892440796,
+    1.9110192954540253,
+    2.010503828525543,
+    2.047707736492157,
+    1.8187103271484375,
+    1.774341732263565,
+    2.115667313337326,
+    2.1247178316116333,
+    2.0494262874126434,
+    2.300173819065094,
+    2.0279658436775208,
+    1.9435496628284454,
+    2.056148022413254,
+    1.9430205821990967,
+    1.9995667338371277,
+    1.5916635990142822,
+    2.1291545033454895,
+    1.991887778043747,
+    1.7915579974651337,
+    2.3111766576766968,
+    2.0212161540985107,
+    2.000482052564621,
+    2.1014957427978516,
+    1.7770590782165527,
+    1.7969960868358612,
+    1.5803268998861313,
+    2.0647006034851074,
+    1.9547259509563446,
+    2.0696517527103424,
+    1.9910634756088257,
+    2.1530147790908813,
+    1.9543803930282593,
+    2.1136884093284607,
+    2.13692843914032,
+    1.9456982016563416,
+    2.0092910528182983,
+    1.9912080764770508,
+    2.0097333788871765,
+    2.083057552576065,
+    1.9318538010120392,
+    2.1256435215473175,
+    2.257496029138565,
+    1.8094933331012726,
+    1.4012587666511536,
+    1.8827908337116241,
+    2.1756474673748016,
+    2.06759449839592,
+    1.5783221572637558,
+    2.39899218082428,
+    2.112785190343857,
+    2.0296302437782288,
+    1.9456823468208313,
+    1.998735636472702,
+    1.7228958010673523,
+    1.8486731946468353,
+    2.2145639061927795,
+    1.7764050662517548,
+    2.197065979242325,
+    1.86238494515419,
+    1.9430935084819794,
+    1.919914335012436,
+    1.913959413766861,
+    1.9404174387454987,
+    1.7771977186203003,
+    1.7064241766929626,
+    1.862898051738739,
+    1.9373095035552979,
+    1.9098209589719772,
+    1.8045717179775238,
+    1.6182075440883636,
+    2.072701245546341,
+    1.968608945608139,
+    1.9726117551326752,
+    2.0110373198986053,
+    2.182236433029175,
+    2.140879124403,
+    1.9428103268146515,
+    2.0359572768211365,
+    2.0655657947063446,
+    2.2953418493270874,
+    2.036114752292633,
+    1.8473150730133057,
+    1.9862259030342102,
+    1.7294813990592957,
+    1.8292729258537292,
+    2.0504174530506134,
+    1.7716586291790009,
+    2.3436484932899475,
+    1.9334840178489685,
+    1.9688337743282318,
+    1.9796840250492096,
+    2.2490031719207764,
+    1.9752130806446075,
+    1.7560361623764038,
+    2.016396939754486,
+    2.108469933271408,
+    2.0847749412059784,
+    1.9710015654563904,
+    1.687826007604599,
+    1.9662368595600128,
+    1.813076376914978,
+    2.1836700439453125,
+    2.0774579644203186,
+    1.8036229610443115,
+    2.1311080157756805,
+    1.9503213465213776,
+    1.7284455001354218,
+    2.085205554962158,
+    1.9091865718364716,
+    2.197684645652771,
+    2.00957328081131,
+    1.8571208119392395,
+    1.5821105539798737,
+    1.955141693353653,
+    2.1221119463443756,
+    1.790937751531601,
+    2.030369132757187,
+    2.0114158987998962,
+    1.9399300813674927,
+    1.4476726800203323,
+    1.8313698768615723,
+    1.9050827622413635,
+    2.224960744380951,
+    1.8724946975708008,
+    2.015827775001526,
+    1.9561944007873535,
+    1.9261743128299713,
+    2.1048684120178223,
+    1.9394489526748657,
+    2.0986559689044952,
+    1.7638024985790253,
+    2.036400258541107,
+    2.1528721749782562,
+    2.073129326105118,
+    1.8411673605442047,
+    1.939580261707306,
+    1.6920580565929413,
+    2.147077739238739,
+    2.057867646217346,
+    1.9146268963813782,
+    1.854831874370575,
+    2.2440930604934692,
+    1.8057295978069305,
+    1.9425169229507446,
+    1.990231841802597,
+    1.8307946920394897,
+    2.145416170358658,
+    1.99793940782547,
+    1.935516893863678,
+    2.1256790161132812,
+    1.9603136777877808,
+    2.151856780052185,
+    1.7466541230678558,
+    2.007850080728531,
+    2.044885069131851,
+    1.9393750727176666,
+    1.7012759149074554,
+    2.1481933295726776,
+    1.9955629408359528,
+    1.930353969335556,
+    1.728232204914093,
+    2.0282465517520905,
+    1.9507566094398499,
+    1.8695492148399353,
+    2.0041505098342896,
+    1.676375225186348,
+    1.968402624130249,
+    1.8427912890911102,
+    1.5847935378551483,
+    1.9024984240531921,
+    1.8487526178359985,
+    2.3884875774383545,
+    1.6445600986480713,
+    2.0323259830474854,
+    1.9654830396175385,
+    1.7109937965869904,
+    2.042092353105545,
+    1.805917888879776,
+    2.049924910068512,
+    1.8928887248039246,
+    1.9888927936553955,
+    1.9198053479194641,
+    1.9256122708320618,
+    1.8431824743747711,
+    1.7066606283187866,
+    2.032832980155945,
+    2.1495854258537292,
+    2.0490241646766663,
+    1.860532820224762,
+    1.753870278596878,
+    2.1220842003822327,
+    2.162771910429001,
+    2.1290297508239746,
+    2.078712522983551,
+    2.0653501749038696,
+    1.9271245002746582,
+    1.8236683309078217,
+    2.074468970298767,
+    2.0969933569431305,
+    1.7224549651145935,
+    1.800935447216034,
+    1.7939541637897491,
+    1.9220082461833954,
+    2.1352640986442566,
+    2.1541234850883484,
+    1.7974610328674316,
+    1.981817603111267,
+    2.2368058264255524,
+    1.9721477627754211,
+    1.9194137752056122,
+    1.9907901585102081,
+    1.8740435242652893,
+    1.9384220242500305,
+    1.9559401869773865,
+    2.055519014596939,
+    2.1836227476596832,
+    1.811684250831604,
+    2.0930145382881165,
+    2.146706461906433,
+    1.976416289806366,
+    2.1783730685710907,
+    1.946544885635376,
+    1.8253210186958313,
+    1.8091734051704407,
+    2.071792960166931,
+    1.8575608134269714,
+    2.113013833761215,
+    1.906772643327713,
+    1.6668291985988617,
+    2.128485381603241,
+    1.8696006536483765,
+    1.8044472634792328,
+    1.9519094824790955,
+    2.0555010437965393,
+    1.985167235136032,
+    1.8908187448978424,
+    1.9573894739151,
+    1.7676569819450378,
+    1.8598586320877075,
+    2.2113934755325317,
+    1.925371676683426,
+    1.8920843601226807,
+    1.9883770048618317,
+    1.8450222313404083,
+    1.8073875904083252,
+    2.0803558826446533,
+    1.8198177218437195,
+    1.825066089630127,
+    1.8360100090503693,
+    2.1978712677955627,
+    1.9036785662174225,
+    2.141344964504242,
+    1.9967727065086365,
+    2.2903705835342407,
+    1.949187994003296,
+    2.0696626007556915,
+    1.9963598549365997,
+    1.6952697336673737,
+    2.1226349771022797,
+    1.8703835606575012,
+    1.2933985143899918,
+    2.2219377756118774,
+    1.8957476913928986,
+    2.269321084022522,
+    2.007890820503235,
+    2.083814024925232,
+    1.9622712135314941,
+    1.7073319852352142,
+    1.847601592540741,
+    2.053486615419388,
+    2.1813484132289886,
+    1.890440970659256,
+    2.027731269598007,
+    1.6174477934837341,
+    1.7919402718544006,
+    1.9463393688201904,
+    1.9896308481693268,
+    2.0289697349071503,
+    1.6240250766277313,
+    1.8370580971240997,
+    2.143990010023117,
+    2.034472703933716,
+    2.006387770175934,
+    2.0914021134376526,
+    2.0233986377716064,
+    2.0465296804904938,
+    1.8912285566329956,
+    1.4061703383922577,
+    1.6956148147583008,
+    1.9879199862480164,
+    2.026995986700058,
+    2.1163467168807983,
+    1.745167225599289,
+    2.064620405435562,
+    2.0390766859054565,
+    1.6615254580974579,
+    1.9635947942733765,
+    1.8338792622089386,
+    2.0168954133987427,
+    2.2045645117759705,
+    2.037251979112625,
+    2.0321780145168304,
+    1.779864877462387,
+    2.1200835406780243,
+    1.965651959180832,
+    1.7456597089767456,
+    1.8997738659381866,
+    1.971346378326416,
+    1.9074249267578125,
+    2.1454968452453613,
+    2.22280752658844,
+    2.430083930492401,
+    2.141452819108963,
+    1.64227962680161,
+    1.9994220733642578,
+    2.036228984594345,
+    2.0477948784828186,
+    1.9701568186283112,
+    1.8482519686222076,
+    1.8488691449165344,
+    1.9171075522899628,
+    1.9093800485134125,
+    2.023425042629242,
+    1.8402385413646698,
+    2.038976401090622,
+    2.0120771527290344,
+    2.151847720146179,
+    2.0497776865959167,
+    1.9586480557918549,
+    2.0237147212028503,
+    1.9801948070526123,
+    2.0151692628860474,
+    1.8029577434062958,
+    2.0051112473011017,
+    1.8663323819637299,
+    2.0179233849048615,
+    1.7519203424453735,
+    1.840652197599411,
+    1.646545797586441,
+    1.9958685636520386,
+    1.9217908382415771,
+    1.8133135735988617,
+    2.100397229194641,
+    1.827485352754593,
+    2.1830170452594757,
+    2.0946406424045563,
+    1.9813707768917084,
+    2.005462735891342,
+    2.0008463859558105,
+    2.1075166165828705,
+    1.8437193930149078,
+    1.940528154373169,
+    2.0984645187854767,
+    2.119257003068924,
+    1.977652668952942,
+    1.485251858830452,
+    2.018624782562256,
+    1.9600473046302795,
+    2.0482820868492126,
+    1.803219497203827,
+    1.9399366080760956,
+    2.2106973826885223,
+    2.0708835124969482,
+    1.772843986749649,
+    2.2345972061157227,
+    2.0929279029369354,
+    2.1459622979164124,
+    1.668130785226822,
+    2.062741607427597,
+    2.1128511130809784,
+    1.889920860528946,
+    2.0712372064590454,
+    1.812429517507553,
+    2.0796378552913666,
+    2.1831634044647217,
+    2.0532161593437195,
+    1.98189777135849,
+    1.5960696637630463,
+    2.0842748880386353,
+    1.93175008893013,
+    2.035250961780548,
+    1.7772827744483948,
+    2.102186381816864,
+    1.936110019683838,
+    2.0190663039684296,
+    2.0467732548713684,
+    1.8976051807403564,
+    1.7496248185634613,
+    1.9593359231948853,
+    1.9533615708351135,
+    2.2826983630657196,
+    1.9272327721118927,
+    1.9620782434940338,
+    1.8481930196285248,
+    1.8081250488758087,
+    1.9669547975063324,
+    1.5353659391403198,
+    1.954338788986206,
+    1.964853674173355,
+    1.738270342350006,
+    1.7745414972305298,
+    2.108994275331497,
+    1.8138256967067719,
+    1.8150640279054642,
+    1.9487068355083466,
+    1.7879611253738403,
+    2.049844413995743,
+    1.9595679342746735,
+    1.9147726893424988
+  ],
+  "dataset_offset": 0,
+  "config": {
+    "king_dir": "/root/king_165_hf",
+    "optimizer": "adamw",
+    "lr": 5e-05,
+    "muon_lr": 0.02,
+    "lora_r": 256,
+    "steps": 5000,
+    "grad_accum": 4,
+    "seq_len": 2048,
+    "unfreeze_head": false,
+    "source_dirs": [
+      "/root/datasets/finewebedu"
+    ],
+    "source_weights": [
+      1.0
+    ]
+  }
+}

best_eval_2/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: ''
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- 'base_model:adapter:'
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

best_eval_2/adapter/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 512,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 256,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "gate",
+    "q_proj",
+    "up",
+    "v_proj",
+    "down",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

best_eval_2/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b981e29791f8a3b09e961bf9568346991768eeed4f368c35d6dadca505e5d877
+size 155198248

best_eval_2/cuda_rng.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfa47d762301992efbdc1b1bd617b22937e18fd1d502354481860eea9ba3be63
+size 1584

best_eval_2/numpy_rng.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bd2bef57f4304c9f2ecf89022498801b55e0cf339fd3b7ec48666907b8ee80a
+size 2675

best_eval_2/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7011ddfc77a1d8a50d54740d7f666b7efa45a571b42109f33728f584ef6e163
+size 310433291

best_eval_2/python_rng.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8648e962b57e6a2136a6fba8892cfc7d625d7a3b4ed99f5561934fc7725f0ab6
+size 3762

best_eval_2/torch_rng.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ef73e8425e75b9b9ed7f1ddc0e8f1658494f12bbec61c3ee76bac7dfaae5c05
+size 6647

best_eval_2/train_state.json

@@ -0,0 +1,124 @@
+{
+  "step": 101,
+  "losses": [
+    2.1343601644039154,
+    2.072067230939865,
+    1.8817282915115356,
+    1.5482478141784668,
+    2.0044714510440826,
+    1.9346624314785004,
+    2.0394379794597626,
+    2.0922091901302338,
+    1.9468883574008942,
+    1.7588119804859161,
+    1.874794840812683,
+    1.911599040031433,
+    1.6168376505374908,
+    1.7635800540447235,
+    2.070902556180954,
+    1.7358599305152893,
+    2.070700913667679,
+    1.9784598052501678,
+    1.9782940447330475,
+    1.8401454091072083,
+    1.6551836282014847,
+    1.9741482436656952,
+    1.990296185016632,
+    1.8002875596284866,
+    2.04819917678833,
+    1.8555984199047089,
+    2.1058743596076965,
+    1.9181069433689117,
+    2.1770740747451782,
+    2.1200993955135345,
+    1.5497342869639397,
+    2.041239619255066,
+    1.7999589145183563,
+    1.8640441000461578,
+    1.8068368434906006,
+    1.958029717206955,
+    2.3048327565193176,
+    2.0624635815620422,
+    1.7916592359542847,
+    2.197334885597229,
+    1.8518704771995544,
+    2.014060288667679,
+    2.3126999735832214,
+    2.0366491973400116,
+    2.1218522787094116,
+    1.9050887823104858,
+    2.140252709388733,
+    2.016432911157608,
+    2.0771953761577606,
+    1.99711474776268,
+    2.2010857462882996,
+    1.906881034374237,
+    1.9355229437351227,
+    1.568751234561205,
+    2.218832314014435,
+    2.282701849937439,
+    2.1470184922218323,
+    1.8618396520614624,
+    2.0435074865818024,
+    2.2046265304088593,
+    2.036127895116806,
+    1.9144421517848969,
+    1.7361707091331482,
+    1.8860499560832977,
+    1.9349249005317688,
+    2.091381222009659,
+    2.2041552662849426,
+    2.0161964893341064,
+    1.6289199590682983,
+    1.7889264225959778,
+    2.021867960691452,
+    2.1548608541488647,
+    1.7794556617736816,
+    1.9493914544582367,
+    1.8160816133022308,
+    1.8409814834594727,
+    2.2246702015399933,
+    2.080227106809616,
+    1.6188245043158531,
+    1.8832258582115173,
+    1.818663239479065,
+    2.0883366763591766,
+    2.00100439786911,
+    2.1166227757930756,
+    2.0359020829200745,
+    2.2903840243816376,
+    2.1877321600914,
+    1.8351092636585236,
+    2.1081435680389404,
+    1.876007080078125,
+    1.6549921333789825,
+    2.0958774387836456,
+    1.93923220038414,
+    1.8541749119758606,
+    2.41263547539711,
+    2.037635773420334,
+    2.0588943660259247,
+    1.969245046377182,
+    1.7285726368427277,
+    1.9336746335029602,
+    2.1948340237140656
+  ],
+  "dataset_offset": 0,
+  "config": {
+    "king_dir": "/root/king_165_hf",
+    "optimizer": "adamw",
+    "lr": 5e-05,
+    "muon_lr": 0.02,
+    "lora_r": 256,
+    "steps": 5000,
+    "grad_accum": 4,
+    "seq_len": 2048,
+    "unfreeze_head": false,
+    "source_dirs": [
+      "/root/datasets/finewebedu"
+    ],
+    "source_weights": [
+      1.0
+    ]
+  }
+}

best_eval_3/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: ''
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- 'base_model:adapter:'
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

best_eval_3/adapter/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 512,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 256,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "gate",
+    "q_proj",
+    "up",
+    "v_proj",
+    "down",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

best_eval_3/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa5894c322d1be93bcf2a48cf5ca57ec429ebb8dbc5b0b363b4e6222660aa09c
+size 155198248

best_eval_3/cuda_rng.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfa47d762301992efbdc1b1bd617b22937e18fd1d502354481860eea9ba3be63
+size 1584

best_eval_3/numpy_rng.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bd2bef57f4304c9f2ecf89022498801b55e0cf339fd3b7ec48666907b8ee80a
+size 2675

best_eval_3/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:314c9fec6d9230258165ccfc5589d7bfe405dfa57e63f14e6bd624198786aed0
+size 310433291

best_eval_3/python_rng.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8648e962b57e6a2136a6fba8892cfc7d625d7a3b4ed99f5561934fc7725f0ab6
+size 3762

best_eval_3/torch_rng.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ef73e8425e75b9b9ed7f1ddc0e8f1658494f12bbec61c3ee76bac7dfaae5c05
+size 6647

best_eval_3/train_state.json

@@ -0,0 +1,224 @@
+{
+  "step": 201,
+  "losses": [
+    2.1343601644039154,
+    2.072067230939865,
+    1.8817282915115356,
+    1.5482478141784668,
+    2.0044714510440826,
+    1.9346624314785004,
+    2.0394379794597626,
+    2.0922091901302338,
+    1.9468883574008942,
+    1.7588119804859161,
+    1.874794840812683,
+    1.911599040031433,
+    1.6168376505374908,
+    1.7635800540447235,
+    2.070902556180954,
+    1.7358599305152893,
+    2.070700913667679,
+    1.9784598052501678,
+    1.9782940447330475,
+    1.8401454091072083,
+    1.6551836282014847,
+    1.9741482436656952,
+    1.990296185016632,
+    1.8002875596284866,
+    2.04819917678833,
+    1.8555984199047089,
+    2.1058743596076965,
+    1.9181069433689117,
+    2.1770740747451782,
+    2.1200993955135345,
+    1.5497342869639397,
+    2.041239619255066,
+    1.7999589145183563,
+    1.8640441000461578,
+    1.8068368434906006,
+    1.958029717206955,
+    2.3048327565193176,
+    2.0624635815620422,
+    1.7916592359542847,
+    2.197334885597229,
+    1.8518704771995544,
+    2.014060288667679,
+    2.3126999735832214,
+    2.0366491973400116,
+    2.1218522787094116,
+    1.9050887823104858,
+    2.140252709388733,
+    2.016432911157608,
+    2.0771953761577606,
+    1.99711474776268,
+    2.2010857462882996,
+    1.906881034374237,
+    1.9355229437351227,
+    1.568751234561205,
+    2.218832314014435,
+    2.282701849937439,
+    2.1470184922218323,
+    1.8618396520614624,
+    2.0435074865818024,
+    2.2046265304088593,
+    2.036127895116806,
+    1.9144421517848969,
+    1.7361707091331482,
+    1.8860499560832977,
+    1.9349249005317688,
+    2.091381222009659,
+    2.2041552662849426,
+    2.0161964893341064,
+    1.6289199590682983,
+    1.7889264225959778,
+    2.021867960691452,
+    2.1548608541488647,
+    1.7794556617736816,
+    1.9493914544582367,
+    1.8160816133022308,
+    1.8409814834594727,
+    2.2246702015399933,
+    2.080227106809616,
+    1.6188245043158531,
+    1.8832258582115173,
+    1.818663239479065,
+    2.0883366763591766,
+    2.00100439786911,
+    2.1166227757930756,
+    2.0359020829200745,
+    2.2903840243816376,
+    2.1877321600914,
+    1.8351092636585236,
+    2.1081435680389404,
+    1.876007080078125,
+    1.6549921333789825,
+    2.0958774387836456,
+    1.93923220038414,
+    1.8541749119758606,
+    2.41263547539711,
+    2.037635773420334,
+    2.0588943660259247,
+    1.969245046377182,
+    1.7285726368427277,
+    1.9336746335029602,
+    2.1948340237140656,
+    1.4296438321471214,
+    1.8817992806434631,
+    1.8943344950675964,
+    1.779104083776474,
+    1.9108014702796936,
+    1.805834949016571,
+    1.834848165512085,
+    1.9906627535820007,
+    2.1963523626327515,
+    2.1539515554904938,
+    1.9684762954711914,
+    1.910222202539444,
+    2.019984722137451,
+    1.9977181553840637,
+    1.6560729146003723,
+    1.9717278629541397,
+    1.8783657252788544,
+    1.8279190361499786,
+    2.0361652076244354,
+    1.8888441324234009,
+    2.0143093168735504,
+    1.7985135614871979,
+    2.2679185271263123,
+    1.9247187972068787,
+    1.7891883850097656,
+    2.112279176712036,
+    2.047613203525543,
+    2.1257645189762115,
+    2.0965375006198883,
+    1.8606078326702118,
+    1.986148178577423,
+    2.235414505004883,
+    2.0115409195423126,
+    1.8105634152889252,
+    2.055979162454605,
+    1.8827663362026215,
+    2.228433668613434,
+    2.168084293603897,
+    1.9350906312465668,
+    2.1963738203048706,
+    1.9545983374118805,
+    2.0042108595371246,
+    2.076897233724594,
+    2.164249151945114,
+    2.2511336505413055,
+    2.147927701473236,
+    1.666139394044876,
+    1.9886944591999054,
+    1.8915810883045197,
+    1.9184699952602386,
+    1.9324964582920074,
+    2.2565895915031433,
+    1.8644168674945831,
+    2.024057060480118,
+    2.1561132073402405,
+    1.831732153892517,
+    2.0300653278827667,
+    1.8685192167758942,
+    1.9508120715618134,
+    2.217629134654999,
+    1.7884540557861328,
+    1.9629022479057312,
+    2.014945089817047,
+    1.773240178823471,
+    1.8480553328990936,
+    2.174328625202179,
+    2.0528362691402435,
+    1.9184516966342926,
+    1.9994118809700012,
+    2.1629433631896973,
+    1.956313133239746,
+    1.4914195835590363,
+    2.0376154482364655,
+    1.9383164942264557,
+    1.8460872173309326,
+    2.1424327194690704,
+    2.047647774219513,
+    2.094540774822235,
+    1.3891073316335678,
+    1.9750277996063232,
+    2.0804302990436554,
+    1.9623984098434448,
+    1.7242400497198105,
+    1.6598125100135803,
+    2.072585940361023,
+    2.003816694021225,
+    1.7694512009620667,
+    2.2525570392608643,
+    1.8560697436332703,
+    2.7015225291252136,
+    2.017684131860733,
+    1.930792212486267,
+    1.9603004455566406,
+    2.167401760816574,
+    2.0757872462272644,
+    2.1265901923179626,
+    2.067963659763336,
+    1.9692361950874329,
+    2.0600834786891937,
+    2.0879141092300415
+  ],
+  "dataset_offset": 0,
+  "config": {
+    "king_dir": "/root/king_165_hf",
+    "optimizer": "adamw",
+    "lr": 5e-05,
+    "muon_lr": 0.02,
+    "lora_r": 256,
+    "steps": 5000,
+    "grad_accum": 4,
+    "seq_len": 2048,
+    "unfreeze_head": false,
+    "source_dirs": [
+      "/root/datasets/finewebedu"
+    ],
+    "source_weights": [
+      1.0
+    ]
+  }
+}

best_eval_4/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: ''
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- 'base_model:adapter:'
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

best_eval_4/adapter/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 512,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 256,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "gate",
+    "q_proj",
+    "up",
+    "v_proj",
+    "down",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

best_eval_4/adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2da78f287c42644c5ea2def705e84192a336d76f0dc9fb02e680ea87f90caca3
+size 155198248

best_eval_4/cuda_rng.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfa47d762301992efbdc1b1bd617b22937e18fd1d502354481860eea9ba3be63
+size 1584

best_eval_4/numpy_rng.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bd2bef57f4304c9f2ecf89022498801b55e0cf339fd3b7ec48666907b8ee80a
+size 2675