Add files using upload-large-folder tool

exp2b/swap_0.3_cost_to_drop_1032/README.md

@@ -0,0 +1,155 @@
+---
+library_name: transformers
+tags:
+- generated_from_trainer
+metrics:
+- accuracy
+model-index:
+- name: exceptions_exp2_swap_0.3_cost_to_drop_1032
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/craaaa/exceptions_exp2/runs/g647l2pm)
+# exceptions_exp2_swap_0.3_cost_to_drop_1032
+
+This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 3.5839
+- Accuracy: 0.3658
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0006
+- train_batch_size: 16
+- eval_batch_size: 16
+- seed: 1032
+- gradient_accumulation_steps: 5
+- total_train_batch_size: 80
+- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.98) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 100
+- num_epochs: 50.0
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch   | Step  | Validation Loss | Accuracy |
+|:-------------:|:-------:|:-----:|:---------------:|:--------:|
+| 4.8221        | 0.2916  | 1000  | 4.7488          | 0.2547   |
+| 4.3555        | 0.5831  | 2000  | 4.2862          | 0.2988   |
+| 4.1468        | 0.8747  | 3000  | 4.1061          | 0.3138   |
+| 4.0089        | 1.1662  | 4000  | 3.9981          | 0.3239   |
+| 3.9422        | 1.4578  | 5000  | 3.9251          | 0.3304   |
+| 3.8787        | 1.7493  | 6000  | 3.8665          | 0.3358   |
+| 3.7557        | 2.0408  | 7000  | 3.8216          | 0.3401   |
+| 3.7567        | 2.3324  | 8000  | 3.7918          | 0.3430   |
+| 3.7487        | 2.6239  | 9000  | 3.7621          | 0.3459   |
+| 3.7323        | 2.9155  | 10000 | 3.7356          | 0.3482   |
+| 3.6476        | 3.2070  | 11000 | 3.7249          | 0.3500   |
+| 3.6577        | 3.4986  | 12000 | 3.7042          | 0.3520   |
+| 3.634         | 3.7901  | 13000 | 3.6858          | 0.3535   |
+| 3.5548        | 4.0816  | 14000 | 3.6784          | 0.3552   |
+| 3.5881        | 4.3732  | 15000 | 3.6678          | 0.3557   |
+| 3.5723        | 4.6648  | 16000 | 3.6532          | 0.3572   |
+| 3.6017        | 4.9563  | 17000 | 3.6420          | 0.3584   |
+| 3.5155        | 5.2478  | 18000 | 3.6461          | 0.3585   |
+| 3.5305        | 5.5394  | 19000 | 3.6327          | 0.3598   |
+| 3.5203        | 5.8310  | 20000 | 3.6215          | 0.3607   |
+| 3.4567        | 6.1225  | 21000 | 3.6236          | 0.3614   |
+| 3.4791        | 6.4140  | 22000 | 3.6179          | 0.3622   |
+| 3.4968        | 6.7056  | 23000 | 3.6079          | 0.3627   |
+| 3.4976        | 6.9971  | 24000 | 3.5998          | 0.3633   |
+| 3.4409        | 7.2886  | 25000 | 3.6082          | 0.3632   |
+| 3.4434        | 7.5802  | 26000 | 3.5985          | 0.3638   |
+| 3.479         | 7.8718  | 27000 | 3.5888          | 0.3649   |
+| 3.3946        | 8.1633  | 28000 | 3.5970          | 0.3644   |
+| 3.4239        | 8.4548  | 29000 | 3.5905          | 0.3653   |
+| 3.437         | 8.7464  | 30000 | 3.5839          | 0.3658   |
+| 3.3318        | 9.0379  | 31000 | 3.5885          | 0.3658   |
+| 3.3781        | 9.3295  | 32000 | 3.5854          | 0.3659   |
+| 3.4108        | 9.6210  | 33000 | 3.5766          | 0.3667   |
+| 3.4274        | 9.9126  | 34000 | 3.5707          | 0.3673   |
+| 3.3505        | 10.2041 | 35000 | 3.5813          | 0.3669   |
+| 3.3773        | 10.4957 | 36000 | 3.5736          | 0.3675   |
+| 3.3963        | 10.7872 | 37000 | 3.5660          | 0.3681   |
+| 3.3042        | 11.0787 | 38000 | 3.5795          | 0.3678   |
+| 3.3448        | 11.3703 | 39000 | 3.5743          | 0.3680   |
+| 3.3706        | 11.6618 | 40000 | 3.5664          | 0.3684   |
+| 3.3846        | 11.9534 | 41000 | 3.5581          | 0.3689   |
+| 3.3202        | 12.2449 | 42000 | 3.5720          | 0.3685   |
+| 3.3493        | 12.5365 | 43000 | 3.5636          | 0.3685   |
+| 3.367         | 12.8280 | 44000 | 3.5583          | 0.3696   |
+| 3.2663        | 13.1195 | 45000 | 3.5691          | 0.3691   |
+| 3.309         | 13.4111 | 46000 | 3.5634          | 0.3692   |
+| 3.3335        | 13.7027 | 47000 | 3.5546          | 0.3698   |
+| 3.3487        | 13.9942 | 48000 | 3.5496          | 0.3701   |
+| 3.2792        | 14.2857 | 49000 | 3.5649          | 0.3696   |
+| 3.3064        | 14.5773 | 50000 | 3.5573          | 0.3699   |
+| 3.3363        | 14.8689 | 51000 | 3.5480          | 0.3706   |
+| 3.2446        | 15.1604 | 52000 | 3.5664          | 0.3700   |
+| 3.2848        | 15.4519 | 53000 | 3.5603          | 0.3702   |
+| 3.3058        | 15.7435 | 54000 | 3.5453          | 0.3710   |
+| 3.2076        | 16.0350 | 55000 | 3.5576          | 0.3706   |
+| 3.2679        | 16.3265 | 56000 | 3.5547          | 0.3709   |
+| 3.2942        | 16.6181 | 57000 | 3.5536          | 0.3706   |
+| 3.3099        | 16.9097 | 58000 | 3.5449          | 0.3713   |
+| 3.2476        | 17.2012 | 59000 | 3.5615          | 0.3707   |
+| 3.2693        | 17.4927 | 60000 | 3.5523          | 0.3713   |
+| 3.2908        | 17.7843 | 61000 | 3.5449          | 0.3714   |
+| 3.205         | 18.0758 | 62000 | 3.5588          | 0.3712   |
+| 3.2554        | 18.3674 | 63000 | 3.5547          | 0.3712   |
+| 3.2703        | 18.6589 | 64000 | 3.5525          | 0.3715   |
+| 3.2909        | 18.9505 | 65000 | 3.5421          | 0.3722   |
+| 3.2095        | 19.2420 | 66000 | 3.5582          | 0.3716   |
+| 3.2457        | 19.5336 | 67000 | 3.5511          | 0.3721   |
+| 3.2676        | 19.8251 | 68000 | 3.5441          | 0.3721   |
+| 3.1825        | 20.1166 | 69000 | 3.5578          | 0.3715   |
+| 3.2339        | 20.4082 | 70000 | 3.5531          | 0.3716   |
+| 3.2581        | 20.6997 | 71000 | 3.5460          | 0.3721   |
+| 3.2604        | 20.9913 | 72000 | 3.5375          | 0.3726   |
+| 3.2032        | 21.2828 | 73000 | 3.5557          | 0.3719   |
+| 3.2375        | 21.5744 | 74000 | 3.5470          | 0.3724   |
+| 3.2491        | 21.8659 | 75000 | 3.5403          | 0.3728   |
+| 3.1692        | 22.1574 | 76000 | 3.5603          | 0.3718   |
+| 3.2165        | 22.4490 | 77000 | 3.5531          | 0.3721   |
+| 3.2322        | 22.7406 | 78000 | 3.5444          | 0.3729   |
+| 3.1449        | 23.0321 | 79000 | 3.5596          | 0.3723   |
+| 3.1896        | 23.3236 | 80000 | 3.5578          | 0.3722   |
+| 3.2083        | 23.6152 | 81000 | 3.5471          | 0.3725   |
+| 3.224         | 23.9068 | 82000 | 3.5402          | 0.3732   |
+| 3.171         | 24.1983 | 83000 | 3.5561          | 0.3726   |
+| 3.2066        | 24.4898 | 84000 | 3.5516          | 0.3730   |
+| 3.2235        | 24.7814 | 85000 | 3.5446          | 0.3731   |
+| 3.1408        | 25.0729 | 86000 | 3.5595          | 0.3726   |
+| 3.1736        | 25.3645 | 87000 | 3.5544          | 0.3728   |
+| 3.1914        | 25.6560 | 88000 | 3.5453          | 0.3734   |
+| 3.2119        | 25.9476 | 89000 | 3.5402          | 0.3735   |
+| 3.1569        | 26.2391 | 90000 | 3.5583          | 0.3727   |
+| 3.1936        | 26.5306 | 91000 | 3.5492          | 0.3728   |
+| 3.1956        | 26.8222 | 92000 | 3.5419          | 0.3735   |
+
+
+### Framework versions
+
+- Transformers 4.55.2
+- Pytorch 2.8.0+cu128
+- Datasets 4.0.0
+- Tokenizers 0.21.4

exp2b/swap_0.3_cost_to_drop_1032/all_results.json

@@ -0,0 +1,16 @@
+{
+    "epoch": 26.82220537640679,
+    "eval_accuracy": 0.36580799960608973,
+    "eval_loss": 3.5838623046875,
+    "eval_runtime": 53.8483,
+    "eval_samples": 16613,
+    "eval_samples_per_second": 308.515,
+    "eval_steps_per_second": 19.295,
+    "perplexity": 36.0123633067796,
+    "total_flos": 1.923000658034688e+18,
+    "train_loss": 3.4192942959329358,
+    "train_runtime": 41176.7228,
+    "train_samples": 274384,
+    "train_samples_per_second": 333.179,
+    "train_steps_per_second": 4.165
+}

exp2b/swap_0.3_cost_to_drop_1032/config.json

@@ -0,0 +1,31 @@
+{
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.2",
+  "use_cache": true,
+  "vocab_size": 50257
+}

exp2b/swap_0.3_cost_to_drop_1032/eval_results.json

@@ -0,0 +1,10 @@
+{
+    "epoch": 26.82220537640679,
+    "eval_accuracy": 0.36580799960608973,
+    "eval_loss": 3.5838623046875,
+    "eval_runtime": 53.8483,
+    "eval_samples": 16613,
+    "eval_samples_per_second": 308.515,
+    "eval_steps_per_second": 19.295,
+    "perplexity": 36.0123633067796
+}

exp2b/swap_0.3_cost_to_drop_1032/generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.55.2"
+}

exp2b/swap_0.3_cost_to_drop_1032/special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

exp2b/swap_0.3_cost_to_drop_1032/tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1024,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

exp2b/swap_0.3_cost_to_drop_1032/train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 26.82220537640679,
+    "total_flos": 1.923000658034688e+18,
+    "train_loss": 3.4192942959329358,
+    "train_runtime": 41176.7228,
+    "train_samples": 274384,
+    "train_samples_per_second": 333.179,
+    "train_steps_per_second": 4.165
+}

exp2b/swap_0.3_cost_to_drop_2128/README.md

@@ -0,0 +1,165 @@
+---
+library_name: transformers
+tags:
+- generated_from_trainer
+metrics:
+- accuracy
+model-index:
+- name: exceptions_exp2_swap_0.3_cost_to_drop_2128
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/craaaa/exceptions_exp2/runs/9mwuqnix)
+# exceptions_exp2_swap_0.3_cost_to_drop_2128
+
+This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 3.5645
+- Accuracy: 0.3686
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0006
+- train_batch_size: 16
+- eval_batch_size: 16
+- seed: 2128
+- gradient_accumulation_steps: 5
+- total_train_batch_size: 80
+- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.98) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 100
+- num_epochs: 50.0
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch   | Step   | Validation Loss | Accuracy |
+|:-------------:|:-------:|:------:|:---------------:|:--------:|
+| 4.8412        | 0.2916  | 1000   | 4.7675          | 0.2525   |
+| 4.3629        | 0.5831  | 2000   | 4.2992          | 0.2974   |
+| 4.1467        | 0.8747  | 3000   | 4.1125          | 0.3142   |
+| 4.0091        | 1.1662  | 4000   | 3.9979          | 0.3240   |
+| 3.9325        | 1.4578  | 5000   | 3.9250          | 0.3302   |
+| 3.8863        | 1.7493  | 6000   | 3.8661          | 0.3356   |
+| 3.7587        | 2.0408  | 7000   | 3.8232          | 0.3400   |
+| 3.7568        | 2.3324  | 8000   | 3.7916          | 0.3431   |
+| 3.7504        | 2.6239  | 9000   | 3.7629          | 0.3457   |
+| 3.7423        | 2.9155  | 10000  | 3.7371          | 0.3482   |
+| 3.6398        | 3.2070  | 11000  | 3.7222          | 0.3501   |
+| 3.6404        | 3.4986  | 12000  | 3.7053          | 0.3518   |
+| 3.6436        | 3.7901  | 13000  | 3.6884          | 0.3535   |
+| 3.5458        | 4.0816  | 14000  | 3.6811          | 0.3546   |
+| 3.5789        | 4.3732  | 15000  | 3.6694          | 0.3560   |
+| 3.5978        | 4.6648  | 16000  | 3.6576          | 0.3570   |
+| 3.5815        | 4.9563  | 17000  | 3.6433          | 0.3581   |
+| 3.5227        | 5.2478  | 18000  | 3.6438          | 0.3590   |
+| 3.5329        | 5.5394  | 19000  | 3.6334          | 0.3598   |
+| 3.5403        | 5.8310  | 20000  | 3.6214          | 0.3607   |
+| 3.4624        | 6.1225  | 21000  | 3.6298          | 0.3607   |
+| 3.4801        | 6.4140  | 22000  | 3.6188          | 0.3618   |
+| 3.499         | 6.7056  | 23000  | 3.6087          | 0.3624   |
+| 3.5005        | 6.9971  | 24000  | 3.5999          | 0.3635   |
+| 3.4326        | 7.2886  | 25000  | 3.6095          | 0.3631   |
+| 3.4619        | 7.5802  | 26000  | 3.5995          | 0.3637   |
+| 3.4767        | 7.8718  | 27000  | 3.5887          | 0.3647   |
+| 3.3875        | 8.1633  | 28000  | 3.5997          | 0.3644   |
+| 3.4172        | 8.4548  | 29000  | 3.5920          | 0.3650   |
+| 3.442         | 8.7464  | 30000  | 3.5813          | 0.3657   |
+| 3.3494        | 9.0379  | 31000  | 3.5922          | 0.3659   |
+| 3.3838        | 9.3295  | 32000  | 3.5890          | 0.3661   |
+| 3.4142        | 9.6210  | 33000  | 3.5799          | 0.3665   |
+| 3.4202        | 9.9126  | 34000  | 3.5699          | 0.3671   |
+| 3.3572        | 10.2041 | 35000  | 3.5847          | 0.3668   |
+| 3.3677        | 10.4957 | 36000  | 3.5785          | 0.3669   |
+| 3.3858        | 10.7872 | 37000  | 3.5684          | 0.3675   |
+| 3.3138        | 11.0787 | 38000  | 3.5779          | 0.3678   |
+| 3.3435        | 11.3703 | 39000  | 3.5729          | 0.3681   |
+| 3.3632        | 11.6618 | 40000  | 3.5645          | 0.3686   |
+| 3.3812        | 11.9534 | 41000  | 3.5576          | 0.3690   |
+| 3.2993        | 12.2449 | 42000  | 3.5710          | 0.3682   |
+| 3.3399        | 12.5365 | 43000  | 3.5659          | 0.3689   |
+| 3.3545        | 12.8280 | 44000  | 3.5561          | 0.3695   |
+| 3.2804        | 13.1195 | 45000  | 3.5699          | 0.3687   |
+| 3.3232        | 13.4111 | 46000  | 3.5670          | 0.3689   |
+| 3.3411        | 13.7027 | 47000  | 3.5578          | 0.3697   |
+| 3.3427        | 13.9942 | 48000  | 3.5520          | 0.3701   |
+| 3.2652        | 14.2857 | 49000  | 3.5674          | 0.3695   |
+| 3.3215        | 14.5773 | 50000  | 3.5607          | 0.3700   |
+| 3.3388        | 14.8689 | 51000  | 3.5499          | 0.3706   |
+| 3.263         | 15.1604 | 52000  | 3.5668          | 0.3698   |
+| 3.2958        | 15.4519 | 53000  | 3.5581          | 0.3703   |
+| 3.3097        | 15.7435 | 54000  | 3.5523          | 0.3708   |
+| 3.2196        | 16.0350 | 55000  | 3.5626          | 0.3702   |
+| 3.2662        | 16.3265 | 56000  | 3.5624          | 0.3704   |
+| 3.2873        | 16.6181 | 57000  | 3.5548          | 0.3705   |
+| 3.3024        | 16.9097 | 58000  | 3.5470          | 0.3713   |
+| 3.2379        | 17.2012 | 59000  | 3.5632          | 0.3707   |
+| 3.2753        | 17.4927 | 60000  | 3.5568          | 0.3709   |
+| 3.2848        | 17.7843 | 61000  | 3.5484          | 0.3714   |
+| 3.2023        | 18.0758 | 62000  | 3.5607          | 0.3713   |
+| 3.2515        | 18.3674 | 63000  | 3.5598          | 0.3710   |
+| 3.2652        | 18.6589 | 64000  | 3.5506          | 0.3717   |
+| 3.2867        | 18.9505 | 65000  | 3.5431          | 0.3719   |
+| 3.2242        | 19.2420 | 66000  | 3.5621          | 0.3713   |
+| 3.2556        | 19.5336 | 67000  | 3.5534          | 0.3716   |
+| 3.2738        | 19.8251 | 68000  | 3.5475          | 0.3721   |
+| 3.2021        | 20.1166 | 69000  | 3.5616          | 0.3713   |
+| 3.2299        | 20.4082 | 70000  | 3.5592          | 0.3716   |
+| 3.2573        | 20.6997 | 71000  | 3.5481          | 0.3723   |
+| 3.2701        | 20.9913 | 72000  | 3.5409          | 0.3725   |
+| 3.2204        | 21.2828 | 73000  | 3.5582          | 0.3716   |
+| 3.2229        | 21.5744 | 74000  | 3.5522          | 0.3720   |
+| 3.2339        | 21.8659 | 75000  | 3.5419          | 0.3727   |
+| 3.1707        | 22.1574 | 76000  | 3.5588          | 0.3722   |
+| 3.2119        | 22.4490 | 77000  | 3.5525          | 0.3723   |
+| 3.2174        | 22.7406 | 78000  | 3.5479          | 0.3725   |
+| 3.1437        | 23.0321 | 79000  | 3.5616          | 0.3721   |
+| 3.197         | 23.3236 | 80000  | 3.5559          | 0.3722   |
+| 3.2125        | 23.6152 | 81000  | 3.5491          | 0.3728   |
+| 3.2511        | 23.9068 | 82000  | 3.5405          | 0.3731   |
+| 3.162         | 24.1983 | 83000  | 3.5605          | 0.3720   |
+| 3.1887        | 24.4898 | 84000  | 3.5551          | 0.3724   |
+| 3.2115        | 24.7814 | 85000  | 3.5455          | 0.3729   |
+| 3.1402        | 25.0729 | 86000  | 3.5610          | 0.3727   |
+| 3.1789        | 25.3645 | 87000  | 3.5575          | 0.3722   |
+| 3.1993        | 25.6560 | 88000  | 3.5498          | 0.3731   |
+| 3.218         | 25.9476 | 89000  | 3.5447          | 0.3734   |
+| 3.1469        | 26.2391 | 90000  | 3.5599          | 0.3725   |
+| 3.1894        | 26.5306 | 91000  | 3.5501          | 0.3733   |
+| 3.1931        | 26.8222 | 92000  | 3.5464          | 0.3733   |
+| 3.1294        | 27.1137 | 93000  | 3.5635          | 0.3724   |
+| 3.1452        | 27.4053 | 94000  | 3.5562          | 0.3731   |
+| 3.1861        | 27.6968 | 95000  | 3.5468          | 0.3732   |
+| 3.1921        | 27.9884 | 96000  | 3.5444          | 0.3735   |
+| 3.1493        | 28.2799 | 97000  | 3.5572          | 0.3730   |
+| 3.1537        | 28.5715 | 98000  | 3.5511          | 0.3730   |
+| 3.1804        | 28.8630 | 99000  | 3.5434          | 0.3739   |
+| 3.1242        | 29.1545 | 100000 | 3.5606          | 0.3727   |
+| 3.1423        | 29.4461 | 101000 | 3.5562          | 0.3731   |
+| 3.1702        | 29.7377 | 102000 | 3.5504          | 0.3735   |
+
+
+### Framework versions
+
+- Transformers 4.55.2
+- Pytorch 2.8.0+cu128
+- Datasets 4.0.0
+- Tokenizers 0.21.4

exp2b/swap_0.3_cost_to_drop_2128/all_results.json

@@ -0,0 +1,16 @@
+{
+    "epoch": 29.737652341244388,
+    "eval_accuracy": 0.36855724356105724,
+    "eval_loss": 3.564516067504883,
+    "eval_runtime": 53.6794,
+    "eval_samples": 16613,
+    "eval_samples_per_second": 309.485,
+    "eval_steps_per_second": 19.356,
+    "perplexity": 35.3223556318352,
+    "total_flos": 2.132021741617152e+18,
+    "train_loss": 3.3939477501663506,
+    "train_runtime": 45494.094,
+    "train_samples": 274384,
+    "train_samples_per_second": 301.56,
+    "train_steps_per_second": 3.77
+}

exp2b/swap_0.3_cost_to_drop_2128/config.json

@@ -0,0 +1,31 @@
+{
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.2",
+  "use_cache": true,
+  "vocab_size": 50257
+}

exp2b/swap_0.3_cost_to_drop_2128/eval_results.json

@@ -0,0 +1,10 @@
+{
+    "epoch": 29.737652341244388,
+    "eval_accuracy": 0.36855724356105724,
+    "eval_loss": 3.564516067504883,
+    "eval_runtime": 53.6794,
+    "eval_samples": 16613,
+    "eval_samples_per_second": 309.485,
+    "eval_steps_per_second": 19.356,
+    "perplexity": 35.3223556318352
+}

exp2b/swap_0.3_cost_to_drop_2128/generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.55.2"
+}

exp2b/swap_0.3_cost_to_drop_2128/special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

exp2b/swap_0.3_cost_to_drop_2128/tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1024,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

exp2b/swap_0.3_cost_to_drop_2128/train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 29.737652341244388,
+    "total_flos": 2.132021741617152e+18,
+    "train_loss": 3.3939477501663506,
+    "train_runtime": 45494.094,
+    "train_samples": 274384,
+    "train_samples_per_second": 301.56,
+    "train_steps_per_second": 3.77
+}

exp2b/swap_0.3_cost_to_drop_3591/README.md

@@ -0,0 +1,163 @@
+---
+library_name: transformers
+tags:
+- generated_from_trainer
+metrics:
+- accuracy
+model-index:
+- name: exceptions_exp2_swap_0.3_cost_to_drop_3591
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/craaaa/exceptions_exp2/runs/eauilxa9)
+# exceptions_exp2_swap_0.3_cost_to_drop_3591
+
+This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 3.5628
+- Accuracy: 0.3685
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0006
+- train_batch_size: 16
+- eval_batch_size: 16
+- seed: 3591
+- gradient_accumulation_steps: 5
+- total_train_batch_size: 80
+- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.98) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 100
+- num_epochs: 50.0
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch   | Step   | Accuracy | Validation Loss |
+|:-------------:|:-------:|:------:|:--------:|:---------------:|
+| 4.8365        | 0.2916  | 1000   | 0.2548   | 4.7491          |
+| 4.3471        | 0.5831  | 2000   | 0.2988   | 4.2858          |
+| 4.1508        | 0.8747  | 3000   | 0.3151   | 4.0989          |
+| 3.9896        | 1.1662  | 4000   | 0.3244   | 3.9914          |
+| 3.9465        | 1.4578  | 5000   | 0.3314   | 3.9160          |
+| 3.8798        | 1.7493  | 6000   | 0.3365   | 3.8590          |
+| 3.7577        | 2.0408  | 7000   | 0.3405   | 3.8170          |
+| 3.7548        | 2.3324  | 8000   | 0.3436   | 3.7865          |
+| 3.7612        | 2.6239  | 9000   | 0.3464   | 3.7582          |
+| 3.7282        | 2.9155  | 10000  | 0.3491   | 3.7295          |
+| 3.6436        | 3.2070  | 11000  | 0.3508   | 3.7194          |
+| 3.6459        | 3.4986  | 12000  | 0.3524   | 3.6989          |
+| 3.6481        | 3.7901  | 13000  | 0.3541   | 3.6826          |
+| 3.5535        | 4.0816  | 14000  | 0.3553   | 3.6771          |
+| 3.5787        | 4.3732  | 15000  | 0.3564   | 3.6656          |
+| 3.5765        | 4.6648  | 16000  | 0.3577   | 3.6524          |
+| 3.584         | 4.9563  | 17000  | 0.3591   | 3.6387          |
+| 3.5087        | 5.2478  | 18000  | 0.3591   | 3.6413          |
+| 3.5284        | 5.5394  | 19000  | 0.3603   | 3.6312          |
+| 3.5434        | 5.8310  | 20000  | 0.3609   | 3.6183          |
+| 3.4492        | 6.1225  | 21000  | 0.3614   | 3.6239          |
+| 3.487         | 6.4140  | 22000  | 0.3624   | 3.6121          |
+| 3.4875        | 6.7056  | 23000  | 0.3631   | 3.6044          |
+| 3.5           | 6.9971  | 24000  | 0.3638   | 3.5953          |
+| 3.4285        | 7.2886  | 25000  | 0.3637   | 3.6057          |
+| 3.4546        | 7.5802  | 26000  | 0.3645   | 3.5960          |
+| 3.4594        | 7.8718  | 27000  | 0.3652   | 3.5860          |
+| 3.3818        | 8.1633  | 28000  | 0.3648   | 3.5961          |
+| 3.4168        | 8.4548  | 29000  | 0.3655   | 3.5885          |
+| 3.4465        | 8.7464  | 30000  | 0.3661   | 3.5820          |
+| 3.3283        | 9.0379  | 31000  | 0.3660   | 3.5848          |
+| 3.3822        | 9.3295  | 32000  | 0.3662   | 3.5840          |
+| 3.402         | 9.6210  | 33000  | 0.3671   | 3.5742          |
+| 3.4218        | 9.9126  | 34000  | 0.3676   | 3.5681          |
+| 3.3471        | 10.2041 | 35000  | 0.3673   | 3.5807          |
+| 3.3754        | 10.4957 | 36000  | 0.3676   | 3.5724          |
+| 3.3889        | 10.7872 | 37000  | 0.3681   | 3.5649          |
+| 3.3111        | 11.0787 | 38000  | 0.3678   | 3.5778          |
+| 3.3445        | 11.3703 | 39000  | 0.3684   | 3.5678          |
+| 3.3621        | 11.6618 | 40000  | 0.3685   | 3.5628          |
+| 3.3843        | 11.9534 | 41000  | 0.3692   | 3.5562          |
+| 3.3039        | 12.2449 | 42000  | 0.3684   | 3.5673          |
+| 3.3371        | 12.5365 | 43000  | 0.3692   | 3.5636          |
+| 3.357         | 12.8280 | 44000  | 0.3698   | 3.5513          |
+| 3.2723        | 13.1195 | 45000  | 0.3691   | 3.5671          |
+| 3.3072        | 13.4111 | 46000  | 0.3695   | 3.5618          |
+| 3.3428        | 13.7027 | 47000  | 0.3702   | 3.5526          |
+| 3.3393        | 13.9942 | 48000  | 0.3705   | 3.5471          |
+| 3.2803        | 14.2857 | 49000  | 0.3699   | 3.5606          |
+| 3.3086        | 14.5773 | 50000  | 0.3704   | 3.5528          |
+| 3.3236        | 14.8689 | 51000  | 0.3710   | 3.5478          |
+| 3.248         | 15.1604 | 52000  | 0.3703   | 3.5615          |
+| 3.2804        | 15.4519 | 53000  | 0.3706   | 3.5556          |
+| 3.3118        | 15.7435 | 54000  | 0.3709   | 3.5484          |
+| 3.2093        | 16.0350 | 55000  | 0.3705   | 3.5584          |
+| 3.2675        | 16.3265 | 56000  | 0.3710   | 3.5577          |
+| 3.2897        | 16.6181 | 57000  | 0.3714   | 3.5507          |
+| 3.2994        | 16.9097 | 58000  | 0.3717   | 3.5419          |
+| 3.235         | 17.2012 | 59000  | 0.3709   | 3.5576          |
+| 3.2628        | 17.4927 | 60000  | 0.3708   | 3.5542          |
+| 3.2733        | 17.7843 | 61000  | 0.3716   | 3.5446          |
+| 3.1962        | 18.0758 | 62000  | 0.3713   | 3.5552          |
+| 3.2368        | 18.3674 | 63000  | 0.3713   | 3.5570          |
+| 3.2627        | 18.6589 | 64000  | 0.3719   | 3.5456          |
+| 3.2811        | 18.9505 | 65000  | 0.3722   | 3.5395          |
+| 3.2051        | 19.2420 | 66000  | 0.3717   | 3.5561          |
+| 3.2372        | 19.5336 | 67000  | 0.3720   | 3.5484          |
+| 3.2646        | 19.8251 | 68000  | 0.3726   | 3.5410          |
+| 3.1758        | 20.1166 | 69000  | 0.3721   | 3.5565          |
+| 3.2262        | 20.4082 | 70000  | 0.3720   | 3.5522          |
+| 3.2448        | 20.6997 | 71000  | 0.3727   | 3.5422          |
+| 3.2535        | 20.9913 | 72000  | 0.3731   | 3.5363          |
+| 3.2035        | 21.2828 | 73000  | 0.3719   | 3.5573          |
+| 3.227         | 21.5744 | 74000  | 0.3725   | 3.5450          |
+| 3.2327        | 21.8659 | 75000  | 0.3731   | 3.5384          |
+| 3.1776        | 22.1574 | 76000  | 0.3723   | 3.5553          |
+| 3.2118        | 22.4490 | 77000  | 0.3730   | 3.5483          |
+| 3.2266        | 22.7406 | 78000  | 0.3730   | 3.5435          |
+| 3.1331        | 23.0321 | 79000  | 0.3724   | 3.5540          |
+| 3.1916        | 23.3236 | 80000  | 0.3729   | 3.5507          |
+| 3.187         | 23.6152 | 81000  | 3.5549   | 0.3726          |
+| 3.2057        | 23.9068 | 82000  | 3.5462   | 0.3730          |
+| 3.1548        | 24.1986 | 83000  | 3.5578   | 0.3727          |
+| 3.1716        | 24.4901 | 84000  | 3.5530   | 0.3727          |
+| 3.2119        | 24.7817 | 85000  | 3.5454   | 0.3734          |
+| 3.1349        | 25.0732 | 86000  | 3.5582   | 0.3726          |
+| 3.1728        | 25.3647 | 87000  | 3.5531   | 0.3734          |
+| 3.1959        | 25.6563 | 88000  | 3.5434   | 0.3732          |
+| 3.2124        | 25.9479 | 89000  | 3.5389   | 0.3738          |
+| 3.144         | 26.2394 | 90000  | 3.5542   | 0.3731          |
+| 3.1763        | 26.5309 | 91000  | 3.5478   | 0.3733          |
+| 3.1947        | 26.8225 | 92000  | 3.5423   | 0.3738          |
+| 3.1318        | 27.1140 | 93000  | 3.5578   | 0.3730          |
+| 3.1523        | 27.4056 | 94000  | 3.5547   | 0.3735          |
+| 3.1788        | 27.6971 | 95000  | 3.5462   | 0.3738          |
+| 3.1949        | 27.9887 | 96000  | 3.5371   | 0.3745          |
+| 3.1268        | 28.2802 | 97000  | 3.5564   | 0.3731          |
+| 3.1661        | 28.5718 | 98000  | 3.5473   | 0.3737          |
+| 3.1665        | 28.8633 | 99000  | 3.5398   | 0.3741          |
+| 3.1139        | 29.1548 | 100000 | 3.5575   | 0.3734          |
+
+
+### Framework versions
+
+- Transformers 4.55.2
+- Pytorch 2.8.0+cu128
+- Datasets 4.0.0
+- Tokenizers 0.21.4

exp2b/swap_0.3_cost_to_drop_3591/all_results.json

@@ -0,0 +1,16 @@
+{
+    "epoch": 29.154819523004257,
+    "eval_accuracy": 0.3685325063730653,
+    "eval_loss": 3.5627942085266113,
+    "eval_runtime": 80.853,
+    "eval_samples": 16613,
+    "eval_samples_per_second": 205.472,
+    "eval_steps_per_second": 12.85,
+    "perplexity": 35.26158784844721,
+    "total_flos": 2.090235919859712e+18,
+    "train_loss": 0.6341165426635742,
+    "train_runtime": 16222.486,
+    "train_samples": 274384,
+    "train_samples_per_second": 845.69,
+    "train_steps_per_second": 10.572
+}

exp2b/swap_0.3_cost_to_drop_3591/config.json

@@ -0,0 +1,31 @@
+{
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.2",
+  "use_cache": true,
+  "vocab_size": 50257
+}

exp2b/swap_0.3_cost_to_drop_3591/eval_results.json

@@ -0,0 +1,10 @@
+{
+    "epoch": 29.154819523004257,
+    "eval_accuracy": 0.3685325063730653,
+    "eval_loss": 3.5627942085266113,
+    "eval_runtime": 80.853,
+    "eval_samples": 16613,
+    "eval_samples_per_second": 205.472,
+    "eval_steps_per_second": 12.85,
+    "perplexity": 35.26158784844721
+}

exp2b/swap_0.3_cost_to_drop_3591/generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.55.2"
+}

exp2b/swap_0.3_cost_to_drop_3591/special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

exp2b/swap_0.3_cost_to_drop_3591/tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1024,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

exp2b/swap_0.3_cost_to_drop_3591/train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 29.154819523004257,
+    "total_flos": 2.090235919859712e+18,
+    "train_loss": 0.6341165426635742,
+    "train_runtime": 16222.486,
+    "train_samples": 274384,
+    "train_samples_per_second": 845.69,
+    "train_steps_per_second": 10.572
+}

exp2b/swap_0.3_cost_to_drop_40817/README.md

@@ -0,0 +1,155 @@
+---
+library_name: transformers
+tags:
+- generated_from_trainer
+metrics:
+- accuracy
+model-index:
+- name: exceptions_exp2_swap_0.3_cost_to_drop_40817
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/craaaa/exceptions_exp2/runs/tevm949y)
+# exceptions_exp2_swap_0.3_cost_to_drop_40817
+
+This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 3.5843
+- Accuracy: 0.3658
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0006
+- train_batch_size: 16
+- eval_batch_size: 16
+- seed: 40817
+- gradient_accumulation_steps: 5
+- total_train_batch_size: 80
+- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.98) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 100
+- num_epochs: 50.0
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch   | Step  | Validation Loss | Accuracy |
+|:-------------:|:-------:|:-----:|:---------------:|:--------:|
+| 4.8253        | 0.2916  | 1000  | 4.7509          | 0.2547   |
+| 4.3464        | 0.5831  | 2000  | 4.2883          | 0.2984   |
+| 4.1463        | 0.8747  | 3000  | 4.1065          | 0.3143   |
+| 4.0014        | 1.1662  | 4000  | 3.9989          | 0.3242   |
+| 3.9414        | 1.4578  | 5000  | 3.9235          | 0.3304   |
+| 3.8969        | 1.7493  | 6000  | 3.8642          | 0.3355   |
+| 3.7561        | 2.0408  | 7000  | 3.8230          | 0.3403   |
+| 3.7663        | 2.3324  | 8000  | 3.7936          | 0.3428   |
+| 3.7423        | 2.6239  | 9000  | 3.7637          | 0.3461   |
+| 3.7236        | 2.9155  | 10000 | 3.7368          | 0.3483   |
+| 3.6389        | 3.2070  | 11000 | 3.7250          | 0.3501   |
+| 3.6502        | 3.4986  | 12000 | 3.7054          | 0.3518   |
+| 3.6544        | 3.7901  | 13000 | 3.6876          | 0.3536   |
+| 3.5502        | 4.0816  | 14000 | 3.6809          | 0.3549   |
+| 3.583         | 4.3732  | 15000 | 3.6712          | 0.3560   |
+| 3.5796        | 4.6648  | 16000 | 3.6538          | 0.3572   |
+| 3.581         | 4.9563  | 17000 | 3.6421          | 0.3586   |
+| 3.5122        | 5.2478  | 18000 | 3.6446          | 0.3584   |
+| 3.525         | 5.5394  | 19000 | 3.6348          | 0.3598   |
+| 3.5368        | 5.8310  | 20000 | 3.6213          | 0.3606   |
+| 3.4484        | 6.1225  | 21000 | 3.6261          | 0.3613   |
+| 3.4845        | 6.4140  | 22000 | 3.6184          | 0.3622   |
+| 3.4977        | 6.7056  | 23000 | 3.6091          | 0.3627   |
+| 3.4924        | 6.9971  | 24000 | 3.6000          | 0.3634   |
+| 3.4321        | 7.2886  | 25000 | 3.6098          | 0.3632   |
+| 3.4541        | 7.5802  | 26000 | 3.5999          | 0.3639   |
+| 3.4603        | 7.8718  | 27000 | 3.5904          | 0.3650   |
+| 3.3916        | 8.1633  | 28000 | 3.5991          | 0.3646   |
+| 3.4293        | 8.4548  | 29000 | 3.5937          | 0.3650   |
+| 3.4353        | 8.7464  | 30000 | 3.5843          | 0.3658   |
+| 3.3256        | 9.0379  | 31000 | 3.5890          | 0.3658   |
+| 3.389         | 9.3295  | 32000 | 3.5875          | 0.3660   |
+| 3.4145        | 9.6210  | 33000 | 3.5771          | 0.3668   |
+| 3.4229        | 9.9126  | 34000 | 3.5694          | 0.3673   |
+| 3.3505        | 10.2041 | 35000 | 3.5863          | 0.3668   |
+| 3.3784        | 10.4957 | 36000 | 3.5753          | 0.3674   |
+| 3.3983        | 10.7872 | 37000 | 3.5668          | 0.3682   |
+| 3.3011        | 11.0787 | 38000 | 3.5799          | 0.3674   |
+| 3.3479        | 11.3703 | 39000 | 3.5759          | 0.3679   |
+| 3.3693        | 11.6618 | 40000 | 3.5683          | 0.3684   |
+| 3.3889        | 11.9534 | 41000 | 3.5575          | 0.3695   |
+| 3.3164        | 12.2449 | 42000 | 3.5740          | 0.3685   |
+| 3.3456        | 12.5365 | 43000 | 3.5672          | 0.3688   |
+| 3.3534        | 12.8280 | 44000 | 3.5567          | 0.3695   |
+| 3.2725        | 13.1195 | 45000 | 3.5710          | 0.3689   |
+| 3.3196        | 13.4111 | 46000 | 3.5633          | 0.3693   |
+| 3.3422        | 13.7027 | 47000 | 3.5570          | 0.3702   |
+| 3.3471        | 13.9942 | 48000 | 3.5477          | 0.3704   |
+| 3.2938        | 14.2857 | 49000 | 3.5640          | 0.3698   |
+| 3.319         | 14.5773 | 50000 | 3.5593          | 0.3703   |
+| 3.3273        | 14.8689 | 51000 | 3.5482          | 0.3708   |
+| 3.2519        | 15.1604 | 52000 | 3.5683          | 0.3700   |
+| 3.2842        | 15.4519 | 53000 | 3.5604          | 0.3704   |
+| 3.2954        | 15.7435 | 54000 | 3.5522          | 0.3707   |
+| 3.2043        | 16.0350 | 55000 | 3.5648          | 0.3704   |
+| 3.2567        | 16.3265 | 56000 | 3.5596          | 0.3707   |
+| 3.292         | 16.6181 | 57000 | 3.5538          | 0.3712   |
+| 3.3029        | 16.9097 | 58000 | 3.5476          | 0.3716   |
+| 3.2451        | 17.2012 | 59000 | 3.5595          | 0.3707   |
+| 3.2652        | 17.4927 | 60000 | 3.5539          | 0.3713   |
+| 3.295         | 17.7843 | 61000 | 3.5509          | 0.3715   |
+| 3.2147        | 18.0758 | 62000 | 3.5609          | 0.3713   |
+| 3.245         | 18.3674 | 63000 | 3.5601          | 0.3711   |
+| 3.2796        | 18.6589 | 64000 | 3.5508          | 0.3717   |
+| 3.2952        | 18.9505 | 65000 | 3.5425          | 0.3721   |
+| 3.216         | 19.2420 | 66000 | 3.5589          | 0.3716   |
+| 3.2585        | 19.5336 | 67000 | 3.5516          | 0.3715   |
+| 3.2594        | 19.8251 | 68000 | 3.5475          | 0.3721   |
+| 3.1957        | 20.1166 | 69000 | 3.5620          | 0.3719   |
+| 3.2145        | 20.4082 | 70000 | 3.5529          | 0.3721   |
+| 3.2456        | 20.6997 | 71000 | 3.5489          | 0.3723   |
+| 3.2628        | 20.9913 | 72000 | 3.5371          | 0.3730   |
+| 3.1993        | 21.2828 | 73000 | 3.5585          | 0.3719   |
+| 3.231         | 21.5744 | 74000 | 3.5494          | 0.3725   |
+| 3.2477        | 21.8659 | 75000 | 3.5436          | 0.3729   |
+| 3.1661        | 22.1574 | 76000 | 3.5601          | 0.3721   |
+| 3.2189        | 22.4490 | 77000 | 3.5512          | 0.3726   |
+| 3.2296        | 22.7406 | 78000 | 3.5443          | 0.3730   |
+| 3.142         | 23.0321 | 79000 | 3.5527          | 0.3727   |
+| 3.1853        | 23.3236 | 80000 | 3.5581          | 0.3724   |
+| 3.2124        | 23.6152 | 81000 | 3.5495          | 0.3729   |
+| 3.2265        | 23.9068 | 82000 | 3.5426          | 0.3734   |
+| 3.1684        | 24.1983 | 83000 | 3.5639          | 0.3723   |
+| 3.2058        | 24.4898 | 84000 | 3.5531          | 0.3729   |
+| 3.1945        | 24.7814 | 85000 | 3.5442          | 0.3733   |
+| 3.1343        | 25.0729 | 86000 | 3.5590          | 0.3727   |
+| 3.1752        | 25.3645 | 87000 | 3.5542          | 0.3730   |
+| 3.1882        | 25.6560 | 88000 | 3.5493          | 0.3734   |
+| 3.2231        | 25.9476 | 89000 | 3.5403          | 0.3736   |
+| 3.1648        | 26.2391 | 90000 | 3.5593          | 0.3726   |
+| 3.1801        | 26.5306 | 91000 | 3.5525          | 0.3732   |
+| 3.1926        | 26.8222 | 92000 | 3.5414          | 0.3738   |
+
+
+### Framework versions
+
+- Transformers 4.55.2
+- Pytorch 2.8.0+cu128
+- Datasets 4.0.0
+- Tokenizers 0.21.4

exp2b/swap_0.3_cost_to_drop_40817/all_results.json

@@ -0,0 +1,16 @@
+{
+    "epoch": 26.82220537640679,
+    "eval_accuracy": 0.365785736136897,
+    "eval_loss": 3.5843005180358887,
+    "eval_runtime": 84.2675,
+    "eval_samples": 16613,
+    "eval_samples_per_second": 197.146,
+    "eval_steps_per_second": 12.33,
+    "perplexity": 36.02814786333675,
+    "total_flos": 1.923000658034688e+18,
+    "train_loss": 3.419246922866158,
+    "train_runtime": 68665.402,
+    "train_samples": 274384,
+    "train_samples_per_second": 199.798,
+    "train_steps_per_second": 2.498
+}

exp2b/swap_0.3_cost_to_drop_40817/config.json

@@ -0,0 +1,31 @@
+{
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.2",
+  "use_cache": true,
+  "vocab_size": 50257
+}

exp2b/swap_0.3_cost_to_drop_40817/eval_results.json

@@ -0,0 +1,10 @@
+{
+    "epoch": 26.82220537640679,
+    "eval_accuracy": 0.365785736136897,
+    "eval_loss": 3.5843005180358887,
+    "eval_runtime": 84.2675,
+    "eval_samples": 16613,
+    "eval_samples_per_second": 197.146,
+    "eval_steps_per_second": 12.33,
+    "perplexity": 36.02814786333675
+}

exp2b/swap_0.3_cost_to_drop_40817/generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.55.2"
+}

exp2b/swap_0.3_cost_to_drop_40817/special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

exp2b/swap_0.3_cost_to_drop_40817/tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1024,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

exp2b/swap_0.3_cost_to_drop_40817/train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 26.82220537640679,
+    "total_flos": 1.923000658034688e+18,
+    "train_loss": 3.419246922866158,
+    "train_runtime": 68665.402,
+    "train_samples": 274384,
+    "train_samples_per_second": 199.798,
+    "train_steps_per_second": 2.498
+}

exp2b/swap_0.3_cost_to_drop_5039/README.md

@@ -0,0 +1,172 @@
+---
+library_name: transformers
+tags:
+- generated_from_trainer
+metrics:
+- accuracy
+model-index:
+- name: exceptions_exp2_swap_0.3_cost_to_drop_5039
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/craaaa/exceptions_exp2/runs/gmmwslh6)
+# exceptions_exp2_swap_0.3_cost_to_drop_5039
+
+This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 3.5648
+- Accuracy: 0.3685
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0006
+- train_batch_size: 16
+- eval_batch_size: 16
+- seed: 5039
+- gradient_accumulation_steps: 5
+- total_train_batch_size: 80
+- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.98) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 100
+- num_epochs: 50.0
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch   | Step   | Validation Loss | Accuracy |
+|:-------------:|:-------:|:------:|:---------------:|:--------:|
+| 4.8389        | 0.2916  | 1000   | 4.7840          | 0.2508   |
+| 4.3668        | 0.5831  | 2000   | 4.2972          | 0.2979   |
+| 4.1742        | 0.8747  | 3000   | 4.1106          | 0.3136   |
+| 4.0092        | 1.1662  | 4000   | 4.0018          | 0.3235   |
+| 3.9283        | 1.4578  | 5000   | 3.9267          | 0.3306   |
+| 3.9025        | 1.7493  | 6000   | 3.8681          | 0.3360   |
+| 3.7392        | 2.0408  | 7000   | 3.8234          | 0.3399   |
+| 3.7636        | 2.3324  | 8000   | 3.7944          | 0.3429   |
+| 3.729         | 2.6239  | 9000   | 3.7647          | 0.3459   |
+| 3.7381        | 2.9155  | 10000  | 3.7375          | 0.3482   |
+| 3.6516        | 3.2070  | 11000  | 3.7226          | 0.3504   |
+| 3.6536        | 3.4986  | 12000  | 3.7043          | 0.3518   |
+| 3.6419        | 3.7901  | 13000  | 3.6872          | 0.3537   |
+| 3.5465        | 4.0816  | 14000  | 3.6797          | 0.3549   |
+| 3.5873        | 4.3732  | 15000  | 3.6692          | 0.3563   |
+| 3.5885        | 4.6648  | 16000  | 3.6552          | 0.3569   |
+| 3.592         | 4.9563  | 17000  | 3.6410          | 0.3584   |
+| 3.5235        | 5.2478  | 18000  | 3.6458          | 0.3588   |
+| 3.5286        | 5.5394  | 19000  | 3.6334          | 0.3599   |
+| 3.5339        | 5.8310  | 20000  | 3.6228          | 0.3610   |
+| 3.4402        | 6.1225  | 21000  | 3.6266          | 0.3614   |
+| 3.488         | 6.4140  | 22000  | 3.6171          | 0.3620   |
+| 3.4822        | 6.7056  | 23000  | 3.6088          | 0.3626   |
+| 3.5088        | 6.9971  | 24000  | 3.5973          | 0.3632   |
+| 3.4569        | 7.2886  | 25000  | 3.6074          | 0.3634   |
+| 3.4598        | 7.5802  | 26000  | 3.6001          | 0.3639   |
+| 3.4805        | 7.8718  | 27000  | 3.5892          | 0.3647   |
+| 3.391         | 8.1633  | 28000  | 3.5987          | 0.3648   |
+| 3.4192        | 8.4548  | 29000  | 3.5941          | 0.3649   |
+| 3.4553        | 8.7464  | 30000  | 3.5835          | 0.3656   |
+| 3.3433        | 9.0379  | 31000  | 3.5879          | 0.3658   |
+| 3.3777        | 9.3295  | 32000  | 3.5863          | 0.3660   |
+| 3.4124        | 9.6210  | 33000  | 3.5779          | 0.3669   |
+| 3.4163        | 9.9126  | 34000  | 3.5696          | 0.3671   |
+| 3.3552        | 10.2041 | 35000  | 3.5840          | 0.3665   |
+| 3.3787        | 10.4957 | 36000  | 3.5789          | 0.3674   |
+| 3.3957        | 10.7872 | 37000  | 3.5704          | 0.3675   |
+| 3.3018        | 11.0787 | 38000  | 3.5795          | 0.3677   |
+| 3.3555        | 11.3703 | 39000  | 3.5742          | 0.3679   |
+| 3.3832        | 11.6618 | 40000  | 3.5648          | 0.3685   |
+| 3.3791        | 11.9534 | 41000  | 3.5585          | 0.3690   |
+| 3.3167        | 12.2449 | 42000  | 3.5723          | 0.3683   |
+| 3.3413        | 12.5365 | 43000  | 3.5661          | 0.3689   |
+| 3.3628        | 12.8280 | 44000  | 3.5583          | 0.3694   |
+| 3.2734        | 13.1195 | 45000  | 3.5724          | 0.3691   |
+| 3.3231        | 13.4111 | 46000  | 3.5654          | 0.3693   |
+| 3.3429        | 13.7027 | 47000  | 3.5575          | 0.3698   |
+| 3.3593        | 13.9942 | 48000  | 3.5502          | 0.3706   |
+| 3.2888        | 14.2857 | 49000  | 3.5649          | 0.3696   |
+| 3.3147        | 14.5773 | 50000  | 3.5584          | 0.3701   |
+| 3.3218        | 14.8689 | 51000  | 3.5499          | 0.3707   |
+| 3.2606        | 15.1604 | 52000  | 3.5661          | 0.3699   |
+| 3.2897        | 15.4519 | 53000  | 3.5600          | 0.3704   |
+| 3.3127        | 15.7435 | 54000  | 3.5523          | 0.3707   |
+| 3.2188        | 16.0350 | 55000  | 3.5637          | 0.3702   |
+| 3.2612        | 16.3265 | 56000  | 3.5612          | 0.3705   |
+| 3.2943        | 16.6181 | 57000  | 3.5506          | 0.3713   |
+| 3.3085        | 16.9097 | 58000  | 3.5449          | 0.3714   |
+| 3.2314        | 17.2012 | 59000  | 3.5642          | 0.3707   |
+| 3.2629        | 17.4927 | 60000  | 3.5536          | 0.3713   |
+| 3.2787        | 17.7843 | 61000  | 3.5458          | 0.3717   |
+| 3.2018        | 18.0758 | 62000  | 3.5633          | 0.3710   |
+| 3.2435        | 18.3674 | 63000  | 3.5584          | 0.3713   |
+| 3.2724        | 18.6589 | 64000  | 3.5488          | 0.3718   |
+| 3.284         | 18.9505 | 65000  | 3.5413          | 0.3723   |
+| 3.2149        | 19.2420 | 66000  | 3.5605          | 0.3713   |
+| 3.2538        | 19.5336 | 67000  | 3.5527          | 0.3717   |
+| 3.2765        | 19.8251 | 68000  | 3.5437          | 0.3726   |
+| 3.1922        | 20.1166 | 69000  | 3.5602          | 0.3712   |
+| 3.2177        | 20.4082 | 70000  | 3.5591          | 0.3715   |
+| 3.2451        | 20.6997 | 71000  | 3.5503          | 0.3720   |
+| 3.2648        | 20.9913 | 72000  | 3.5401          | 0.3729   |
+| 3.206         | 21.2828 | 73000  | 3.5584          | 0.3716   |
+| 3.2347        | 21.5744 | 74000  | 3.5520          | 0.3721   |
+| 3.2503        | 21.8659 | 75000  | 3.5425          | 0.3728   |
+| 3.1884        | 22.1574 | 76000  | 3.5599          | 0.3720   |
+| 3.201         | 22.4490 | 77000  | 3.5554          | 0.3722   |
+| 3.2315        | 22.7406 | 78000  | 3.5435          | 0.3729   |
+| 3.1297        | 23.0321 | 79000  | 3.5574          | 0.3722   |
+| 3.181         | 23.3236 | 80000  | 3.5602          | 0.3721   |
+| 3.2171        | 23.6152 | 81000  | 3.5474          | 0.3729   |
+| 3.2329        | 23.9068 | 82000  | 3.5409          | 0.3731   |
+| 3.1634        | 24.1983 | 83000  | 3.5631          | 0.3721   |
+| 3.2056        | 24.4898 | 84000  | 3.5525          | 0.3726   |
+| 3.2099        | 24.7814 | 85000  | 3.5461          | 0.3732   |
+| 3.137         | 25.0729 | 86000  | 3.5613          | 0.3723   |
+| 3.1779        | 25.3645 | 87000  | 3.5577          | 0.3723   |
+| 3.1919        | 25.6560 | 88000  | 3.5501          | 0.3730   |
+| 3.2131        | 25.9476 | 89000  | 3.5394          | 0.3736   |
+| 3.1418        | 26.2391 | 90000  | 3.5594          | 0.3726   |
+| 3.1775        | 26.5306 | 91000  | 3.5563          | 0.3727   |
+| 3.1992        | 26.8222 | 92000  | 3.5488          | 0.3734   |
+| 3.1247        | 27.1137 | 93000  | 3.5643          | 0.3725   |
+| 3.1699        | 27.4053 | 94000  | 3.5572          | 0.3729   |
+| 3.1893        | 27.6968 | 95000  | 3.5476          | 0.3734   |
+| 3.1944        | 27.9884 | 96000  | 3.5439          | 0.3739   |
+| 3.1462        | 28.2799 | 97000  | 3.5595          | 0.3729   |
+| 3.1687        | 28.5715 | 98000  | 3.5537          | 0.3733   |
+| 3.1761        | 28.8630 | 99000  | 3.5453          | 0.3738   |
+| 3.1207        | 29.1545 | 100000 | 3.5631          | 0.3731   |
+| 3.1545        | 29.4461 | 101000 | 3.5610          | 0.3732   |
+| 3.1726        | 29.7377 | 102000 | 3.5489          | 0.3735   |
+| 3.0964        | 30.0292 | 103000 | 3.5597          | 0.3731   |
+| 3.1257        | 30.3207 | 104000 | 3.5614          | 0.3733   |
+| 3.1518        | 30.6123 | 105000 | 3.5540          | 0.3735   |
+| 3.1596        | 30.9038 | 106000 | 3.5481          | 0.3741   |
+| 3.1093        | 31.1953 | 107000 | 3.5633          | 0.3731   |
+| 3.131         | 31.4869 | 108000 | 3.5570          | 0.3737   |
+| 3.1623        | 31.7785 | 109000 | 3.5498          | 0.3740   |
+
+
+### Framework versions
+
+- Transformers 4.55.2
+- Pytorch 2.8.0+cu128
+- Datasets 4.0.0
+- Tokenizers 0.21.4

exp2b/swap_0.3_cost_to_drop_5039/all_results.json

@@ -0,0 +1,16 @@
+{
+    "epoch": 31.77847104787451,
+    "eval_accuracy": 0.3685449927631946,
+    "eval_loss": 3.564793586730957,
+    "eval_runtime": 88.1112,
+    "eval_samples": 16613,
+    "eval_samples_per_second": 188.546,
+    "eval_steps_per_second": 11.792,
+    "perplexity": 35.33215962496863,
+    "total_flos": 2.278336918192128e+18,
+    "train_loss": 3.3781057582470253,
+    "train_runtime": 81761.4476,
+    "train_samples": 274384,
+    "train_samples_per_second": 167.795,
+    "train_steps_per_second": 2.098
+}