craa's picture
Final models for exp1b, exp2a, exp3 (Leong & Linzen exceptions project)
48e1bd5
|
Raw History Blame Contribute Delete
11.2 kB
metadata
library_name: transformers
tags:
  - generated_from_trainer
datasets:
  - craa/100M
metrics:
  - accuracy
model-index:
  - name: high_2000_495
    results:
      - task:
          name: Causal Language Modeling
          type: text-generation
        dataset:
          name: craa/100M high_2000
          type: craa/100M
          args: high_2000
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.3976544447645905

Visualize in Weights & Biases

high_2000_495

This model is a fine-tuned version of on the craa/100M high_2000 dataset. It achieves the following results on the evaluation set:

  • Loss: 3.2715
  • Accuracy: 0.3977

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0006
  • train_batch_size: 16
  • eval_batch_size: 16
  • seed: 495
  • gradient_accumulation_steps: 5
  • total_train_batch_size: 80
  • optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.98) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 100
  • num_epochs: 40.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Accuracy Validation Loss
4.5772 0.2692 1000 0.2680 4.5105
4.0956 0.5383 2000 0.3163 4.0294
3.8948 0.8075 3000 0.3333 3.8446
3.744 1.0764 4000 0.3443 3.7341
3.6945 1.3456 5000 0.3510 3.6635
3.6366 1.6148 6000 0.3560 3.6113
3.6008 1.8839 7000 0.3602 3.5670
3.5165 2.1529 8000 0.3637 3.5360
3.5007 2.4220 9000 0.3667 3.5102
3.4864 2.6912 10000 0.3689 3.4849
3.464 2.9604 11000 0.3714 3.4628
3.3924 3.2293 12000 0.3729 3.4515
3.4073 3.4985 13000 0.3748 3.4358
3.3945 3.7677 14000 0.3761 3.4203
3.294 4.0366 15000 0.3773 3.4138
3.33 4.3058 16000 0.3783 3.4069
3.3293 4.5749 17000 0.3796 3.3921
3.3378 4.8441 18000 0.3809 3.3793
3.2494 5.1130 19000 0.3814 3.3809
3.2682 5.3822 20000 0.3820 3.3743
3.3043 5.6514 21000 0.3832 3.3640
3.2926 5.9205 22000 0.3838 3.3563
3.2291 6.1895 23000 0.3840 3.3612
3.2504 6.4587 24000 0.3845 3.3558
3.268 6.7278 25000 0.3856 3.3441
3.2672 6.9970 26000 0.3865 3.3354
3.2013 7.2659 27000 0.3861 3.3429
3.22 7.5351 28000 0.3870 3.3352
3.2162 7.8043 29000 0.3876 3.3296
3.1399 8.0732 30000 0.3875 3.3348
3.1783 8.3424 31000 0.3883 3.3289
3.1991 8.6115 32000 0.3886 3.3223
3.2035 8.8807 33000 0.3892 3.3164
3.1348 9.1497 34000 0.3886 3.3264
3.1709 9.4188 35000 0.3892 3.3198
3.1825 9.6880 36000 0.3899 3.3119
3.1852 9.9571 37000 0.3903 3.3085
3.1129 10.2261 38000 0.3901 3.3170
3.1422 10.4953 39000 0.3905 3.3121
3.1647 10.7644 40000 0.3910 3.3064
3.0573 11.0334 41000 0.3910 3.3119
3.1104 11.3025 42000 0.3911 3.3092
3.1398 11.5717 43000 0.3913 3.3048
3.1452 11.8409 44000 0.3920 3.2982
3.0687 12.1098 45000 0.3916 3.3099
3.098 12.3790 46000 0.3918 3.3046
3.1211 12.6481 47000 0.3923 3.2978
3.1252 12.9173 48000 0.3929 3.2917
3.0567 13.1863 49000 0.3920 3.3073
3.0894 13.4554 50000 0.3927 3.3016
3.101 13.7246 51000 0.3930 3.2926
3.1171 13.9938 52000 0.3935 3.2877
3.0578 14.2627 53000 0.3930 3.3017
3.0831 14.5319 54000 0.3930 3.2959
3.0911 14.8010 55000 0.3938 3.2865
3.0138 15.0700 56000 0.3936 3.3001
3.05 15.3391 57000 0.3934 3.2969
3.0701 15.6083 58000 0.3939 3.2907
3.0819 15.8775 59000 0.3944 3.2846
3.0113 16.1464 60000 0.3939 3.2997
3.051 16.4156 61000 0.3941 3.2935
3.0578 16.6848 62000 0.3946 3.2842
3.065 16.9539 63000 0.3950 3.2802
3.0174 17.2229 64000 0.3943 3.2938
3.0362 17.4920 65000 0.3942 3.2918
3.0606 17.7612 66000 0.3952 3.2804
2.9624 18.0301 67000 0.3944 3.2948
3.0074 18.2993 68000 0.3947 3.2947
3.0378 18.5685 69000 0.3948 3.2866
3.0442 18.8376 70000 0.3956 3.2790
2.9767 19.1066 71000 0.3947 3.2948
2.9956 19.3758 72000 0.3947 3.2918
3.0241 19.6449 73000 0.3956 3.2833
3.036 19.9141 74000 0.3960 3.2782
2.9768 20.1830 75000 0.3952 3.2936
3.007 20.4522 76000 0.3952 3.2892
3.0079 20.7214 77000 0.3958 3.2842
3.0312 20.9905 78000 0.3964 3.2761
2.9761 21.2595 79000 0.3955 3.2894
2.9983 21.5286 80000 0.3957 3.2856
3.0083 21.7978 81000 0.3961 3.2783
2.9479 22.0668 82000 0.3956 3.2914
2.9673 22.3359 83000 0.3955 3.2900
2.9974 22.6051 84000 0.3963 3.2822
2.9881 22.8742 85000 0.3964 3.2775
2.9401 23.1432 86000 0.3959 3.2894
2.9705 23.4124 87000 0.3962 3.2870
2.9741 23.6815 88000 0.3964 3.2801
2.9865 23.9507 89000 0.3970 3.2750
2.9469 24.2196 90000 0.3961 3.2907
2.9622 24.4888 91000 0.3968 3.2826
2.9704 24.7580 92000 0.3968 3.2799
2.9062 25.0269 93000 0.3964 3.2864
2.9439 25.2961 94000 0.3965 3.2879
2.9474 25.5652 95000 0.3969 3.2830
2.966 25.8344 96000 0.3969 3.2790
2.9114 26.1034 97000 0.3964 3.2921
2.9351 26.3725 98000 0.3967 3.2854
2.955 26.6417 99000 0.3972 3.2821
2.9637 26.9109 100000 0.3977 3.2715
2.9047 27.1798 101000 0.3965 3.2911
2.9261 27.4490 102000 0.3970 3.2869
2.9289 27.7181 103000 0.3974 3.2797
2.9416 27.9873 104000 0.3978 3.2729
2.9046 28.2562 105000 0.3970 3.2879
2.9343 28.5254 106000 0.3973 3.2808
2.9354 28.7946 107000 0.3976 3.2798
2.871 29.0635 108000 0.3971 3.2864
2.9031 29.3327 109000 0.3974 3.2867
2.9175 29.6019 110000 0.3976 3.2807
2.9304 29.8710 111000 0.3981 3.2767
2.8797 30.1400 112000 0.3975 3.2886
2.9046 30.4091 113000 0.3975 3.2851
2.9068 30.6783 114000 0.3982 3.2775
2.92 30.9475 115000 0.3982 3.2751
2.8819 31.2164 116000 0.3974 3.2877
2.9002 31.4856 117000 0.3978 3.2845
2.8976 31.7547 118000 0.3983 3.2789
2.8533 32.0237 119000 0.3978 3.2854
2.8772 32.2929 120000 0.3977 3.2869
2.8754 32.5620 121000 3.2869 0.3975
2.889 32.8312 122000 3.2823 0.3980
2.8458 33.1004 123000 3.2873 0.3979
2.8752 33.3696 124000 3.2858 0.3980
2.8878 33.6387 125000 3.2829 0.3982
2.8909 33.9079 126000 3.2790 0.3985
2.8487 34.1768 127000 3.2886 0.3979
2.8584 34.4460 128000 3.2839 0.3983
2.8711 34.7152 129000 3.2790 0.3987
2.8672 34.9843 130000 3.2769 0.3988
2.8497 35.2533 131000 3.2835 0.3985
2.8554 35.5224 132000 3.2841 0.3987
2.8521 35.7916 133000 3.2779 0.3988
2.8176 36.0606 134000 3.2855 0.3984
2.8358 36.3297 135000 3.2839 0.3986
2.8456 36.5989 136000 3.2820 0.3987
2.8367 36.8681 137000 3.2767 0.3991
2.8204 37.1370 138000 3.2842 0.3987
2.8306 37.4062 139000 3.2827 0.3989
2.8403 37.6753 140000 3.2797 0.3991

Framework versions

  • Transformers 4.55.2
  • Pytorch 2.8.0+cu128
  • Datasets 4.0.0
  • Tokenizers 0.21.4