IFM/TxT360-v2
Viewer • Updated • 1.84B • 68.9k • 108
~50M model pretrained on a preliminary new, larger datamix, and midtrained on a larger version of the previous mix.
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 0 | acc | ↑ | 0.2116 | ± | 0.0119 |
| none | 0 | acc_norm | ↑ | 0.2577 | ± | 0.0128 | ||
| arc_easy | 1 | none | 0 | acc | ↑ | 0.4924 | ± | 0.0103 |
| none | 0 | acc_norm | ↑ | 0.4499 | ± | 0.0102 | ||
| hellaswag | 1 | none | 0 | acc | ↑ | 0.3005 | ± | 0.0046 |
| none | 0 | acc_norm | ↑ | 0.3281 | ± | 0.0047 | ||
| piqa | 1 | none | 0 | acc | ↑ | 0.6360 | ± | 0.0112 |
| none | 0 | acc_norm | ↑ | 0.6376 | ± | 0.0112 |
Current sub-50M champion on the glossia SLM leaderboard!
And 3rd on the AxiomicLabs SLM leaderboard as of my unmerged PR:
Only clusters 2, 6, 12, and 15 of ClimbMix were used; notably this excludes the mathematics cluster and most of the STEM clusters.
Pretraining codebase: https://code.allura.moe/fizz/lobotomy
49,718,600 parameters (torch.bfloat16)
arithmark-3: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:01<00:00, 21.60it/s]
arithmark-3: raw 34.90% (349/1000) normalized 34.90% (349/1000)
speed: 674.7 examples/s (tokenize 0.24s, evaluate 1.48s)
Category N Raw Normalized
----------------------------------------------------------------------------------------------
elementary_school_math_continuation::addition::grades_1_2::easy 128 20.31% 20.31%
elementary_school_math_continuation::comparison::grades_2_3::medium 44 20.45% 20.45%
elementary_school_math_continuation::comparison_difference::grades_2_3::medium 48 27.08% 27.08%
elementary_school_math_continuation::data::grades_2_3::easy 43 37.21% 32.56%
elementary_school_math_continuation::division::grades_3_4::medium 54 35.19% 35.19%
elementary_school_math_continuation::fractions_counting::grades_3_4::medium 50 20.00% 22.00%
elementary_school_math_continuation::geometry_area::grades_4_5::medium 52 69.23% 71.15%
elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium 45 46.67% 46.67%
elementary_school_math_continuation::measurement::grades_2_3::easy 76 32.89% 32.89%
elementary_school_math_continuation::money::grades_3_4::medium 64 31.25% 31.25%
elementary_school_math_continuation::multiplication::grades_3_4::medium 74 40.54% 40.54%
elementary_school_math_continuation::patterns::grades_3_4::medium 53 33.96% 33.96%
elementary_school_math_continuation::subtraction::grades_1_2::easy 117 23.93% 23.93%
elementary_school_math_continuation::time::grades_2_3::easy 55 100.00% 100.00%
elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium 46 23.91% 23.91%
elementary_school_math_continuation::two_step_addition::grades_2_3::medium 19 10.53% 10.53%
elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium 32 31.25% 31.25%
====================================================================
../exported_hf_2/ (49,718,600 params) RESULTS
====================================================================
Raw continuation accuracy 34.90%
Length-normalized accuracy 34.90%
Primary (acc_norm) 34.90%
====================================================================
Results saved to benchmark_cache/.._exported_hf_2__arithmark-3_results.json
Base model
glossia/Rambley-V2-Next-50M-Pretrain