Safetensors
rambley_v2
custom_code

4d696582-ab0b-44b9-b827-01ea7d71d55b

~50M model pretrained on a preliminary new, larger datamix, and midtrained on a larger version of the previous mix.

Tasks Version Filter n-shot Metric Value Stderr
arc_challenge 1 none 0 acc ↑ 0.2116 ± 0.0119
none 0 acc_norm ↑ 0.2577 ± 0.0128
arc_easy 1 none 0 acc ↑ 0.4924 ± 0.0103
none 0 acc_norm ↑ 0.4499 ± 0.0102
hellaswag 1 none 0 acc ↑ 0.3005 ± 0.0046
none 0 acc_norm ↑ 0.3281 ± 0.0047
piqa 1 none 0 acc ↑ 0.6360 ± 0.0112
none 0 acc_norm ↑ 0.6376 ± 0.0112

Current sub-50M champion on the glossia SLM leaderboard!

Screenshot 2026-11-15 at 9.21.59 PM

And 3rd on the AxiomicLabs SLM leaderboard as of my unmerged PR:

image

Architecture Details

  • Natively trained at FP8
  • SwiGLU MLPs
  • Virtual Head Attention with 32 fake query heads and 8 physical query heads (Ding et al., 2026)
  • Headwise gated attention (headwise chosen instead of elementwise to save params for the smaller model families; Qiu et al., 2025)
  • Learned attention sinks on every attention head (various, but mainly OpenAI, 2026, LLM-Core Xiaomi, 2026)
  • SWA:SWA:GQA attention pattern, with the final layer forced to GQA
  • 128 sliding window size (Qiao et al., 2026)
  • 2048 native context length
  • HoPE positional embeddings (on both SWA and GQA layers; Chen et al., 2024)
  • Identity Hyperconnections (涮月亮的谪仙人, 2026)
  • Patch-level pretraining for 60% of the pretraining token budget (Shao et al., 2025; Peng et al., 2026)
  • Optimizers:
    • Muon at 0.02 LR for matrices with Hyperball weight norm constraints (Wen et al., 2026)
    • AdamW at 0.002 LR for embeddings and vectors, norms, etc. with no weight decay or Hyperball
    • AIR LR scheduler (Yu et al., 2026)

Notes on Data

Only clusters 2, 6, 12, and 15 of ClimbMix were used; notably this excludes the mathematics cluster and most of the STEM clusters.

Code

Pretraining codebase: https://code.allura.moe/fizz/lobotomy

Other Benchmarks

  49,718,600 parameters (torch.bfloat16)                                                                                                                                   
  arithmark-3: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:01<00:00, 21.60it/s] 
  arithmark-3: raw 34.90% (349/1000)  normalized 34.90% (349/1000)                                                                                                         
  speed: 674.7 examples/s  (tokenize 0.24s, evaluate 1.48s)                                                                                                                
                                                                                                                                                                           
  Category                                                             N       Raw   Normalized                                                                            
  ----------------------------------------------------------------------------------------------                                                                           
  elementary_school_math_continuation::addition::grades_1_2::easy    128    20.31%       20.31%                                                                            
  elementary_school_math_continuation::comparison::grades_2_3::medium    44    20.45%       20.45%                                                                         
  elementary_school_math_continuation::comparison_difference::grades_2_3::medium    48    27.08%       27.08%                                                              
  elementary_school_math_continuation::data::grades_2_3::easy         43    37.21%       32.56%                                                                            
  elementary_school_math_continuation::division::grades_3_4::medium    54    35.19%       35.19%
  elementary_school_math_continuation::fractions_counting::grades_3_4::medium    50    20.00%       22.00%
  elementary_school_math_continuation::geometry_area::grades_4_5::medium    52    69.23%       71.15%
  elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium    45    46.67%       46.67%
  elementary_school_math_continuation::measurement::grades_2_3::easy    76    32.89%       32.89%
  elementary_school_math_continuation::money::grades_3_4::medium      64    31.25%       31.25%
  elementary_school_math_continuation::multiplication::grades_3_4::medium    74    40.54%       40.54%
  elementary_school_math_continuation::patterns::grades_3_4::medium    53    33.96%       33.96%
  elementary_school_math_continuation::subtraction::grades_1_2::easy   117    23.93%       23.93%
  elementary_school_math_continuation::time::grades_2_3::easy         55   100.00%      100.00%
  elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium    46    23.91%       23.91%
  elementary_school_math_continuation::two_step_addition::grades_2_3::medium    19    10.53%       10.53%
  elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium    32    31.25%       31.25%

====================================================================
  ../exported_hf_2/ (49,718,600 params) RESULTS
====================================================================
  Raw continuation accuracy        34.90%
  Length-normalized accuracy       34.90%
  Primary (acc_norm)           34.90%
====================================================================
Results saved to benchmark_cache/.._exported_hf_2__arithmark-3_results.json
Downloads last month
41
Safetensors
Model size
49.7M params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for glossia/Rambley-V2-Next-50M

Finetuned
(1)
this model

Datasets used to train glossia/Rambley-V2-Next-50M

Collection including glossia/Rambley-V2-Next-50M

Papers for glossia/Rambley-V2-Next-50M