prateekiiest's picture
Add model card
abb17cf verified
|
Raw History Blame Contribute Delete
1.4 kB
metadata
base_model: Qwen/Qwen2.5-0.5B
library_name: transformers
tags:
  - qwen2
  - pretraining
  - mu-greats

muon-gradientMatching

Qwen2.5-0.5B models pretrained from scratch on openwebtext-100k, from the Mu-GREATS subset-selection / Muon gradient-matching experiments. Each run is a subfolder; load one with:

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "efficiency-learning/muon-gradientMatching"
sub = "full_muon_lr0.001"  # any folder below
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub)
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=sub)
Subfolder Optimizer LR Mode Step dtype
full_adamw_lr0.0001 adamw 0.0001 full 6317 ?
full_adamw_lr0.0005 adamw 0.0005 full 6317 ?
full_adamw_lr0.001 adamw 0.001 full 6317 ?
full_adamw_lr0.005 adamw 0.005 full 6317 ?
full_adamw_lr5e-05 adamw 5e-05 full 6317 ?
full_muon_lr0.0001 muon 0.0001 full 6317 ?
full_muon_lr0.0002 muon 0.0002 full 6317 ?
full_muon_lr0.0005 muon 0.0005 full 6317 ?
full_muon_lr0.001 muon 0.001 full 6317 ?
full_muon_lr0.005 muon 0.005 full 6317 ?
full_muon_lr5e-05 muon 5e-05 full 6317 ?
random_muon_lr0.001 muon 0.001 random 6317 ?