prateekiiest commited on
Commit
a64715e
·
verified ·
1 Parent(s): 643153f

Add model card

Browse files
Files changed (1) hide show
  1. README.md +35 -0
README.md ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen2.5-0.5B
3
+ library_name: transformers
4
+ tags:
5
+ - qwen2
6
+ - pretraining
7
+ - mu-greats
8
+ ---
9
+
10
+ # muon-gradientMatching
11
+
12
+ Qwen2.5-0.5B models pretrained from scratch on `openwebtext-100k`, from the Mu-GREATS subset-selection / Muon gradient-matching experiments.
13
+ Each run is a subfolder; load one with:
14
+
15
+ ```python
16
+ from transformers import AutoModelForCausalLM, AutoTokenizer
17
+ repo = "efficiency-learning/muon-gradientMatching"
18
+ sub = "full_muon_lr0.001" # any folder below
19
+ model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub)
20
+ tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=sub)
21
+ ```
22
+
23
+ | Subfolder | Optimizer | LR | Mode | Step | dtype |
24
+ |---|---|---|---|---|---|
25
+ | `full_adamw_lr0.0001` | adamw | 0.0001 | full | 6317 | ? |
26
+ | `full_adamw_lr0.0005` | adamw | 0.0005 | full | 6317 | ? |
27
+ | `full_adamw_lr0.001` | adamw | 0.001 | full | 6317 | ? |
28
+ | `full_adamw_lr0.005` | adamw | 0.005 | full | 6317 | ? |
29
+ | `full_adamw_lr5e-05` | adamw | 5e-05 | full | 6317 | ? |
30
+ | `full_muon_lr0.0001` | muon | 0.0001 | full | 6317 | ? |
31
+ | `full_muon_lr0.0002` | muon | 0.0002 | full | 6317 | ? |
32
+ | `full_muon_lr0.0005` | muon | 0.0005 | full | 6317 | ? |
33
+ | `full_muon_lr0.001` | muon | 0.001 | full | 6317 | ? |
34
+ | `full_muon_lr0.005` | muon | 0.005 | full | 6317 | ? |
35
+ | `full_muon_lr5e-05` | muon | 5e-05 | full | 6317 | ? |