Commit History
-
fix bug in fsdp
811726c
-
Update torch-ext/optimizer/muon.py
TaehyunKimUnverified signatureb0230e7 -
Update torch-ext/optimizer/muon.py
TaehyunKimUnverified signatureff2fcfb -
Update muon.py
TaehyunKimUnverified signaturec16b438
-
fix assert in a2a gather scatter
3dafb3e
-
delete state in split_func
15336dc -
change owner_params to owned_params
6943c45 -
modify pre step (overlap step) can get from arsgs
589b763 -
add doc strings + init self rank on init_assign_params
267e8a0
-
license added for flash_muon
d7cd571 -
apply pre-commit hook
fceb334 -
consider multi node
39c42e0
-
misc
35894d1 -
use inpalce op in update_g
6e9baad -
use COMM_DTYPE instead of hardcoded dtype
2a8631f -
apply all2all scatter gather
ff6d675 -
feat(muon_clip) : add muon clip (#6)
Unverified signature* feat(muon_clip) : add muon clip * fix(muon_clip): delete comment * fix(muon_clip): delete comment * fix(muon_clip): considering when nkvheadgroup>1 * docs(muon_clip): refine __init__ docstring and add clip_info argument description * refactor(muon_clip): refactor clip info using dataclass * fix(muon_clip): change min -> new_scaling compare * test(muon): add qk_clip=False case to model comparison * test(muon): show results * fix(muon_clip): change default is muon func * Add built binary [ci skip] --------- Co-authored-by: dongseokmotif <dongseokkim@motiftech.io> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>d65066c
-
feat(muon) : add tuned-abc-values & blfoat16 communication
f7faa93
-
feat: update muon to receive paramgroups, not model (#4)
Unverified signature* feat: update muon to receive paramgroups, not model * feat: update message formats * feat: remove boilerplate assertion * chore: run pre-commits * test: fix testscript * fix: fix misc bugs * feat: add get_default_muon_param_groups helper function * fix: fix readme * fix: raise error if parametergroup does not follow instructions * chore: upload binary --------- Co-authored-by: junhyeok.lee <junhyeok.lee@moreh.io> Co-authored-by: WyldeCat <skan1543@gmail.com>b0f46c7
-
fix(muon): add update_p stage and dealloc tensors properly
99e7c0c
-
chore: add .gitignore
79fc8ba
-
applied lint
db36e39
-
fix(optimizer): resolve bug where weight decay was multiplied by wrong lr value (#5)
Verified signature- fix(optimizer): resolve bug where weight decay was multiplied by wrong lr value (33568272536def07cf6095dadff78ca2d4b182b5)671b033
-
Support HSDP (#4)
Verified signature- feat(muon): support HSDP (dd04b0152725e7b55970171706d5d16f59352cda)8447fd1
-
fix(muon): handle un-distributed env
1f13dae -
refactor(muon): change argument adam_wd to weight_decay and handle params' type
02ac540
-
fix(muon): free tensors that are no longer needed
64757cb -
chore(muon): update comment
036642a -
chore(muon): clean build and update doc
febdf5b
-
fix(muon): delete intermediate tensors immediately to lower peak mem usage
bdd2678 -
chore: initial commit
8535e80