MoWE Paper Releases associated with the MoWE paper written by Tavish, Vardhaman, Keshava, Dr. Deepan. Coded and trained with openlanguagemodel. FAIRC/mowe-mowe-768-19l 0.3B • Updated Aug 6 • 5 FAIRC/mowe-dense-768-19l 0.1B • Updated Aug 6 • 4 FAIRC/mowe-moe-768-19l 0.4B • Updated Aug 6 • 5
Token Averaging A comprehensive research framework for analyzing whether averaging adjacent tokens in a LLM can reduce the compute compared to standard model. FAIRC/token-averaging-matched_mean-avg_50m_k8 Updated Aug 7 • 6 FAIRC/token-averaging-model1_50m Updated Aug 18 • 8 FAIRC/token-averaging-avg_50m_k8_wexp Updated Aug 7 • 5 FAIRC/token-averaging-avg_50m_k8_learnable_pos Updated Aug 7 • 5
Token Averaging A comprehensive research framework for analyzing whether averaging adjacent tokens in a LLM can reduce the compute compared to standard model. FAIRC/token-averaging-matched_mean-avg_50m_k8 Updated Aug 7 • 6 FAIRC/token-averaging-model1_50m Updated Aug 18 • 8 FAIRC/token-averaging-avg_50m_k8_wexp Updated Aug 7 • 5 FAIRC/token-averaging-avg_50m_k8_learnable_pos Updated Aug 7 • 5
MoWE Paper Releases associated with the MoWE paper written by Tavish, Vardhaman, Keshava, Dr. Deepan. Coded and trained with openlanguagemodel. FAIRC/mowe-mowe-768-19l 0.3B • Updated Aug 6 • 5 FAIRC/mowe-dense-768-19l 0.1B • Updated Aug 6 • 4 FAIRC/mowe-moe-768-19l 0.4B • Updated Aug 6 • 5
