--- license: mit tags: - approx-gelu - constant-warmup - generation - grouped-query - huge - hybrid - lamb - scalenorm - trunc-normal - tucker --- # train.py ## Model Overview A **huge**-scale implementation of the **hybrid** architecture, built for **generation** tasks. ## Architecture - **Architecture**: hybrid - **Scale**: huge - **Attention**: grouped query - **Fusion strategy**: tucker - **Task head**: generation - **Activation**: approx gelu - **Normalization**: scalenorm - **Initialization**: trunc normal ## Training - **Optimizer**: lamb - **LR scheduler**: constant warmup ## Files - `train.py` — main artifact of this repository ## License See the license field above.