metadata
license: cc-by-4.0
tags:
- adam
- generation
- grouped-query
- low-rank
- mish
- orthogonal
- poolformer
- rmsnorm
- step
- xlarge
model.py
Model Overview
A xlarge-scale implementation of the poolformer architecture, built for generation tasks.
Architecture
- Architecture: poolformer
- Scale: xlarge
- Attention: grouped query
- Fusion strategy: low rank
- Task head: generation
- Activation: mish
- Normalization: rmsnorm
- Initialization: orthogonal
Training
- Optimizer: adam
- LR scheduler: step
Files
model.py— main artifact of this repository
License
See the license field above.