embedding-lite73 / README.md
kennynakamura's picture
Upload 2 files
898ccf3 verified
|
Raw History Blame Contribute Delete
742 Bytes
metadata
license: mit
tags:
  - approx-gelu
  - constant-warmup
  - generation
  - grouped-query
  - huge
  - hybrid
  - lamb
  - scalenorm
  - trunc-normal
  - tucker

train.py

Model Overview

A huge-scale implementation of the hybrid architecture, built for generation tasks.

Architecture

  • Architecture: hybrid
  • Scale: huge
  • Attention: grouped query
  • Fusion strategy: tucker
  • Task head: generation
  • Activation: approx gelu
  • Normalization: scalenorm
  • Initialization: trunc normal

Training

  • Optimizer: lamb
  • LR scheduler: constant warmup

Files

  • train.py — main artifact of this repository

License

See the license field above.