dmsmirnov
/

diffusion-qa

constant-warmup

Model card Files Files and versions

diffusion-qa / README.md

dmsmirnov's picture

Upload 2 files

e8b8e01 verified about 1 month ago

|

Raw History Blame Contribute Delete

711 Bytes

license: mit
tags:
  - constant-warmup
  - giant
  - kaiming
  - lamb
  - layernorm
  - linear
  - mixer
  - multitask
  - swish
  - tucker

inference.py

Model Overview

A giant-scale implementation of the mixer architecture, built for multitask tasks.

Architecture

Architecture: mixer
Scale: giant
Attention: linear
Fusion strategy: tucker
Task head: multitask
Activation: swish
Normalization: layernorm
Initialization: kaiming

Training

Optimizer: lamb
LR scheduler: constant warmup

Files

inference.py — main artifact of this repository

License

See the license field above.