Spaces:
Sleeping
Sleeping
Commit History
-
feat: Add FLAN-T5 compatibility with relative position bias
Major changes: - Implement T5RelativePositionBias for encoder/decoder self-attention - T5 uses unscaled attention (no sqrt(d_k) scaling) - Add float32 softmax path for numerical stability - Switch to aot_eager compile backend (inductor causes NaN in decoder backward) - Add gated-gelu activation support for T5 FFN - Fix vocab size handling (32100 vs 32128) - Update model configs for T5-base architecture - Add dev/medium training configs for faster iteration - Optimize training for ~4 min dev runs on RTX 4070 The model now correctly loads FLAN-T5-base weights and generates coherent summaries with proper encoder-decoder architecture.486475d
-
Made necessary .json files available for hugging face hosting
37769a5