Spaces:
Running
[Join Request] RMS-normalized Block Attention Residuals
Hi! I would like to join Blog Explorers to publish an original technical research note on RMS-normalized Block Attention Residuals.
The article studies a Key–Value mismatch in Block AttnRes and reports matched pretraining experiments at 57M, 157M, 512M, and 4B parameters. At 4B, RMS Block reduces validation perplexity by 4.54% relative to the original Block method and reaches the same validation loss with 20.75% fewer training tokens.
We are preparing a public reproduction repository containing the implementation, configurations, machine-readable results, figures, and evidence mapping. The article clearly distinguishes validation from test results and invites independent 4B+ replication.
A Chinese version of the research note has already been published on Zhihu:
Zhihu article: [ZHIHU_URL]
Reproduction repository: https://github.com/kimmichtank/rms-block-attnres