| ---
|
| license: mit
|
| library_name: pytorch
|
| tags:
|
| - physics
|
| - simulation
|
| - attention-residuals
|
| - gphyt
|
| ---
|
| # GPhyT-S with Block Attention Residuals
|
|
|
| **Study headline: GPhyT + Block Attention Residuals beats GPhyT without Attention Residuals and all other tested baselines on radiative flow.**
|
|
|
| On this study's radiative-flow full test set, GPhyT + Block Attention Residuals has the lowest mean 12-step NMSE among all nine tested trained, initialized, and persistence models: 0.28917. That is 26.38% lower than GPhyT without Attention Residuals (0.39277) and 30.99% lower than persistence (0.41901).
|
|
|
| This is a descriptive mean ranking, emphasized after results, not a win in every seed. Standard/Block use six seeds and Full/Uniform three; initialized controls are post-hoc. The preregistered two-dataset test remains separate and does not establish significance against every comparator.
|
|
|
|
|
| Final fixed-update checkpoints from the paired GPhyT Attention Residuals study by Libor Burian.
|
| The architecture uses twelve layers, width 192, MLP width 768, three heads, four context frames,
|
| 128x64 fields, and five canonical channels. Mode: `block`. Training updates: 3175.
|
|
|
| The [GPhyT without Attention Residuals baseline](https://huggingface.co/BurnyCoder/gphyt-attnres-standard)
|
| contains all six seeds (0–5). It keeps the original GPhyT additive residual connections and
|
| transformer self-attention. The primary Block Attention Residuals comparison uses the same
|
| six seeds, identical shared backbone initialization and paired training examples, AdamW and
|
| learning-rate schedule, effective batch size 16, BF16 autocast, and fixed update count.
|
| The models have matched training procedures and updates, not identical GPU time.
|
| The [baseline table and all-model physics-field examples](https://github.com/BurnyCoder/general-physics-attention-residuals-transformer#readme)
|
| show the measured comparison; Full and Uniform Block use three exploratory seeds.
|
|
|
| ## Use and evaluation
|
|
|
| Clone the [study repository](https://github.com/BurnyCoder/general-physics-attention-residuals-transformer),
|
| install with `uv sync --locked`, and construct `make_model(config, residual_mode, seed)` from
|
| `gphyt_attnres_study.model`. Load `seedN/model.safetensors` with `safetensors.torch.load_file`.
|
| Normalize each active channel using this release's per-dataset `normalization.json`; absent
|
| channels remain zero and must be masked out of metrics and autoregressive feedback.
|
| The CLI evaluation phase and tests specify the exact inference and scoring procedure.
|
|
|
| Each `seedN/evaluation.json`, when available, reports measured one-step, 12-step, and 24-step
|
| errors. `runs.json` retains failed or omitted seeds. No unmeasured improvement is asserted here.
|
| These models were trained from scratch on the preregistered subset of
|
| [The Well](https://arxiv.org/abs/2412.00568), with official trajectory splits and train-only normalization.
|
| They are research surrogates, not numerical solvers or validated predictors for unseen physics.
|
| Coarse spatial resolution, a bounded local compute budget, and six primary seed pairs limit inference.
|
| The completed study finds a relative Block-versus-Standard error reduction, but all trained
|
| variants lose to persistence on convection under the fixed metric, and radiative predictions
|
| frequently have negative density. See the [measured interpretation](https://github.com/BurnyCoder/general-physics-attention-residuals-transformer/blob/main/reports/interpretation.md)
|
| for effect sizes, uncertainty, exploratory controls, and absolute failures.
|
| Buoyancy is a separate channel and is not temperature.
|
| Training used BF16 autocast. Eligible FLA routing contractions inherit autocast despite explicit
|
| FP32 tensor casts; raw Block sums accumulate in the FP32 residual stream. The post-freeze
|
| precision-description disclosure and pre-evaluation intervention correction are documented in
|
| the [architecture audit](https://github.com/BurnyCoder/general-physics-attention-residuals-transformer/blob/main/reports/architecture-audit.md).
|
| All held-out trajectories share their first frame with training at another physical parameter setting.
|
| The official-split evaluation does not establish generalization to entirely unseen initial conditions;
|
| see the repository's `reports/data-preparation.md` and `reports/data_audit.json` for the exact overlap audit.
|
|
|
| ## Sources and attribution
|
|
|
| - [GPhyT paper](https://arxiv.org/abs/2509.13805) and [official MIT code](https://github.com/FloWsnr/General-Physics-Transformer).
|
| - [Attention Residuals](https://arxiv.org/abs/2603.15031) and [FLA MIT operator](https://github.com/fla-org/flash-linear-attention).
|
| - Exact pinned revisions, licenses, reports, and the paper are in the study repository.
|
|
|
| ## Initialization-only baselines (post-hoc)
|
|
|
| 6/6 initialization-only evaluations completed. `untrained/seedN/model.safetensors` contains reconstructed initialized weights with **zero optimizer updates**, not a trained checkpoint. Use `untrained/config.json` and the same top-level normalization statistics. `untrained/seedN/evaluation.json` and `initialization_status.json` preserve exact hashes and measured scores. These user-requested controls were specified after the trained results were known and do not change the original confirmatory comparison. The original constructor and seeds reconstruct the states; historical step-zero files were not saved during training.
|
|
|
| See the [initialized-versus-trained scores and field examples](https://github.com/BurnyCoder/general-physics-attention-residuals-transformer/blob/main/reports/untrained/summary.md) for the complete measured comparison. Initialized Block and Uniform routing are identical for the same seed; do not count them as independent evidence.
|
|
|
| In these post-hoc 12-step comparisons, training lowers radiative-flow error for all four architectures, but increases convection error for Block, Full, and Uniform Block in every tested seed. GPhyT without Attention Residuals improves on both datasets. Final trained-model rankings therefore do not establish improvement over each architecture's own initialized state. These are descriptive findings on the fixed test population.
|
|
|
