WeatherAI / models /fuxi_ens.md
LonghaoWang's picture
Sync from GitHub 67168d0
09af8e7 verified
|
Raw History Blame Contribute Delete
6.52 kB

FuXi-ENS — native PyTorch (reverse-engineered from the official ONNX; numerically checked vs onnxruntime)

← Model zoo · All model pages · Paper: Science Advances 2025, doi:10.1126/sciadv.adu2854

FuXi-ENS (Science Advances 2025) is a latent-noise ensemble forecaster. The only official artefact is fuxi_ens.onnx (Zenodo 10.5281/zenodo.15124541, CC-BY-NC-4.0 — non-commercial; weights are never committed: download with python scripts/fetch_fuxi_ens.py --out /workspace/ckpt/fuxi_ens, 9.9 GB). weatherai/models/fuxi_ens/model.py is an explicit PyTorch re-implementation whose structure, constants and parameter names were read out of the 26,018-node graph; the official code has no PyTorch definition, so the ONNX graph itself is the oracle.

input [B,2,78,721,1440] (2 steps x (5 vars x 13 levels + 13 surface))  --(x-mean)/std, NaN->0, accumulated channels 73-77 := 0-->  xn
 dist_p  : PatchEmbed (Conv 8x8 on [state;static const] -> LN) -> dropout "drop0" -> layers.0 (7 AdaLN blocks) -> dropout "drop1"
           -> layers.1 (7 blocks) -> final AdaLN (+patch-embed residual) -> ConvTranspose 8x8 heads: mean, logvar     (14 blocks)
 sample  = mean + exp(logvar/2) * eps                                  <- the ensemble noise ("RandomNormalLike")
 decoder.0: z = xn + sample (channels 73-77 := 0) -> PatchEmbed -> 6 layers x 7 AdaLN blocks -> final AdaLN (+residual)
           -> pl_head (65 ch) || sf_head (13 ch)  -> x*std+mean ; tp = exp(clip(.,0,7)) - 1                          (42 blocks)
output [B,2,78,721,1440] = concat(input[:, -1], new state)           (== the ONNX output; inference.py feeds it back as next input)

AdaLN block = adaln = Linear(SiLU(cond)) -> (shift, scale, gate) x 2; x += g1*Attn(LN(x)(1+s1)+b1), x += g2*MLP(LN(x)(1+s2)+b2); cond = MLP(sin/cos(step)) + MLP(hour/24) + MLP(doy/365). Attention: 24 heads x 64, 18x18 windows over the 90x180 token grid, odd blocks cyclic-shifted by 9 with the stored −100 mask, 1-D RoPE on the window-flattened index (not 2-D), q and k scaled by d^-1/4; MLP is a gated GELU (a*gelu(b), fc1 1536→8192 no bias); LayerNorm is the unbiased-variance form (eps 1e-6). All of this was recovered from the graph and then verified numerically (below). 2,474,861,958 parameters + buffers mean/std [78,1,1] and const [6,721,1440] (parameter count equals the ONNX weight elements exactly; the 22 attn_mask initialisers are derived by make_shift_mask and tested equal to the ONNX tensors).

from weatherai.models.fuxi_ens import load_official, FuXiENSNoise, FuXiENS_lite, FuXiENSConfig
model = load_official("/workspace/ckpt/fuxi_ens/fuxi_ens.onnx")          # strict state-dict load, weights memory-mapped (low RAM)
g = torch.Generator().manual_seed(0)
noise = FuXiENSNoise.sample(model.cfg, 1, g)                             # 2 dropout masks + latent normal noise, all explicit
y = model(x, step, hour, doy, noise)                                     # x: [1,2,78,721,1440] physical units; step=t, hour=h/24, doy=min(365,doy)/365
y2 = model(x, step, hour, doy, generator=torch.Generator().manual_seed(1))  # another ensemble member
lite = FuXiENS_lite()                                                    # ~2.1 M params, same code, trainable (see tests)

Random ops in the exported graph (handled): the ONNX has no eval switch — RandomUniformLike (×2: dropout p=0.2 on the patch embedding drop0 and on the output of dist_p.layers.0 drop1, keep-prob 0.8, scale 1/0.8) and RandomNormalLike (latent noise) are always active, so two ONNX runs differ. For a deterministic comparison scripts/fuxi_ens_fixed_noise_onnx.py rewrites the 108 MB graph so these three tensors are graph inputs (no op is changed), and both runtimes receive identical tensors (torch generator seed 1234). FuXiENSNoise is the corresponding explicit input of the PyTorch model; FuXiENS.forward without noise draws all three from a torch.Generator.

Verified (CPU, fp32; scripts/fuxi_ens_ort_segments.py + scripts/fuxi_ens_verify.py; official input.nc 2018-09-13 18Z/2018-09-14 00Z, step=0, fixed noise): one full forward (all 14 + 42 blocks), strict load of all 498 initialisers. Stage-by-stage vs onnxruntime 1.30 (max |Δ| / max |ref|): normalised input 0 (exact), patch embed 5e-7, dist_p layer 0 1.8e-6, dist_p out 1.4e-5, mean/logvar 4.7e-7 / 1.2e-6, sample 1.2e-7, decoder input z 6e-8, decoder layers 0–3 ≤ 1.5e-6, layer 4–5 1.7e-4–2.3e-4 (a few outlier tokens, rel-RMSE 1e-5), decoder out 8e-5. Final output (physical units, all 78 channels × 2 steps): max |Δ| = 1.66 on |values| up to 2.06e5 (geopotential; max |Δ|/max|ref| = 8.1e-6); per-channel max |Δ| / channel-std ≤ 3.3e-3 (precipitation tp, whose exp(clip)-1 amplifies logit errors) and ≤ 1.8e-3 for every other channel; relative RMSE ≤ 1.1e-5 for every channel; NaN (SST land) positions identical; the pass-through step is bit-identical. That is fp32-rounding level for a 2.5 B-parameter network, not bit-exactness.

HF GPU (ZeroGPU Space, NVIDIA RTX PRO 6000 Blackwell MIG 2g.48gb, torch 2.13+cu130; scripts/gpu_smoke.py fuxi_ens): PASS. The 9.9 GB weights were fetched on the Space from Zenodo with scripts/fetch_fuxi_ens.py (parallel HTTP range requests + local inflate: ≈ 430–480 s versus ≈ 3 MB/s ≈ 55 min for a single stream; /tmp has ≈ 3 TB free) and strict-loaded into CPU RAM (0.8 s, memory-mapped); then, inside the 120 s ZeroGPU window, moved to the GPU in fp32 (no reduced precision was needed): 3 s, 11.5 GB resident, 16.0 GB peak, one forward 2.5 s. Same official input.nc and the same fixed noise (torch CPU generator, seed 1234) as the CPU run; compared on the committed stride-12 slice of the onnxruntime output (all 78 channels × both steps): max |Δ| = 0.078 (max |Δ|/channel-std = 5.9e-4, tolerance 1e-2·std); step-0 pass-through equal; a second member with different noise differs. TF32 disabled. The Space also passes the 10 synthetic unit tests (2 file-dependent tests skipped there). Only this single forward step was run on the GPU (no 15-day rollout, no bf16/fp16). Caveat: the Space weights are preloaded in a background thread (fetch_fuxi_ens endpoint) because a redeploy wipes /tmp and downloading does not fit into one 120 s GPU call.