Download VALIDATION.md from windborne/weathermesh-5c: direct link, hf CLI and curl.
- Browser
- Download file 20.6 kB
-
https://huggingface.co/windborne/weathermesh-5c/resolve/main/VALIDATION.md
- Command line
-
hf download hf://windborne/weathermesh-5c/VALIDATION.md
-
curl -L -o VALIDATION.md https://huggingface.co/windborne/weathermesh-5c/resolve/main/VALIDATION.md
Validation record
2026-08-21 live observation cycle
- Source model commit:
5f41011701988aa84f05279041a6eb2a0b3a3783 - NOAA feed: GDAS
- Download cycle:
2026082106 - AIDA valid time:
2026082107 - GPU host: Baga
- GPU: NVIDIA GeForce RTX 4090, 24 GB
All public-source downloads and processors completed:
| Source | Processed observations | Hourly files |
|---|---|---|
1bamua |
161,669 | 6 |
1bmhs |
1,438,688 | 6 |
atms |
2,320,268 | 6 |
mtiasi |
641,821 | 6 |
gpsro |
5,051 profiles | 6 |
satwnd |
1,964,327 | 6 |
adpupa |
2,493 | 2 |
adpsfc |
51,271 | 6 |
The radiosonde output matched the source processor row for row and value for value, including missing values. The static IASI projection matrix and bias matched the source PCA and scaler calculations exactly. Automated tests also compare every stored schema with its WeatherMesh-5c mesh contract.
RTX 4090 inference
The smoke used a 20,000-observation limit per source. GPSRO supplied 2,520 profiles, radiosonde supplied 194 levels, and WindBorne was empty. The other six enabled source groups supplied 20,000 observations each.
- AIDA output: finite, shape
(1, 12288, 6, 1024) - Three-hour forecast latent: finite, shape
(1, 12288, 6, 1024) - Three-hour global forecast: finite, shape
(1, 720, 1440, 1247) - Peak allocated VRAM: 14.26 GiB
The background was a normalized synthetic tensor. This validation covers the complete public-observation path and model execution, but it does not close the IFS input task.
2026-08-21 inference-only cleanup
The retained source core is 5,904 lines, down from 29,209 lines in the imported core. Direct runtime dependencies fell from 15 to 5. The removed code includes training, losses, evaluation, plotting, checkpoint assembly, credentials, internal server control, cloud control, and unused model variants.
The cleanup was compared with an unmodified numerical reference from repository commit
acd7b7a. A run on Baga used the live processed GDAS observations above, with at most
20,000 observations per source. For the AIDA analysis, three-hour forecast latent, and
global decoded forecast, all 16,384 deterministic samples and all mean, standard
deviation, and maximum-absolute-value statistics matched bit for bit.
The same RTX 4090 run also loaded the checkpoint with strict matching and verified the learned ensemble forker:
- 12-hour ensemble-forker output: finite, shape
(1, 12288, 6, 1024)
2026-08-22 Open IFS adapter
The production adapter downloaded and processed ECMWF Open IFS ensemble member 1 for
cycle 2026082100. Its selected GRIB stream and NOAA OISST file matched the earlier
research downloads byte for byte:
- Open IFS SHA-256:
0525e8bc1d62b5e13884b911dffe02b3ce38c8318759768d2b42f2013351ebbe - OISST SHA-256:
9d61425e85bb46109bf51376d75aeace2ccde30569d2f1f01128577d57ade460
Two independent production processing runs made equal background arrays. Each array was
finite float16 with shape (720, 1440, 137). The production checkpoint then completed
AIDA with empty observations, a three-hour forecast step, and a global decode on one
RTX 4090:
- AIDA output: finite, shape
(1, 12288, 6, 1024) - Three-hour forecast latent: finite, shape
(1, 12288, 6, 1024) - Three-hour global forecast: finite, shape
(1, 720, 1440, 1247) - Peak allocated VRAM: 14.96 GiB
The exact production background, including its open land-sea mask, was also compared with the matched paid member through a 24-hour model rollout. Physical RMSE versus the paid-input forecast was:
| Field | 3 hours | 24 hours |
|---|---|---|
| Z500 height | 0.86 m | 1.35 m |
| T850 | 0.073 K | 0.142 K |
| U850 | 0.103 m/s | 0.241 m/s |
| V850 | 0.105 m/s | 0.230 m/s |
| q850 | 0.066 g/kg | 0.140 g/kg |
| MSLP | 0.113 hPa | 0.159 hPa |
| 10 m U | 0.128 m/s | 0.206 m/s |
| 10 m V | 0.121 m/s | 0.201 m/s |
| 2 m temperature | 0.217 K | 0.285 K |
| 2 m dewpoint | 0.167 K | 0.223 K |
| 100 m U | 0.129 m/s | 0.238 m/s |
| 100 m V | 0.128 m/s | 0.233 m/s |
A second matched rollout used OISST from two days before the IFS cycle, which matches the delay seen in the live test below. It caused no material degradation: 24-hour RMSE was 1.34 m for Z500 height, 0.143 K for T850, and 0.294 K for 2 m temperature.
The latest-cycle path independently discovered and ingested Open IFS cycle 2026082206.
NOAA had not yet published OISST for August 21, so the adapter correctly selected the
latest available field from August 20 and recorded that date. That current background
also completed finite AIDA and three-hour forecast passes on an RTX 4090, with 11.73 GiB
of peak allocated VRAM.
The repository suite has 16 passing tests, including exact member and variable
selection, missing-field rejection, pressure interpolation, humidity positivity,
channel order, legacy vertical expansion, OISST fallback, and SST land masking. The
built wheel's wm5c-background command also processed the fixed cycle and produced an
array equal to the source-tree result.
2026-08-22 end-to-end command
The packaged wm5c-run command used Open IFS ensemble member 1 and all eight public
NOAA sources for cycle 2026082100. Four isolated workers completed raw BUFR decoding
in about 15 minutes. They produced:
| Source | Processed observations | Hourly files |
|---|---|---|
1bamua |
162,040 | 6 |
1bmhs |
1,445,348 | 6 |
atms |
2,317,501 | 6 |
mtiasi |
645,259 | 6 |
gpsro |
5,104 profiles | 6 |
satwnd |
3,312,191 | 6 |
adpupa |
61,590 levels | 4 |
adpsfc |
49,993 | 6 |
A concurrent job claimed GPU 0 during ingestion, so the first model attempt stopped with an out-of-memory error. The command now checks for at least 16 GiB of free memory immediately before model loading. The prepared-input rerun on GPU 1 completed AIDA, one three-hour WeatherMesh-5c step, global decoding, physical-unit conversion, atomic NetCDF output, and manifest output in 41.0 seconds.
- AIDA valid time:
2026082103 - Forecast valid time:
2026082106 - Output dimensions: time 1, pressure 25, latitude 720, longitude 1440
- Output fields: five pressure fields and 12 surface fields
- All output samples: finite
- NetCDF size: 345,819,669 bytes
- NetCDF SHA-256:
f3b3925f67fe8f9556f3d28b0a0b0f80b782a9ac0f0dd525f2d9d50fa1af308c - Peak allocated VRAM: 11.77 GiB
The output preserves the model's de-normalized values without silent physical clamps. This exposes small bound violations that downstream product policy must settle: specific humidity reached -0.000249 kg/kg, total cloud cover ranged from -0.0847 to 1.162, and three-hour precipitation reached -0.0499 mm. These are post-processing questions, not non-finite or tensor-contract failures.
The suite now has 18 passing tests. The two new tests cover command argument safety, atomic output handling, physical-unit conversion, the precipitation inverse transform, CF coordinates, and the full 30-quantile channel order on a small grid.
2026-08-22 environment isolation
The project launcher replaced an inherited external UV_PROJECT_ENVIRONMENT with the
checkout-local .venv, synchronized from the committed lock, and installed PyTorch
2.11 with CUDA 12.8. The resolved Python prefix and both environment variables matched
the checkout-local directory. The launcher also rejected a symlinked environment and
an explicit environment override.
The full suite passed in the isolated environment. The source distribution and wheel also built there without isolated build-dependency installation. The suite now has 22 passing tests, including the launcher and NCEPLIBS-bufr installation guards.
2026-08-22 operational ensemble path
The portable runner now follows the realtime ensemble calculation. A bounded live run used Open IFS member 1, the public observation set, two learned branches at hour 12, and pressure and surface grid output.
- AIDA and both forked latent trajectories completed on one NVIDIA GeForce RTX 4090.
- The member surface file averaged the two learned branches.
- The pressure supermember averaged the same two branches.
- The overall file contained mean and sample standard deviation from both raw paths.
- Surface and pressure samples were finite.
- The member and supermember means matched the overall mean exactly.
- Standard deviation was non-zero for both product families.
- Peak allocated VRAM was 12.91 GiB.
- Inference took 93.8 seconds; total command time was 94.7 seconds.
- The ensemble file was 808,304,374 bytes and recorded two trajectories.
- Temporary disk-backed statistics were removed after successful output.
An independent pre-fork run used distinct Open IFS members 1 and 2. It completed two
AIDA analyses and the surface-only decoder path at hour 3. The ensemble mean matched
the arithmetic mean of the two published members to float32 precision. The sample
standard deviation matched the two-member reference to within 1.2e-5, and the member
forecasts differed materially. Peak allocated VRAM was 11.77 GiB. Inference took 39.5
seconds, including both AIDA passes, and the full command took 59.4 seconds, including
preparation of the second Open IFS background.
The automated ensemble tests verify the learned-fork boundary, coherent branch advancement, repeatable seeding, Welford mean and sample standard deviation, single-member output, ensemble output, pressure fields, surface fields, the operational precipitation expectation, derived wind speed, and trajectory metadata.
2026-08-22 operational efficiency parity
A matched live benchmark used Open IFS member 1, the processed public observations, two learned branches at hour 12, surface leads 12, 15, and 18, and pressure leads 12 and 18. Each run used one NVIDIA GeForce RTX 4090.
| Runner | Total time | Peak allocated VRAM |
|---|---|---|
| Original clean ensemble path | 254.9 s | 12.91 GiB |
| Bounded transfer, serial output | 206.7 s | 12.95 GiB |
| Bounded transfer, four output workers | 106.4 s | 12.95 GiB |
The final path was 2.40 times as fast as the original path, with no material GPU-memory increase. The changes mirror the realtime producer and aggregator: reusable prepared inputs, compact float16 product transfer, a three-slot pinned-memory queue, CPU aggregation that overlaps GPU work, mean-only member stores, and independent output writers.
The serial and parallel writer runs matched for the retained gridded fields. Compared
with the original full-float transfer, the largest normalized RMSE was 2.73e-4, in
three-hour precipitation. The largest absolute difference was 0.0392 mm, also in
three-hour precipitation. These bounded differences come from the realtime float16
transfer representation; the ensemble schedule and output schema did not change.
The suite now has 30 passing tests. The added production checks cover compact transfer, ordered asynchronous consumption, cached-observation isolation, mean-only statistics, and file writing from disk-backed statistic references.
2026-08-23 inference-core cleanup
The production package and tests fell from 12,516 to 10,408 Python lines, a reduction
of 2,108 lines or 16.8 percent. The cleanup removed inactive training branches, unused
model variants, obsolete deterministic output APIs, and source-only metadata. It also
replaced internal project nicknames with model-domain names and moved the retained core
from model_heal to model.
A matched RTX 4090 run used Open IFS member 1, all available public observations, two learned branches at hour 12, surface leads 12, 15, and 18, and pressure leads 12 and 18. Background encoding, the three-hour advance, all six observation encoders, AIDA, and the first forecast step matched the reference tensor hashes exactly.
The cleanup exposed and then preserved the checkpoint's trained radiation-feature convention: UTC hour selects the climatology day and the source hour is zero. The final lock check, source build, wheel build, lint, and 30-test suite passed in the isolated project environment.
2026-08-23 native surface BUFR decoding
The ADPSFC path now uses the pinned NOAA NCEPLIBS-bufr Python interface. It has no
external executable, configuration file, environment variable, or subprocess. A fixed
public GDAS 2026082100 surface BUFR file was processed through both the retired UCAR
decoder and the native path.
- Both paths selected 49,993 reports across the same six observation hours.
- All 13 float32 model-facing columns matched bit for bit, including missing values.
- Atomic storage removed the same three duplicate rows and retained 49,990 rows.
- All 49,990 prepared AIDA surface rows were finite and matched the reference bit for bit.
- The public
wm5c-obs process adpsfccommand completed without decoder-specific options. - A live RTX 4090 smoke run consumed 12,100 native surface rows, completed AIDA and one three-hour WeatherMesh-5c step, and peaked at 11.73 GiB of allocated VRAM.
- The complete isolated test suite passed with 31 tests.
2026-08-23 portable runtime bundle
The production checkpoint and assets were converted into the portable bundle format. The independent verifier checked every manifest SHA-256 and then inspected the safe state dictionary.
| Role | Files | Bytes |
|---|---|---|
| Checkpoint | 1 | 2,841,543,169 |
| Global assets | 8,817 | 37,223,897,436 |
| Total | 8,818 | 40,065,440,605 |
The runtime contains no pickle loader. The bundle manifest test proves that checksum changes fail and that every required file is present.
A global-only smoke test used the production Open IFS background and all available public observations. It completed AIDA, one three-hour forecast step, and the global decoder on one NVIDIA GeForce RTX 4090. All outputs were finite with the expected shapes, and peak allocated VRAM was 14.41 GiB.
The matched run used Open IFS member 1, two learned branches at hour 12, surface leads 12, 15, and 18, and pressure leads 12 and 18. The retained global fields matched the pre-bundle cleanup reference bit for bit.
2026-08-25 Earth2Studio package contract
The package now has a stable 0.1.0 interface for the companion Earth2Studio adapter.
Remote runtime packages materialize into a normal bundle directory without changing the
manifest contract. The materializer installs files atomically, preserves nested paths,
and reuses valid cache entries.
The 37-test isolated suite, lint, source build, and wheel build passed in the dedicated repository environment. The private Earth2Studio lock selected PyTorch 2.13.0 with CUDA 13.0 and installed this package from its pinned Git revision.
A final run used Open IFS member 1 and all eight public NOAA sources for cycle
2026082506. AIDA analyzed the observations at 2026082509, and the learned fork made
two trajectories at 12 and 15 hours. One RTX 5090 completed the workflow in 111.9
seconds with 12.38 GiB of peak allocated memory. The two global NetCDF files contained
41 numeric variables and 317,265,172 checked numeric values; every unmasked value was
finite. Their sizes matched the manifest, and both forecast records reported two
trajectories. The run used the single-output-worker path because unrelated jobs had
filled the host's shared-memory device; the four-process operational default is
unchanged.
2026-08-25 release model boundary
The release model and checkpoint now contain only AIDA, the forecast processor, the
learned ensemble forker, and the gridded global decoder. The checkpoint has 984 keys,
762,997,823 tensor elements, and 2,841,543,169 bytes. Its SHA-256 is
20d6a06cf75f6fa42c8796bc39fa7e426ea08be390a895b6236a6f03aabe73d2.
The bundle verifier checked all 8,818 files and inspected the full state dictionary.
An inference-state audit removed four decoder buffers used only for training losses and
the unused variable-weight JSON file. A repeated two-branch forecast matched all 24
NetCDF variables and 288,232,586 numeric values from the validated reference bit for bit.
A live RTX 4090 smoke test loaded the checkpoint with strict matching, consumed the prepared Open IFS member and public observations, and completed AIDA, one three-hour forecast step, and the global decode. All outputs were finite and had their expected shapes. Peak allocated memory was 13.21 GiB.
A second live run made two learned branches at hour 12 and wrote the gridded surface and pressure ensemble product. It completed in 56.7 seconds with 11.16 GiB of peak allocated memory. The NetCDF file recorded two trajectories and contained 23 numeric variables and 288,232,586 finite, unmasked numeric values. The isolated 35-test suite and lint passed.
2026-08-25 compact release bundle
The Hugging Face release layout packs the runtime assets into 13 files instead of 8,818. The schema-version 2 bundle has 40,068,460,765 bytes:
| Role | Files | Bytes |
|---|---|---|
| Checkpoint | 1 | 2,841,543,169 |
| Global assets | 12 | 37,226,917,596 |
| Total | 13 | 40,068,460,765 |
The independent verifier checked every SHA-256 and inspected all 984 checkpoint entries and 762,997,823 tensor elements. All 8,809 source radiation and solar-angle fields matched their packed float32 slices exactly. A separate comparison checked 16,588,800 derived model-input values, including both sides of the solar shard boundary and UTC hour 23; every float16 value matched the production calculation bit for bit. The hour-23 case also verifies the added radiation field 24 that the earlier portable bundle omitted.
A live RTX 4090 run used Open IFS member 1, the processed public observations, and two
learned branches at hour 12. It completed in 74.2 seconds with 11.16 GiB of peak allocated
memory and wrote a 776,849,543-byte ensemble forecast. The complete NetCDF file matched
the validated pre-packing reference bit for bit at SHA-256
eabe72d21686cce9955c5c52363ba992e39de3b2636125abbd3a119bd664a405.
The isolated 38-test suite, Ruff, lock check, source build, and wheel build passed. The additional test verifies same-file-system cache reuse through the atomic hard-link path.
A fresh authenticated Earth2Studio download from private Hugging Face revision
cbad309a6515bb8957b5e33b415a586a6a8ac3f0 resolved the default package without an
override. It downloaded all 13 files into an empty cache and passed a second complete
checksum pass at manifest SHA-256
9ba96c09c9abb8b260d67186be0995bdfb49d3d2fa200069a7b505ab0b352ab2.
The original copy path used 37 GB for downloads and another 37 GB for the runtime layout
on NFS. Rematerialization through the release hard-link path shared all 14 manifest and
runtime-file inodes; both layouts occupied 37 GB in total.
The downloaded bundle then ran in a new frozen environment on an idle RTX 4090. AIDA
used the prepared Open IFS member and all available processed NOAA observations, and the
learned fork produced two trajectories at 12 hours. The run completed inference in 121.1
seconds with 11.16 GiB of peak allocated memory. Its 776,849,543-byte surface and
pressure NetCDF matched the pre-release numerical reference bit for bit at SHA-256
eabe72d21686cce9955c5c52363ba992e39de3b2636125abbd3a119bd664a405.
2026-08-26 generated temporal features
Schema version 3 removes the stored solar-angle and direct-radiation climatologies. The runtime generates each requested global field from the original production formula and keeps a bounded in-memory cache. The bundle now has 10 runtime files and 3,535,775,581 bytes, down from 40,068,460,765 bytes.
Generated float32 fields matched the stored production arrays exactly for representative days, every file boundary, day 365, and hour 23. Five additional-variable tensors across the leap year compared 20,736,000 float16 values; every value matched the file-backed calculation bit for bit. A generated global solar field took about 0.06 seconds on CPU.
The bundle verifier checked all 10 manifest entries and inspected all 984 checkpoint entries and 762,997,823 tensor elements. The isolated 41-test suite and Ruff passed. A fresh frozen environment used only its repository-local Python, Torch, Pytest, and Ruff; the shared Deep environment was absent from its Python path.
The final RTX 4090 run used Open IFS member 1, the processed public observations, and two
learned branches at hour 12. It completed inference in 122.6 seconds with 11.16 GiB of
peak allocated memory. The 776,849,543-byte surface and pressure NetCDF matched the
release reference bit for bit at SHA-256
eabe72d21686cce9955c5c52363ba992e39de3b2636125abbd3a119bd664a405.