|
Download VALIDATION.md from windborne/weathermesh-5c: direct link, hf CLI and curl.
- Browser
- Download file 20.6 kB
-
https://huggingface.co/windborne/weathermesh-5c/resolve/main/VALIDATION.md
- Command line
-
hf download hf://windborne/weathermesh-5c/VALIDATION.md
-
curl -L -o VALIDATION.md https://huggingface.co/windborne/weathermesh-5c/resolve/main/VALIDATION.md
20.6 kB
| # Validation record | |
| ## 2026-08-21 live observation cycle | |
| - Source model commit: `5f41011701988aa84f05279041a6eb2a0b3a3783` | |
| - NOAA feed: GDAS | |
| - Download cycle: `2026082106` | |
| - AIDA valid time: `2026082107` | |
| - GPU host: Baga | |
| - GPU: NVIDIA GeForce RTX 4090, 24 GB | |
| All public-source downloads and processors completed: | |
| | Source | Processed observations | Hourly files | | |
| | --- | ---: | ---: | | |
| | `1bamua` | 161,669 | 6 | | |
| | `1bmhs` | 1,438,688 | 6 | | |
| | `atms` | 2,320,268 | 6 | | |
| | `mtiasi` | 641,821 | 6 | | |
| | `gpsro` | 5,051 profiles | 6 | | |
| | `satwnd` | 1,964,327 | 6 | | |
| | `adpupa` | 2,493 | 2 | | |
| | `adpsfc` | 51,271 | 6 | | |
| The radiosonde output matched the source processor row for row and value for value, | |
| including missing values. The static IASI projection matrix and bias matched the source | |
| PCA and scaler calculations exactly. Automated tests also compare every stored schema | |
| with its WeatherMesh-5c mesh contract. | |
| ## RTX 4090 inference | |
| The smoke used a 20,000-observation limit per source. GPSRO supplied 2,520 profiles, | |
| radiosonde supplied 194 levels, and WindBorne was empty. The other six enabled | |
| source groups supplied 20,000 observations each. | |
| - AIDA output: finite, shape `(1, 12288, 6, 1024)` | |
| - Three-hour forecast latent: finite, shape `(1, 12288, 6, 1024)` | |
| - Three-hour global forecast: finite, shape `(1, 720, 1440, 1247)` | |
| - Peak allocated VRAM: 14.26 GiB | |
| The background was a normalized synthetic tensor. This validation covers the complete | |
| public-observation path and model execution, but it does not close the IFS input task. | |
| ## 2026-08-21 inference-only cleanup | |
| The retained source core is 5,904 lines, down from 29,209 lines in the imported core. | |
| Direct runtime dependencies fell from 15 to 5. The removed code includes training, | |
| losses, evaluation, plotting, checkpoint assembly, credentials, internal server | |
| control, cloud control, and unused model variants. | |
| The cleanup was compared with an unmodified numerical reference from repository commit | |
| `acd7b7a`. A run on Baga used the live processed GDAS observations above, with at most | |
| 20,000 observations per source. For the AIDA analysis, three-hour forecast latent, and | |
| global decoded forecast, all 16,384 deterministic samples and all mean, standard | |
| deviation, and maximum-absolute-value statistics matched bit for bit. | |
| The same RTX 4090 run also loaded the checkpoint with strict matching and verified the | |
| learned ensemble forker: | |
| - 12-hour ensemble-forker output: finite, shape `(1, 12288, 6, 1024)` | |
| ## 2026-08-22 Open IFS adapter | |
| The production adapter downloaded and processed ECMWF Open IFS ensemble member 1 for | |
| cycle `2026082100`. Its selected GRIB stream and NOAA OISST file matched the earlier | |
| research downloads byte for byte: | |
| - Open IFS SHA-256: `0525e8bc1d62b5e13884b911dffe02b3ce38c8318759768d2b42f2013351ebbe` | |
| - OISST SHA-256: `9d61425e85bb46109bf51376d75aeace2ccde30569d2f1f01128577d57ade460` | |
| Two independent production processing runs made equal background arrays. Each array was | |
| finite float16 with shape `(720, 1440, 137)`. The production checkpoint then completed | |
| AIDA with empty observations, a three-hour forecast step, and a global decode on one | |
| RTX 4090: | |
| - AIDA output: finite, shape `(1, 12288, 6, 1024)` | |
| - Three-hour forecast latent: finite, shape `(1, 12288, 6, 1024)` | |
| - Three-hour global forecast: finite, shape `(1, 720, 1440, 1247)` | |
| - Peak allocated VRAM: 14.96 GiB | |
| The exact production background, including its open land-sea mask, was also compared | |
| with the matched paid member through a 24-hour model rollout. Physical RMSE versus the | |
| paid-input forecast was: | |
| | Field | 3 hours | 24 hours | | |
| | --- | ---: | ---: | | |
| | Z500 height | 0.86 m | 1.35 m | | |
| | T850 | 0.073 K | 0.142 K | | |
| | U850 | 0.103 m/s | 0.241 m/s | | |
| | V850 | 0.105 m/s | 0.230 m/s | | |
| | q850 | 0.066 g/kg | 0.140 g/kg | | |
| | MSLP | 0.113 hPa | 0.159 hPa | | |
| | 10 m U | 0.128 m/s | 0.206 m/s | | |
| | 10 m V | 0.121 m/s | 0.201 m/s | | |
| | 2 m temperature | 0.217 K | 0.285 K | | |
| | 2 m dewpoint | 0.167 K | 0.223 K | | |
| | 100 m U | 0.129 m/s | 0.238 m/s | | |
| | 100 m V | 0.128 m/s | 0.233 m/s | | |
| A second matched rollout used OISST from two days before the IFS cycle, which matches | |
| the delay seen in the live test below. It caused no material degradation: 24-hour RMSE | |
| was 1.34 m for Z500 height, 0.143 K for T850, and 0.294 K for 2 m temperature. | |
| The latest-cycle path independently discovered and ingested Open IFS cycle `2026082206`. | |
| NOAA had not yet published OISST for August 21, so the adapter correctly selected the | |
| latest available field from August 20 and recorded that date. That current background | |
| also completed finite AIDA and three-hour forecast passes on an RTX 4090, with 11.73 GiB | |
| of peak allocated VRAM. | |
| The repository suite has 16 passing tests, including exact member and variable | |
| selection, missing-field rejection, pressure interpolation, humidity positivity, | |
| channel order, legacy vertical expansion, OISST fallback, and SST land masking. The | |
| built wheel's `wm5c-background` command also processed the fixed cycle and produced an | |
| array equal to the source-tree result. | |
| ## 2026-08-22 end-to-end command | |
| The packaged `wm5c-run` command used Open IFS ensemble member 1 and all eight public | |
| NOAA sources for cycle `2026082100`. Four isolated workers completed raw BUFR decoding | |
| in about 15 minutes. They produced: | |
| | Source | Processed observations | Hourly files | | |
| | --- | ---: | ---: | | |
| | `1bamua` | 162,040 | 6 | | |
| | `1bmhs` | 1,445,348 | 6 | | |
| | `atms` | 2,317,501 | 6 | | |
| | `mtiasi` | 645,259 | 6 | | |
| | `gpsro` | 5,104 profiles | 6 | | |
| | `satwnd` | 3,312,191 | 6 | | |
| | `adpupa` | 61,590 levels | 4 | | |
| | `adpsfc` | 49,993 | 6 | | |
| A concurrent job claimed GPU 0 during ingestion, so the first model attempt stopped | |
| with an out-of-memory error. The command now checks for at least 16 GiB of free memory | |
| immediately before model loading. The prepared-input rerun on GPU 1 completed AIDA, | |
| one three-hour WeatherMesh-5c step, global decoding, physical-unit conversion, atomic | |
| NetCDF output, and manifest output in 41.0 seconds. | |
| - AIDA valid time: `2026082103` | |
| - Forecast valid time: `2026082106` | |
| - Output dimensions: time 1, pressure 25, latitude 720, longitude 1440 | |
| - Output fields: five pressure fields and 12 surface fields | |
| - All output samples: finite | |
| - NetCDF size: 345,819,669 bytes | |
| - NetCDF SHA-256: `f3b3925f67fe8f9556f3d28b0a0b0f80b782a9ac0f0dd525f2d9d50fa1af308c` | |
| - Peak allocated VRAM: 11.77 GiB | |
| The output preserves the model's de-normalized values without silent physical clamps. | |
| This exposes small bound violations that downstream product policy must settle: | |
| specific humidity reached -0.000249 kg/kg, total cloud cover ranged from -0.0847 to | |
| 1.162, and three-hour precipitation reached -0.0499 mm. These are post-processing | |
| questions, not non-finite or tensor-contract failures. | |
| The suite now has 18 passing tests. The two new tests cover command argument safety, | |
| atomic output handling, physical-unit conversion, the precipitation inverse transform, | |
| CF coordinates, and the full 30-quantile channel order on a small grid. | |
| ## 2026-08-22 environment isolation | |
| The project launcher replaced an inherited external `UV_PROJECT_ENVIRONMENT` with the | |
| checkout-local `.venv`, synchronized from the committed lock, and installed PyTorch | |
| 2.11 with CUDA 12.8. The resolved Python prefix and both environment variables matched | |
| the checkout-local directory. The launcher also rejected a symlinked environment and | |
| an explicit environment override. | |
| The full suite passed in the isolated environment. The source distribution and wheel | |
| also built there without isolated build-dependency installation. The suite now has 22 | |
| passing tests, including the launcher and NCEPLIBS-bufr installation guards. | |
| ## 2026-08-22 operational ensemble path | |
| The portable runner now follows the realtime ensemble calculation. A bounded live run | |
| used Open IFS member 1, the public observation set, two learned branches at hour 12, and | |
| pressure and surface grid output. | |
| - AIDA and both forked latent trajectories completed on one NVIDIA GeForce RTX 4090. | |
| - The member surface file averaged the two learned branches. | |
| - The pressure supermember averaged the same two branches. | |
| - The overall file contained mean and sample standard deviation from both raw paths. | |
| - Surface and pressure samples were finite. | |
| - The member and supermember means matched the overall mean exactly. | |
| - Standard deviation was non-zero for both product families. | |
| - Peak allocated VRAM was 12.91 GiB. | |
| - Inference took 93.8 seconds; total command time was 94.7 seconds. | |
| - The ensemble file was 808,304,374 bytes and recorded two trajectories. | |
| - Temporary disk-backed statistics were removed after successful output. | |
| An independent pre-fork run used distinct Open IFS members 1 and 2. It completed two | |
| AIDA analyses and the surface-only decoder path at hour 3. The ensemble mean matched | |
| the arithmetic mean of the two published members to float32 precision. The sample | |
| standard deviation matched the two-member reference to within `1.2e-5`, and the member | |
| forecasts differed materially. Peak allocated VRAM was 11.77 GiB. Inference took 39.5 | |
| seconds, including both AIDA passes, and the full command took 59.4 seconds, including | |
| preparation of the second Open IFS background. | |
| The automated ensemble tests verify the learned-fork boundary, coherent branch | |
| advancement, repeatable seeding, Welford mean and sample standard deviation, | |
| single-member output, ensemble output, pressure fields, surface fields, the operational | |
| precipitation expectation, derived wind speed, and trajectory metadata. | |
| ## 2026-08-22 operational efficiency parity | |
| A matched live benchmark used Open IFS member 1, the processed public observations, two | |
| learned branches at hour 12, surface leads 12, 15, and 18, and pressure leads 12 and 18. | |
| Each run used one NVIDIA GeForce RTX 4090. | |
| | Runner | Total time | Peak allocated VRAM | | |
| | --- | ---: | ---: | | |
| | Original clean ensemble path | 254.9 s | 12.91 GiB | | |
| | Bounded transfer, serial output | 206.7 s | 12.95 GiB | | |
| | Bounded transfer, four output workers | 106.4 s | 12.95 GiB | | |
| The final path was 2.40 times as fast as the original path, with no material GPU-memory | |
| increase. The changes mirror the realtime producer and aggregator: reusable prepared | |
| inputs, compact float16 product transfer, a three-slot pinned-memory queue, CPU | |
| aggregation that overlaps GPU work, mean-only member stores, and independent output | |
| writers. | |
| The serial and parallel writer runs matched for the retained gridded fields. Compared | |
| with the original full-float transfer, the largest normalized RMSE was `2.73e-4`, in | |
| three-hour precipitation. The largest absolute difference was `0.0392 mm`, also in | |
| three-hour precipitation. These bounded differences come from the realtime float16 | |
| transfer representation; the ensemble schedule and output schema did not change. | |
| The suite now has 30 passing tests. The added production checks cover compact transfer, | |
| ordered asynchronous consumption, cached-observation isolation, mean-only statistics, | |
| and file writing from disk-backed statistic references. | |
| ## 2026-08-23 inference-core cleanup | |
| The production package and tests fell from 12,516 to 10,408 Python lines, a reduction | |
| of 2,108 lines or 16.8 percent. The cleanup removed inactive training branches, unused | |
| model variants, obsolete deterministic output APIs, and source-only metadata. It also | |
| replaced internal project nicknames with model-domain names and moved the retained core | |
| from `model_heal` to `model`. | |
| A matched RTX 4090 run used Open IFS member 1, all available public observations, two | |
| learned branches at hour 12, surface leads 12, 15, and 18, and pressure leads 12 and 18. | |
| Background encoding, the three-hour advance, all six observation encoders, AIDA, and the | |
| first forecast step matched the reference tensor hashes exactly. | |
| The cleanup exposed and then preserved the checkpoint's trained radiation-feature | |
| convention: UTC hour selects the climatology day and the source hour is zero. The final | |
| lock check, source build, wheel build, lint, and 30-test suite passed in the isolated | |
| project environment. | |
| ## 2026-08-23 native surface BUFR decoding | |
| The ADPSFC path now uses the pinned NOAA NCEPLIBS-bufr Python interface. It has no | |
| external executable, configuration file, environment variable, or subprocess. A fixed | |
| public GDAS `2026082100` surface BUFR file was processed through both the retired UCAR | |
| decoder and the native path. | |
| - Both paths selected 49,993 reports across the same six observation hours. | |
| - All 13 float32 model-facing columns matched bit for bit, including missing values. | |
| - Atomic storage removed the same three duplicate rows and retained 49,990 rows. | |
| - All 49,990 prepared AIDA surface rows were finite and matched the reference bit for bit. | |
| - The public `wm5c-obs process adpsfc` command completed without decoder-specific options. | |
| - A live RTX 4090 smoke run consumed 12,100 native surface rows, completed AIDA and one | |
| three-hour WeatherMesh-5c step, and peaked at 11.73 GiB of allocated VRAM. | |
| - The complete isolated test suite passed with 31 tests. | |
| ## 2026-08-23 portable runtime bundle | |
| The production checkpoint and assets were converted into the portable bundle format. | |
| The independent verifier checked every manifest SHA-256 and then inspected the safe | |
| state dictionary. | |
| | Role | Files | Bytes | | |
| | --- | ---: | ---: | | |
| | Checkpoint | 1 | 2,841,543,169 | | |
| | Global assets | 8,817 | 37,223,897,436 | | |
| | Total | 8,818 | 40,065,440,605 | | |
| The runtime contains no pickle loader. The bundle manifest test proves that checksum | |
| changes fail and that every required file is present. | |
| A global-only smoke test used the production Open IFS background and all available | |
| public observations. It completed AIDA, one three-hour forecast step, and the global | |
| decoder on one NVIDIA GeForce RTX 4090. All outputs were finite with the expected | |
| shapes, and peak allocated VRAM was 14.41 GiB. | |
| The matched run used Open IFS member 1, two learned branches at hour 12, surface leads | |
| 12, 15, and 18, and pressure leads 12 and 18. The retained global fields matched the | |
| pre-bundle cleanup reference bit for bit. | |
| ## 2026-08-25 Earth2Studio package contract | |
| The package now has a stable `0.1.0` interface for the companion Earth2Studio adapter. | |
| Remote runtime packages materialize into a normal bundle directory without changing the | |
| manifest contract. The materializer installs files atomically, preserves nested paths, | |
| and reuses valid cache entries. | |
| The 37-test isolated suite, lint, source build, and wheel build passed in the dedicated | |
| repository environment. The private Earth2Studio lock selected PyTorch 2.13.0 with | |
| CUDA 13.0 and installed this package from its pinned Git revision. | |
| A final run used Open IFS member 1 and all eight public NOAA sources for cycle | |
| `2026082506`. AIDA analyzed the observations at `2026082509`, and the learned fork made | |
| two trajectories at 12 and 15 hours. One RTX 5090 completed the workflow in 111.9 | |
| seconds with 12.38 GiB of peak allocated memory. The two global NetCDF files contained | |
| 41 numeric variables and 317,265,172 checked numeric values; every unmasked value was | |
| finite. Their sizes matched the manifest, and both forecast records reported two | |
| trajectories. The run used the single-output-worker path because unrelated jobs had | |
| filled the host's shared-memory device; the four-process operational default is | |
| unchanged. | |
| ## 2026-08-25 release model boundary | |
| The release model and checkpoint now contain only AIDA, the forecast processor, the | |
| learned ensemble forker, and the gridded global decoder. The checkpoint has 984 keys, | |
| 762,997,823 tensor elements, and 2,841,543,169 bytes. Its SHA-256 is | |
| `20d6a06cf75f6fa42c8796bc39fa7e426ea08be390a895b6236a6f03aabe73d2`. | |
| The bundle verifier checked all 8,818 files and inspected the full state dictionary. | |
| An inference-state audit removed four decoder buffers used only for training losses and | |
| the unused variable-weight JSON file. A repeated two-branch forecast matched all 24 | |
| NetCDF variables and 288,232,586 numeric values from the validated reference bit for bit. | |
| A live RTX 4090 smoke test loaded the checkpoint with strict matching, consumed the | |
| prepared Open IFS member and public observations, and completed AIDA, one three-hour | |
| forecast step, and the global decode. All outputs were finite and had their expected | |
| shapes. Peak allocated memory was 13.21 GiB. | |
| A second live run made two learned branches at hour 12 and wrote the gridded surface and | |
| pressure ensemble product. It completed in 56.7 seconds with 11.16 GiB of peak allocated | |
| memory. The NetCDF file recorded two trajectories and contained 23 numeric variables and | |
| 288,232,586 finite, unmasked numeric values. The isolated 35-test suite and lint passed. | |
| ## 2026-08-25 compact release bundle | |
| The Hugging Face release layout packs the runtime assets into 13 files instead of 8,818. | |
| The schema-version 2 bundle has 40,068,460,765 bytes: | |
| | Role | Files | Bytes | | |
| | --- | ---: | ---: | | |
| | Checkpoint | 1 | 2,841,543,169 | | |
| | Global assets | 12 | 37,226,917,596 | | |
| | Total | 13 | 40,068,460,765 | | |
| The independent verifier checked every SHA-256 and inspected all 984 checkpoint entries | |
| and 762,997,823 tensor elements. All 8,809 source radiation and solar-angle fields matched | |
| their packed float32 slices exactly. A separate comparison checked 16,588,800 derived | |
| model-input values, including both sides of the solar shard boundary and UTC hour 23; | |
| every float16 value matched the production calculation bit for bit. The hour-23 case also | |
| verifies the added radiation field 24 that the earlier portable bundle omitted. | |
| A live RTX 4090 run used Open IFS member 1, the processed public observations, and two | |
| learned branches at hour 12. It completed in 74.2 seconds with 11.16 GiB of peak allocated | |
| memory and wrote a 776,849,543-byte ensemble forecast. The complete NetCDF file matched | |
| the validated pre-packing reference bit for bit at SHA-256 | |
| `eabe72d21686cce9955c5c52363ba992e39de3b2636125abbd3a119bd664a405`. | |
| The isolated 38-test suite, Ruff, lock check, source build, and wheel build passed. The | |
| additional test verifies same-file-system cache reuse through the atomic hard-link path. | |
| A fresh authenticated Earth2Studio download from private Hugging Face revision | |
| `cbad309a6515bb8957b5e33b415a586a6a8ac3f0` resolved the default package without an | |
| override. It downloaded all 13 files into an empty cache and passed a second complete | |
| checksum pass at manifest SHA-256 | |
| `9ba96c09c9abb8b260d67186be0995bdfb49d3d2fa200069a7b505ab0b352ab2`. | |
| The original copy path used 37 GB for downloads and another 37 GB for the runtime layout | |
| on NFS. Rematerialization through the release hard-link path shared all 14 manifest and | |
| runtime-file inodes; both layouts occupied 37 GB in total. | |
| The downloaded bundle then ran in a new frozen environment on an idle RTX 4090. AIDA | |
| used the prepared Open IFS member and all available processed NOAA observations, and the | |
| learned fork produced two trajectories at 12 hours. The run completed inference in 121.1 | |
| seconds with 11.16 GiB of peak allocated memory. Its 776,849,543-byte surface and | |
| pressure NetCDF matched the pre-release numerical reference bit for bit at SHA-256 | |
| `eabe72d21686cce9955c5c52363ba992e39de3b2636125abbd3a119bd664a405`. | |
| ## 2026-08-26 generated temporal features | |
| Schema version 3 removes the stored solar-angle and direct-radiation climatologies. The | |
| runtime generates each requested global field from the original production formula and | |
| keeps a bounded in-memory cache. The bundle now has 10 runtime files and 3,535,775,581 | |
| bytes, down from 40,068,460,765 bytes. | |
| Generated float32 fields matched the stored production arrays exactly for representative | |
| days, every file boundary, day 365, and hour 23. Five additional-variable tensors across | |
| the leap year compared 20,736,000 float16 values; every value matched the file-backed | |
| calculation bit for bit. A generated global solar field took about 0.06 seconds on CPU. | |
| The bundle verifier checked all 10 manifest entries and inspected all 984 checkpoint | |
| entries and 762,997,823 tensor elements. The isolated 41-test suite and Ruff passed. A | |
| fresh frozen environment used only its repository-local Python, Torch, Pytest, and Ruff; | |
| the shared Deep environment was absent from its Python path. | |
| The final RTX 4090 run used Open IFS member 1, the processed public observations, and two | |
| learned branches at hour 12. It completed inference in 122.6 seconds with 11.16 GiB of | |
| peak allocated memory. The 776,849,543-byte surface and pressure NetCDF matched the | |
| release reference bit for bit at SHA-256 | |
| `eabe72d21686cce9955c5c52363ba992e39de3b2636125abbd3a119bd664a405`. | |