File size: 20,567 Bytes
3938e59
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
# Validation record

## 2026-08-21 live observation cycle

- Source model commit: `5f41011701988aa84f05279041a6eb2a0b3a3783`
- NOAA feed: GDAS
- Download cycle: `2026082106`
- AIDA valid time: `2026082107`
- GPU host: Baga
- GPU: NVIDIA GeForce RTX 4090, 24 GB

All public-source downloads and processors completed:

| Source | Processed observations | Hourly files |
| --- | ---: | ---: |
| `1bamua` | 161,669 | 6 |
| `1bmhs` | 1,438,688 | 6 |
| `atms` | 2,320,268 | 6 |
| `mtiasi` | 641,821 | 6 |
| `gpsro` | 5,051 profiles | 6 |
| `satwnd` | 1,964,327 | 6 |
| `adpupa` | 2,493 | 2 |
| `adpsfc` | 51,271 | 6 |

The radiosonde output matched the source processor row for row and value for value,
including missing values. The static IASI projection matrix and bias matched the source
PCA and scaler calculations exactly. Automated tests also compare every stored schema
with its WeatherMesh-5c mesh contract.

## RTX 4090 inference

The smoke used a 20,000-observation limit per source. GPSRO supplied 2,520 profiles,
radiosonde supplied 194 levels, and WindBorne was empty. The other six enabled
source groups supplied 20,000 observations each.

- AIDA output: finite, shape `(1, 12288, 6, 1024)`
- Three-hour forecast latent: finite, shape `(1, 12288, 6, 1024)`
- Three-hour global forecast: finite, shape `(1, 720, 1440, 1247)`
- Peak allocated VRAM: 14.26 GiB

The background was a normalized synthetic tensor. This validation covers the complete
public-observation path and model execution, but it does not close the IFS input task.

## 2026-08-21 inference-only cleanup

The retained source core is 5,904 lines, down from 29,209 lines in the imported core.
Direct runtime dependencies fell from 15 to 5. The removed code includes training,
losses, evaluation, plotting, checkpoint assembly, credentials, internal server
control, cloud control, and unused model variants.

The cleanup was compared with an unmodified numerical reference from repository commit
`acd7b7a`. A run on Baga used the live processed GDAS observations above, with at most
20,000 observations per source. For the AIDA analysis, three-hour forecast latent, and
global decoded forecast, all 16,384 deterministic samples and all mean, standard
deviation, and maximum-absolute-value statistics matched bit for bit.

The same RTX 4090 run also loaded the checkpoint with strict matching and verified the
learned ensemble forker:

- 12-hour ensemble-forker output: finite, shape `(1, 12288, 6, 1024)`

## 2026-08-22 Open IFS adapter

The production adapter downloaded and processed ECMWF Open IFS ensemble member 1 for
cycle `2026082100`. Its selected GRIB stream and NOAA OISST file matched the earlier
research downloads byte for byte:

- Open IFS SHA-256: `0525e8bc1d62b5e13884b911dffe02b3ce38c8318759768d2b42f2013351ebbe`
- OISST SHA-256: `9d61425e85bb46109bf51376d75aeace2ccde30569d2f1f01128577d57ade460`

Two independent production processing runs made equal background arrays. Each array was
finite float16 with shape `(720, 1440, 137)`. The production checkpoint then completed
AIDA with empty observations, a three-hour forecast step, and a global decode on one
RTX 4090:

- AIDA output: finite, shape `(1, 12288, 6, 1024)`
- Three-hour forecast latent: finite, shape `(1, 12288, 6, 1024)`
- Three-hour global forecast: finite, shape `(1, 720, 1440, 1247)`
- Peak allocated VRAM: 14.96 GiB

The exact production background, including its open land-sea mask, was also compared
with the matched paid member through a 24-hour model rollout. Physical RMSE versus the
paid-input forecast was:

| Field | 3 hours | 24 hours |
| --- | ---: | ---: |
| Z500 height | 0.86 m | 1.35 m |
| T850 | 0.073 K | 0.142 K |
| U850 | 0.103 m/s | 0.241 m/s |
| V850 | 0.105 m/s | 0.230 m/s |
| q850 | 0.066 g/kg | 0.140 g/kg |
| MSLP | 0.113 hPa | 0.159 hPa |
| 10 m U | 0.128 m/s | 0.206 m/s |
| 10 m V | 0.121 m/s | 0.201 m/s |
| 2 m temperature | 0.217 K | 0.285 K |
| 2 m dewpoint | 0.167 K | 0.223 K |
| 100 m U | 0.129 m/s | 0.238 m/s |
| 100 m V | 0.128 m/s | 0.233 m/s |

A second matched rollout used OISST from two days before the IFS cycle, which matches
the delay seen in the live test below. It caused no material degradation: 24-hour RMSE
was 1.34 m for Z500 height, 0.143 K for T850, and 0.294 K for 2 m temperature.

The latest-cycle path independently discovered and ingested Open IFS cycle `2026082206`.
NOAA had not yet published OISST for August 21, so the adapter correctly selected the
latest available field from August 20 and recorded that date. That current background
also completed finite AIDA and three-hour forecast passes on an RTX 4090, with 11.73 GiB
of peak allocated VRAM.

The repository suite has 16 passing tests, including exact member and variable
selection, missing-field rejection, pressure interpolation, humidity positivity,
channel order, legacy vertical expansion, OISST fallback, and SST land masking. The
built wheel's `wm5c-background` command also processed the fixed cycle and produced an
array equal to the source-tree result.

## 2026-08-22 end-to-end command

The packaged `wm5c-run` command used Open IFS ensemble member 1 and all eight public
NOAA sources for cycle `2026082100`. Four isolated workers completed raw BUFR decoding
in about 15 minutes. They produced:

| Source | Processed observations | Hourly files |
| --- | ---: | ---: |
| `1bamua` | 162,040 | 6 |
| `1bmhs` | 1,445,348 | 6 |
| `atms` | 2,317,501 | 6 |
| `mtiasi` | 645,259 | 6 |
| `gpsro` | 5,104 profiles | 6 |
| `satwnd` | 3,312,191 | 6 |
| `adpupa` | 61,590 levels | 4 |
| `adpsfc` | 49,993 | 6 |

A concurrent job claimed GPU 0 during ingestion, so the first model attempt stopped
with an out-of-memory error. The command now checks for at least 16 GiB of free memory
immediately before model loading. The prepared-input rerun on GPU 1 completed AIDA,
one three-hour WeatherMesh-5c step, global decoding, physical-unit conversion, atomic
NetCDF output, and manifest output in 41.0 seconds.

- AIDA valid time: `2026082103`
- Forecast valid time: `2026082106`
- Output dimensions: time 1, pressure 25, latitude 720, longitude 1440
- Output fields: five pressure fields and 12 surface fields
- All output samples: finite
- NetCDF size: 345,819,669 bytes
- NetCDF SHA-256: `f3b3925f67fe8f9556f3d28b0a0b0f80b782a9ac0f0dd525f2d9d50fa1af308c`
- Peak allocated VRAM: 11.77 GiB

The output preserves the model's de-normalized values without silent physical clamps.
This exposes small bound violations that downstream product policy must settle:
specific humidity reached -0.000249 kg/kg, total cloud cover ranged from -0.0847 to
1.162, and three-hour precipitation reached -0.0499 mm. These are post-processing
questions, not non-finite or tensor-contract failures.

The suite now has 18 passing tests. The two new tests cover command argument safety,
atomic output handling, physical-unit conversion, the precipitation inverse transform,
CF coordinates, and the full 30-quantile channel order on a small grid.

## 2026-08-22 environment isolation

The project launcher replaced an inherited external `UV_PROJECT_ENVIRONMENT` with the
checkout-local `.venv`, synchronized from the committed lock, and installed PyTorch
2.11 with CUDA 12.8. The resolved Python prefix and both environment variables matched
the checkout-local directory. The launcher also rejected a symlinked environment and
an explicit environment override.

The full suite passed in the isolated environment. The source distribution and wheel
also built there without isolated build-dependency installation. The suite now has 22
passing tests, including the launcher and NCEPLIBS-bufr installation guards.

## 2026-08-22 operational ensemble path

The portable runner now follows the realtime ensemble calculation. A bounded live run
used Open IFS member 1, the public observation set, two learned branches at hour 12, and
pressure and surface grid output.

- AIDA and both forked latent trajectories completed on one NVIDIA GeForce RTX 4090.
- The member surface file averaged the two learned branches.
- The pressure supermember averaged the same two branches.
- The overall file contained mean and sample standard deviation from both raw paths.
- Surface and pressure samples were finite.
- The member and supermember means matched the overall mean exactly.
- Standard deviation was non-zero for both product families.
- Peak allocated VRAM was 12.91 GiB.
- Inference took 93.8 seconds; total command time was 94.7 seconds.
- The ensemble file was 808,304,374 bytes and recorded two trajectories.
- Temporary disk-backed statistics were removed after successful output.

An independent pre-fork run used distinct Open IFS members 1 and 2. It completed two
AIDA analyses and the surface-only decoder path at hour 3. The ensemble mean matched
the arithmetic mean of the two published members to float32 precision. The sample
standard deviation matched the two-member reference to within `1.2e-5`, and the member
forecasts differed materially. Peak allocated VRAM was 11.77 GiB. Inference took 39.5
seconds, including both AIDA passes, and the full command took 59.4 seconds, including
preparation of the second Open IFS background.

The automated ensemble tests verify the learned-fork boundary, coherent branch
advancement, repeatable seeding, Welford mean and sample standard deviation,
single-member output, ensemble output, pressure fields, surface fields, the operational
precipitation expectation, derived wind speed, and trajectory metadata.

## 2026-08-22 operational efficiency parity

A matched live benchmark used Open IFS member 1, the processed public observations, two
learned branches at hour 12, surface leads 12, 15, and 18, and pressure leads 12 and 18.
Each run used one NVIDIA GeForce RTX 4090.

| Runner | Total time | Peak allocated VRAM |
| --- | ---: | ---: |
| Original clean ensemble path | 254.9 s | 12.91 GiB |
| Bounded transfer, serial output | 206.7 s | 12.95 GiB |
| Bounded transfer, four output workers | 106.4 s | 12.95 GiB |

The final path was 2.40 times as fast as the original path, with no material GPU-memory
increase. The changes mirror the realtime producer and aggregator: reusable prepared
inputs, compact float16 product transfer, a three-slot pinned-memory queue, CPU
aggregation that overlaps GPU work, mean-only member stores, and independent output
writers.

The serial and parallel writer runs matched for the retained gridded fields. Compared
with the original full-float transfer, the largest normalized RMSE was `2.73e-4`, in
three-hour precipitation. The largest absolute difference was `0.0392 mm`, also in
three-hour precipitation. These bounded differences come from the realtime float16
transfer representation; the ensemble schedule and output schema did not change.

The suite now has 30 passing tests. The added production checks cover compact transfer,
ordered asynchronous consumption, cached-observation isolation, mean-only statistics,
and file writing from disk-backed statistic references.

## 2026-08-23 inference-core cleanup

The production package and tests fell from 12,516 to 10,408 Python lines, a reduction
of 2,108 lines or 16.8 percent. The cleanup removed inactive training branches, unused
model variants, obsolete deterministic output APIs, and source-only metadata. It also
replaced internal project nicknames with model-domain names and moved the retained core
from `model_heal` to `model`.

A matched RTX 4090 run used Open IFS member 1, all available public observations, two
learned branches at hour 12, surface leads 12, 15, and 18, and pressure leads 12 and 18.
Background encoding, the three-hour advance, all six observation encoders, AIDA, and the
first forecast step matched the reference tensor hashes exactly.

The cleanup exposed and then preserved the checkpoint's trained radiation-feature
convention: UTC hour selects the climatology day and the source hour is zero. The final
lock check, source build, wheel build, lint, and 30-test suite passed in the isolated
project environment.

## 2026-08-23 native surface BUFR decoding

The ADPSFC path now uses the pinned NOAA NCEPLIBS-bufr Python interface. It has no
external executable, configuration file, environment variable, or subprocess. A fixed
public GDAS `2026082100` surface BUFR file was processed through both the retired UCAR
decoder and the native path.

- Both paths selected 49,993 reports across the same six observation hours.
- All 13 float32 model-facing columns matched bit for bit, including missing values.
- Atomic storage removed the same three duplicate rows and retained 49,990 rows.
- All 49,990 prepared AIDA surface rows were finite and matched the reference bit for bit.
- The public `wm5c-obs process adpsfc` command completed without decoder-specific options.
- A live RTX 4090 smoke run consumed 12,100 native surface rows, completed AIDA and one
  three-hour WeatherMesh-5c step, and peaked at 11.73 GiB of allocated VRAM.
- The complete isolated test suite passed with 31 tests.

## 2026-08-23 portable runtime bundle

The production checkpoint and assets were converted into the portable bundle format.
The independent verifier checked every manifest SHA-256 and then inspected the safe
state dictionary.

| Role | Files | Bytes |
| --- | ---: | ---: |
| Checkpoint | 1 | 2,841,543,169 |
| Global assets | 8,817 | 37,223,897,436 |
| Total | 8,818 | 40,065,440,605 |

The runtime contains no pickle loader. The bundle manifest test proves that checksum
changes fail and that every required file is present.

A global-only smoke test used the production Open IFS background and all available
public observations. It completed AIDA, one three-hour forecast step, and the global
decoder on one NVIDIA GeForce RTX 4090. All outputs were finite with the expected
shapes, and peak allocated VRAM was 14.41 GiB.

The matched run used Open IFS member 1, two learned branches at hour 12, surface leads
12, 15, and 18, and pressure leads 12 and 18. The retained global fields matched the
pre-bundle cleanup reference bit for bit.

## 2026-08-25 Earth2Studio package contract

The package now has a stable `0.1.0` interface for the companion Earth2Studio adapter.
Remote runtime packages materialize into a normal bundle directory without changing the
manifest contract. The materializer installs files atomically, preserves nested paths,
and reuses valid cache entries.

The 37-test isolated suite, lint, source build, and wheel build passed in the dedicated
repository environment. The private Earth2Studio lock selected PyTorch 2.13.0 with
CUDA 13.0 and installed this package from its pinned Git revision.

A final run used Open IFS member 1 and all eight public NOAA sources for cycle
`2026082506`. AIDA analyzed the observations at `2026082509`, and the learned fork made
two trajectories at 12 and 15 hours. One RTX 5090 completed the workflow in 111.9
seconds with 12.38 GiB of peak allocated memory. The two global NetCDF files contained
41 numeric variables and 317,265,172 checked numeric values; every unmasked value was
finite. Their sizes matched the manifest, and both forecast records reported two
trajectories. The run used the single-output-worker path because unrelated jobs had
filled the host's shared-memory device; the four-process operational default is
unchanged.

## 2026-08-25 release model boundary

The release model and checkpoint now contain only AIDA, the forecast processor, the
learned ensemble forker, and the gridded global decoder. The checkpoint has 984 keys,
762,997,823 tensor elements, and 2,841,543,169 bytes. Its SHA-256 is
`20d6a06cf75f6fa42c8796bc39fa7e426ea08be390a895b6236a6f03aabe73d2`.
The bundle verifier checked all 8,818 files and inspected the full state dictionary.
An inference-state audit removed four decoder buffers used only for training losses and
the unused variable-weight JSON file. A repeated two-branch forecast matched all 24
NetCDF variables and 288,232,586 numeric values from the validated reference bit for bit.

A live RTX 4090 smoke test loaded the checkpoint with strict matching, consumed the
prepared Open IFS member and public observations, and completed AIDA, one three-hour
forecast step, and the global decode. All outputs were finite and had their expected
shapes. Peak allocated memory was 13.21 GiB.

A second live run made two learned branches at hour 12 and wrote the gridded surface and
pressure ensemble product. It completed in 56.7 seconds with 11.16 GiB of peak allocated
memory. The NetCDF file recorded two trajectories and contained 23 numeric variables and
288,232,586 finite, unmasked numeric values. The isolated 35-test suite and lint passed.

## 2026-08-25 compact release bundle

The Hugging Face release layout packs the runtime assets into 13 files instead of 8,818.
The schema-version 2 bundle has 40,068,460,765 bytes:

| Role | Files | Bytes |
| --- | ---: | ---: |
| Checkpoint | 1 | 2,841,543,169 |
| Global assets | 12 | 37,226,917,596 |
| Total | 13 | 40,068,460,765 |

The independent verifier checked every SHA-256 and inspected all 984 checkpoint entries
and 762,997,823 tensor elements. All 8,809 source radiation and solar-angle fields matched
their packed float32 slices exactly. A separate comparison checked 16,588,800 derived
model-input values, including both sides of the solar shard boundary and UTC hour 23;
every float16 value matched the production calculation bit for bit. The hour-23 case also
verifies the added radiation field 24 that the earlier portable bundle omitted.

A live RTX 4090 run used Open IFS member 1, the processed public observations, and two
learned branches at hour 12. It completed in 74.2 seconds with 11.16 GiB of peak allocated
memory and wrote a 776,849,543-byte ensemble forecast. The complete NetCDF file matched
the validated pre-packing reference bit for bit at SHA-256
`eabe72d21686cce9955c5c52363ba992e39de3b2636125abbd3a119bd664a405`.

The isolated 38-test suite, Ruff, lock check, source build, and wheel build passed. The
additional test verifies same-file-system cache reuse through the atomic hard-link path.

A fresh authenticated Earth2Studio download from private Hugging Face revision
`cbad309a6515bb8957b5e33b415a586a6a8ac3f0` resolved the default package without an
override. It downloaded all 13 files into an empty cache and passed a second complete
checksum pass at manifest SHA-256
`9ba96c09c9abb8b260d67186be0995bdfb49d3d2fa200069a7b505ab0b352ab2`.
The original copy path used 37 GB for downloads and another 37 GB for the runtime layout
on NFS. Rematerialization through the release hard-link path shared all 14 manifest and
runtime-file inodes; both layouts occupied 37 GB in total.

The downloaded bundle then ran in a new frozen environment on an idle RTX 4090. AIDA
used the prepared Open IFS member and all available processed NOAA observations, and the
learned fork produced two trajectories at 12 hours. The run completed inference in 121.1
seconds with 11.16 GiB of peak allocated memory. Its 776,849,543-byte surface and
pressure NetCDF matched the pre-release numerical reference bit for bit at SHA-256
`eabe72d21686cce9955c5c52363ba992e39de3b2636125abbd3a119bd664a405`.

## 2026-08-26 generated temporal features

Schema version 3 removes the stored solar-angle and direct-radiation climatologies. The
runtime generates each requested global field from the original production formula and
keeps a bounded in-memory cache. The bundle now has 10 runtime files and 3,535,775,581
bytes, down from 40,068,460,765 bytes.

Generated float32 fields matched the stored production arrays exactly for representative
days, every file boundary, day 365, and hour 23. Five additional-variable tensors across
the leap year compared 20,736,000 float16 values; every value matched the file-backed
calculation bit for bit. A generated global solar field took about 0.06 seconds on CPU.

The bundle verifier checked all 10 manifest entries and inspected all 984 checkpoint
entries and 762,997,823 tensor elements. The isolated 41-test suite and Ruff passed. A
fresh frozen environment used only its repository-local Python, Torch, Pytest, and Ruff;
the shared Deep environment was absent from its Python path.

The final RTX 4090 run used Open IFS member 1, the processed public observations, and two
learned branches at hour 12. It completed inference in 122.6 seconds with 11.16 GiB of
peak allocated memory. The 776,849,543-byte surface and pressure NetCDF matched the
release reference bit for bit at SHA-256
`eabe72d21686cce9955c5c52363ba992e39de3b2636125abbd3a119bd664a405`.