Publish completed WeatherNext Cyclones Mini daily comparison and audited three-model chart

RELEASE_NOTES_TRACKFORMER_1_2.md

@@ -28,14 +28,16 @@ Inference weight SHA-256: `db49f36e85a3766defc4c172746897a1f783705d1ce8e6f9dfb8e
 
 ## Matched development results
 
-| Measure | 1.1 | 1.2 mean of 50 | Matched coverage |
-| --- | ---: | ---: | --- |
-| Mean track error | 798.4 km | 471.2 km | 1,473 daily starts / 270 storms |
-| Direction error | 51.58° | 34.96° | Same daily starts |
-| Central-pressure MAE against JMA | 13.53 hPa | 12.84 hPa | 134 common starts / 40 storms |
+| Measure | 1.1 | 1.2 mean of 50 | DeepMind Mini, one member | Matched coverage |
+| --- | ---: | ---: | ---: | --- |
+| Mean track error | 798.4 km | 471.2 km | 498.4 km | 1,473 daily starts / 270 storms |
+| Direction error | 51.58° | 34.96° | 39.76° | Same daily starts |
+| Central-pressure MAE against JMA | 13.53 hPa | 12.84 hPa | 10.20 hPa | 134 common starts / 40 storms |
 
 Storms receive equal weight after valid leads and daily starts are averaged. The mean pressure reduction is small and its paired whole-storm uncertainty includes no improvement. These repeatedly inspected results are development evidence, not a certified untouched holdout. [Verified common-support metrics](https://github.com/yu314-coder/typhoon-predict/blob/main/evaluation/released_daily/released_daily_benchmark.json).
 
+**The DeepMind comparison is complete and verified:** the official **WeatherNext Cyclones Mini `<2024`** checkpoint (1° resolution, trained through 2023), using **WeatherNext software v0.3.0**, completed all 1,473 daily starts / 270 storms on an RTX 3070 using CUDA. Mini's recent-only mean track error is **288.5 km**, better than 1.2's **477.2 km**. Historical fitting-year overlap, different weather inputs and unequal member counts prevent a general superiority claim. [Completed results and model identity](https://github.com/yu314-coder/typhoon-predict/blob/main/docs/deepmind_daily_benchmark.md) · [CUDA completion receipt](https://github.com/yu314-coder/typhoon-predict/blob/main/evaluation/deepmind_daily/verification.json) · [Independent publication audit](https://github.com/yu314-coder/typhoon-predict/blob/main/evaluation/deepmind_daily/publication_audit.json).
+
 The [model announcement](https://github.com/yu314-coder/typhoon-predict) includes the selected Mangkhut pressure-map animation and architecture. Selected examples are not representative skill. Auxiliary wind and pressure-derived radius diagnostics remain unvalidated; there is no native wind-radius forecast head in 1.2.
 
 This is a research model, not an operational warning service or a safety-critical forecast. Trackformer 1.1 remains a separate release. No historical archive, media, benchmark predictions or training run is modified by this exporter update.

docs/deepmind_daily_benchmark.md

@@ -69,3 +69,5 @@ python release_tools/import_deepmind_release_results.py \
 ```
 
 Completed scores are also exposed by the existing [public benchmark API](https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/api/benchmarks/released). Documentation publication does not replace immutable forecasts, change either model's weights, restart the benchmark, modify active training, or redeploy the website. The original two-model verification receipt remains unchanged; the new publication audit identifies the merged snapshot's exact hash.
+
+The returned worker's `benchmark.json` is retained byte-for-byte for its audit hashes. Its `published_to_site: false` and `integration_note` describe the handoff before import, not the current benchmark status. The completed CUDA receipt, independent publication audit and shared public snapshot above are the current completion/publication evidence; those historical worker fields do not mean the comparison is unfinished.

docs/trackformer_1_2_evaluation.md

@@ -20,6 +20,12 @@ The original track run's 1.2-only central-pressure MAE is **12.62 hPa on 40 stor
 
 The other **1,339 starts lack valid frozen 1.1 issue-time intensity inputs**, so this is not 270-storm pressure coverage. The original pressure-only mask is not substituted into the new common-mask comparison. A basin-wide average does not establish cyclone-core accuracy. The frozen cohort contains 40 recent and 230 historical storms from 1980–1999; retrospective analyses, complete-five-day eligibility and prior model selection limit interpretation.
 
+## Completed DeepMind comparison
+
+Google's official **WeatherNext Cyclones Mini `<2024`**, using **WeatherNext software v0.3.0**, completed the same **1,473 daily starts / 270 storms** on an RTX 3070 with CUDA. Mini is one seeded member, not the full-sized WeatherNext model; 1.2 remains a 50-member mean. The common-support results are **498.4 km** mean track error, **39.76°** direction error and **10.20 hPa** JMA central-pressure MAE on **134 starts / 40 storms**. Its JMA pressure-curve similarity is **0.8088**.
+
+Mini's recent-only mean track error is **288.5 km** versus **477.2 km** for 1.2. Historical fitting-year overlap and different input/member policies limit the combined-cohort comparison; no general superiority claim is justified. The [completed protocol and period breakdowns](deepmind_daily_benchmark.md), [CUDA receipt](../evaluation/deepmind_daily/verification.json) and [independent three-model publication audit](../evaluation/deepmind_daily/publication_audit.json) establish completion. The older two-model receipt remains the original Trackformer audit, not the hash receipt for the subsequently merged snapshot. No missing or unsupported score is treated as zero.
+
 ## Recent pressure-line example
 
 ### Fung-wong MP4: selected route-and-pressure example

evaluation/README.md

@@ -1,12 +1,18 @@
 # Evaluation assets
 
-This directory holds the current published comparison figures, metrics, selected forecast previews and the numerical inputs under [`release_data/`](release_data/). The architecture paper is separate in [`../paper/`](../paper/).
+The current **completed three-model comparison** scores Trackformer 1.1, Trackformer 1.2 and Google's **WeatherNext Cyclones Mini `<2024` (software v0.3.0)** on **1,473 daily starts / 270 Western Pacific storms**, through +120 h. Mini's RTX 3070 CUDA run is complete and verified; Trackformer 1.2 is a mean of 50 members and Mini is one member.
 
-- [270-case comparison and definitions](../docs/trackformer_1_2_evaluation.md): 270 issue times from 90 storms; the 1.2 output is a 50-member input-perturbation mean.
-- [Expanded daily-issue protocol](../docs/daily_storm_benchmark.md): 1,473 daily issues from 270 storms; results are not implied by the existing 270-case charts.
-- [TIP diagnostic](trackformer_1_2_vs_1_1_tip_metrics.json): retain pressure regressions as well as improvements.
-- [Showcase selection](trackformer_1_2_showcase_selection.json): deliberately selected examples, not typical performance or untouched validation.
+- [Announcement chart](released_daily/model_1_2_benchmark.png) and [shared three-model scores](released_daily/released_daily_benchmark.json).
+- [Completed DeepMind results](deepmind_daily/benchmark.json), [CUDA completion receipt](deepmind_daily/verification.json) and [independent publication audit](deepmind_daily/publication_audit.json).
+- [Daily-case protocol](../docs/daily_storm_benchmark.md), [DeepMind model identity and period breakdowns](../docs/deepmind_daily_benchmark.md) and [Trackformer evaluation definitions](../docs/trackformer_1_2_evaluation.md).
+- [Matched pressure, wind and radius evaluation](../docs/intensity_benchmark.md): pressure uses 134 shared daily starts / 40 storms, not all 270 storms. Unsupported scores stay unavailable, never zero.
 
-For geographically aligned routes, use errors between forecasts and observations at the **same valid times on the same map**. Centred/scale-normalized shape similarity alone cannot establish alignment. Never translate, rotate or rescale a forecast to make it match the observed route.
+Scores average valid leads within each day, days within each storm, then storms equally. The historical 1980–1999 group overlaps Mini's fitting years; recent 2024+ results are reported separately. This is development evidence, not a certified untouched holdout or an equal-compute architecture ablation.
 
-From the repository root, reproduce the 270-case figures and Surigae isobars using `python release_tools/plot_release_270.py`. The script defaults to this directory and uses its saved plot inputs; it does not run model inference. Rendering requires NumPy and Matplotlib. The [Fung-wong video builder](../release_tools/build_fung_wong_video.py) likewise uses saved model fields, not synthetic illustrations. Its [provenance](release_data/fung_wong_video.json) reports geographic route alignment separately from pressure error.
+For geographically aligned routes, compare forecasts and observations at the **same valid times on the same map**. Centred shape similarity alone cannot establish alignment. Never translate, rotate or rescale a forecast to make it match the observed route.
+
+Reproduce the current three-model announcement chart from saved results with `python release_tools/plot_model_announcement.py`. This renders audited scores without new inference or weather retrieval; NumPy and Matplotlib are required. The paper is in [`../paper/`](../paper/), and saved pressure arrays and preview provenance are under [`release_data/`](release_data/).
+
+## Archived diagnostics and selected examples
+
+Older figures, the [TIP diagnostic](trackformer_1_2_vs_1_1_tip_metrics.json) and [showcase selection](trackformer_1_2_showcase_selection.json) are preserved for provenance, not presented as the current benchmark. See the [showcase archive](../docs/showcase_archive.md) for selected pressure-map films and their limitations.

models/trackformer_1_2_field/README.md

@@ -78,6 +78,8 @@ No model weights, learned architecture or existing benchmark results changed.
 
 The published 50-member mean is a separate evaluation policy: 50 deterministic seeds perturb only normalized historical basin/regional fields with smooth zero-centred noise, then average routes, central pressures and common-grid fields. `predict.py` computes one clean forecast; do not label it a 50-member mean.
 
-**Matched development evaluation:** the released 1.1 and 1.2 route comparison uses the same frozen **1,473 daily starts / 270 equal-weight storms**, +6 to +120 h. The completed native-pressure comparison has **134 common starts / 40 storms**; the other 1,339 starts lack valid frozen 1.1 intensity inputs and are not zero-scored. The Site defaults to central-pressure MAE: **13.53 / 12.84 hPa** against JMA and **12.84 / 12.55 hPa** against USA (1.1 / 1.2), slight mean improvements with paired whole-storm uncertainty including zero. The optional curve-similarity diagnostic is **(1 + centred cosine) / 2 at exact common valid times**, without time shifting or warping. JMA similarity is **0.7074 / 0.7118**; USA is **0.7237 / 0.6637** on 131 eligible non-flat curves. Removing level/amplitude makes this a shape diagnostic, not proof of correct pressure levels. Actual unshifted hPa timelines, agency masks and uncertainty remain separate. See the [shared snapshot](../../evaluation/released_daily/released_daily_benchmark.json), [verification receipt](../../evaluation/released_daily/released_daily_verification.json), [pressure protocol](../../docs/intensity_benchmark.md) and [public benchmark API](https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/api/benchmarks/released). A new matched WeatherNext Cyclones Mini run is in progress on these exact daily starts; results remain pending. See the [frozen protocol](../../docs/deepmind_daily_benchmark.md); no old scores are transferred.
+**Matched development evaluation:** the released 1.1 and 1.2 route comparison uses the same frozen **1,473 daily starts / 270 equal-weight storms**, +6 to +120 h. The completed native-pressure comparison has **134 common starts / 40 storms**; the other 1,339 starts lack valid frozen 1.1 intensity inputs and are not zero-scored. The Site defaults to central-pressure MAE: **13.53 / 12.84 hPa** against JMA and **12.84 / 12.55 hPa** against USA (1.1 / 1.2), slight mean improvements with paired whole-storm uncertainty including zero. The optional curve-similarity diagnostic is **(1 + centred cosine) / 2 at exact common valid times**, without time shifting or warping. JMA similarity is **0.7074 / 0.7118**; USA is **0.7237 / 0.6637** on 131 eligible non-flat curves. Removing level/amplitude makes this a shape diagnostic, not proof of correct pressure levels. Actual unshifted hPa timelines, agency masks and uncertainty remain separate. See the [shared snapshot](../../evaluation/released_daily/released_daily_benchmark.json), [original Trackformer audit](../../evaluation/released_daily/released_daily_verification.json), [pressure protocol](../../docs/intensity_benchmark.md) and [public benchmark API](https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/api/benchmarks/released).
+
+**Completed DeepMind comparison:** Google's official **WeatherNext Cyclones Mini `<2024`**, run with **WeatherNext software v0.3.0**, completed all **1,473 daily starts / 270 storms** on an **RTX 3070 using CUDA**. This is the 1° Mini checkpoint trained through 2023, not the full-sized WeatherNext model. Mini uses one seeded member; Trackformer 1.2 uses a 50-member mean. On shared support, Mini's mean track error is **498.4 km**, direction error **39.76°**, and JMA central-pressure MAE **10.20 hPa** on **134 starts / 40 storms**. Mini's recent-only track error is **288.5 km**, better than 1.2's **477.2 km**; the combined historical/recent result is not a general superiority claim. Missing outputs are unavailable, never zero errors. [Completed results and model identity](../../docs/deepmind_daily_benchmark.md) · [CUDA completion receipt](../../evaluation/deepmind_daily/verification.json) · [Independent three-model publication audit](../../evaluation/deepmind_daily/publication_audit.json). No older benchmark's DeepMind scores were substituted.
 
 **Pressure-display coverage:** `regional_mslp_hpa` remains the original fixed issue-relative composite. Use the separately exported `core_mslp_hpa` and its per-lead coordinates/mask when the storm moves away from that fixed patch. Do not extend it by inventing a vortex or moving it onto a route. If the moving core leaves supported basin geography, the corresponding masked cells remain unavailable. For an ensemble, register each physical member field onto a common geographic grid **before** averaging; do not average moving-frame array indices. The Mangkhut film uses that separate 50-member policy. Older fixed-patch films retain their original data in [the showcase archive](../../docs/showcase_archive.md). The automatic History archive and its separate core recovery expose geography and coverage through [the public API](https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/history-api).

release_tools/sync_public_model_cards.py

@@ -50,6 +50,7 @@ SYNC_FILES = (
     'docs/trackformer_1_2_evaluation.md',
     'docs/deepmind_daily_benchmark.md',
     'docs/showcase_archive.md',
+    'evaluation/README.md',
     'docs/trackformer_1_2_architecture.svg',
     'docs/trackformer_1_2_mangkhut.gif',
     'docs/trackformer_1_2_mangkhut.gif.json',

release_tools/test_public_model_cards.py

@@ -1,4 +1,5 @@
 """Portable regression checks for complete shared README publication."""
+import json
 import re
 import unittest
 from sync_public_model_cards import CURRENT_FIGURES, SYNC_FILES, MEDIA_REVISION, ROOT, media_url, render_card
@@ -79,6 +80,83 @@ class PublicModelCardsTest(unittest.TestCase):
         for path in SYNC_FILES:
             self.assertTrue((ROOT / path).is_file(), path)
 
+    def test_linked_current_docs_cannot_call_completed_run_pending(self):
+        for name in ('README.md', 'models/trackformer_1_2_field/README.md',
+                     'evaluation/README.md', 'docs/deepmind_daily_benchmark.md',
+                     'docs/daily_storm_benchmark.md', 'docs/trackformer_1_2_evaluation.md',
+                     'RELEASE_NOTES_TRACKFORMER_1_2.md'):
+            with self.subTest(path=name):
+                source = (ROOT / name).read_text()
+                self.assertIn('Mini', source)
+                self.assertIn('1,473', source)
+                self.assertIn('270', source)
+                for stale in ('results remain pending', 'run is in progress',
+                              'scores are **pending**', 'deepmind comparison has not been run'):
+                    self.assertNotIn(stale, source.lower())
+                if name != 'README.md':
+                    self.assertIn(name, SYNC_FILES)
+
+    def assert_metric_row(self, source, label, values, places):
+        rows = [line for line in source.splitlines() if line.startswith('| ')
+                and line.split('|')[1].strip().startswith(label)]
+        self.assertEqual(len(rows), 1, label)
+        cells = [cell.strip() for cell in rows[0].strip().strip('|').split('|')]
+        self.assertEqual(len(cells), 5, label)
+        for index, key in enumerate(('1.1', '1.2', 'deepmind'), 1):
+            match = re.search(r'\d+(?:\.\d+)?', cells[index].replace('**', '').replace(',', ''))
+            self.assertIsNotNone(match, (label, key))
+            self.assertEqual(match.group(), f'{values[key]:.{places}f}', (label, key))
+
+    def test_readme_and_release_tables_match_completed_metrics_not_estimates(self):
+        exported = json.loads((ROOT / 'evaluation/deepmind_daily/benchmark.json').read_text())
+        receipt = json.loads((ROOT / 'evaluation/deepmind_daily/verification.json').read_text())
+        metrics = {row['key']: row['values'] for row in exported['metrics']}
+        self.assertEqual(exported['status'], 'complete_verified')
+        self.assertEqual(receipt['verified_daily_cases'], 1473)
+        self.assertEqual(receipt['verified_storms'], 270)
+        specs = (
+            ('Mean track error, +6 to +120 h', 'mean_track_error_km', 1),
+            ('Six-hour track-direction error', 'direction_error_deg', 2),
+            ('Central-pressure MAE · JMA', 'pressure_JMA_hpa', 2),
+            ('Centred route-shape similarity', 'shape_similarity', 4),
+            ('Pressure-curve similarity · JMA', 'pressure_JMA_curve_similarity', 4),
+        )
+        for source in (self.source, render_card(self.original, self.source)):
+            for label, key, places in specs:
+                with self.subTest(label=label):
+                    self.assert_metric_row(source, label, metrics[key], places)
+        release = (ROOT / 'RELEASE_NOTES_TRACKFORMER_1_2.md').read_text()
+        for label, key, places in (
+            ('Mean track error', 'mean_track_error_km', 1),
+            ('Direction error', 'direction_error_deg', 2),
+            ('Central-pressure MAE against JMA', 'pressure_JMA_hpa', 2),
+        ):
+            self.assert_metric_row(release, label, metrics[key], places)
+
+    def test_recent_group_is_not_confused_with_combined_cohort(self):
+        exported = json.loads((ROOT / 'evaluation/deepmind_daily/benchmark.json').read_text())
+        recent = exported['periods']['recent_2024_onward']['route']
+        section = self.source.split('| Recent storms beginning in 2024+', 1)[1]
+        for label, key, places in (('Mean track error', 'mean_track_error_km', 1),
+                                   ('Six-hour direction error', 'direction_error_deg', 2)):
+            values = {model: recent[model][key]['value'] for model in ('1.1', '1.2', 'deepmind')}
+            # The recent-only table has four columns, unlike the main coverage table.
+            rows = [line for line in section.splitlines() if line.startswith('| ')
+                    and line.split('|')[1].strip().startswith(label)]
+            self.assertEqual(len(rows), 1)
+            row = rows[0].rstrip('|').rstrip() + ' | recent-only |'
+            self.assert_metric_row(row, label, values, places)
+
+    def test_evaluation_index_and_worker_handoff_metadata_are_explained(self):
+        index = (ROOT / 'evaluation/README.md').read_text()
+        self.assertIn('completed three-model comparison', index)
+        self.assertNotIn('270 issue times from 90 storms', index)
+        self.assertNotIn('results are not implied', index)
+        self.assertIn('evaluation/README.md', SYNC_FILES)
+        notes = (ROOT / 'docs/deepmind_daily_benchmark.md').read_text()
+        self.assertIn('retained byte-for-byte', notes)
+        self.assertIn('handoff before import, not the current benchmark status', notes)
+
 
 if __name__ == '__main__':
     unittest.main()