euler314 commited on
Commit
35ff794
·
verified ·
1 Parent(s): 2025f05

Publish verified matched pressure-curve benchmark and revised technical paper

Browse files
.gitattributes CHANGED
@@ -194,3 +194,4 @@ evaluation/storm_structure/mangkhut_wind_radius.png filter=lfs diff=lfs merge=lf
194
  evaluation/storm_structure/meranti_wind_radius.png filter=lfs diff=lfs merge=lfs -text
195
  evaluation/storm_structure/soudelor_wind_radius.png filter=lfs diff=lfs merge=lfs -text
196
  evaluation/release_data/pressure_core/mangkhut/common-pressure.npz filter=lfs diff=lfs merge=lfs -text
 
 
194
  evaluation/storm_structure/meranti_wind_radius.png filter=lfs diff=lfs merge=lfs -text
195
  evaluation/storm_structure/soudelor_wind_radius.png filter=lfs diff=lfs merge=lfs -text
196
  evaluation/release_data/pressure_core/mangkhut/common-pressure.npz filter=lfs diff=lfs merge=lfs -text
197
+ evaluation/released_daily/pressure_comparison.png filter=lfs diff=lfs merge=lfs -text
docs/daily_storm_benchmark.md CHANGED
@@ -2,7 +2,11 @@
2
 
3
  Status: **complete** at 2026-09-29T08:22:39Z. All **1,473 daily cases / 270 storms** finished; the SHA-256 of every saved forecast was verified. [Final equal-storm results and all 270 storm scores](../evaluation/daily_storm_final.json).
4
 
5
- Mean track error is 798.4 km for 1.1 and 471.2 km for 1.2 (50-member mean); +120 h error is 1,646.4 versus 1,031.6 km. Direction error is 51.58° versus 34.96°, centred shape similarity 0.7544 versus 0.8837, and geographic path similarity 0.5345 versus 0.6560. Central-pressure MAE for 1.2 is 12.62 hPa on 40 pressure-labelled storms; basin MSLP MAE is 2.72 hPa on 270 storms. No matched 1.1 pressure result or confidence interval is claimed.
 
 
 
 
6
 
7
  ## What counts as a case and a score
8
 
@@ -33,6 +37,6 @@ Future weather/track/pressure are used for eligibility coverage and scoring only
33
 
34
  ## Pressure and failures
35
 
36
- For 1.2, report central-pressure MAE over valid labels and cosine-area-weighted basin MSLP MAE on the native basin grid, daily then equally per storm. Keep pressure label coverage visible. No matched 1.1 pressure score is fabricated; this run does not establish superiority on pressure. Regional/core field scores need their own valid coverage and are not implied by basin-wide error.
37
 
38
  Completed daily forecasts are saved atomically with hashes. A nonfinite forecast or source mismatch stops the run visibly rather than silently deleting the case. Resume skips completed cases without rerunning them. Partial aggregate scores include only fully completed storms, identify their counts, and must not be presented as the final 270-storm result. Final uncertainty should resample whole storms, not overlapping daily issues independently.
 
2
 
3
  Status: **complete** at 2026-09-29T08:22:39Z. All **1,473 daily cases / 270 storms** finished; the SHA-256 of every saved forecast was verified. [Final equal-storm results and all 270 storm scores](../evaluation/daily_storm_final.json).
4
 
5
+ Mean track error is 798.4 km for 1.1 and 471.2 km for 1.2 (50-member mean); +120 h error is 1,646.4 versus 1,031.6 km. Direction error is 51.58° versus 34.96°, centred shape similarity 0.7544 versus 0.8837, and geographic path similarity 0.5345 versus 0.6560. The original track run reported 1.2-only central-pressure MAE of 12.62 hPa on 40 pressure-labelled storms and basin MSLP MAE of 2.72 hPa on 270 storms. It did not produce 1.1 intensity outputs.
6
+
7
+ The completed [native pressure follow-up](intensity_benchmark.md) now provides a matched 1.1 comparison on 134 daily starts / 40 storms from the same frozen plan. JMA MAE is 13.53 / 12.84 hPa for 1.1 / 1.2; USA MAE is 12.84 / 12.55 hPa. Each uses exact common agency-specific labels. The 1,339 remaining issue inputs are unavailable to the frozen 1.1 intensity pipeline, never zero-scored or filled with future observations. Paired whole-storm intervals include zero. These are not all-270-storm pressure scores.
8
+
9
+ On 1 October, `release_tools/build_release_benchmark.py` independently rechecked all forecast hashes, coordinate conversions and daily/storm route aggregates, reproduced the native pressure postprocessor and generated the [shared Site snapshot](../evaluation/released_daily/released_daily_benchmark.json) and [audit receipt](../evaluation/released_daily/released_daily_verification.json). No inference or weight changes were necessary. DeepMind has not been evaluated on this larger daily plan; an older cohort's score cannot be transferred into it.
10
 
11
  ## What counts as a case and a score
12
 
 
37
 
38
  ## Pressure and failures
39
 
40
+ For the original 1.2 run, central-pressure MAE uses valid labels and cosine-area-weighted basin MSLP MAE uses the native basin grid, daily then equally per storm. Keep pressure label coverage visible. The separate native-pressure follow-up uses a paired mask for both models at each agency's exact valid times; it does not substitute its smaller coverage into all-storm track results. Regional/core field scores need their own valid coverage and are not implied by basin-wide error. No same-cohort 1.1 basin-field score is claimed.
41
 
42
  Completed daily forecasts are saved atomically with hashes. A nonfinite forecast or source mismatch stops the run visibly rather than silently deleting the case. Resume skips completed cases without rerunning them. Partial aggregate scores include only fully completed storms, identify their counts, and must not be presented as the final 270-storm result. Final uncertainty should resample whole storms, not overlapping daily issues independently.
docs/intensity_benchmark.md CHANGED
@@ -63,6 +63,8 @@ all counts are printed. Observational RMW has reporting/provisional limitations.
63
 
64
  ## Curve comparison, analogous to route direction and shape
65
 
 
 
66
  Magnitude MAE is retained alongside time-curve diagnostics, without optimizing
67
  time lag, warping time, shifting observations or adjusting forecasts to truth.
68
  These are separate, versioned post-hoc descriptive metrics, not a preregistered
 
63
 
64
  ## Curve comparison, analogous to route direction and shape
65
 
66
+ The released Site defaults to **central-pressure MAE**, with slight descriptive mean improvements (JMA 13.53 / 12.84 hPa; USA 12.84 / 12.55 hPa, for 1.1 / 1.2) and paired uncertainty including zero. Its secondary **similarity** view is the exact complement of the shape-error metric below: `(1 + centred cosine) / 2 = 1 − shape_error`. This is a presentation transform, not a new fit or score. JMA similarity is 0.7074 / 0.7118 on 40 storms and 134 starts; USA similarity is 0.7237 / 0.6637 on 40 storms and 131 non-flat starts. Unshifted physical hPa timelines remain visible separately. The public snapshot includes every input-eligible pressure timeline.
67
+
68
  Magnitude MAE is retained alongside time-curve diagnostics, without optimizing
69
  time lag, warping time, shifting observations or adjusting forecasts to truth.
70
  These are separate, versioned post-hoc descriptive metrics, not a preregistered
docs/trackformer_1_2_evaluation.md CHANGED
@@ -16,7 +16,9 @@ Trackformer 1.2 uses the mean of 50 distinct causal input perturbations, not 50
16
 
17
  Direction, centred shape, geographic path and same-time position error measure different aspects of a forecast. A translated or mistimed route can have high shape similarity without overlapping the observed route. No combined weighted ranking is introduced.
18
 
19
- The 1.2 central-pressure MAE is **12.62 hPa on 40 storms with valid labels**; area-weighted basin MSLP MAE is **2.72 hPa on 270 storms**. No matched 1.1 pressure output is available. A basin-wide average does not establish cyclone-core accuracy. The frozen cohort contains 40 recent and 230 historical storms from 1980–1999; retrospective analyses, complete-five-day eligibility and prior model selection limit interpretation.
 
 
20
 
21
  ## Recent pressure-line example
22
 
 
16
 
17
  Direction, centred shape, geographic path and same-time position error measure different aspects of a forecast. A translated or mistimed route can have high shape similarity without overlapping the observed route. No combined weighted ranking is introduced.
18
 
19
+ The original track run's 1.2-only central-pressure MAE is **12.62 hPa on 40 storms with valid labels**; area-weighted basin MSLP MAE is **2.72 hPa on 270 storms**. The completed native 1.1 follow-up now supplies a matched pressure comparison on **134 common daily starts / 40 storms** from the same frozen plan. The default pressure chart is MAE: JMA **13.53 / 12.84 hPa**, USA **12.84 / 12.55 hPa** for 1.1 / 1.2, with paired uncertainty including zero. The optional same-time pressure-curve similarity diagnostic uses **(1 + centred cosine) / 2**, with no lag optimization or time warping: JMA scores are **0.7074 / 0.7118**, while USA scores are **0.7237 / 0.6637** on 131 non-flat eligible starts. Undefined curves remain unavailable, not zero. Both paired whole-storm similarity-difference intervals also include zero. This shape metric removes level and amplitude, so inspect the actual hPa overlays and MAE separately. [Shared verified snapshot](../evaluation/released_daily/released_daily_benchmark.json) · [Pressure protocol](intensity_benchmark.md) · [Public API](https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/api/benchmarks/released).
20
+
21
+ The other **1,339 starts lack valid frozen 1.1 issue-time intensity inputs**, so this is not 270-storm pressure coverage. The original pressure-only mask is not substituted into the new common-mask comparison. A basin-wide average does not establish cyclone-core accuracy. The frozen cohort contains 40 recent and 230 historical storms from 1980–1999; retrospective analyses, complete-five-day eligibility and prior model selection limit interpretation.
22
 
23
  ## Recent pressure-line example
24
 
evaluation/released_daily/pressure_comparison.png ADDED

Git LFS Details

  • SHA256: c19ee5ab8f36ea4630f110db5058a89549927cb98c0f26eb1e69d57fe132932f
  • Pointer size: 131 Bytes
  • Size of remote file: 218 kB
evaluation/released_daily/released_daily_benchmark.json ADDED
The diff for this file is too large to render. See raw diff
 
evaluation/released_daily/released_daily_verification.json ADDED
The diff for this file is too large to render. See raw diff
 
evaluation/trackformer_1_2_vs_1_1_270_storms_bars.png CHANGED

Git LFS Details

  • SHA256: 053110494b50bb92dd1c40a806530d1c557befcc92fb229276489d5dec8c3094
  • Pointer size: 131 Bytes
  • Size of remote file: 169 kB

Git LFS Details

  • SHA256: c389511009ccf50c737bd40d1f4185c0c315b97f49a439be1c193570d4291954
  • Pointer size: 131 Bytes
  • Size of remote file: 173 kB
models/trackformer_1_2_field/README.md CHANGED
@@ -45,4 +45,6 @@ No model weights, learned architecture or existing benchmark results changed.
45
 
46
  The published 50-member mean is a separate evaluation policy: 50 deterministic seeds perturb only normalized historical basin/regional fields with smooth zero-centred noise, then average routes, central pressures and common-grid fields. `predict.py` computes one clean forecast; do not label it a 50-member mean.
47
 
 
 
48
  **Pressure-display coverage:** `regional_mslp_hpa` is the original fixed issue-relative composite, not an automatically extended moving-core map. When the forecast centre leaves that patch, a scalar central-pressure readout must not be inserted into the image to make it appear consistent. The corrected Mangkhut film separately exports the actual moving model cores and registers member fields before averaging; [the current model card](https://huggingface.co/euler314/typhoon-predict) links the verified pressure arrays and shows which other films retain their older fixed patches. The automatic History archive and its separate core recovery expose exact geography and coverage through [the public API](https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/history-api).
 
45
 
46
  The published 50-member mean is a separate evaluation policy: 50 deterministic seeds perturb only normalized historical basin/regional fields with smooth zero-centred noise, then average routes, central pressures and common-grid fields. `predict.py` computes one clean forecast; do not label it a 50-member mean.
47
 
48
+ **Matched development evaluation:** the released 1.1 and 1.2 route comparison uses the same frozen **1,473 daily starts / 270 equal-weight storms**, +6 to +120 h. The completed native-pressure comparison has **134 common starts / 40 storms**; the other 1,339 starts lack valid frozen 1.1 intensity inputs and are not zero-scored. The Site defaults to central-pressure MAE: **13.53 / 12.84 hPa** against JMA and **12.84 / 12.55 hPa** against USA (1.1 / 1.2), slight mean improvements with paired whole-storm uncertainty including zero. The optional curve-similarity diagnostic is **(1 + centred cosine) / 2 at exact common valid times**, without time shifting or warping. JMA similarity is **0.7074 / 0.7118**; USA is **0.7237 / 0.6637** on 131 eligible non-flat curves. Removing level/amplitude makes this a shape diagnostic, not proof of correct pressure levels. Actual unshifted hPa timelines, agency masks and uncertainty remain separate. See the [shared snapshot](../../evaluation/released_daily/released_daily_benchmark.json), [verification receipt](../../evaluation/released_daily/released_daily_verification.json), [pressure protocol](../../docs/intensity_benchmark.md) and [public benchmark API](https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/api/benchmarks/released). DeepMind Mini has not run on this daily cohort; no old score is transferred into it.
49
+
50
  **Pressure-display coverage:** `regional_mslp_hpa` is the original fixed issue-relative composite, not an automatically extended moving-core map. When the forecast centre leaves that patch, a scalar central-pressure readout must not be inserted into the image to make it appear consistent. The corrected Mangkhut film separately exports the actual moving model cores and registers member fields before averaging; [the current model card](https://huggingface.co/euler314/typhoon-predict) links the verified pressure arrays and shows which other films retain their older fixed patches. The automatic History archive and its separate core recovery expose exact geography and coverage through [the public API](https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/history-api).
paper/trackformer.pdf CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:d50499cb0a55b190cfccb9d97739cf055500c215ac22b16d0d4fdc9cf99ab45f
3
- size 416129
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ac371a8d9a577f352cfaaeb322c8e1f4803f815f4727cbf234977cebfd1e1f45
3
+ size 431222
paper/trackformer.tex CHANGED
@@ -39,7 +39,7 @@
39
  \begin{document}
40
  \fontsize{10.3}{13}\selectfont
41
  \thispagestyle{empty}
42
- {\sffamily\small\color{modelteal}RESEARCH TECHNICAL REPORT\hfill 29 SEPTEMBER 2026\par}
43
  \vspace{13pt}
44
  {\sffamily\bfseries\color{ink}\fontsize{26}{29}\selectfont Trackformer 1.2\par}
45
  \vspace{4pt}
@@ -62,6 +62,9 @@ This report specifies the released architecture, state updates, learning
62
  objectives and 50-member input-perturbation policy. Across 1,473 daily forecasts
63
  from 270 storms, equal-storm mean track error is 471.2 km for the 1.2 ensemble
64
  and 798.4 km for the causal 1.1 route pipeline.
 
 
 
65
  The pipelines differ; this is not an architecture ablation or an untouched-test
66
  result. A selected Fung-wong case illustrates actual geographic alignment,
67
  intensity evolution and model-generated pressure contours, including their
@@ -460,10 +463,171 @@ look similar; the shape score cannot replace geographic error or an unshifted
460
  overlay. Path similarity responds to displacement but is not a strict
461
  same-time distance.
462
 
463
- \keyline{Pressure coverage.}{The 1.2 central-pressure MAE is 12.62 hPa on
464
- 40 storms with valid labels; basin MSLP MAE is 2.72 hPa on 270 storms.
465
- No matched 1.1 pressure comparison is available. The SHA-256 hashes of all
466
- 1,473 saved daily forecasts were verified.}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
467
  \reportpage
468
  \section{Fung-wong: route alignment and intensity}
469
  The selected forecast begins at 00 UTC on 7 November 2025 and uses the saved
@@ -676,32 +840,30 @@ animates the saved fields; no additional inference or route adjustment is used.
676
  \label{fig:pressuremap}
677
  \end{figure}
678
  \subsection{Intensity labels and map coverage}
679
- Pressure truth is JMA best track from IBTrACS \texttt{TOKYO\_PRES}, not a JMA
680
- forecast. Native-detail history was unavailable; $0.25^\circ$ output sampling
681
- does not establish native resolution. From +66 h the centre leaves the fixed
682
- regional patch. The video then shows only the saved coarse basin field there,
683
- while the moving-core readout supplies central pressure. No detailed core is
684
- invented beyond coverage.
685
-
686
- There is no verified matched 1.1 pressure array for this benchmark. On the
687
  repeatedly inspected TIP ten-start diagnostic, 1.2 pressure MAE is worse
688
  (30.7 versus 20.1 hPa). Those overlapping starts are not independent tests;
689
  route gains do not establish an all-metric improvement.
690
 
691
  \subsection{What remains unverified}
692
- The completed cohort includes 40 recent and 230 historical storms (1980--1999),
693
- outside this checkpoint's fitting and validation years but not certified unused
694
- across prior experiments. Complete five-day truth excludes short remaining
695
- lifetimes. Historical hindcasts use a model trained on later years and
696
- retrospective analyses. Generalization still requires an unused whole-storm
697
- holdout, training-only preprocessing and whole-storm uncertainty resampling.
698
 
699
  \subsection{Conclusion}
700
- Trackformer 1.2 links recurrent environmental evolution to a translating
701
- pressure anomaly and coupled geographic readout. Its development results
702
- improve aggregate route measures over the archived 1.1 pipeline, but leave
703
- large absolute errors and intensity failures; the older cohort also showed
704
- a short-lead regression. The
705
- released model is a reproducible research artifact, not a substitute for
706
  official warnings.
707
  \end{document}
 
39
  \begin{document}
40
  \fontsize{10.3}{13}\selectfont
41
  \thispagestyle{empty}
42
+ {\sffamily\small\color{modelteal}RESEARCH TECHNICAL REPORT\hfill REVISED 1 OCTOBER 2026\par}
43
  \vspace{13pt}
44
  {\sffamily\bfseries\color{ink}\fontsize{26}{29}\selectfont Trackformer 1.2\par}
45
  \vspace{4pt}
 
62
  objectives and 50-member input-perturbation policy. Across 1,473 daily forecasts
63
  from 270 storms, equal-storm mean track error is 471.2 km for the 1.2 ensemble
64
  and 798.4 km for the causal 1.1 route pipeline.
65
+ On 134 matched daily starts from 40 storms, same-time JMA pressure MAE
66
+ is 13.53 and 12.84 hPa for 1.1 and 1.2, respectively; paired
67
+ whole-storm uncertainty includes zero improvement.
68
  The pipelines differ; this is not an architecture ablation or an untouched-test
69
  result. A selected Fung-wong case illustrates actual geographic alignment,
70
  intensity evolution and model-generated pressure contours, including their
 
463
  overlay. Path similarity responds to displacement but is not a strict
464
  same-time distance.
465
 
466
+ \keyline{Verified same benchmark.}{All 1,473 original daily forecasts were
467
+ hash-checked again on 1 October. Absolute routes were converted to the same
468
+ issue-centred kilometre projection, and every daily and equal-storm route score
469
+ was reproduced. The native-pressure comparison below uses a common subset of
470
+ this same frozen plan, not a different storm sample. DeepMind Mini has not been
471
+ run on this daily plan; its older scores are not transferred.}
472
+ \reportpage
473
+ \section{Matched central-pressure comparison}
474
+ \subsection{Common starts, agency labels and missing inputs}
475
+ The frozen 1.1 intensity ensemble requires valid issue-time wind and central
476
+ pressure. These inputs are present in \textbf{134 daily starts from 40 storms}
477
+ within the 1,473-start plan. The remaining \textbf{1,339 starts are unavailable},
478
+ not zero-scored and not replaced with later observations. The 1.1 primary,
479
+ structure and gated temporal experts retain their released calibration.
480
+ For 1.2, all 134 fifty-member causal replays reproduce the originally saved
481
+ central-pressure means exactly (maximum difference 0.0 hPa).
482
+
483
+ JMA pressure uses \texttt{TOKYO\_PRES}; USA pressure uses \texttt{USA\_PRES}
484
+ from the same IBTrACS storm identifiers at issue time plus the exact forecast
485
+ lead. Agencies remain separate. Each comparison uses the intersection of valid
486
+ truth and both model outputs: lead errors average within a day, days within a
487
+ storm, then storms equally. Missing observations remain missing. This preserves
488
+ the track benchmark's hierarchy without claiming 270-storm intensity coverage.
489
+
490
+ The default pressure chart uses same-time central-pressure MAE in hPa.
491
+ JMA mean error decreases by approximately 5.1\%; USA decreases by 2.2\%.
492
+ These are slight descriptive differences, not established universal gains.
493
+ A secondary graph metric is same-time pressure-curve similarity,
494
+ $S=(1+\rho_c)/2$, where $\rho_c$ is the centred cosine between the forecast
495
+ and observed pressure sequences on the common valid leads. No lag is optimized;
496
+ there is no time shift, warping or fitting to truth. At least six common points
497
+ and non-flat truth and both forecasts are required. JMA has 134 eligible
498
+ starts, USA 131; undefined shape scores are unavailable, not zero. Removing
499
+ level and amplitude makes this a shape diagnostic, not proof of correct
500
+ absolute pressure. Physical hPa curves and MAE remain separate checks.
501
+
502
+ \begin{figure}[H]
503
+ \centering
504
+ \begin{tikzpicture}[x=1cm,y=1cm,font=\scriptsize]
505
+ \foreach \offset/\title/\old/\new/\oldlo/\oldhi/\newlo/\newhi in {
506
+ 0/JMA/13.5323/12.8371/11.6118/15.5213/10.5684/15.3174,
507
+ 7.2/USA/12.8356/12.5529/11.0010/14.6486/10.5056/14.7379}{
508
+ \begin{scope}[shift={(\offset,0)}]
509
+ \node[font=\small\bfseries] at (2.5,4.2) {\title\ central-pressure MAE};
510
+ \foreach \tick in {0,5,10,15,20}{
511
+ \draw[black!10] (0,\tick*3.5/20)--(5,\tick*3.5/20);
512
+ \node[left] at (0,\tick*3.5/20) {\tick};}
513
+ \fill[baselinegray] (1.05,0) rectangle (1.75,\old*3.5/20);
514
+ \fill[modelteal] (3.25,0) rectangle (3.95,\new*3.5/20);
515
+ \draw[ink,thick] (1.4,\oldlo*3.5/20)--(1.4,\oldhi*3.5/20);
516
+ \draw[ink,thick] (1.25,\oldlo*3.5/20)--(1.55,\oldlo*3.5/20);
517
+ \draw[ink,thick] (1.25,\oldhi*3.5/20)--(1.55,\oldhi*3.5/20);
518
+ \draw[ink,thick] (3.6,\newlo*3.5/20)--(3.6,\newhi*3.5/20);
519
+ \draw[ink,thick] (3.45,\newlo*3.5/20)--(3.75,\newlo*3.5/20);
520
+ \draw[ink,thick] (3.45,\newhi*3.5/20)--(3.75,\newhi*3.5/20);
521
+ \node[above=4pt] at (1.4,\oldhi*3.5/20) {\pgfmathprintnumber[fixed,precision=2]{\old}};
522
+ \node[above=4pt] at (3.6,\newhi*3.5/20) {\pgfmathprintnumber[fixed,precision=2]{\new}};
523
+ \node[below=3pt] at (1.4,0) {1.1};
524
+ \node[below=3pt] at (3.6,0) {1.2 mean of 50};
525
+ \draw[black!55] (0,3.5)--(0,0)--(5,0);
526
+ \node[rotate=90] at (-.75,1.75) {MAE (hPa); lower is better};
527
+ \end{scope}}
528
+ \end{tikzpicture}
529
+ \caption{Central-pressure MAE at exact common times, 40 equal-weight
530
+ storms and 134 shared starts. Error bars are 95\% whole-storm bootstrap intervals
531
+ (2,000 replicates, fixed seed 4712). They are development uncertainty, not an
532
+ untouched-test certification. Each agency uses its own common label mask.}
533
+ \label{fig:matchedpressure}
534
+ \end{figure}
535
+
536
+ \reportpage
537
+ \subsection{Physical pressure levels and same-time overlays}
538
+ The following overlay shows the first input-eligible frozen daily case,
539
+ not a selected best case. Forecast and observed pressure share the same
540
+ physical hPa axis and exact +6--120 h leads. No translation, normalization or
541
+ temporal adjustment is applied to the drawing. It demonstrates why a shape
542
+ score must be read with actual levels and timing, including poor predictions.
543
+ \begin{figure}[H]
544
+ \centering
545
+ \begin{tikzpicture}[x=1cm,y=1cm,font=\scriptsize]
546
+ \begin{scope}[shift={(0,0)}]
547
+ \node[font=\small\bfseries] at (2.5,4.1) {JMA pressure: EWINIAR};
548
+ \draw[black!10] (0,0)--(5,0);
549
+ \node[left] at (0,0) {965};
550
+ \draw[black!10] (0,1.75)--(5,1.75);
551
+ \node[left] at (0,1.75) {990};
552
+ \draw[black!10] (0,3.5)--(5,3.5);
553
+ \node[left] at (0,3.5) {1015};
554
+ \node[below=3pt] at (0,0) {6};
555
+ \node[below=3pt] at (1.8421052631578947,0) {48};
556
+ \node[below=3pt] at (3.9473684210526314,0) {96};
557
+ \node[below=3pt] at (5,0) {120};
558
+ \draw[ink,solid,thick] (1.5789,2.7300)--(1.8421,2.8700)--(2.1053,2.7300)--(2.3684,2.8700)--(2.6316,2.7300)--(2.8947,2.8700)--(3.1579,2.7300)--(3.4211,2.5900)--(3.6842,2.3100)--(3.9474,2.0300)--(4.2105,1.7500)--(4.4737,1.0500)--(4.7368,0.7000)--(5.0000,0.3500);
559
+ \draw[baselinegray,dashed,thick] (0.0000,2.8399)--(0.2632,2.9400)--(0.5263,2.9680)--(0.7895,2.9456)--(1.0526,2.8945)--(1.3158,2.8329)--(1.5789,2.7629)--(1.8421,2.6740)--(2.1053,2.5809)--(2.3684,2.4920)--(2.6316,2.4010)--(2.8947,2.2834)--(3.1579,2.1756)--(3.4211,2.0678)--(3.6842,1.9943)--(3.9474,1.9264)--(4.2105,1.8753)--(4.4737,1.8193)--(4.7368,1.7850)--(5.0000,1.8067);
560
+ \draw[modelteal,solid,thick] (0.0000,2.4672)--(0.2632,2.4474)--(0.5263,2.4338)--(0.7895,2.4479)--(1.0526,2.3913)--(1.3158,2.3234)--(1.5789,2.1839)--(1.8421,1.9551)--(2.1053,1.6817)--(2.3684,1.3776)--(2.6316,1.1448)--(2.8947,0.9760)--(3.1579,0.9345)--(3.4211,0.8983)--(3.6842,0.8637)--(3.9474,0.8436)--(4.2105,0.8260)--(4.4737,0.8174)--(4.7368,0.8217)--(5.0000,0.8158);
561
+ \draw[black!55] (0,3.5)--(0,0)--(5,0);
562
+ \node[rotate=90] at (-.75,1.75) {Central pressure (hPa)};
563
+ \node at (2.5,-.65) {Forecast lead (hours)};
564
+ \end{scope}
565
+ \begin{scope}[shift={(7.2,0)}]
566
+ \node[font=\small\bfseries] at (2.5,4.1) {USA pressure: EWINIAR};
567
+ \draw[black!10] (0,0)--(5,0);
568
+ \node[left] at (0,0) {955};
569
+ \draw[black!10] (0,1.75)--(5,1.75);
570
+ \node[left] at (0,1.75) {985};
571
+ \draw[black!10] (0,3.5)--(5,3.5);
572
+ \node[left] at (0,3.5) {1015};
573
+ \node[below=3pt] at (0,0) {6};
574
+ \node[below=3pt] at (1.8421052631578947,0) {48};
575
+ \node[below=3pt] at (3.9473684210526314,0) {96};
576
+ \node[below=3pt] at (5,0) {120};
577
+ \draw[ink,solid,thick] (0.0000,2.6250)--(0.2632,2.6833)--(0.5263,2.6833)--(0.7895,2.6833)--(1.0526,2.6833)--(1.3158,2.6833)--(1.5789,2.5667)--(1.8421,2.6833)--(2.1053,2.6250)--(2.3684,2.5083)--(2.6316,2.4500)--(2.8947,2.3917)--(3.1579,2.8000)--(3.4211,2.2167)--(3.6842,2.1583)--(3.9474,2.0417)--(4.2105,1.9250)--(4.4737,0.8167)--(4.7368,0.5833)--(5.0000,0.4083);
578
+ \draw[baselinegray,dashed,thick] (0.0000,2.9499)--(0.2632,3.0333)--(0.5263,3.0567)--(0.7895,3.0380)--(1.0526,2.9954)--(1.3158,2.9441)--(1.5789,2.8858)--(1.8421,2.8117)--(2.1053,2.7341)--(2.3684,2.6600)--(2.6316,2.5842)--(2.8947,2.4862)--(3.1579,2.3963)--(3.4211,2.3065)--(3.6842,2.2453)--(3.9474,2.1887)--(4.2105,2.1461)--(4.4737,2.0994)--(4.7368,2.0708)--(5.0000,2.0889);
579
+ \draw[modelteal,solid,thick] (0.0000,2.6393)--(0.2632,2.6229)--(0.5263,2.6115)--(0.7895,2.6232)--(1.0526,2.5761)--(1.3158,2.5195)--(1.5789,2.4033)--(1.8421,2.2126)--(2.1053,1.9848)--(2.3684,1.7313)--(2.6316,1.5374)--(2.8947,1.3966)--(3.1579,1.3621)--(3.4211,1.3319)--(3.6842,1.3031)--(3.9474,1.2864)--(4.2105,1.2717)--(4.4737,1.2645)--(4.7368,1.2680)--(5.0000,1.2632);
580
+ \draw[black!55] (0,3.5)--(0,0)--(5,0);
581
+ \node[rotate=90] at (-.75,1.75) {Central pressure (hPa)};
582
+ \node at (2.5,-.65) {Forecast lead (hours)};
583
+ \end{scope}
584
+ \end{tikzpicture}\par\vspace{4pt}
585
+ {\small Black: best track; gray dashed: 1.1; teal: 1.2 mean of 50.}
586
+ \caption{EWINIAR, issue 2024-05-22 00:00 UTC. Observations are verification only, not model inputs. Agencies remain separate and gaps are not filled.}
587
+ \label{fig:pressurecurves}
588
+ \end{figure}
589
+
590
+ \begin{table}[H]
591
+ \centering\small
592
+ \caption{Same-time pressure errors. Values are hPa; each lead retains the
593
+ daily-to-storm weighting and exact common labels.}
594
+ \begin{tabular}{lrrrr}
595
+ \toprule
596
+ & \multicolumn{2}{c}{JMA best track} & \multicolumn{2}{c}{USA best track}\\
597
+ \cmidrule(lr){2-3}\cmidrule(lr){4-5}
598
+ Measure & 1.1 & 1.2 & 1.1 & 1.2\\
599
+ \midrule
600
+ All valid +6--120 h & 13.53 & 12.84 & 12.84 & 12.55\\
601
+ At +120 h & 15.95 & 12.44 & 13.52 & 15.55\\
602
+ \bottomrule
603
+ \end{tabular}
604
+ \end{table}
605
+
606
+ \subsection{Modest mean differences, not an all-metric win}
607
+ JMA pressure-curve similarity changes only from 0.7074 to 0.7118;
608
+ USA similarity worsens from 0.7237 to 0.6637. Both paired whole-storm
609
+ similarity-difference intervals include zero. These numbers are not accuracy
610
+ percentages or a proof that the pressure graphs align in hPa.
611
+ The paired 1.2 minus 1.1 pressure difference is $-0.70$ hPa for JMA
612
+ (95\% interval $[-2.68,1.20]$ hPa) and $-0.28$ hPa for USA
613
+ ($[-2.10,1.45]$ hPa). Both include zero. The USA +120 h comparison worsens,
614
+ whereas JMA improves at that lead; aggregate route gains do not imply
615
+ universally improved intensity. The separate auxiliary-wind diagnostic is
616
+ also worse for 1.2 (25.03 versus 16.98 kt against USA one-minute wind).
617
+ Its surface averaging period remains unvalidated; JMA ten-minute wind is not
618
+ pooled with USA wind. The released 1.2 network has no native wind-radius head.
619
+
620
+ The original 1.2-only pressure score, 12.62 hPa, uses a different label mask
621
+ and is not substituted into these paired bars. Its basin MSLP error is
622
+ 2.72 hPa on 270 storms, a separate field diagnostic with no matched 1.1
623
+ field result. A basin mean does not establish central-core accuracy.
624
+
625
+ \keyline{Reproducible publication.}{The shared daily snapshot and audit
626
+ receipt preserve source hashes, checkpoint identity, 20 exact leads, common
627
+ pressure coverage and all storm scores. A read-only public endpoint,
628
+ \href{https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/api/benchmarks/released}{\texttt{/api/benchmarks/released}},
629
+ serves the same verified values as the Site charts and documentation.
630
+ No forecasts, model weights or calibration were changed for this update.}
631
  \reportpage
632
  \section{Fung-wong: route alignment and intensity}
633
  The selected forecast begins at 00 UTC on 7 November 2025 and uses the saved
 
840
  \label{fig:pressuremap}
841
  \end{figure}
842
  \subsection{Intensity labels and map coverage}
843
+ Pressure truth is IBTrACS \texttt{TOKYO\_PRES} (JMA best track), not a JMA
844
+ forecast. Native-detail history was unavailable; $0.25^\circ$ sampling is not
845
+ native resolution. Beyond +66 h, the fixed patch no longer covers the centre:
846
+ the film displays only coarse basin pressure there, while the moving-core
847
+ readout supplies central pressure. No detailed core is invented.
848
+
849
+ The native-pressure follow-up in Figure~\ref{fig:matchedpressure} is restricted
850
+ to its common 40-storm support. On the
851
  repeatedly inspected TIP ten-start diagnostic, 1.2 pressure MAE is worse
852
  (30.7 versus 20.1 hPa). Those overlapping starts are not independent tests;
853
  route gains do not establish an all-metric improvement.
854
 
855
  \subsection{What remains unverified}
856
+ The cohort's 40 recent and 230 historical storms (1980--1999) are outside this
857
+ checkpoint's fitting/validation years, but not certified unused across prior
858
+ experiments. Five-day eligibility excludes short lifetimes; historical
859
+ hindcasts use retrospective analyses and later-year training. Generalization
860
+ requires an unused whole-storm holdout and training-only preprocessing.
 
861
 
862
  \subsection{Conclusion}
863
+ Trackformer 1.2 couples recurrent environmental evolution, a moving pressure
864
+ anomaly and geographic readout. Aggregate route scores improve, but absolute
865
+ errors and intensity failures remain. Pressure-curve results are mixed, with
866
+ paired uncertainty including zero; auxiliary-wind errors are large and native
867
+ radius skill is unavailable. This reproducible research model does not replace
 
868
  official warnings.
869
  \end{document}
release_tools/build_release_benchmark.py ADDED
@@ -0,0 +1,220 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Verify saved daily forecasts and publish a small, cohort-explicit Site snapshot.
2
+
3
+ Read-only evaluation: no inference, fitting, forecast replacement or model changes.
4
+ The pressure postprocessor checks the original inputs and 50-member replay files.
5
+ DeepMind is explicitly unscored on this daily cohort, not borrowed from an older one.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import argparse
10
+ from collections import defaultdict
11
+ from datetime import datetime, timezone
12
+ import hashlib
13
+ import json
14
+ from pathlib import Path
15
+ import sys
16
+
17
+ import numpy as np
18
+
19
+ ROOT = Path(__file__).resolve().parents[1]
20
+ CHECKPOINT = 'f194a23d3f91ea76ad776dfad942fabd669eeae8b3fd665815463095367e9ee0'
21
+ TRACK_METRICS = (
22
+ ('mean_track_error_km', 'Mean track error', 'km', 1, 'lower'),
23
+ ('track_error_120h_km', '+120 h track error', 'km', 1, 'lower'),
24
+ ('direction_error_deg', 'Six-hour direction error', 'degrees', 2, 'lower'),
25
+ ('shape_similarity', 'Centred route-shape similarity', 'similarity', 4, 'higher'),
26
+ ('path_similarity', 'Geographic path similarity', 'similarity', 4, 'higher'),
27
+ ('frechet_distance_km', 'Frechet path distance', 'km', 1, 'lower'),
28
+ )
29
+
30
+
31
+ def sha(path):
32
+ with Path(path).open('rb') as stream:
33
+ return hashlib.file_digest(stream, 'sha256').hexdigest()
34
+
35
+
36
+ def read(path):
37
+ return json.loads(Path(path).read_text())
38
+
39
+
40
+ def close(actual, expected):
41
+ if actual is None or expected is None:
42
+ if actual is not expected:
43
+ raise ValueError('Unavailable value replaced by a score')
44
+ elif not np.allclose(actual, expected, rtol=0, atol=1e-8):
45
+ raise ValueError('Saved metric does not reproduce: ' + str((actual, expected)))
46
+
47
+
48
+ def check_daily(project, report, plan):
49
+ """Recompute routes, coordinate conversion, days and equal-storm means."""
50
+ sys.path.insert(0, str(project))
51
+ from benchmark_daily_storms_1_2 import metrics, local
52
+ if report['status'] != 'complete' or report['completed_daily_cases'] != len(plan['cases']):
53
+ raise ValueError('Daily benchmark is not complete')
54
+ if report['cohort_sha256'] != sha(project/'benchmark_daily_storms_1_2/cohort.json'):
55
+ raise ValueError('Daily cohort identity differs')
56
+ if plan['models']['1.2']['checkpoint_sha256'] != CHECKPOINT or plan['models']['1.2']['members'] != 50:
57
+ raise ValueError('Wrong release checkpoint or member count')
58
+ groups = defaultdict(list)
59
+ files = {}
60
+ daily_keys = [(c['storm_id'], c['day_utc']) for c in plan['cases']]
61
+ if len(set(daily_keys)) != len(daily_keys):
62
+ raise ValueError('More than one start per storm-day')
63
+ for case in plan['cases']:
64
+ stem = f"{case['case_index']:05d}"
65
+ path = project/'benchmark_daily_storms_1_2/cases'/stem
66
+ saved = read(path.with_suffix('.json'))
67
+ if any(saved[k] != v for k, v in case.items()):
68
+ raise ValueError('Daily case identity changed')
69
+ digest = sha(path.with_suffix('.npz'))
70
+ if digest != saved['forecast_sha256'] or saved['members'] != 50:
71
+ raise ValueError('Saved forecast or member count changed')
72
+ files[stem] = digest
73
+ with np.load(path.with_suffix('.npz'), allow_pickle=False) as z:
74
+ arrays = [z[k] for k in ('v11_local', 'v12_local', 'truth_local')]
75
+ if any(a.shape != (20, 2) or not np.isfinite(a).all() for a in arrays):
76
+ raise ValueError('Missing route must not be zero-scored')
77
+ for model, route in zip(('v11', 'v12'), arrays[:2]):
78
+ projected = local(z[model+'_lat_lon'], case['base_lat'], case['base_lon'])
79
+ close(route, projected)
80
+ calculated = metrics(*arrays)
81
+ for model in ('1.1', '1.2'):
82
+ for key in [k[0] for k in TRACK_METRICS] + ['track_error_by_lead_km']:
83
+ close(calculated[model][key], saved['metrics'][model][key])
84
+ groups[case['storm_id']].append(calculated)
85
+ if len(groups) != plan['storm_count'] or len(groups) != report['completed_storms']:
86
+ raise ValueError('Incomplete storm coverage')
87
+ expected_storms = {s['storm_id']: s for s in report['storm_scores']}
88
+ for model in ('1.1', '1.2'):
89
+ for key in [k[0] for k in TRACK_METRICS] + ['track_error_by_lead_km']:
90
+ storm_values = []
91
+ for sid, days in groups.items():
92
+ values = [d[model][key] for d in days if d[model][key] is not None]
93
+ value = np.mean(values, axis=0).tolist() if values else None
94
+ close(value, expected_storms[sid]['models'][model][key])
95
+ if value is not None:
96
+ storm_values.append(value)
97
+ close(np.mean(storm_values, axis=0).tolist(), report['aggregate_equal_storm'][model][key])
98
+ return files
99
+
100
+
101
+ def snapshot(daily, plan, intensity, runner, identities):
102
+ if daily['cohort_sha256'] != intensity['source_hashes']['cohort']:
103
+ raise ValueError('Track and pressure use different frozen daily plans')
104
+ if daily['completed_storms'] != 270 or daily['completed_daily_cases'] != 1473:
105
+ raise ValueError('Unexpected released daily coverage')
106
+ if intensity['status'] != 'complete_verified' or not intensity['verification']['all_frozen_original_forecasts_sha256_valid']:
107
+ raise ValueError('Pressure comparison is not verified')
108
+ metrics = []
109
+ curve_metrics = []
110
+ for key, label, unit, digits, preferred in TRACK_METRICS:
111
+ metrics.append(dict(key=key, label=label, unit=unit, decimals=digits,
112
+ preferred=preferred, coverage={'storms': 270, 'daily_issues': 1473},
113
+ values={m: daily['aggregate_equal_storm'][m][key] for m in ('1.1', '1.2')}))
114
+ for agency in ('JMA', 'USA'):
115
+ key = 'pressure_'+agency+'_hpa'
116
+ source = intensity['metrics'][key]['mae']
117
+ shape = intensity['metrics'][key]['shape_error']
118
+ # The published error is (1 - centred cosine) / 2. Its complement is
119
+ # a similarity; this linear transform preserves the exact paired mask.
120
+ delta = shape['paired_delta_1_2_minus_1_1']
121
+ curve_metrics.append(dict(key='pressure_'+agency+'_curve_similarity',
122
+ label=agency+' pressure-curve similarity', unit='similarity',
123
+ decimals=4, preferred='higher', reference=agency,
124
+ coverage={'storms': shape['valid_storms'], 'daily_issues': shape['valid_daily_issues']},
125
+ values={m: 1-shape['models'][m]['equal_storm_mean'] for m in ('1.1', '1.2')},
126
+ intervals={m: [1-shape['models'][m]['bootstrap_95_percent_interval'][1],
127
+ 1-shape['models'][m]['bootstrap_95_percent_interval'][0]] for m in ('1.1', '1.2')},
128
+ paired_delta={'mean': -delta['mean'],
129
+ 'bootstrap_95_percent_interval': [-delta['bootstrap_95_percent_interval'][1],
130
+ -delta['bootstrap_95_percent_interval'][0]]},
131
+ definition='(1 + centred cosine similarity) / 2 at exact common UTC leads; no shift, time warp or lag optimization. At least six common points and non-flat truth and both forecasts. Removes level and amplitude, so read with pressure MAE and actual hPa curves.'))
132
+ metrics.append(dict(key=key, label=agency+' central-pressure MAE', unit='hPa',
133
+ decimals=2, preferred='lower',
134
+ coverage={'storms': source['valid_storms'], 'daily_issues': source['valid_daily_issues'],
135
+ 'valid_issue_leads': sum(runner[key]['paired_issue_leads_by_lead']),
136
+ 'issue_leads_by_lead': runner[key]['paired_issue_leads_by_lead']},
137
+ values={m: source['models'][m]['equal_storm_mean'] for m in ('1.1', '1.2')},
138
+ intervals={m: source['models'][m]['bootstrap_95_percent_interval'] for m in ('1.1', '1.2')},
139
+ paired_delta=source['paired_delta_1_2_minus_1_1'],
140
+ per_lead={m: runner[key]['models'][m]['equal_storm_mae_by_lead'] for m in ('1.1', '1.2')}))
141
+ return dict(schema='released-daily-benchmark-v1', status='complete_verified',
142
+ verified_at_utc=datetime.now(timezone.utc).isoformat(),
143
+ cohort={'sha256': daily['cohort_sha256'], 'storms': 270, 'daily_issues': 1473,
144
+ 'leads_hours': plan['leads_hours'], 'aggregation': daily['aggregation']},
145
+ models=[{'key': 'v11', 'label': 'Trackformer 1.1',
146
+ 'method': 'Frozen weighted route / native intensity ensemble'},
147
+ {'key': 'v12', 'label': 'Trackformer 1.2', 'method': '50 distinct input-perturbed members',
148
+ 'member_count': 50, 'checkpoint_sha256': CHECKPOINT},
149
+ {'key': 'deepmind', 'label': 'DeepMind Mini', 'method': 'Not run on this daily cohort',
150
+ 'status': 'unavailable', 'reason': 'Older starts and different weighting; no transferred score.'}],
151
+ metrics=curve_metrics+metrics,
152
+ intensity_coverage={'eligible_daily_issues': intensity['eligible_intensity_daily_issues'],
153
+ 'unavailable_daily_issues': intensity['excluded_missing_current_intensity'],
154
+ 'reason': 'Frozen 1.1 pipeline requires valid issue-time wind and pressure.'},
155
+ per_lead_track={m: daily['aggregate_equal_storm'][m]['track_error_by_lead_km'] for m in ('1.1', '1.2')},
156
+ sources=identities, notes=[
157
+ 'Development evidence, not a certified unused holdout or an architecture ablation.',
158
+ 'Metric-specific common truth masks; missing outputs and labels never become zero error.',
159
+ 'No clean one-member daily evaluation: these 1.2 scores are always the verified 50-member mean.',
160
+ 'Pressure metrics use 40 storms / 134 issues, not all 270 storms / 1473 issues.',
161
+ 'No wind or radius bars in the Site main chart; diagnostic reports remain separately downloadable.',
162
+ 'The pressure display repair does not change these immutable forecasts or central-pressure results.'])
163
+
164
+
165
+ def build(project, output):
166
+ from plot_intensity_benchmark import verify_and_summarize
167
+ plan = read(project/'benchmark_daily_storms_1_2/cohort.json')
168
+ daily = read(ROOT/'evaluation/daily_storm_final.json')
169
+ originals = check_daily(project, daily, plan)
170
+ source = project/'output/intensity-v12-v11-20261001'
171
+ intensity = read(ROOT/'evaluation/intensity/intensity_final.json')
172
+ receipt = read(ROOT/'evaluation/intensity/verification.json')
173
+ if sha(ROOT/'evaluation/intensity/intensity_final.json') != receipt['summary_sha256']:
174
+ raise ValueError('Published pressure report hash differs')
175
+ reproduced = verify_and_summarize(source, project)
176
+ for key in ('metrics', 'native_1_1_radius_USA_km', 'source_hashes', 'verification'):
177
+ if reproduced[key] != intensity[key]:
178
+ raise ValueError('Verified pressure report no longer reproduces: '+key)
179
+ runner = read(source/'report.json')
180
+ identities = {'daily_report_sha256': sha(ROOT/'evaluation/daily_storm_final.json'),
181
+ 'intensity_report_sha256': sha(ROOT/'evaluation/intensity/intensity_final.json'),
182
+ 'intensity_receipt_sha256': sha(ROOT/'evaluation/intensity/verification.json'),
183
+ 'intensity_protocol_sha256': sha(source/'protocol.json'),
184
+ 'intensity_runner_report_sha256': sha(source/'report.json')}
185
+ data = snapshot(daily, plan, intensity, runner, identities)
186
+ import csv
187
+ names = {}
188
+ with (project/'data/ibtracs/ibtracs.WP.list.v04r01.csv').open() as stream:
189
+ rows = csv.DictReader(stream)
190
+ next(rows) # units row, not a storm
191
+ for row in rows:
192
+ if row['NAME'].strip() not in ('', 'NOT_NAMED'):
193
+ names[row['SID'].strip()] = row['NAME'].strip()
194
+ records = [read(p) for p in sorted((source/'cases').glob('*.json'))]
195
+ data['pressure_curves'] = [dict(case_index=r['case_index'], storm_id=r['storm_id'],
196
+ name=names.get(r['storm_id'], r['storm_id']), issue_time_utc=r['issue_time_utc'],
197
+ observed={agency:r['truth']['pressure_'+agency+'_hpa'] for agency in ('JMA','USA')},
198
+ forecasts={'1.1':[p['central_pressure_hpa'] for p in r['v11']],
199
+ '1.2':r['v12_pressure_hpa']}) for r in records if r['v11'] is not None]
200
+ output = output.resolve()
201
+ if not output.is_relative_to(Path('/Volumes/D')):
202
+ raise ValueError('Artifacts must stay on D')
203
+ output.mkdir(parents=True, exist_ok=True)
204
+ (output/'released_daily_benchmark.json').write_text(json.dumps(data, indent=2, allow_nan=False)+'\n')
205
+ audit = {'status': 'complete_verified', **identities,
206
+ 'snapshot_sha256': sha(output/'released_daily_benchmark.json'),
207
+ 'verified_original_daily_forecasts': len(originals),
208
+ 'all_route_scores_and_absolute_local_conversions_reproduced': True,
209
+ 'original_forecast_sha256': originals, 'verified_pressure_member_replays': 134,
210
+ 'missing_results_scored_as_zero': 0, 'new_inference_performed': False}
211
+ (output/'released_daily_verification.json').write_text(json.dumps(audit, indent=2)+'\n')
212
+ print(json.dumps({k:v for k,v in audit.items() if k != 'original_forecast_sha256'}), flush=True)
213
+
214
+
215
+ if __name__ == '__main__':
216
+ parser = argparse.ArgumentParser(description=__doc__)
217
+ parser.add_argument('--project', type=Path, default=ROOT.parent)
218
+ parser.add_argument('--output', type=Path, required=True)
219
+ args = parser.parse_args()
220
+ build(args.project, args.output)
release_tools/plot_daily_storm_final.py CHANGED
@@ -30,6 +30,6 @@ for ax, (key, title, unit, digits) in zip(axes.flat, panels):
30
  ax.grid(axis='y', alpha=.16)
31
  fig.suptitle('Trackformer 1.2 vs 1.1 | 270 storms · 1,473 daily forecasts', x=.07, ha='left', fontsize=17, fontweight='bold')
32
  fig.text(.07, .922, 'Complete +6 to +120 h rollouts · each storm has equal weight · broader development evidence', color='#526174')
33
- fig.text(.07, .022, 'Different input pipelines; not an architecture ablation or certified untouched test. No matched 1.1 pressure comparison.', fontsize=9, color='#526174')
34
  fig.subplots_adjust(top=.84, bottom=.11, left=.07, right=.98, hspace=.42, wspace=.35)
35
  fig.savefig(ROOT / 'evaluation/trackformer_1_2_vs_1_1_270_storms_bars.png', dpi=180, facecolor='white')
 
30
  ax.grid(axis='y', alpha=.16)
31
  fig.suptitle('Trackformer 1.2 vs 1.1 | 270 storms · 1,473 daily forecasts', x=.07, ha='left', fontsize=17, fontweight='bold')
32
  fig.text(.07, .922, 'Complete +6 to +120 h rollouts · each storm has equal weight · broader development evidence', color='#526174')
33
+ fig.text(.07, .022, 'Different input pipelines; development evidence, not an untouched test. Matched pressure uses a separate 40-storm / 134-day support.', fontsize=9, color='#526174')
34
  fig.subplots_adjust(top=.84, bottom=.11, left=.07, right=.98, hspace=.42, wspace=.35)
35
  fig.savefig(ROOT / 'evaluation/trackformer_1_2_vs_1_1_270_storms_bars.png', dpi=180, facecolor='white')
release_tools/plot_release_pressure_benchmark.py ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Plot matched central-pressure MAE and unshifted pressure-versus-time curves."""
2
+ import json
3
+ from pathlib import Path
4
+ import matplotlib
5
+ matplotlib.use('Agg')
6
+ import matplotlib.pyplot as plt
7
+ import numpy as np
8
+
9
+ ROOT=Path(__file__).resolve().parents[1]
10
+ data=json.loads((ROOT/'evaluation/released_daily/released_daily_benchmark.json').read_text())
11
+ assert data['status']=='complete_verified'
12
+ fig,axes=plt.subplots(2,2,figsize=(12,8))
13
+ colors=['#66758b','#007d78']
14
+ for column,agency in enumerate(('JMA','USA')):
15
+ metric=next(m for m in data['metrics'] if m['key']=='pressure_'+agency+'_hpa')
16
+ assert metric['coverage']['storms']==40
17
+ axis=axes[0,column]
18
+ values=np.array([metric['values'][m] for m in ('1.1','1.2')])
19
+ intervals=np.array([metric['intervals'][m] for m in ('1.1','1.2')])
20
+ errors=np.stack([values-intervals[:,0],intervals[:,1]-values])
21
+ bars=axis.bar(['1.1','1.2 · mean of 50'],values,yerr=errors,color=colors,width=.52,capsize=5)
22
+ axis.bar_label(bars,labels=[f'{v:.2f}' for v in values],padding=5,fontweight='bold')
23
+ axis.set_ylim(0,20)
24
+ axis.set_title(f"{agency} pressure MAE · {metric['coverage']['daily_issues']} shared starts",loc='left',fontweight='bold',fontsize=12)
25
+ axis.set_ylabel('Central-pressure MAE (hPa) · lower is better')
26
+ lower=axes[1,column]
27
+ issue=data['pressure_curves'][0]
28
+ observed=np.array([np.nan if v is None else v for v in issue['observed'][agency]])
29
+ lower.plot(data['cohort']['leads_hours'],observed,label=agency+' best track',color='#253643',linewidth=2.2)
30
+ for model,color,style in zip(('1.1','1.2'),colors,('--','-')):
31
+ lower.plot(data['cohort']['leads_hours'],issue['forecasts'][model],label=model,color=color,linestyle=style,linewidth=2.2)
32
+ lower.set_xticks([6,24,48,72,96,120])
33
+ lower.set_title(issue['name']+' · '+issue['issue_time_utc'][:16].replace('T',' ')+' UTC',loc='left',fontsize=10)
34
+ lower.set_xlabel('Forecast lead (hours)')
35
+ lower.set_ylabel('Actual central pressure (hPa)')
36
+ lower.legend(frameon=False)
37
+ for a in (axis,lower):
38
+ a.spines[['top','right']].set_visible(False)
39
+ a.set_axisbelow(True);a.grid(axis='y',alpha=.15)
40
+ fig.suptitle('Trackformer 1.1 vs 1.2 | matched central pressure',x=.075,ha='left',fontsize=17,fontweight='bold')
41
+ fig.text(.075,.923,'Same exact valid times · daily → storm → equal storm · 40 storms · JMA and USA separate',color='#526174')
42
+ fig.text(.075,.025,'Slight descriptive mean improvement; paired difference intervals include zero. 95% whole-storm intervals.\nOverlay: first eligible frozen case, not a best example. Exact-time physical hPa curves; no shift or warping. Missing is not zero.',fontsize=9,color='#526174')
43
+ fig.subplots_adjust(left=.075,right=.98,top=.85,bottom=.12,hspace=.43,wspace=.25)
44
+ fig.savefig(ROOT/'evaluation/released_daily/pressure_comparison.png',dpi=180,facecolor='white')
release_tools/sync_public_model_cards.py CHANGED
@@ -22,6 +22,10 @@ CACHE = '/Volumes/D/typhoon_predict/.cache/huggingface-intensity'
22
  HF = 'https://huggingface.co/' + REPO
23
  GALLERY = '| Soudelor (2015) | Mangkhut (2018) | Meranti (2016) |\n'
24
  FILMS = ('mangkhut', 'fung_wong', 'soudelor', 'meranti')
 
 
 
 
25
  SYNC_FILES = (
26
  'models/trackformer_1_2_field/README.md',
27
  'models/trackformer_1_2_field/WIND_ESTIMATION.md',
@@ -29,6 +33,21 @@ SYNC_FILES = (
29
  'models/trackformer_1_2_field/wind_estimation.py',
30
  'release_tools/sync_public_model_cards.py',
31
  'release_tools/test_public_model_cards.py',
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
  )
33
 
34
 
@@ -50,7 +69,7 @@ def player(stem, title):
50
  'src="' + media_url(stem) + '"></video>')
51
 
52
 
53
- def render_card(original, github):
54
  metadata = re.match(r'\A---\n.*?\n---\n', original, re.S)
55
  if not metadata:
56
  raise ValueError('Preserve existing HF YAML metadata; missing header')
@@ -96,7 +115,8 @@ def render_card(original, github):
96
  resolved = (ROOT / path).resolve()
97
  if not resolved.is_relative_to(ROOT) or not resolved.exists():
98
  raise ValueError('Invalid or missing relative public link: ' + path)
99
- kind = 'resolve/' + MEDIA_REVISION if image else ('tree/main' if resolved.is_dir() else 'blob/main')
 
100
  target = HF + '/' + kind + '/' + path + ('#' + fragment if fragment else '')
101
  return image + '[' + label + '](' + target + ')'
102
  card = re.sub(r'(!?)\[([^\[\]\n]*)\]\(([^)\s]+)\)', link, card)
@@ -160,9 +180,18 @@ def run(output, publish=False):
160
  dirty = subprocess.check_output(['git', 'status', '--porcelain', '--', 'README.md', *SYNC_FILES], cwd=ROOT, text=True)
161
  if dirty:
162
  raise ValueError('Commit canonical README and matching source files before publishing')
163
- committed = api.create_commit(repo_id=REPO, repo_type='model', parent_commit=parent,
164
- commit_message='Refresh complete model card, pin corrected pressure movie, and sync inference documentation',
165
- operations=[CommitOperationAdd(path_in_repo=name, path_or_fileobj=str(path)) for name, path in files.items()])
 
 
 
 
 
 
 
 
 
166
  for name, expected in receipt['files'].items():
167
  saved = hf_hub_download(REPO, name, revision=committed.oid, cache_dir=CACHE)
168
  if sha(saved) != expected:
 
22
  HF = 'https://huggingface.co/' + REPO
23
  GALLERY = '| Soudelor (2015) | Mangkhut (2018) | Meranti (2016) |\n'
24
  FILMS = ('mangkhut', 'fung_wong', 'soudelor', 'meranti')
25
+ CURRENT_FIGURES = frozenset((
26
+ 'evaluation/trackformer_1_2_vs_1_1_270_storms_bars.png',
27
+ 'evaluation/released_daily/pressure_comparison.png',
28
+ ))
29
  SYNC_FILES = (
30
  'models/trackformer_1_2_field/README.md',
31
  'models/trackformer_1_2_field/WIND_ESTIMATION.md',
 
33
  'models/trackformer_1_2_field/wind_estimation.py',
34
  'release_tools/sync_public_model_cards.py',
35
  'release_tools/test_public_model_cards.py',
36
+ 'release_tools/build_release_benchmark.py',
37
+ 'release_tools/plot_release_pressure_benchmark.py',
38
+ 'release_tools/plot_daily_storm_final.py',
39
+ 'docs/daily_storm_benchmark.md',
40
+ 'docs/intensity_benchmark.md',
41
+ 'docs/trackformer_1_2_evaluation.md',
42
+ 'evaluation/daily_storm_final.json',
43
+ 'evaluation/released_daily/released_daily_benchmark.json',
44
+ 'evaluation/released_daily/released_daily_verification.json',
45
+ 'evaluation/intensity/intensity_final.json',
46
+ 'evaluation/intensity/verification.json',
47
+ 'evaluation/trackformer_1_2_vs_1_1_270_storms_bars.png',
48
+ 'evaluation/released_daily/pressure_comparison.png',
49
+ 'paper/trackformer.tex',
50
+ 'paper/trackformer.pdf',
51
  )
52
 
53
 
 
69
  'src="' + media_url(stem) + '"></video>')
70
 
71
 
72
+ def render_card(original, github, figure_revision='main'):
73
  metadata = re.match(r'\A---\n.*?\n---\n', original, re.S)
74
  if not metadata:
75
  raise ValueError('Preserve existing HF YAML metadata; missing header')
 
115
  resolved = (ROOT / path).resolve()
116
  if not resolved.is_relative_to(ROOT) or not resolved.exists():
117
  raise ValueError('Invalid or missing relative public link: ' + path)
118
+ revision = figure_revision if path in CURRENT_FIGURES else MEDIA_REVISION
119
+ kind = 'resolve/' + revision if image else ('tree/main' if resolved.is_dir() else 'blob/main')
120
  target = HF + '/' + kind + '/' + path + ('#' + fragment if fragment else '')
121
  return image + '[' + label + '](' + target + ')'
122
  card = re.sub(r'(!?)\[([^\[\]\n]*)\]\(([^)\s]+)\)', link, card)
 
180
  dirty = subprocess.check_output(['git', 'status', '--porcelain', '--', 'README.md', *SYNC_FILES], cwd=ROOT, text=True)
181
  if dirty:
182
  raise ValueError('Commit canonical README and matching source files before publishing')
183
+ # Publish scientific assets first, then pin the new figures to that
184
+ # immutable commit in the card. Never pin new bytes to the old movie
185
+ # revision or let a missing future image render as a stale figure.
186
+ assets = api.create_commit(repo_id=REPO, repo_type='model', parent_commit=parent,
187
+ commit_message='Publish verified matched pressure-curve benchmark and revised technical paper',
188
+ operations=[CommitOperationAdd(path_in_repo=name, path_or_fileobj=str(ROOT/name)) for name in SYNC_FILES])
189
+ card_path.write_text(render_card(original.read_text(), github, assets.oid))
190
+ receipt['files']['README.md'] = sha(card_path)
191
+ receipt['figure_revision'] = assets.oid
192
+ committed = api.create_commit(repo_id=REPO, repo_type='model', parent_commit=assets.oid,
193
+ commit_message='Refresh model card with exact-time pressure graph similarity and pinned figures',
194
+ operations=[CommitOperationAdd(path_in_repo='README.md', path_or_fileobj=str(card_path))])
195
  for name, expected in receipt['files'].items():
196
  saved = hf_hub_download(REPO, name, revision=committed.oid, cache_dir=CACHE)
197
  if sha(saved) != expected:
release_tools/test_public_model_cards.py CHANGED
@@ -1,7 +1,7 @@
1
  """Portable regression checks for complete shared README publication."""
2
  import re
3
  import unittest
4
- from sync_public_model_cards import MEDIA_REVISION, ROOT, media_url, render_card
5
 
6
 
7
  class PublicModelCardsTest(unittest.TestCase):
@@ -47,6 +47,22 @@ class PublicModelCardsTest(unittest.TestCase):
47
  with self.assertRaisesRegex(ValueError, 'relative public link'):
48
  render_card(self.original, self.source + '\n[No secrets](../../private-file.txt)\n')
49
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50
 
51
  if __name__ == '__main__':
52
  unittest.main()
 
1
  """Portable regression checks for complete shared README publication."""
2
  import re
3
  import unittest
4
+ from sync_public_model_cards import CURRENT_FIGURES, SYNC_FILES, MEDIA_REVISION, ROOT, media_url, render_card
5
 
6
 
7
  class PublicModelCardsTest(unittest.TestCase):
 
47
  with self.assertRaisesRegex(ValueError, 'relative public link'):
48
  render_card(self.original, self.source + '\n[No secrets](../../private-file.txt)\n')
49
 
50
+ def test_updated_pressure_graph_and_paper_are_published_together(self):
51
+ revision = 'a' * 40
52
+ rendered = render_card(self.original, self.source, revision)
53
+ for text in ('Pressure graph similarity', '0.7074', '0.7118', '0.7237', '0.6637', '131 days', 'time-warped', 'slight mean improvement', '13.53 to 12.84'):
54
+ self.assertIn(text, rendered)
55
+ for path in CURRENT_FIGURES:
56
+ self.assertIn('/resolve/' + revision + '/' + path, rendered)
57
+ self.assertIn(path, SYNC_FILES)
58
+ for path in ('paper/trackformer.tex', 'paper/trackformer.pdf',
59
+ 'evaluation/released_daily/released_daily_benchmark.json',
60
+ 'evaluation/released_daily/released_daily_verification.json'):
61
+ self.assertIn(path, SYNC_FILES)
62
+ self.assertNotIn('models/trackformer_1_2_field/weights.pt', SYNC_FILES)
63
+ for path in SYNC_FILES:
64
+ self.assertTrue((ROOT / path).is_file(), path)
65
+
66
 
67
  if __name__ == '__main__':
68
  unittest.main()