Publish verified matched pressure-curve benchmark and revised technical paper
Browse files- .gitattributes +1 -0
- docs/daily_storm_benchmark.md +6 -2
- docs/intensity_benchmark.md +2 -0
- docs/trackformer_1_2_evaluation.md +3 -1
- evaluation/released_daily/pressure_comparison.png +3 -0
- evaluation/released_daily/released_daily_benchmark.json +0 -0
- evaluation/released_daily/released_daily_verification.json +0 -0
- evaluation/trackformer_1_2_vs_1_1_270_storms_bars.png +2 -2
- models/trackformer_1_2_field/README.md +2 -0
- paper/trackformer.pdf +2 -2
- paper/trackformer.tex +187 -25
- release_tools/build_release_benchmark.py +220 -0
- release_tools/plot_daily_storm_final.py +1 -1
- release_tools/plot_release_pressure_benchmark.py +44 -0
- release_tools/sync_public_model_cards.py +34 -5
- release_tools/test_public_model_cards.py +17 -1
.gitattributes
CHANGED
|
@@ -194,3 +194,4 @@ evaluation/storm_structure/mangkhut_wind_radius.png filter=lfs diff=lfs merge=lf
|
|
| 194 |
evaluation/storm_structure/meranti_wind_radius.png filter=lfs diff=lfs merge=lfs -text
|
| 195 |
evaluation/storm_structure/soudelor_wind_radius.png filter=lfs diff=lfs merge=lfs -text
|
| 196 |
evaluation/release_data/pressure_core/mangkhut/common-pressure.npz filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 194 |
evaluation/storm_structure/meranti_wind_radius.png filter=lfs diff=lfs merge=lfs -text
|
| 195 |
evaluation/storm_structure/soudelor_wind_radius.png filter=lfs diff=lfs merge=lfs -text
|
| 196 |
evaluation/release_data/pressure_core/mangkhut/common-pressure.npz filter=lfs diff=lfs merge=lfs -text
|
| 197 |
+
evaluation/released_daily/pressure_comparison.png filter=lfs diff=lfs merge=lfs -text
|
docs/daily_storm_benchmark.md
CHANGED
|
@@ -2,7 +2,11 @@
|
|
| 2 |
|
| 3 |
Status: **complete** at 2026-09-29T08:22:39Z. All **1,473 daily cases / 270 storms** finished; the SHA-256 of every saved forecast was verified. [Final equal-storm results and all 270 storm scores](../evaluation/daily_storm_final.json).
|
| 4 |
|
| 5 |
-
Mean track error is 798.4 km for 1.1 and 471.2 km for 1.2 (50-member mean); +120 h error is 1,646.4 versus 1,031.6 km. Direction error is 51.58° versus 34.96°, centred shape similarity 0.7544 versus 0.8837, and geographic path similarity 0.5345 versus 0.6560.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
|
| 7 |
## What counts as a case and a score
|
| 8 |
|
|
@@ -33,6 +37,6 @@ Future weather/track/pressure are used for eligibility coverage and scoring only
|
|
| 33 |
|
| 34 |
## Pressure and failures
|
| 35 |
|
| 36 |
-
For 1.2,
|
| 37 |
|
| 38 |
Completed daily forecasts are saved atomically with hashes. A nonfinite forecast or source mismatch stops the run visibly rather than silently deleting the case. Resume skips completed cases without rerunning them. Partial aggregate scores include only fully completed storms, identify their counts, and must not be presented as the final 270-storm result. Final uncertainty should resample whole storms, not overlapping daily issues independently.
|
|
|
|
| 2 |
|
| 3 |
Status: **complete** at 2026-09-29T08:22:39Z. All **1,473 daily cases / 270 storms** finished; the SHA-256 of every saved forecast was verified. [Final equal-storm results and all 270 storm scores](../evaluation/daily_storm_final.json).
|
| 4 |
|
| 5 |
+
Mean track error is 798.4 km for 1.1 and 471.2 km for 1.2 (50-member mean); +120 h error is 1,646.4 versus 1,031.6 km. Direction error is 51.58° versus 34.96°, centred shape similarity 0.7544 versus 0.8837, and geographic path similarity 0.5345 versus 0.6560. The original track run reported 1.2-only central-pressure MAE of 12.62 hPa on 40 pressure-labelled storms and basin MSLP MAE of 2.72 hPa on 270 storms. It did not produce 1.1 intensity outputs.
|
| 6 |
+
|
| 7 |
+
The completed [native pressure follow-up](intensity_benchmark.md) now provides a matched 1.1 comparison on 134 daily starts / 40 storms from the same frozen plan. JMA MAE is 13.53 / 12.84 hPa for 1.1 / 1.2; USA MAE is 12.84 / 12.55 hPa. Each uses exact common agency-specific labels. The 1,339 remaining issue inputs are unavailable to the frozen 1.1 intensity pipeline, never zero-scored or filled with future observations. Paired whole-storm intervals include zero. These are not all-270-storm pressure scores.
|
| 8 |
+
|
| 9 |
+
On 1 October, `release_tools/build_release_benchmark.py` independently rechecked all forecast hashes, coordinate conversions and daily/storm route aggregates, reproduced the native pressure postprocessor and generated the [shared Site snapshot](../evaluation/released_daily/released_daily_benchmark.json) and [audit receipt](../evaluation/released_daily/released_daily_verification.json). No inference or weight changes were necessary. DeepMind has not been evaluated on this larger daily plan; an older cohort's score cannot be transferred into it.
|
| 10 |
|
| 11 |
## What counts as a case and a score
|
| 12 |
|
|
|
|
| 37 |
|
| 38 |
## Pressure and failures
|
| 39 |
|
| 40 |
+
For the original 1.2 run, central-pressure MAE uses valid labels and cosine-area-weighted basin MSLP MAE uses the native basin grid, daily then equally per storm. Keep pressure label coverage visible. The separate native-pressure follow-up uses a paired mask for both models at each agency's exact valid times; it does not substitute its smaller coverage into all-storm track results. Regional/core field scores need their own valid coverage and are not implied by basin-wide error. No same-cohort 1.1 basin-field score is claimed.
|
| 41 |
|
| 42 |
Completed daily forecasts are saved atomically with hashes. A nonfinite forecast or source mismatch stops the run visibly rather than silently deleting the case. Resume skips completed cases without rerunning them. Partial aggregate scores include only fully completed storms, identify their counts, and must not be presented as the final 270-storm result. Final uncertainty should resample whole storms, not overlapping daily issues independently.
|
docs/intensity_benchmark.md
CHANGED
|
@@ -63,6 +63,8 @@ all counts are printed. Observational RMW has reporting/provisional limitations.
|
|
| 63 |
|
| 64 |
## Curve comparison, analogous to route direction and shape
|
| 65 |
|
|
|
|
|
|
|
| 66 |
Magnitude MAE is retained alongside time-curve diagnostics, without optimizing
|
| 67 |
time lag, warping time, shifting observations or adjusting forecasts to truth.
|
| 68 |
These are separate, versioned post-hoc descriptive metrics, not a preregistered
|
|
|
|
| 63 |
|
| 64 |
## Curve comparison, analogous to route direction and shape
|
| 65 |
|
| 66 |
+
The released Site defaults to **central-pressure MAE**, with slight descriptive mean improvements (JMA 13.53 / 12.84 hPa; USA 12.84 / 12.55 hPa, for 1.1 / 1.2) and paired uncertainty including zero. Its secondary **similarity** view is the exact complement of the shape-error metric below: `(1 + centred cosine) / 2 = 1 − shape_error`. This is a presentation transform, not a new fit or score. JMA similarity is 0.7074 / 0.7118 on 40 storms and 134 starts; USA similarity is 0.7237 / 0.6637 on 40 storms and 131 non-flat starts. Unshifted physical hPa timelines remain visible separately. The public snapshot includes every input-eligible pressure timeline.
|
| 67 |
+
|
| 68 |
Magnitude MAE is retained alongside time-curve diagnostics, without optimizing
|
| 69 |
time lag, warping time, shifting observations or adjusting forecasts to truth.
|
| 70 |
These are separate, versioned post-hoc descriptive metrics, not a preregistered
|
docs/trackformer_1_2_evaluation.md
CHANGED
|
@@ -16,7 +16,9 @@ Trackformer 1.2 uses the mean of 50 distinct causal input perturbations, not 50
|
|
| 16 |
|
| 17 |
Direction, centred shape, geographic path and same-time position error measure different aspects of a forecast. A translated or mistimed route can have high shape similarity without overlapping the observed route. No combined weighted ranking is introduced.
|
| 18 |
|
| 19 |
-
The 1.2 central-pressure MAE is **12.62 hPa on 40 storms with valid labels**; area-weighted basin MSLP MAE is **2.72 hPa on 270 storms**.
|
|
|
|
|
|
|
| 20 |
|
| 21 |
## Recent pressure-line example
|
| 22 |
|
|
|
|
| 16 |
|
| 17 |
Direction, centred shape, geographic path and same-time position error measure different aspects of a forecast. A translated or mistimed route can have high shape similarity without overlapping the observed route. No combined weighted ranking is introduced.
|
| 18 |
|
| 19 |
+
The original track run's 1.2-only central-pressure MAE is **12.62 hPa on 40 storms with valid labels**; area-weighted basin MSLP MAE is **2.72 hPa on 270 storms**. The completed native 1.1 follow-up now supplies a matched pressure comparison on **134 common daily starts / 40 storms** from the same frozen plan. The default pressure chart is MAE: JMA **13.53 / 12.84 hPa**, USA **12.84 / 12.55 hPa** for 1.1 / 1.2, with paired uncertainty including zero. The optional same-time pressure-curve similarity diagnostic uses **(1 + centred cosine) / 2**, with no lag optimization or time warping: JMA scores are **0.7074 / 0.7118**, while USA scores are **0.7237 / 0.6637** on 131 non-flat eligible starts. Undefined curves remain unavailable, not zero. Both paired whole-storm similarity-difference intervals also include zero. This shape metric removes level and amplitude, so inspect the actual hPa overlays and MAE separately. [Shared verified snapshot](../evaluation/released_daily/released_daily_benchmark.json) · [Pressure protocol](intensity_benchmark.md) · [Public API](https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/api/benchmarks/released).
|
| 20 |
+
|
| 21 |
+
The other **1,339 starts lack valid frozen 1.1 issue-time intensity inputs**, so this is not 270-storm pressure coverage. The original pressure-only mask is not substituted into the new common-mask comparison. A basin-wide average does not establish cyclone-core accuracy. The frozen cohort contains 40 recent and 230 historical storms from 1980–1999; retrospective analyses, complete-five-day eligibility and prior model selection limit interpretation.
|
| 22 |
|
| 23 |
## Recent pressure-line example
|
| 24 |
|
evaluation/released_daily/pressure_comparison.png
ADDED
|
Git LFS Details
|
evaluation/released_daily/released_daily_benchmark.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
evaluation/released_daily/released_daily_verification.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
evaluation/trackformer_1_2_vs_1_1_270_storms_bars.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
models/trackformer_1_2_field/README.md
CHANGED
|
@@ -45,4 +45,6 @@ No model weights, learned architecture or existing benchmark results changed.
|
|
| 45 |
|
| 46 |
The published 50-member mean is a separate evaluation policy: 50 deterministic seeds perturb only normalized historical basin/regional fields with smooth zero-centred noise, then average routes, central pressures and common-grid fields. `predict.py` computes one clean forecast; do not label it a 50-member mean.
|
| 47 |
|
|
|
|
|
|
|
| 48 |
**Pressure-display coverage:** `regional_mslp_hpa` is the original fixed issue-relative composite, not an automatically extended moving-core map. When the forecast centre leaves that patch, a scalar central-pressure readout must not be inserted into the image to make it appear consistent. The corrected Mangkhut film separately exports the actual moving model cores and registers member fields before averaging; [the current model card](https://huggingface.co/euler314/typhoon-predict) links the verified pressure arrays and shows which other films retain their older fixed patches. The automatic History archive and its separate core recovery expose exact geography and coverage through [the public API](https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/history-api).
|
|
|
|
| 45 |
|
| 46 |
The published 50-member mean is a separate evaluation policy: 50 deterministic seeds perturb only normalized historical basin/regional fields with smooth zero-centred noise, then average routes, central pressures and common-grid fields. `predict.py` computes one clean forecast; do not label it a 50-member mean.
|
| 47 |
|
| 48 |
+
**Matched development evaluation:** the released 1.1 and 1.2 route comparison uses the same frozen **1,473 daily starts / 270 equal-weight storms**, +6 to +120 h. The completed native-pressure comparison has **134 common starts / 40 storms**; the other 1,339 starts lack valid frozen 1.1 intensity inputs and are not zero-scored. The Site defaults to central-pressure MAE: **13.53 / 12.84 hPa** against JMA and **12.84 / 12.55 hPa** against USA (1.1 / 1.2), slight mean improvements with paired whole-storm uncertainty including zero. The optional curve-similarity diagnostic is **(1 + centred cosine) / 2 at exact common valid times**, without time shifting or warping. JMA similarity is **0.7074 / 0.7118**; USA is **0.7237 / 0.6637** on 131 eligible non-flat curves. Removing level/amplitude makes this a shape diagnostic, not proof of correct pressure levels. Actual unshifted hPa timelines, agency masks and uncertainty remain separate. See the [shared snapshot](../../evaluation/released_daily/released_daily_benchmark.json), [verification receipt](../../evaluation/released_daily/released_daily_verification.json), [pressure protocol](../../docs/intensity_benchmark.md) and [public benchmark API](https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/api/benchmarks/released). DeepMind Mini has not run on this daily cohort; no old score is transferred into it.
|
| 49 |
+
|
| 50 |
**Pressure-display coverage:** `regional_mslp_hpa` is the original fixed issue-relative composite, not an automatically extended moving-core map. When the forecast centre leaves that patch, a scalar central-pressure readout must not be inserted into the image to make it appear consistent. The corrected Mangkhut film separately exports the actual moving model cores and registers member fields before averaging; [the current model card](https://huggingface.co/euler314/typhoon-predict) links the verified pressure arrays and shows which other films retain their older fixed patches. The automatic History archive and its separate core recovery expose exact geography and coverage through [the public API](https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/history-api).
|
paper/trackformer.pdf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ac371a8d9a577f352cfaaeb322c8e1f4803f815f4727cbf234977cebfd1e1f45
|
| 3 |
+
size 431222
|
paper/trackformer.tex
CHANGED
|
@@ -39,7 +39,7 @@
|
|
| 39 |
\begin{document}
|
| 40 |
\fontsize{10.3}{13}\selectfont
|
| 41 |
\thispagestyle{empty}
|
| 42 |
-
{\sffamily\small\color{modelteal}RESEARCH TECHNICAL REPORT\hfill
|
| 43 |
\vspace{13pt}
|
| 44 |
{\sffamily\bfseries\color{ink}\fontsize{26}{29}\selectfont Trackformer 1.2\par}
|
| 45 |
\vspace{4pt}
|
|
@@ -62,6 +62,9 @@ This report specifies the released architecture, state updates, learning
|
|
| 62 |
objectives and 50-member input-perturbation policy. Across 1,473 daily forecasts
|
| 63 |
from 270 storms, equal-storm mean track error is 471.2 km for the 1.2 ensemble
|
| 64 |
and 798.4 km for the causal 1.1 route pipeline.
|
|
|
|
|
|
|
|
|
|
| 65 |
The pipelines differ; this is not an architecture ablation or an untouched-test
|
| 66 |
result. A selected Fung-wong case illustrates actual geographic alignment,
|
| 67 |
intensity evolution and model-generated pressure contours, including their
|
|
@@ -460,10 +463,171 @@ look similar; the shape score cannot replace geographic error or an unshifted
|
|
| 460 |
overlay. Path similarity responds to displacement but is not a strict
|
| 461 |
same-time distance.
|
| 462 |
|
| 463 |
-
\keyline{
|
| 464 |
-
|
| 465 |
-
|
| 466 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 467 |
\reportpage
|
| 468 |
\section{Fung-wong: route alignment and intensity}
|
| 469 |
The selected forecast begins at 00 UTC on 7 November 2025 and uses the saved
|
|
@@ -676,32 +840,30 @@ animates the saved fields; no additional inference or route adjustment is used.
|
|
| 676 |
\label{fig:pressuremap}
|
| 677 |
\end{figure}
|
| 678 |
\subsection{Intensity labels and map coverage}
|
| 679 |
-
Pressure truth is
|
| 680 |
-
forecast. Native-detail history was unavailable; $0.25^\circ$
|
| 681 |
-
|
| 682 |
-
|
| 683 |
-
|
| 684 |
-
|
| 685 |
-
|
| 686 |
-
|
| 687 |
repeatedly inspected TIP ten-start diagnostic, 1.2 pressure MAE is worse
|
| 688 |
(30.7 versus 20.1 hPa). Those overlapping starts are not independent tests;
|
| 689 |
route gains do not establish an all-metric improvement.
|
| 690 |
|
| 691 |
\subsection{What remains unverified}
|
| 692 |
-
The
|
| 693 |
-
|
| 694 |
-
|
| 695 |
-
|
| 696 |
-
|
| 697 |
-
holdout, training-only preprocessing and whole-storm uncertainty resampling.
|
| 698 |
|
| 699 |
\subsection{Conclusion}
|
| 700 |
-
Trackformer 1.2
|
| 701 |
-
|
| 702 |
-
|
| 703 |
-
|
| 704 |
-
|
| 705 |
-
released model is a reproducible research artifact, not a substitute for
|
| 706 |
official warnings.
|
| 707 |
\end{document}
|
|
|
|
| 39 |
\begin{document}
|
| 40 |
\fontsize{10.3}{13}\selectfont
|
| 41 |
\thispagestyle{empty}
|
| 42 |
+
{\sffamily\small\color{modelteal}RESEARCH TECHNICAL REPORT\hfill REVISED 1 OCTOBER 2026\par}
|
| 43 |
\vspace{13pt}
|
| 44 |
{\sffamily\bfseries\color{ink}\fontsize{26}{29}\selectfont Trackformer 1.2\par}
|
| 45 |
\vspace{4pt}
|
|
|
|
| 62 |
objectives and 50-member input-perturbation policy. Across 1,473 daily forecasts
|
| 63 |
from 270 storms, equal-storm mean track error is 471.2 km for the 1.2 ensemble
|
| 64 |
and 798.4 km for the causal 1.1 route pipeline.
|
| 65 |
+
On 134 matched daily starts from 40 storms, same-time JMA pressure MAE
|
| 66 |
+
is 13.53 and 12.84 hPa for 1.1 and 1.2, respectively; paired
|
| 67 |
+
whole-storm uncertainty includes zero improvement.
|
| 68 |
The pipelines differ; this is not an architecture ablation or an untouched-test
|
| 69 |
result. A selected Fung-wong case illustrates actual geographic alignment,
|
| 70 |
intensity evolution and model-generated pressure contours, including their
|
|
|
|
| 463 |
overlay. Path similarity responds to displacement but is not a strict
|
| 464 |
same-time distance.
|
| 465 |
|
| 466 |
+
\keyline{Verified same benchmark.}{All 1,473 original daily forecasts were
|
| 467 |
+
hash-checked again on 1 October. Absolute routes were converted to the same
|
| 468 |
+
issue-centred kilometre projection, and every daily and equal-storm route score
|
| 469 |
+
was reproduced. The native-pressure comparison below uses a common subset of
|
| 470 |
+
this same frozen plan, not a different storm sample. DeepMind Mini has not been
|
| 471 |
+
run on this daily plan; its older scores are not transferred.}
|
| 472 |
+
\reportpage
|
| 473 |
+
\section{Matched central-pressure comparison}
|
| 474 |
+
\subsection{Common starts, agency labels and missing inputs}
|
| 475 |
+
The frozen 1.1 intensity ensemble requires valid issue-time wind and central
|
| 476 |
+
pressure. These inputs are present in \textbf{134 daily starts from 40 storms}
|
| 477 |
+
within the 1,473-start plan. The remaining \textbf{1,339 starts are unavailable},
|
| 478 |
+
not zero-scored and not replaced with later observations. The 1.1 primary,
|
| 479 |
+
structure and gated temporal experts retain their released calibration.
|
| 480 |
+
For 1.2, all 134 fifty-member causal replays reproduce the originally saved
|
| 481 |
+
central-pressure means exactly (maximum difference 0.0 hPa).
|
| 482 |
+
|
| 483 |
+
JMA pressure uses \texttt{TOKYO\_PRES}; USA pressure uses \texttt{USA\_PRES}
|
| 484 |
+
from the same IBTrACS storm identifiers at issue time plus the exact forecast
|
| 485 |
+
lead. Agencies remain separate. Each comparison uses the intersection of valid
|
| 486 |
+
truth and both model outputs: lead errors average within a day, days within a
|
| 487 |
+
storm, then storms equally. Missing observations remain missing. This preserves
|
| 488 |
+
the track benchmark's hierarchy without claiming 270-storm intensity coverage.
|
| 489 |
+
|
| 490 |
+
The default pressure chart uses same-time central-pressure MAE in hPa.
|
| 491 |
+
JMA mean error decreases by approximately 5.1\%; USA decreases by 2.2\%.
|
| 492 |
+
These are slight descriptive differences, not established universal gains.
|
| 493 |
+
A secondary graph metric is same-time pressure-curve similarity,
|
| 494 |
+
$S=(1+\rho_c)/2$, where $\rho_c$ is the centred cosine between the forecast
|
| 495 |
+
and observed pressure sequences on the common valid leads. No lag is optimized;
|
| 496 |
+
there is no time shift, warping or fitting to truth. At least six common points
|
| 497 |
+
and non-flat truth and both forecasts are required. JMA has 134 eligible
|
| 498 |
+
starts, USA 131; undefined shape scores are unavailable, not zero. Removing
|
| 499 |
+
level and amplitude makes this a shape diagnostic, not proof of correct
|
| 500 |
+
absolute pressure. Physical hPa curves and MAE remain separate checks.
|
| 501 |
+
|
| 502 |
+
\begin{figure}[H]
|
| 503 |
+
\centering
|
| 504 |
+
\begin{tikzpicture}[x=1cm,y=1cm,font=\scriptsize]
|
| 505 |
+
\foreach \offset/\title/\old/\new/\oldlo/\oldhi/\newlo/\newhi in {
|
| 506 |
+
0/JMA/13.5323/12.8371/11.6118/15.5213/10.5684/15.3174,
|
| 507 |
+
7.2/USA/12.8356/12.5529/11.0010/14.6486/10.5056/14.7379}{
|
| 508 |
+
\begin{scope}[shift={(\offset,0)}]
|
| 509 |
+
\node[font=\small\bfseries] at (2.5,4.2) {\title\ central-pressure MAE};
|
| 510 |
+
\foreach \tick in {0,5,10,15,20}{
|
| 511 |
+
\draw[black!10] (0,\tick*3.5/20)--(5,\tick*3.5/20);
|
| 512 |
+
\node[left] at (0,\tick*3.5/20) {\tick};}
|
| 513 |
+
\fill[baselinegray] (1.05,0) rectangle (1.75,\old*3.5/20);
|
| 514 |
+
\fill[modelteal] (3.25,0) rectangle (3.95,\new*3.5/20);
|
| 515 |
+
\draw[ink,thick] (1.4,\oldlo*3.5/20)--(1.4,\oldhi*3.5/20);
|
| 516 |
+
\draw[ink,thick] (1.25,\oldlo*3.5/20)--(1.55,\oldlo*3.5/20);
|
| 517 |
+
\draw[ink,thick] (1.25,\oldhi*3.5/20)--(1.55,\oldhi*3.5/20);
|
| 518 |
+
\draw[ink,thick] (3.6,\newlo*3.5/20)--(3.6,\newhi*3.5/20);
|
| 519 |
+
\draw[ink,thick] (3.45,\newlo*3.5/20)--(3.75,\newlo*3.5/20);
|
| 520 |
+
\draw[ink,thick] (3.45,\newhi*3.5/20)--(3.75,\newhi*3.5/20);
|
| 521 |
+
\node[above=4pt] at (1.4,\oldhi*3.5/20) {\pgfmathprintnumber[fixed,precision=2]{\old}};
|
| 522 |
+
\node[above=4pt] at (3.6,\newhi*3.5/20) {\pgfmathprintnumber[fixed,precision=2]{\new}};
|
| 523 |
+
\node[below=3pt] at (1.4,0) {1.1};
|
| 524 |
+
\node[below=3pt] at (3.6,0) {1.2 mean of 50};
|
| 525 |
+
\draw[black!55] (0,3.5)--(0,0)--(5,0);
|
| 526 |
+
\node[rotate=90] at (-.75,1.75) {MAE (hPa); lower is better};
|
| 527 |
+
\end{scope}}
|
| 528 |
+
\end{tikzpicture}
|
| 529 |
+
\caption{Central-pressure MAE at exact common times, 40 equal-weight
|
| 530 |
+
storms and 134 shared starts. Error bars are 95\% whole-storm bootstrap intervals
|
| 531 |
+
(2,000 replicates, fixed seed 4712). They are development uncertainty, not an
|
| 532 |
+
untouched-test certification. Each agency uses its own common label mask.}
|
| 533 |
+
\label{fig:matchedpressure}
|
| 534 |
+
\end{figure}
|
| 535 |
+
|
| 536 |
+
\reportpage
|
| 537 |
+
\subsection{Physical pressure levels and same-time overlays}
|
| 538 |
+
The following overlay shows the first input-eligible frozen daily case,
|
| 539 |
+
not a selected best case. Forecast and observed pressure share the same
|
| 540 |
+
physical hPa axis and exact +6--120 h leads. No translation, normalization or
|
| 541 |
+
temporal adjustment is applied to the drawing. It demonstrates why a shape
|
| 542 |
+
score must be read with actual levels and timing, including poor predictions.
|
| 543 |
+
\begin{figure}[H]
|
| 544 |
+
\centering
|
| 545 |
+
\begin{tikzpicture}[x=1cm,y=1cm,font=\scriptsize]
|
| 546 |
+
\begin{scope}[shift={(0,0)}]
|
| 547 |
+
\node[font=\small\bfseries] at (2.5,4.1) {JMA pressure: EWINIAR};
|
| 548 |
+
\draw[black!10] (0,0)--(5,0);
|
| 549 |
+
\node[left] at (0,0) {965};
|
| 550 |
+
\draw[black!10] (0,1.75)--(5,1.75);
|
| 551 |
+
\node[left] at (0,1.75) {990};
|
| 552 |
+
\draw[black!10] (0,3.5)--(5,3.5);
|
| 553 |
+
\node[left] at (0,3.5) {1015};
|
| 554 |
+
\node[below=3pt] at (0,0) {6};
|
| 555 |
+
\node[below=3pt] at (1.8421052631578947,0) {48};
|
| 556 |
+
\node[below=3pt] at (3.9473684210526314,0) {96};
|
| 557 |
+
\node[below=3pt] at (5,0) {120};
|
| 558 |
+
\draw[ink,solid,thick] (1.5789,2.7300)--(1.8421,2.8700)--(2.1053,2.7300)--(2.3684,2.8700)--(2.6316,2.7300)--(2.8947,2.8700)--(3.1579,2.7300)--(3.4211,2.5900)--(3.6842,2.3100)--(3.9474,2.0300)--(4.2105,1.7500)--(4.4737,1.0500)--(4.7368,0.7000)--(5.0000,0.3500);
|
| 559 |
+
\draw[baselinegray,dashed,thick] (0.0000,2.8399)--(0.2632,2.9400)--(0.5263,2.9680)--(0.7895,2.9456)--(1.0526,2.8945)--(1.3158,2.8329)--(1.5789,2.7629)--(1.8421,2.6740)--(2.1053,2.5809)--(2.3684,2.4920)--(2.6316,2.4010)--(2.8947,2.2834)--(3.1579,2.1756)--(3.4211,2.0678)--(3.6842,1.9943)--(3.9474,1.9264)--(4.2105,1.8753)--(4.4737,1.8193)--(4.7368,1.7850)--(5.0000,1.8067);
|
| 560 |
+
\draw[modelteal,solid,thick] (0.0000,2.4672)--(0.2632,2.4474)--(0.5263,2.4338)--(0.7895,2.4479)--(1.0526,2.3913)--(1.3158,2.3234)--(1.5789,2.1839)--(1.8421,1.9551)--(2.1053,1.6817)--(2.3684,1.3776)--(2.6316,1.1448)--(2.8947,0.9760)--(3.1579,0.9345)--(3.4211,0.8983)--(3.6842,0.8637)--(3.9474,0.8436)--(4.2105,0.8260)--(4.4737,0.8174)--(4.7368,0.8217)--(5.0000,0.8158);
|
| 561 |
+
\draw[black!55] (0,3.5)--(0,0)--(5,0);
|
| 562 |
+
\node[rotate=90] at (-.75,1.75) {Central pressure (hPa)};
|
| 563 |
+
\node at (2.5,-.65) {Forecast lead (hours)};
|
| 564 |
+
\end{scope}
|
| 565 |
+
\begin{scope}[shift={(7.2,0)}]
|
| 566 |
+
\node[font=\small\bfseries] at (2.5,4.1) {USA pressure: EWINIAR};
|
| 567 |
+
\draw[black!10] (0,0)--(5,0);
|
| 568 |
+
\node[left] at (0,0) {955};
|
| 569 |
+
\draw[black!10] (0,1.75)--(5,1.75);
|
| 570 |
+
\node[left] at (0,1.75) {985};
|
| 571 |
+
\draw[black!10] (0,3.5)--(5,3.5);
|
| 572 |
+
\node[left] at (0,3.5) {1015};
|
| 573 |
+
\node[below=3pt] at (0,0) {6};
|
| 574 |
+
\node[below=3pt] at (1.8421052631578947,0) {48};
|
| 575 |
+
\node[below=3pt] at (3.9473684210526314,0) {96};
|
| 576 |
+
\node[below=3pt] at (5,0) {120};
|
| 577 |
+
\draw[ink,solid,thick] (0.0000,2.6250)--(0.2632,2.6833)--(0.5263,2.6833)--(0.7895,2.6833)--(1.0526,2.6833)--(1.3158,2.6833)--(1.5789,2.5667)--(1.8421,2.6833)--(2.1053,2.6250)--(2.3684,2.5083)--(2.6316,2.4500)--(2.8947,2.3917)--(3.1579,2.8000)--(3.4211,2.2167)--(3.6842,2.1583)--(3.9474,2.0417)--(4.2105,1.9250)--(4.4737,0.8167)--(4.7368,0.5833)--(5.0000,0.4083);
|
| 578 |
+
\draw[baselinegray,dashed,thick] (0.0000,2.9499)--(0.2632,3.0333)--(0.5263,3.0567)--(0.7895,3.0380)--(1.0526,2.9954)--(1.3158,2.9441)--(1.5789,2.8858)--(1.8421,2.8117)--(2.1053,2.7341)--(2.3684,2.6600)--(2.6316,2.5842)--(2.8947,2.4862)--(3.1579,2.3963)--(3.4211,2.3065)--(3.6842,2.2453)--(3.9474,2.1887)--(4.2105,2.1461)--(4.4737,2.0994)--(4.7368,2.0708)--(5.0000,2.0889);
|
| 579 |
+
\draw[modelteal,solid,thick] (0.0000,2.6393)--(0.2632,2.6229)--(0.5263,2.6115)--(0.7895,2.6232)--(1.0526,2.5761)--(1.3158,2.5195)--(1.5789,2.4033)--(1.8421,2.2126)--(2.1053,1.9848)--(2.3684,1.7313)--(2.6316,1.5374)--(2.8947,1.3966)--(3.1579,1.3621)--(3.4211,1.3319)--(3.6842,1.3031)--(3.9474,1.2864)--(4.2105,1.2717)--(4.4737,1.2645)--(4.7368,1.2680)--(5.0000,1.2632);
|
| 580 |
+
\draw[black!55] (0,3.5)--(0,0)--(5,0);
|
| 581 |
+
\node[rotate=90] at (-.75,1.75) {Central pressure (hPa)};
|
| 582 |
+
\node at (2.5,-.65) {Forecast lead (hours)};
|
| 583 |
+
\end{scope}
|
| 584 |
+
\end{tikzpicture}\par\vspace{4pt}
|
| 585 |
+
{\small Black: best track; gray dashed: 1.1; teal: 1.2 mean of 50.}
|
| 586 |
+
\caption{EWINIAR, issue 2024-05-22 00:00 UTC. Observations are verification only, not model inputs. Agencies remain separate and gaps are not filled.}
|
| 587 |
+
\label{fig:pressurecurves}
|
| 588 |
+
\end{figure}
|
| 589 |
+
|
| 590 |
+
\begin{table}[H]
|
| 591 |
+
\centering\small
|
| 592 |
+
\caption{Same-time pressure errors. Values are hPa; each lead retains the
|
| 593 |
+
daily-to-storm weighting and exact common labels.}
|
| 594 |
+
\begin{tabular}{lrrrr}
|
| 595 |
+
\toprule
|
| 596 |
+
& \multicolumn{2}{c}{JMA best track} & \multicolumn{2}{c}{USA best track}\\
|
| 597 |
+
\cmidrule(lr){2-3}\cmidrule(lr){4-5}
|
| 598 |
+
Measure & 1.1 & 1.2 & 1.1 & 1.2\\
|
| 599 |
+
\midrule
|
| 600 |
+
All valid +6--120 h & 13.53 & 12.84 & 12.84 & 12.55\\
|
| 601 |
+
At +120 h & 15.95 & 12.44 & 13.52 & 15.55\\
|
| 602 |
+
\bottomrule
|
| 603 |
+
\end{tabular}
|
| 604 |
+
\end{table}
|
| 605 |
+
|
| 606 |
+
\subsection{Modest mean differences, not an all-metric win}
|
| 607 |
+
JMA pressure-curve similarity changes only from 0.7074 to 0.7118;
|
| 608 |
+
USA similarity worsens from 0.7237 to 0.6637. Both paired whole-storm
|
| 609 |
+
similarity-difference intervals include zero. These numbers are not accuracy
|
| 610 |
+
percentages or a proof that the pressure graphs align in hPa.
|
| 611 |
+
The paired 1.2 minus 1.1 pressure difference is $-0.70$ hPa for JMA
|
| 612 |
+
(95\% interval $[-2.68,1.20]$ hPa) and $-0.28$ hPa for USA
|
| 613 |
+
($[-2.10,1.45]$ hPa). Both include zero. The USA +120 h comparison worsens,
|
| 614 |
+
whereas JMA improves at that lead; aggregate route gains do not imply
|
| 615 |
+
universally improved intensity. The separate auxiliary-wind diagnostic is
|
| 616 |
+
also worse for 1.2 (25.03 versus 16.98 kt against USA one-minute wind).
|
| 617 |
+
Its surface averaging period remains unvalidated; JMA ten-minute wind is not
|
| 618 |
+
pooled with USA wind. The released 1.2 network has no native wind-radius head.
|
| 619 |
+
|
| 620 |
+
The original 1.2-only pressure score, 12.62 hPa, uses a different label mask
|
| 621 |
+
and is not substituted into these paired bars. Its basin MSLP error is
|
| 622 |
+
2.72 hPa on 270 storms, a separate field diagnostic with no matched 1.1
|
| 623 |
+
field result. A basin mean does not establish central-core accuracy.
|
| 624 |
+
|
| 625 |
+
\keyline{Reproducible publication.}{The shared daily snapshot and audit
|
| 626 |
+
receipt preserve source hashes, checkpoint identity, 20 exact leads, common
|
| 627 |
+
pressure coverage and all storm scores. A read-only public endpoint,
|
| 628 |
+
\href{https://trackformer-weatherlab.rudin-euler-8253.chatgpt.site/api/benchmarks/released}{\texttt{/api/benchmarks/released}},
|
| 629 |
+
serves the same verified values as the Site charts and documentation.
|
| 630 |
+
No forecasts, model weights or calibration were changed for this update.}
|
| 631 |
\reportpage
|
| 632 |
\section{Fung-wong: route alignment and intensity}
|
| 633 |
The selected forecast begins at 00 UTC on 7 November 2025 and uses the saved
|
|
|
|
| 840 |
\label{fig:pressuremap}
|
| 841 |
\end{figure}
|
| 842 |
\subsection{Intensity labels and map coverage}
|
| 843 |
+
Pressure truth is IBTrACS \texttt{TOKYO\_PRES} (JMA best track), not a JMA
|
| 844 |
+
forecast. Native-detail history was unavailable; $0.25^\circ$ sampling is not
|
| 845 |
+
native resolution. Beyond +66 h, the fixed patch no longer covers the centre:
|
| 846 |
+
the film displays only coarse basin pressure there, while the moving-core
|
| 847 |
+
readout supplies central pressure. No detailed core is invented.
|
| 848 |
+
|
| 849 |
+
The native-pressure follow-up in Figure~\ref{fig:matchedpressure} is restricted
|
| 850 |
+
to its common 40-storm support. On the
|
| 851 |
repeatedly inspected TIP ten-start diagnostic, 1.2 pressure MAE is worse
|
| 852 |
(30.7 versus 20.1 hPa). Those overlapping starts are not independent tests;
|
| 853 |
route gains do not establish an all-metric improvement.
|
| 854 |
|
| 855 |
\subsection{What remains unverified}
|
| 856 |
+
The cohort's 40 recent and 230 historical storms (1980--1999) are outside this
|
| 857 |
+
checkpoint's fitting/validation years, but not certified unused across prior
|
| 858 |
+
experiments. Five-day eligibility excludes short lifetimes; historical
|
| 859 |
+
hindcasts use retrospective analyses and later-year training. Generalization
|
| 860 |
+
requires an unused whole-storm holdout and training-only preprocessing.
|
|
|
|
| 861 |
|
| 862 |
\subsection{Conclusion}
|
| 863 |
+
Trackformer 1.2 couples recurrent environmental evolution, a moving pressure
|
| 864 |
+
anomaly and geographic readout. Aggregate route scores improve, but absolute
|
| 865 |
+
errors and intensity failures remain. Pressure-curve results are mixed, with
|
| 866 |
+
paired uncertainty including zero; auxiliary-wind errors are large and native
|
| 867 |
+
radius skill is unavailable. This reproducible research model does not replace
|
|
|
|
| 868 |
official warnings.
|
| 869 |
\end{document}
|
release_tools/build_release_benchmark.py
ADDED
|
@@ -0,0 +1,220 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Verify saved daily forecasts and publish a small, cohort-explicit Site snapshot.
|
| 2 |
+
|
| 3 |
+
Read-only evaluation: no inference, fitting, forecast replacement or model changes.
|
| 4 |
+
The pressure postprocessor checks the original inputs and 50-member replay files.
|
| 5 |
+
DeepMind is explicitly unscored on this daily cohort, not borrowed from an older one.
|
| 6 |
+
"""
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import argparse
|
| 10 |
+
from collections import defaultdict
|
| 11 |
+
from datetime import datetime, timezone
|
| 12 |
+
import hashlib
|
| 13 |
+
import json
|
| 14 |
+
from pathlib import Path
|
| 15 |
+
import sys
|
| 16 |
+
|
| 17 |
+
import numpy as np
|
| 18 |
+
|
| 19 |
+
ROOT = Path(__file__).resolve().parents[1]
|
| 20 |
+
CHECKPOINT = 'f194a23d3f91ea76ad776dfad942fabd669eeae8b3fd665815463095367e9ee0'
|
| 21 |
+
TRACK_METRICS = (
|
| 22 |
+
('mean_track_error_km', 'Mean track error', 'km', 1, 'lower'),
|
| 23 |
+
('track_error_120h_km', '+120 h track error', 'km', 1, 'lower'),
|
| 24 |
+
('direction_error_deg', 'Six-hour direction error', 'degrees', 2, 'lower'),
|
| 25 |
+
('shape_similarity', 'Centred route-shape similarity', 'similarity', 4, 'higher'),
|
| 26 |
+
('path_similarity', 'Geographic path similarity', 'similarity', 4, 'higher'),
|
| 27 |
+
('frechet_distance_km', 'Frechet path distance', 'km', 1, 'lower'),
|
| 28 |
+
)
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def sha(path):
|
| 32 |
+
with Path(path).open('rb') as stream:
|
| 33 |
+
return hashlib.file_digest(stream, 'sha256').hexdigest()
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def read(path):
|
| 37 |
+
return json.loads(Path(path).read_text())
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def close(actual, expected):
|
| 41 |
+
if actual is None or expected is None:
|
| 42 |
+
if actual is not expected:
|
| 43 |
+
raise ValueError('Unavailable value replaced by a score')
|
| 44 |
+
elif not np.allclose(actual, expected, rtol=0, atol=1e-8):
|
| 45 |
+
raise ValueError('Saved metric does not reproduce: ' + str((actual, expected)))
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def check_daily(project, report, plan):
|
| 49 |
+
"""Recompute routes, coordinate conversion, days and equal-storm means."""
|
| 50 |
+
sys.path.insert(0, str(project))
|
| 51 |
+
from benchmark_daily_storms_1_2 import metrics, local
|
| 52 |
+
if report['status'] != 'complete' or report['completed_daily_cases'] != len(plan['cases']):
|
| 53 |
+
raise ValueError('Daily benchmark is not complete')
|
| 54 |
+
if report['cohort_sha256'] != sha(project/'benchmark_daily_storms_1_2/cohort.json'):
|
| 55 |
+
raise ValueError('Daily cohort identity differs')
|
| 56 |
+
if plan['models']['1.2']['checkpoint_sha256'] != CHECKPOINT or plan['models']['1.2']['members'] != 50:
|
| 57 |
+
raise ValueError('Wrong release checkpoint or member count')
|
| 58 |
+
groups = defaultdict(list)
|
| 59 |
+
files = {}
|
| 60 |
+
daily_keys = [(c['storm_id'], c['day_utc']) for c in plan['cases']]
|
| 61 |
+
if len(set(daily_keys)) != len(daily_keys):
|
| 62 |
+
raise ValueError('More than one start per storm-day')
|
| 63 |
+
for case in plan['cases']:
|
| 64 |
+
stem = f"{case['case_index']:05d}"
|
| 65 |
+
path = project/'benchmark_daily_storms_1_2/cases'/stem
|
| 66 |
+
saved = read(path.with_suffix('.json'))
|
| 67 |
+
if any(saved[k] != v for k, v in case.items()):
|
| 68 |
+
raise ValueError('Daily case identity changed')
|
| 69 |
+
digest = sha(path.with_suffix('.npz'))
|
| 70 |
+
if digest != saved['forecast_sha256'] or saved['members'] != 50:
|
| 71 |
+
raise ValueError('Saved forecast or member count changed')
|
| 72 |
+
files[stem] = digest
|
| 73 |
+
with np.load(path.with_suffix('.npz'), allow_pickle=False) as z:
|
| 74 |
+
arrays = [z[k] for k in ('v11_local', 'v12_local', 'truth_local')]
|
| 75 |
+
if any(a.shape != (20, 2) or not np.isfinite(a).all() for a in arrays):
|
| 76 |
+
raise ValueError('Missing route must not be zero-scored')
|
| 77 |
+
for model, route in zip(('v11', 'v12'), arrays[:2]):
|
| 78 |
+
projected = local(z[model+'_lat_lon'], case['base_lat'], case['base_lon'])
|
| 79 |
+
close(route, projected)
|
| 80 |
+
calculated = metrics(*arrays)
|
| 81 |
+
for model in ('1.1', '1.2'):
|
| 82 |
+
for key in [k[0] for k in TRACK_METRICS] + ['track_error_by_lead_km']:
|
| 83 |
+
close(calculated[model][key], saved['metrics'][model][key])
|
| 84 |
+
groups[case['storm_id']].append(calculated)
|
| 85 |
+
if len(groups) != plan['storm_count'] or len(groups) != report['completed_storms']:
|
| 86 |
+
raise ValueError('Incomplete storm coverage')
|
| 87 |
+
expected_storms = {s['storm_id']: s for s in report['storm_scores']}
|
| 88 |
+
for model in ('1.1', '1.2'):
|
| 89 |
+
for key in [k[0] for k in TRACK_METRICS] + ['track_error_by_lead_km']:
|
| 90 |
+
storm_values = []
|
| 91 |
+
for sid, days in groups.items():
|
| 92 |
+
values = [d[model][key] for d in days if d[model][key] is not None]
|
| 93 |
+
value = np.mean(values, axis=0).tolist() if values else None
|
| 94 |
+
close(value, expected_storms[sid]['models'][model][key])
|
| 95 |
+
if value is not None:
|
| 96 |
+
storm_values.append(value)
|
| 97 |
+
close(np.mean(storm_values, axis=0).tolist(), report['aggregate_equal_storm'][model][key])
|
| 98 |
+
return files
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
def snapshot(daily, plan, intensity, runner, identities):
|
| 102 |
+
if daily['cohort_sha256'] != intensity['source_hashes']['cohort']:
|
| 103 |
+
raise ValueError('Track and pressure use different frozen daily plans')
|
| 104 |
+
if daily['completed_storms'] != 270 or daily['completed_daily_cases'] != 1473:
|
| 105 |
+
raise ValueError('Unexpected released daily coverage')
|
| 106 |
+
if intensity['status'] != 'complete_verified' or not intensity['verification']['all_frozen_original_forecasts_sha256_valid']:
|
| 107 |
+
raise ValueError('Pressure comparison is not verified')
|
| 108 |
+
metrics = []
|
| 109 |
+
curve_metrics = []
|
| 110 |
+
for key, label, unit, digits, preferred in TRACK_METRICS:
|
| 111 |
+
metrics.append(dict(key=key, label=label, unit=unit, decimals=digits,
|
| 112 |
+
preferred=preferred, coverage={'storms': 270, 'daily_issues': 1473},
|
| 113 |
+
values={m: daily['aggregate_equal_storm'][m][key] for m in ('1.1', '1.2')}))
|
| 114 |
+
for agency in ('JMA', 'USA'):
|
| 115 |
+
key = 'pressure_'+agency+'_hpa'
|
| 116 |
+
source = intensity['metrics'][key]['mae']
|
| 117 |
+
shape = intensity['metrics'][key]['shape_error']
|
| 118 |
+
# The published error is (1 - centred cosine) / 2. Its complement is
|
| 119 |
+
# a similarity; this linear transform preserves the exact paired mask.
|
| 120 |
+
delta = shape['paired_delta_1_2_minus_1_1']
|
| 121 |
+
curve_metrics.append(dict(key='pressure_'+agency+'_curve_similarity',
|
| 122 |
+
label=agency+' pressure-curve similarity', unit='similarity',
|
| 123 |
+
decimals=4, preferred='higher', reference=agency,
|
| 124 |
+
coverage={'storms': shape['valid_storms'], 'daily_issues': shape['valid_daily_issues']},
|
| 125 |
+
values={m: 1-shape['models'][m]['equal_storm_mean'] for m in ('1.1', '1.2')},
|
| 126 |
+
intervals={m: [1-shape['models'][m]['bootstrap_95_percent_interval'][1],
|
| 127 |
+
1-shape['models'][m]['bootstrap_95_percent_interval'][0]] for m in ('1.1', '1.2')},
|
| 128 |
+
paired_delta={'mean': -delta['mean'],
|
| 129 |
+
'bootstrap_95_percent_interval': [-delta['bootstrap_95_percent_interval'][1],
|
| 130 |
+
-delta['bootstrap_95_percent_interval'][0]]},
|
| 131 |
+
definition='(1 + centred cosine similarity) / 2 at exact common UTC leads; no shift, time warp or lag optimization. At least six common points and non-flat truth and both forecasts. Removes level and amplitude, so read with pressure MAE and actual hPa curves.'))
|
| 132 |
+
metrics.append(dict(key=key, label=agency+' central-pressure MAE', unit='hPa',
|
| 133 |
+
decimals=2, preferred='lower',
|
| 134 |
+
coverage={'storms': source['valid_storms'], 'daily_issues': source['valid_daily_issues'],
|
| 135 |
+
'valid_issue_leads': sum(runner[key]['paired_issue_leads_by_lead']),
|
| 136 |
+
'issue_leads_by_lead': runner[key]['paired_issue_leads_by_lead']},
|
| 137 |
+
values={m: source['models'][m]['equal_storm_mean'] for m in ('1.1', '1.2')},
|
| 138 |
+
intervals={m: source['models'][m]['bootstrap_95_percent_interval'] for m in ('1.1', '1.2')},
|
| 139 |
+
paired_delta=source['paired_delta_1_2_minus_1_1'],
|
| 140 |
+
per_lead={m: runner[key]['models'][m]['equal_storm_mae_by_lead'] for m in ('1.1', '1.2')}))
|
| 141 |
+
return dict(schema='released-daily-benchmark-v1', status='complete_verified',
|
| 142 |
+
verified_at_utc=datetime.now(timezone.utc).isoformat(),
|
| 143 |
+
cohort={'sha256': daily['cohort_sha256'], 'storms': 270, 'daily_issues': 1473,
|
| 144 |
+
'leads_hours': plan['leads_hours'], 'aggregation': daily['aggregation']},
|
| 145 |
+
models=[{'key': 'v11', 'label': 'Trackformer 1.1',
|
| 146 |
+
'method': 'Frozen weighted route / native intensity ensemble'},
|
| 147 |
+
{'key': 'v12', 'label': 'Trackformer 1.2', 'method': '50 distinct input-perturbed members',
|
| 148 |
+
'member_count': 50, 'checkpoint_sha256': CHECKPOINT},
|
| 149 |
+
{'key': 'deepmind', 'label': 'DeepMind Mini', 'method': 'Not run on this daily cohort',
|
| 150 |
+
'status': 'unavailable', 'reason': 'Older starts and different weighting; no transferred score.'}],
|
| 151 |
+
metrics=curve_metrics+metrics,
|
| 152 |
+
intensity_coverage={'eligible_daily_issues': intensity['eligible_intensity_daily_issues'],
|
| 153 |
+
'unavailable_daily_issues': intensity['excluded_missing_current_intensity'],
|
| 154 |
+
'reason': 'Frozen 1.1 pipeline requires valid issue-time wind and pressure.'},
|
| 155 |
+
per_lead_track={m: daily['aggregate_equal_storm'][m]['track_error_by_lead_km'] for m in ('1.1', '1.2')},
|
| 156 |
+
sources=identities, notes=[
|
| 157 |
+
'Development evidence, not a certified unused holdout or an architecture ablation.',
|
| 158 |
+
'Metric-specific common truth masks; missing outputs and labels never become zero error.',
|
| 159 |
+
'No clean one-member daily evaluation: these 1.2 scores are always the verified 50-member mean.',
|
| 160 |
+
'Pressure metrics use 40 storms / 134 issues, not all 270 storms / 1473 issues.',
|
| 161 |
+
'No wind or radius bars in the Site main chart; diagnostic reports remain separately downloadable.',
|
| 162 |
+
'The pressure display repair does not change these immutable forecasts or central-pressure results.'])
|
| 163 |
+
|
| 164 |
+
|
| 165 |
+
def build(project, output):
|
| 166 |
+
from plot_intensity_benchmark import verify_and_summarize
|
| 167 |
+
plan = read(project/'benchmark_daily_storms_1_2/cohort.json')
|
| 168 |
+
daily = read(ROOT/'evaluation/daily_storm_final.json')
|
| 169 |
+
originals = check_daily(project, daily, plan)
|
| 170 |
+
source = project/'output/intensity-v12-v11-20261001'
|
| 171 |
+
intensity = read(ROOT/'evaluation/intensity/intensity_final.json')
|
| 172 |
+
receipt = read(ROOT/'evaluation/intensity/verification.json')
|
| 173 |
+
if sha(ROOT/'evaluation/intensity/intensity_final.json') != receipt['summary_sha256']:
|
| 174 |
+
raise ValueError('Published pressure report hash differs')
|
| 175 |
+
reproduced = verify_and_summarize(source, project)
|
| 176 |
+
for key in ('metrics', 'native_1_1_radius_USA_km', 'source_hashes', 'verification'):
|
| 177 |
+
if reproduced[key] != intensity[key]:
|
| 178 |
+
raise ValueError('Verified pressure report no longer reproduces: '+key)
|
| 179 |
+
runner = read(source/'report.json')
|
| 180 |
+
identities = {'daily_report_sha256': sha(ROOT/'evaluation/daily_storm_final.json'),
|
| 181 |
+
'intensity_report_sha256': sha(ROOT/'evaluation/intensity/intensity_final.json'),
|
| 182 |
+
'intensity_receipt_sha256': sha(ROOT/'evaluation/intensity/verification.json'),
|
| 183 |
+
'intensity_protocol_sha256': sha(source/'protocol.json'),
|
| 184 |
+
'intensity_runner_report_sha256': sha(source/'report.json')}
|
| 185 |
+
data = snapshot(daily, plan, intensity, runner, identities)
|
| 186 |
+
import csv
|
| 187 |
+
names = {}
|
| 188 |
+
with (project/'data/ibtracs/ibtracs.WP.list.v04r01.csv').open() as stream:
|
| 189 |
+
rows = csv.DictReader(stream)
|
| 190 |
+
next(rows) # units row, not a storm
|
| 191 |
+
for row in rows:
|
| 192 |
+
if row['NAME'].strip() not in ('', 'NOT_NAMED'):
|
| 193 |
+
names[row['SID'].strip()] = row['NAME'].strip()
|
| 194 |
+
records = [read(p) for p in sorted((source/'cases').glob('*.json'))]
|
| 195 |
+
data['pressure_curves'] = [dict(case_index=r['case_index'], storm_id=r['storm_id'],
|
| 196 |
+
name=names.get(r['storm_id'], r['storm_id']), issue_time_utc=r['issue_time_utc'],
|
| 197 |
+
observed={agency:r['truth']['pressure_'+agency+'_hpa'] for agency in ('JMA','USA')},
|
| 198 |
+
forecasts={'1.1':[p['central_pressure_hpa'] for p in r['v11']],
|
| 199 |
+
'1.2':r['v12_pressure_hpa']}) for r in records if r['v11'] is not None]
|
| 200 |
+
output = output.resolve()
|
| 201 |
+
if not output.is_relative_to(Path('/Volumes/D')):
|
| 202 |
+
raise ValueError('Artifacts must stay on D')
|
| 203 |
+
output.mkdir(parents=True, exist_ok=True)
|
| 204 |
+
(output/'released_daily_benchmark.json').write_text(json.dumps(data, indent=2, allow_nan=False)+'\n')
|
| 205 |
+
audit = {'status': 'complete_verified', **identities,
|
| 206 |
+
'snapshot_sha256': sha(output/'released_daily_benchmark.json'),
|
| 207 |
+
'verified_original_daily_forecasts': len(originals),
|
| 208 |
+
'all_route_scores_and_absolute_local_conversions_reproduced': True,
|
| 209 |
+
'original_forecast_sha256': originals, 'verified_pressure_member_replays': 134,
|
| 210 |
+
'missing_results_scored_as_zero': 0, 'new_inference_performed': False}
|
| 211 |
+
(output/'released_daily_verification.json').write_text(json.dumps(audit, indent=2)+'\n')
|
| 212 |
+
print(json.dumps({k:v for k,v in audit.items() if k != 'original_forecast_sha256'}), flush=True)
|
| 213 |
+
|
| 214 |
+
|
| 215 |
+
if __name__ == '__main__':
|
| 216 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 217 |
+
parser.add_argument('--project', type=Path, default=ROOT.parent)
|
| 218 |
+
parser.add_argument('--output', type=Path, required=True)
|
| 219 |
+
args = parser.parse_args()
|
| 220 |
+
build(args.project, args.output)
|
release_tools/plot_daily_storm_final.py
CHANGED
|
@@ -30,6 +30,6 @@ for ax, (key, title, unit, digits) in zip(axes.flat, panels):
|
|
| 30 |
ax.grid(axis='y', alpha=.16)
|
| 31 |
fig.suptitle('Trackformer 1.2 vs 1.1 | 270 storms · 1,473 daily forecasts', x=.07, ha='left', fontsize=17, fontweight='bold')
|
| 32 |
fig.text(.07, .922, 'Complete +6 to +120 h rollouts · each storm has equal weight · broader development evidence', color='#526174')
|
| 33 |
-
fig.text(.07, .022, 'Different input pipelines; not an
|
| 34 |
fig.subplots_adjust(top=.84, bottom=.11, left=.07, right=.98, hspace=.42, wspace=.35)
|
| 35 |
fig.savefig(ROOT / 'evaluation/trackformer_1_2_vs_1_1_270_storms_bars.png', dpi=180, facecolor='white')
|
|
|
|
| 30 |
ax.grid(axis='y', alpha=.16)
|
| 31 |
fig.suptitle('Trackformer 1.2 vs 1.1 | 270 storms · 1,473 daily forecasts', x=.07, ha='left', fontsize=17, fontweight='bold')
|
| 32 |
fig.text(.07, .922, 'Complete +6 to +120 h rollouts · each storm has equal weight · broader development evidence', color='#526174')
|
| 33 |
+
fig.text(.07, .022, 'Different input pipelines; development evidence, not an untouched test. Matched pressure uses a separate 40-storm / 134-day support.', fontsize=9, color='#526174')
|
| 34 |
fig.subplots_adjust(top=.84, bottom=.11, left=.07, right=.98, hspace=.42, wspace=.35)
|
| 35 |
fig.savefig(ROOT / 'evaluation/trackformer_1_2_vs_1_1_270_storms_bars.png', dpi=180, facecolor='white')
|
release_tools/plot_release_pressure_benchmark.py
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Plot matched central-pressure MAE and unshifted pressure-versus-time curves."""
|
| 2 |
+
import json
|
| 3 |
+
from pathlib import Path
|
| 4 |
+
import matplotlib
|
| 5 |
+
matplotlib.use('Agg')
|
| 6 |
+
import matplotlib.pyplot as plt
|
| 7 |
+
import numpy as np
|
| 8 |
+
|
| 9 |
+
ROOT=Path(__file__).resolve().parents[1]
|
| 10 |
+
data=json.loads((ROOT/'evaluation/released_daily/released_daily_benchmark.json').read_text())
|
| 11 |
+
assert data['status']=='complete_verified'
|
| 12 |
+
fig,axes=plt.subplots(2,2,figsize=(12,8))
|
| 13 |
+
colors=['#66758b','#007d78']
|
| 14 |
+
for column,agency in enumerate(('JMA','USA')):
|
| 15 |
+
metric=next(m for m in data['metrics'] if m['key']=='pressure_'+agency+'_hpa')
|
| 16 |
+
assert metric['coverage']['storms']==40
|
| 17 |
+
axis=axes[0,column]
|
| 18 |
+
values=np.array([metric['values'][m] for m in ('1.1','1.2')])
|
| 19 |
+
intervals=np.array([metric['intervals'][m] for m in ('1.1','1.2')])
|
| 20 |
+
errors=np.stack([values-intervals[:,0],intervals[:,1]-values])
|
| 21 |
+
bars=axis.bar(['1.1','1.2 · mean of 50'],values,yerr=errors,color=colors,width=.52,capsize=5)
|
| 22 |
+
axis.bar_label(bars,labels=[f'{v:.2f}' for v in values],padding=5,fontweight='bold')
|
| 23 |
+
axis.set_ylim(0,20)
|
| 24 |
+
axis.set_title(f"{agency} pressure MAE · {metric['coverage']['daily_issues']} shared starts",loc='left',fontweight='bold',fontsize=12)
|
| 25 |
+
axis.set_ylabel('Central-pressure MAE (hPa) · lower is better')
|
| 26 |
+
lower=axes[1,column]
|
| 27 |
+
issue=data['pressure_curves'][0]
|
| 28 |
+
observed=np.array([np.nan if v is None else v for v in issue['observed'][agency]])
|
| 29 |
+
lower.plot(data['cohort']['leads_hours'],observed,label=agency+' best track',color='#253643',linewidth=2.2)
|
| 30 |
+
for model,color,style in zip(('1.1','1.2'),colors,('--','-')):
|
| 31 |
+
lower.plot(data['cohort']['leads_hours'],issue['forecasts'][model],label=model,color=color,linestyle=style,linewidth=2.2)
|
| 32 |
+
lower.set_xticks([6,24,48,72,96,120])
|
| 33 |
+
lower.set_title(issue['name']+' · '+issue['issue_time_utc'][:16].replace('T',' ')+' UTC',loc='left',fontsize=10)
|
| 34 |
+
lower.set_xlabel('Forecast lead (hours)')
|
| 35 |
+
lower.set_ylabel('Actual central pressure (hPa)')
|
| 36 |
+
lower.legend(frameon=False)
|
| 37 |
+
for a in (axis,lower):
|
| 38 |
+
a.spines[['top','right']].set_visible(False)
|
| 39 |
+
a.set_axisbelow(True);a.grid(axis='y',alpha=.15)
|
| 40 |
+
fig.suptitle('Trackformer 1.1 vs 1.2 | matched central pressure',x=.075,ha='left',fontsize=17,fontweight='bold')
|
| 41 |
+
fig.text(.075,.923,'Same exact valid times · daily → storm → equal storm · 40 storms · JMA and USA separate',color='#526174')
|
| 42 |
+
fig.text(.075,.025,'Slight descriptive mean improvement; paired difference intervals include zero. 95% whole-storm intervals.\nOverlay: first eligible frozen case, not a best example. Exact-time physical hPa curves; no shift or warping. Missing is not zero.',fontsize=9,color='#526174')
|
| 43 |
+
fig.subplots_adjust(left=.075,right=.98,top=.85,bottom=.12,hspace=.43,wspace=.25)
|
| 44 |
+
fig.savefig(ROOT/'evaluation/released_daily/pressure_comparison.png',dpi=180,facecolor='white')
|
release_tools/sync_public_model_cards.py
CHANGED
|
@@ -22,6 +22,10 @@ CACHE = '/Volumes/D/typhoon_predict/.cache/huggingface-intensity'
|
|
| 22 |
HF = 'https://huggingface.co/' + REPO
|
| 23 |
GALLERY = '| Soudelor (2015) | Mangkhut (2018) | Meranti (2016) |\n'
|
| 24 |
FILMS = ('mangkhut', 'fung_wong', 'soudelor', 'meranti')
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
SYNC_FILES = (
|
| 26 |
'models/trackformer_1_2_field/README.md',
|
| 27 |
'models/trackformer_1_2_field/WIND_ESTIMATION.md',
|
|
@@ -29,6 +33,21 @@ SYNC_FILES = (
|
|
| 29 |
'models/trackformer_1_2_field/wind_estimation.py',
|
| 30 |
'release_tools/sync_public_model_cards.py',
|
| 31 |
'release_tools/test_public_model_cards.py',
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
)
|
| 33 |
|
| 34 |
|
|
@@ -50,7 +69,7 @@ def player(stem, title):
|
|
| 50 |
'src="' + media_url(stem) + '"></video>')
|
| 51 |
|
| 52 |
|
| 53 |
-
def render_card(original, github):
|
| 54 |
metadata = re.match(r'\A---\n.*?\n---\n', original, re.S)
|
| 55 |
if not metadata:
|
| 56 |
raise ValueError('Preserve existing HF YAML metadata; missing header')
|
|
@@ -96,7 +115,8 @@ def render_card(original, github):
|
|
| 96 |
resolved = (ROOT / path).resolve()
|
| 97 |
if not resolved.is_relative_to(ROOT) or not resolved.exists():
|
| 98 |
raise ValueError('Invalid or missing relative public link: ' + path)
|
| 99 |
-
|
|
|
|
| 100 |
target = HF + '/' + kind + '/' + path + ('#' + fragment if fragment else '')
|
| 101 |
return image + '[' + label + '](' + target + ')'
|
| 102 |
card = re.sub(r'(!?)\[([^\[\]\n]*)\]\(([^)\s]+)\)', link, card)
|
|
@@ -160,9 +180,18 @@ def run(output, publish=False):
|
|
| 160 |
dirty = subprocess.check_output(['git', 'status', '--porcelain', '--', 'README.md', *SYNC_FILES], cwd=ROOT, text=True)
|
| 161 |
if dirty:
|
| 162 |
raise ValueError('Commit canonical README and matching source files before publishing')
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
for name, expected in receipt['files'].items():
|
| 167 |
saved = hf_hub_download(REPO, name, revision=committed.oid, cache_dir=CACHE)
|
| 168 |
if sha(saved) != expected:
|
|
|
|
| 22 |
HF = 'https://huggingface.co/' + REPO
|
| 23 |
GALLERY = '| Soudelor (2015) | Mangkhut (2018) | Meranti (2016) |\n'
|
| 24 |
FILMS = ('mangkhut', 'fung_wong', 'soudelor', 'meranti')
|
| 25 |
+
CURRENT_FIGURES = frozenset((
|
| 26 |
+
'evaluation/trackformer_1_2_vs_1_1_270_storms_bars.png',
|
| 27 |
+
'evaluation/released_daily/pressure_comparison.png',
|
| 28 |
+
))
|
| 29 |
SYNC_FILES = (
|
| 30 |
'models/trackformer_1_2_field/README.md',
|
| 31 |
'models/trackformer_1_2_field/WIND_ESTIMATION.md',
|
|
|
|
| 33 |
'models/trackformer_1_2_field/wind_estimation.py',
|
| 34 |
'release_tools/sync_public_model_cards.py',
|
| 35 |
'release_tools/test_public_model_cards.py',
|
| 36 |
+
'release_tools/build_release_benchmark.py',
|
| 37 |
+
'release_tools/plot_release_pressure_benchmark.py',
|
| 38 |
+
'release_tools/plot_daily_storm_final.py',
|
| 39 |
+
'docs/daily_storm_benchmark.md',
|
| 40 |
+
'docs/intensity_benchmark.md',
|
| 41 |
+
'docs/trackformer_1_2_evaluation.md',
|
| 42 |
+
'evaluation/daily_storm_final.json',
|
| 43 |
+
'evaluation/released_daily/released_daily_benchmark.json',
|
| 44 |
+
'evaluation/released_daily/released_daily_verification.json',
|
| 45 |
+
'evaluation/intensity/intensity_final.json',
|
| 46 |
+
'evaluation/intensity/verification.json',
|
| 47 |
+
'evaluation/trackformer_1_2_vs_1_1_270_storms_bars.png',
|
| 48 |
+
'evaluation/released_daily/pressure_comparison.png',
|
| 49 |
+
'paper/trackformer.tex',
|
| 50 |
+
'paper/trackformer.pdf',
|
| 51 |
)
|
| 52 |
|
| 53 |
|
|
|
|
| 69 |
'src="' + media_url(stem) + '"></video>')
|
| 70 |
|
| 71 |
|
| 72 |
+
def render_card(original, github, figure_revision='main'):
|
| 73 |
metadata = re.match(r'\A---\n.*?\n---\n', original, re.S)
|
| 74 |
if not metadata:
|
| 75 |
raise ValueError('Preserve existing HF YAML metadata; missing header')
|
|
|
|
| 115 |
resolved = (ROOT / path).resolve()
|
| 116 |
if not resolved.is_relative_to(ROOT) or not resolved.exists():
|
| 117 |
raise ValueError('Invalid or missing relative public link: ' + path)
|
| 118 |
+
revision = figure_revision if path in CURRENT_FIGURES else MEDIA_REVISION
|
| 119 |
+
kind = 'resolve/' + revision if image else ('tree/main' if resolved.is_dir() else 'blob/main')
|
| 120 |
target = HF + '/' + kind + '/' + path + ('#' + fragment if fragment else '')
|
| 121 |
return image + '[' + label + '](' + target + ')'
|
| 122 |
card = re.sub(r'(!?)\[([^\[\]\n]*)\]\(([^)\s]+)\)', link, card)
|
|
|
|
| 180 |
dirty = subprocess.check_output(['git', 'status', '--porcelain', '--', 'README.md', *SYNC_FILES], cwd=ROOT, text=True)
|
| 181 |
if dirty:
|
| 182 |
raise ValueError('Commit canonical README and matching source files before publishing')
|
| 183 |
+
# Publish scientific assets first, then pin the new figures to that
|
| 184 |
+
# immutable commit in the card. Never pin new bytes to the old movie
|
| 185 |
+
# revision or let a missing future image render as a stale figure.
|
| 186 |
+
assets = api.create_commit(repo_id=REPO, repo_type='model', parent_commit=parent,
|
| 187 |
+
commit_message='Publish verified matched pressure-curve benchmark and revised technical paper',
|
| 188 |
+
operations=[CommitOperationAdd(path_in_repo=name, path_or_fileobj=str(ROOT/name)) for name in SYNC_FILES])
|
| 189 |
+
card_path.write_text(render_card(original.read_text(), github, assets.oid))
|
| 190 |
+
receipt['files']['README.md'] = sha(card_path)
|
| 191 |
+
receipt['figure_revision'] = assets.oid
|
| 192 |
+
committed = api.create_commit(repo_id=REPO, repo_type='model', parent_commit=assets.oid,
|
| 193 |
+
commit_message='Refresh model card with exact-time pressure graph similarity and pinned figures',
|
| 194 |
+
operations=[CommitOperationAdd(path_in_repo='README.md', path_or_fileobj=str(card_path))])
|
| 195 |
for name, expected in receipt['files'].items():
|
| 196 |
saved = hf_hub_download(REPO, name, revision=committed.oid, cache_dir=CACHE)
|
| 197 |
if sha(saved) != expected:
|
release_tools/test_public_model_cards.py
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
"""Portable regression checks for complete shared README publication."""
|
| 2 |
import re
|
| 3 |
import unittest
|
| 4 |
-
from sync_public_model_cards import MEDIA_REVISION, ROOT, media_url, render_card
|
| 5 |
|
| 6 |
|
| 7 |
class PublicModelCardsTest(unittest.TestCase):
|
|
@@ -47,6 +47,22 @@ class PublicModelCardsTest(unittest.TestCase):
|
|
| 47 |
with self.assertRaisesRegex(ValueError, 'relative public link'):
|
| 48 |
render_card(self.original, self.source + '\n[No secrets](../../private-file.txt)\n')
|
| 49 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
if __name__ == '__main__':
|
| 52 |
unittest.main()
|
|
|
|
| 1 |
"""Portable regression checks for complete shared README publication."""
|
| 2 |
import re
|
| 3 |
import unittest
|
| 4 |
+
from sync_public_model_cards import CURRENT_FIGURES, SYNC_FILES, MEDIA_REVISION, ROOT, media_url, render_card
|
| 5 |
|
| 6 |
|
| 7 |
class PublicModelCardsTest(unittest.TestCase):
|
|
|
|
| 47 |
with self.assertRaisesRegex(ValueError, 'relative public link'):
|
| 48 |
render_card(self.original, self.source + '\n[No secrets](../../private-file.txt)\n')
|
| 49 |
|
| 50 |
+
def test_updated_pressure_graph_and_paper_are_published_together(self):
|
| 51 |
+
revision = 'a' * 40
|
| 52 |
+
rendered = render_card(self.original, self.source, revision)
|
| 53 |
+
for text in ('Pressure graph similarity', '0.7074', '0.7118', '0.7237', '0.6637', '131 days', 'time-warped', 'slight mean improvement', '13.53 to 12.84'):
|
| 54 |
+
self.assertIn(text, rendered)
|
| 55 |
+
for path in CURRENT_FIGURES:
|
| 56 |
+
self.assertIn('/resolve/' + revision + '/' + path, rendered)
|
| 57 |
+
self.assertIn(path, SYNC_FILES)
|
| 58 |
+
for path in ('paper/trackformer.tex', 'paper/trackformer.pdf',
|
| 59 |
+
'evaluation/released_daily/released_daily_benchmark.json',
|
| 60 |
+
'evaluation/released_daily/released_daily_verification.json'):
|
| 61 |
+
self.assertIn(path, SYNC_FILES)
|
| 62 |
+
self.assertNotIn('models/trackformer_1_2_field/weights.pt', SYNC_FILES)
|
| 63 |
+
for path in SYNC_FILES:
|
| 64 |
+
self.assertTrue((ROOT / path).is_file(), path)
|
| 65 |
+
|
| 66 |
|
| 67 |
if __name__ == '__main__':
|
| 68 |
unittest.main()
|