Remove retired 270-case benchmark; retain completed 1473-forecast evaluation

README.md

@@ -77,35 +77,7 @@ Mean track error is **41.0% lower** for 1.2 on this cohort. Direction, shape and
 
 [Final report and all storm scores](evaluation/daily_storm_final.json) · [Protocol](docs/daily_storm_benchmark.md) · [Frozen issues](evaluation/release_data/daily_storm_cohort.json) · [Reproduce chart](release_tools/plot_daily_storm_final.py)
 
-## Earlier 270-case comparison: 90 storms
-
-This comparison uses **270 forecast cases from 90 storms**, with the same issue rows, observed routes and 20 six-hour leads through +120 h. The number 270 counts forecast cases. The 1.2 prediction for each case is a **mean of 50 causal input-perturbation members**; 1.1 uses its archived causal route pipeline.
-
-| Metric | 1.1 | 1.2 · mean of 50 | Preferred |
-| --- | ---: | ---: | --- |
-| Mean six-hour direction error | 56.22° | **44.82°** | Lower |
-| Route-shape similarity | 0.7273 | **0.8249** | Higher |
-| Path similarity (Fréchet) | 0.4983 | **0.5727** | Higher |
-| Mean track error, +6 to +120 h | 902.3 km | **714.4 km** | Lower |
-| Track error at +120 h | 1,833.2 km | **1,433.4 km** | Lower |
-
-![270-case direction, route-shape, path and position comparison](evaluation/trackformer_1_2_vs_1_1_270_bars.png)
-
-**What “similarity” means:** direction error compares the heading of each six-hour step, on 5,382 common valid case-leads where truth and both models move more than 1 km. Shape similarity compares the centred route curves after removing overall scale, while preserving their orientation; a score of 1 means identical shape under that comparison. Path similarity uses Fréchet distance and the observed route length, so it also responds to geographic displacement. These scores measure different aspects of the forecast and are reported separately.
-
-**Actual route alignment is the goal:** predicted and observed positions should coincide geographically at the same valid times. A high centred-shape score alone does **not** demonstrate that alignment; a displaced forecast can have the right shape and still be wrong. Position error and the unshifted route overlay must therefore be read alongside the shape score.
-
-The saved 1.1 archive had a coordinate-label bug: its `v11_local` array held absolute latitude/longitude, while the original scorer interpreted it as east/north kilometres. The new comparison converts those coordinates to the same local projection used by 1.2 and verifies a round trip back to the original coordinates. **The old 1,036.3 km score is superseded by 902.3 km.** Original forecast archives are preserved. The [published metrics](evaluation/trackformer_1_2_vs_1_1_270_metrics.json), small plot-data arrays and [reproduction script](release_tools/plot_release_270.py) record the correction.
-
-### What the routes look like
-
-These are **six selected best-performing examples from six distinct storms**, not a representative sample. Among in-domain cases with at least 300 km of observed travel, shape similarity ≥0.90 and direction error ≤30° on at least 18 common valid leads, we select the lowest 1.2 mean track errors, with one case per storm. Every curve starts at the same issue-time origin. The [selection manifest and all 270 case scores](evaluation/trackformer_1_2_showcase_selection.json) make the choice auditable; the aggregate bars above still include every case, including poor forecasts.
-
-![Observed, 1.1 and 1.2 routes on six selected best-performing examples](evaluation/trackformer_1_2_vs_1_1_route_examples.png)
-
-The 1.2 mean has better aggregate direction, shape and position scores on this previously inspected development cohort. That does not establish an overall win on every storm or pressure metric. A same-270 1.1 central-pressure prediction array was not verified in these route artifacts, so it is not assigned a pressure-error bar. The separate 1.2 report gives 15.73 hPa central-pressure MAE and 2.54 hPa area-weighted regional MSLP MAE. TIP remains a separate diagnostic: its pressure error was worse for 1.2 (30.7 vs 20.1 hPa). See [metric definitions and limitations](docs/trackformer_1_2_evaluation.md).
-
-The earlier dated partial snapshot is retained for provenance only. The completed 270-storm result at the top of this document supersedes it; the earlier 270-case comparison remains a different cohort.
+**Reading route similarity:** direction error compares headings of common moving six-hour steps. Centred shape similarity removes translation and scale but preserves orientation; it does not establish geographic overlap. Path similarity also responds to displacement. Predicted and observed routes should align at the same valid times, so these metrics must be read alongside position error and unshifted route overlays.
 
 ## Fung-wong pressure forecast — MP4
 

RELEASE_NOTES_TRACKFORMER_1_2.md

@@ -4,11 +4,7 @@
 
 This release packages a verified moving-pressure-core model as **Trackformer 1.2**. It supersedes the withdrawn, unrelated route/scalar 1.2 candidate. Use `models/trackformer_1_2_field/predict.py`; the incompatible root-level candidate and example are no longer in the current source tree. Trackformer 1.1 remains available as a separate release.
 
-The attached archive includes inference-only weights, exact source modules, an input wrapper, a data-contract/provenance manifest, detailed documentation, **270-case direction/route-shape benchmark bars**, paired route examples and recent **Surigae pressure maps with labelled isobars**. The training checkpoint was internally labelled version `1.2.73`, epoch 4; its SHA-256 is `f194a23d3f91ea76ad776dfad942fabd669eeae8b3fd665815463095367e9ee0`. The exported inference weights SHA-256 is `db49f36e85a3766defc4c172746897a1f783705d1ce8e6f9dfb8e87ae1d902cb`.
-
-On the 270-case development cohort, 1.1 versus 1.2 mean-of-50 scores are: direction error **56.22° vs 44.82°**, route-shape similarity **0.7273 vs 0.8249**, path similarity **0.4983 vs 0.5727**, and mean track error **902.3 vs 714.4 km**. Direction uses 5,382 common valid steps. The old 1.1 report had stored latitude/longitude under a local-kilometre key; the new comparison repairs that coordinate interpretation and records the correction. Original artifacts remain available. These results are development comparisons, and no matched 1.1 pressure-field bar is claimed.
-
-The route gallery shows six explicitly selected best-performing examples from distinct storms, with the selection rule and all case scores supplied. It is not representative evidence; the earlier aggregate bars retain all 270 cases. The new primary benchmark below uses **270 distinct storms / 1,473 daily issues**, with daily scores averaged per storm and storms weighted equally. The paper now presents this completed larger cohort, separately from the earlier 270-case results.
+The attached archive includes inference-only weights, exact source modules, an input wrapper, a data-contract/provenance manifest, detailed documentation, **1,473-daily-forecast / 270-storm benchmark bars**, the selected Fung-wong route-and-pressure example and recent **Surigae pressure maps with labelled isobars**. The training checkpoint was internally labelled version `1.2.73`, epoch 4; its SHA-256 is `f194a23d3f91ea76ad776dfad942fabd669eeae8b3fd665815463095367e9ee0`. The exported inference weights SHA-256 is `db49f36e85a3766defc4c172746897a1f783705d1ce8e6f9dfb8e87ae1d902cb`.
 
 The README shows the user-selected **Fung-wong MP4** in place of Yagi; the interactive PRAPIROON showcase remains removed. The forecast starts 7 November 2025 at 00 UTC and uses actual saved 50-member mean pressure fields and routes through +120 h. Central-pressure MAE is 7.36 hPa against JMA best track from IBTrACS. The broad route is similar but not perfectly overlapping: mean geographic error is 130.9 km and +120 h error is 142.3 km. This is a selected development example, not typical skill. From +66 h the forecast centre leaves the regional patch; the video explicitly labels that only the saved coarse basin field is shown there. No detailed core is invented outside coverage.
 
@@ -32,4 +28,4 @@ All **1,473 daily forecasts across 270 distinct storms** completed at 2026-09-29
 
 This is a **41.0% reduction in mean track error** on a broader development cohort, not a certified untouched test. The 1.2-only central-pressure MAE is 12.62 hPa over 40 pressure-labelled storms; basin MSLP MAE is 2.72 hPa over 270 storms. There is no matched 1.1 pressure comparison. Retrospective analyses, different input pipelines, prior model selection and complete-five-day eligibility limit interpretation.
 
-The main README chart and redesigned eight-page paper now use these final equal-storm results. The earlier 270-case/90-storm comparison remains explicitly separate. `evaluation/daily_storm_final.json` includes all storm scores; `release_tools/plot_daily_storm_final.py` reproduces the new bars. The paper includes the architecture, equations, lead-error chart, selected route/pressure curves and model isobars. **Weights and inference implementation are unchanged.**
+The main README chart and redesigned eight-page paper now use these final equal-storm results. `evaluation/daily_storm_final.json` includes all storm scores; `release_tools/plot_daily_storm_final.py` reproduces the new bars. The paper includes the architecture, equations, lead-error chart, selected route/pressure curves and model isobars. **Weights and inference implementation are unchanged.**

docs/trackformer_1_2_evaluation.md

@@ -1,33 +1,22 @@
 # Trackformer 1.2 evaluation notes
 
-These are archived development results, not an untouched generalization test.
+## Completed equal-storm benchmark
 
-## Primary 270-case route comparison
+The current comparison uses **1,473 daily forecasts across 270 Western Pacific storms**, with 20 six-hour leads from +6 through +120 h. Scores average leads within issues, daily issues within each storm, then storms equally. The completed results and all forecast SHA-256 checks are in [the final report](../evaluation/daily_storm_final.json); the [daily-storm protocol](daily_storm_benchmark.md) defines the frozen cohort and metrics.
 
-The main chart compares 270 issue cases from 90 storms and all 20 leads (+6 through +120 h). Preparation asserts exact equality of source-row order and truth arrays between the saved 1.1 cohort and the 1.2 ensemble archive. All position and shape scores use all 270 cases. The 1.2 forecasts are means of 50 input-perturbation members; 1.1 uses the archived causal route pipeline. The forecasts use different input pipelines, so the comparison is not a controlled architecture ablation.
+Trackformer 1.2 uses the mean of 50 distinct causal input perturbations, not 50 trained networks. Trackformer 1.1 uses its archived causal route pipeline. These are matched forecast outputs with different input pipelines, not a controlled architecture ablation or an untouched generalization test.
 
-### Coordinate correction
+| Equal-storm metric | 1.1 | 1.2 mean of 50 |
+| --- | ---: | ---: |
+| Mean track error | 798.4 km | 471.2 km |
+| +120 h track error | 1,646.4 km | 1,031.6 km |
+| Six-hour direction error | 51.58° | 34.96° |
+| Centred shape similarity | 0.7544 | 0.8837 |
+| Geographic path similarity | 0.5345 | 0.6560 |
 
-`benchmark_100_storms_3_days_gpu.py::build_v11_routes` returned absolute latitude/longitude, but `benchmark_strict_causal_post2021.py` treated this return value as local kilometres and saved it under `v11_local`. The new reproduction script reads that original array as degrees and projects it using the same definition as the 1.2 benchmark: east = wrapped longitude difference × 111.2 × max(cos(issue latitude), 0.2); north = latitude difference × 111.2. A round trip back to the archived latitude/longitude is asserted. The 1.2 arrays and observed local displacements are unchanged. This correction replaces the old 1.1 mean track error of 1,036.3 km with **902.3 km**. It also replaces the old direction/shape comparisons. The old reports are not overwritten.
+Direction, centred shape, geographic path and same-time position error measure different aspects of a forecast. A translated or mistimed route can have high shape similarity without overlapping the observed route. No combined weighted ranking is introduced.
 
-### Separate direction, shape and path measures
-
-- **Direction error:** absolute wrapped angle between predicted and observed six-hour displacement vectors, with the issue origin prepended. Both models use the same mask: each model and truth must move more than 1 km. There are **5,382 valid steps out of 5,400**. The mean is 56.2241° for 1.1 and 44.8241° for 1.2. Lower is better.
-- **Route-shape similarity:** centre each 20-point route, flatten its east/north coordinates, calculate cosine similarity to the centred truth and map it to [0, 1] using `(1 + cosine)/2`. Translation and a uniform scale factor do not affect this measure; orientation and the ordering of points do. It is not an accuracy percentage. Means: **0.727339 for 1.1; 0.824862 for 1.2**.
-- **Path similarity:** calculate discrete Fréchet distance on the ordered 20-point forecast/truth curves, then `exp(-distance / observed path length)`, and average over cases. The observed length includes the step from the issue origin. Means: **0.498254 for 1.1; 0.572741 for 1.2**. This responds to geographic displacement as well as the curve.
-- **Position error:** mean Euclidean error in the saved benchmark's local-kilometre projection, **902.3001 vs 714.4444 km**. At +120 h it is **1,833.2481 vs 1,433.3863 km**. These are local-coordinate distances, not newly calculated geodesic distances.
-
-No combined weighted ranking is introduced. The route gallery shows six deliberately selected best-performing examples, one per distinct storm, not a representative sample. Eligibility requires an issue inside 0–60°N / 100–180°E, observed travel ≥300 km, 1.2 shape similarity ≥0.90, and direction error ≤30° on ≥18 common valid leads. Eligible cases are sorted by ascending 1.2 mean track error, then the first six distinct storms are selected. `evaluation/trackformer_1_2_showcase_selection.json` records the rule, chosen cases and scores for all 270 cases. Axes retain actual kilometre displacements; no route is moved or rotated to improve the visual comparison. Full-cohort bars are unchanged.
-
-The full 270 cohort includes cases beyond the intended Western Pacific model domain. Its aggregate is a legacy development comparison, not a basin-specific validation claim. The previously inspected 133-case subset is separate and is not substituted for the requested 270 cases.
-
-## 270-case field benchmark
-
-The saved `benchmark_ensemble50/v173_e4/report.json` has 270 development cases (90 storms with three issue times each), 20 six-hour leads, checkpoint SHA-256 `f194a23d3f91ea76ad776dfad942fabd669eeae8b3fd665815463095367e9ee0`, 50 distinct input-perturbation member IDs and deterministic seed base 2043. Each member uses separate Gaussian fields smoothed with kernel sizes 5 (basin) and 9 (regional), centred and unit-scaled per map, with normalized amplitudes 0.025 and 0.015 respectively. Only `global_history` and `regional_history` are perturbed. Means are taken of routes, central pressures and fields on the same grids. These are neither 50 trained models nor learned latent members, and their spread is not a calibrated probability.
-
-The mean-of-50 result is 714.4 km all-lead mean track error, 1433.4 km +120 h track error, 15.73 hPa central-pressure MAE and 2.536 hPa area-weighted regional MSLP MAE. The regional field metric is on a fixed grid; only 2627 of 5400 case-leads (48.6%) had valid in-domain centres for the separately reported centre-local metric. Do not conflate low area-weighted field error with accurate cyclone-core placement. The TIP pressure error is much higher than the 270-case aggregate.
-
-No same-270 1.1 central-pressure prediction array was verified in the route artifacts, and 1.1 has no model-generated MSLP field there; neither receives an invented comparison bar. The separate TIP ten-start diagnostic still gives 383.9 vs 370.5 km track error and 20.1 vs 30.7 hPa central-pressure MAE for 1.1 vs 1.2. TIP's overlapping starts are one repeatedly inspected storm; those pressure scores are not mixed into the 270-case chart.
+The 1.2 central-pressure MAE is **12.62 hPa on 40 storms with valid labels**; area-weighted basin MSLP MAE is **2.72 hPa on 270 storms**. No matched 1.1 pressure output is available. A basin-wide average does not establish cyclone-core accuracy. The frozen cohort contains 40 recent and 230 historical storms from 1980–1999; retrospective analyses, complete-five-day eligibility and prior model selection limit interpretation.
 
 ## Recent pressure-line example
 
@@ -43,19 +32,13 @@ The Surigae archive issued **2026-09-27 12 UTC** supplies real model regional MS
 
 This example is one deterministic member from the released checkpoint. Inputs contain nine GFS analyses ending at 06 UTC (six hours before issue), issue-time JMA centre/central pressure, and static geography. Issue-time wind, prior motion and native regional history were unavailable and masked/zeroed. The stored +0 field is an input analysis; it is excluded from these forecast panels. The GFS transfer and input lag remain limitations. The selected leads keep the centre within the fixed 114–144°E, 11–41°N map. Names, times, member count and checkpoint/source hashes are in `evaluation/release_data/surigae_provenance.json`.
 
-## Reproduce the figures
-
-The public repository includes the small numerical plot inputs under `evaluation/release_data/`, including original 1.1 latitude/longitude, corrected routes, 1.2 mean routes, truth, the three pressure grids and map boundaries. After installing NumPy and Matplotlib, run:
+## Reproduce the benchmark chart
 
 ```bash
-python release_tools/plot_release_270.py
+python release_tools/plot_daily_storm_final.py
 ```
 
-This regenerates the four-panel benchmark, per-lead curve, selected-best route examples, full selection audit, isobar figure and metrics JSON. `--source-root` is optional and is only used to rebuild the small plot inputs from the original local research archives. No inference, fitting or new weather retrieval is performed by this plotting script.
-
-## Expanded benchmark in progress
-
-The [daily-storm protocol](daily_storm_benchmark.md) freezes 270 distinct Western Pacific storms and 1,473 daily issue cases. It averages leads within each daily issue, daily issues within each storm, and then storms equally. This new run does not reuse the old 270 cases as though they were 270 storms. Results remain pending until the whole frozen cohort is evaluated.
+This plots the saved completed equal-storm results; it performs no new inference or weather retrieval. Original research arrays and reproduction tools remain preserved for provenance, but the retired smaller-cohort comparison is not presented as the current benchmark.
 
 ## Validation boundary
 

paper/trackformer.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d5797fc86961d7163bba57903d0bed3da47f29f585aa567f24b75a6403c4a8c2
-size 416164
+oid sha256:d50499cb0a55b190cfccb9d97739cf055500c215ac22b16d0d4fdc9cf99ab45f
+size 416129

paper/trackformer.tex

@@ -462,8 +462,8 @@ same-time distance.
 
 \keyline{Pressure coverage.}{The 1.2 central-pressure MAE is 12.62 hPa on
 40 storms with valid labels; basin MSLP MAE is 2.72 hPa on 270 storms.
-No matched 1.1 pressure comparison is available. All 1,473 saved forecast
-hashes were verified; the earlier 270-case/90-storm study remains separate.}
+No matched 1.1 pressure comparison is available. The SHA-256 hashes of all
+1,473 saved daily forecasts were verified.}
 \reportpage
 \section{Fung-wong: route alignment and intensity}
 The selected forecast begins at 00 UTC on 7 November 2025 and uses the saved