Consolidate full 1.2 training documentation and source archive into one PDF

.gitattributes

@@ -216,3 +216,4 @@ training/trackformer_1_2/data/pressure_hpa.npy filter=lfs diff=lfs merge=lfs -te
 training/trackformer_1_2/data/geography.npz filter=lfs diff=lfs merge=lfs -text
 training/trackformer_1_2/checkpoints/v168_epoch_003.pt filter=lfs diff=lfs merge=lfs -text
 training/trackformer_1_2/checkpoints/v173_epoch_004.pt filter=lfs diff=lfs merge=lfs -text
+docs/TrackFormer_1.2_Training_and_Reproducibility.pdf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -150,11 +150,13 @@ Routes and central pressures are averaged separately. Physical member pressure f
 
 Code and project weights are **MIT licensed**, including modification, redistribution and commercial use. Source, training arrays and checkpoints are public without account approval or a download gate. [Openness inventory and remaining provenance gaps](OPENNESS.md) · [Contributing](CONTRIBUTING.md) · [Software citation](CITATION.cff). Upstream weather data retain their own terms and attribution.
 
-## Train the released 1.2 architecture
+## Complete training manual (PDF)
 
-The [open training package](training/trackformer_1_2/README.md) contains the original trainer, architecture, losses, sampler, resume logic, tests, data-preparation utilities and explicit launch configuration. The **exact six derived training-array files**, original manifest, **v168 epoch-3 warm start** and selected **v173 epoch-4 full checkpoint with optimizer/RNG state** are public on [Hugging Face](https://huggingface.co/euler314/typhoon-predict/tree/main/training/trackformer_1_2). A checksum-verifying downloader works without the old Drive setup. Source and instructions are mirrored on both repositories; large binary assets are served by Hugging Face and linked from GitHub.
+**[Download the single 106-page training and reproducibility PDF](docs/TrackFormer_1.2_Training_and_Reproducibility.pdf).** It contains data sources and processing, whole-storm partitions, normalization, tensor construction, architecture, initialization, all active losses, sampling, batches, truncated backpropagation, optimizer settings, curriculum, validation, checkpoint/resume behavior, recorded history, reproduction commands and limitations. A complete 40-file source/provenance appendix and an embedded exact-source ZIP are included; no separate Markdown pages are needed to read the method.
 
-This opens the actual released training path; it does not claim a newly reproduced training run or an untouched benchmark. The archived raw IBTrACS snapshot ID was not recorded. The report's selected epoch 4 is **not** the minimum aggregate validation-loss checkpoint: the saved history's minimum is epoch 2. See the training package for historical metadata and remaining reproducibility limits.
+The checkpoint audit distinguishes facts from the archived recipe: the selected epoch-4 optimizer records **4,384 updates**, compatible with effective batch **16** under the published four-epoch sampler; the archived notebook recipe uses batch **10** and would produce 7,016 updates. Original argv/microbatch/worker settings were not recorded. The PDF explains that discrepancy, the epoch-2 minimum validation loss versus selected epoch 4, and the remaining raw-provenance/runtime gaps.
+
+[Training source and asset index](training/trackformer_1_2/) · [Public exact corpus and optimizer checkpoints](https://huggingface.co/euler314/typhoon-predict/tree/main/training/trackformer_1_2). Project code/weights remain MIT; weather data retain upstream terms. This documentation publication does not rerun or modify any trainer.
 
 ## Get started
 

docs/TrackFormer_1.2_Training_and_Reproducibility.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ff4dfe29509fa62e5cffd91e4046d5ca2d436e606cd8ea0c14f15df33adf7b1
+size 499229

docs/trackformer_1_2_training_data.md

@@ -1,62 +1,5 @@
-# Trackformer 1.2 training data
+# TrackFormer 1.2 training data and process
 
-The **released 1.2 model's fitting cutoff is 2021**, not 2025 or 2026. Its selected checkpoint is development version 1.2.73, epoch 4. The dataset and normalization identities are frozen in the [release manifest](../models/trackformer_1_2_field/manifest.json). This document describes that release, not later 1.3 training or live-input downloads.
+The complete explanation is consolidated in **[one 106-page PDF](TrackFormer_1.2_Training_and_Reproducibility.pdf)**, rather than split across Markdown pages. It includes sources, exact corpus, train/validation/test partitions, eligibility, normalization, all input/target tensors and masks, training stages, active losses, optimizer and checkpoint details, reproduction commands, full original text-source appendices and provenance limits.
 
-## Sources actually consumed by the released training loader
-
-| Component | Dataset | Variables and sampling |
-| --- | --- | --- |
-| Storm observations / supervised labels | [NOAA IBTrACS](https://www.ncei.noaa.gov/products/international-best-track-archive)-derived `track_windows_v13.npz` | Position, prior motion, available maximum wind and central pressure, exact storm/time identities and label masks. Only eligible Western Pacific-domain windows enter this release. |
-| Basin weather | [NOAA PSL NCEP/NCAR Reanalysis 1](https://psl.noaa.gov/data/gridded/data.ncep.reanalysis.html), `basin_all_int8.npz` and the exact-time SLP atlas | MSLP, 500 hPa geopotential height, zonal/meridional winds at 850/500/200 hPa. Six-hour analyses on 100–180°E, 0–60°N; 2.5° grid, 25 × 33 cells. |
-| Native regional pressure | [ARCO-ERA5](https://github.com/google-research/arco-era5), `pressure_hpa.npy` with its frozen `plan.json` | Mean sea-level pressure on 0.25° fixed issue-centred 121 × 121 patches. 1,000 original windows across all three partitions, not native detail for every coarse window. |
-| Static geography | Same ARCO-ERA5 source, `geography.npz` | Land fraction and surface geopotential / 9.80665 to obtain elevation in metres; latitude/longitude encodings. Static context is not an additional temporal fitting period. |
-
-The eight weather channels are **MSLP, HGT500, U850, V850, U500, V500, U200, V200**. No satellite cloud imagery, humidity/vapour, SST or ocean-heat-content channel is present in this released input contract. Such features in other experiments must not be attributed to 1.2.
-
-The eastward/northward **u/v pairs encode wind direction and speed together**, not direction alone. These environmental winds evolve during the model rollout; their presence is not a separate claim of validated monsoon or surface-wind skill. The [Chinese research explanation](research_report_zh.md) uses these same source variables and partition counts.
-
-The selected dataset manifest identifies the **derived IBTrACS archive by SHA-256**, not by an independently recorded raw IBTrACS release/download ID. A newer downloaded IBTrACS file elsewhere is not evidence of which raw snapshot built these frozen windows. Available wind labels and the learned auxiliary head do not establish a harmonized agency averaging period or a validated surface-wind/radius forecast.
-
-## Temporal partitions
-
-| Role | Whole-storm years | Eligible windows / storms | Native-pressure windows | First → last eligible UTC issue |
-| --- | --- | --- | ---: | --- |
-| Fitting and normalization | 2000–2021 | 13,949 / 611 | 800 | 2000-05-05 18:00 → 2021-12-20 12:00 |
-| Validation / checkpoint selection | 2022–2023 | 1,041 / 46 | 100 | 2022-03-31 00:00 → 2023-12-17 18:00 |
-| Original test / evaluation | 2024–2025 | 1,195 / 56 | 100 | 2024-05-22 00:00 → 2025-12-02 12:00 |
-
-These are loader-eligible window counts, not independent storms or the number of gradient updates. Native-pressure windows are a **subset** of the corresponding eligible windows, not additional cases. Storm membership uses each storm's first year in the frozen archive. Windows whose −48 h history or +120 h targets cross a partition boundary are excluded. Weather normalization uses the 2000–2021 basin frames only.
-
-Every forecast history comprises nine consecutive six-hour analyses at **−48, −42, …, 0 h**. Supervised fields and storm labels at **+6, +12, …, +120 h** are targets only; they cannot be inference inputs. Missing native detail and missing intensity labels retain their masks. Validation is not gradient fitting, but it does inform checkpoint selection. Previously inspected original tests and showcases are development evidence, not a new unused holdout.
-
-The two-day span is the input format chosen for this implementation. No matched 24/48/72-hour experiment establishes it as the best history length. Likewise, the evaluation's 50 input-perturbed forecasts share one set of weights; the count is not a training-data multiplier or proof of a calibrated ensemble.
-
-## Newer data do not change the fitting cutoff
-
-The checksum-pinned caches contain records beyond the fitting years:
-
-- Derived storm-window archive: **1980-01-03 00 UTC to 2026-07-13 00 UTC**.
-- Coarse NCEP weather archive: **1980-01-01 00 UTC to 2026-03-17 18 UTC**.
-- Original native-pressure plan: fitting, validation and test windows in **2000–2025**, separated as above.
-
-Those are **source-cache endpoints**, not years all used to fit 1.2. The 2026 live/backfill pipeline uses the unchanged released model with later causal inputs; GFS transfer is experimental, and generating a new forecast does not retrain the weights. Retrospective reanalysis availability is not proof that the same inputs were available operationally at each historical issue time.
-
-## Verified identity and reproduction
-
-The [training-data receipt](../evaluation/training_data/trackformer_1_2_provenance.json) rechecks all six derived dataset file hashes, the released dataset-manifest hash, exact split counts, storm separation and native-patch membership. It records the original cache endpoints and exact earliest histories/latest targets. This is not a new raw-source-download audit.
-
-| Frozen identity | SHA-256 |
-| --- | --- |
-| Dataset manifest | `04fd0d30620137aad20173dcd59e22da2a6c20fae2f17de593e91a59517459c1` |
-| Selected training checkpoint | `f194a23d3f91ea76ad776dfad942fabd669eeae8b3fd665815463095367e9ee0` |
-| Inference-only weights | `db49f36e85a3766defc4c172746897a1f783705d1ce8e6f9dfb8e87ae1d902cb` |
-
-With the original frozen dataset available, reproduce the small receipt without inference, downloads or refreshing the training cache:
-
-```bash
-python release_tools/audit_training_data.py --dataset-root /Volumes/D/typhoon_predict/data/v164_reuse --output /Volumes/D/typhoon_predict/remote_typhoon_predict_repo/evaluation/training_data/trackformer_1_2_provenance.json
-```
-
-The exact derived arrays, original manifest, warm start and selected full training checkpoint are now publicly available in [the open training package](../training/trackformer_1_2/README.md). Original source bytes and frozen model weights are unchanged. The additive publication does not run training or modify any active trainer.
-
-The source archive includes historical ERA5 preparation utilities; its small full-field pilot is not the released six-file corpus. Use the published exact corpus to reproduce the released data boundary. No missing raw IBTrACS provenance has been invented.
+The original training-data audit remains machine-readable at [evaluation/training_data/trackformer_1_2_provenance.json](../evaluation/training_data/trackformer_1_2_provenance.json). Exact public assets and source are indexed at [training/trackformer_1_2/](../training/trackformer_1_2/). The PDF reproduces the full earlier data explanation and audit, including the unrecorded original raw IBTrACS snapshot ID; it does not invent missing provenance.

training/trackformer_1_2/README.md

@@ -1,63 +1,9 @@
-# TrackFormer 1.2: open training package
+# TrackFormer 1.2 training
 
-This is the actual released pressure-field model: development **1.2.73 (v173), epoch 4**, 21,452,595 parameters. It is separate from older route-only 1.2 experiments and later rainfall models.
+**[Read / download the complete training manual (106-page PDF)](../../docs/TrackFormer_1.2_Training_and_Reproducibility.pdf).**
 
-## Published contents
+The single PDF contains the entire training method and data process, active losses, settings, checkpoint history, reproduction commands, limitations and all 40 text-source/provenance files. Its embedded ZIP preserves their exact bytes. You do not need to open separate Markdown documents to read the method. The original source files remain executable and inspectable here.
 
-- `source/v173/`: original, byte-identical model, baseline architecture, archive loader, objective, curriculum, sampler, trainer, resume logic, design and original Metal tests.
-- `source/data_preparation/`: original historical ERA5 download/plan utilities and dataset contracts. The small full-field pilot described there is **not** the six-file training corpus.
-- `assets.json`: exact sizes and SHA-256 values for all six original derived archives, the original manifest, warm-start checkpoint and selected full training checkpoint.
-- `data/`: those exact public arrays, including original split plan, labels/masks and static geography. Binary files are hosted on Hugging Face at the identical path, not committed into GitHub source history.
-- `checkpoints/v168_epoch_003.pt`: the original, mandatory warm start, previously split across Drive files.
-- `checkpoints/v173_epoch_004.pt`: the selected full training checkpoint, with model, optimizer, normalization, four-epoch history and Python/NumPy/Torch CPU RNG states.
-- `config.json`, `checkpoint-history.json`, `SOURCE_MANIFEST.json`: archived recipe, exact saved loss history and source hashes.
-- `train.py`: a path-portability wrapper; it calls the unmodified original trainer with explicit paths and settings. No module import starts training.
-- Model export, physical pressure-map rendering and benchmark tools remain in the repository's `release_tools/` and `models/trackformer_1_2_field/`.
+Large assets are public at the same [Hugging Face training path](https://huggingface.co/euler314/typhoon-predict/tree/main/training/trackformer_1_2); `assets.json` and `download_assets.py` verify their hashes without tokens or Drive permissions. Code and weights use MIT; upstream data attribution remains in `THIRD_PARTY_NOTICES.md` and the PDF.
 
-Both [GitHub](https://github.com/yu314-coder/typhoon-predict/tree/main/training/trackformer_1_2) and [Hugging Face](https://huggingface.co/euler314/typhoon-predict/tree/main/training/trackformer_1_2) carry the source and instructions. Hugging Face serves the large assets (2,644,261,044 bytes in total); no Drive permission is required.
-
-## Download and verify
-
-From the repository root, after obtaining this source folder:
-
-```bash
-python training/trackformer_1_2/download_assets.py
-python training/trackformer_1_2/download_assets.py --verify-only
-python -m pip install -r training/trackformer_1_2/requirements.txt
-```
-
-The downloader uses only Python's standard library. It publishes each local file only after exact size/SHA verification and does not overwrite a differing existing file. Use `--revision <immutable-HF-commit>` from `PUBLICATION.md` to pin the upload; even the default `main` URL must match the fixed hashes. For a writable data/output directory on an external disk, copy the package there or set `--destination`; then pass its `data` and `checkpoints` paths to the launcher.
-
-## Preflight, then explicitly start training
-
-```bash
-python training/trackformer_1_2/train.py --device mps --output runs/retrain-1.2 --preflight
-python training/trackformer_1_2/train.py --device mps --output runs/retrain-1.2
-```
-
-Use `--device cuda` for NVIDIA or `--device cpu` for CPU. This wrapper explicitly sets the archived notebook/design recipe: effective batch 10, microbatch 4, workers 3, four epochs; pass flags to change execution settings deliberately. It uses FP32 even though the historical notebook's compatibility AMP positional flag was 1. The unchanged original entry point is also available:
-
-```bash
-python training/trackformer_1_2/source/v173/v173_pressure_train.py 10 3 0 \
-  --microbatch 4 --epochs 4 --device mps \
-  --manifest training/trackformer_1_2/data/manifest.json \
-  --warm-start training/trackformer_1_2/checkpoints/v168_epoch_003.pt \
-  --output runs/retrain-1.2
-```
-
-The original loader hashes all arrays, reconstructs a writable basin mmap beside the dataset, recomputes train-only normalization and enforces counts **13,949 / 1,041 / 1,195** with native-pressure subsets **800 / 100 / 100**. Allow space for that extra approximately 390 MB mmap and approximately 260 MB checkpoint files. Do not point preflight/training at an active trainer's data/output directories. The trainer uses a per-output lock; the same output resumes its `last.pt`. SIGINT/SIGTERM request a checkpoint at an optimizer boundary. Training horizons are 4/8/12/20 six-hour steps; validation always uses 20 steps. Four-step truncated backpropagation, gradient clipping, AdamW settings and all losses are in the original source.
-
-## Provenance and honest reproduction limits
-
-- Dataset manifest: `04fd0d30620137aad20173dcd59e22da2a6c20fae2f17de593e91a59517459c1`.
-- Warm start: `99ed776097e05ad6df03375465d9618741bed63a4ee57724d3ba98e295039d30`.
-- Selected full checkpoint: `f194a23d3f91ea76ad776dfad942fabd669eeae8b3fd665815463095367e9ee0`.
-- Inference-only weights: `db49f36e85a3766defc4c172746897a1f783705d1ce8e6f9dfb8e87ae1d902cb`.
-
-The selected full checkpoint is epoch **4**, used by the report; **it is not the smallest saved aggregate validation loss** (epoch 2 is 0.6939101; epoch 4 is 0.7283264). A fresh run's `best.pt` follows the original minimum-validation criterion and need not be the report's epoch-4 checkpoint. Do not conflate these choices. The supplied warm start makes the actual v173 training path runnable without reconstructing every earlier experimental model. Historical v168 optimization logs and complete original runtime package versions/accelerator RNG states are not archived here. The recipe is preserved from the original notebook/design, not a newly recovered complete original process argv. Bitwise end-to-end retraining across devices has **not** been demonstrated.
-
-The exact derived data are published; the raw IBTrACS release/download ID was not recorded in the frozen manifest. Current upstream downloads may differ and cannot be claimed byte-identical. Fitting uses 2000–2021, validation 2022–2023, original test 2024–2025, with complete storms separated. Later cache records are not fitting years. All future fields/positions are labels only. Repeatedly inspected tests/showcases are development evidence, not a fresh unused holdout. This publication does not rerun training, change released weights or modify active rainfall training.
-
-## Licenses and attribution
-
-Project source and project checkpoints retain the repository's MIT license. Upstream weather/track data retain their own terms and attributions, not the MIT software license; see `THIRD_PARTY_NOTICES.md`. Downloaded full checkpoints use Python pickle containers: load these identified, checksum-verified project artifacts only.
+Important: the archived notebook/config recipe uses batch 10, while the selected full checkpoint records 4,384 updates, compatible with batch 16 for the published four-epoch sampler. The PDF documents the evidence and provides both commands without claiming recovery of the original argv. See `records/selected-checkpoint-training.json` for the machine-readable inspection. The report selects epoch 4; the saved validation-loss minimum is epoch 2.

training/trackformer_1_2/SOURCE_MANIFEST.json

@@ -20,8 +20,8 @@
     },
     {
       "path": "training/trackformer_1_2/README.md",
-      "sha256": "342c43d94710355101ddc5bcad44923f47707f473eb803398f44fe2d77e8ace9",
-      "bytes": 6560
+      "sha256": "97ac00bff15891218956a4743ae0d12cddad56106683dbc3c14b4922a463023c",
+      "bytes": 1328
     },
     {
       "path": "training/trackformer_1_2/THIRD_PARTY_NOTICES.md",
@@ -40,8 +40,8 @@
     },
     {
       "path": "training/trackformer_1_2/config.json",
-      "sha256": "813f5d05a3dec5ec160b4e91eb80970fb11d88416fc7709ffc9d147fc880316b",
-      "bytes": 820
+      "sha256": "6dddaba9c17720eedb6ab8ad2564c6da926ff37ba490257a585facb59d477e4e",
+      "bytes": 1064
     },
     {
       "path": "training/trackformer_1_2/download_assets.py",
@@ -132,6 +132,11 @@
       "path": "training/trackformer_1_2/train.py",
       "sha256": "4502a87bfafb11667015070c6fad37b723b4a53723789febda075fba1c7770e8",
       "bytes": 1436
+    },
+    {
+      "path": "training/trackformer_1_2/records/selected-checkpoint-training.json",
+      "sha256": "28a8b86dfd66cc718d7cb0b165af885cbee3f39627fdb9cda0be91c021260180",
+      "bytes": 1537
     }
   ],
   "assets": [
@@ -189,5 +194,11 @@
       "bytes": 257969971,
       "group": "checkpoints"
     }
-  ]
+  ],
+  "documentation_pdf": {
+    "path": "docs/TrackFormer_1.2_Training_and_Reproducibility.pdf",
+    "sha256": "2ff4dfe29509fa62e5cffd91e4046d5ca2d436e606cd8ea0c14f15df33adf7b1",
+    "pages": 106,
+    "source_snapshot_huggingface_revision": "1f4ed985a05589076d37f170cc8053e26cec9fba"
+  }
 }

training/trackformer_1_2/config.json

@@ -25,5 +25,9 @@
   "selected_epoch": 4,
   "minimum_saved_validation_loss_epoch": 2,
   "warm_start_sha256": "99ed776097e05ad6df03375465d9618741bed63a4ee57724d3ba98e295039d30",
-  "selected_checkpoint_sha256": "f194a23d3f91ea76ad776dfad942fabd669eeae8b3fd665815463095367e9ee0"
+  "selected_checkpoint_sha256": "f194a23d3f91ea76ad776dfad942fabd669eeae8b3fd665815463095367e9ee0",
+  "recorded_selected_optimizer_steps": 4384,
+  "checkpoint_compatible_inferred_effective_batch": 16,
+  "archived_recipe_batch_is_not_verified_original_argv": true,
+  "documentation_pdf": "docs/TrackFormer_1.2_Training_and_Reproducibility.pdf"
 }

training/trackformer_1_2/records/selected-checkpoint-training.json

@@ -0,0 +1,63 @@
+{
+  "source_checkpoint_sha256": "f194a23d3f91ea76ad776dfad942fabd669eeae8b3fd665815463095367e9ee0",
+  "version": "1.2.73",
+  "architecture": "v173-moving-core-multiscale-attention",
+  "completed_epoch": 4,
+  "history": [
+    {
+      "epoch": 1,
+      "horizon": 4,
+      "train": 0.1411035786937901,
+      "validation": 0.7017406511544373
+    },
+    {
+      "epoch": 2,
+      "horizon": 8,
+      "train": 0.18676476989154306,
+      "validation": 0.6939100875934489
+    },
+    {
+      "epoch": 3,
+      "horizon": 12,
+      "train": 0.23248198316925273,
+      "validation": 0.7200751373592149
+    },
+    {
+      "epoch": 4,
+      "horizon": 20,
+      "train": 0.3557772162892601,
+      "validation": 0.7283263609942383
+    }
+  ],
+  "saved_best": 0.6939100875934489,
+  "progress": null,
+  "optimizer_step_values": [
+    4384
+  ],
+  "optimizer_parameter_states": 375,
+  "optimizer_groups": [
+    {
+      "lr": 3e-05,
+      "betas": [
+        0.9,
+        0.999
+      ],
+      "eps": 1e-08,
+      "weight_decay": 0.01,
+      "amsgrad": false,
+      "maximize": false,
+      "foreach": null,
+      "capturable": false,
+      "differentiable": false,
+      "fused": null,
+      "decoupled_weight_decay": true
+    }
+  ],
+  "epoch_draws_from_public_sampler": 17532,
+  "archived_notebook_effective_batch": 10,
+  "expected_steps_four_epochs_batch10": 7016,
+  "inferred_checkpoint_compatible_effective_batch": 16,
+  "expected_steps_four_epochs_batch16": 4384,
+  "inference_is_not_recorded_original_argv": true,
+  "optimizer_updates_performed": 0
+}