Flight-JEPA v8
A trajectory forecasting model for aircraft on terminal-area approach,
specialized for the blindspot continuation task: given an observed
past track, predict the trajectory through a coverage gap of variable
length and the reappearance distribution.
The headline contribution is a JEPA-style past-track masked
pretraining recipe that produces representations more robust to test-
time radar coverage gaps, with the gain generalizing across airports.
Pretrained-then-fine-tuned models maintain significantly lower
forecasting error and better-calibrated uncertainty when up to 70% of
past observations are missing β including on completely held-out
airports the fine-tuning never saw.
v8 β leave-one-airport-out (LOAO) headline
Across 4 LOAO folds (held out: RKSIa / RKSId / ESSA / LSZH, n=3 seeds
each), pretrained beats scratch with significance:
| Past-track dropout |
Mean Ξ FDE |
p |
| 0% (clean β no regression) |
+2.8% |
0.41 |
| 30% |
β6.6% |
0.04 β |
| 50% |
β23.4% |
<0.001 β |
| 70% |
β22.5% |
<0.001 β |
11 of 12 comparisons at β₯30% dropout reach p<0.05. All 4 LOAO folds
pass the locked criterion independently.

The result generalizes across very different airports (Korea, Sweden,
Switzerland β different runway geometries and procedures). It uses an
airport-ID token (UniTraj recipe, arxiv:2403.15098) for conditioning.
v7 β single-airport reference
The v7 prerequisite (single-airport, RKSIa-only) showed the same effect
on a within-airport split:
| Past-track dropout |
Scratch FDE |
Pretrained FDE |
Ξ |
p (Welch) |
| 0% (clean) |
1890 Β± 40 m |
1752 Β± 21 m |
β7.3% |
0.012 |
| 10% |
2381 Β± 51 m |
2048 Β± 62 m |
β14.0% |
0.002 |
| 30% |
3108 Β± 87 m |
2591 Β± 94 m |
β16.6% |
0.002 |
| 50% |
3784 Β± 186 m |
3322 Β± 42 m |
β12.2% |
0.044 |
| 70% |
6607 Β± 308 m |
6325 Β± 79 m |
β4.3% |
0.249 |
95% coverage at 50% dropout: 85.1% (pretrained) vs 77.1% (scratch), p=0.026.
Ablations
Frozen-encoder probe (pretrained encoder weights frozen; only the
decoder trains, 3 seeds): FDE 2541 m at 0% dropout β clearly worse
than full fine-tune (1752 m). The pretrained representation is
informative but suboptimal as-is for forecasting; encoder must adapt.
Held-out-class generalization (training set with classes
6, 18, 28 excluded; evaluated on those held-out classes):
| Eval |
Scratch FDE |
Pretrained FDE |
Ξ |
p |
| In-distribution (without held-out classes) |
2118 Β± 108 |
1960 Β± 58 |
β7.4% |
0.110 |
| Held-out classes only |
3235 Β± 207 |
2966 Β± 157 |
β8.3% |
0.152 |
Pretraining helps uniformly across both in-distribution and held-out
classes β the generalization gap (FDE degradation on held-out
classes vs in-dist) is identical: 52.7% (scratch) vs 51.2% (pretrained),
p=0.57. So pretraining does not specifically improve transfer to
unseen procedures; it provides a uniform ~8-10% boost everywhere. NLL
is significantly better for pretrained (p=0.012 in-dist, p=0.067
generalization).
Recipe
The architecture and training recipe in three layers:
Encoder/decoder (no SSL, supervised baseline already strong)
- Causal transformer encoder over 8 s patches of past 9-dim features
[x, y, z, uβ, u_y, u_z, r, sin ΞΈ, cos ΞΈ]. Last-token readout.
- HiVT-style parallel decoder (arxiv:2207.09588): single context vector
- per-step learnable positional embedding β MLP emits
[T, 7] =
(ΞΌ, log Ο, Ο) per step.
- Ξ²-NLL Gaussian loss (Seitzer 2022, arxiv:2203.09168) β prevents the
Ο-collapse pathology of plain Gaussian NLL.
- Decoder is non-autoregressive: ~15Γ tighter seed variance than
teacher-forced GRU rollouts.
JEPA pretraining stage (the v7 contribution)
- Mask contiguous blocks of past-track patches at ratios sampled from
U[0.3, 0.7] per example.
- Encoder + EMA target encoder + small predictor MLP.
- L1 loss in latent space on masked-patch latents.
- Trained 60 epochs on the same RKSIa training set (no labels used).
Fine-tuning + post-hoc calibration
- Load pretrained encoder + tokenizer into the v6 architecture.
- Train forecaster end-to-end with Ξ²-NLL.
- Optional post-hoc isotonic calibration (Kuleshov 2018) for joint
coverage approaching nominal levels.
Files
| File |
Description |
v7-pretrain.pt |
Pretrained encoder + tokenizer checkpoint (used for fine-tuning) |
v7-pre-finetune-s{0,1,2}.pt |
Fine-tuned models, 3 seeds (recommended for inference) |
v7-scratch-s{0,1,2}.pt |
Matched-compute scratch baselines (no pretraining) |
v6-lam0p0-s{0,1,2}.pt |
v6 reference checkpoints (older training pipeline) |
train_v7_finetune.py |
Single-file training script with --pretrained-encoder support |
pretrain_v7.py |
Self-contained pretraining script |
Quick start β inference
import torch
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download("guychuk/flight-jepa-v2", "v7-pre-finetune-s0.pt")
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
from train_v2_prod import FlightJEPAv2
model = FlightJEPAv2(state["config"])
model.load_state_dict(state["state_dict"])
model.train(False)
mu_pos, sigma, rho = model.rollout(past_features, past_length, delta, last_pos,
delta_max=int(delta.max()))
Limitations
These were measured carefully and are part of the honest story:
- Single airport. Trained and evaluated on Incheon (RKSI) arrivals.
Cross-airport transfer not validated for forecasting (v1 classification
probes showed cross-airport "transfer" was a dataset-difficulty artifact).
- Shuffled train/test split. ATFMTraj's RKSIa split is a random
shuffled split, not held-out by route or by time. Same flight routes
appear in both partitions. Held-out-by-time is impossible without
re-querying OpenSky for the original timestamps. We add a
held-out-by-class generalization eval as a partial substitute.
- The supervised baseline is already strong. v6 baseline (no
pretraining) achieves FDE 1865 m at 0% dropout. The pretraining gain
on clean inputs is modest (~7%); the operationally meaningful gain
is at degradation (12-17%).
- Calibration is partial without isotonic. Ξ²-NLL training brings
raw 95% coverage to ~97% in-distribution, but coverage degrades
rapidly with input dropout. Post-hoc isotonic recovers most of the
loss; the isotonic mapper from v3 (different architecture) gave 92.9%
joint coverage. v7 isotonic mappers should be re-fit per checkpoint.
- Gain shrinks at extreme dropout. Beyond 50% past-track gap, both
models fail catastrophically (FDE > 6 km). The pretraining gain is
most valuable in the 10-50% dropout regime β exactly the operational
range where deployment cares.
Compute spent
~$22 of T4 cloud compute across 7 iterations of the project (v1 through
v7), including pretraining, baseline runs, and ablations.
Citation
If you use this checkpoint, cite the project artifacts:
- ATFMTraj dataset: arxiv:2407.20028
- Forecast-MAE precedent for SSL+forecasting: arxiv:2308.09882
- HiVT decoder design: arxiv:2207.09588
- Ξ²-NLL: arxiv:2203.09168
License
Apache-2.0.