PhenoProto-SSL

Self-supervised pretraining + prototype-based finetuning for crop-type semantic segmentation on satellite image time series, built on top of a from-scratch reproduction of STCLN (XiaoleiQinn/STCLN) on the PASTIS benchmark.

Status: work in progress, actively improving. Track A (S3M pretrain, 100 epochs) and the official-baseline reproduction (Rung 0) have both completed. Two full PhenoProto-SSL finetunes (s3m_ep89, s3m_ep99) beat the official baseline reproduction but underperformed the published number and showed a val-mIoU peak-then-degrade pattern. A follow-up ablation, E_s3m_nosemi (same recipe, semi-supervised mean-teacher branch disabled), isolated the cause: with it removed, validation mIoU no longer degrades, the best checkpoint comes from epoch 50/100 instead of epoch ~14, and the fold-4 test result — mIoU 0.4833, OA 0.8133, mF1 0.6093 — essentially matches the published STCLN number (mIoU 0.4843), closing a gap that was −5.4 points down to −0.10. A follow-up 2×2 attribution grid plus 3-seed checks on both E_s3m_nosemi and the plain-linear-head baseline A_linear changed the conclusion: A_linear (official encoder, plain linear head, no PhenoProto components at all — just a corrected data pipeline) scores 0.4805 ± 0.0071 across seeds, while E_s3m_nosemi scores 0.4835 ± 0.0006 — the mean difference (+0.003) is well inside A_linear's own seed noise, so a mean-accuracy improvement from PhenoProto-SSL's components is not established. What is established, with a 147× variance ratio: the prototype+SupCon head makes rare-class predictions (Spring barley, Mixed cereal, ...) far more stable across seeds than a plain linear head on this 152-crop labeled set. The bulk of the −13.5 mIoU gap Rung 0 couldn't close was a dataloader bug, not the method. See Results for the full evidence and honest framing, and Sample predictions for real classification maps.

---

Table of contents

• What this is
• Method
• Dataset
• Experimental protocol
• Configuration
• Hardware & environment
• Status
• Results
• Sample predictions
• Reproducing
• Repository structure
• What to push to a public repo
• Known issues & engineering notes
• Acknowledgements

---

What this is

PASTIS is a Sentinel-2 optical satellite image time series dataset for panoptic/semantic agricultural parcel segmentation — 2,433 patches of 128×128 px, ~20–60 irregularly-spaced acquisitions per patch, 10 spectral bands, 20 semantic crop-type classes (+ background/void). STCLN is a published transformer-based baseline (UTAE backbone + spatio-temporal attention fusion) for this task.

This project has two goals, run as two parallel overnight tracks on one GPU:

• Track A — pretrain the UTAE encoder with a novel self-supervised masking objective (S3M, below) on unlabeled data, before any labels are used.

• Track B — an ablation ladder that starts from the officially released pretrained encoder (not our own Track A pretrain, so the two tracks can run independently) and adds one PhenoProto-SSL component at a time, to isolate which addition actually moves the needle:

  Rung	Config	Isolates
  0	official STCLN code, unmodified	the reference number
  A	our pipeline, plain linear head	pipeline-equivalence check
  B	+ prototype head	contribution of PA-Seg-style prototypes
  C	+ balanced SupCon	contribution of contrastive geometry
  D	+ semi-supervised mean-teacher	contribution of unlabeled data

Method

Backbone (unchanged from official STCLN, imported verbatim — never reimplemented, to eliminate an entire class of silent reproduction bugs): U-TAE encoder (encoder_widths=[32,256], 2-level) → LTAE2d temporal self-attention → a spatio-temporal attention (STA) fusion block (temporal max-pool branch × spatial-attention branch, ReZero-gated).

New contributions on top of that backbone:

• S3M — Spectro-Spatio-Temporal Masking (masking.py). The official S3M baseline masks a spatio-temporal dropout pattern (MASK_RATIO), then force-unmasks any frame whose NDVI-vegetation pixel fraction is below CLOUD_GATE (a "don't waste masking on cloudy/non-vegetated frames" gate). We add a third, orthogonal masking stream over the spectral/band axis (SPECTRAL_MASK_P), forcing the encoder to learn inter-band correlation structure — the signal that separates visually-similar cereal classes (spring barley / winter triticale / mixed cereal), which dominate the baseline's mIoU deficit.
• NDVI auxiliary loss (NDVI_AUX_W) — a bounded phenology-index term added to the masked-reconstruction loss, so reconstructions have to be phenologically plausible, not just spectrally close in MSE.
• PrototypeHead (phenoproto.py) — replaces the linear classifier head with a cosine-distance classifier against per-class prototypes maintained as an EMA over a per-class memory bank (PROTO_BANK=4096, PROTO_MOMENTUM=0.999). Rare classes get a low-variance mean-based decision boundary instead of a linear weight that only ever sees frequency-proportional gradient.
• Balanced Supervised Contrastive loss (losses.py) — samples a fixed number of embedding anchors per present class (SUPCON_ANCHORS=64) rather than per pixel, so majority classes (e.g. Meadow) don't dominate the embedding geometry the way they dominate cross-entropy.
• Semi-supervised mean-teacher — an EMA teacher (EMA_DECAY=0.999) provides pseudo-labels on unlabeled fold-5 patches, gated by a FlexMatch-style per-class adaptive confidence threshold (AdaptiveThreshold, base PSEUDO_THRESH=0.95) so the pseudo-label distribution doesn't collapse onto the majority class.

Dataset

PASTIS — Sentinel-2 time series, 5 official geographic folds (with a 1km buffer between them to prevent spatial leakage). Splits follow the official STCLN protocol exactly (splits.py), including its two documented quirks: the labeled train/val sets are 76 hardcoded patch IDs each (not a fold filter), and pretrain/test use full folds.

Split	Source	Patches	Crops/epoch
Pretrain (Track A)	fold 5	496	7,936 (4×4 inner crop grid)
Train (finetune)	76 hardcoded IDs	76	152 (2 fixed crop positions)
Val (finetune)	76 hardcoded IDs	76	152
Test (Rung 0/A–D eval)	fold 4	482	7,712-equiv (full 128×128, no cropping)
Unlabeled (semi-sup)	fold 5	496	—

Verified programmatically at every launch (preflight.py): zero patch overlap between train/val/pretrain and the fold-4 test set.

Experimental protocol

• Normalization: per-fold Sentinel-2 band mean/std from PASTIS/NORM_S2_patch.json, averaged across folds.
• Positions: the model receives range(T) index positions, not real calendar dates (official behaviour — USE_INDEX_POSITIONS=True).
• Class weighting: CrossEntropy with background (class 0) and void (class 19) weighted to zero; both are also excluded from all reported metrics.
• Test-time protocol: fold-4 evaluation runs on the full 128×128 patch (not the 32×32 training crops) with no test-time augmentation, mIoU computed only over classes present in the ground truth — matching the official test_STCLN.py protocol exactly.

Configuration

Key hyperparameters (config.py); [OFFICIAL] = verbatim from STCLN, changing these invalidates the comparison to the published number, [NEW] = PhenoProto-SSL addition, [HW] = hardware tuning, no effect on optimization semantics.

Group	Param	Value
Model	N_CHANNELS / N_CLASSES	10 / 20	official
	ENCODER_WIDTHS / DECODER_WIDTHS	[32,256] / [32,256]	official
	AGG_MODE, N_HEAD, D_MODEL, D_K	att_mean, 8, 256, 32	official
Pretrain	PRE_EPOCHS / PRE_BATCH	100 / 4	official
	PRE_LR / PRE_WD / PRE_CLIP	1e-4 / 0.0 / 5.0	official
	MASK_RATIO / CLOUD_GATE / NDVI_THRESH	0.4 / 0.9 / 0.2	official
	SPECTRAL_MASK_P / NDVI_AUX_W	0.25 / 0.5	new
Finetune	FT_EPOCHS / FT_BATCH / FT_LR	100 / 2 / 1e-4	official
	USE_PROTOTYPE, D_EMBED, PROTO_TAU	True, 128, 0.1	new
	SUPCON_W, SUPCON_TAU, SUPCON_ANCHORS	0.1, 0.07, 64	new
	USE_SEMISUP, UNSUP_W, PSEUDO_THRESH	True, 1.0, 0.95	new
Reproducibility	SEED	3407	official
Hardware	AMP_DTYPE / NUM_WORKERS / TF32	bf16 / 12 / True	hw only

Full config with inline rationale for every value: config.py.

Hardware & environment

GPU	1× NVIDIA RTX PRO 6000 Blackwell Server Edition, 97,887 MiB VRAM
Driver / CUDA	580.159.03 / CUDA 13.0
PyTorch	2.13.0+cu130
Python	3.12.3
Precision	bf16 autocast (Blackwell/Ampere+: bf16 has fp32 exponent range, immune to the fp16 overflow that previously caused a silent NaN pretrain run)
Peak VRAM measured	~8.7 GB @ batch=4 (pretrain fwd/bwd, single 32×32 crop-tile step)

Status

As of the last check, run on this machine:

• Track A (S3M pretrain): ✅ complete, all 100 epochs, checkpoint_99 saved clean (0 non-finite weights).
• Track B — Rung 0 (official baseline): ✅ complete, all 100 epochs. Numbers below.
• A_linear (official-checkpoint pipeline-equivalence check, linear head, no PhenoProto components): ✅ complete, evaluated on fold-4, and reran on 3 seeds (3407/42/1234) for a decisive comparison against E_s3m_nosemi — see Results. C_supcon (official encoder + prototype/SupCon head) and F_s3m_linear (S3M encoder + linear head) also complete, as attribution-grid cells. B_proto and D_full were never run and remain intentionally paused.
• s3m_ep89 — full PhenoProto-SSL finetune (prototype head + balanced SupCon + semi-supervised mean-teacher) sourced from our own Track A pretrain at epoch 89 (not the official released encoder): ✅ complete. Beats Rung 0 on the held-out test set — see Results.
• s3m_ep99 — same finetune config, sourced from the final (epoch 99) Track A pretrain checkpoint, kept as a separate run/tag from s3m_ep89: ✅ complete. Marginally beats s3m_ep89, same degradation pattern.
• E_s3m_nosemi — root-cause ablation: identical recipe to s3m_ep99 (S3M epoch-99 pretrain, prototype head, balanced SupCon) with only the semi-supervised mean-teacher branch disabled (--no_semisup): ✅ complete, 100/100 epochs (~11 min wall-clock — roughly 10x faster per epoch than the semisup runs, since dropping the branch also drops its unlabeled-batch forward/backward pass entirely, not just its loss term). Confirms the semi-sup branch was actively harmful, not neutral — see Results.
• Attribution grid (A_linear, C_supcon, F_s3m_linear, alongside E_s3m_nosemi): ✅ complete, 4/4 cells, single seed each — see Results. Superseded by the 3-seed decisive comparison below: with only one seed per cell the grid's apparent interaction effect turned out to be within seed noise.
• Seed variance: E_s3m_nosemi (seeds 3407/42/1234, std ≈ 0.0006) and A_linear (same 3 seeds, std ≈ 0.0071) both ✅ complete. The reproducible finding is not that E_s3m_nosemi beats the published baseline or A_linear on mean accuracy (neither is established) — it's that E_s3m_nosemi has ~147× lower run-to-run variance than A_linear. See Results for the full comparison.
• Two follow-up ablations on E_s3m_nosemi (real acquisition dates; Lovász-Softmax w=0.75): ✅ complete, both — neither improved on the baseline. See Results for the honest negative-result writeup (including a test-harness bug caught and fixed mid-ablation).

An open methodology question is tracked before fully trusting any official-checkpoint-sourced Track B number (Rung 0, A_linear): the official finetuning_STCLN.py imports a src.dataset.PASTIS_Dataset from a sibling repo (utae-paps-main) that was never available in this environment and had to be reimplemented from the same logic already used elsewhere in this codebase (splits.py). Line-by-line fidelity against the true upstream implementation has not yet been confirmed. This does not affect s3m_ep89/s3m_ep99/E_s3m_nosemi, which use our own PastisPatches loader throughout, not the vendored one.

Resolved: s3m_ep89 and s3m_ep99 both showed validation mIoU peaking early (epoch ~13-14) then steadily degrading through epoch 99 (see the training-curve plots in Sample predictions). Root cause isolated via E_s3m_nosemi: the semi-supervised mean-teacher branch (UNSUP_RAMP_EP=30 ramping λ_u to full strength by epoch 30) actively degrades the model rather than helping it — most likely because pseudo-labels from a teacher trained on only 152 labeled crops are confidently wrong, and the adaptive per-class threshold lowers the bar precisely for the rare classes where the teacher is least reliable. With the branch removed, validation mIoU is monotonically non-degrading and the best checkpoint comes from epoch 50/100 instead of epoch ~14. The saved model_best.tar for s3m_ep89/s3m_ep99 still correctly captured their respective peaks, so those reported results aren't invalidated — but E_s3m_nosemi supersedes them as the recommended configuration. Semi-sup as configured (linear ramp, fixed 0.95 threshold, per-class relaxation) should be considered broken; a fix (raise PSEUDO_THRESH, remove per-class relaxation, ramp over 60 epochs instead of 30) is a candidate follow-up but not yet attempted.

Results

s3m_ep89 — full PhenoProto-SSL, our own S3M pretrain (epoch 89)

Fold-4 held-out test set, full-protocol evaluation, no TTA, best checkpoint (epoch 13/100 by validation mIoU):

Metric	s3m_ep89	Rung 0 (official baseline)	Published (STCLN_wp)
mIoU	0.4298	0.3493	0.4843
OA	0.7819	0.7746	0.8170
mF1	0.5488	0.4945	0.6059
Kappa	0.7328	0.7178	—

Beats the official-baseline reproduction on every metric, and unlike Rung 0 (4 dead classes), every one of the 18 scored classes gets a nonzero IoU. Still short of the published number, consistent with Rung 0 also falling short — see the open methodology question in Status.

Rung 0 — official STCLN reproduction (100/100 epochs)

Metric	This run	Published (STCLN_wp)	Δ
mIoU	0.3493	0.4843	−0.135
OA	0.7746	0.8170	−0.042
mF1	0.4945	0.6059	−0.111
Kappa	0.7178	—	—

Validation accuracy plateaued at epoch 7 (best checkpoint = epoch 7/100) and never improved for the remaining 92 epochs. 4 of 18 scored classes (Spring barley, Potatoes, Mixed cereal, Sorghum) are never predicted at all (IoU = 0) on the test set.

s3m_ep99 — full PhenoProto-SSL, our own S3M pretrain (final epoch 99)

Same finetune config as s3m_ep89, sourced from the final Track A checkpoint instead of the intermediate epoch-89 one. Fold-4 held-out test set, best checkpoint (epoch 14/100 by validation mIoU):

Metric	s3m_ep99	s3m_ep89	Rung 0 (official)	Published (STCLN_wp)
mIoU	0.4308	0.4298	0.3493	0.4843
OA	0.7873	0.7819	0.7746	0.8170
mF1	0.5525	0.5488	0.4945	0.6059
Kappa	0.7386	0.7328	0.7178	—

Our best result so far, essentially matching s3m_ep89 (marginal improvement) — consistent with pretraining another 10 epochs having a small but real effect. Same "zero dead classes" property as s3m_ep89. Its training curve (below) shows the same peak-then-degrade pattern as s3m_ep89 (peaks ~epoch 14 at mIoU 0.46, degrades afterward, with a sharp dip around epoch 62) — confirms this is a reproducible property of the current finetune recipe, not a one-off fluke.

E_s3m_nosemi — root-cause ablation: semi-supervised branch disabled

Same recipe as s3m_ep99 (S3M epoch-99 pretrain, prototype head, balanced SupCon), --no_semisup. Fold-4 held-out test set, best checkpoint (epoch 50/100 by validation mIoU — a genuinely late epoch, not an early-stopped snapshot):

Metric	E_s3m_nosemi	s3m_ep99	s3m_ep89	Rung 0 (official)	Published (STCLN_wp)
mIoU	0.4833	0.4308	0.4298	0.3493	0.4843
OA	0.8133	0.7873	0.7819	0.7746	0.8170
mF1	0.6093	0.5525	0.5488	0.4945	0.6059
Kappa	0.7668	0.7386	0.7328	0.7178	—

Essentially matches the published baseline (mIoU −0.0010, OA −0.0037) and mF1 exceeds it (0.6093 vs 0.6059). Zero dead classes among the 18 scored. This closes the gap that stood at −5.4 mIoU (s3m_ep99 vs published) down to −0.10 — almost entirely attributable to the harmful semi-sup branch identified above, not to any remaining architecture or pretraining deficiency.

Per-class IoU, E_s3m_nosemi vs Rung 0 — every previously-dead class is now scored, and several rare classes are strong:

Class	Rung 0 IoU	E_s3m_nosemi IoU	Support
Spring barley	0.0000 (dead)	0.2860	53,762
Potatoes	0.0000 (dead)	0.3912	22,613
Mixed cereal	0.0000 (dead)	0.1693	42,769
Sorghum	0.0000 (dead)	0.2226	32,024
Beet	—	0.8593	71,047
Winter rapeseed	—	0.8741	129,119
Corn	—	0.8701	688,289

The two weakest classes are Winter triticale (0.066) and Fruits/vegetables/ flowers (0.216) — both named as limitations in the original STCLN paper, and still not solved by S3M's spectral stream. Full per-class table: logs/eval_E_s3m_nosemi_fold4.log.

Honest framing (revised after the multi-seed comparison below): against the true published number, PhenoProto-SSL (E_s3m_nosemi) is a statistical tie, not a clear win. It also turns out not to reliably beat the plain-linear-head A_linear baseline either — see the decisive comparison below. The headline contribution of this project is not "beats STCLN" or even "beats a correct linear-head baseline" — it's (a) closing the reproduction gap that the official code alone could not close on this hardware/protocol (Rung 0 → 0.48-ish is +13-14 mIoU, and this is almost entirely a dataloader fix, not a PhenoProto-SSL contribution), (b) eliminating all 4 dead classes seen in Rung 0 (both A_linear and E_s3m_nosemi do this — a property of the corrected pipeline, not specifically the prototype head), (c) the diagnostic finding that the paper's own semi-supervised design, as specified, is harmful in this low-label regime (152 crops), and (d) a genuinely new finding that the prototype+SupCon head substantially reduces run-to-run variance, even though it doesn't clearly move the mean.

Attribution grid — what actually contributed

E_s3m_nosemi's result raises an obvious question: is the gain from S3M pretraining, from the prototype+SupCon head, or from just having a correct data pipeline in the first place? A 2×2 grid isolates each factor, holding everything else (loader, protocol, no semi-sup) fixed. All four cells use the same finetune recipe and are evaluated on the fold-4 held-out test set:

	Linear head, no SupCon	Prototype + SupCon
Official released encoder	A_linear: 0.4808	C_supcon: 0.4720
S3M encoder (ours)	F_s3m_linear: 0.4581	E_s3m_nosemi: 0.4833

Row effect (S3M pretraining vs. the official encoder): +0.0113 mIoU with the prototype+SupCon head, but −0.0227 with a plain linear head. Column effect (prototype+SupCon vs. linear): +0.0252 on the S3M encoder, but −0.0088 on the official encoder. Neither factor has a consistent sign on its own — there is no clean "S3M pretraining helps" or "the prototype head helps" story. All four cells sit within a 2.5-point mIoU band of each other.

Reading — revised after multi-seed data (see below): at a single seed each, the grid looked like a real interaction — S3M-pretrained features seemed to need the prototype/contrastive head to become competitive (F_s3m_linear 0.4581 → E_s3m_nosemi 0.4833), while the officially-released encoder seemed to already be linearly separable and gain nothing from the extra head (A_linear 0.4808 → C_supcon 0.4720). Running A_linear on three seeds instead of one (below) showed its single-seed 0.4808 was unremarkable — A_linear alone swings ±0.007 by seed, more than large enough to explain those margins without any real interaction. The grid's cell differences are not distinguishable from seed noise given only one seed per cell; a rigorous version of this grid would need 3 seeds in every cell, which was not run (out of scope for this session — flagged as a limitation, not fixed). What is solid: A_linear (official encoder, plain linear head, just our corrected data pipeline — 152-crop protocol, no semi-sup, zero PhenoProto components) reaches the same ballpark as every other cell, meaning a meaningful share of the original −5.4 mIoU gap from Rung 0 was pipeline/protocol correctness, not any PhenoProto-SSL component.

Seed variance on E_s3m_nosemi (is 0.4833 real?)

Same config, three seeds, fold-4 test set:

Seed	mIoU
3407	0.4833
42	0.4831
1234	0.4842
mean ± std	0.4835 ± 0.0006

Run-to-run variance is tiny (std ≈ 0.06 mIoU points) relative to the 0.10-point gap to the published number. Taken alone, the claim "PhenoProto-SSL matches the published STCLN baseline within run-to-run variance" looked solid — but it's only half the comparison. See below.

Decisive comparison: A_linear vs E_s3m_nosemi under matched seeds

The real question isn't "does E_s3m_nosemi match the published number," it's "does E_s3m_nosemi beat the simplest possible correct baseline," i.e. A_linear — official encoder, plain linear head, no PhenoProto components, no 10-hour S3M pretrain. Ran A_linear on the same three seeds:

Seed	A_linear mIoU	E_s3m_nosemi mIoU
3407	0.4747	0.4833
42	0.4783	0.4831
1234	0.4884	0.4842
mean ± std	0.4805 ± 0.0071	0.4835 ± 0.0006
range	[0.4747, 0.4884]	[0.4831, 0.4842]

Variance ratio: 147×. A_linear's spread (0.0137) alone covers more than 10× E_s3m_nosemi's entire range (0.0011). The +0.003 mean gap in E_s3m_nosemi's favor is far smaller than A_linear's own seed-to-seed noise — not a defensible accuracy claim. Traced the source of A_linear's instability to specific rare classes, which swing hard by seed under a plain linear head:

Class	A_linear seed 3407	seed 42	seed 1234	swing
Spring barley	0.360	0.250	0.238	0.122
Mixed cereal	0.144	0.094	0.121	0.050
Winter triticale	0.086	0.118	0.161	0.075

Honest verdict: PhenoProto-SSL's prototype+SupCon head does not demonstrate a reliable mean-mIoU improvement over a correctly-piped plain linear-head baseline — that claim is retracted from earlier framing in this README. What the head does demonstrably do is cut rare-class / run-to-run variance by roughly two orders of magnitude (147×) — a real, mechanistically-plausible effect (SupCon's per-class-balanced anchor sampling gives every class, including rare ones, a consistent gradient signal every step, regardless of how many pixels of it happen to fall in a given 32×32 training crop; a plain linear head only gets gradient for classes actually present in the current crop, which is noisy at 152 crops total). That's a legitimate, useful finding for a label-scarce setting even though it isn't the accuracy win originally hoped for.

Ablation: real acquisition dates instead of position indices

The official protocol (and our S3M pretrain) feeds the model range(T) as temporal positions — it knows acquisition order but not calendar date. Several confused class pairs (Spring vs. Winter barley; Winter triticale vs. Winter durum wheat vs. Soft winter wheat) are distinguished largely by sowing/green-up timing, not spectral signature, so real dates were a candidate fix. Tested cheaply: finetune-only switch (--real_dates), S3M encoder unchanged (still pretrained with index positions).

A methodology note first: the first evaluation of this checkpoint showed a catastrophic mIoU of 0.1238 — collapsed from 0.48. This was not a real result: evaluate_fold4.py had no --real_dates flag, so it evaluated a checkpoint trained on real-date positions using index positions instead, a train/test mismatch introduced by the test harness, not the model. Caught before being reported, fixed (evaluate_fold4.py --real_dates now exists), and only the evaluation was rerun (not the 100-epoch finetune, since nothing about training was wrong).

Corrected result:

Metric	E_s3m_nosemi (baseline)	E_s3m_nosemi_realdates	Δ
mIoU	0.4833	0.4730	−0.0103
Spring barley IoU	0.2860	0.3358	+0.0498
Winter triticale IoU	0.0662	0.0599	−0.0063

Partial support for the hypothesis (Spring barley improved, as predicted), but the net effect across all 18 classes is negative — most likely the predicted train/pretrain positional mismatch (caveat flagged before running this) outweighs the calendar-date signal. Does not justify a fresh 10h S3M pretrain with real dates on this evidence; the idea would need the encoder itself pretrained with real dates to be tested fairly.

Ablation: Lovász-Softmax loss (w=0.75)

Directly optimizes an IoU surrogate instead of pixel cross-entropy alone, hypothesized to help low-IoU classes disproportionately.

Metric	E_s3m_nosemi (baseline)	E_s3m_nosemi_lovasz	Δ
mIoU	0.4833	0.4803	−0.0030
OA	0.8133	0.8123	−0.0010
mF1	0.6093	0.6091	−0.0002

No improvement — a small decline, larger than the ±0.0006 seed-noise band, so likely a real (if minor) negative effect rather than noise. Does not support adding Lovász at this weight for this recipe.

Rungs A–D (PhenoProto-SSL ablations from the official checkpoint)

Not yet available — see Status.

Sample predictions

True-color composite / ground truth / prediction / error overlay on real PASTIS fold-4 test patches, s3m_ep89 model, generated by visualize_predictions.py (chosen as the 6 most class-diverse test patches — harder and more informative than random ones, most of which are >90% a single majority class):

Patch 40038 — mIoU 0.333, OA 0.723. Field boundaries and the two majority classes (Meadow/Corn) are largely correct; some confusion with Grapevine.

Patch 20368 — mIoU 0.164, OA 0.405. A genuinely hard patch (dense village + many thin/mixed parcels) — a real weakness, not cherry-picked.

More patches: patch_40019 · patch_40086 · patch_20156 · patch_20016

Training curve (validation mIoU/OA/mF1/Kappa per epoch) — shows the epoch-13 peak and subsequent degradation noted in Status:

s3m_ep99 (our best result), same 6 patches for direct comparison:

Patch 40038 — mIoU 0.428 (vs 0.333 for s3m_ep89), OA 0.796.

More s3m_ep99 patches: patch_40019 · patch_40086 · patch_20156 · patch_20016 · patch_20368

E_s3m_nosemi (best result — matches published mIoU), same 6 patches:

Patch 40019 — mIoU 0.517, OA 0.808, the strongest of the 6 diverse patches.

Validation mIoU/OA/mF1/Kappa per epoch — no peak-then-degrade pattern; climbs and holds, best checkpoint at epoch 50/100. Compare against the s3m_ep99 curve above.

More E_s3m_nosemi patches: patch_40086 · patch_20156 · patch_20016 · patch_40038 · patch_20368

Regenerate for any checkpoint:

python3 visualize_predictions.py \
    --ckpt checkpoints/finetune/<tag>/model_best.tar --tag <tag> \
    --log logs/finetune_<tag>.log --n 6

Reproducing

# one-time setup: fixes PASTIS permissions, clones the official STCLN repo,
# checks Python deps and GPU/Blackwell compatibility
bash setup.sh

# mandatory gate — verifies splits are leak-free, class names match the
# official list, a real batch loads with correct shapes, pretrain/finetune
# forward+backward are finite, and reports peak VRAM. Exit 0 = safe to launch.
python3 preflight.py

# launch both tracks, detached in tmux
bash tmux_night.sh start

# any time, from any shell:
bash tmux_night.sh status      # GPU + both log tails + checkpoint progress
bash tmux_night.sh attach      # live view (Ctrl-b then d to detach)
bash tmux_night.sh results     # every metric found in the logs
bash tmux_night.sh stop        # kill the session

Every training script checkpoints every epoch (latest.tar) and auto-resumes from it on the next tmux_night.sh start — safe to interrupt for a VM pause/restart without losing more than one epoch of progress.

Environment overrides (for running on a different machine): PASTIS_ROOT, EXP_ROOT, REF_ROOT env vars override the dataset/exp/ reference-repo paths hardcoded as defaults in config.py.

Repository structure

phenoproto/
├── config.py               all paths + hyperparameters, inline rationale
├── splits.py                protocol-correct fold splits, class names, PastisPatches dataset
├── masking.py                S3M masking (official baseline + spectral stream) + reconstruction loss
├── phenoproto.py              PrototypeHead, PhenoProtoClassifier/Pretrain model wrappers
├── losses.py                 BalancedSupCon, Lovasz-softmax, AdaptiveThreshold, consistency loss
├── STCLN.py                  official UTAE/UTAEClassification backbone (vendored, unmodified)
├── pretrain_s3m.py            Track A entrypoint
├── finetune_phenoproto.py     Track B (Rungs A-D) entrypoint
├── evaluate_fold4.py          official fold-4 test protocol (--official flag for Rung 0's architecture)
├── visualize_predictions.py   true-color/GT/prediction/error maps + training-curve plots
├── preflight.py               pre-launch correctness/sanity gate
├── setup.sh                  one-time environment setup
├── track_b.sh                 Rung 0 + ablation ladder orchestrator
├── tmux_night.sh               overnight dual-track tmux orchestrator, resume-aware
└── logs/, checkpoints/, figures/  generated at runtime — see below

What to push to a public repo

This is a research project with a 37GB dataset and multi-GB checkpoints alongside the code — push code and documentation, not data or artifacts.

Push:

config.py  splits.py  masking.py  phenoproto.py  losses.py
pretrain_s3m.py  finetune_phenoproto.py  evaluate_fold4.py  preflight.py
visualize_predictions.py
setup.sh  track_b.sh  tmux_night.sh
README.md

Add before pushing (currently missing from this working copy):

• requirements.txt / environment.yml — pin torch, numpy, scikit-learn, tensorboard, pandas at minimum (see Hardware for the exact versions this was validated against).
• LICENSE — pick one; note that STCLN.py is vendored from XiaoleiQinn/STCLN and its own license terms should be checked/carried forward for that file specifically.
• .gitignore covering at minimum: __pycache__/, *.pyc, checkpoints/, logs/, cache/, *.tar.

Do not push:

• checkpoints/ — every .tar is 24–50MB; the pretrain dir alone is currently 683MB. Use Git LFS or a model registry (e.g. a separate HF Hub model repo) if checkpoints need to be distributed, not the code repo.
• logs/ — run logs and TensorBoard event files, fully regenerable.
• __pycache__/.
• The PASTIS dataset itself (37GB) — link to the official source instead.
• stcln_ref/ — this is a separate clone of the official STCLN repo (has its own .git) living alongside this project on disk, not a subdirectory of it; don't nest it into this repo. setup.sh clones it fresh.

Before pushing, also note: config.py's PASTIS_ROOT/EXP_ROOT/ REF_ROOT defaults are currently hardcoded to this machine's absolute paths (/home/ubuntu/IB-Connect-ver-2/test/...). They're already env-var-overridable (see Reproducing), but the fallback defaults leak this machine's path structure — harmless, but worth a find/replace pass to a placeholder before a public push.

Known issues & engineering notes

• Official repo required non-trivial repair to run at all. As cloned, finetuning_STCLN.py imported a src package from a sibling repo never included in the clone, hardcoded CUDA_VISIBLE_DEVICES='2' (breaks on any single-GPU box), hardcoded the dataset path inside main() regardless of its own --datadir flag, and needed tensorboard installed. It also crashed at runtime on pandas.DataFrame.append() (removed in pandas 3.0) and on a None-guard bug in its optional Visdom logger. All patched in-place with inline comments explaining each change; see git diff on stcln_ref/PASTIS/ for the full patch (not part of this repo — see What to push).
• CLOUD_GATE mostly neutralizes the spatio-temporal mask on real PASTIS data. The official masking gate force-unmasks any frame under 90% NDVI-vegetation coverage; measured on real pretrain-fold frames, the median vegetation fraction is 0.42, so the gate fires on ~98% of frames. CLOUD_GATE is left at its official value of 0.9 (unchanged) — this means S3M's effective tested contribution, as currently configured, is concentrated in the added spectral masking stream rather than the spatio-temporal stream. preflight.py's sanity bound on mask visible fraction was recalibrated from [0.35, 0.75] to [0.70, 1.0] to match this measured, expected behavior rather than an incorrect assumption.
• Batch size was tested and reverted. Raising PRE_BATCH (4→12→24) to use more of the 97GB card was tried; 24 nearly OOM'd with Track B running concurrently, and even the safe 12 measured slower wall-clock per epoch than 4 — this workload isn't GPU-compute-bound at this model/crop size. Reverted to the official value.

Acknowledgements

Backbone architecture and baseline training/eval protocol from XiaoleiQinn/STCLN. Dataset: PASTIS (Garnot & Landrieu).