Karar-git commited on
Commit
2b9ed19
·
verified ·
1 Parent(s): 47263de

Winner models, code, notebooks, data and results

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ models/zero_shot_lgbm.txt filter=lfs diff=lfs merge=lfs -text
37
+ results/f1_vs_n_full_city.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,296 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - tabular-classification
4
+ - lightgbm
5
+ - remote-sensing
6
+ - landsat
7
+ - domain-adaptation
8
+ - few-shot
9
+ library_name: lightgbm
10
+ ---
11
+
12
+ # Building age from Landsat: Madrid → Amsterdam (Hack4Dev 2026)
13
+
14
+ This repo holds the two winning models from our Iraq Hackathon 2026 / Hack4Dev
15
+ entry "Building Age Classification from Satellite Imagery", together with
16
+ the code, data and notebooks needed to reproduce them or compare against them.
17
+
18
+ **The task.** Predict a building-age class (1–4) for each 30 m Landsat pixel.
19
+ Train on Madrid and transfer to Amsterdam. Scoring is macro F1 on Amsterdam
20
+ after you are given **n = 5, 25, 50, 100 or 200 labelled Amsterdam pixels per
21
+ class** (the few-shot protocol).
22
+
23
+ | class | Amsterdam | Madrid |
24
+ |---|---|---|
25
+ | 1 | before 1945 | before 1960 |
26
+ | 2 | 1945–1984 | 1960–1984 |
27
+ | 3 | 1984–2004 | 1984–2004 |
28
+ | 4 | 2004–2024 | 2004–2024 |
29
+
30
+ ## The two winners
31
+
32
+ | | **Zero-shot LightGBM** | **Retrained LightGBM** |
33
+ |---|---|---|
34
+ | What it is | LightGBM trained on Madrid only. It never sees an Amsterdam label. | A fresh LightGBM trained on all of Madrid plus the n labelled Amsterdam pixels per class, with those pixels weighted 30×. |
35
+ | When to use it | You may not use any Amsterdam labels | You are given a few labelled target pixels, which is the official protocol |
36
+ | Amsterdam macro F1 | **0.590–0.593** at every n (it ignores the labels; it drifts only because each draw's labelled pixels leave the scored set) | **0.656** average over n, **0.720** at n=200 |
37
+ | Saved model | `models/zero_shot_lgbm.txt` | Refit for every draw, so there is no single model. `models/retrained_n{25,200}_rep0_lgbm.txt` are two example draws. |
38
+ | Training cost | A few minutes on a 12-thread CPU | ~1–4 min per draw (84 s mean on a 12-thread CPU) |
39
+
40
+ Both models use the same 393 features and the same decision rule. All numbers
41
+ are on **all of Amsterdam** (25,992 pixels), minus the labelled pixels of each
42
+ draw.
43
+
44
+ ### Results on all of Amsterdam
45
+
46
+ | model | n=5 | n=25 | n=50 | n=100 | n=200 | average | peak |
47
+ |---|---|---|---|---|---|---|---|
48
+ | **Retrained LightGBM** (5 draws per n) | 0.5658 ± 0.0108 | 0.6284 ± 0.0120 | 0.6697 ± 0.0087 | 0.6973 ± 0.0038 | **0.7204 ± 0.0060** | **0.6563** | **0.7204** |
49
+ | Prototype, within-class Mahalanobis (12 draws) | 0.5040 ± 0.0402 | **0.6361 ± 0.0170** | 0.6733 ± 0.0135 | 0.7023 ± 0.0067 | 0.7141 ± 0.0031 | 0.6460 | 0.7141 |
50
+ | **Zero-shot LightGBM** (20 draws) | **0.5928** | 0.5925 | 0.5922 | 0.5918 | 0.5903 | 0.5919 | 0.5928 |
51
+ | Organisers' prototype method, re-run by us on our 393 features (20 draws) | 0.5066 | 0.5821 | 0.5965 | 0.6034 | 0.5972 | 0.5772 | 0.6034 |
52
+ | *Organisers' prototype, as reported in their `4-Modelling.ipynb` (their 60 features, their draws)* | *0.5437* | *0.5986* | *0.6079* | *0.6129* | *0.6150* | *0.5956* | *0.6150* |
53
+ | *Organisers' reported zero-shot Random Forest (60 features)* | *0.3427* | | | | | | |
54
+
55
+ Mean ± std over draws. Each draw is a fresh random sample of labelled pixels,
56
+ so the ± is the spread you get from which pixels you happen to be handed.
57
+
58
+ The two italic rows are the organisers' own figures, copied from their
59
+ notebook. Their draws and evaluation set are not ours, so compare them
60
+ loosely. Their prototype scores higher on their 60 features than on our 393.
61
+ Both of our winners still beat it at every n.
62
+
63
+ **Best by prize category:**
64
+ - **Average:** retrained LightGBM, 0.6563.
65
+ - **Peak:** retrained LightGBM at n=200, 0.7204.
66
+ - **n=25:** the prototype, 0.6361. It is implemented in `src/gbdt.py`
67
+ (`mc_prototype(metric="within")`) but not packaged as a model here.
68
+ - **n=5:** zero-shot LightGBM, 0.5928.
69
+
70
+ ![macro F1 against labelling budget](results/f1_vs_n_full_city.png)
71
+
72
+ ### The fine-tuned LLM we also tried, for comparison
73
+
74
+ We also fine-tuned Qwen3-4B-Base with LoRA on text prompts that describe each
75
+ Madrid pixel. It took about 2 h on an A100: 1,513 steps at ~0.2 it/s.
76
+ - **Held-out Madrid areas:** it went from 0.305 untrained to 0.537 macro F1.
77
+ These come from the fine-tuning notebook's validation output, which is not
78
+ in this repo.
79
+ - **Amsterdam:** 0.554 with no labels, on a 6,000-pixel random subset. 0.622
80
+ at best (n=200), on a fixed 2,000-pixel subset with 6 draws. These are not
81
+ full-city numbers like the table above.
82
+
83
+ It does not beat the zero-shot LightGBM below n=200, so it is not one of the
84
+ winners. Details are in `results/llm_protocol_results_colab.md`.
85
+
86
+ ## Quick start
87
+
88
+ ```bash
89
+ pip install numpy pandas pyarrow scikit-learn lightgbm matplotlib
90
+ python winners.py zero-shot # ~15-40 s
91
+ python winners.py retrained --n 25 --rep 0 # ~2-4 min
92
+ ```
93
+
94
+ Expected output, from the verification run on our machine:
95
+
96
+ ```
97
+ zero-shot, all 25,992 Amsterdam pixels: macro F1 0.5928
98
+ retrained, n=25 rep=0: macro F1 0.6303
99
+ ```
100
+
101
+ To re-run the whole protocol: `python winners.py protocol --model retrained
102
+ --reps 5` (25 refits: ~35 min on our machine, up to ~1.5 h on a slower
103
+ one). Or open **`notebooks/winners_demo.ipynb`**, which is
104
+ already executed, for a walk-through of both models with outputs and the
105
+ chart.
106
+
107
+ LightGBM is multi-threaded, so on a different machine the numbers can differ
108
+ in the 4th decimal.
109
+
110
+ ## What we did, exactly
111
+
112
+ ### 1. Features (`notebooks/01_build_features.ipynb`, `src/features.py`)
113
+
114
+ We started from the organisers' raw per-pixel Landsat time series and built
115
+ one row per pixel with 393 usable features:
116
+
117
+ | group | count | what |
118
+ |---|---|---|
119
+ | official | 59 | The organisers' 60 features, reproduced exactly. One has zero variance in a city and was dropped. |
120
+ | reflectance | 276 | Surface reflectance per band; correct spectral indices; tasseled cap; chromaticity; per-decade and early/late-period means |
121
+ | events | 15 | Construction-event detection: when and how sharply the pixel's spectrum stepped |
122
+ | neighbourhood | 15 | The same signals averaged over surrounding pixels |
123
+ | other | 28 | Disturbance spikes and post-construction trends (12), Landsat sensor shares (9), observation coverage (4), position in the city: radial distance and distance to water (3) |
124
+
125
+ Two features with zero variance in one city were dropped, leaving 393 of 395.
126
+ The pixel coordinates and the label are never features. The resulting tables
127
+ are in `data/processed/` (Madrid 76,263 pixels, Amsterdam 25,992). Notebook 1
128
+ needs the raw hackathon data, which is not in this repo; the processed tables
129
+ are.
130
+
131
+ ### 2. Model selection, on Madrid only (`notebooks/03_gbdt_models.ipynb`)
132
+
133
+ The rule we followed was that **Amsterdam labels are used only to draw the
134
+ permitted n-per-class sample and to score**. Every design choice was made on
135
+ Madrid.
136
+
137
+ **Spatial cross-validation.** Madrid was split into 25 geographic blocks, with
138
+ 5-fold cross-validation over whole blocks. Random-pixel folds leak, because
139
+ neighbouring pixels share a construction era. That is why the published 0.618
140
+ Madrid baseline looks better than it transfers.
141
+
142
+ | model | 59 official | 393 all features |
143
+ |---|---|---|
144
+ | LightGBM, class-balanced | 0.556 | **0.570** |
145
+ | LightGBM, unweighted | 0.550 | 0.569 |
146
+ | LightGBM, ordinal | 0.546 | 0.562 |
147
+ | CatBoost, class-balanced | 0.545 | 0.562 * |
148
+
149
+ This comparison used cheap settings (400 rounds, learning rate 0.06) and
150
+ robust-z scaling per block. We were reading the ranking here, not the final
151
+ scores. The normalisation test below then dropped the scaling.
152
+ \* From an earlier run of the same notebook
153
+ (`results/logs/03_gbdt_models_fullscore_run.log`).
154
+
155
+ **Per-city normalisation, decided on Madrid.** Each held-out block was scaled
156
+ with its own statistics, the same way a new city would be.
157
+ - No normalisation: **0.599**.
158
+ - Robust z (median/IQR): 0.569.
159
+
160
+ So the zero-shot model uses raw features. This went against our starting
161
+ intuition, and Amsterdam agreed: robust z scored 0.527 there against 0.593
162
+ with no normalisation.
163
+
164
+ **Decision rule.** Plain argmax maximises accuracy, not macro F1. A simulated
165
+ shift harness makes held-out Madrid blocks look like a new city, by resampling
166
+ the class mix and mapping the features onto Amsterdam's unlabelled
167
+ distribution. In it, the most robust rule was:
168
+
169
+ 1. Divide the probabilities by the Madrid class prior raised to 0.5 (tau-logit
170
+ adjustment).
171
+ 2. Apply the macro-F1 plug-in rule (Ramaswamy et al.).
172
+
173
+ It never lost to argmax in any simulated regime: +0.012 on average. EM-based
174
+ prior estimation was the worst option tested (−0.099) and is not used.
175
+
176
+ ### 3. The two winners
177
+
178
+ **Zero-shot LightGBM** (`winners.train_zero_shot`)
179
+ - 1,200 rounds, learning rate 0.03, 31 leaves, `min_child_samples` 80,
180
+ subsample 0.8, `colsample_bytree` 0.6, L1 0.5, L2 5.0, seed 42.
181
+ - Class-balanced weights, raw features.
182
+ - Prediction is the decision rule above, applied to all of Amsterdam's
183
+ probabilities at once. The rule uses the batch's predicted class shares,
184
+ never its labels.
185
+
186
+ **Retrained LightGBM** (`winners.train_retrained`). This follows the
187
+ organisers' `modelCreate` spec. The deck
188
+ `Building-Age-from-Space-A-Transferable-Classifier.pptx` is in their
189
+ `hackathon.zip` download, and its slide 4 says:
190
+
191
+ > "We are NOT building a classifier. We are building a METHOD that creates
192
+ > classifiers."
193
+
194
+ Slide 5 says that `modelCreate` takes N labelled pixels per class from the new
195
+ city and outputs "a fully functional classifier for that city". There is one
196
+ liberty beyond the tightest reading of slides 5 and 7, which give
197
+ `modelCreate` only the labelled sample: step 1 below also reads Amsterdam's
198
+ **unlabelled** feature table to compute scaling statistics (see Caveats). For
199
+ each draw of n labelled Amsterdam pixels per class:
200
+ 1. Robust-z scale Madrid with Madrid's median/IQR. Scale Amsterdam with
201
+ Amsterdam's median/IQR, computed from its whole unlabelled feature table.
202
+ 2. Stack the 76,263 Madrid rows and the 4n Amsterdam rows. The Amsterdam rows
203
+ get weight 30, then everything is multiplied by class-balanced weights.
204
+ 3. Fit LightGBM with the same parameters as the zero-shot model, but 400 rounds
205
+ at learning rate 0.06. This is the cheaper setting the protocol was run
206
+ with, because it refits on every draw.
207
+ 4. Predict every other Amsterdam pixel and apply the same decision rule.
208
+
209
+ ### 4. The protocol, i.e. how every number was measured
210
+
211
+ - **Labelled sample:** for each n and each repeat r, n pixels per class are
212
+ drawn with `numpy.random.default_rng(42 + 1000·n + r)`. Use
213
+ `winners.shots_for(y_ams, n, r)` to get the identical pixels.
214
+ - **Scoring:** those pixels are removed and macro F1 is computed on every
215
+ remaining Amsterdam pixel.
216
+ - **Repeats:** 5 for the retrained model, since each draw is a full refit. 12
217
+ for the prototype, 20 for the zero-shot and the organisers' baseline.
218
+ - **Exploration subsample:** notebook 3 explored 14 variants on a fixed
219
+ 8,000-pixel subsample.
220
+ - **Full-city re-scoring:** `tools/full_score.py` re-scored the retrained
221
+ LightGBM and the within-class prototype on all of Amsterdam. In the same run
222
+ it re-scored the 8,000-pixel subsample, and that reproduced notebook 3's
223
+ table to the 4th decimal, so the full-city numbers come from identical
224
+ models. The full city reads 0.002–0.006 higher per n, and 0.003–0.004 on the
225
+ average.
226
+ - **The other full-city rows:** the zero-shot and organisers' baseline rows
227
+ come from an earlier full-city run of notebook 3
228
+ (`results/logs/03_gbdt_models_fullscore_run.log`, 20 draws).
229
+
230
+ ### 5. The other 12 Amsterdam variants (8,000-pixel subsample, average over n)
231
+
232
+ | variant | average |
233
+ |---|---|
234
+ | refit leaf values, tree structure frozen † | 0.612 |
235
+ | prototype, total-covariance Mahalanobis | 0.612 |
236
+ | prototype, rank + Euclidean | 0.591 |
237
+ | pure k-NN on the labelled pixels only | 0.589 |
238
+ | zero-shot, no normalisation (the zero-shot winner, before full-city scoring) | 0.586 |
239
+ | organisers' Euclidean prototype (re-run on our 393 features) | 0.575 |
240
+ | zero-shot, per-city rank / robust z / robust z + CORAL | 0.553 / 0.524 / 0.537 |
241
+ | recalibrated LightGBM / CatBoost | 0.544 / 0.550 |
242
+ | prototype, robust z + Euclidean | 0.401 |
243
+
244
+ † Measured with a normalisation bug: it applied robust z when its label says
245
+ none.
246
+
247
+ The models that actually learn from the labelled Amsterdam pixels win. The
248
+ ones that only rescale or recalibrate Madrid's output score below the
249
+ organisers' baseline (0.524–0.553 against 0.575). The exception is the plain
250
+ zero-shot model with no scaling at all.
251
+
252
+ ## Caveats
253
+
254
+ - **Transductive scaling.** The retrained model scales Amsterdam with the
255
+ median/IQR of Amsterdam's whole unlabelled feature table. That uses features
256
+ only, never labels, and the protocol passes the table in. The organisers'
257
+ baseline does **not** do this: it scales Amsterdam with Madrid's scaler. On a
258
+ strict reading, where `modelCreate` gets only the labelled sample, you would
259
+ compute those statistics from the shots alone. On a new test set, compute
260
+ them on that set's features. The zero-shot model is unaffected, because it
261
+ uses no scaling.
262
+ - **Batch-level decision rule.** The rule looks at the predicted class shares
263
+ of the batch it labels. With plain argmax the zero-shot model scores 0.5680
264
+ on all of Amsterdam instead of 0.5928.
265
+ - **Few repeats.** The retrained model's ± comes from 5 draws per n.
266
+ - **Hidden test sets.** The organisers score on held-out Amsterdam and Madrid
267
+ sets we never saw. Expect those numbers to be lower.
268
+
269
+ ## Repo layout
270
+
271
+ ```
272
+ winners.py self-contained: both models, the decision rule, the protocol
273
+ models/
274
+ zero_shot_lgbm.txt the zero-shot model (LightGBM text format)
275
+ retrained_n25_rep0_lgbm.txt example retrained models, draw rep 0
276
+ retrained_n200_rep0_lgbm.txt
277
+ amsterdam_robustz_stats.npz Amsterdam median/IQR the retrained models expect
278
+ retrained_example_shot_indices.json which Amsterdam rows they were trained on
279
+ meta.json feature order, Madrid prior, verification results
280
+ data/processed/ madrid_features.parquet, amsterdam_features.parquet
281
+ notebooks/
282
+ winners_demo.ipynb executed walk-through of both winners (start here)
283
+ 03_gbdt_models.ipynb full model study: CV, normalisation, decision rule, all 14 variants
284
+ 01_build_features.ipynb raw data -> features (needs the hackathon raw data)
285
+ src/ the same notebooks as plain .py
286
+ src/ project code the notebooks import (gbdt.py, evaluation.py, ...)
287
+ tools/full_score.py full-city re-scoring of the two winners
288
+ results/ per-draw JSON, logs of every run, the LLM results
289
+ ```
290
+
291
+ `models/meta.json` → `verification` records the checks run when these files
292
+ were exported:
293
+ - the ported code in `winners.py` gives the same probabilities as the project
294
+ code (max abs diff 0);
295
+ - the saved models reload identically;
296
+ - the two example draws reproduce the reported per-draw scores exactly.
data/processed/amsterdam_features.parquet ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3d5a8953b2560b30fe2453363b07408cd870f293d0bb2d0c93e24803f32e0262
3
+ size 56157092
data/processed/madrid_features.parquet ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0dd77b81c5314bb4380160442edeced09b7e9c3eec94dfd13d720366eba43161
3
+ size 165478500
models/amsterdam_robustz_stats.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:035cfe85473152b23f9bb5076adb78a2f8106b547bb18f8226d84612d990f931
3
+ size 6792
models/meta.json ADDED
@@ -0,0 +1,430 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "feature_names": [
3
+ "Blue_mean",
4
+ "Green_mean",
5
+ "Red_mean",
6
+ "NIR_mean",
7
+ "SWIR1_mean",
8
+ "SWIR2_mean",
9
+ "Blue_std",
10
+ "Green_std",
11
+ "Red_std",
12
+ "NIR_std",
13
+ "SWIR1_std",
14
+ "SWIR2_std",
15
+ "NDVI_mean",
16
+ "NDBI_mean",
17
+ "UI_mean",
18
+ "MNDWI_mean",
19
+ "BSI_mean",
20
+ "NDVI_std",
21
+ "NDBI_std",
22
+ "UI_std",
23
+ "MNDWI_std",
24
+ "BSI_std",
25
+ "Blue_early_mean",
26
+ "Green_early_mean",
27
+ "Red_early_mean",
28
+ "NIR_early_mean",
29
+ "SWIR1_early_mean",
30
+ "SWIR2_early_mean",
31
+ "Blue_early_std",
32
+ "Green_early_std",
33
+ "Red_early_std",
34
+ "NIR_early_std",
35
+ "SWIR1_early_std",
36
+ "SWIR2_early_std",
37
+ "Blue_late_mean",
38
+ "Green_late_mean",
39
+ "Red_late_mean",
40
+ "NIR_late_mean",
41
+ "SWIR1_late_mean",
42
+ "SWIR2_late_mean",
43
+ "Blue_late_std",
44
+ "Green_late_std",
45
+ "Red_late_std",
46
+ "NIR_late_std",
47
+ "SWIR1_late_std",
48
+ "SWIR2_late_std",
49
+ "d_Blue_mean",
50
+ "d_Green_mean",
51
+ "d_Red_mean",
52
+ "d_NIR_mean",
53
+ "d_SWIR1_mean",
54
+ "d_SWIR2_mean",
55
+ "d_Blue_std",
56
+ "d_Green_std",
57
+ "d_Red_std",
58
+ "d_NIR_std",
59
+ "d_SWIR1_std",
60
+ "d_SWIR2_std",
61
+ "has_late_data",
62
+ "sr_brightness_mean",
63
+ "sr_brightness_std",
64
+ "sr_brightness_early_mean",
65
+ "sr_brightness_late_mean",
66
+ "sr_brightness_late_minus_early",
67
+ "sr_brightness_cv",
68
+ "sr_brightness_early_std",
69
+ "sr_brightness_late_std",
70
+ "sr_brightness_slope_decade",
71
+ "sr_brightness_min",
72
+ "sr_brightness_max",
73
+ "sr_brightness_tm_mean",
74
+ "sr_brightness_oli_mean",
75
+ "sr_brightness_oli_minus_tm",
76
+ "sr_brightness_dec84_93_mean",
77
+ "sr_brightness_dec94_03_mean",
78
+ "sr_brightness_dec04_13_mean",
79
+ "sr_brightness_dec14_24_mean",
80
+ "sr_Blue_mean",
81
+ "sr_Blue_std",
82
+ "sr_Blue_early_mean",
83
+ "sr_Blue_late_mean",
84
+ "sr_Blue_late_minus_early",
85
+ "sr_Blue_cv",
86
+ "sr_Blue_early_std",
87
+ "sr_Blue_late_std",
88
+ "sr_Blue_slope_decade",
89
+ "sr_Blue_min",
90
+ "sr_Blue_max",
91
+ "sr_Blue_tm_mean",
92
+ "sr_Blue_oli_mean",
93
+ "sr_Blue_oli_minus_tm",
94
+ "sr_Blue_dec84_93_mean",
95
+ "sr_Blue_dec94_03_mean",
96
+ "sr_Blue_dec04_13_mean",
97
+ "sr_Blue_dec14_24_mean",
98
+ "sr_Green_mean",
99
+ "sr_Green_std",
100
+ "sr_Green_early_mean",
101
+ "sr_Green_late_mean",
102
+ "sr_Green_late_minus_early",
103
+ "sr_Green_cv",
104
+ "sr_Green_early_std",
105
+ "sr_Green_late_std",
106
+ "sr_Green_slope_decade",
107
+ "sr_Green_min",
108
+ "sr_Green_max",
109
+ "sr_Green_tm_mean",
110
+ "sr_Green_oli_mean",
111
+ "sr_Green_oli_minus_tm",
112
+ "sr_Green_dec84_93_mean",
113
+ "sr_Green_dec94_03_mean",
114
+ "sr_Green_dec04_13_mean",
115
+ "sr_Green_dec14_24_mean",
116
+ "sr_Red_mean",
117
+ "sr_Red_std",
118
+ "sr_Red_early_mean",
119
+ "sr_Red_late_mean",
120
+ "sr_Red_late_minus_early",
121
+ "sr_Red_cv",
122
+ "sr_Red_early_std",
123
+ "sr_Red_late_std",
124
+ "sr_Red_slope_decade",
125
+ "sr_Red_min",
126
+ "sr_Red_max",
127
+ "sr_Red_tm_mean",
128
+ "sr_Red_oli_mean",
129
+ "sr_Red_oli_minus_tm",
130
+ "sr_Red_dec84_93_mean",
131
+ "sr_Red_dec94_03_mean",
132
+ "sr_Red_dec04_13_mean",
133
+ "sr_Red_dec14_24_mean",
134
+ "sr_NIR_mean",
135
+ "sr_NIR_std",
136
+ "sr_NIR_early_mean",
137
+ "sr_NIR_late_mean",
138
+ "sr_NIR_late_minus_early",
139
+ "sr_NIR_cv",
140
+ "sr_NIR_early_std",
141
+ "sr_NIR_late_std",
142
+ "sr_NIR_slope_decade",
143
+ "sr_NIR_min",
144
+ "sr_NIR_max",
145
+ "sr_NIR_tm_mean",
146
+ "sr_NIR_oli_mean",
147
+ "sr_NIR_oli_minus_tm",
148
+ "sr_NIR_dec84_93_mean",
149
+ "sr_NIR_dec94_03_mean",
150
+ "sr_NIR_dec04_13_mean",
151
+ "sr_NIR_dec14_24_mean",
152
+ "sr_SWIR1_mean",
153
+ "sr_SWIR1_std",
154
+ "sr_SWIR1_early_mean",
155
+ "sr_SWIR1_late_mean",
156
+ "sr_SWIR1_late_minus_early",
157
+ "sr_SWIR1_cv",
158
+ "sr_SWIR1_early_std",
159
+ "sr_SWIR1_late_std",
160
+ "sr_SWIR1_slope_decade",
161
+ "sr_SWIR1_min",
162
+ "sr_SWIR1_max",
163
+ "sr_SWIR1_tm_mean",
164
+ "sr_SWIR1_oli_mean",
165
+ "sr_SWIR1_oli_minus_tm",
166
+ "sr_SWIR1_dec84_93_mean",
167
+ "sr_SWIR1_dec94_03_mean",
168
+ "sr_SWIR1_dec04_13_mean",
169
+ "sr_SWIR1_dec14_24_mean",
170
+ "sr_SWIR2_mean",
171
+ "sr_SWIR2_std",
172
+ "sr_SWIR2_early_mean",
173
+ "sr_SWIR2_late_mean",
174
+ "sr_SWIR2_late_minus_early",
175
+ "sr_SWIR2_cv",
176
+ "sr_SWIR2_early_std",
177
+ "sr_SWIR2_late_std",
178
+ "sr_SWIR2_slope_decade",
179
+ "sr_SWIR2_min",
180
+ "sr_SWIR2_max",
181
+ "sr_SWIR2_tm_mean",
182
+ "sr_SWIR2_oli_mean",
183
+ "sr_SWIR2_oli_minus_tm",
184
+ "sr_SWIR2_dec84_93_mean",
185
+ "sr_SWIR2_dec94_03_mean",
186
+ "sr_SWIR2_dec04_13_mean",
187
+ "sr_SWIR2_dec14_24_mean",
188
+ "sr_NDVI_mean",
189
+ "sr_NDVI_std",
190
+ "sr_NDVI_early_mean",
191
+ "sr_NDVI_late_mean",
192
+ "sr_NDVI_late_minus_early",
193
+ "sr_NDVI_cv",
194
+ "sr_NDVI_early_std",
195
+ "sr_NDVI_late_std",
196
+ "sr_NDVI_slope_decade",
197
+ "sr_NDVI_min",
198
+ "sr_NDVI_max",
199
+ "sr_NDVI_tm_mean",
200
+ "sr_NDVI_oli_mean",
201
+ "sr_NDVI_oli_minus_tm",
202
+ "sr_NDVI_dec84_93_mean",
203
+ "sr_NDVI_dec94_03_mean",
204
+ "sr_NDVI_dec04_13_mean",
205
+ "sr_NDVI_dec14_24_mean",
206
+ "sr_NDBI_mean",
207
+ "sr_NDBI_std",
208
+ "sr_NDBI_early_mean",
209
+ "sr_NDBI_late_mean",
210
+ "sr_NDBI_late_minus_early",
211
+ "sr_NDBI_cv",
212
+ "sr_NDBI_early_std",
213
+ "sr_NDBI_late_std",
214
+ "sr_NDBI_slope_decade",
215
+ "sr_NDBI_min",
216
+ "sr_NDBI_max",
217
+ "sr_NDBI_tm_mean",
218
+ "sr_NDBI_oli_mean",
219
+ "sr_NDBI_oli_minus_tm",
220
+ "sr_NDBI_dec84_93_mean",
221
+ "sr_NDBI_dec94_03_mean",
222
+ "sr_NDBI_dec04_13_mean",
223
+ "sr_NDBI_dec14_24_mean",
224
+ "sr_UI_mean",
225
+ "sr_UI_std",
226
+ "sr_UI_early_mean",
227
+ "sr_UI_late_mean",
228
+ "sr_UI_late_minus_early",
229
+ "sr_UI_cv",
230
+ "sr_UI_early_std",
231
+ "sr_UI_late_std",
232
+ "sr_UI_slope_decade",
233
+ "sr_UI_min",
234
+ "sr_UI_max",
235
+ "sr_UI_tm_mean",
236
+ "sr_UI_oli_mean",
237
+ "sr_UI_oli_minus_tm",
238
+ "sr_UI_dec84_93_mean",
239
+ "sr_UI_dec94_03_mean",
240
+ "sr_UI_dec04_13_mean",
241
+ "sr_UI_dec14_24_mean",
242
+ "sr_MNDWI_mean",
243
+ "sr_MNDWI_std",
244
+ "sr_MNDWI_early_mean",
245
+ "sr_MNDWI_late_mean",
246
+ "sr_MNDWI_late_minus_early",
247
+ "sr_MNDWI_cv",
248
+ "sr_MNDWI_early_std",
249
+ "sr_MNDWI_late_std",
250
+ "sr_MNDWI_slope_decade",
251
+ "sr_MNDWI_min",
252
+ "sr_MNDWI_max",
253
+ "sr_MNDWI_tm_mean",
254
+ "sr_MNDWI_oli_mean",
255
+ "sr_MNDWI_oli_minus_tm",
256
+ "sr_MNDWI_dec84_93_mean",
257
+ "sr_MNDWI_dec94_03_mean",
258
+ "sr_MNDWI_dec04_13_mean",
259
+ "sr_MNDWI_dec14_24_mean",
260
+ "sr_BSI_mean",
261
+ "sr_BSI_std",
262
+ "sr_BSI_early_mean",
263
+ "sr_BSI_late_mean",
264
+ "sr_BSI_late_minus_early",
265
+ "sr_BSI_cv",
266
+ "sr_BSI_early_std",
267
+ "sr_BSI_late_std",
268
+ "sr_BSI_slope_decade",
269
+ "sr_BSI_min",
270
+ "sr_BSI_max",
271
+ "sr_BSI_tm_mean",
272
+ "sr_BSI_oli_mean",
273
+ "sr_BSI_oli_minus_tm",
274
+ "sr_BSI_dec84_93_mean",
275
+ "sr_BSI_dec94_03_mean",
276
+ "sr_BSI_dec04_13_mean",
277
+ "sr_BSI_dec14_24_mean",
278
+ "sr_NBR_mean",
279
+ "sr_NBR_std",
280
+ "sr_NBR_early_mean",
281
+ "sr_NBR_late_mean",
282
+ "sr_NBR_late_minus_early",
283
+ "sr_NBR_cv",
284
+ "sr_TCB_mean",
285
+ "sr_TCB_std",
286
+ "sr_TCB_early_mean",
287
+ "sr_TCB_late_mean",
288
+ "sr_TCB_late_minus_early",
289
+ "sr_TCB_cv",
290
+ "sr_TCG_mean",
291
+ "sr_TCG_std",
292
+ "sr_TCG_early_mean",
293
+ "sr_TCG_late_mean",
294
+ "sr_TCG_late_minus_early",
295
+ "sr_TCG_cv",
296
+ "sr_TCW_mean",
297
+ "sr_TCW_std",
298
+ "sr_TCW_early_mean",
299
+ "sr_TCW_late_mean",
300
+ "sr_TCW_late_minus_early",
301
+ "sr_TCW_cv",
302
+ "sr_chromBlue_mean",
303
+ "sr_chromBlue_std",
304
+ "sr_chromBlue_early_mean",
305
+ "sr_chromBlue_late_mean",
306
+ "sr_chromBlue_late_minus_early",
307
+ "sr_chromBlue_cv",
308
+ "sr_chromGreen_mean",
309
+ "sr_chromGreen_std",
310
+ "sr_chromGreen_early_mean",
311
+ "sr_chromGreen_late_mean",
312
+ "sr_chromGreen_late_minus_early",
313
+ "sr_chromGreen_cv",
314
+ "sr_chromRed_mean",
315
+ "sr_chromRed_std",
316
+ "sr_chromRed_early_mean",
317
+ "sr_chromRed_late_mean",
318
+ "sr_chromRed_late_minus_early",
319
+ "sr_chromRed_cv",
320
+ "sr_chromNIR_mean",
321
+ "sr_chromNIR_std",
322
+ "sr_chromNIR_early_mean",
323
+ "sr_chromNIR_late_mean",
324
+ "sr_chromNIR_late_minus_early",
325
+ "sr_chromNIR_cv",
326
+ "sr_chromSWIR1_mean",
327
+ "sr_chromSWIR1_std",
328
+ "sr_chromSWIR1_early_mean",
329
+ "sr_chromSWIR1_late_mean",
330
+ "sr_chromSWIR1_late_minus_early",
331
+ "sr_chromSWIR1_cv",
332
+ "sr_chromSWIR2_mean",
333
+ "sr_chromSWIR2_std",
334
+ "sr_chromSWIR2_early_mean",
335
+ "sr_chromSWIR2_late_mean",
336
+ "sr_chromSWIR2_late_minus_early",
337
+ "sr_chromSWIR2_cv",
338
+ "step_brightness_year",
339
+ "step_brightness_mag",
340
+ "step_brightness_njumps",
341
+ "step_brightness_t",
342
+ "step_brightness_snr",
343
+ "step_SWIR1_year",
344
+ "step_SWIR1_mag",
345
+ "step_SWIR1_njumps",
346
+ "step_SWIR1_t",
347
+ "step_SWIR1_snr",
348
+ "step_NDVI_year",
349
+ "step_NDVI_mag",
350
+ "step_NDVI_njumps",
351
+ "step_NDVI_t",
352
+ "step_NDVI_snr",
353
+ "spike_BSI_year",
354
+ "spike_BSI_z",
355
+ "spike_BSI_returns",
356
+ "spike_SWIR1_year",
357
+ "spike_SWIR1_z",
358
+ "spike_SWIR1_returns",
359
+ "spike_brightness_year",
360
+ "spike_brightness_z",
361
+ "spike_brightness_returns",
362
+ "post_step_NDVI_slope_decade",
363
+ "years_since_step",
364
+ "pre_state",
365
+ "sat_share_tm",
366
+ "sat_share_tm_early",
367
+ "sat_share_tm_late",
368
+ "sat_share_etm",
369
+ "sat_share_etm_early",
370
+ "sat_share_etm_late",
371
+ "sat_share_oli",
372
+ "sat_share_oli_late",
373
+ "sat_switches",
374
+ "n_observed_years",
375
+ "first_observed_year",
376
+ "last_observed_year",
377
+ "coverage",
378
+ "radial_px",
379
+ "radial_frac",
380
+ "dist_to_water_px",
381
+ "nbr_sr_brightness_mean",
382
+ "nbr_sr_brightness_mean_diff",
383
+ "nbr_sr_NDVI_mean",
384
+ "nbr_sr_NDVI_mean_diff",
385
+ "nbr_sr_SWIR1_mean",
386
+ "nbr_sr_SWIR1_mean_diff",
387
+ "nbr_sr_brightness_late_minus_early",
388
+ "nbr_sr_brightness_late_minus_early_diff",
389
+ "nbr_step_brightness_year",
390
+ "nbr_step_brightness_year_diff",
391
+ "nbr_step_brightness_mag",
392
+ "nbr_step_brightness_mag_diff",
393
+ "nbr_sr_brightness_std",
394
+ "nbr_sr_brightness_std_diff",
395
+ "nbr_n_within_radius"
396
+ ],
397
+ "dropped_dead_features": [
398
+ "has_early_data",
399
+ "sat_share_oli_early"
400
+ ],
401
+ "classes": [
402
+ 1,
403
+ 2,
404
+ 3,
405
+ 4
406
+ ],
407
+ "seed": 42,
408
+ "madrid_prior": [
409
+ 0.18087408048463868,
410
+ 0.35603110289393286,
411
+ 0.244836945832186,
412
+ 0.2182578707892425
413
+ ],
414
+ "verification": {
415
+ "zero_shot_port_max_abs_diff": 0.0,
416
+ "zero_shot_reload_max_abs_diff": 0.0,
417
+ "decision_rule_port_identical": true,
418
+ "zero_shot_all_pixels": 0.592839846672412,
419
+ "zero_shot_n5_rep0": 0.592823816444138,
420
+ "zero_shot_all_pixels_plain_argmax": 0.5679842811565695,
421
+ "retrained_n25_port_max_abs_diff": 0.0,
422
+ "retrained_n25_reload_max_abs_diff": 0.0,
423
+ "retrained_n25_rep0": 0.6302737351779462,
424
+ "retrained_n25_rep0_reported": 0.6302737351779462,
425
+ "retrained_n200_port_max_abs_diff": 0.0,
426
+ "retrained_n200_reload_max_abs_diff": 0.0,
427
+ "retrained_n200_rep0": 0.7264520284114167,
428
+ "retrained_n200_rep0_reported": 0.7264520284114167
429
+ }
430
+ }
models/retrained_example_shot_indices.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"note": "Row indices into amsterdam_features.parquet of the labelled pixels each example model was trained on (draw rep 0).", "n25_rep0": [18209, 14793, 13081, 16589, 12889, 8381, 24399, 17181, 8471, 8515, 21652, 13638, 10185, 20225, 16664, 9001, 12752, 21310, 15707, 16958, 18041, 10554, 10847, 17240, 10859, 6669, 19686, 5601, 5623, 24595, 6035, 20979, 18754, 10980, 24335, 8212, 19981, 7277, 19086, 8830, 5902, 23524, 22079, 4517, 24030, 2259, 18522, 23554, 20420, 22163, 3346, 2215, 24890, 25644, 12506, 1931, 21353, 19457, 5595, 16020, 2685, 1390, 23123, 20915, 1603, 16994, 25794, 14681, 4879, 22097, 1574, 4700, 19263, 380, 20591, 15820, 25444, 20409, 2552, 19403, 2896, 3170, 6392, 15664, 6626, 22649, 8717, 15469, 25184, 25752, 1609, 203, 11321, 8911, 106, 851, 3206, 2821, 11818, 2823], "n200_rep0": [20654, 20967, 11154, 15646, 15977, 13403, 17039, 16593, 12370, 17313, 17984, 12118, 21891, 13184, 16304, 8086, 8067, 11131, 11704, 9913, 11579, 16044, 10758, 11957, 11988, 11464, 8811, 15911, 3389, 8363, 11027, 13754, 14215, 12225, 15645, 17666, 9020, 13148, 19873, 9224, 8676, 20079, 20758, 13620, 19097, 16442, 13763, 11616, 7858, 12522, 8171, 16562, 8796, 19447, 16275, 9336, 16260, 19687, 17233, 11175, 11444, 13228, 13577, 12539, 16172, 3300, 8823, 20700, 13990, 15500, 10045, 12519, 10335, 15305, 17747, 17282, 20831, 15850, 20167, 10270, 14653, 13760, 25756, 13152, 11256, 13076, 18640, 15876, 9448, 22074, 8113, 10362, 13113, 12192, 22773, 17441, 8740, 8426, 10553, 17135, 11016, 2352, 9426, 9093, 17847, 9596, 16776, 11069, 20815, 14459, 8364, 14606, 21497, 11426, 15315, 8872, 15400, 18130, 14526, 7266, 10382, 9350, 11534, 16035, 12045, 15504, 9543, 9187, 8897, 16242, 8068, 16796, 8070, 16130, 10907, 9819, 16966, 20706, 10621, 12196, 7634, 18552, 14149, 12149, 15863, 12062, 19763, 19126, 7983, 7981, 13770, 8304, 13334, 20785, 19342, 9532, 11784, 13396, 11173, 21238, 10778, 10859, 8423, 15870, 9605, 9871, 13623, 21631, 11537, 13144, 18301, 11989, 8669, 16602, 10189, 19069, 18700, 14953, 8675, 9274, 9973, 3998, 11440, 21736, 20582, 9610, 25596, 17099, 20500, 14723, 17363, 12133, 9225, 16732, 17094, 10606, 12633, 16102, 23531, 16126, 24788, 1207, 2411, 5724, 15590, 15291, 1011, 12725, 5769, 5405, 5192, 4450, 23979, 22540, 6712, 3995, 6061, 18941, 19122, 7081, 20440, 2239, 20097, 6928, 23056, 14025, 18895, 7117, 20115, 24627, 12551, 19265, 1583, 25014, 25893, 1749, 2634, 24388, 8156, 12419, 4877, 13434, 18125, 21958, 14268, 23363, 13544, 19470, 17795, 11412, 7946, 24246, 24809, 25850, 6244, 18527, 14263, 15900, 9397, 17058, 641, 1912, 20017, 10742, 11224, 13890, 13666, 20246, 23776, 24969, 18383, 5358, 15616, 21477, 17722, 20897, 24241, 16318, 5302, 5493, 15601, 7038, 5509, 21162, 2424, 25020, 18307, 19187, 12413, 18016, 1595, 12847, 12346, 11930, 5456, 18717, 6275, 5744, 2747, 18493, 1453, 18136, 2528, 23318, 18005, 24122, 24344, 23171, 18173, 2159, 23934, 2990, 18542, 23788, 20003, 24083, 15318, 20751, 25108, 6101, 10597, 15347, 6136, 18075, 20980, 5310, 7958, 13918, 23700, 11063, 25212, 21247, 5891, 9627, 7265, 23576, 24205, 11272, 16980, 6797, 22920, 6781, 13832, 20890, 10815, 23258, 24375, 13791, 10138, 10439, 13484, 1208, 12324, 11008, 22322, 8469, 22967, 23655, 23618, 3947, 22860, 6617, 21457, 12906, 2886, 2234, 18842, 6373, 5754, 3865, 15427, 19464, 23600, 21779, 20649, 13903, 22966, 12169, 5263, 15665, 1598, 21545, 23996, 9106, 23443, 5127, 4954, 4045, 23981, 3198, 18573, 23720, 6594, 5408, 5586, 24736, 5402, 24064, 20601, 5231, 1961, 22298, 2451, 1115, 22553, 5090, 24684, 25206, 7407, 7037, 3440, 1644, 12508, 833, 25302, 4499, 7834, 20431, 6992, 22832, 9177, 16153, 18968, 21896, 17037, 10339, 17449, 3966, 2008, 7413, 957, 2379, 20993, 376, 6428, 537, 2224, 8858, 3128, 25257, 3286, 5941, 5470, 23307, 4928, 6228, 25279, 14232, 12076, 553, 3770, 2146, 22589, 9315, 6058, 16757, 11387, 4751, 25385, 22437, 14439, 14926, 8229, 13204, 22375, 24149, 5545, 22142, 7104, 444, 25454, 2592, 1180, 23159, 24671, 16847, 1611, 10541, 12082, 3197, 6890, 18805, 3559, 24115, 852, 3716, 17893, 1503, 8349, 3089, 6640, 2230, 22596, 6408, 1842, 21486, 19990, 19159, 23764, 1023, 16626, 3548, 8552, 11376, 16767, 5800, 1835, 23148, 12921, 25008, 20029, 450, 18421, 17057, 354, 4168, 1432, 24898, 13253, 6779, 20028, 3914, 14636, 3289, 21334, 9969, 4371, 8083, 18980, 23420, 22574, 5596, 23965, 6316, 3343, 13530, 1449, 274, 1296, 4179, 5570, 4551, 7962, 25076, 18733, 23563, 15449, 4005, 1374, 19663, 19942, 22548, 6639, 24589, 2012, 23377, 7225, 3963, 7033, 14702, 4661, 419, 12577, 24437, 1120, 1247, 395, 19727, 15127, 13847, 12858, 22361, 22690, 1058, 15234, 1744, 12965, 19730, 15139, 960, 490, 3809, 2742, 1361, 2269, 24556, 20973, 2192, 24402, 954, 3898, 24590, 3451, 18692, 22414, 4395, 19605, 1091, 20620, 20012, 1567, 629, 22536, 11902, 3313, 9923, 660, 898, 14584, 11584, 2983, 21780, 10727, 25618, 24391, 23543, 25616, 11400, 11662, 156, 10219, 19787, 12487, 17250, 19753, 1697, 25687, 7156, 43, 10721, 15649, 22923, 651, 24544, 6500, 25253, 15584, 3169, 25532, 15605, 65, 17682, 25382, 630, 24382, 6798, 23007, 4145, 1548, 2831, 22231, 9801, 4584, 14985, 23076, 24485, 10649, 23674, 195, 25309, 2466, 2407, 22121, 801, 24742, 19594, 25503, 3252, 24907, 24364, 2898, 20349, 23714, 23270, 208, 935, 25667, 15946, 22, 25119, 17423, 7186, 3532, 22649, 6576, 17112, 11656, 25524, 23198, 160, 2252, 3231, 22389, 10, 24361, 209, 15652, 15738, 1504, 22529, 920, 24506, 24717, 22388, 6876, 15736, 1366, 116, 25505, 18364, 589, 2895, 18392, 18529, 187, 2335, 17863, 23187, 25449, 10895, 47, 2333, 16473, 2584, 6542, 1256, 23312, 6598, 2900, 1383, 19883, 1650, 11495, 6184, 22946, 3971, 25273, 23176, 16313, 950, 2667, 23993, 25635, 2, 19934, 25556, 11128, 148, 17457, 13345, 7441, 1892, 22199, 25979, 25627, 10000, 17555, 12246, 23036, 25397, 22470, 6586, 7233, 3016, 15467, 1204, 94, 7323, 25713, 3303, 3321, 15819, 6610, 3105, 2304, 19845, 2934, 130, 140, 6455, 169, 25363, 2127, 8712, 24833, 18926, 22959, 1307, 21381, 2695, 1151, 18766, 63, 25159, 2313, 25247, 9626, 16979, 2699, 9641]}
models/retrained_n200_rep0_lgbm.txt ADDED
The diff for this file is too large to render. See raw diff
 
models/retrained_n25_rep0_lgbm.txt ADDED
The diff for this file is too large to render. See raw diff
 
models/zero_shot_lgbm.txt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:02f51d07177c68c0c8fbb6b5753195737b664498d2b875138188749d634374c6
3
+ size 17080833
notebooks/01_build_features.ipynb ADDED
@@ -0,0 +1,437 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "metadata": {},
6
+ "source": [
7
+ "# 1 - Features: from 40 years of Landsat to one row per building pixel\n",
8
+ "\n",
9
+ "This notebook turns the two raw parquet files into two feature tables.\n",
10
+ "\n",
11
+ "It does **two** things at once:\n",
12
+ "\n",
13
+ "1. **Reproduces the organisers' 60 features exactly** (Notebook 3), so every\n",
14
+ " number we report stays comparable with the published Random-Forest\n",
15
+ " baseline of 0.618 +- 0.004 macro F1 on Madrid.\n",
16
+ "2. **Adds an extended set** on top: proper surface reflectance instead of raw\n",
17
+ " digital numbers, correct spectral indices, construction-event detection,\n",
18
+ " per-decade means, trends, and label-free neighbourhood context.\n",
19
+ "\n",
20
+ "Nothing here touches `y_amsterdam`. Amsterdam's *features* are used (for\n",
21
+ "per-city statistics), its *labels* are not.\n",
22
+ "\n",
23
+ "Runs on CPU in a few minutes - no GPU needed. Do this once, locally, and the\n",
24
+ "GPU notebooks just load the parquet files it writes."
25
+ ]
26
+ },
27
+ {
28
+ "cell_type": "code",
29
+ "metadata": {},
30
+ "source": [
31
+ "import sys\n",
32
+ "from pathlib import Path\n",
33
+ "\n",
34
+ "# Make src/ importable whether we are local, on Kaggle, or on Colab.\n",
35
+ "for cand in [Path.cwd() / \"src\", Path.cwd().parent / \"src\",\n",
36
+ " Path(\"/kaggle/working/src\"), Path(\"/content/hack4dev/src\")]:\n",
37
+ " if cand.exists():\n",
38
+ " sys.path.insert(0, str(cand))\n",
39
+ " break\n",
40
+ "\n",
41
+ "import numpy as np\n",
42
+ "import pandas as pd\n",
43
+ "import matplotlib.pyplot as plt\n",
44
+ "\n",
45
+ "import config as C\n",
46
+ "import features as F\n",
47
+ "import retrieval as R\n",
48
+ "\n",
49
+ "pd.set_option(\"display.width\", 200)\n",
50
+ "pd.set_option(\"display.max_columns\", 60)\n",
51
+ "print(\"platform:\", C.PLATFORM)\n",
52
+ "print(\"looking for raw parquet under:\", C.RAW_DIR)"
53
+ ],
54
+ "execution_count": null,
55
+ "outputs": []
56
+ },
57
+ {
58
+ "cell_type": "markdown",
59
+ "metadata": {},
60
+ "source": [
61
+ "## 1.1 Find the raw files\n",
62
+ "\n",
63
+ "Expected: `madrid_train.parquet` (about 3.17 M rows) and\n",
64
+ "`amsterdam_data.parquet` (about 1.06 M rows). On Kaggle they will be inside\n",
65
+ "`/kaggle/input/<dataset-name>/`; the search below handles that."
66
+ ]
67
+ },
68
+ {
69
+ "cell_type": "code",
70
+ "metadata": {},
71
+ "source": [
72
+ "def find_parquet(pattern: str) -> Path:\n",
73
+ " roots = [C.RAW_DIR, C.DATA_DIR, C.PROJECT_ROOT, Path(\"/kaggle/input\")]\n",
74
+ " for root in roots:\n",
75
+ " if not root.exists():\n",
76
+ " continue\n",
77
+ " hits = sorted(root.rglob(pattern))\n",
78
+ " if hits:\n",
79
+ " return hits[0]\n",
80
+ " raise FileNotFoundError(\n",
81
+ " f\"{pattern} not found. Put the two parquet files in {C.RAW_DIR} \"\n",
82
+ " \"(local) or attach them as a Kaggle dataset.\")\n",
83
+ "\n",
84
+ "MADRID_PARQUET = find_parquet(\"madrid_train.parquet\")\n",
85
+ "AMS_PARQUET = find_parquet(\"amsterdam_data.parquet\")\n",
86
+ "print(\"Madrid :\", MADRID_PARQUET)\n",
87
+ "print(\"Amsterdam:\", AMS_PARQUET)"
88
+ ],
89
+ "execution_count": null,
90
+ "outputs": []
91
+ },
92
+ {
93
+ "cell_type": "markdown",
94
+ "metadata": {},
95
+ "source": [
96
+ "## 1.2 What is actually in the raw file\n",
97
+ "\n",
98
+ "One row per *(pixel, year)*. Up to three summer observations per year are\n",
99
+ "stored side by side as `Blue_1/Blue_2/Blue_3`, each with a `qa_valid_N`\n",
100
+ "cloud flag. Band values are raw Landsat Collection-2 integers:\n",
101
+ "\n",
102
+ "$$\\text{surface reflectance} = \\text{DN} \\times 2.75\\times10^{-5} - 0.2$$"
103
+ ]
104
+ },
105
+ {
106
+ "cell_type": "code",
107
+ "metadata": {},
108
+ "source": [
109
+ "peek = pd.read_parquet(MADRID_PARQUET)\n",
110
+ "print(f\"{len(peek):,} rows x {peek.shape[1]} columns\")\n",
111
+ "print(\"\\ncolumns:\")\n",
112
+ "print(list(peek.columns))\n",
113
+ "print(\"\\nyears:\", peek[\"year\"].min(), \"-\", peek[\"year\"].max(),\n",
114
+ " f\"({peek['year'].nunique()} distinct)\")\n",
115
+ "print(\"unique pixels:\", f\"{peek.groupby(C.PID).ngroups:,}\")\n",
116
+ "print(\"\\nfirst row:\")\n",
117
+ "print(peek.iloc[0].to_string())"
118
+ ],
119
+ "execution_count": null,
120
+ "outputs": []
121
+ },
122
+ {
123
+ "cell_type": "code",
124
+ "metadata": {},
125
+ "source": [
126
+ "# Reflectance sanity check on one band, and the QA-flag hit rate.\n",
127
+ "dn = peek[\"Red_1\"].dropna()\n",
128
+ "sr = dn * C.SR_SCALE + C.SR_OFFSET\n",
129
+ "print(\"Red_1 digital numbers :\", dn.quantile([0, .01, .5, .99, 1]).round(0).to_dict())\n",
130
+ "print(\"Red_1 as reflectance :\", sr.quantile([0, .01, .5, .99, 1]).round(3).to_dict())\n",
131
+ "for s in (1, 2, 3):\n",
132
+ " col = f\"qa_valid_{s}\"\n",
133
+ " if col in peek.columns:\n",
134
+ " print(f\"{col}: {peek[col].mean():.1%} valid\")\n",
135
+ "del peek"
136
+ ],
137
+ "execution_count": null,
138
+ "outputs": []
139
+ },
140
+ {
141
+ "cell_type": "markdown",
142
+ "metadata": {},
143
+ "source": [
144
+ "## 1.3 Build both feature sets\n",
145
+ "\n",
146
+ "The heavy lifting lives in `src/features.py`. Per city it:\n",
147
+ "\n",
148
+ "* picks the first QA-valid observation per band per year, drops haze\n",
149
+ " (`Blue > 15000`);\n",
150
+ "* records `has_early_data` / `has_late_data` on the *observed* years;\n",
151
+ "* fills the annual series (linear inside, nearest at the edges);\n",
152
+ "* pivots to one `(pixels x years)` matrix per band, then computes every\n",
153
+ " statistic with numpy - a whole city in about a minute."
154
+ ]
155
+ },
156
+ {
157
+ "cell_type": "code",
158
+ "metadata": {},
159
+ "source": [
160
+ "madrid = F.build_city(MADRID_PARQUET, city_hint=\"Madrid\")"
161
+ ],
162
+ "execution_count": null,
163
+ "outputs": []
164
+ },
165
+ {
166
+ "cell_type": "code",
167
+ "metadata": {},
168
+ "source": [
169
+ "amsterdam = F.build_city(AMS_PARQUET, city_hint=\"Amsterdam\")"
170
+ ],
171
+ "execution_count": null,
172
+ "outputs": []
173
+ },
174
+ {
175
+ "cell_type": "markdown",
176
+ "metadata": {},
177
+ "source": [
178
+ "## 1.4 Does this match the organisers' own output?\n",
179
+ "\n",
180
+ "If `preprocessed_data.pkl` from the organisers' Notebook 3 is around, compare\n",
181
+ "the 60 official features against it. Their arrays are standardised with\n",
182
+ "Madrid's `StandardScaler`, so we compare **correlation per feature**, which is\n",
183
+ "invariant to that. Anything below 0.999 means our pipeline has drifted."
184
+ ]
185
+ },
186
+ {
187
+ "cell_type": "code",
188
+ "metadata": {},
189
+ "source": [
190
+ "pkl = None\n",
191
+ "for root in [C.DATA_DIR, C.PROJECT_ROOT, Path(\"/kaggle/input\")]:\n",
192
+ " if root.exists():\n",
193
+ " hits = sorted(root.rglob(\"preprocessed_data.pkl\"))\n",
194
+ " if hits:\n",
195
+ " pkl = hits[0]\n",
196
+ " break\n",
197
+ "\n",
198
+ "if pkl is None:\n",
199
+ " print(\"preprocessed_data.pkl not found - skipping the cross-check.\")\n",
200
+ "else:\n",
201
+ " import pickle\n",
202
+ " with open(pkl, \"rb\") as fh:\n",
203
+ " ref = pickle.load(fh)\n",
204
+ " print(\"reference file:\", pkl)\n",
205
+ " ours = madrid.set_index([\"px_key\", \"py_key\"])\n",
206
+ " ref_ids = pd.MultiIndex.from_arrays(\n",
207
+ " [ref[\"pixel_ids_madrid\"][:, 0], ref[\"pixel_ids_madrid\"][:, 1]],\n",
208
+ " names=[\"px_key\", \"py_key\"])\n",
209
+ " common = ref_ids.intersection(ours.index)\n",
210
+ " print(f\"pixels: ours {len(ours):,}, theirs {len(ref_ids):,}, \"\n",
211
+ " f\"in common {len(common):,}\")\n",
212
+ " ref_df = pd.DataFrame(ref[\"X_madrid\"], index=ref_ids,\n",
213
+ " columns=ref[\"feature_names\"]).loc[common]\n",
214
+ " our_df = ours.loc[common, ref[\"feature_names\"]]\n",
215
+ " corr = {c: float(np.corrcoef(ref_df[c], our_df[c])[0, 1])\n",
216
+ " for c in ref[\"feature_names\"]\n",
217
+ " if ref_df[c].std() > 0 and our_df[c].std() > 0}\n",
218
+ " bad = {k: v for k, v in corr.items() if v < 0.999}\n",
219
+ " print(f\"features compared: {len(corr)} | min correlation: \"\n",
220
+ " f\"{min(corr.values()):.5f}\")\n",
221
+ " print(\"dead features in their file (zero variance):\",\n",
222
+ " [c for c in ref[\"feature_names\"] if ref_df[c].std() == 0])\n",
223
+ " print(\"features below 0.999:\", bad if bad else \"none - pipeline matches\")\n",
224
+ " # Labels must agree exactly.\n",
225
+ " y_ref = pd.Series(ref[\"y_madrid\"], index=ref_ids).loc[common]\n",
226
+ " print(\"label agreement:\",\n",
227
+ " f\"{(y_ref.to_numpy() == our_df.index.map(ours['age_class']).to_numpy()).mean():.4%}\")"
228
+ ],
229
+ "execution_count": null,
230
+ "outputs": []
231
+ },
232
+ {
233
+ "cell_type": "markdown",
234
+ "metadata": {},
235
+ "source": [
236
+ "## 1.5 Class balance, and the domain gap\n",
237
+ "\n",
238
+ "Two facts drive every later design decision:\n",
239
+ "\n",
240
+ "* the classes are unbalanced, differently in each city;\n",
241
+ "* the same feature has a different *level* in each city, by much more than\n",
242
+ " the gap between classes - which is why a Madrid model collapses to ~0.34 on\n",
243
+ " Amsterdam."
244
+ ]
245
+ },
246
+ {
247
+ "cell_type": "code",
248
+ "metadata": {},
249
+ "source": [
250
+ "for name, df in [(\"Madrid\", madrid), (\"Amsterdam\", amsterdam)]:\n",
251
+ " n = len(df)\n",
252
+ " counts = df[\"age_class\"].value_counts().sort_index()\n",
253
+ " print(f\"\\n{name}: {n:,} labelled pixels\")\n",
254
+ " for c in C.CLASS_LABELS[name]:\n",
255
+ " k = int(counts.get(c, 0))\n",
256
+ " print(f\" class {c} ({C.CLASS_LABELS[name][c]:>11}): {k:6,} {k / n:6.1%}\")"
257
+ ],
258
+ "execution_count": null,
259
+ "outputs": []
260
+ },
261
+ {
262
+ "cell_type": "code",
263
+ "metadata": {},
264
+ "source": [
265
+ "# How far apart are the two cities, feature by feature?\n",
266
+ "# Expressed in Madrid standard deviations, so it is directly comparable.\n",
267
+ "key = [\"sr_brightness_mean\", \"sr_Red_mean\", \"sr_NIR_mean\", \"sr_SWIR1_mean\",\n",
268
+ " \"sr_NDVI_mean\", \"sr_NDBI_mean\", \"sr_UI_mean\", \"sr_MNDWI_mean\",\n",
269
+ " \"sr_BSI_mean\", \"sr_brightness_std\", \"sr_brightness_late_minus_early\",\n",
270
+ " \"step_brightness_mag\", \"step_brightness_njumps\", \"coverage\"]\n",
271
+ "key = [k for k in key if k in madrid.columns and k in amsterdam.columns]\n",
272
+ "gap = pd.DataFrame({\n",
273
+ " \"madrid_mean\": [madrid[k].mean() for k in key],\n",
274
+ " \"amsterdam_mean\": [amsterdam[k].mean() for k in key],\n",
275
+ "}, index=key)\n",
276
+ "gap[\"madrid_sd\"] = [madrid[k].std() for k in key]\n",
277
+ "gap[\"gap_in_madrid_sd\"] = (gap.amsterdam_mean - gap.madrid_mean) / gap.madrid_sd\n",
278
+ "print(gap.round(3).sort_values(\"gap_in_madrid_sd\").to_string())"
279
+ ],
280
+ "execution_count": null,
281
+ "outputs": []
282
+ },
283
+ {
284
+ "cell_type": "markdown",
285
+ "metadata": {},
286
+ "source": [
287
+ "A feature whose `gap_in_madrid_sd` is near zero is one a Madrid-trained model\n",
288
+ "can still read in Amsterdam. A feature at +/-2 is telling the model which\n",
289
+ "*city* it is looking at, not how old the building is."
290
+ ]
291
+ },
292
+ {
293
+ "cell_type": "code",
294
+ "metadata": {},
295
+ "source": [
296
+ "# Within-class spread versus between-city gap, for the top offenders.\n",
297
+ "fig, axes = plt.subplots(2, 4, figsize=(18, 7))\n",
298
+ "for ax, k in zip(axes.ravel(), key[:8]):\n",
299
+ " for name, df, col in [(\"Madrid\", madrid, \"tab:red\"),\n",
300
+ " (\"Amsterdam\", amsterdam, \"tab:blue\")]:\n",
301
+ " ax.hist(df[k], bins=60, density=True, alpha=0.45, label=name, color=col)\n",
302
+ " ax.set_title(k, fontsize=9)\n",
303
+ " ax.tick_params(labelsize=7)\n",
304
+ "axes.ravel()[0].legend(fontsize=8)\n",
305
+ "plt.suptitle(\"The same feature in the two cities\", fontweight=\"bold\")\n",
306
+ "plt.tight_layout()\n",
307
+ "plt.show()"
308
+ ],
309
+ "execution_count": null,
310
+ "outputs": []
311
+ },
312
+ {
313
+ "cell_type": "markdown",
314
+ "metadata": {},
315
+ "source": [
316
+ "## 1.6 Does the construction-event detector actually work?\n",
317
+ "\n",
318
+ "A genuine test of the extended features. For classes 3 and 4 the building went\n",
319
+ "up *during* the satellite record, so `step_brightness_year` should land near\n",
320
+ "`weighted_mean_year`. For classes 1 and 2 it was already standing in 1984, so\n",
321
+ "the detector should find nothing meaningful. If that pattern does not show up,\n",
322
+ "the detector is noise and should be dropped."
323
+ ]
324
+ },
325
+ {
326
+ "cell_type": "code",
327
+ "metadata": {},
328
+ "source": [
329
+ "for name, df in [(\"Madrid\", madrid), (\"Amsterdam\", amsterdam)]:\n",
330
+ " print(f\"\\n{name}\")\n",
331
+ " for sig in (\"brightness\", \"SWIR1\", \"NDVI\"):\n",
332
+ " sub = df[df.age_class.isin([3, 4])]\n",
333
+ " err = (sub[f\"step_{sig}_year\"] - sub[\"weighted_mean_year\"]).abs()\n",
334
+ " r = np.corrcoef(sub[f\"step_{sig}_year\"], sub[\"weighted_mean_year\"])[0, 1]\n",
335
+ " print(f\" step_{sig}_year vs true year (classes 3+4): r={r:+.3f} \"\n",
336
+ " f\"median |error| = {err.median():.1f} years \"\n",
337
+ " f\"within 5 y: {(err <= 5).mean():.1%}\")\n",
338
+ " print(\" mean |step magnitude| by class:\",\n",
339
+ " df.groupby(\"age_class\")[\"step_brightness_mag\"].apply(\n",
340
+ " lambda s: round(float(s.abs().mean()), 4)).to_dict())"
341
+ ],
342
+ "execution_count": null,
343
+ "outputs": []
344
+ },
345
+ {
346
+ "cell_type": "code",
347
+ "metadata": {},
348
+ "source": [
349
+ "fig, axes = plt.subplots(1, 2, figsize=(13, 4.5))\n",
350
+ "for ax, (name, df) in zip(axes, [(\"Madrid\", madrid), (\"Amsterdam\", amsterdam)]):\n",
351
+ " sub = df[df.age_class.isin([3, 4])].sample(min(4000, len(df)), random_state=0)\n",
352
+ " ax.scatter(sub[\"weighted_mean_year\"], sub[\"step_brightness_year\"], s=3,\n",
353
+ " alpha=0.25)\n",
354
+ " lims = [1984, 2024]\n",
355
+ " ax.plot(lims, lims, \"r--\", lw=1)\n",
356
+ " ax.set(xlabel=\"true construction year\", ylabel=\"detected brightness step year\",\n",
357
+ " title=f\"{name}: classes 3 and 4\", xlim=lims, ylim=lims)\n",
358
+ "plt.tight_layout()\n",
359
+ "plt.show()"
360
+ ],
361
+ "execution_count": null,
362
+ "outputs": []
363
+ },
364
+ {
365
+ "cell_type": "markdown",
366
+ "metadata": {},
367
+ "source": [
368
+ "## 1.7 Neighbourhood context - free information, no labels\n",
369
+ "\n",
370
+ "Buildings go up a street at a time, so a pixel's neighbours carry real signal\n",
371
+ "about its era. These features average a few key quantities over the 8 nearest\n",
372
+ "pixels (within 10 pixel units = 300 m) and also store the *difference* between\n",
373
+ "the pixel and its surroundings. No labels are involved, so this is available\n",
374
+ "in Amsterdam at n = 0."
375
+ ]
376
+ },
377
+ {
378
+ "cell_type": "code",
379
+ "metadata": {},
380
+ "source": [
381
+ "madrid = R.add_neighbourhood_features(madrid)\n",
382
+ "amsterdam = R.add_neighbourhood_features(amsterdam)\n",
383
+ "nbr_cols = [c for c in madrid.columns if c.startswith(\"nbr_\")]\n",
384
+ "print(f\"added {len(nbr_cols)} neighbourhood features\")\n",
385
+ "print(madrid[[\"age_class\"] + nbr_cols[:6]].groupby(\"age_class\").mean().round(4).to_string())"
386
+ ],
387
+ "execution_count": null,
388
+ "outputs": []
389
+ },
390
+ {
391
+ "cell_type": "markdown",
392
+ "metadata": {},
393
+ "source": [
394
+ "## 1.8 Save\n",
395
+ "\n",
396
+ "One parquet per city. Every later notebook starts from these."
397
+ ]
398
+ },
399
+ {
400
+ "cell_type": "code",
401
+ "metadata": {},
402
+ "source": [
403
+ "C.PROC_DIR.mkdir(parents=True, exist_ok=True)\n",
404
+ "madrid.to_parquet(C.PROC_DIR / \"madrid_features.parquet\", index=False)\n",
405
+ "amsterdam.to_parquet(C.PROC_DIR / \"amsterdam_features.parquet\", index=False)\n",
406
+ "for n, df in [(\"madrid\", madrid), (\"amsterdam\", amsterdam)]:\n",
407
+ " p = C.PROC_DIR / f\"{n}_features.parquet\"\n",
408
+ " print(f\"{p} {p.stat().st_size / 1e6:.1f} MB {df.shape}\")\n",
409
+ "print(\"\\nfeature groups:\",\n",
410
+ " f\"{len(C.FEATURE_COLS_60)} official,\",\n",
411
+ " f\"{sum(1 for c in madrid.columns if c.startswith('sr_'))} reflectance-based,\",\n",
412
+ " f\"{sum(1 for c in madrid.columns if c.startswith('step_'))} change-event,\",\n",
413
+ " f\"{len(nbr_cols)} neighbourhood\")"
414
+ ],
415
+ "execution_count": null,
416
+ "outputs": []
417
+ }
418
+ ],
419
+ "metadata": {
420
+ "kernelspec": {
421
+ "display_name": "Python 3",
422
+ "language": "python",
423
+ "name": "python3"
424
+ },
425
+ "language_info": {
426
+ "name": "python",
427
+ "version": "3.11"
428
+ },
429
+ "accelerator": "GPU",
430
+ "colab": {
431
+ "provenance": [],
432
+ "gpuType": "L4"
433
+ }
434
+ },
435
+ "nbformat": 4,
436
+ "nbformat_minor": 5
437
+ }
notebooks/03_gbdt_models.ipynb ADDED
@@ -0,0 +1,747 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "metadata": {},
6
+ "source": [
7
+ "# 3 - LightGBM and CatBoost: the two classical models\n",
8
+ "\n",
9
+ "The organisers ask for at least two models, chosen on Madrid. These are them.\n",
10
+ "They are also the honest yardstick the fine-tuned LLM has to beat, and the\n",
11
+ "source of the probabilities we later ensemble with it.\n",
12
+ "\n",
13
+ "Running order:\n",
14
+ "\n",
15
+ "1. **Spatial** cross-validation on Madrid - random folds are dishonest here,\n",
16
+ " because neighbouring pixels share a construction era.\n",
17
+ "2. **Adversarial validation** - find the features that only say \"this is\n",
18
+ " Madrid\" and drop them.\n",
19
+ "3. **Per-city normalisation** - the single biggest transfer win, and it needs\n",
20
+ " no labels.\n",
21
+ "4. **LightGBM vs CatBoost vs ordinal LightGBM**, all judged on Madrid only.\n",
22
+ "5. **The official few-shot protocol** at n = 5, 25, 50, 100, 200.\n",
23
+ "\n",
24
+ "Runs on CPU. About 20 minutes on a laptop."
25
+ ]
26
+ },
27
+ {
28
+ "cell_type": "code",
29
+ "metadata": {},
30
+ "source": [
31
+ "import sys\n",
32
+ "from pathlib import Path\n",
33
+ "\n",
34
+ "for cand in [Path.cwd() / \"src\", Path.cwd().parent / \"src\",\n",
35
+ " Path(\"/kaggle/working/src\"), Path(\"/content/hack4dev/src\")]:\n",
36
+ " if cand.exists():\n",
37
+ " sys.path.insert(0, str(cand))\n",
38
+ " break\n",
39
+ "\n",
40
+ "import numpy as np\n",
41
+ "import pandas as pd\n",
42
+ "import matplotlib.pyplot as plt\n",
43
+ "\n",
44
+ "import config as C\n",
45
+ "import evaluation as E\n",
46
+ "import gbdt as G\n",
47
+ "import retrieval as R\n",
48
+ "\n",
49
+ "RNG = np.random.default_rng(C.RANDOM_SEED)\n",
50
+ "\n",
51
+ "# Prefer the full feature tables from notebook 1. Fall back to the organisers'\n",
52
+ "# preprocessed_data.pkl, which carries the official 60 features and the pixel\n",
53
+ "# coordinates - enough for everything in THIS notebook, and only ~50 MB, so it\n",
54
+ "# is the right thing to grab first if the raw parquet is hard to get hold of.\n",
55
+ "import features as F\n",
56
+ "\n",
57
+ "if (C.PROC_DIR / \"madrid_features.parquet\").exists():\n",
58
+ " madrid = pd.read_parquet(C.PROC_DIR / \"madrid_features.parquet\")\n",
59
+ " amsterdam = pd.read_parquet(C.PROC_DIR / \"amsterdam_features.parquet\")\n",
60
+ " SOURCE = \"notebook 1 (official 60 + extended)\"\n",
61
+ "else:\n",
62
+ " pkl = None\n",
63
+ " for root in [C.DATA_DIR, C.PROJECT_ROOT, Path(\"/kaggle/input\")]:\n",
64
+ " if root.exists():\n",
65
+ " hits = sorted(root.rglob(\"preprocessed_data.pkl\"))\n",
66
+ " if hits:\n",
67
+ " pkl = hits[0]\n",
68
+ " break\n",
69
+ " if pkl is None:\n",
70
+ " raise FileNotFoundError(\n",
71
+ " \"Neither data/processed/madrid_features.parquet (run notebook 1) \"\n",
72
+ " \"nor preprocessed_data.pkl were found.\")\n",
73
+ " madrid, amsterdam = F.from_official_pkl(pkl)\n",
74
+ " SOURCE = \"organisers' preprocessed_data.pkl (official 60 only)\"\n",
75
+ "\n",
76
+ "print(f\"\\nfeature source: {SOURCE}\")\n",
77
+ "print(madrid.shape, amsterdam.shape)"
78
+ ],
79
+ "execution_count": null,
80
+ "outputs": []
81
+ },
82
+ {
83
+ "cell_type": "code",
84
+ "metadata": {},
85
+ "source": [
86
+ "# Feature sets we will compare. Keeping them explicit makes the ablation table\n",
87
+ "# in the report trivial to produce.\n",
88
+ "DROP = {\"px_key\", \"py_key\", \"age_class\", \"weighted_mean_year\", \"city\"}\n",
89
+ "ALL_FEATS = [c for c in madrid.columns\n",
90
+ " if c not in DROP and madrid[c].dtype.kind in \"fiu\"]\n",
91
+ "DEAD = [c for c in ALL_FEATS if madrid[c].std() == 0 or amsterdam[c].std() == 0]\n",
92
+ "ALL_FEATS = [c for c in ALL_FEATS if c not in DEAD]\n",
93
+ "\n",
94
+ "FEATURE_SETS = {\n",
95
+ " \"official60\": [c for c in C.FEATURE_COLS_60 if c in ALL_FEATS],\n",
96
+ " \"reflectance\": [c for c in ALL_FEATS if c.startswith(\"sr_\")],\n",
97
+ " \"events\": [c for c in ALL_FEATS if c.startswith(\"step_\")],\n",
98
+ " \"neighbourhood\": [c for c in ALL_FEATS if c.startswith(\"nbr_\")],\n",
99
+ " \"everything\": ALL_FEATS,\n",
100
+ "}\n",
101
+ "print(\"dead (zero variance in at least one city):\", DEAD)\n",
102
+ "for k, v in FEATURE_SETS.items():\n",
103
+ " print(f\" {k:<14} {len(v)} features\")\n",
104
+ "\n",
105
+ "y_mad = madrid[\"age_class\"].to_numpy()\n",
106
+ "y_ams = amsterdam[\"age_class\"].to_numpy() # opened only for scoring\n",
107
+ "coords_mad = madrid[C.PID].to_numpy(float)\n",
108
+ "coords_ams = amsterdam[C.PID].to_numpy(float)"
109
+ ],
110
+ "execution_count": null,
111
+ "outputs": []
112
+ },
113
+ {
114
+ "cell_type": "markdown",
115
+ "metadata": {},
116
+ "source": [
117
+ "## 3.1 An honest validation split\n",
118
+ "\n",
119
+ "`RepeatedStratifiedKFold` on pixels puts a pixel's own neighbours in the\n",
120
+ "training fold. Neighbours share an era, so the model can almost look up the\n",
121
+ "answer, and Madrid CV comes out flattering - part of why 0.63 on Madrid turns\n",
122
+ "into 0.34 on Amsterdam.\n",
123
+ "\n",
124
+ "Instead: cut Madrid into geographic blocks and hold out whole blocks. That is\n",
125
+ "our stand-in for \"a city the model has never seen\", and it is the only thing\n",
126
+ "we are allowed to choose models on."
127
+ ]
128
+ },
129
+ {
130
+ "cell_type": "code",
131
+ "metadata": {},
132
+ "source": [
133
+ "blocks_mad = R.spatial_blocks(coords_mad, n_blocks=25)\n",
134
+ "print(\"block sizes:\", np.bincount(blocks_mad))\n",
135
+ "\n",
136
+ "fig, ax = plt.subplots(figsize=(7, 5))\n",
137
+ "ax.scatter(coords_mad[:, 0], coords_mad[:, 1], c=blocks_mad, s=1, cmap=\"tab20\")\n",
138
+ "ax.set(title=\"Madrid split into spatial blocks\", xlabel=\"px_key\", ylabel=\"py_key\")\n",
139
+ "plt.tight_layout()\n",
140
+ "plt.show()\n",
141
+ "\n",
142
+ "\n",
143
+ "def block_cv(X, y, blocks, fit_fn, n_folds=5, normalise=\"robustz\", seed=0):\n",
144
+ " \"\"\"Hold out whole geographic blocks, score macro F1 on each.\"\"\"\n",
145
+ " uniq = np.unique(blocks)\n",
146
+ " rng = np.random.default_rng(seed)\n",
147
+ " order = rng.permutation(uniq)\n",
148
+ " folds = np.array_split(order, n_folds)\n",
149
+ " scores, oof = [], np.zeros((len(y), 4))\n",
150
+ " for f in folds:\n",
151
+ " te = np.isin(blocks, f)\n",
152
+ " tr = ~te\n",
153
+ " nrm = G.CityNormaliser(normalise).fit(X[tr]) if normalise else None\n",
154
+ " Xtr = nrm.transform(X[tr]) if nrm else X[tr]\n",
155
+ " # The held-out block plays the part of a new city: it gets normalised\n",
156
+ " # with its OWN statistics, exactly as Amsterdam will be.\n",
157
+ " nte = G.CityNormaliser(normalise).fit(X[te]) if normalise else None\n",
158
+ " Xte = nte.transform(X[te]) if nte else X[te]\n",
159
+ " m = fit_fn(Xtr, y[tr])\n",
160
+ " p = m.predict_proba(Xte)\n",
161
+ " oof[te] = p\n",
162
+ " scores.append(E.macro_f1(y[te], E.CLASSES[np.argmax(p, 1)]))\n",
163
+ " return np.array(scores), oof"
164
+ ],
165
+ "execution_count": null,
166
+ "outputs": []
167
+ },
168
+ {
169
+ "cell_type": "markdown",
170
+ "metadata": {},
171
+ "source": [
172
+ "## 3.2 Which features betray the city?\n",
173
+ "\n",
174
+ "Train a classifier to separate Madrid pixels from Amsterdam pixels. If it\n",
175
+ "succeeds easily, the features carry a strong city signature. The most\n",
176
+ "important features in *that* task are the ones most likely to mislead the age\n",
177
+ "model, and are candidates for removal."
178
+ ]
179
+ },
180
+ {
181
+ "cell_type": "code",
182
+ "metadata": {},
183
+ "source": [
184
+ "auc, imp = G.adversarial_validation(madrid[ALL_FEATS], amsterdam[ALL_FEATS], ALL_FEATS)\n",
185
+ "print(f\"city-vs-city AUC: {auc:.4f} (1.0 = the two cities are trivially told apart)\")\n",
186
+ "print(imp.head(20).to_string(index=False))"
187
+ ],
188
+ "execution_count": null,
189
+ "outputs": []
190
+ },
191
+ {
192
+ "cell_type": "markdown",
193
+ "metadata": {},
194
+ "source": [
195
+ "The joint model tells us the cities are separable; it does not tell us which\n",
196
+ "feature is to blame, because importance is shared among correlated features.\n",
197
+ "A single-feature AUC does, and the before/after table is the most convincing\n",
198
+ "figure in the write-up: 0.5 means a feature is domain-invariant, 1.0 means it\n",
199
+ "is a pure city marker and any Madrid split on it is meaningless in Amsterdam."
200
+ ]
201
+ },
202
+ {
203
+ "cell_type": "code",
204
+ "metadata": {},
205
+ "source": [
206
+ "Xm_all = madrid[ALL_FEATS].to_numpy(float)\n",
207
+ "Xa_all = amsterdam[ALL_FEATS].to_numpy(float)\n",
208
+ "sf_raw = G.single_feature_domain_auc(Xm_all, Xa_all, ALL_FEATS)\n",
209
+ "\n",
210
+ "# The rankable mask is a STRUCTURAL decision about the features, so it is\n",
211
+ "# computed once on Madrid - the design city - and shared. Fitting it per city\n",
212
+ "# lets a column be ranked in one and z-scored in the other, which puts them on\n",
213
+ "# different scales and reports a domain AUC of 1.000 for a feature that is\n",
214
+ "# actually domain-invariant. `n_observed_years` did exactly that.\n",
215
+ "RANKABLE = G.rankable_mask(Xm_all)\n",
216
+ "print(f\"rank-transformable on Madrid: {RANKABLE.sum()} of {len(ALL_FEATS)}; \"\n",
217
+ " f\"too heavily tied to rank: {[ALL_FEATS[j] for j in np.flatnonzero(~RANKABLE)]}\")\n",
218
+ "nm_ = G.CityNormaliser(\"rank\", rankable=RANKABLE).fit(Xm_all).transform(Xm_all)\n",
219
+ "na_ = G.CityNormaliser(\"rank\", rankable=RANKABLE).fit(Xa_all).transform(Xa_all)\n",
220
+ "sf_rank = G.single_feature_domain_auc(nm_, na_, ALL_FEATS)\n",
221
+ "screen = (sf_raw.rename(columns={\"domain_auc\": \"auc_raw\"})\n",
222
+ " .merge(sf_rank.rename(columns={\"domain_auc\": \"auc_after_rank\"}), on=\"feature\")\n",
223
+ " .sort_values(\"auc_raw\", ascending=False))\n",
224
+ "print(\"worst offenders, raw vs after per-city rank normalisation:\")\n",
225
+ "print(screen.head(25).round(3).to_string(index=False))\n",
226
+ "print(f\"\\nmedian domain AUC: raw {screen.auc_raw.median():.3f} -> \"\n",
227
+ " f\"after rank {screen.auc_after_rank.median():.3f}\")\n",
228
+ "print(f\"features still above 0.90 after normalisation: \"\n",
229
+ " f\"{(screen.auc_after_rank > 0.90).sum()} of {len(screen)}\")\n",
230
+ "screen.to_csv(C.REPORT_DIR / \"domain_auc_screen.csv\", index=False)\n",
231
+ "\n",
232
+ "fig, ax = plt.subplots(figsize=(7, 5))\n",
233
+ "ax.scatter(screen.auc_raw, screen.auc_after_rank, s=8, alpha=0.5)\n",
234
+ "ax.plot([0.5, 1], [0.5, 1], \"k--\", lw=1)\n",
235
+ "ax.set(xlabel=\"domain AUC, raw\", ylabel=\"domain AUC, after per-city rank\",\n",
236
+ " title=\"Per-city ranking removes the city signature (points below the line)\")\n",
237
+ "plt.tight_layout()\n",
238
+ "plt.show()"
239
+ ],
240
+ "execution_count": null,
241
+ "outputs": []
242
+ },
243
+ {
244
+ "cell_type": "code",
245
+ "metadata": {},
246
+ "source": [
247
+ "# Repeat after per-city normalisation: a good normaliser should make the cities\n",
248
+ "# harder to tell apart without destroying the age signal.\n",
249
+ "nm = G.CityNormaliser(\"robustz\").fit(Xm_all).transform(Xm_all)\n",
250
+ "na = G.CityNormaliser(\"robustz\").fit(Xa_all).transform(Xa_all)\n",
251
+ "auc_z, imp_z = G.adversarial_validation(nm, na, ALL_FEATS)\n",
252
+ "auc_r, imp_r = G.adversarial_validation(nm_, na_, ALL_FEATS) # shared mask\n",
253
+ "print(f\"AUC raw {auc:.4f} | robust-z {auc_z:.4f} | rank {auc_r:.4f}\")\n",
254
+ "print(\"\\nstill city-specific after rank normalisation:\")\n",
255
+ "print(imp_r.head(12).to_string(index=False))"
256
+ ],
257
+ "execution_count": null,
258
+ "outputs": []
259
+ },
260
+ {
261
+ "cell_type": "markdown",
262
+ "metadata": {},
263
+ "source": [
264
+ "Read the two screens together, because they say different things and the\n",
265
+ "difference is the point.\n",
266
+ "\n",
267
+ "Per-city **rank** normalisation forces every feature's marginal distribution\n",
268
+ "to be uniform in both cities, so the *single-feature* domain AUC becomes\n",
269
+ "exactly 0.5 by construction. That is not evidence that the domain gap is\n",
270
+ "gone. If the *joint* adversarial AUC above is still near 1.0 after ranking,\n",
271
+ "the city signature has simply moved into the **correlations between**\n",
272
+ "features, where a marginal transform cannot reach it.\n",
273
+ "\n",
274
+ "That is precisely what CORAL addresses - it matches second-order statistics\n",
275
+ "rather than marginals. So the honest conclusion from these two cells is:\n",
276
+ "ranking is necessary and cheap, and it is not sufficient. Whether CORAL on\n",
277
+ "top actually helps is an empirical question answered in section 3.5, not a\n",
278
+ "foregone conclusion: in synthetic testing CORAL slightly *hurt* when the\n",
279
+ "shift was close to diagonal."
280
+ ]
281
+ },
282
+ {
283
+ "cell_type": "markdown",
284
+ "metadata": {},
285
+ "source": [
286
+ "### The drop rule, and why it must be conservative\n",
287
+ "\n",
288
+ "The tempting rule is \"drop the top features by adversarial importance\". It is\n",
289
+ "wrong. With 59 features the average importance share is 1/59 = 0.017, so any\n",
290
+ "threshold near that drops whatever happens to be slightly above average -\n",
291
+ "and once normalisation has pushed the joint AUC down to ~0.5, those\n",
292
+ "importances are *noise*. In an earlier run this rule dropped three features\n",
293
+ "whose single-feature domain AUC was exactly 0.500, i.e. perfectly\n",
294
+ "domain-invariant ones.\n",
295
+ "\n",
296
+ "The principled rule needs two conditions, both of them absolute rather than\n",
297
+ "relative: a feature is dropped only if it is **still** strongly\n",
298
+ "city-discriminative on its own after normalisation, **and** it is not\n",
299
+ "carrying much of the age signal on Madrid. Usually that drops nothing, which\n",
300
+ "is the correct answer."
301
+ ]
302
+ },
303
+ {
304
+ "cell_type": "code",
305
+ "metadata": {},
306
+ "source": [
307
+ "AUC_CUT = 0.90 # still a city marker after normalisation\n",
308
+ "IMP_CUT = 0.005 # and contributes little to the Madrid model\n",
309
+ "\n",
310
+ "X_all = madrid[ALL_FEATS].to_numpy(float)\n",
311
+ "probe = G.fit_lgbm(G.CityNormaliser(\"rank\").fit_transform(X_all), y_mad,\n",
312
+ " params=dict(n_estimators=300))\n",
313
+ "age_imp = pd.DataFrame({\n",
314
+ " \"feature\": ALL_FEATS,\n",
315
+ " \"age_importance\": probe.feature_importances_ / max(probe.feature_importances_.sum(), 1)\n",
316
+ "})\n",
317
+ "drop_tbl = screen.merge(age_imp, on=\"feature\")\n",
318
+ "leaky = drop_tbl[(drop_tbl.auc_after_rank > AUC_CUT)\n",
319
+ " & (drop_tbl.age_importance < IMP_CUT)].feature.tolist()\n",
320
+ "FEATURE_SETS[\"pruned\"] = [c for c in ALL_FEATS if c not in leaky]\n",
321
+ "print(f\"features still above AUC {AUC_CUT} after normalisation: \"\n",
322
+ " f\"{(drop_tbl.auc_after_rank > AUC_CUT).sum()}\")\n",
323
+ "print(f\"dropping {len(leaky)}: {leaky if leaky else 'nothing - as expected'}\")\n",
324
+ "print(f\"pruned set: {len(FEATURE_SETS['pruned'])} features\")\n",
325
+ "print(\"\\nmost city-specific features that we KEPT because they carry age signal:\")\n",
326
+ "print(drop_tbl.sort_values(\"auc_after_rank\", ascending=False)\n",
327
+ " .head(8).round(4).to_string(index=False))"
328
+ ],
329
+ "execution_count": null,
330
+ "outputs": []
331
+ },
332
+ {
333
+ "cell_type": "markdown",
334
+ "metadata": {},
335
+ "source": [
336
+ "## 3.3 Model and feature-set comparison, on Madrid only\n",
337
+ "\n",
338
+ "Every row of this table is decided with Madrid block-CV. Amsterdam is not\n",
339
+ "consulted. This is the table that justifies the final model choice."
340
+ ]
341
+ },
342
+ {
343
+ "cell_type": "code",
344
+ "metadata": {},
345
+ "source": [
346
+ "# The comparison runs 4 models x 3 feature sets x 5 blocked folds = 60 fits.\n",
347
+ "# At full strength (1200 LightGBM rounds, 2000 CatBoost iterations) that is\n",
348
+ "# tens of minutes on 76k rows, and it buys nothing: we are reading the\n",
349
+ "# ORDERING of configurations here, not their final scores. So compare cheaply,\n",
350
+ "# then refit the winner at full strength in section 3.7.\n",
351
+ "CHEAP_LGB = dict(n_estimators=400, learning_rate=0.06)\n",
352
+ "CHEAP_CAT = dict(iterations=500, learning_rate=0.06)\n",
353
+ "\n",
354
+ "\n",
355
+ "def lgbm_balanced(X, y):\n",
356
+ " return G.fit_lgbm(X, y, params=CHEAP_LGB,\n",
357
+ " class_weight=dict(zip(range(4), G.balanced_class_weights(y))))\n",
358
+ "\n",
359
+ "def lgbm_plain(X, y):\n",
360
+ " return G.fit_lgbm(X, y, params=CHEAP_LGB)\n",
361
+ "\n",
362
+ "def cat_balanced(X, y):\n",
363
+ " return G.fit_catboost(X, y, params=CHEAP_CAT,\n",
364
+ " class_weights=G.balanced_class_weights(y))\n",
365
+ "\n",
366
+ "def ordinal_lgbm(X, y):\n",
367
+ " return G.OrdinalGBDT(params=CHEAP_LGB).fit(X, y)\n",
368
+ "\n",
369
+ "import time\n",
370
+ "\n",
371
+ "FEATURE_SETS_TO_TRY = [k for k in (\"official60\", \"everything\", \"pruned\")\n",
372
+ " if len(FEATURE_SETS.get(k, []))]\n",
373
+ "# Drop a duplicate: with the pkl-only fallback, \"everything\" == \"official60\".\n",
374
+ "seen, uniq = set(), []\n",
375
+ "for k in FEATURE_SETS_TO_TRY:\n",
376
+ " sig = tuple(FEATURE_SETS[k])\n",
377
+ " if sig not in seen:\n",
378
+ " seen.add(sig)\n",
379
+ " uniq.append(k)\n",
380
+ " else:\n",
381
+ " print(f\" (skipping '{k}' - identical to an earlier set)\")\n",
382
+ "FEATURE_SETS_TO_TRY = uniq\n",
383
+ "\n",
384
+ "rows = []\n",
385
+ "for fs_name in FEATURE_SETS_TO_TRY:\n",
386
+ " X = madrid[FEATURE_SETS[fs_name]].to_numpy(float)\n",
387
+ " models = [(\"lgbm\", lgbm_balanced), (\"lgbm_unweighted\", lgbm_plain),\n",
388
+ " (\"lgbm_ordinal\", ordinal_lgbm)]\n",
389
+ " if fs_name == FEATURE_SETS_TO_TRY[0]:\n",
390
+ " # CatBoost costs 35 min per feature set against LightGBM's 10 and came\n",
391
+ " # in about 0.008 behind on every set, so it runs on one set only. It\n",
392
+ " # still ships as the second required model.\n",
393
+ " models.append((\"catboost\", cat_balanced))\n",
394
+ " for mdl_name, fn in models:\n",
395
+ " t0 = time.time()\n",
396
+ " s, oof = block_cv(X, y_mad, blocks_mad, fn)\n",
397
+ " rows.append(dict(features=fs_name, model=mdl_name, n_feat=X.shape[1],\n",
398
+ " macro_f1=s.mean(), std=s.std(ddof=1),\n",
399
+ " minutes=(time.time() - t0) / 60))\n",
400
+ " print(f\" {fs_name:<12} {mdl_name:<16} {s.mean():.4f} +- \"\n",
401
+ " f\"{s.std(ddof=1):.4f} ({rows[-1]['minutes']:.1f} min)\")\n",
402
+ "table = pd.DataFrame(rows).sort_values(\"macro_f1\", ascending=False)\n",
403
+ "print(\"\\n\" + table.round(4).to_string(index=False))"
404
+ ],
405
+ "execution_count": null,
406
+ "outputs": []
407
+ },
408
+ {
409
+ "cell_type": "markdown",
410
+ "metadata": {},
411
+ "source": [
412
+ "### Does per-city normalisation actually help? Decide it on Madrid.\n",
413
+ "\n",
414
+ "This has to be settled here and not on the Amsterdam score, because the\n",
415
+ "Amsterdam score is not ours to choose on. `block_cv` is the right instrument:\n",
416
+ "it normalises the held-out block with the block's OWN statistics, exactly as\n",
417
+ "Amsterdam will be normalised, so each fold is a miniature transfer problem.\n",
418
+ "\n",
419
+ "Worth doing carefully, because the intuition (\"absolute reflectance does not\n",
420
+ "transfer, so normalise it away\") is strong and may still be wrong. Per-city\n",
421
+ "normalisation also destroys genuine between-city differences in LEVEL, and if\n",
422
+ "a city's overall darkness happens to correlate with its class mix, removing\n",
423
+ "the level removes a real cue along with the nuisance."
424
+ ]
425
+ },
426
+ {
427
+ "cell_type": "code",
428
+ "metadata": {},
429
+ "source": [
430
+ "norm_rows = []\n",
431
+ "X_best = madrid[FEATURE_SETS[table.iloc[0].features]].to_numpy(float)\n",
432
+ "for norm in (None, \"robustz\", \"rank\"):\n",
433
+ " s, _ = block_cv(X_best, y_mad, blocks_mad, lgbm_plain, normalise=norm)\n",
434
+ " norm_rows.append(dict(normalise=str(norm), macro_f1=s.mean(),\n",
435
+ " std=s.std(ddof=1)))\n",
436
+ " print(f\" normalise={str(norm):8} {s.mean():.4f} +- {s.std(ddof=1):.4f}\")\n",
437
+ "norm_table = pd.DataFrame(norm_rows).sort_values(\"macro_f1\", ascending=False)\n",
438
+ "print(\"\\n\" + norm_table.round(4).to_string(index=False))\n",
439
+ "BEST_NORM = None if norm_table.iloc[0].normalise == \"None\" else norm_table.iloc[0].normalise\n",
440
+ "print(f\"\\nnormalisation chosen on Madrid: {BEST_NORM}\")\n",
441
+ "print(\"Whatever Amsterdam says later, this is the choice the rules allow us to\")\n",
442
+ "print(\"make. If Amsterdam disagrees, that is a finding to report, not a reason\")\n",
443
+ "print(\"to switch.\")"
444
+ ],
445
+ "execution_count": null,
446
+ "outputs": []
447
+ },
448
+ {
449
+ "cell_type": "markdown",
450
+ "metadata": {},
451
+ "source": [
452
+ "## 3.4 The decision rule matters as much as the model\n",
453
+ "\n",
454
+ "Macro F1 is not maximised by `argmax`. Three cheap post-processing steps,\n",
455
+ "each measured separately on Madrid block-CV:\n",
456
+ "\n",
457
+ "* **class weights** tuned by coordinate ascent on out-of-fold probabilities;\n",
458
+ "* **prior correction** towards the class prior EM estimates from *unlabelled*\n",
459
+ " target features;\n",
460
+ "* **spatial smoothing** of the probabilities over neighbouring pixels."
461
+ ]
462
+ },
463
+ {
464
+ "cell_type": "code",
465
+ "metadata": {},
466
+ "source": [
467
+ "BEST_FS = table.iloc[0].features\n",
468
+ "X_mad = madrid[FEATURE_SETS[BEST_FS]].to_numpy(float)\n",
469
+ "X_ams = amsterdam[FEATURE_SETS[BEST_FS]].to_numpy(float)\n",
470
+ "scores, oof = block_cv(X_mad, y_mad, blocks_mad, lgbm_balanced)\n",
471
+ "print(f\"baseline argmax, Madrid block-CV: {E.macro_f1(y_mad, E.CLASSES[np.argmax(oof, 1)]):.4f}\")\n",
472
+ "\n",
473
+ "w = E.tune_class_weights(oof, y_mad)\n",
474
+ "print(\"tuned class weights:\", np.round(w, 3))\n",
475
+ "print(\"after weighting: \",\n",
476
+ " f\"{E.macro_f1(y_mad, E.predict_weighted(oof, w)):.4f}\")\n",
477
+ "\n",
478
+ "for radius in (2.0, 4.0, 8.0):\n",
479
+ " sm = E.spatial_smooth(oof, coords_mad, radius=radius, alpha=0.5)\n",
480
+ " print(f\"spatial smoothing r={radius:>4}: \"\n",
481
+ " f\"{E.macro_f1(y_mad, E.predict_weighted(sm, w)):.4f}\")"
482
+ ],
483
+ "execution_count": null,
484
+ "outputs": []
485
+ },
486
+ {
487
+ "cell_type": "code",
488
+ "metadata": {},
489
+ "source": [
490
+ "print(pd.DataFrame(E.confusion_proportions(y_mad, E.predict_weighted(oof, w)),\n",
491
+ " index=[f\"true {c}\" for c in E.CLASSES],\n",
492
+ " columns=[f\"pred {c}\" for c in E.CLASSES]).round(3).to_string())"
493
+ ],
494
+ "execution_count": null,
495
+ "outputs": []
496
+ },
497
+ {
498
+ "cell_type": "markdown",
499
+ "metadata": {},
500
+ "source": [
501
+ "## 3.5 The simulated-shift harness - choosing the decision rule on Madrid\n",
502
+ "\n",
503
+ "This is the most important cell in the notebook, and the thing I would most\n",
504
+ "want the team to copy.\n",
505
+ "\n",
506
+ "None of the adaptation tricks above can be chosen honestly without a way to\n",
507
+ "*simulate* Amsterdam from Madrid. So: hold out a spatial block, and build four\n",
508
+ "versions of it.\n",
509
+ "\n",
510
+ "* `clean` - as-is;\n",
511
+ "* `label` - resampled to a random Dirichlet class prior (pure label shift);\n",
512
+ "* `covariate` - every feature quantile-mapped onto Amsterdam's own marginal\n",
513
+ " (pure covariate shift, using `X_amsterdam` only, no labels);\n",
514
+ "* `both`.\n",
515
+ "\n",
516
+ "Then score every candidate decision rule on all four. A rule that helps under\n",
517
+ "label shift can lose badly under covariate shift, and without this there is no\n",
518
+ "way to tell which regime you are actually in. **Ship only rules that are\n",
519
+ "non-negative in the worst case.**\n",
520
+ "\n",
521
+ "In synthetic testing this harness caught a trap: free SLD-EM prior estimation\n",
522
+ "looked excellent under label shift and lost 0.19 macro F1 on average (0.35 at\n",
523
+ "worst) under covariate shift, because EM assumes label shift only. The\n",
524
+ "macro-F1 plug-in rule driven by the *source* prior was the robust winner."
525
+ ]
526
+ },
527
+ {
528
+ "cell_type": "code",
529
+ "metadata": {},
530
+ "source": [
531
+ "# Note the UNWEIGHTED model here. The whole point of this cell is to choose the\n",
532
+ "# operating point, so the probabilities going into it must be an honest estimate\n",
533
+ "# of the Madrid posterior. class_weight='balanced' bakes a decision into the fit,\n",
534
+ "# where no decision rule can undo it.\n",
535
+ "harness_df, harness_table = E.shift_harness(\n",
536
+ " lgbm_plain, X_mad, y_mad, blocks_mad, X_ams,\n",
537
+ " E.DEFAULT_STRATEGIES, n_folds=5, n_rep=4, seed=C.RANDOM_SEED)\n",
538
+ "harness_table.to_csv(C.REPORT_DIR / \"shift_harness.csv\")"
539
+ ],
540
+ "execution_count": null,
541
+ "outputs": []
542
+ },
543
+ {
544
+ "cell_type": "code",
545
+ "metadata": {},
546
+ "source": [
547
+ "# The criterion is the PER-REGIME worst case: for each rule, its deficit\n",
548
+ "# against argmax in whichever regime it handles worst. An earlier version used\n",
549
+ "# a column that compared each rule's worst cell anywhere against argmax's worst\n",
550
+ "# cell anywhere - two different regimes and two different folds - so a rule\n",
551
+ "# could look safe merely because its bad regime was also argmax's bad regime.\n",
552
+ "WORST = \"delta_vs_argmax_worst_regime\"\n",
553
+ "safe = harness_table[harness_table.get(WORST, 0) >= 0]\n",
554
+ "print(\"rules that never lose to plain argmax in ANY regime:\")\n",
555
+ "print(safe.round(4).to_string() if len(safe) else \" none - keep plain argmax\")\n",
556
+ "print(\"\\nrules to avoid:\")\n",
557
+ "risky = harness_table[harness_table.get(WORST, 0) < -0.01]\n",
558
+ "print(risky.round(4).to_string() if len(risky) else \" none\")\n",
559
+ "\n",
560
+ "CHOSEN_RULE = (harness_table[WORST].idxmax()\n",
561
+ " if WORST in harness_table else \"argmax\")\n",
562
+ "print(f\"\\nrule chosen on Madrid alone: {CHOSEN_RULE}\")\n",
563
+ "DECISION = E.DEFAULT_STRATEGIES[CHOSEN_RULE]"
564
+ ],
565
+ "execution_count": null,
566
+ "outputs": []
567
+ },
568
+ {
569
+ "cell_type": "markdown",
570
+ "metadata": {},
571
+ "source": [
572
+ "## 3.6 The official few-shot protocol\n",
573
+ "\n",
574
+ "`y_amsterdam` enters here and only here: to draw the n labelled pixels per\n",
575
+ "class that the rules allow, and to score the remainder. Nothing above was\n",
576
+ "tuned on it."
577
+ ]
578
+ },
579
+ {
580
+ "cell_type": "code",
581
+ "metadata": {},
582
+ "source": [
583
+ "N_LIST = (5, 25, 50, 100, 200)\n",
584
+ "N_REPEATS = 12\n",
585
+ "# Score a fixed random 8,000-pixel sample of Amsterdam rather than all 25,992.\n",
586
+ "# Full scoring cost 19 s per draw - 32 minutes for one variant and hours for the\n",
587
+ "# table - and bought nothing: the variants that ignore the labelled sample have\n",
588
+ "# a standard deviation of 0.0001 across draws. The subsample is fixed, so every\n",
589
+ "# row of the table is compared on identical pixels. Re-run the winner on all\n",
590
+ "# 25,992 once, at the end, for the number that goes in the report.\n",
591
+ "EVAL_SUBSAMPLE = 8000\n",
592
+ "src_prior = np.array([(y_mad == c).mean() for c in E.CLASSES])\n",
593
+ "print('Madrid class prior:', np.round(src_prior, 3))\n",
594
+ "\n",
595
+ "# Cost control. Most variants cache the Madrid-trained model across repeats, so\n",
596
+ "# 20 draws are cheap. `upweighted shots, retrained` refits from scratch on every\n",
597
+ "# draw - 100 full fits on 76k rows - so it gets cheap params and fewer draws,\n",
598
+ "# and its std is correspondingly noisier. That is noted in the table.\n",
599
+ "REPEATS = {\"upweighted shots, retrained\": 5}\n",
600
+ "\n",
601
+ "variants = {\n",
602
+ " # the two baselines everything must beat\n",
603
+ " \"organisers' prototype (Euclidean, Madrid scaler)\": G.mc_prototype_official(),\n",
604
+ " \"pure retrieval k-NN (no source model)\": G.mc_knn(),\n",
605
+ " # zero-shot: does normalisation alone carry us?\n",
606
+ " \"zero-shot, no normalisation\": G.mc_zero_shot(normalise=None),\n",
607
+ " \"zero-shot, robust z\": G.mc_zero_shot(normalise=\"robustz\"),\n",
608
+ " \"zero-shot, rank\": G.mc_zero_shot(normalise=\"rank\"),\n",
609
+ " \"zero-shot, robust z + CORAL\": G.mc_zero_shot(normalise=\"robustz\", use_coral=True),\n",
610
+ " # the prototype metric ablation. On synthetic anisotropic data the pooled\n",
611
+ " # WITHIN-class Mahalanobis metric beat the organisers' Euclidean prototype\n",
612
+ " # by +0.11 macro F1 at n=25 - the largest single gain of anything tried.\n",
613
+ " # The total-covariance version is worse because the target city's total\n",
614
+ " # covariance is dominated by BETWEEN-class spread, so it shrinks exactly the\n",
615
+ " # directions that separate the classes. Confirm the ordering here on the\n",
616
+ " # real features before believing it.\n",
617
+ " \"prototype, robust z + euclidean\": G.mc_prototype(metric=\"euclidean\"),\n",
618
+ " \"prototype, rank + euclidean\": G.mc_prototype(normalise=\"rank\", metric=\"euclidean\"),\n",
619
+ " \"prototype, robust z + total-cov Mahalanobis\": G.mc_prototype(metric=\"total\"),\n",
620
+ " \"prototype, robust z + within-class Mahalanobis\": G.mc_prototype(metric=\"within\"),\n",
621
+ " # using the labelled sample with the source model\n",
622
+ " \"recalibrated (stratification-corrected)\": G.mc_recalibrated(normalise=\"robustz\"),\n",
623
+ " \"refit leaf values (structure frozen)\": G.mc_refit(),\n",
624
+ " \"upweighted shots, retrained\": G.mc_shot_weighted(\n",
625
+ " normalise=\"robustz\", fit=lambda X, y, **kw: G.fit_lgbm(X, y, params=CHEAP_LGB, **kw)),\n",
626
+ " \"catboost recalibrated\": G.mc_recalibrated(fit=G.fit_catboost, normalise=\"robustz\"),\n",
627
+ "}\n",
628
+ "\n",
629
+ "protocol = {}\n",
630
+ "for name, mc in variants.items():\n",
631
+ " print(f\"\\n{name}\")\n",
632
+ " reps = REPEATS.get(name, N_REPEATS)\n",
633
+ " t0 = time.time()\n",
634
+ " protocol[name] = E.evaluate_protocol(\n",
635
+ " mc, X_mad, y_mad, X_ams, y_ams, n_list=N_LIST, n_repeats=reps,\n",
636
+ " decision=lambda p: DECISION(p, src_prior),\n",
637
+ " eval_subsample=EVAL_SUBSAMPLE)\n",
638
+ " print(f\" ({reps} draws per n, {(time.time() - t0) / 60:.1f} min)\")"
639
+ ],
640
+ "execution_count": null,
641
+ "outputs": []
642
+ },
643
+ {
644
+ "cell_type": "code",
645
+ "metadata": {},
646
+ "source": [
647
+ "summary = pd.DataFrame({name: {n: r[\"mean\"] for n, r in res.items()}\n",
648
+ " for name, res in protocol.items()}).T\n",
649
+ "summary[\"average\"] = summary.mean(axis=1)\n",
650
+ "summary[\"peak\"] = summary[list(N_LIST)].max(axis=1)\n",
651
+ "print(summary.round(4).sort_values(\"average\", ascending=False).to_string())"
652
+ ],
653
+ "execution_count": null,
654
+ "outputs": []
655
+ },
656
+ {
657
+ "cell_type": "code",
658
+ "metadata": {},
659
+ "source": [
660
+ "fig, ax = plt.subplots(figsize=(8, 5))\n",
661
+ "for name, res in protocol.items():\n",
662
+ " m = [res[n][\"mean\"] for n in N_LIST]\n",
663
+ " s = [res[n][\"std\"] for n in N_LIST]\n",
664
+ " ax.errorbar(np.log2(N_LIST), m, yerr=s, marker=\"o\", capsize=3, label=name)\n",
665
+ "ax.axhline(0.618, ls=\"--\", c=\"k\", lw=1, label=\"published Madrid baseline 0.618\")\n",
666
+ "ax.set(xlabel=\"log2(labelled Amsterdam pixels per class)\", ylabel=\"macro F1 on Amsterdam\",\n",
667
+ " title=\"Macro F1 against labelling budget\")\n",
668
+ "ax.legend(fontsize=7)\n",
669
+ "plt.tight_layout()\n",
670
+ "plt.show()"
671
+ ],
672
+ "execution_count": null,
673
+ "outputs": []
674
+ },
675
+ {
676
+ "cell_type": "markdown",
677
+ "metadata": {},
678
+ "source": [
679
+ "## 3.7 Save the probabilities for the ensemble\n",
680
+ "\n",
681
+ "The LLM notebook loads these and blends them with its own predictions."
682
+ ]
683
+ },
684
+ {
685
+ "cell_type": "code",
686
+ "metadata": {},
687
+ "source": [
688
+ "import pickle\n",
689
+ "\n",
690
+ "best_name = summary.sort_values(\"average\", ascending=False).index[0]\n",
691
+ "print(\"best classical variant on the protocol:\", best_name)\n",
692
+ "\n",
693
+ "nrm_m = G.CityNormaliser(\"robustz\").fit(X_mad)\n",
694
+ "nrm_a = G.CityNormaliser(\"robustz\").fit(X_ams)\n",
695
+ "final = G.fit_lgbm(nrm_m.transform(X_mad), y_mad,\n",
696
+ " class_weight=dict(zip(range(4), G.balanced_class_weights(y_mad))))\n",
697
+ "final_cat = G.fit_catboost(nrm_m.transform(X_mad), y_mad,\n",
698
+ " class_weights=G.balanced_class_weights(y_mad))\n",
699
+ "\n",
700
+ "out = {\n",
701
+ " \"feature_set\": BEST_FS,\n",
702
+ " \"feature_names\": FEATURE_SETS[BEST_FS],\n",
703
+ " \"madrid_oof_proba\": oof,\n",
704
+ " \"madrid_blocks\": blocks_mad,\n",
705
+ " \"class_weights\": w,\n",
706
+ " \"amsterdam_proba_lgbm\": final.predict_proba(nrm_a.transform(X_ams)),\n",
707
+ " \"amsterdam_proba_catboost\": final_cat.predict_proba(nrm_a.transform(X_ams)),\n",
708
+ " \"protocol\": {k: {n: dict(mean=r[\"mean\"], std=r[\"std\"],\n",
709
+ " per_class=r[\"per_class\"].tolist())\n",
710
+ " for n, r in v.items()} for k, v in protocol.items()},\n",
711
+ " # Plain lists, not a DataFrame: this file is read on Colab, whose pandas\n",
712
+ " # 2.x cannot unpickle a pandas 3 DataFrame with string columns\n",
713
+ " # (\"StringDtype.__init__() takes from 1 to 2 positional arguments\").\n",
714
+ " # Rebuild with pd.DataFrame(out[\"madrid_block_cv_table\"]).\n",
715
+ " \"madrid_block_cv_table\": {c: table[c].tolist() for c in table.columns},\n",
716
+ " \"adversarial_auc\": dict(raw=auc, robustz=auc_z, rank=auc_r),\n",
717
+ " \"dropped_city_specific\": leaky,\n",
718
+ "}\n",
719
+ "with open(C.PROC_DIR / \"gbdt_outputs.pkl\", \"wb\") as fh:\n",
720
+ " pickle.dump(out, fh)\n",
721
+ "final.booster_.save_model(str(C.PROC_DIR / \"model_lgbm.txt\"))\n",
722
+ "final_cat.save_model(str(C.PROC_DIR / \"model_catboost.cbm\"))\n",
723
+ "print(\"saved:\", C.PROC_DIR / \"gbdt_outputs.pkl\")"
724
+ ],
725
+ "execution_count": null,
726
+ "outputs": []
727
+ }
728
+ ],
729
+ "metadata": {
730
+ "kernelspec": {
731
+ "display_name": "Python 3",
732
+ "language": "python",
733
+ "name": "python3"
734
+ },
735
+ "language_info": {
736
+ "name": "python",
737
+ "version": "3.11"
738
+ },
739
+ "accelerator": "GPU",
740
+ "colab": {
741
+ "provenance": [],
742
+ "gpuType": "L4"
743
+ }
744
+ },
745
+ "nbformat": 4,
746
+ "nbformat_minor": 5
747
+ }
notebooks/src/01_build_features.py ADDED
@@ -0,0 +1,290 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # %% [markdown]
2
+ # # 1 - Features: from 40 years of Landsat to one row per building pixel
3
+ #
4
+ # This notebook turns the two raw parquet files into two feature tables.
5
+ #
6
+ # It does **two** things at once:
7
+ #
8
+ # 1. **Reproduces the organisers' 60 features exactly** (Notebook 3), so every
9
+ # number we report stays comparable with the published Random-Forest
10
+ # baseline of 0.618 +- 0.004 macro F1 on Madrid.
11
+ # 2. **Adds an extended set** on top: proper surface reflectance instead of raw
12
+ # digital numbers, correct spectral indices, construction-event detection,
13
+ # per-decade means, trends, and label-free neighbourhood context.
14
+ #
15
+ # Nothing here touches `y_amsterdam`. Amsterdam's *features* are used (for
16
+ # per-city statistics), its *labels* are not.
17
+ #
18
+ # Runs on CPU in a few minutes - no GPU needed. Do this once, locally, and the
19
+ # GPU notebooks just load the parquet files it writes.
20
+
21
+ # %%
22
+ import sys
23
+ from pathlib import Path
24
+
25
+ # Make src/ importable whether we are local, on Kaggle, or on Colab.
26
+ for cand in [Path.cwd() / "src", Path.cwd().parent / "src",
27
+ Path("/kaggle/working/src"), Path("/content/hack4dev/src")]:
28
+ if cand.exists():
29
+ sys.path.insert(0, str(cand))
30
+ break
31
+
32
+ import numpy as np
33
+ import pandas as pd
34
+ import matplotlib.pyplot as plt
35
+
36
+ import config as C
37
+ import features as F
38
+ import retrieval as R
39
+
40
+ pd.set_option("display.width", 200)
41
+ pd.set_option("display.max_columns", 60)
42
+ print("platform:", C.PLATFORM)
43
+ print("looking for raw parquet under:", C.RAW_DIR)
44
+
45
+ # %% [markdown]
46
+ # ## 1.1 Find the raw files
47
+ #
48
+ # Expected: `madrid_train.parquet` (about 3.17 M rows) and
49
+ # `amsterdam_data.parquet` (about 1.06 M rows). On Kaggle they will be inside
50
+ # `/kaggle/input/<dataset-name>/`; the search below handles that.
51
+
52
+ # %%
53
+ def find_parquet(pattern: str) -> Path:
54
+ roots = [C.RAW_DIR, C.DATA_DIR, C.PROJECT_ROOT, Path("/kaggle/input")]
55
+ for root in roots:
56
+ if not root.exists():
57
+ continue
58
+ hits = sorted(root.rglob(pattern))
59
+ if hits:
60
+ return hits[0]
61
+ raise FileNotFoundError(
62
+ f"{pattern} not found. Put the two parquet files in {C.RAW_DIR} "
63
+ "(local) or attach them as a Kaggle dataset.")
64
+
65
+ MADRID_PARQUET = find_parquet("madrid_train.parquet")
66
+ AMS_PARQUET = find_parquet("amsterdam_data.parquet")
67
+ print("Madrid :", MADRID_PARQUET)
68
+ print("Amsterdam:", AMS_PARQUET)
69
+
70
+ # %% [markdown]
71
+ # ## 1.2 What is actually in the raw file
72
+ #
73
+ # One row per *(pixel, year)*. Up to three summer observations per year are
74
+ # stored side by side as `Blue_1/Blue_2/Blue_3`, each with a `qa_valid_N`
75
+ # cloud flag. Band values are raw Landsat Collection-2 integers:
76
+ #
77
+ # $$\text{surface reflectance} = \text{DN} \times 2.75\times10^{-5} - 0.2$$
78
+
79
+ # %%
80
+ peek = pd.read_parquet(MADRID_PARQUET)
81
+ print(f"{len(peek):,} rows x {peek.shape[1]} columns")
82
+ print("\ncolumns:")
83
+ print(list(peek.columns))
84
+ print("\nyears:", peek["year"].min(), "-", peek["year"].max(),
85
+ f"({peek['year'].nunique()} distinct)")
86
+ print("unique pixels:", f"{peek.groupby(C.PID).ngroups:,}")
87
+ print("\nfirst row:")
88
+ print(peek.iloc[0].to_string())
89
+
90
+ # %%
91
+ # Reflectance sanity check on one band, and the QA-flag hit rate.
92
+ dn = peek["Red_1"].dropna()
93
+ sr = dn * C.SR_SCALE + C.SR_OFFSET
94
+ print("Red_1 digital numbers :", dn.quantile([0, .01, .5, .99, 1]).round(0).to_dict())
95
+ print("Red_1 as reflectance :", sr.quantile([0, .01, .5, .99, 1]).round(3).to_dict())
96
+ for s in (1, 2, 3):
97
+ col = f"qa_valid_{s}"
98
+ if col in peek.columns:
99
+ print(f"{col}: {peek[col].mean():.1%} valid")
100
+ del peek
101
+
102
+ # %% [markdown]
103
+ # ## 1.3 Build both feature sets
104
+ #
105
+ # The heavy lifting lives in `src/features.py`. Per city it:
106
+ #
107
+ # * picks the first QA-valid observation per band per year, drops haze
108
+ # (`Blue > 15000`);
109
+ # * records `has_early_data` / `has_late_data` on the *observed* years;
110
+ # * fills the annual series (linear inside, nearest at the edges);
111
+ # * pivots to one `(pixels x years)` matrix per band, then computes every
112
+ # statistic with numpy - a whole city in about a minute.
113
+
114
+ # %%
115
+ madrid = F.build_city(MADRID_PARQUET, city_hint="Madrid")
116
+
117
+ # %%
118
+ amsterdam = F.build_city(AMS_PARQUET, city_hint="Amsterdam")
119
+
120
+ # %% [markdown]
121
+ # ## 1.4 Does this match the organisers' own output?
122
+ #
123
+ # If `preprocessed_data.pkl` from the organisers' Notebook 3 is around, compare
124
+ # the 60 official features against it. Their arrays are standardised with
125
+ # Madrid's `StandardScaler`, so we compare **correlation per feature**, which is
126
+ # invariant to that. Anything below 0.999 means our pipeline has drifted.
127
+
128
+ # %%
129
+ pkl = None
130
+ for root in [C.DATA_DIR, C.PROJECT_ROOT, Path("/kaggle/input")]:
131
+ if root.exists():
132
+ hits = sorted(root.rglob("preprocessed_data.pkl"))
133
+ if hits:
134
+ pkl = hits[0]
135
+ break
136
+
137
+ if pkl is None:
138
+ print("preprocessed_data.pkl not found - skipping the cross-check.")
139
+ else:
140
+ import pickle
141
+ with open(pkl, "rb") as fh:
142
+ ref = pickle.load(fh)
143
+ print("reference file:", pkl)
144
+ ours = madrid.set_index(["px_key", "py_key"])
145
+ ref_ids = pd.MultiIndex.from_arrays(
146
+ [ref["pixel_ids_madrid"][:, 0], ref["pixel_ids_madrid"][:, 1]],
147
+ names=["px_key", "py_key"])
148
+ common = ref_ids.intersection(ours.index)
149
+ print(f"pixels: ours {len(ours):,}, theirs {len(ref_ids):,}, "
150
+ f"in common {len(common):,}")
151
+ ref_df = pd.DataFrame(ref["X_madrid"], index=ref_ids,
152
+ columns=ref["feature_names"]).loc[common]
153
+ our_df = ours.loc[common, ref["feature_names"]]
154
+ corr = {c: float(np.corrcoef(ref_df[c], our_df[c])[0, 1])
155
+ for c in ref["feature_names"]
156
+ if ref_df[c].std() > 0 and our_df[c].std() > 0}
157
+ bad = {k: v for k, v in corr.items() if v < 0.999}
158
+ print(f"features compared: {len(corr)} | min correlation: "
159
+ f"{min(corr.values()):.5f}")
160
+ print("dead features in their file (zero variance):",
161
+ [c for c in ref["feature_names"] if ref_df[c].std() == 0])
162
+ print("features below 0.999:", bad if bad else "none - pipeline matches")
163
+ # Labels must agree exactly.
164
+ y_ref = pd.Series(ref["y_madrid"], index=ref_ids).loc[common]
165
+ print("label agreement:",
166
+ f"{(y_ref.to_numpy() == our_df.index.map(ours['age_class']).to_numpy()).mean():.4%}")
167
+
168
+ # %% [markdown]
169
+ # ## 1.5 Class balance, and the domain gap
170
+ #
171
+ # Two facts drive every later design decision:
172
+ #
173
+ # * the classes are unbalanced, differently in each city;
174
+ # * the same feature has a different *level* in each city, by much more than
175
+ # the gap between classes - which is why a Madrid model collapses to ~0.34 on
176
+ # Amsterdam.
177
+
178
+ # %%
179
+ for name, df in [("Madrid", madrid), ("Amsterdam", amsterdam)]:
180
+ n = len(df)
181
+ counts = df["age_class"].value_counts().sort_index()
182
+ print(f"\n{name}: {n:,} labelled pixels")
183
+ for c in C.CLASS_LABELS[name]:
184
+ k = int(counts.get(c, 0))
185
+ print(f" class {c} ({C.CLASS_LABELS[name][c]:>11}): {k:6,} {k / n:6.1%}")
186
+
187
+ # %%
188
+ # How far apart are the two cities, feature by feature?
189
+ # Expressed in Madrid standard deviations, so it is directly comparable.
190
+ key = ["sr_brightness_mean", "sr_Red_mean", "sr_NIR_mean", "sr_SWIR1_mean",
191
+ "sr_NDVI_mean", "sr_NDBI_mean", "sr_UI_mean", "sr_MNDWI_mean",
192
+ "sr_BSI_mean", "sr_brightness_std", "sr_brightness_late_minus_early",
193
+ "step_brightness_mag", "step_brightness_njumps", "coverage"]
194
+ key = [k for k in key if k in madrid.columns and k in amsterdam.columns]
195
+ gap = pd.DataFrame({
196
+ "madrid_mean": [madrid[k].mean() for k in key],
197
+ "amsterdam_mean": [amsterdam[k].mean() for k in key],
198
+ }, index=key)
199
+ gap["madrid_sd"] = [madrid[k].std() for k in key]
200
+ gap["gap_in_madrid_sd"] = (gap.amsterdam_mean - gap.madrid_mean) / gap.madrid_sd
201
+ print(gap.round(3).sort_values("gap_in_madrid_sd").to_string())
202
+
203
+ # %% [markdown]
204
+ # A feature whose `gap_in_madrid_sd` is near zero is one a Madrid-trained model
205
+ # can still read in Amsterdam. A feature at +/-2 is telling the model which
206
+ # *city* it is looking at, not how old the building is.
207
+
208
+ # %%
209
+ # Within-class spread versus between-city gap, for the top offenders.
210
+ fig, axes = plt.subplots(2, 4, figsize=(18, 7))
211
+ for ax, k in zip(axes.ravel(), key[:8]):
212
+ for name, df, col in [("Madrid", madrid, "tab:red"),
213
+ ("Amsterdam", amsterdam, "tab:blue")]:
214
+ ax.hist(df[k], bins=60, density=True, alpha=0.45, label=name, color=col)
215
+ ax.set_title(k, fontsize=9)
216
+ ax.tick_params(labelsize=7)
217
+ axes.ravel()[0].legend(fontsize=8)
218
+ plt.suptitle("The same feature in the two cities", fontweight="bold")
219
+ plt.tight_layout()
220
+ plt.show()
221
+
222
+ # %% [markdown]
223
+ # ## 1.6 Does the construction-event detector actually work?
224
+ #
225
+ # A genuine test of the extended features. For classes 3 and 4 the building went
226
+ # up *during* the satellite record, so `step_brightness_year` should land near
227
+ # `weighted_mean_year`. For classes 1 and 2 it was already standing in 1984, so
228
+ # the detector should find nothing meaningful. If that pattern does not show up,
229
+ # the detector is noise and should be dropped.
230
+
231
+ # %%
232
+ for name, df in [("Madrid", madrid), ("Amsterdam", amsterdam)]:
233
+ print(f"\n{name}")
234
+ for sig in ("brightness", "SWIR1", "NDVI"):
235
+ sub = df[df.age_class.isin([3, 4])]
236
+ err = (sub[f"step_{sig}_year"] - sub["weighted_mean_year"]).abs()
237
+ r = np.corrcoef(sub[f"step_{sig}_year"], sub["weighted_mean_year"])[0, 1]
238
+ print(f" step_{sig}_year vs true year (classes 3+4): r={r:+.3f} "
239
+ f"median |error| = {err.median():.1f} years "
240
+ f"within 5 y: {(err <= 5).mean():.1%}")
241
+ print(" mean |step magnitude| by class:",
242
+ df.groupby("age_class")["step_brightness_mag"].apply(
243
+ lambda s: round(float(s.abs().mean()), 4)).to_dict())
244
+
245
+ # %%
246
+ fig, axes = plt.subplots(1, 2, figsize=(13, 4.5))
247
+ for ax, (name, df) in zip(axes, [("Madrid", madrid), ("Amsterdam", amsterdam)]):
248
+ sub = df[df.age_class.isin([3, 4])].sample(min(4000, len(df)), random_state=0)
249
+ ax.scatter(sub["weighted_mean_year"], sub["step_brightness_year"], s=3,
250
+ alpha=0.25)
251
+ lims = [1984, 2024]
252
+ ax.plot(lims, lims, "r--", lw=1)
253
+ ax.set(xlabel="true construction year", ylabel="detected brightness step year",
254
+ title=f"{name}: classes 3 and 4", xlim=lims, ylim=lims)
255
+ plt.tight_layout()
256
+ plt.show()
257
+
258
+ # %% [markdown]
259
+ # ## 1.7 Neighbourhood context - free information, no labels
260
+ #
261
+ # Buildings go up a street at a time, so a pixel's neighbours carry real signal
262
+ # about its era. These features average a few key quantities over the 8 nearest
263
+ # pixels (within 10 pixel units = 300 m) and also store the *difference* between
264
+ # the pixel and its surroundings. No labels are involved, so this is available
265
+ # in Amsterdam at n = 0.
266
+
267
+ # %%
268
+ madrid = R.add_neighbourhood_features(madrid)
269
+ amsterdam = R.add_neighbourhood_features(amsterdam)
270
+ nbr_cols = [c for c in madrid.columns if c.startswith("nbr_")]
271
+ print(f"added {len(nbr_cols)} neighbourhood features")
272
+ print(madrid[["age_class"] + nbr_cols[:6]].groupby("age_class").mean().round(4).to_string())
273
+
274
+ # %% [markdown]
275
+ # ## 1.8 Save
276
+ #
277
+ # One parquet per city. Every later notebook starts from these.
278
+
279
+ # %%
280
+ C.PROC_DIR.mkdir(parents=True, exist_ok=True)
281
+ madrid.to_parquet(C.PROC_DIR / "madrid_features.parquet", index=False)
282
+ amsterdam.to_parquet(C.PROC_DIR / "amsterdam_features.parquet", index=False)
283
+ for n, df in [("madrid", madrid), ("amsterdam", amsterdam)]:
284
+ p = C.PROC_DIR / f"{n}_features.parquet"
285
+ print(f"{p} {p.stat().st_size / 1e6:.1f} MB {df.shape}")
286
+ print("\nfeature groups:",
287
+ f"{len(C.FEATURE_COLS_60)} official,",
288
+ f"{sum(1 for c in madrid.columns if c.startswith('sr_'))} reflectance-based,",
289
+ f"{sum(1 for c in madrid.columns if c.startswith('step_'))} change-event,",
290
+ f"{len(nbr_cols)} neighbourhood")
notebooks/src/03_gbdt_models.py ADDED
@@ -0,0 +1,574 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # %% [markdown]
2
+ # # 3 - LightGBM and CatBoost: the two classical models
3
+ #
4
+ # The organisers ask for at least two models, chosen on Madrid. These are them.
5
+ # They are also the honest yardstick the fine-tuned LLM has to beat, and the
6
+ # source of the probabilities we later ensemble with it.
7
+ #
8
+ # Running order:
9
+ #
10
+ # 1. **Spatial** cross-validation on Madrid - random folds are dishonest here,
11
+ # because neighbouring pixels share a construction era.
12
+ # 2. **Adversarial validation** - find the features that only say "this is
13
+ # Madrid" and drop them.
14
+ # 3. **Per-city normalisation** - the single biggest transfer win, and it needs
15
+ # no labels.
16
+ # 4. **LightGBM vs CatBoost vs ordinal LightGBM**, all judged on Madrid only.
17
+ # 5. **The official few-shot protocol** at n = 5, 25, 50, 100, 200.
18
+ #
19
+ # Runs on CPU. About 20 minutes on a laptop.
20
+
21
+ # %%
22
+ import sys
23
+ from pathlib import Path
24
+
25
+ for cand in [Path.cwd() / "src", Path.cwd().parent / "src",
26
+ Path("/kaggle/working/src"), Path("/content/hack4dev/src")]:
27
+ if cand.exists():
28
+ sys.path.insert(0, str(cand))
29
+ break
30
+
31
+ import numpy as np
32
+ import pandas as pd
33
+ import matplotlib.pyplot as plt
34
+
35
+ import config as C
36
+ import evaluation as E
37
+ import gbdt as G
38
+ import retrieval as R
39
+
40
+ RNG = np.random.default_rng(C.RANDOM_SEED)
41
+
42
+ # Prefer the full feature tables from notebook 1. Fall back to the organisers'
43
+ # preprocessed_data.pkl, which carries the official 60 features and the pixel
44
+ # coordinates - enough for everything in THIS notebook, and only ~50 MB, so it
45
+ # is the right thing to grab first if the raw parquet is hard to get hold of.
46
+ import features as F
47
+
48
+ if (C.PROC_DIR / "madrid_features.parquet").exists():
49
+ madrid = pd.read_parquet(C.PROC_DIR / "madrid_features.parquet")
50
+ amsterdam = pd.read_parquet(C.PROC_DIR / "amsterdam_features.parquet")
51
+ SOURCE = "notebook 1 (official 60 + extended)"
52
+ else:
53
+ pkl = None
54
+ for root in [C.DATA_DIR, C.PROJECT_ROOT, Path("/kaggle/input")]:
55
+ if root.exists():
56
+ hits = sorted(root.rglob("preprocessed_data.pkl"))
57
+ if hits:
58
+ pkl = hits[0]
59
+ break
60
+ if pkl is None:
61
+ raise FileNotFoundError(
62
+ "Neither data/processed/madrid_features.parquet (run notebook 1) "
63
+ "nor preprocessed_data.pkl were found.")
64
+ madrid, amsterdam = F.from_official_pkl(pkl)
65
+ SOURCE = "organisers' preprocessed_data.pkl (official 60 only)"
66
+
67
+ print(f"\nfeature source: {SOURCE}")
68
+ print(madrid.shape, amsterdam.shape)
69
+
70
+ # %%
71
+ # Feature sets we will compare. Keeping them explicit makes the ablation table
72
+ # in the report trivial to produce.
73
+ DROP = {"px_key", "py_key", "age_class", "weighted_mean_year", "city"}
74
+ ALL_FEATS = [c for c in madrid.columns
75
+ if c not in DROP and madrid[c].dtype.kind in "fiu"]
76
+ DEAD = [c for c in ALL_FEATS if madrid[c].std() == 0 or amsterdam[c].std() == 0]
77
+ ALL_FEATS = [c for c in ALL_FEATS if c not in DEAD]
78
+
79
+ FEATURE_SETS = {
80
+ "official60": [c for c in C.FEATURE_COLS_60 if c in ALL_FEATS],
81
+ "reflectance": [c for c in ALL_FEATS if c.startswith("sr_")],
82
+ "events": [c for c in ALL_FEATS if c.startswith("step_")],
83
+ "neighbourhood": [c for c in ALL_FEATS if c.startswith("nbr_")],
84
+ "everything": ALL_FEATS,
85
+ }
86
+ print("dead (zero variance in at least one city):", DEAD)
87
+ for k, v in FEATURE_SETS.items():
88
+ print(f" {k:<14} {len(v)} features")
89
+
90
+ y_mad = madrid["age_class"].to_numpy()
91
+ y_ams = amsterdam["age_class"].to_numpy() # opened only for scoring
92
+ coords_mad = madrid[C.PID].to_numpy(float)
93
+ coords_ams = amsterdam[C.PID].to_numpy(float)
94
+
95
+ # %% [markdown]
96
+ # ## 3.1 An honest validation split
97
+ #
98
+ # `RepeatedStratifiedKFold` on pixels puts a pixel's own neighbours in the
99
+ # training fold. Neighbours share an era, so the model can almost look up the
100
+ # answer, and Madrid CV comes out flattering - part of why 0.63 on Madrid turns
101
+ # into 0.34 on Amsterdam.
102
+ #
103
+ # Instead: cut Madrid into geographic blocks and hold out whole blocks. That is
104
+ # our stand-in for "a city the model has never seen", and it is the only thing
105
+ # we are allowed to choose models on.
106
+
107
+ # %%
108
+ blocks_mad = R.spatial_blocks(coords_mad, n_blocks=25)
109
+ print("block sizes:", np.bincount(blocks_mad))
110
+
111
+ fig, ax = plt.subplots(figsize=(7, 5))
112
+ ax.scatter(coords_mad[:, 0], coords_mad[:, 1], c=blocks_mad, s=1, cmap="tab20")
113
+ ax.set(title="Madrid split into spatial blocks", xlabel="px_key", ylabel="py_key")
114
+ plt.tight_layout()
115
+ plt.show()
116
+
117
+
118
+ def block_cv(X, y, blocks, fit_fn, n_folds=5, normalise="robustz", seed=0):
119
+ """Hold out whole geographic blocks, score macro F1 on each."""
120
+ uniq = np.unique(blocks)
121
+ rng = np.random.default_rng(seed)
122
+ order = rng.permutation(uniq)
123
+ folds = np.array_split(order, n_folds)
124
+ scores, oof = [], np.zeros((len(y), 4))
125
+ for f in folds:
126
+ te = np.isin(blocks, f)
127
+ tr = ~te
128
+ nrm = G.CityNormaliser(normalise).fit(X[tr]) if normalise else None
129
+ Xtr = nrm.transform(X[tr]) if nrm else X[tr]
130
+ # The held-out block plays the part of a new city: it gets normalised
131
+ # with its OWN statistics, exactly as Amsterdam will be.
132
+ nte = G.CityNormaliser(normalise).fit(X[te]) if normalise else None
133
+ Xte = nte.transform(X[te]) if nte else X[te]
134
+ m = fit_fn(Xtr, y[tr])
135
+ p = m.predict_proba(Xte)
136
+ oof[te] = p
137
+ scores.append(E.macro_f1(y[te], E.CLASSES[np.argmax(p, 1)]))
138
+ return np.array(scores), oof
139
+
140
+ # %% [markdown]
141
+ # ## 3.2 Which features betray the city?
142
+ #
143
+ # Train a classifier to separate Madrid pixels from Amsterdam pixels. If it
144
+ # succeeds easily, the features carry a strong city signature. The most
145
+ # important features in *that* task are the ones most likely to mislead the age
146
+ # model, and are candidates for removal.
147
+
148
+ # %%
149
+ auc, imp = G.adversarial_validation(madrid[ALL_FEATS], amsterdam[ALL_FEATS], ALL_FEATS)
150
+ print(f"city-vs-city AUC: {auc:.4f} (1.0 = the two cities are trivially told apart)")
151
+ print(imp.head(20).to_string(index=False))
152
+
153
+ # %% [markdown]
154
+ # The joint model tells us the cities are separable; it does not tell us which
155
+ # feature is to blame, because importance is shared among correlated features.
156
+ # A single-feature AUC does, and the before/after table is the most convincing
157
+ # figure in the write-up: 0.5 means a feature is domain-invariant, 1.0 means it
158
+ # is a pure city marker and any Madrid split on it is meaningless in Amsterdam.
159
+
160
+ # %%
161
+ Xm_all = madrid[ALL_FEATS].to_numpy(float)
162
+ Xa_all = amsterdam[ALL_FEATS].to_numpy(float)
163
+ sf_raw = G.single_feature_domain_auc(Xm_all, Xa_all, ALL_FEATS)
164
+
165
+ # The rankable mask is a STRUCTURAL decision about the features, so it is
166
+ # computed once on Madrid - the design city - and shared. Fitting it per city
167
+ # lets a column be ranked in one and z-scored in the other, which puts them on
168
+ # different scales and reports a domain AUC of 1.000 for a feature that is
169
+ # actually domain-invariant. `n_observed_years` did exactly that.
170
+ RANKABLE = G.rankable_mask(Xm_all)
171
+ print(f"rank-transformable on Madrid: {RANKABLE.sum()} of {len(ALL_FEATS)}; "
172
+ f"too heavily tied to rank: {[ALL_FEATS[j] for j in np.flatnonzero(~RANKABLE)]}")
173
+ nm_ = G.CityNormaliser("rank", rankable=RANKABLE).fit(Xm_all).transform(Xm_all)
174
+ na_ = G.CityNormaliser("rank", rankable=RANKABLE).fit(Xa_all).transform(Xa_all)
175
+ sf_rank = G.single_feature_domain_auc(nm_, na_, ALL_FEATS)
176
+ screen = (sf_raw.rename(columns={"domain_auc": "auc_raw"})
177
+ .merge(sf_rank.rename(columns={"domain_auc": "auc_after_rank"}), on="feature")
178
+ .sort_values("auc_raw", ascending=False))
179
+ print("worst offenders, raw vs after per-city rank normalisation:")
180
+ print(screen.head(25).round(3).to_string(index=False))
181
+ print(f"\nmedian domain AUC: raw {screen.auc_raw.median():.3f} -> "
182
+ f"after rank {screen.auc_after_rank.median():.3f}")
183
+ print(f"features still above 0.90 after normalisation: "
184
+ f"{(screen.auc_after_rank > 0.90).sum()} of {len(screen)}")
185
+ screen.to_csv(C.REPORT_DIR / "domain_auc_screen.csv", index=False)
186
+
187
+ fig, ax = plt.subplots(figsize=(7, 5))
188
+ ax.scatter(screen.auc_raw, screen.auc_after_rank, s=8, alpha=0.5)
189
+ ax.plot([0.5, 1], [0.5, 1], "k--", lw=1)
190
+ ax.set(xlabel="domain AUC, raw", ylabel="domain AUC, after per-city rank",
191
+ title="Per-city ranking removes the city signature (points below the line)")
192
+ plt.tight_layout()
193
+ plt.show()
194
+
195
+ # %%
196
+ # Repeat after per-city normalisation: a good normaliser should make the cities
197
+ # harder to tell apart without destroying the age signal.
198
+ nm = G.CityNormaliser("robustz").fit(Xm_all).transform(Xm_all)
199
+ na = G.CityNormaliser("robustz").fit(Xa_all).transform(Xa_all)
200
+ auc_z, imp_z = G.adversarial_validation(nm, na, ALL_FEATS)
201
+ auc_r, imp_r = G.adversarial_validation(nm_, na_, ALL_FEATS) # shared mask
202
+ print(f"AUC raw {auc:.4f} | robust-z {auc_z:.4f} | rank {auc_r:.4f}")
203
+ print("\nstill city-specific after rank normalisation:")
204
+ print(imp_r.head(12).to_string(index=False))
205
+
206
+ # %% [markdown]
207
+ # Read the two screens together, because they say different things and the
208
+ # difference is the point.
209
+ #
210
+ # Per-city **rank** normalisation forces every feature's marginal distribution
211
+ # to be uniform in both cities, so the *single-feature* domain AUC becomes
212
+ # exactly 0.5 by construction. That is not evidence that the domain gap is
213
+ # gone. If the *joint* adversarial AUC above is still near 1.0 after ranking,
214
+ # the city signature has simply moved into the **correlations between**
215
+ # features, where a marginal transform cannot reach it.
216
+ #
217
+ # That is precisely what CORAL addresses - it matches second-order statistics
218
+ # rather than marginals. So the honest conclusion from these two cells is:
219
+ # ranking is necessary and cheap, and it is not sufficient. Whether CORAL on
220
+ # top actually helps is an empirical question answered in section 3.5, not a
221
+ # foregone conclusion: in synthetic testing CORAL slightly *hurt* when the
222
+ # shift was close to diagonal.
223
+
224
+ # %% [markdown]
225
+ # ### The drop rule, and why it must be conservative
226
+ #
227
+ # The tempting rule is "drop the top features by adversarial importance". It is
228
+ # wrong. With 59 features the average importance share is 1/59 = 0.017, so any
229
+ # threshold near that drops whatever happens to be slightly above average -
230
+ # and once normalisation has pushed the joint AUC down to ~0.5, those
231
+ # importances are *noise*. In an earlier run this rule dropped three features
232
+ # whose single-feature domain AUC was exactly 0.500, i.e. perfectly
233
+ # domain-invariant ones.
234
+ #
235
+ # The principled rule needs two conditions, both of them absolute rather than
236
+ # relative: a feature is dropped only if it is **still** strongly
237
+ # city-discriminative on its own after normalisation, **and** it is not
238
+ # carrying much of the age signal on Madrid. Usually that drops nothing, which
239
+ # is the correct answer.
240
+
241
+ # %%
242
+ AUC_CUT = 0.90 # still a city marker after normalisation
243
+ IMP_CUT = 0.005 # and contributes little to the Madrid model
244
+
245
+ X_all = madrid[ALL_FEATS].to_numpy(float)
246
+ probe = G.fit_lgbm(G.CityNormaliser("rank").fit_transform(X_all), y_mad,
247
+ params=dict(n_estimators=300))
248
+ age_imp = pd.DataFrame({
249
+ "feature": ALL_FEATS,
250
+ "age_importance": probe.feature_importances_ / max(probe.feature_importances_.sum(), 1)
251
+ })
252
+ drop_tbl = screen.merge(age_imp, on="feature")
253
+ leaky = drop_tbl[(drop_tbl.auc_after_rank > AUC_CUT)
254
+ & (drop_tbl.age_importance < IMP_CUT)].feature.tolist()
255
+ FEATURE_SETS["pruned"] = [c for c in ALL_FEATS if c not in leaky]
256
+ print(f"features still above AUC {AUC_CUT} after normalisation: "
257
+ f"{(drop_tbl.auc_after_rank > AUC_CUT).sum()}")
258
+ print(f"dropping {len(leaky)}: {leaky if leaky else 'nothing - as expected'}")
259
+ print(f"pruned set: {len(FEATURE_SETS['pruned'])} features")
260
+ print("\nmost city-specific features that we KEPT because they carry age signal:")
261
+ print(drop_tbl.sort_values("auc_after_rank", ascending=False)
262
+ .head(8).round(4).to_string(index=False))
263
+
264
+ # %% [markdown]
265
+ # ## 3.3 Model and feature-set comparison, on Madrid only
266
+ #
267
+ # Every row of this table is decided with Madrid block-CV. Amsterdam is not
268
+ # consulted. This is the table that justifies the final model choice.
269
+
270
+ # %%
271
+ # The comparison runs 4 models x 3 feature sets x 5 blocked folds = 60 fits.
272
+ # At full strength (1200 LightGBM rounds, 2000 CatBoost iterations) that is
273
+ # tens of minutes on 76k rows, and it buys nothing: we are reading the
274
+ # ORDERING of configurations here, not their final scores. So compare cheaply,
275
+ # then refit the winner at full strength in section 3.7.
276
+ CHEAP_LGB = dict(n_estimators=400, learning_rate=0.06)
277
+ CHEAP_CAT = dict(iterations=500, learning_rate=0.06)
278
+
279
+
280
+ def lgbm_balanced(X, y):
281
+ return G.fit_lgbm(X, y, params=CHEAP_LGB,
282
+ class_weight=dict(zip(range(4), G.balanced_class_weights(y))))
283
+
284
+ def lgbm_plain(X, y):
285
+ return G.fit_lgbm(X, y, params=CHEAP_LGB)
286
+
287
+ def cat_balanced(X, y):
288
+ return G.fit_catboost(X, y, params=CHEAP_CAT,
289
+ class_weights=G.balanced_class_weights(y))
290
+
291
+ def ordinal_lgbm(X, y):
292
+ return G.OrdinalGBDT(params=CHEAP_LGB).fit(X, y)
293
+
294
+ import time
295
+
296
+ FEATURE_SETS_TO_TRY = [k for k in ("official60", "everything", "pruned")
297
+ if len(FEATURE_SETS.get(k, []))]
298
+ # Drop a duplicate: with the pkl-only fallback, "everything" == "official60".
299
+ seen, uniq = set(), []
300
+ for k in FEATURE_SETS_TO_TRY:
301
+ sig = tuple(FEATURE_SETS[k])
302
+ if sig not in seen:
303
+ seen.add(sig)
304
+ uniq.append(k)
305
+ else:
306
+ print(f" (skipping '{k}' - identical to an earlier set)")
307
+ FEATURE_SETS_TO_TRY = uniq
308
+
309
+ rows = []
310
+ for fs_name in FEATURE_SETS_TO_TRY:
311
+ X = madrid[FEATURE_SETS[fs_name]].to_numpy(float)
312
+ models = [("lgbm", lgbm_balanced), ("lgbm_unweighted", lgbm_plain),
313
+ ("lgbm_ordinal", ordinal_lgbm)]
314
+ if fs_name == FEATURE_SETS_TO_TRY[0]:
315
+ # CatBoost costs 35 min per feature set against LightGBM's 10 and came
316
+ # in about 0.008 behind on every set, so it runs on one set only. It
317
+ # still ships as the second required model.
318
+ models.append(("catboost", cat_balanced))
319
+ for mdl_name, fn in models:
320
+ t0 = time.time()
321
+ s, oof = block_cv(X, y_mad, blocks_mad, fn)
322
+ rows.append(dict(features=fs_name, model=mdl_name, n_feat=X.shape[1],
323
+ macro_f1=s.mean(), std=s.std(ddof=1),
324
+ minutes=(time.time() - t0) / 60))
325
+ print(f" {fs_name:<12} {mdl_name:<16} {s.mean():.4f} +- "
326
+ f"{s.std(ddof=1):.4f} ({rows[-1]['minutes']:.1f} min)")
327
+ table = pd.DataFrame(rows).sort_values("macro_f1", ascending=False)
328
+ print("\n" + table.round(4).to_string(index=False))
329
+
330
+ # %% [markdown]
331
+ # ### Does per-city normalisation actually help? Decide it on Madrid.
332
+ #
333
+ # This has to be settled here and not on the Amsterdam score, because the
334
+ # Amsterdam score is not ours to choose on. `block_cv` is the right instrument:
335
+ # it normalises the held-out block with the block's OWN statistics, exactly as
336
+ # Amsterdam will be normalised, so each fold is a miniature transfer problem.
337
+ #
338
+ # Worth doing carefully, because the intuition ("absolute reflectance does not
339
+ # transfer, so normalise it away") is strong and may still be wrong. Per-city
340
+ # normalisation also destroys genuine between-city differences in LEVEL, and if
341
+ # a city's overall darkness happens to correlate with its class mix, removing
342
+ # the level removes a real cue along with the nuisance.
343
+
344
+ # %%
345
+ norm_rows = []
346
+ X_best = madrid[FEATURE_SETS[table.iloc[0].features]].to_numpy(float)
347
+ for norm in (None, "robustz", "rank"):
348
+ s, _ = block_cv(X_best, y_mad, blocks_mad, lgbm_plain, normalise=norm)
349
+ norm_rows.append(dict(normalise=str(norm), macro_f1=s.mean(),
350
+ std=s.std(ddof=1)))
351
+ print(f" normalise={str(norm):8} {s.mean():.4f} +- {s.std(ddof=1):.4f}")
352
+ norm_table = pd.DataFrame(norm_rows).sort_values("macro_f1", ascending=False)
353
+ print("\n" + norm_table.round(4).to_string(index=False))
354
+ BEST_NORM = None if norm_table.iloc[0].normalise == "None" else norm_table.iloc[0].normalise
355
+ print(f"\nnormalisation chosen on Madrid: {BEST_NORM}")
356
+ print("Whatever Amsterdam says later, this is the choice the rules allow us to")
357
+ print("make. If Amsterdam disagrees, that is a finding to report, not a reason")
358
+ print("to switch.")
359
+
360
+ # %% [markdown]
361
+ # ## 3.4 The decision rule matters as much as the model
362
+ #
363
+ # Macro F1 is not maximised by `argmax`. Three cheap post-processing steps,
364
+ # each measured separately on Madrid block-CV:
365
+ #
366
+ # * **class weights** tuned by coordinate ascent on out-of-fold probabilities;
367
+ # * **prior correction** towards the class prior EM estimates from *unlabelled*
368
+ # target features;
369
+ # * **spatial smoothing** of the probabilities over neighbouring pixels.
370
+
371
+ # %%
372
+ BEST_FS = table.iloc[0].features
373
+ X_mad = madrid[FEATURE_SETS[BEST_FS]].to_numpy(float)
374
+ X_ams = amsterdam[FEATURE_SETS[BEST_FS]].to_numpy(float)
375
+ scores, oof = block_cv(X_mad, y_mad, blocks_mad, lgbm_balanced)
376
+ print(f"baseline argmax, Madrid block-CV: {E.macro_f1(y_mad, E.CLASSES[np.argmax(oof, 1)]):.4f}")
377
+
378
+ w = E.tune_class_weights(oof, y_mad)
379
+ print("tuned class weights:", np.round(w, 3))
380
+ print("after weighting: ",
381
+ f"{E.macro_f1(y_mad, E.predict_weighted(oof, w)):.4f}")
382
+
383
+ for radius in (2.0, 4.0, 8.0):
384
+ sm = E.spatial_smooth(oof, coords_mad, radius=radius, alpha=0.5)
385
+ print(f"spatial smoothing r={radius:>4}: "
386
+ f"{E.macro_f1(y_mad, E.predict_weighted(sm, w)):.4f}")
387
+
388
+ # %%
389
+ print(pd.DataFrame(E.confusion_proportions(y_mad, E.predict_weighted(oof, w)),
390
+ index=[f"true {c}" for c in E.CLASSES],
391
+ columns=[f"pred {c}" for c in E.CLASSES]).round(3).to_string())
392
+
393
+ # %% [markdown]
394
+ # ## 3.5 The simulated-shift harness - choosing the decision rule on Madrid
395
+ #
396
+ # This is the most important cell in the notebook, and the thing I would most
397
+ # want the team to copy.
398
+ #
399
+ # None of the adaptation tricks above can be chosen honestly without a way to
400
+ # *simulate* Amsterdam from Madrid. So: hold out a spatial block, and build four
401
+ # versions of it.
402
+ #
403
+ # * `clean` - as-is;
404
+ # * `label` - resampled to a random Dirichlet class prior (pure label shift);
405
+ # * `covariate` - every feature quantile-mapped onto Amsterdam's own marginal
406
+ # (pure covariate shift, using `X_amsterdam` only, no labels);
407
+ # * `both`.
408
+ #
409
+ # Then score every candidate decision rule on all four. A rule that helps under
410
+ # label shift can lose badly under covariate shift, and without this there is no
411
+ # way to tell which regime you are actually in. **Ship only rules that are
412
+ # non-negative in the worst case.**
413
+ #
414
+ # In synthetic testing this harness caught a trap: free SLD-EM prior estimation
415
+ # looked excellent under label shift and lost 0.19 macro F1 on average (0.35 at
416
+ # worst) under covariate shift, because EM assumes label shift only. The
417
+ # macro-F1 plug-in rule driven by the *source* prior was the robust winner.
418
+
419
+ # %%
420
+ # Note the UNWEIGHTED model here. The whole point of this cell is to choose the
421
+ # operating point, so the probabilities going into it must be an honest estimate
422
+ # of the Madrid posterior. class_weight='balanced' bakes a decision into the fit,
423
+ # where no decision rule can undo it.
424
+ harness_df, harness_table = E.shift_harness(
425
+ lgbm_plain, X_mad, y_mad, blocks_mad, X_ams,
426
+ E.DEFAULT_STRATEGIES, n_folds=5, n_rep=4, seed=C.RANDOM_SEED)
427
+ harness_table.to_csv(C.REPORT_DIR / "shift_harness.csv")
428
+
429
+ # %%
430
+ # The criterion is the PER-REGIME worst case: for each rule, its deficit
431
+ # against argmax in whichever regime it handles worst. An earlier version used
432
+ # a column that compared each rule's worst cell anywhere against argmax's worst
433
+ # cell anywhere - two different regimes and two different folds - so a rule
434
+ # could look safe merely because its bad regime was also argmax's bad regime.
435
+ WORST = "delta_vs_argmax_worst_regime"
436
+ safe = harness_table[harness_table.get(WORST, 0) >= 0]
437
+ print("rules that never lose to plain argmax in ANY regime:")
438
+ print(safe.round(4).to_string() if len(safe) else " none - keep plain argmax")
439
+ print("\nrules to avoid:")
440
+ risky = harness_table[harness_table.get(WORST, 0) < -0.01]
441
+ print(risky.round(4).to_string() if len(risky) else " none")
442
+
443
+ CHOSEN_RULE = (harness_table[WORST].idxmax()
444
+ if WORST in harness_table else "argmax")
445
+ print(f"\nrule chosen on Madrid alone: {CHOSEN_RULE}")
446
+ DECISION = E.DEFAULT_STRATEGIES[CHOSEN_RULE]
447
+
448
+ # %% [markdown]
449
+ # ## 3.6 The official few-shot protocol
450
+ #
451
+ # `y_amsterdam` enters here and only here: to draw the n labelled pixels per
452
+ # class that the rules allow, and to score the remainder. Nothing above was
453
+ # tuned on it.
454
+
455
+ # %%
456
+ N_LIST = (5, 25, 50, 100, 200)
457
+ N_REPEATS = 12
458
+ # Score a fixed random 8,000-pixel sample of Amsterdam rather than all 25,992.
459
+ # Full scoring cost 19 s per draw - 32 minutes for one variant and hours for the
460
+ # table - and bought nothing: the variants that ignore the labelled sample have
461
+ # a standard deviation of 0.0001 across draws. The subsample is fixed, so every
462
+ # row of the table is compared on identical pixels. Re-run the winner on all
463
+ # 25,992 once, at the end, for the number that goes in the report.
464
+ EVAL_SUBSAMPLE = 8000
465
+ src_prior = np.array([(y_mad == c).mean() for c in E.CLASSES])
466
+ print('Madrid class prior:', np.round(src_prior, 3))
467
+
468
+ # Cost control. Most variants cache the Madrid-trained model across repeats, so
469
+ # 20 draws are cheap. `upweighted shots, retrained` refits from scratch on every
470
+ # draw - 100 full fits on 76k rows - so it gets cheap params and fewer draws,
471
+ # and its std is correspondingly noisier. That is noted in the table.
472
+ REPEATS = {"upweighted shots, retrained": 5}
473
+
474
+ variants = {
475
+ # the two baselines everything must beat
476
+ "organisers' prototype (Euclidean, Madrid scaler)": G.mc_prototype_official(),
477
+ "pure retrieval k-NN (no source model)": G.mc_knn(),
478
+ # zero-shot: does normalisation alone carry us?
479
+ "zero-shot, no normalisation": G.mc_zero_shot(normalise=None),
480
+ "zero-shot, robust z": G.mc_zero_shot(normalise="robustz"),
481
+ "zero-shot, rank": G.mc_zero_shot(normalise="rank"),
482
+ "zero-shot, robust z + CORAL": G.mc_zero_shot(normalise="robustz", use_coral=True),
483
+ # the prototype metric ablation. On synthetic anisotropic data the pooled
484
+ # WITHIN-class Mahalanobis metric beat the organisers' Euclidean prototype
485
+ # by +0.11 macro F1 at n=25 - the largest single gain of anything tried.
486
+ # The total-covariance version is worse because the target city's total
487
+ # covariance is dominated by BETWEEN-class spread, so it shrinks exactly the
488
+ # directions that separate the classes. Confirm the ordering here on the
489
+ # real features before believing it.
490
+ "prototype, robust z + euclidean": G.mc_prototype(metric="euclidean"),
491
+ "prototype, rank + euclidean": G.mc_prototype(normalise="rank", metric="euclidean"),
492
+ "prototype, robust z + total-cov Mahalanobis": G.mc_prototype(metric="total"),
493
+ "prototype, robust z + within-class Mahalanobis": G.mc_prototype(metric="within"),
494
+ # using the labelled sample with the source model
495
+ "recalibrated (stratification-corrected)": G.mc_recalibrated(normalise="robustz"),
496
+ "refit leaf values (structure frozen)": G.mc_refit(),
497
+ "upweighted shots, retrained": G.mc_shot_weighted(
498
+ normalise="robustz", fit=lambda X, y, **kw: G.fit_lgbm(X, y, params=CHEAP_LGB, **kw)),
499
+ "catboost recalibrated": G.mc_recalibrated(fit=G.fit_catboost, normalise="robustz"),
500
+ }
501
+
502
+ protocol = {}
503
+ for name, mc in variants.items():
504
+ print(f"\n{name}")
505
+ reps = REPEATS.get(name, N_REPEATS)
506
+ t0 = time.time()
507
+ protocol[name] = E.evaluate_protocol(
508
+ mc, X_mad, y_mad, X_ams, y_ams, n_list=N_LIST, n_repeats=reps,
509
+ decision=lambda p: DECISION(p, src_prior),
510
+ eval_subsample=EVAL_SUBSAMPLE)
511
+ print(f" ({reps} draws per n, {(time.time() - t0) / 60:.1f} min)")
512
+
513
+ # %%
514
+ summary = pd.DataFrame({name: {n: r["mean"] for n, r in res.items()}
515
+ for name, res in protocol.items()}).T
516
+ summary["average"] = summary.mean(axis=1)
517
+ summary["peak"] = summary[list(N_LIST)].max(axis=1)
518
+ print(summary.round(4).sort_values("average", ascending=False).to_string())
519
+
520
+ # %%
521
+ fig, ax = plt.subplots(figsize=(8, 5))
522
+ for name, res in protocol.items():
523
+ m = [res[n]["mean"] for n in N_LIST]
524
+ s = [res[n]["std"] for n in N_LIST]
525
+ ax.errorbar(np.log2(N_LIST), m, yerr=s, marker="o", capsize=3, label=name)
526
+ ax.axhline(0.618, ls="--", c="k", lw=1, label="published Madrid baseline 0.618")
527
+ ax.set(xlabel="log2(labelled Amsterdam pixels per class)", ylabel="macro F1 on Amsterdam",
528
+ title="Macro F1 against labelling budget")
529
+ ax.legend(fontsize=7)
530
+ plt.tight_layout()
531
+ plt.show()
532
+
533
+ # %% [markdown]
534
+ # ## 3.7 Save the probabilities for the ensemble
535
+ #
536
+ # The LLM notebook loads these and blends them with its own predictions.
537
+
538
+ # %%
539
+ import pickle
540
+
541
+ best_name = summary.sort_values("average", ascending=False).index[0]
542
+ print("best classical variant on the protocol:", best_name)
543
+
544
+ nrm_m = G.CityNormaliser("robustz").fit(X_mad)
545
+ nrm_a = G.CityNormaliser("robustz").fit(X_ams)
546
+ final = G.fit_lgbm(nrm_m.transform(X_mad), y_mad,
547
+ class_weight=dict(zip(range(4), G.balanced_class_weights(y_mad))))
548
+ final_cat = G.fit_catboost(nrm_m.transform(X_mad), y_mad,
549
+ class_weights=G.balanced_class_weights(y_mad))
550
+
551
+ out = {
552
+ "feature_set": BEST_FS,
553
+ "feature_names": FEATURE_SETS[BEST_FS],
554
+ "madrid_oof_proba": oof,
555
+ "madrid_blocks": blocks_mad,
556
+ "class_weights": w,
557
+ "amsterdam_proba_lgbm": final.predict_proba(nrm_a.transform(X_ams)),
558
+ "amsterdam_proba_catboost": final_cat.predict_proba(nrm_a.transform(X_ams)),
559
+ "protocol": {k: {n: dict(mean=r["mean"], std=r["std"],
560
+ per_class=r["per_class"].tolist())
561
+ for n, r in v.items()} for k, v in protocol.items()},
562
+ # Plain lists, not a DataFrame: this file is read on Colab, whose pandas
563
+ # 2.x cannot unpickle a pandas 3 DataFrame with string columns
564
+ # ("StringDtype.__init__() takes from 1 to 2 positional arguments").
565
+ # Rebuild with pd.DataFrame(out["madrid_block_cv_table"]).
566
+ "madrid_block_cv_table": {c: table[c].tolist() for c in table.columns},
567
+ "adversarial_auc": dict(raw=auc, robustz=auc_z, rank=auc_r),
568
+ "dropped_city_specific": leaky,
569
+ }
570
+ with open(C.PROC_DIR / "gbdt_outputs.pkl", "wb") as fh:
571
+ pickle.dump(out, fh)
572
+ final.booster_.save_model(str(C.PROC_DIR / "model_lgbm.txt"))
573
+ final_cat.save_model(str(C.PROC_DIR / "model_catboost.cbm"))
574
+ print("saved:", C.PROC_DIR / "gbdt_outputs.pkl")
notebooks/src/winners_demo.py ADDED
@@ -0,0 +1,171 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # %% [markdown]
2
+ # # The two winners, end to end
3
+ #
4
+ # Building age class (1-4) per 30 m Landsat pixel. Train on Madrid, predict
5
+ # Amsterdam, scored by macro F1 at n = 5, 25, 50, 100, 200 labelled Amsterdam
6
+ # pixels per class.
7
+ #
8
+ # * **Zero-shot LightGBM** - trained on Madrid only, never sees an Amsterdam
9
+ # label. Best model if you may not use Amsterdam at all: **0.593**.
10
+ # * **Retrained LightGBM** - refit on Madrid plus the n labelled Amsterdam pixels
11
+ # per class the protocol gives you, those pixels weighted 30x. Best average
12
+ # (**0.656**) and best peak (**0.720** at n=200) over the whole protocol.
13
+ #
14
+ # Everything here runs on a laptop CPU. The heavy lifting is in `winners.py`;
15
+ # this notebook just calls it. All numbers are on ALL of Amsterdam minus the
16
+ # labelled pixels of each draw.
17
+
18
+ # %%
19
+ import json
20
+ import sys
21
+ from pathlib import Path
22
+
23
+ import numpy as np
24
+ import pandas as pd
25
+
26
+ ROOT = Path.cwd() if (Path.cwd() / "winners.py").exists() else Path.cwd().parent
27
+ sys.path.insert(0, str(ROOT))
28
+ import winners as W
29
+
30
+ d = W.load_data()
31
+ print("Madrid ", d["X_mad"].shape, " Amsterdam", d["X_ams"].shape,
32
+ " features", len(d["feature_names"]))
33
+ print("Madrid class shares ", np.round(d["madrid_prior"], 3))
34
+ print("Amsterdam class shares", np.round([(d["y_ams"] == c).mean() for c in W.CLASSES], 3))
35
+
36
+ # %% [markdown]
37
+ # ## 1. Zero-shot LightGBM: Madrid only
38
+ #
39
+ # 1,200 boosting rounds, class-balanced, raw features (per-city scaling made
40
+ # things *worse* on Madrid's own spatial cross-validation, so none is applied).
41
+ # The final step is the decision rule: tau = 0.5 logit adjustment by the Madrid
42
+ # class prior, then the macro-F1 plug-in rule. It uses no Amsterdam labels, only
43
+ # the predicted class shares of the batch being labelled.
44
+
45
+ # %%
46
+ zs = W.load_zero_shot()
47
+ P = W.predict(zs, d["X_ams"])
48
+ lab = W.decide(P, d["madrid_prior"])
49
+ from sklearn.metrics import confusion_matrix, f1_score
50
+ print(f"macro F1, all {len(lab):,} Amsterdam pixels: {W.macro_f1(d['y_ams'], lab):.4f}")
51
+ print(f" same model, plain argmax instead of the rule: "
52
+ f"{W.macro_f1(d['y_ams'], W.CLASSES[P.argmax(1)]):.4f}")
53
+ print("per-class F1:", np.round(f1_score(d["y_ams"], lab, labels=W.CLASSES, average=None), 3))
54
+ cm = confusion_matrix(d["y_ams"], lab, labels=W.CLASSES, normalize="true")
55
+ print(pd.DataFrame(cm, index=[f"true {c}" for c in W.CLASSES],
56
+ columns=[f"pred {c}" for c in W.CLASSES]).round(3).to_string())
57
+
58
+ # %% [markdown]
59
+ # ## 2. Retrained LightGBM: one draw
60
+ #
61
+ # The protocol hands you n labelled Amsterdam pixels per class. Here n = 25,
62
+ # draw 0 - the exact pixels behind the reported number. A fresh LightGBM
63
+ # (400 rounds) is fit on all 76k Madrid pixels plus these 100, the 100 weighted
64
+ # 30x. Each city is robust-z scaled with its own median and IQR, Amsterdam's
65
+ # taken from its unlabelled features. The 100 pixels are removed before scoring.
66
+
67
+ # %%
68
+ import time
69
+ n, rep = 25, 0
70
+ shot = W.shots_for(d["y_ams"], n, rep)
71
+ keep = np.setdiff1d(np.arange(len(d["y_ams"])), shot)
72
+ print(f"{len(shot)} labelled pixels, classes {np.bincount(d['y_ams'][shot])[1:]}; "
73
+ f"scoring the other {len(keep):,}")
74
+ t0 = time.time()
75
+ model, ams_stats = W.train_retrained(d["X_mad"], d["y_mad"], d["X_ams"],
76
+ d["X_ams"][shot], d["y_ams"][shot])
77
+ P = W.predict_retrained(model, ams_stats, d["X_ams"][keep])
78
+ lab = W.decide(P, d["madrid_prior"])
79
+ print(f"trained in {time.time() - t0:.0f} s; macro F1 {W.macro_f1(d['y_ams'][keep], lab):.4f}")
80
+ print("per-class F1:", np.round(f1_score(d["y_ams"][keep], lab, labels=W.CLASSES, average=None), 3))
81
+
82
+ # %% [markdown]
83
+ # ## 3. The full protocol, as reported
84
+ #
85
+ # Every n, several draws, scored on all of Amsterdam. These come from
86
+ # `results/full_score_*.json` (produced by `tools/full_score.py`); to recompute
87
+ # them run `python winners.py protocol --model retrained --reps 5` (~35 min).
88
+ # The zero-shot and organisers' rows come from an earlier full-city run
89
+ # (`results/logs/03_gbdt_models_fullscore_run.log`, 20 draws per n).
90
+
91
+ # %%
92
+ N = [5, 25, 50, 100, 200]
93
+ R = ROOT / "results"
94
+ rows = {}
95
+ for key, label in [("upweighted", "retrained LightGBM (5 draws per n)"),
96
+ ("prototype", "prototype, within-class Mahalanobis (12 draws)")]:
97
+ s = json.loads((R / f"full_score_{key}.json").read_text())["summary"]["full"]
98
+ rows[label] = {n: s[str(n)]["mean"] for n in N} | {"std@25": s["25"]["std"]}
99
+ rows["zero-shot LightGBM (20 draws)"] = dict(zip(N, [0.5928, 0.5925, 0.5922, 0.5918, 0.5903]))
100
+ rows["organisers' prototype, our re-run on 393 features (20 draws)"] = dict(zip(N, [0.5066, 0.5821, 0.5965, 0.6034, 0.5972]))
101
+ table = pd.DataFrame(rows).T
102
+ table["average"] = table[N].mean(axis=1)
103
+ table["peak"] = table[N].max(axis=1)
104
+ print(table.round(4).to_string())
105
+
106
+ # %%
107
+ import matplotlib.pyplot as plt
108
+
109
+ INK, INK2, MUTED, GRID, SURF = "#0b0b0b", "#52514e", "#898781", "#e1e0d9", "#fcfcfb"
110
+ series = [ # fixed categorical order: blue, orange, aqua
111
+ ("upweighted", "Retrained LightGBM", "#2a78d6"),
112
+ ("prototype", "Prototype (within-class Mahalanobis)", "#eb6834"),
113
+ ]
114
+ fig, ax = plt.subplots(figsize=(8.5, 5), facecolor=SURF)
115
+ ax.set_facecolor(SURF)
116
+ x = np.log2(N)
117
+ for key, name, col in series:
118
+ s = json.loads((R / f"full_score_{key}.json").read_text())["summary"]["full"]
119
+ m = np.array([s[str(n)]["mean"] for n in N])
120
+ e = np.array([s[str(n)]["std"] for n in N])
121
+ ax.fill_between(x, m - e, m + e, color=col, alpha=0.12, lw=0)
122
+ ax.plot(x, m, color=col, lw=2, marker="o", ms=8, mec=SURF, mew=2, label=name)
123
+ dy = 7 if key == "upweighted" else -7 # keep the two end labels apart
124
+ ax.annotate(name, (x[-1], m[-1]), xytext=(10, dy), textcoords="offset points",
125
+ va="center", color=INK2, fontsize=9)
126
+ zs_m = np.array(list(rows["zero-shot LightGBM (20 draws)"].values()))
127
+ ax.plot(x, zs_m, color="#1baf7a", lw=2, marker="o", ms=8, mec=SURF, mew=2,
128
+ label="Zero-shot LightGBM (Madrid only)")
129
+ ax.annotate("Zero-shot LightGBM", (x[-1], zs_m[-1]), xytext=(10, -9),
130
+ textcoords="offset points", va="center", color=INK2, fontsize=9)
131
+ base = np.array(list(rows["organisers' prototype, our re-run on 393 features (20 draws)"].values()))
132
+ ax.plot(x, base, color=MUTED, lw=1.5, ls="--", label="Organisers' prototype (our re-run)")
133
+ ax.annotate("Organisers' prototype (re-run)", (x[-1], base[-1]), xytext=(10, 8),
134
+ textcoords="offset points", va="center", color=MUTED, fontsize=9)
135
+ ax.set_xticks(x, [str(n) for n in N])
136
+ ax.set_xlabel("Labelled Amsterdam pixels per class (log scale)", color=INK2)
137
+ ax.set_ylabel("Macro F1, all of Amsterdam", color=INK2)
138
+ ax.set_title("Macro F1 against labelling budget", loc="left", color=INK, fontsize=12)
139
+ ax.grid(axis="y", color=GRID, lw=0.8)
140
+ ax.set_axisbelow(True)
141
+ for sp in ("top", "right"):
142
+ ax.spines[sp].set_visible(False)
143
+ for sp in ("left", "bottom"):
144
+ ax.spines[sp].set_color(GRID)
145
+ ax.tick_params(colors=MUTED)
146
+ ax.set_xlim(x[0] - 0.3, x[-1] + 2.6)
147
+ ax.legend(loc="lower right", frameon=False, fontsize=8, labelcolor=INK2)
148
+ plt.tight_layout()
149
+ fig.savefig(R / "f1_vs_n_full_city.png", dpi=160, facecolor=SURF)
150
+ plt.show()
151
+
152
+ # %% [markdown]
153
+ # ## 4. Using them on your own data
154
+ #
155
+ # Your features must be the same 393 columns, in `models/meta.json` order
156
+ # (`W.feature_names()`); `src/features.py` / notebook 1 builds them from the
157
+ # raw hackathon parquet files.
158
+ #
159
+ # ```python
160
+ # # zero-shot: no target labels at all
161
+ # P = W.predict(W.load_zero_shot(), X_target)
162
+ # labels = W.decide(P, madrid_prior) # or P.argmax(1) + 1
163
+ #
164
+ # # retrained: you have n labelled target pixels per class
165
+ # model, stats = W.train_retrained(X_mad, y_mad, X_target_all, X_shot, y_shot)
166
+ # labels = W.decide(W.predict_retrained(model, stats, X_rest), madrid_prior)
167
+ # ```
168
+ #
169
+ # To compare with your own model on identical draws, use `W.shots_for(y_ams, n,
170
+ # rep)` for the labelled pixels and score on the rest of Amsterdam - that is the
171
+ # setup behind every number in the table above.
notebooks/winners_demo.ipynb ADDED
The diff for this file is too large to render. See raw diff
 
results/f1_vs_n_full_city.png ADDED

Git LFS Details

  • SHA256: 8347ddf331b7ff0a40da5a9e24865a8774d5fbc59d89b1155a939e3097026422
  • Pointer size: 131 Bytes
  • Size of remote file: 118 kB
results/full_score_prototype.json ADDED
@@ -0,0 +1,1884 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "variant": "prototype, robust z + within-class Mahalanobis",
3
+ "key": "prototype",
4
+ "seed": 42,
5
+ "n_repeats": 12,
6
+ "complete": true,
7
+ "minutes": 1.0,
8
+ "summary": {
9
+ "full": {
10
+ "5": {
11
+ "mean": 0.5040134544059031,
12
+ "std": 0.04021859373385611
13
+ },
14
+ "25": {
15
+ "mean": 0.6361018748678128,
16
+ "std": 0.017000180812003537
17
+ },
18
+ "50": {
19
+ "mean": 0.6732725857102647,
20
+ "std": 0.01345779825462668
21
+ },
22
+ "100": {
23
+ "mean": 0.7023483182129454,
24
+ "std": 0.006656872378539378
25
+ },
26
+ "200": {
27
+ "mean": 0.7140912076382394,
28
+ "std": 0.003112708631110763
29
+ },
30
+ "average": 0.6459654881670331,
31
+ "peak": 0.7140912076382394
32
+ },
33
+ "sub2000": {
34
+ "5": {
35
+ "mean": 0.502055921409773,
36
+ "std": 0.03767702957238527
37
+ },
38
+ "25": {
39
+ "mean": 0.6343220327127315,
40
+ "std": 0.020571814234869437
41
+ },
42
+ "50": {
43
+ "mean": 0.6706418571061313,
44
+ "std": 0.012976731694065123
45
+ },
46
+ "100": {
47
+ "mean": 0.6935287159302076,
48
+ "std": 0.005102727884790501
49
+ },
50
+ "200": {
51
+ "mean": 0.706650164661898,
52
+ "std": 0.004865731656175073
53
+ },
54
+ "average": 0.6414397383641484,
55
+ "peak": 0.706650164661898
56
+ },
57
+ "sub8000": {
58
+ "5": {
59
+ "mean": 0.5024695988340874,
60
+ "std": 0.04055649098254216
61
+ },
62
+ "25": {
63
+ "mean": 0.6334132402958002,
64
+ "std": 0.019190847360900567
65
+ },
66
+ "50": {
67
+ "mean": 0.6694949059810421,
68
+ "std": 0.013232307229607042
69
+ },
70
+ "100": {
71
+ "mean": 0.6967362989316824,
72
+ "std": 0.005985273723055242
73
+ },
74
+ "200": {
75
+ "mean": 0.7085324191993009,
76
+ "std": 0.002440110517091129
77
+ },
78
+ "average": 0.6421292926483826,
79
+ "peak": 0.7085324191993009
80
+ }
81
+ },
82
+ "draws": [
83
+ {
84
+ "n": 5,
85
+ "rep": 0,
86
+ "n_shots": 20,
87
+ "full": 0.5154424688500305,
88
+ "full_n_scored": 25972,
89
+ "full_per_class": [
90
+ 0.6124050317598705,
91
+ 0.5037013196009011,
92
+ 0.5162102612527542,
93
+ 0.4294532627865961
94
+ ],
95
+ "sub2000": 0.5257512950480522,
96
+ "sub2000_n_scored": 2000,
97
+ "sub2000_per_class": [
98
+ 0.605414273995078,
99
+ 0.5209742895805142,
100
+ 0.49513513513513513,
101
+ 0.48148148148148145
102
+ ],
103
+ "sub8000": 0.5134392880785523,
104
+ "sub8000_n_scored": 7995,
105
+ "sub8000_per_class": [
106
+ 0.6041834271922768,
107
+ 0.5072587532023911,
108
+ 0.5009256810367627,
109
+ 0.44138929088277856
110
+ ],
111
+ "seconds": 39.8
112
+ },
113
+ {
114
+ "n": 5,
115
+ "rep": 1,
116
+ "n_shots": 20,
117
+ "full": 0.5326295828708851,
118
+ "full_n_scored": 25972,
119
+ "full_per_class": [
120
+ 0.5560405918702874,
121
+ 0.5418573898891262,
122
+ 0.40981810212579445,
123
+ 0.6228022475983325
124
+ ],
125
+ "sub2000": 0.5267168025537948,
126
+ "sub2000_n_scored": 1998,
127
+ "sub2000_per_class": [
128
+ 0.5521835677276091,
129
+ 0.5410691003911343,
130
+ 0.4094955489614243,
131
+ 0.6041189931350115
132
+ ],
133
+ "sub8000": 0.5269621001826816,
134
+ "sub8000_n_scored": 7989,
135
+ "sub8000_per_class": [
136
+ 0.5532614653754796,
137
+ 0.5414490861618799,
138
+ 0.40131578947368424,
139
+ 0.6118220597196831
140
+ ],
141
+ "seconds": 35.2
142
+ },
143
+ {
144
+ "n": 5,
145
+ "rep": 2,
146
+ "n_shots": 20,
147
+ "full": 0.5137821179640637,
148
+ "full_n_scored": 25972,
149
+ "full_per_class": [
150
+ 0.5787100814026299,
151
+ 0.5056806444949391,
152
+ 0.5766300425054134,
153
+ 0.3941077034532722
154
+ ],
155
+ "sub2000": 0.4876887682276281,
156
+ "sub2000_n_scored": 1999,
157
+ "sub2000_per_class": [
158
+ 0.5804837364470392,
159
+ 0.5074821080026025,
160
+ 0.5479744136460555,
161
+ 0.3148148148148148
162
+ ],
163
+ "sub8000": 0.5131155290628532,
164
+ "sub8000_n_scored": 7994,
165
+ "sub8000_per_class": [
166
+ 0.5848521668691778,
167
+ 0.5130234091658424,
168
+ 0.5740197385969592,
169
+ 0.3805668016194332
170
+ ],
171
+ "seconds": 27.8
172
+ },
173
+ {
174
+ "n": 5,
175
+ "rep": 3,
176
+ "n_shots": 20,
177
+ "full": 0.5246630594618923,
178
+ "full_n_scored": 25972,
179
+ "full_per_class": [
180
+ 0.5900469706879125,
181
+ 0.5285010147868947,
182
+ 0.5155802984613006,
183
+ 0.4645239539114615
184
+ ],
185
+ "sub2000": 0.5102206127354992,
186
+ "sub2000_n_scored": 1997,
187
+ "sub2000_per_class": [
188
+ 0.5787432117920869,
189
+ 0.5407788390889052,
190
+ 0.48370136698212407,
191
+ 0.43765903307888043
192
+ ],
193
+ "sub8000": 0.5209184933017256,
194
+ "sub8000_n_scored": 7990,
195
+ "sub8000_per_class": [
196
+ 0.5779394867866717,
197
+ 0.5269195189639223,
198
+ 0.5113871635610766,
199
+ 0.4674278038952317
200
+ ],
201
+ "seconds": 27.2
202
+ },
203
+ {
204
+ "n": 5,
205
+ "rep": 4,
206
+ "n_shots": 20,
207
+ "full": 0.5268593323268136,
208
+ "full_n_scored": 25972,
209
+ "full_per_class": [
210
+ 0.5894369450289997,
211
+ 0.5089075630252101,
212
+ 0.4094387755102041,
213
+ 0.5996540457428406
214
+ ],
215
+ "sub2000": 0.5213589711822051,
216
+ "sub2000_n_scored": 1997,
217
+ "sub2000_per_class": [
218
+ 0.5939799331103679,
219
+ 0.5292439372325249,
220
+ 0.39710144927536234,
221
+ 0.5651105651105651
222
+ ],
223
+ "sub8000": 0.520376795826306,
224
+ "sub8000_n_scored": 7992,
225
+ "sub8000_per_class": [
226
+ 0.5819983484723369,
227
+ 0.5062016897357541,
228
+ 0.40028388928317954,
229
+ 0.5930232558139535
230
+ ],
231
+ "seconds": 27.6
232
+ },
233
+ {
234
+ "n": 5,
235
+ "rep": 5,
236
+ "n_shots": 20,
237
+ "full": 0.5471240648440912,
238
+ "full_n_scored": 25972,
239
+ "full_per_class": [
240
+ 0.6156672995651375,
241
+ 0.5271687321258341,
242
+ 0.5570912183712524,
243
+ 0.48856900931414055
244
+ ],
245
+ "sub2000": 0.5534035506246425,
246
+ "sub2000_n_scored": 1998,
247
+ "sub2000_per_class": [
248
+ 0.6097560975609756,
249
+ 0.5557251908396946,
250
+ 0.5255198487712666,
251
+ 0.5226130653266332
252
+ ],
253
+ "sub8000": 0.5484807588173956,
254
+ "sub8000_n_scored": 7994,
255
+ "sub8000_per_class": [
256
+ 0.615263261684086,
257
+ 0.5338706582229898,
258
+ 0.5482661668228679,
259
+ 0.4965229485396384
260
+ ],
261
+ "seconds": 27.6
262
+ },
263
+ {
264
+ "n": 5,
265
+ "rep": 6,
266
+ "n_shots": 20,
267
+ "full": 0.46987007052344576,
268
+ "full_n_scored": 25972,
269
+ "full_per_class": [
270
+ 0.45782431646305993,
271
+ 0.47055798405759836,
272
+ 0.4908051305825993,
273
+ 0.4602928509905254
274
+ ],
275
+ "sub2000": 0.4803513443960111,
276
+ "sub2000_n_scored": 1997,
277
+ "sub2000_per_class": [
278
+ 0.45048543689320386,
279
+ 0.4915693904020752,
280
+ 0.4968944099378882,
281
+ 0.4824561403508772
282
+ ],
283
+ "sub8000": 0.4765089772639029,
284
+ "sub8000_n_scored": 7992,
285
+ "sub8000_per_class": [
286
+ 0.457655107336636,
287
+ 0.4862204724409449,
288
+ 0.5025614754098361,
289
+ 0.4595988538681948
290
+ ],
291
+ "seconds": 34.9
292
+ },
293
+ {
294
+ "n": 5,
295
+ "rep": 7,
296
+ "n_shots": 20,
297
+ "full": 0.4917213402701698,
298
+ "full_n_scored": 25972,
299
+ "full_per_class": [
300
+ 0.5128599302311831,
301
+ 0.5368910239674168,
302
+ 0.5493389370457928,
303
+ 0.36779546983628614
304
+ ],
305
+ "sub2000": 0.49679984874412175,
306
+ "sub2000_n_scored": 1999,
307
+ "sub2000_per_class": [
308
+ 0.5030769230769231,
309
+ 0.5381526104417671,
310
+ 0.5437352245862884,
311
+ 0.4022346368715084
312
+ ],
313
+ "sub8000": 0.4965408085174481,
314
+ "sub8000_n_scored": 7994,
315
+ "sub8000_per_class": [
316
+ 0.5141329258976318,
317
+ 0.5416877210712481,
318
+ 0.5490423679628554,
319
+ 0.38130021913805695
320
+ ],
321
+ "seconds": 45.0
322
+ },
323
+ {
324
+ "n": 5,
325
+ "rep": 8,
326
+ "n_shots": 20,
327
+ "full": 0.4951747256979879,
328
+ "full_n_scored": 25972,
329
+ "full_per_class": [
330
+ 0.5407663348976329,
331
+ 0.47215399554463927,
332
+ 0.4596954314720812,
333
+ 0.5080831408775982
334
+ ],
335
+ "sub2000": 0.5003407962157336,
336
+ "sub2000_n_scored": 1999,
337
+ "sub2000_per_class": [
338
+ 0.5452109845947756,
339
+ 0.48520710059171596,
340
+ 0.46132971506105835,
341
+ 0.5096153846153846
342
+ ],
343
+ "sub8000": 0.49407311500154427,
344
+ "sub8000_n_scored": 7994,
345
+ "sub8000_per_class": [
346
+ 0.5343511450381679,
347
+ 0.4681793027116768,
348
+ 0.46380697050938335,
349
+ 0.5099550417469493
350
+ ],
351
+ "seconds": 61.9
352
+ },
353
+ {
354
+ "n": 5,
355
+ "rep": 9,
356
+ "n_shots": 20,
357
+ "full": 0.5105027094316081,
358
+ "full_n_scored": 25972,
359
+ "full_per_class": [
360
+ 0.5187257856220404,
361
+ 0.44122837980938934,
362
+ 0.5274665702927358,
363
+ 0.5545901020022668
364
+ ],
365
+ "sub2000": 0.4913459895921074,
366
+ "sub2000_n_scored": 1998,
367
+ "sub2000_per_class": [
368
+ 0.4989339019189765,
369
+ 0.4460856720827179,
370
+ 0.4902439024390244,
371
+ 0.5301204819277109
372
+ ],
373
+ "sub8000": 0.5075218336707972,
374
+ "sub8000_n_scored": 7995,
375
+ "sub8000_per_class": [
376
+ 0.5158371040723982,
377
+ 0.4420180722891566,
378
+ 0.5225171488219504,
379
+ 0.5497150094996833
380
+ ],
381
+ "seconds": 55.9
382
+ },
383
+ {
384
+ "n": 5,
385
+ "rep": 10,
386
+ "n_shots": 20,
387
+ "full": 0.3943778679352975,
388
+ "full_n_scored": 25972,
389
+ "full_per_class": [
390
+ 0.5376032331751889,
391
+ 0.3708198667634061,
392
+ 0.298419173890872,
393
+ 0.3706691979117228
394
+ ],
395
+ "sub2000": 0.40268980129722615,
396
+ "sub2000_n_scored": 1999,
397
+ "sub2000_per_class": [
398
+ 0.533256880733945,
399
+ 0.3829787234042553,
400
+ 0.3004059539918809,
401
+ 0.3941176470588235
402
+ ],
403
+ "sub8000": 0.3875502403605131,
404
+ "sub8000_n_scored": 7995,
405
+ "sub8000_per_class": [
406
+ 0.530970942593905,
407
+ 0.3642902475142797,
408
+ 0.2881413523615359,
409
+ 0.366798418972332
410
+ ],
411
+ "seconds": 68.2
412
+ },
413
+ {
414
+ "n": 5,
415
+ "rep": 11,
416
+ "n_shots": 20,
417
+ "full": 0.5260141126945512,
418
+ "full_n_scored": 25972,
419
+ "full_per_class": [
420
+ 0.5339580866072807,
421
+ 0.4479087452471483,
422
+ 0.5311679470971507,
423
+ 0.5910216718266253
424
+ ],
425
+ "sub2000": 0.5280032763002531,
426
+ "sub2000_n_scored": 1999,
427
+ "sub2000_per_class": [
428
+ 0.552,
429
+ 0.48097165991902835,
430
+ 0.48787061994609165,
431
+ 0.5911708253358925
432
+ ],
433
+ "sub8000": 0.5241472459253292,
434
+ "sub8000_n_scored": 7995,
435
+ "sub8000_per_class": [
436
+ 0.5301645338208409,
437
+ 0.45690004046944555,
438
+ 0.5241334629089731,
439
+ 0.5853909465020576
440
+ ],
441
+ "seconds": 59.9
442
+ },
443
+ {
444
+ "n": 25,
445
+ "rep": 0,
446
+ "n_shots": 100,
447
+ "full": 0.640875282604727,
448
+ "full_n_scored": 25892,
449
+ "full_per_class": [
450
+ 0.6361733563161783,
451
+ 0.5780404285872086,
452
+ 0.6706143597335307,
453
+ 0.6786729857819905
454
+ ],
455
+ "sub2000": 0.6400644522880872,
456
+ "sub2000_n_scored": 1992,
457
+ "sub2000_per_class": [
458
+ 0.6412593206296603,
459
+ 0.5989010989010989,
460
+ 0.6644370122630993,
461
+ 0.6556603773584906
462
+ ],
463
+ "sub8000": 0.6375518275829706,
464
+ "sub8000_n_scored": 7974,
465
+ "sub8000_per_class": [
466
+ 0.6381789137380192,
467
+ 0.5861887190300474,
468
+ 0.6650246305418719,
469
+ 0.6608150470219436
470
+ ],
471
+ "seconds": 68.6
472
+ },
473
+ {
474
+ "n": 25,
475
+ "rep": 1,
476
+ "n_shots": 100,
477
+ "full": 0.6462353536164829,
478
+ "full_n_scored": 25892,
479
+ "full_per_class": [
480
+ 0.6434660898333121,
481
+ 0.579883027390121,
482
+ 0.7014319626311356,
483
+ 0.6601603346113628
484
+ ],
485
+ "sub2000": 0.6543340178062504,
486
+ "sub2000_n_scored": 1989,
487
+ "sub2000_per_class": [
488
+ 0.6503791069924179,
489
+ 0.6036363636363636,
490
+ 0.6951983298538622,
491
+ 0.6681222707423581
492
+ ],
493
+ "sub8000": 0.6498008979870565,
494
+ "sub8000_n_scored": 7965,
495
+ "sub8000_per_class": [
496
+ 0.645227788084931,
497
+ 0.5950321987120515,
498
+ 0.7054322876817138,
499
+ 0.6535113174695298
500
+ ],
501
+ "seconds": 61.2
502
+ },
503
+ {
504
+ "n": 25,
505
+ "rep": 2,
506
+ "n_shots": 100,
507
+ "full": 0.6049705877379048,
508
+ "full_n_scored": 25892,
509
+ "full_per_class": [
510
+ 0.6285833637930134,
511
+ 0.4990295160966468,
512
+ 0.678342089900759,
513
+ 0.6139273811611998
514
+ ],
515
+ "sub2000": 0.5999868198166421,
516
+ "sub2000_n_scored": 1997,
517
+ "sub2000_per_class": [
518
+ 0.629553264604811,
519
+ 0.4970414201183432,
520
+ 0.6562847608453838,
521
+ 0.6170678336980306
522
+ ],
523
+ "sub8000": 0.600316966641765,
524
+ "sub8000_n_scored": 7970,
525
+ "sub8000_per_class": [
526
+ 0.6238655462184874,
527
+ 0.49592449592449594,
528
+ 0.6746053724729991,
529
+ 0.6068724519510774
530
+ ],
531
+ "seconds": 91.6
532
+ },
533
+ {
534
+ "n": 25,
535
+ "rep": 3,
536
+ "n_shots": 100,
537
+ "full": 0.6219319589295691,
538
+ "full_n_scored": 25892,
539
+ "full_per_class": [
540
+ 0.624221844934918,
541
+ 0.581067472306143,
542
+ 0.6577428663033024,
543
+ 0.6246956521739131
544
+ ],
545
+ "sub2000": 0.6287105382480286,
546
+ "sub2000_n_scored": 1992,
547
+ "sub2000_per_class": [
548
+ 0.6223880597014926,
549
+ 0.6126984126984127,
550
+ 0.6616541353383458,
551
+ 0.6181015452538632
552
+ ],
553
+ "sub8000": 0.6203997388313052,
554
+ "sub8000_n_scored": 7971,
555
+ "sub8000_per_class": [
556
+ 0.6186471663619744,
557
+ 0.5868215501702383,
558
+ 0.6595801222428913,
559
+ 0.6165501165501166
560
+ ],
561
+ "seconds": 64.1
562
+ },
563
+ {
564
+ "n": 25,
565
+ "rep": 4,
566
+ "n_shots": 100,
567
+ "full": 0.6076829582379857,
568
+ "full_n_scored": 25892,
569
+ "full_per_class": [
570
+ 0.6168258859784284,
571
+ 0.5792551230047938,
572
+ 0.614207650273224,
573
+ 0.6204431736954967
574
+ ],
575
+ "sub2000": 0.6012618929248565,
576
+ "sub2000_n_scored": 1992,
577
+ "sub2000_per_class": [
578
+ 0.6019417475728155,
579
+ 0.5868983957219251,
580
+ 0.5987577639751552,
581
+ 0.6174496644295302
582
+ ],
583
+ "sub8000": 0.5991046254256172,
584
+ "sub8000_n_scored": 7962,
585
+ "sub8000_per_class": [
586
+ 0.6048741510187775,
587
+ 0.58025520483546,
588
+ 0.6010995723885155,
589
+ 0.6101895734597157
590
+ ],
591
+ "seconds": 52.0
592
+ },
593
+ {
594
+ "n": 25,
595
+ "rep": 5,
596
+ "n_shots": 100,
597
+ "full": 0.6520096487766935,
598
+ "full_n_scored": 25892,
599
+ "full_per_class": [
600
+ 0.6863331737673528,
601
+ 0.6187339510996985,
602
+ 0.6821985233798196,
603
+ 0.6207729468599034
604
+ ],
605
+ "sub2000": 0.6490206537222208,
606
+ "sub2000_n_scored": 1992,
607
+ "sub2000_per_class": [
608
+ 0.6833463643471462,
609
+ 0.6182336182336182,
610
+ 0.6511111111111111,
611
+ 0.6433915211970075
612
+ ],
613
+ "sub8000": 0.6512923308153209,
614
+ "sub8000_n_scored": 7970,
615
+ "sub8000_per_class": [
616
+ 0.68145861470191,
617
+ 0.6198288159771754,
618
+ 0.6781232951445717,
619
+ 0.6257585974376264
620
+ ],
621
+ "seconds": 23.6
622
+ },
623
+ {
624
+ "n": 25,
625
+ "rep": 6,
626
+ "n_shots": 100,
627
+ "full": 0.6534945461223763,
628
+ "full_n_scored": 25892,
629
+ "full_per_class": [
630
+ 0.6531046569854783,
631
+ 0.5963722664858451,
632
+ 0.6980535684578273,
633
+ 0.6664476925603547
634
+ ],
635
+ "sub2000": 0.6631011526610081,
636
+ "sub2000_n_scored": 1994,
637
+ "sub2000_per_class": [
638
+ 0.6583541147132169,
639
+ 0.6167146974063401,
640
+ 0.7079646017699115,
641
+ 0.6693711967545639
642
+ ],
643
+ "sub8000": 0.6553572804150688,
644
+ "sub8000_n_scored": 7971,
645
+ "sub8000_per_class": [
646
+ 0.6585907530789421,
647
+ 0.6051486584481508,
648
+ 0.7012345679012346,
649
+ 0.6564551422319475
650
+ ],
651
+ "seconds": 24.5
652
+ },
653
+ {
654
+ "n": 25,
655
+ "rep": 7,
656
+ "n_shots": 100,
657
+ "full": 0.6382474330533008,
658
+ "full_n_scored": 25892,
659
+ "full_per_class": [
660
+ 0.6490729295426453,
661
+ 0.594467333725721,
662
+ 0.6640413056795309,
663
+ 0.6454081632653061
664
+ ],
665
+ "sub2000": 0.629982052018506,
666
+ "sub2000_n_scored": 1990,
667
+ "sub2000_per_class": [
668
+ 0.6415396952686447,
669
+ 0.6004112405757368,
670
+ 0.6587112171837709,
671
+ 0.6192660550458715
672
+ ],
673
+ "sub8000": 0.6365399735546273,
674
+ "sub8000_n_scored": 7962,
675
+ "sub8000_per_class": [
676
+ 0.6452642073778664,
677
+ 0.5956349888296958,
678
+ 0.6660854402789886,
679
+ 0.6391752577319587
680
+ ],
681
+ "seconds": 27.0
682
+ },
683
+ {
684
+ "n": 25,
685
+ "rep": 8,
686
+ "n_shots": 100,
687
+ "full": 0.6442699308876451,
688
+ "full_n_scored": 25892,
689
+ "full_per_class": [
690
+ 0.7078610603290676,
691
+ 0.6310214659980844,
692
+ 0.6169136878814299,
693
+ 0.6212835093419984
694
+ ],
695
+ "sub2000": 0.6364944946680255,
696
+ "sub2000_n_scored": 1995,
697
+ "sub2000_per_class": [
698
+ 0.6973365617433414,
699
+ 0.6278409090909091,
700
+ 0.5992865636147443,
701
+ 0.6215139442231076
702
+ ],
703
+ "sub8000": 0.637549048329748,
704
+ "sub8000_n_scored": 7969,
705
+ "sub8000_per_class": [
706
+ 0.6984500686678439,
707
+ 0.6288603937150081,
708
+ 0.6132484813422042,
709
+ 0.6096372495939361
710
+ ],
711
+ "seconds": 25.4
712
+ },
713
+ {
714
+ "n": 25,
715
+ "rep": 9,
716
+ "n_shots": 100,
717
+ "full": 0.6337083777720867,
718
+ "full_n_scored": 25892,
719
+ "full_per_class": [
720
+ 0.5752280611589361,
721
+ 0.5840117593848937,
722
+ 0.6890940384916289,
723
+ 0.686499652052888
724
+ ],
725
+ "sub2000": 0.6472006053609829,
726
+ "sub2000_n_scored": 1990,
727
+ "sub2000_per_class": [
728
+ 0.5823223570190641,
729
+ 0.6200997861724875,
730
+ 0.6804979253112033,
731
+ 0.7058823529411765
732
+ ],
733
+ "sub8000": 0.6350662432396932,
734
+ "sub8000_n_scored": 7970,
735
+ "sub8000_per_class": [
736
+ 0.571667363142499,
737
+ 0.5942159152146578,
738
+ 0.6847265221878225,
739
+ 0.6896551724137931
740
+ ],
741
+ "seconds": 28.9
742
+ },
743
+ {
744
+ "n": 25,
745
+ "rep": 10,
746
+ "n_shots": 100,
747
+ "full": 0.6328845987720971,
748
+ "full_n_scored": 25892,
749
+ "full_per_class": [
750
+ 0.6445723588580732,
751
+ 0.6005224834639542,
752
+ 0.6450345828708052,
753
+ 0.641408969895556
754
+ ],
755
+ "sub2000": 0.6126794912079919,
756
+ "sub2000_n_scored": 1996,
757
+ "sub2000_per_class": [
758
+ 0.6372093023255814,
759
+ 0.6146341463414634,
760
+ 0.6127562642369021,
761
+ 0.5861182519280206
762
+ ],
763
+ "sub8000": 0.624315516214057,
764
+ "sub8000_n_scored": 7978,
765
+ "sub8000_per_class": [
766
+ 0.6394761459307764,
767
+ 0.6042037762736017,
768
+ 0.628668171557562,
769
+ 0.6249139710942877
770
+ ],
771
+ "seconds": 28.2
772
+ },
773
+ {
774
+ "n": 25,
775
+ "rep": 11,
776
+ "n_shots": 100,
777
+ "full": 0.6569118219028849,
778
+ "full_n_scored": 25892,
779
+ "full_per_class": [
780
+ 0.6585581043609781,
781
+ 0.6374685207489325,
782
+ 0.6773131379650972,
783
+ 0.6543075245365322
784
+ ],
785
+ "sub2000": 0.6490282218301774,
786
+ "sub2000_n_scored": 1986,
787
+ "sub2000_per_class": [
788
+ 0.6488294314381271,
789
+ 0.6411682892906815,
790
+ 0.6747787610619469,
791
+ 0.631336405529954
792
+ ],
793
+ "sub8000": 0.6536644345123719,
794
+ "sub8000_n_scored": 7963,
795
+ "sub8000_per_class": [
796
+ 0.6559446254071661,
797
+ 0.6368577941434332,
798
+ 0.6795640326975477,
799
+ 0.6422912858013407
800
+ ],
801
+ "seconds": 28.2
802
+ },
803
+ {
804
+ "n": 50,
805
+ "rep": 0,
806
+ "n_shots": 200,
807
+ "full": 0.6880158094676923,
808
+ "full_n_scored": 25792,
809
+ "full_per_class": [
810
+ 0.7072850989450028,
811
+ 0.660347728547392,
812
+ 0.6904608443441831,
813
+ 0.6939695660341912
814
+ ],
815
+ "sub2000": 0.6816534505616696,
816
+ "sub2000_n_scored": 1990,
817
+ "sub2000_per_class": [
818
+ 0.6990131578947368,
819
+ 0.6652631578947369,
820
+ 0.6652221018418202,
821
+ 0.6971153846153846
822
+ ],
823
+ "sub8000": 0.6822419934147161,
824
+ "sub8000_n_scored": 7937,
825
+ "sub8000_per_class": [
826
+ 0.6990369181380417,
827
+ 0.6651121076233184,
828
+ 0.688367129135539,
829
+ 0.6764518187619656
830
+ ],
831
+ "seconds": 31.1
832
+ },
833
+ {
834
+ "n": 50,
835
+ "rep": 1,
836
+ "n_shots": 200,
837
+ "full": 0.6868617534245177,
838
+ "full_n_scored": 25792,
839
+ "full_per_class": [
840
+ 0.7091687041564793,
841
+ 0.6400612160809935,
842
+ 0.7025649195475545,
843
+ 0.6956521739130435
844
+ ],
845
+ "sub2000": 0.6815708796518979,
846
+ "sub2000_n_scored": 1989,
847
+ "sub2000_per_class": [
848
+ 0.6987951807228916,
849
+ 0.653817642698295,
850
+ 0.6981740064446831,
851
+ 0.6754966887417219
852
+ ],
853
+ "sub8000": 0.6806289615933406,
854
+ "sub8000_n_scored": 7944,
855
+ "sub8000_per_class": [
856
+ 0.6986571879936809,
857
+ 0.640598690364827,
858
+ 0.7044791942751126,
859
+ 0.6787807737397421
860
+ ],
861
+ "seconds": 38.5
862
+ },
863
+ {
864
+ "n": 50,
865
+ "rep": 2,
866
+ "n_shots": 200,
867
+ "full": 0.6761533776528186,
868
+ "full_n_scored": 25792,
869
+ "full_per_class": [
870
+ 0.7218229838462011,
871
+ 0.6528491620111732,
872
+ 0.7049630774514855,
873
+ 0.6249782873024144
874
+ ],
875
+ "sub2000": 0.684266315844529,
876
+ "sub2000_n_scored": 1982,
877
+ "sub2000_per_class": [
878
+ 0.7095709570957096,
879
+ 0.6713881019830028,
880
+ 0.7053469852104665,
881
+ 0.6507592190889371
882
+ ],
883
+ "sub8000": 0.6776552885349774,
884
+ "sub8000_n_scored": 7952,
885
+ "sub8000_per_class": [
886
+ 0.7142005958291956,
887
+ 0.6607015032211883,
888
+ 0.7030542986425339,
889
+ 0.6326647564469914
890
+ ],
891
+ "seconds": 46.1
892
+ },
893
+ {
894
+ "n": 50,
895
+ "rep": 3,
896
+ "n_shots": 200,
897
+ "full": 0.6751606852878018,
898
+ "full_n_scored": 25792,
899
+ "full_per_class": [
900
+ 0.6960825234034941,
901
+ 0.6345070823141248,
902
+ 0.6862860506180106,
903
+ 0.6837670848155776
904
+ ],
905
+ "sub2000": 0.6592988990639563,
906
+ "sub2000_n_scored": 1984,
907
+ "sub2000_per_class": [
908
+ 0.6766561514195584,
909
+ 0.6344628695025234,
910
+ 0.6704161979752531,
911
+ 0.6556603773584906
912
+ ],
913
+ "sub8000": 0.6707430962063111,
914
+ "sub8000_n_scored": 7921,
915
+ "sub8000_per_class": [
916
+ 0.6888888888888889,
917
+ 0.6409655917852639,
918
+ 0.6868686868686869,
919
+ 0.6662492172824045
920
+ ],
921
+ "seconds": 45.4
922
+ },
923
+ {
924
+ "n": 50,
925
+ "rep": 4,
926
+ "n_shots": 200,
927
+ "full": 0.6800384167589169,
928
+ "full_n_scored": 25792,
929
+ "full_per_class": [
930
+ 0.6942928039702233,
931
+ 0.6617647058823529,
932
+ 0.6892243623112962,
933
+ 0.6748717948717948
934
+ ],
935
+ "sub2000": 0.6796779663644648,
936
+ "sub2000_n_scored": 1986,
937
+ "sub2000_per_class": [
938
+ 0.6995884773662552,
939
+ 0.6782006920415224,
940
+ 0.6713947990543735,
941
+ 0.6695278969957081
942
+ ],
943
+ "sub8000": 0.6780713521299202,
944
+ "sub8000_n_scored": 7930,
945
+ "sub8000_per_class": [
946
+ 0.6851628468033776,
947
+ 0.6670180987524161,
948
+ 0.6865412445730825,
949
+ 0.6735632183908046
950
+ ],
951
+ "seconds": 45.5
952
+ },
953
+ {
954
+ "n": 50,
955
+ "rep": 5,
956
+ "n_shots": 200,
957
+ "full": 0.674532717112151,
958
+ "full_n_scored": 25792,
959
+ "full_per_class": [
960
+ 0.7144553465408412,
961
+ 0.6238737478314399,
962
+ 0.6625876978299886,
963
+ 0.6972140762463344
964
+ ],
965
+ "sub2000": 0.6655712354880495,
966
+ "sub2000_n_scored": 1984,
967
+ "sub2000_per_class": [
968
+ 0.7127746135069162,
969
+ 0.6275911365260901,
970
+ 0.6355555555555555,
971
+ 0.6863636363636364
972
+ ],
973
+ "sub8000": 0.6650162387328394,
974
+ "sub8000_n_scored": 7954,
975
+ "sub8000_per_class": [
976
+ 0.7032438926712055,
977
+ 0.6213176218532405,
978
+ 0.6550408719346049,
979
+ 0.6804625684723068
980
+ ],
981
+ "seconds": 46.3
982
+ },
983
+ {
984
+ "n": 50,
985
+ "rep": 6,
986
+ "n_shots": 200,
987
+ "full": 0.6507196917015792,
988
+ "full_n_scored": 25792,
989
+ "full_per_class": [
990
+ 0.6819444444444445,
991
+ 0.6029737036126812,
992
+ 0.700801282051282,
993
+ 0.6171593366979091
994
+ ],
995
+ "sub2000": 0.647336654012965,
996
+ "sub2000_n_scored": 1989,
997
+ "sub2000_per_class": [
998
+ 0.6671802773497689,
999
+ 0.6004654771140419,
1000
+ 0.7017913593256059,
1001
+ 0.6199095022624435
1002
+ ],
1003
+ "sub8000": 0.6466769599736414,
1004
+ "sub8000_n_scored": 7950,
1005
+ "sub8000_per_class": [
1006
+ 0.6756352765321375,
1007
+ 0.5990964447063445,
1008
+ 0.7034883720930233,
1009
+ 0.6084877465630604
1010
+ ],
1011
+ "seconds": 45.0
1012
+ },
1013
+ {
1014
+ "n": 50,
1015
+ "rep": 7,
1016
+ "n_shots": 200,
1017
+ "full": 0.647643882382783,
1018
+ "full_n_scored": 25792,
1019
+ "full_per_class": [
1020
+ 0.6565767079990275,
1021
+ 0.5822250864139669,
1022
+ 0.6838989410489452,
1023
+ 0.6678747940691927
1024
+ ],
1025
+ "sub2000": 0.6529559435392345,
1026
+ "sub2000_n_scored": 1988,
1027
+ "sub2000_per_class": [
1028
+ 0.6493083807973963,
1029
+ 0.6031976744186046,
1030
+ 0.675,
1031
+ 0.6843177189409368
1032
+ ],
1033
+ "sub8000": 0.6467267201387928,
1034
+ "sub8000_n_scored": 7947,
1035
+ "sub8000_per_class": [
1036
+ 0.6532052538717898,
1037
+ 0.5883014623172104,
1038
+ 0.6838032061912659,
1039
+ 0.6615969581749049
1040
+ ],
1041
+ "seconds": 45.8
1042
+ },
1043
+ {
1044
+ "n": 50,
1045
+ "rep": 8,
1046
+ "n_shots": 200,
1047
+ "full": 0.6856794797538986,
1048
+ "full_n_scored": 25792,
1049
+ "full_per_class": [
1050
+ 0.7064713064713065,
1051
+ 0.6492123114873577,
1052
+ 0.7034019319613608,
1053
+ 0.6836323690955693
1054
+ ],
1055
+ "sub2000": 0.6787286673127968,
1056
+ "sub2000_n_scored": 1978,
1057
+ "sub2000_per_class": [
1058
+ 0.6871463217461601,
1059
+ 0.6475644699140402,
1060
+ 0.6968325791855203,
1061
+ 0.683371298405467
1062
+ ],
1063
+ "sub8000": 0.6853339089700422,
1064
+ "sub8000_n_scored": 7928,
1065
+ "sub8000_per_class": [
1066
+ 0.6972404208854477,
1067
+ 0.6550797348145494,
1068
+ 0.7109874826147426,
1069
+ 0.6780279975654291
1070
+ ],
1071
+ "seconds": 43.8
1072
+ },
1073
+ {
1074
+ "n": 50,
1075
+ "rep": 9,
1076
+ "n_shots": 200,
1077
+ "full": 0.665854189312306,
1078
+ "full_n_scored": 25792,
1079
+ "full_per_class": [
1080
+ 0.6847287627668033,
1081
+ 0.6492554151624549,
1082
+ 0.6882217090069284,
1083
+ 0.6412108703130375
1084
+ ],
1085
+ "sub2000": 0.6644871528777947,
1086
+ "sub2000_n_scored": 1985,
1087
+ "sub2000_per_class": [
1088
+ 0.6780210867802109,
1089
+ 0.6657101865136298,
1090
+ 0.6902857142857143,
1091
+ 0.6239316239316239
1092
+ ],
1093
+ "sub8000": 0.6623832546172512,
1094
+ "sub8000_n_scored": 7943,
1095
+ "sub8000_per_class": [
1096
+ 0.6786419265692855,
1097
+ 0.6528478731074261,
1098
+ 0.6880317640385706,
1099
+ 0.6300114547537228
1100
+ ],
1101
+ "seconds": 44.9
1102
+ },
1103
+ {
1104
+ "n": 50,
1105
+ "rep": 10,
1106
+ "n_shots": 200,
1107
+ "full": 0.683092386543684,
1108
+ "full_n_scored": 25792,
1109
+ "full_per_class": [
1110
+ 0.7214936143405303,
1111
+ 0.6243424632686377,
1112
+ 0.7066171505739365,
1113
+ 0.6799163179916318
1114
+ ],
1115
+ "sub2000": 0.6853432711878056,
1116
+ "sub2000_n_scored": 1982,
1117
+ "sub2000_per_class": [
1118
+ 0.7136294027565084,
1119
+ 0.6338880484114977,
1120
+ 0.7037457434733257,
1121
+ 0.6901098901098901
1122
+ ],
1123
+ "sub8000": 0.6775123371964548,
1124
+ "sub8000_n_scored": 7933,
1125
+ "sub8000_per_class": [
1126
+ 0.7108052817556259,
1127
+ 0.6208228703882558,
1128
+ 0.707880813143971,
1129
+ 0.6705403834979663
1130
+ ],
1131
+ "seconds": 45.5
1132
+ },
1133
+ {
1134
+ "n": 50,
1135
+ "rep": 11,
1136
+ "n_shots": 200,
1137
+ "full": 0.6655186391250288,
1138
+ "full_n_scored": 25792,
1139
+ "full_per_class": [
1140
+ 0.6858205171539632,
1141
+ 0.612555026231683,
1142
+ 0.6933265839871763,
1143
+ 0.6703724291272929
1144
+ ],
1145
+ "sub2000": 0.6668118493684136,
1146
+ "sub2000_n_scored": 1987,
1147
+ "sub2000_per_class": [
1148
+ 0.6793025018953753,
1149
+ 0.6318147871545929,
1150
+ 0.6818181818181818,
1151
+ 0.6743119266055045
1152
+ ],
1153
+ "sub8000": 0.6609487602642184,
1154
+ "sub8000_n_scored": 7947,
1155
+ "sub8000_per_class": [
1156
+ 0.6763742264288315,
1157
+ 0.6167113836719049,
1158
+ 0.6898290837769684,
1159
+ 0.6608803471791692
1160
+ ],
1161
+ "seconds": 45.4
1162
+ },
1163
+ {
1164
+ "n": 100,
1165
+ "rep": 0,
1166
+ "n_shots": 400,
1167
+ "full": 0.6956605327583154,
1168
+ "full_n_scored": 25592,
1169
+ "full_per_class": [
1170
+ 0.7147677771642432,
1171
+ 0.6651706082053053,
1172
+ 0.6988453858841913,
1173
+ 0.7038583597795223
1174
+ ],
1175
+ "sub2000": 0.6911555855989419,
1176
+ "sub2000_n_scored": 1979,
1177
+ "sub2000_per_class": [
1178
+ 0.7035256410256411,
1179
+ 0.6505847953216374,
1180
+ 0.6877192982456141,
1181
+ 0.7227926078028748
1182
+ ],
1183
+ "sub8000": 0.6886388554939394,
1184
+ "sub8000_n_scored": 7885,
1185
+ "sub8000_per_class": [
1186
+ 0.7018375815056312,
1187
+ 0.6601584085466937,
1188
+ 0.6996547756041427,
1189
+ 0.6929046563192904
1190
+ ],
1191
+ "seconds": 45.2
1192
+ },
1193
+ {
1194
+ "n": 100,
1195
+ "rep": 1,
1196
+ "n_shots": 400,
1197
+ "full": 0.7031505961965816,
1198
+ "full_n_scored": 25592,
1199
+ "full_per_class": [
1200
+ 0.7306061361935645,
1201
+ 0.6826262981869389,
1202
+ 0.7102562030227108,
1203
+ 0.6891137473831124
1204
+ ],
1205
+ "sub2000": 0.6991707855706939,
1206
+ "sub2000_n_scored": 1963,
1207
+ "sub2000_per_class": [
1208
+ 0.7275720164609053,
1209
+ 0.6887218045112782,
1210
+ 0.7006507592190889,
1211
+ 0.6797385620915033
1212
+ ],
1213
+ "sub8000": 0.6970262639275365,
1214
+ "sub8000_n_scored": 7873,
1215
+ "sub8000_per_class": [
1216
+ 0.7239782564928529,
1217
+ 0.6821179574147352,
1218
+ 0.7091442282058118,
1219
+ 0.6728646135967461
1220
+ ],
1221
+ "seconds": 44.2
1222
+ },
1223
+ {
1224
+ "n": 100,
1225
+ "rep": 2,
1226
+ "n_shots": 400,
1227
+ "full": 0.6995821570081839,
1228
+ "full_n_scored": 25592,
1229
+ "full_per_class": [
1230
+ 0.731018024241663,
1231
+ 0.6614014251781473,
1232
+ 0.6998252026060703,
1233
+ 0.7060839760068551
1234
+ ],
1235
+ "sub2000": 0.6893029935349051,
1236
+ "sub2000_n_scored": 1971,
1237
+ "sub2000_per_class": [
1238
+ 0.7252931323283082,
1239
+ 0.6636904761904762,
1240
+ 0.6915688367129136,
1241
+ 0.6766595289079229
1242
+ ],
1243
+ "sub8000": 0.6921564711310434,
1244
+ "sub8000_n_scored": 7873,
1245
+ "sub8000_per_class": [
1246
+ 0.725422348870344,
1247
+ 0.6627643504531722,
1248
+ 0.6942322886489334,
1249
+ 0.6862068965517242
1250
+ ],
1251
+ "seconds": 44.7
1252
+ },
1253
+ {
1254
+ "n": 100,
1255
+ "rep": 3,
1256
+ "n_shots": 400,
1257
+ "full": 0.7156568141977783,
1258
+ "full_n_scored": 25592,
1259
+ "full_per_class": [
1260
+ 0.7436728587457923,
1261
+ 0.6855245075584059,
1262
+ 0.7246083880747852,
1263
+ 0.7088215024121296
1264
+ ],
1265
+ "sub2000": 0.7004211598844011,
1266
+ "sub2000_n_scored": 1963,
1267
+ "sub2000_per_class": [
1268
+ 0.7213930348258707,
1269
+ 0.6734842951059167,
1270
+ 0.7026406429391504,
1271
+ 0.7041666666666667
1272
+ ],
1273
+ "sub8000": 0.7069554378068514,
1274
+ "sub8000_n_scored": 7869,
1275
+ "sub8000_per_class": [
1276
+ 0.7330097087378641,
1277
+ 0.6855059252506837,
1278
+ 0.7150055991041433,
1279
+ 0.694300518134715
1280
+ ],
1281
+ "seconds": 44.9
1282
+ },
1283
+ {
1284
+ "n": 100,
1285
+ "rep": 4,
1286
+ "n_shots": 400,
1287
+ "full": 0.7102531753981858,
1288
+ "full_n_scored": 25592,
1289
+ "full_per_class": [
1290
+ 0.7277865404837014,
1291
+ 0.6795448237579794,
1292
+ 0.7163207163207164,
1293
+ 0.7173606210303458
1294
+ ],
1295
+ "sub2000": 0.7030202908862202,
1296
+ "sub2000_n_scored": 1977,
1297
+ "sub2000_per_class": [
1298
+ 0.7226596675415573,
1299
+ 0.6847977684797768,
1300
+ 0.6956521739130435,
1301
+ 0.7089715536105032
1302
+ ],
1303
+ "sub8000": 0.7039681905253619,
1304
+ "sub8000_n_scored": 7873,
1305
+ "sub8000_per_class": [
1306
+ 0.7210739399105051,
1307
+ 0.6818181818181818,
1308
+ 0.7183060841597427,
1309
+ 0.6946745562130178
1310
+ ],
1311
+ "seconds": 44.4
1312
+ },
1313
+ {
1314
+ "n": 100,
1315
+ "rep": 5,
1316
+ "n_shots": 400,
1317
+ "full": 0.6970234156619769,
1318
+ "full_n_scored": 25592,
1319
+ "full_per_class": [
1320
+ 0.7195786414820197,
1321
+ 0.6650714911000876,
1322
+ 0.7104996623902768,
1323
+ 0.6929438676755235
1324
+ ],
1325
+ "sub2000": 0.688869420562793,
1326
+ "sub2000_n_scored": 1975,
1327
+ "sub2000_per_class": [
1328
+ 0.7059748427672956,
1329
+ 0.6607142857142857,
1330
+ 0.7029478458049887,
1331
+ 0.6858407079646017
1332
+ ],
1333
+ "sub8000": 0.6916788231103608,
1334
+ "sub8000_n_scored": 7893,
1335
+ "sub8000_per_class": [
1336
+ 0.7110162916989915,
1337
+ 0.6641665104069004,
1338
+ 0.7119777158774373,
1339
+ 0.6795547744581136
1340
+ ],
1341
+ "seconds": 43.7
1342
+ },
1343
+ {
1344
+ "n": 100,
1345
+ "rep": 6,
1346
+ "n_shots": 400,
1347
+ "full": 0.7015577779706541,
1348
+ "full_n_scored": 25592,
1349
+ "full_per_class": [
1350
+ 0.7212658227848101,
1351
+ 0.6631010021393987,
1352
+ 0.7084815887463798,
1353
+ 0.7133826982120282
1354
+ ],
1355
+ "sub2000": 0.6977101464965301,
1356
+ "sub2000_n_scored": 1962,
1357
+ "sub2000_per_class": [
1358
+ 0.7040816326530612,
1359
+ 0.6741731175228712,
1360
+ 0.7034949267192785,
1361
+ 0.7090909090909091
1362
+ ],
1363
+ "sub8000": 0.6957511266536937,
1364
+ "sub8000_n_scored": 7872,
1365
+ "sub8000_per_class": [
1366
+ 0.7123793882159721,
1367
+ 0.6647574727044926,
1368
+ 0.7050754458161865,
1369
+ 0.7007921998781231
1370
+ ],
1371
+ "seconds": 45.0
1372
+ },
1373
+ {
1374
+ "n": 100,
1375
+ "rep": 7,
1376
+ "n_shots": 400,
1377
+ "full": 0.6993930933138065,
1378
+ "full_n_scored": 25592,
1379
+ "full_per_class": [
1380
+ 0.7156596794081381,
1381
+ 0.6734982732265187,
1382
+ 0.7083440971325239,
1383
+ 0.700070323488045
1384
+ ],
1385
+ "sub2000": 0.6927554786238601,
1386
+ "sub2000_n_scored": 1977,
1387
+ "sub2000_per_class": [
1388
+ 0.7064516129032258,
1389
+ 0.6714801444043321,
1390
+ 0.6972477064220184,
1391
+ 0.6958424507658644
1392
+ ],
1393
+ "sub8000": 0.6971786214673333,
1394
+ "sub8000_n_scored": 7893,
1395
+ "sub8000_per_class": [
1396
+ 0.7149375123983337,
1397
+ 0.6808510638297872,
1398
+ 0.7087461081234079,
1399
+ 0.684179801517805
1400
+ ],
1401
+ "seconds": 43.7
1402
+ },
1403
+ {
1404
+ "n": 100,
1405
+ "rep": 8,
1406
+ "n_shots": 400,
1407
+ "full": 0.6991747950902661,
1408
+ "full_n_scored": 25592,
1409
+ "full_per_class": [
1410
+ 0.7125256673511293,
1411
+ 0.667913196215083,
1412
+ 0.7251976848455205,
1413
+ 0.6910626319493315
1414
+ ],
1415
+ "sub2000": 0.6887564297762729,
1416
+ "sub2000_n_scored": 1970,
1417
+ "sub2000_per_class": [
1418
+ 0.6941870261162595,
1419
+ 0.661849710982659,
1420
+ 0.7048458149779736,
1421
+ 0.6941431670281996
1422
+ ],
1423
+ "sub8000": 0.6941301594269301,
1424
+ "sub8000_n_scored": 7885,
1425
+ "sub8000_per_class": [
1426
+ 0.7019429516329062,
1427
+ 0.667871340802313,
1428
+ 0.7200866269626421,
1429
+ 0.6866197183098591
1430
+ ],
1431
+ "seconds": 44.4
1432
+ },
1433
+ {
1434
+ "n": 100,
1435
+ "rep": 9,
1436
+ "n_shots": 400,
1437
+ "full": 0.6925297553913796,
1438
+ "full_n_scored": 25592,
1439
+ "full_per_class": [
1440
+ 0.7182587666263603,
1441
+ 0.6724587315377932,
1442
+ 0.7003060633633882,
1443
+ 0.6790954600379768
1444
+ ],
1445
+ "sub2000": 0.6901187483592387,
1446
+ "sub2000_n_scored": 1972,
1447
+ "sub2000_per_class": [
1448
+ 0.7092709270927092,
1449
+ 0.6815415821501014,
1450
+ 0.6957494407158836,
1451
+ 0.6739130434782609
1452
+ ],
1453
+ "sub8000": 0.6894561113837702,
1454
+ "sub8000_n_scored": 7871,
1455
+ "sub8000_per_class": [
1456
+ 0.7115761935905821,
1457
+ 0.6777546777546778,
1458
+ 0.6991247264770241,
1459
+ 0.6693688477127968
1460
+ ],
1461
+ "seconds": 44.8
1462
+ },
1463
+ {
1464
+ "n": 100,
1465
+ "rep": 10,
1466
+ "n_shots": 400,
1467
+ "full": 0.7087579739936903,
1468
+ "full_n_scored": 25592,
1469
+ "full_per_class": [
1470
+ 0.731912721704734,
1471
+ 0.6695611358835954,
1472
+ 0.7198021381841392,
1473
+ 0.7137559002022926
1474
+ ],
1475
+ "sub2000": 0.6897127029583479,
1476
+ "sub2000_n_scored": 1965,
1477
+ "sub2000_per_class": [
1478
+ 0.7061812023708721,
1479
+ 0.6558105107327905,
1480
+ 0.7019438444924406,
1481
+ 0.6949152542372882
1482
+ ],
1483
+ "sub8000": 0.704166307882562,
1484
+ "sub8000_n_scored": 7871,
1485
+ "sub8000_per_class": [
1486
+ 0.7248101785347835,
1487
+ 0.6714232069095005,
1488
+ 0.7198093220338984,
1489
+ 0.7006225240520656
1490
+ ],
1491
+ "seconds": 43.5
1492
+ },
1493
+ {
1494
+ "n": 100,
1495
+ "rep": 11,
1496
+ "n_shots": 400,
1497
+ "full": 0.7054397315745257,
1498
+ "full_n_scored": 25592,
1499
+ "full_per_class": [
1500
+ 0.7395912187736563,
1501
+ 0.6750114836931557,
1502
+ 0.7118727050183599,
1503
+ 0.6952835188129306
1504
+ ],
1505
+ "sub2000": 0.6913508489102866,
1506
+ "sub2000_n_scored": 1975,
1507
+ "sub2000_per_class": [
1508
+ 0.7257663628831814,
1509
+ 0.6647230320699709,
1510
+ 0.6951754385964912,
1511
+ 0.6797385620915033
1512
+ ],
1513
+ "sub8000": 0.6997292183708061,
1514
+ "sub8000_n_scored": 7865,
1515
+ "sub8000_per_class": [
1516
+ 0.7301587301587301,
1517
+ 0.6732782369146005,
1518
+ 0.7130434782608696,
1519
+ 0.6824364281490243
1520
+ ],
1521
+ "seconds": 45.0
1522
+ },
1523
+ {
1524
+ "n": 200,
1525
+ "rep": 0,
1526
+ "n_shots": 800,
1527
+ "full": 0.7126308732675979,
1528
+ "full_n_scored": 25192,
1529
+ "full_per_class": [
1530
+ 0.7305864739959643,
1531
+ 0.6934458444110507,
1532
+ 0.7190252737301391,
1533
+ 0.7074659009332376
1534
+ ],
1535
+ "sub2000": 0.7085476058479627,
1536
+ "sub2000_n_scored": 1944,
1537
+ "sub2000_per_class": [
1538
+ 0.7174103237095363,
1539
+ 0.6960926193921853,
1540
+ 0.7142857142857143,
1541
+ 0.7064017660044151
1542
+ ],
1543
+ "sub8000": 0.7088247983564959,
1544
+ "sub8000_n_scored": 7767,
1545
+ "sub8000_per_class": [
1546
+ 0.7246682115020012,
1547
+ 0.7003131331736968,
1548
+ 0.7191500953418687,
1549
+ 0.6911677534084173
1550
+ ],
1551
+ "seconds": 48.2
1552
+ },
1553
+ {
1554
+ "n": 200,
1555
+ "rep": 1,
1556
+ "n_shots": 800,
1557
+ "full": 0.712263430940032,
1558
+ "full_n_scored": 25192,
1559
+ "full_per_class": [
1560
+ 0.7431735726561008,
1561
+ 0.6780061422159225,
1562
+ 0.7214309373964889,
1563
+ 0.7064430714916152
1564
+ ],
1565
+ "sub2000": 0.7039979119751034,
1566
+ "sub2000_n_scored": 1948,
1567
+ "sub2000_per_class": [
1568
+ 0.7225042301184433,
1569
+ 0.6731054977711739,
1570
+ 0.7155361050328227,
1571
+ 0.7048458149779736
1572
+ ],
1573
+ "sub8000": 0.7097915636104388,
1574
+ "sub8000_n_scored": 7763,
1575
+ "sub8000_per_class": [
1576
+ 0.7340885684860968,
1577
+ 0.6783625730994152,
1578
+ 0.730349344978166,
1579
+ 0.6963657678780774
1580
+ ],
1581
+ "seconds": 41.2
1582
+ },
1583
+ {
1584
+ "n": 200,
1585
+ "rep": 2,
1586
+ "n_shots": 800,
1587
+ "full": 0.710147876753136,
1588
+ "full_n_scored": 25192,
1589
+ "full_per_class": [
1590
+ 0.7323607427055703,
1591
+ 0.6860651974288338,
1592
+ 0.7245699014531485,
1593
+ 0.6975956654249915
1594
+ ],
1595
+ "sub2000": 0.7071637332744507,
1596
+ "sub2000_n_scored": 1935,
1597
+ "sub2000_per_class": [
1598
+ 0.7272727272727273,
1599
+ 0.6814486326681448,
1600
+ 0.7228381374722838,
1601
+ 0.6970954356846473
1602
+ ],
1603
+ "sub8000": 0.7027890498369546,
1604
+ "sub8000_n_scored": 7774,
1605
+ "sub8000_per_class": [
1606
+ 0.7264975484971221,
1607
+ 0.6817429674572532,
1608
+ 0.7204832509610104,
1609
+ 0.6824324324324325
1610
+ ],
1611
+ "seconds": 41.1
1612
+ },
1613
+ {
1614
+ "n": 200,
1615
+ "rep": 3,
1616
+ "n_shots": 800,
1617
+ "full": 0.7139186438938164,
1618
+ "full_n_scored": 25192,
1619
+ "full_per_class": [
1620
+ 0.7215813350615684,
1621
+ 0.6846558426709353,
1622
+ 0.7344632768361582,
1623
+ 0.7149741210066036
1624
+ ],
1625
+ "sub2000": 0.7086510412647167,
1626
+ "sub2000_n_scored": 1951,
1627
+ "sub2000_per_class": [
1628
+ 0.7182978723404255,
1629
+ 0.6808199121522694,
1630
+ 0.724669603524229,
1631
+ 0.7108167770419426
1632
+ ],
1633
+ "sub8000": 0.707901303466429,
1634
+ "sub8000_n_scored": 7752,
1635
+ "sub8000_per_class": [
1636
+ 0.7096234309623431,
1637
+ 0.6837763519706691,
1638
+ 0.7345454545454545,
1639
+ 0.7036599763872491
1640
+ ],
1641
+ "seconds": 41.4
1642
+ },
1643
+ {
1644
+ "n": 200,
1645
+ "rep": 4,
1646
+ "n_shots": 800,
1647
+ "full": 0.718636301061127,
1648
+ "full_n_scored": 25192,
1649
+ "full_per_class": [
1650
+ 0.7340804890473764,
1651
+ 0.6870220094576449,
1652
+ 0.7302932137220699,
1653
+ 0.7231494920174165
1654
+ ],
1655
+ "sub2000": 0.7081455493129629,
1656
+ "sub2000_n_scored": 1929,
1657
+ "sub2000_per_class": [
1658
+ 0.7281964436917866,
1659
+ 0.6933333333333334,
1660
+ 0.7138047138047138,
1661
+ 0.6972477064220184
1662
+ ],
1663
+ "sub8000": 0.7116828386105452,
1664
+ "sub8000_n_scored": 7770,
1665
+ "sub8000_per_class": [
1666
+ 0.7318275154004107,
1667
+ 0.6885368126747438,
1668
+ 0.7296703296703296,
1669
+ 0.6966966966966966
1670
+ ],
1671
+ "seconds": 41.4
1672
+ },
1673
+ {
1674
+ "n": 200,
1675
+ "rep": 5,
1676
+ "n_shots": 800,
1677
+ "full": 0.7159053357442459,
1678
+ "full_n_scored": 25192,
1679
+ "full_per_class": [
1680
+ 0.7451309170643622,
1681
+ 0.6943847627896174,
1682
+ 0.7167630057803468,
1683
+ 0.7073426573426573
1684
+ ],
1685
+ "sub2000": 0.709536245263495,
1686
+ "sub2000_n_scored": 1938,
1687
+ "sub2000_per_class": [
1688
+ 0.7291666666666666,
1689
+ 0.6964809384164223,
1690
+ 0.7096774193548387,
1691
+ 0.702819956616052
1692
+ ],
1693
+ "sub8000": 0.7095375989140849,
1694
+ "sub8000_n_scored": 7753,
1695
+ "sub8000_per_class": [
1696
+ 0.7353493222106361,
1697
+ 0.690511489992587,
1698
+ 0.717096148517595,
1699
+ 0.6951934349355217
1700
+ ],
1701
+ "seconds": 40.9
1702
+ },
1703
+ {
1704
+ "n": 200,
1705
+ "rep": 6,
1706
+ "n_shots": 800,
1707
+ "full": 0.720830638139817,
1708
+ "full_n_scored": 25192,
1709
+ "full_per_class": [
1710
+ 0.758879694462126,
1711
+ 0.6939720004628023,
1712
+ 0.7177783456853224,
1713
+ 0.7126925119490175
1714
+ ],
1715
+ "sub2000": 0.700926963819813,
1716
+ "sub2000_n_scored": 1947,
1717
+ "sub2000_per_class": [
1718
+ 0.7421808960270498,
1719
+ 0.6870451237263464,
1720
+ 0.6945701357466063,
1721
+ 0.6799116997792495
1722
+ ],
1723
+ "sub8000": 0.7113216854149917,
1724
+ "sub8000_n_scored": 7771,
1725
+ "sub8000_per_class": [
1726
+ 0.7491789819376026,
1727
+ 0.689706154130475,
1728
+ 0.7140039447731755,
1729
+ 0.6923976608187135
1730
+ ],
1731
+ "seconds": 41.2
1732
+ },
1733
+ {
1734
+ "n": 200,
1735
+ "rep": 7,
1736
+ "n_shots": 800,
1737
+ "full": 0.7118383061544808,
1738
+ "full_n_scored": 25192,
1739
+ "full_per_class": [
1740
+ 0.7403766358123205,
1741
+ 0.6793680847287142,
1742
+ 0.7180671229487919,
1743
+ 0.709541381128097
1744
+ ],
1745
+ "sub2000": 0.7099530096487899,
1746
+ "sub2000_n_scored": 1922,
1747
+ "sub2000_per_class": [
1748
+ 0.7345890410958904,
1749
+ 0.6852409638554217,
1750
+ 0.7166853303471444,
1751
+ 0.7032967032967034
1752
+ ],
1753
+ "sub8000": 0.7065874945196432,
1754
+ "sub8000_n_scored": 7739,
1755
+ "sub8000_per_class": [
1756
+ 0.7332228666114333,
1757
+ 0.6789724072312083,
1758
+ 0.7178930219929406,
1759
+ 0.6962616822429907
1760
+ ],
1761
+ "seconds": 43.7
1762
+ },
1763
+ {
1764
+ "n": 200,
1765
+ "rep": 8,
1766
+ "n_shots": 800,
1767
+ "full": 0.7149652917652629,
1768
+ "full_n_scored": 25192,
1769
+ "full_per_class": [
1770
+ 0.7509486465975209,
1771
+ 0.6841824328331653,
1772
+ 0.7204743864272772,
1773
+ 0.7042557012030886
1774
+ ],
1775
+ "sub2000": 0.7057496330827919,
1776
+ "sub2000_n_scored": 1933,
1777
+ "sub2000_per_class": [
1778
+ 0.7422334172963896,
1779
+ 0.6833333333333333,
1780
+ 0.7142857142857143,
1781
+ 0.6831460674157304
1782
+ ],
1783
+ "sub8000": 0.7079957561410746,
1784
+ "sub8000_n_scored": 7764,
1785
+ "sub8000_per_class": [
1786
+ 0.7466340269277846,
1787
+ 0.6825667234525837,
1788
+ 0.7163774403470716,
1789
+ 0.686404833836858
1790
+ ],
1791
+ "seconds": 42.9
1792
+ },
1793
+ {
1794
+ "n": 200,
1795
+ "rep": 9,
1796
+ "n_shots": 800,
1797
+ "full": 0.7125345871281019,
1798
+ "full_n_scored": 25192,
1799
+ "full_per_class": [
1800
+ 0.7208996448770222,
1801
+ 0.6787582839204743,
1802
+ 0.7336075489958868,
1803
+ 0.7168728707190246
1804
+ ],
1805
+ "sub2000": 0.6945571524997171,
1806
+ "sub2000_n_scored": 1947,
1807
+ "sub2000_per_class": [
1808
+ 0.7053649956024626,
1809
+ 0.6686046511627907,
1810
+ 0.7155172413793104,
1811
+ 0.6887417218543046
1812
+ ],
1813
+ "sub8000": 0.7067249502339726,
1814
+ "sub8000_n_scored": 7757,
1815
+ "sub8000_per_class": [
1816
+ 0.7136770345121431,
1817
+ 0.6779033154287831,
1818
+ 0.7330655957161981,
1819
+ 0.7022538552787663
1820
+ ],
1821
+ "seconds": 42.9
1822
+ },
1823
+ {
1824
+ "n": 200,
1825
+ "rep": 10,
1826
+ "n_shots": 800,
1827
+ "full": 0.7141411828291568,
1828
+ "full_n_scored": 25192,
1829
+ "full_per_class": [
1830
+ 0.7355947652841982,
1831
+ 0.6862779228314476,
1832
+ 0.7277310924369748,
1833
+ 0.7069609507640068
1834
+ ],
1835
+ "sub2000": 0.7104950356445686,
1836
+ "sub2000_n_scored": 1938,
1837
+ "sub2000_per_class": [
1838
+ 0.729636048526863,
1839
+ 0.6873614190687362,
1840
+ 0.713963963963964,
1841
+ 0.7110187110187111
1842
+ ],
1843
+ "sub8000": 0.7106255772139292,
1844
+ "sub8000_n_scored": 7736,
1845
+ "sub8000_per_class": [
1846
+ 0.7296554639611076,
1847
+ 0.6921790822961174,
1848
+ 0.725752508361204,
1849
+ 0.6949152542372882
1850
+ ],
1851
+ "seconds": 28.6
1852
+ },
1853
+ {
1854
+ "n": 200,
1855
+ "rep": 11,
1856
+ "n_shots": 800,
1857
+ "full": 0.7112820239820997,
1858
+ "full_n_scored": 25192,
1859
+ "full_per_class": [
1860
+ 0.7345115556418722,
1861
+ 0.683256600034967,
1862
+ 0.7239469709682833,
1863
+ 0.7034129692832765
1864
+ ],
1865
+ "sub2000": 0.7120780943084037,
1866
+ "sub2000_n_scored": 1947,
1867
+ "sub2000_per_class": [
1868
+ 0.7297527706734868,
1869
+ 0.6962305986696231,
1870
+ 0.7239819004524887,
1871
+ 0.6983471074380165
1872
+ ],
1873
+ "sub8000": 0.7086064140730504,
1874
+ "sub8000_n_scored": 7763,
1875
+ "sub8000_per_class": [
1876
+ 0.7280334728033473,
1877
+ 0.687037037037037,
1878
+ 0.7248471372984991,
1879
+ 0.6945080091533181
1880
+ ],
1881
+ "seconds": 32.9
1882
+ }
1883
+ ]
1884
+ }
results/full_score_upweighted.json ADDED
@@ -0,0 +1,834 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "variant": "upweighted shots, retrained",
3
+ "key": "upweighted",
4
+ "seed": 42,
5
+ "n_repeats": 5,
6
+ "complete": true,
7
+ "minutes": 35.2,
8
+ "summary": {
9
+ "full": {
10
+ "5": {
11
+ "mean": 0.5657712701607671,
12
+ "std": 0.010763078238031729
13
+ },
14
+ "25": {
15
+ "mean": 0.6283681875535173,
16
+ "std": 0.011971948185350032
17
+ },
18
+ "50": {
19
+ "mean": 0.66966152039057,
20
+ "std": 0.008673459655728825
21
+ },
22
+ "100": {
23
+ "mean": 0.6973059347465853,
24
+ "std": 0.003834880518714044
25
+ },
26
+ "200": {
27
+ "mean": 0.7203877077810819,
28
+ "std": 0.0060460647590979385
29
+ },
30
+ "average": 0.6562989241265044,
31
+ "peak": 0.7203877077810819
32
+ },
33
+ "sub2000": {
34
+ "5": {
35
+ "mean": 0.5659755325794811,
36
+ "std": 0.011462273225451226
37
+ },
38
+ "25": {
39
+ "mean": 0.6291022015454792,
40
+ "std": 0.014826792915094249
41
+ },
42
+ "50": {
43
+ "mean": 0.6689620638167193,
44
+ "std": 0.008095887938584241
45
+ },
46
+ "100": {
47
+ "mean": 0.6961219395643318,
48
+ "std": 0.006245123984700372
49
+ },
50
+ "200": {
51
+ "mean": 0.7230323369848072,
52
+ "std": 0.011642660688516835
53
+ },
54
+ "average": 0.6566388148981637,
55
+ "peak": 0.7230323369848072
56
+ },
57
+ "sub8000": {
58
+ "5": {
59
+ "mean": 0.5609268681598608,
60
+ "std": 0.01047234508091067
61
+ },
62
+ "25": {
63
+ "mean": 0.6259357981888064,
64
+ "std": 0.01260466365218293
65
+ },
66
+ "50": {
67
+ "mean": 0.6663725210444166,
68
+ "std": 0.010295292878052831
69
+ },
70
+ "100": {
71
+ "mean": 0.6948528587219436,
72
+ "std": 0.0029177058143673947
73
+ },
74
+ "200": {
75
+ "mean": 0.7181758724416476,
76
+ "std": 0.005218893626224835
77
+ },
78
+ "average": 0.6532527837113349,
79
+ "peak": 0.7181758724416476
80
+ }
81
+ },
82
+ "draws": [
83
+ {
84
+ "n": 5,
85
+ "rep": 0,
86
+ "n_shots": 20,
87
+ "full": 0.5658086665322979,
88
+ "full_n_scored": 25972,
89
+ "full_per_class": [
90
+ 0.5928475235672602,
91
+ 0.5579149456047472,
92
+ 0.5534818067754078,
93
+ 0.5589903901817761
94
+ ],
95
+ "sub2000": 0.57501534222845,
96
+ "sub2000_n_scored": 2000,
97
+ "sub2000_per_class": [
98
+ 0.5899705014749262,
99
+ 0.5680297397769517,
100
+ 0.5660377358490566,
101
+ 0.5760233918128655
102
+ ],
103
+ "sub8000": 0.56315166517848,
104
+ "sub8000_n_scored": 7995,
105
+ "sub8000_per_class": [
106
+ 0.5910636231180184,
107
+ 0.5671808214086601,
108
+ 0.5535301668806162,
109
+ 0.5408320493066255
110
+ ],
111
+ "seconds": 74.1
112
+ },
113
+ {
114
+ "n": 5,
115
+ "rep": 1,
116
+ "n_shots": 20,
117
+ "full": 0.5755307163274996,
118
+ "full_n_scored": 25972,
119
+ "full_per_class": [
120
+ 0.5934548649242258,
121
+ 0.5673783486663195,
122
+ 0.5781237460877939,
123
+ 0.563165905631659
124
+ ],
125
+ "sub2000": 0.5755930780881062,
126
+ "sub2000_n_scored": 1998,
127
+ "sub2000_per_class": [
128
+ 0.5883495145631068,
129
+ 0.5701624815361891,
130
+ 0.564755838641189,
131
+ 0.5791044776119403
132
+ ],
133
+ "sub8000": 0.5699378066813413,
134
+ "sub8000_n_scored": 7989,
135
+ "sub8000_per_class": [
136
+ 0.580845474086458,
137
+ 0.5711648513937604,
138
+ 0.5795811518324607,
139
+ 0.548159749412686
140
+ ],
141
+ "seconds": 51.6
142
+ },
143
+ {
144
+ "n": 5,
145
+ "rep": 2,
146
+ "n_shots": 20,
147
+ "full": 0.5476248161842202,
148
+ "full_n_scored": 25972,
149
+ "full_per_class": [
150
+ 0.5695845697329377,
151
+ 0.5309744985503816,
152
+ 0.5586407614253127,
153
+ 0.5312994350282486
154
+ ],
155
+ "sub2000": 0.548263502745409,
156
+ "sub2000_n_scored": 1999,
157
+ "sub2000_per_class": [
158
+ 0.5621301775147929,
159
+ 0.5330330330330331,
160
+ 0.5615141955835962,
161
+ 0.536376604850214
162
+ ],
163
+ "sub8000": 0.5430038132164742,
164
+ "sub8000_n_scored": 7994,
165
+ "sub8000_per_class": [
166
+ 0.5634824667472793,
167
+ 0.5402450518378887,
168
+ 0.5556984821198868,
169
+ 0.5125892521608418
170
+ ],
171
+ "seconds": 50.4
172
+ },
173
+ {
174
+ "n": 5,
175
+ "rep": 3,
176
+ "n_shots": 20,
177
+ "full": 0.5685362129160143,
178
+ "full_n_scored": 25972,
179
+ "full_per_class": [
180
+ 0.6126534190531853,
181
+ 0.5615434234445552,
182
+ 0.5600506689889954,
183
+ 0.5398973401773215
184
+ ],
185
+ "sub2000": 0.5611926106997139,
186
+ "sub2000_n_scored": 1997,
187
+ "sub2000_per_class": [
188
+ 0.5916585838991271,
189
+ 0.5439519158527423,
190
+ 0.5581395348837209,
191
+ 0.5510204081632653
192
+ ],
193
+ "sub8000": 0.5621566106140353,
194
+ "sub8000_n_scored": 7990,
195
+ "sub8000_per_class": [
196
+ 0.6044086276368807,
197
+ 0.5598796766309456,
198
+ 0.5587170201758924,
199
+ 0.5256211180124224
200
+ ],
201
+ "seconds": 116.9
202
+ },
203
+ {
204
+ "n": 5,
205
+ "rep": 4,
206
+ "n_shots": 20,
207
+ "full": 0.5713559388438034,
208
+ "full_n_scored": 25972,
209
+ "full_per_class": [
210
+ 0.5914583802446897,
211
+ 0.5779437560119957,
212
+ 0.5587102983638114,
213
+ 0.557311320754717
214
+ ],
215
+ "sub2000": 0.569813129135726,
216
+ "sub2000_n_scored": 1997,
217
+ "sub2000_per_class": [
218
+ 0.5804540967423495,
219
+ 0.5753823743627094,
220
+ 0.5570971184631803,
221
+ 0.5663189269746647
222
+ ],
223
+ "sub8000": 0.5663844451089731,
224
+ "sub8000_n_scored": 7992,
225
+ "sub8000_per_class": [
226
+ 0.5850804485616772,
227
+ 0.5869762379829494,
228
+ 0.5545073375262054,
229
+ 0.5389737563650607
230
+ ],
231
+ "seconds": 179.8
232
+ },
233
+ {
234
+ "n": 25,
235
+ "rep": 0,
236
+ "n_shots": 100,
237
+ "full": 0.6302737351779462,
238
+ "full_n_scored": 25892,
239
+ "full_per_class": [
240
+ 0.6702978952323314,
241
+ 0.5876472688325598,
242
+ 0.6476267748478702,
243
+ 0.6155230017990234
244
+ ],
245
+ "sub2000": 0.6323146358738826,
246
+ "sub2000_n_scored": 1992,
247
+ "sub2000_per_class": [
248
+ 0.6725820763087844,
249
+ 0.5865457294028723,
250
+ 0.6477644492911668,
251
+ 0.6223662884927067
252
+ ],
253
+ "sub8000": 0.6322177365617214,
254
+ "sub8000_n_scored": 7974,
255
+ "sub8000_per_class": [
256
+ 0.673232654847888,
257
+ 0.5971671388101983,
258
+ 0.6550079491255962,
259
+ 0.6034632034632035
260
+ ],
261
+ "seconds": 161.0
262
+ },
263
+ {
264
+ "n": 25,
265
+ "rep": 1,
266
+ "n_shots": 100,
267
+ "full": 0.6184984160990994,
268
+ "full_n_scored": 25892,
269
+ "full_per_class": [
270
+ 0.675191466083151,
271
+ 0.5813030469560969,
272
+ 0.6292063492063492,
273
+ 0.5882928021508005
274
+ ],
275
+ "sub2000": 0.6252231849389315,
276
+ "sub2000_n_scored": 1989,
277
+ "sub2000_per_class": [
278
+ 0.6745240253853128,
279
+ 0.594383775351014,
280
+ 0.6378947368421053,
281
+ 0.5940902021772939
282
+ ],
283
+ "sub8000": 0.6161232340806198,
284
+ "sub8000_n_scored": 7965,
285
+ "sub8000_per_class": [
286
+ 0.6718472468916519,
287
+ 0.5854517012123582,
288
+ 0.6351842241826674,
289
+ 0.5720097640358015
290
+ ],
291
+ "seconds": 252.8
292
+ },
293
+ {
294
+ "n": 25,
295
+ "rep": 2,
296
+ "n_shots": 100,
297
+ "full": 0.6265213468503279,
298
+ "full_n_scored": 25892,
299
+ "full_per_class": [
300
+ 0.6902949267481288,
301
+ 0.594574780058651,
302
+ 0.6242608548741342,
303
+ 0.5969548257203976
304
+ ],
305
+ "sub2000": 0.6143179742821969,
306
+ "sub2000_n_scored": 1997,
307
+ "sub2000_per_class": [
308
+ 0.6788990825688074,
309
+ 0.5727773406766326,
310
+ 0.6078212290502794,
311
+ 0.5977742448330684
312
+ ],
313
+ "sub8000": 0.6196619541280792,
314
+ "sub8000_n_scored": 7970,
315
+ "sub8000_per_class": [
316
+ 0.6834233485666805,
317
+ 0.5927513639906469,
318
+ 0.6243415580815082,
319
+ 0.5781315458734814
320
+ ],
321
+ "seconds": 46.6
322
+ },
323
+ {
324
+ "n": 25,
325
+ "rep": 3,
326
+ "n_shots": 100,
327
+ "full": 0.6477776392710577,
328
+ "full_n_scored": 25892,
329
+ "full_per_class": [
330
+ 0.7180120812740253,
331
+ 0.6435045317220544,
332
+ 0.6294688659142925,
333
+ 0.6001250781738586
334
+ ],
335
+ "sub2000": 0.652890475221296,
336
+ "sub2000_n_scored": 1992,
337
+ "sub2000_per_class": [
338
+ 0.7223230490018149,
339
+ 0.656298600311042,
340
+ 0.6291666666666667,
341
+ 0.6037735849056604
342
+ ],
343
+ "sub8000": 0.6451965860388205,
344
+ "sub8000_n_scored": 7971,
345
+ "sub8000_per_class": [
346
+ 0.7131991051454138,
347
+ 0.6509652509652509,
348
+ 0.6342592592592593,
349
+ 0.5823627287853578
350
+ ],
351
+ "seconds": 46.6
352
+ },
353
+ {
354
+ "n": 25,
355
+ "rep": 4,
356
+ "n_shots": 100,
357
+ "full": 0.6187698003691553,
358
+ "full_n_scored": 25892,
359
+ "full_per_class": [
360
+ 0.642973221225182,
361
+ 0.596907753778447,
362
+ 0.6185633058380716,
363
+ 0.6166349206349206
364
+ ],
365
+ "sub2000": 0.6207647374110891,
366
+ "sub2000_n_scored": 1992,
367
+ "sub2000_per_class": [
368
+ 0.6409048067860509,
369
+ 0.6066420664206642,
370
+ 0.6288770053475936,
371
+ 0.6066350710900474
372
+ ],
373
+ "sub8000": 0.6164794801347909,
374
+ "sub8000_n_scored": 7962,
375
+ "sub8000_per_class": [
376
+ 0.6382488479262672,
377
+ 0.5981928821685414,
378
+ 0.6276987888362296,
379
+ 0.6017774016081253
380
+ ],
381
+ "seconds": 49.6
382
+ },
383
+ {
384
+ "n": 50,
385
+ "rep": 0,
386
+ "n_shots": 200,
387
+ "full": 0.6657850238597829,
388
+ "full_n_scored": 25792,
389
+ "full_per_class": [
390
+ 0.7333892852147154,
391
+ 0.6585910854992356,
392
+ 0.6566231268531132,
393
+ 0.6145365978720677
394
+ ],
395
+ "sub2000": 0.6693202889467833,
396
+ "sub2000_n_scored": 1990,
397
+ "sub2000_per_class": [
398
+ 0.7379182156133829,
399
+ 0.6661665416354089,
400
+ 0.6489925768822906,
401
+ 0.6242038216560509
402
+ ],
403
+ "sub8000": 0.6587402393105977,
404
+ "sub8000_n_scored": 7937,
405
+ "sub8000_per_class": [
406
+ 0.7293421352151149,
407
+ 0.6570193572636722,
408
+ 0.6536842105263158,
409
+ 0.5949152542372881
410
+ ],
411
+ "seconds": 50.7
412
+ },
413
+ {
414
+ "n": 50,
415
+ "rep": 1,
416
+ "n_shots": 200,
417
+ "full": 0.6740992113506705,
418
+ "full_n_scored": 25792,
419
+ "full_per_class": [
420
+ 0.7241239892183289,
421
+ 0.6627476338839574,
422
+ 0.6629086809470124,
423
+ 0.6466165413533834
424
+ ],
425
+ "sub2000": 0.6748175232223903,
426
+ "sub2000_n_scored": 1989,
427
+ "sub2000_per_class": [
428
+ 0.7133872416891285,
429
+ 0.6736370425690814,
430
+ 0.6582010582010582,
431
+ 0.6540447504302926
432
+ ],
433
+ "sub8000": 0.670791186902067,
434
+ "sub8000_n_scored": 7944,
435
+ "sub8000_per_class": [
436
+ 0.7223919685726757,
437
+ 0.6660384470410856,
438
+ 0.6708727655099895,
439
+ 0.6238615664845173
440
+ ],
441
+ "seconds": 75.8
442
+ },
443
+ {
444
+ "n": 50,
445
+ "rep": 2,
446
+ "n_shots": 200,
447
+ "full": 0.6785650147371174,
448
+ "full_n_scored": 25792,
449
+ "full_per_class": [
450
+ 0.7314858096828046,
451
+ 0.6651240061706419,
452
+ 0.6872041850037366,
453
+ 0.6304460580912863
454
+ ],
455
+ "sub2000": 0.6779123990533649,
456
+ "sub2000_n_scored": 1982,
457
+ "sub2000_per_class": [
458
+ 0.7371225577264654,
459
+ 0.6772486772486772,
460
+ 0.6674082313681868,
461
+ 0.6298701298701299
462
+ ],
463
+ "sub8000": 0.6807228928512371,
464
+ "sub8000_n_scored": 7952,
465
+ "sub8000_per_class": [
466
+ 0.7365916955017301,
467
+ 0.6726201786053582,
468
+ 0.6998098342841619,
469
+ 0.6138698630136986
470
+ ],
471
+ "seconds": 79.3
472
+ },
473
+ {
474
+ "n": 50,
475
+ "rep": 3,
476
+ "n_shots": 200,
477
+ "full": 0.6733592680779585,
478
+ "full_n_scored": 25792,
479
+ "full_per_class": [
480
+ 0.7138979010792558,
481
+ 0.6354566385264774,
482
+ 0.6879363533041079,
483
+ 0.6561461794019934
484
+ ],
485
+ "sub2000": 0.6654221219213152,
486
+ "sub2000_n_scored": 1984,
487
+ "sub2000_per_class": [
488
+ 0.7006145741878841,
489
+ 0.6328538985616957,
490
+ 0.670926517571885,
491
+ 0.6572934973637962
492
+ ],
493
+ "sub8000": 0.6670591031920785,
494
+ "sub8000_n_scored": 7921,
495
+ "sub8000_per_class": [
496
+ 0.7130285468984761,
497
+ 0.634025528672128,
498
+ 0.6862326574172892,
499
+ 0.6349496797804208
500
+ ],
501
+ "seconds": 75.8
502
+ },
503
+ {
504
+ "n": 50,
505
+ "rep": 4,
506
+ "n_shots": 200,
507
+ "full": 0.6564990839273208,
508
+ "full_n_scored": 25792,
509
+ "full_per_class": [
510
+ 0.6987918536416983,
511
+ 0.6346898647440863,
512
+ 0.6606104072037613,
513
+ 0.6319042101197373
514
+ ],
515
+ "sub2000": 0.6573379859397431,
516
+ "sub2000_n_scored": 1986,
517
+ "sub2000_per_class": [
518
+ 0.7039413382218148,
519
+ 0.6536144578313253,
520
+ 0.6437768240343348,
521
+ 0.6280193236714976
522
+ ],
523
+ "sub8000": 0.6545491829661025,
524
+ "sub8000_n_scored": 7930,
525
+ "sub8000_per_class": [
526
+ 0.6958867161159811,
527
+ 0.6416382252559727,
528
+ 0.6666666666666666,
529
+ 0.61400512382579
530
+ ],
531
+ "seconds": 75.8
532
+ },
533
+ {
534
+ "n": 100,
535
+ "rep": 0,
536
+ "n_shots": 400,
537
+ "full": 0.7015724606580533,
538
+ "full_n_scored": 25592,
539
+ "full_per_class": [
540
+ 0.7581886989429727,
541
+ 0.6852300242130751,
542
+ 0.7042970996631337,
543
+ 0.658574019813032
544
+ ],
545
+ "sub2000": 0.7054498451890512,
546
+ "sub2000_n_scored": 1979,
547
+ "sub2000_per_class": [
548
+ 0.7592752372735116,
549
+ 0.6898148148148148,
550
+ 0.6991341991341992,
551
+ 0.6735751295336787
552
+ ],
553
+ "sub8000": 0.6991970661760609,
554
+ "sub8000_n_scored": 7885,
555
+ "sub8000_per_class": [
556
+ 0.7551972846839202,
557
+ 0.6843326885880078,
558
+ 0.7103355704697987,
559
+ 0.6469227209625174
560
+ ],
561
+ "seconds": 75.6
562
+ },
563
+ {
564
+ "n": 100,
565
+ "rep": 1,
566
+ "n_shots": 400,
567
+ "full": 0.6984739849914747,
568
+ "full_n_scored": 25592,
569
+ "full_per_class": [
570
+ 0.757210417638047,
571
+ 0.681917211328976,
572
+ 0.699968030690537,
573
+ 0.6548002803083391
574
+ ],
575
+ "sub2000": 0.6901250639689709,
576
+ "sub2000_n_scored": 1963,
577
+ "sub2000_per_class": [
578
+ 0.7511071744906997,
579
+ 0.6833073322932918,
580
+ 0.6780021253985122,
581
+ 0.6480836236933798
582
+ ],
583
+ "sub8000": 0.6932988533770741,
584
+ "sub8000_n_scored": 7873,
585
+ "sub8000_per_class": [
586
+ 0.7516254876462939,
587
+ 0.6824534161490683,
588
+ 0.7043137254901961,
589
+ 0.6348027842227378
590
+ ],
591
+ "seconds": 71.3
592
+ },
593
+ {
594
+ "n": 100,
595
+ "rep": 2,
596
+ "n_shots": 400,
597
+ "full": 0.6913773887112747,
598
+ "full_n_scored": 25592,
599
+ "full_per_class": [
600
+ 0.7481803005008347,
601
+ 0.6678057114962167,
602
+ 0.6891409534385659,
603
+ 0.6603825894094816
604
+ ],
605
+ "sub2000": 0.6964998689575894,
606
+ "sub2000_n_scored": 1971,
607
+ "sub2000_per_class": [
608
+ 0.7528991971454059,
609
+ 0.6872586872586872,
610
+ 0.6861924686192469,
611
+ 0.6596491228070176
612
+ ],
613
+ "sub8000": 0.6913190180359224,
614
+ "sub8000_n_scored": 7873,
615
+ "sub8000_per_class": [
616
+ 0.7485329276244295,
617
+ 0.675223648385842,
618
+ 0.6954936181297213,
619
+ 0.6460258780036968
620
+ ],
621
+ "seconds": 69.3
622
+ },
623
+ {
624
+ "n": 100,
625
+ "rep": 3,
626
+ "n_shots": 400,
627
+ "full": 0.6989650082863244,
628
+ "full_n_scored": 25592,
629
+ "full_per_class": [
630
+ 0.7544545273895475,
631
+ 0.683009708737864,
632
+ 0.7019051479529793,
633
+ 0.6564906490649065
634
+ ],
635
+ "sub2000": 0.6978739077912046,
636
+ "sub2000_n_scored": 1963,
637
+ "sub2000_per_class": [
638
+ 0.743978590544157,
639
+ 0.6902927580893683,
640
+ 0.7007616974972797,
641
+ 0.6564625850340136
642
+ ],
643
+ "sub8000": 0.6952100373158328,
644
+ "sub8000_n_scored": 7869,
645
+ "sub8000_per_class": [
646
+ 0.7509727626459144,
647
+ 0.6890691386635767,
648
+ 0.7073886369698587,
649
+ 0.6334096109839817
650
+ ],
651
+ "seconds": 70.0
652
+ },
653
+ {
654
+ "n": 100,
655
+ "rep": 4,
656
+ "n_shots": 400,
657
+ "full": 0.6961408310857994,
658
+ "full_n_scored": 25592,
659
+ "full_per_class": [
660
+ 0.7297572914542118,
661
+ 0.6625096502167587,
662
+ 0.7057907845579079,
663
+ 0.6865055981143194
664
+ ],
665
+ "sub2000": 0.6906610119148429,
666
+ "sub2000_n_scored": 1977,
667
+ "sub2000_per_class": [
668
+ 0.7171532846715328,
669
+ 0.6661698956780924,
670
+ 0.6907962771458118,
671
+ 0.6885245901639344
672
+ ],
673
+ "sub8000": 0.6952393187048275,
674
+ "sub8000_n_scored": 7873,
675
+ "sub8000_per_class": [
676
+ 0.7325066430469442,
677
+ 0.6708333333333333,
678
+ 0.7066939192641799,
679
+ 0.6709233791748527
680
+ ],
681
+ "seconds": 69.8
682
+ },
683
+ {
684
+ "n": 200,
685
+ "rep": 0,
686
+ "n_shots": 800,
687
+ "full": 0.7264520284114167,
688
+ "full_n_scored": 25192,
689
+ "full_per_class": [
690
+ 0.7794158170294847,
691
+ 0.716264012948864,
692
+ 0.7321570497536163,
693
+ 0.6779712339137017
694
+ ],
695
+ "sub2000": 0.7356052106575132,
696
+ "sub2000_n_scored": 1944,
697
+ "sub2000_per_class": [
698
+ 0.7804428044280443,
699
+ 0.7388342165026495,
700
+ 0.7276550998948476,
701
+ 0.6954887218045113
702
+ ],
703
+ "sub8000": 0.7244993307381107,
704
+ "sub8000_n_scored": 7767,
705
+ "sub8000_per_class": [
706
+ 0.7783516730294183,
707
+ 0.7232500476826245,
708
+ 0.7360749609578344,
709
+ 0.6603206412825652
710
+ ],
711
+ "seconds": 71.0
712
+ },
713
+ {
714
+ "n": 200,
715
+ "rep": 1,
716
+ "n_shots": 800,
717
+ "full": 0.712509515337747,
718
+ "full_n_scored": 25192,
719
+ "full_per_class": [
720
+ 0.7726071936598252,
721
+ 0.6942394741737459,
722
+ 0.71399209486166,
723
+ 0.6691992986557569
724
+ ],
725
+ "sub2000": 0.7080344138130289,
726
+ "sub2000_n_scored": 1948,
727
+ "sub2000_per_class": [
728
+ 0.7576301615798923,
729
+ 0.6897637795275591,
730
+ 0.7144340602284528,
731
+ 0.6703096539162113
732
+ ],
733
+ "sub8000": 0.7113541517684345,
734
+ "sub8000_n_scored": 7763,
735
+ "sub8000_per_class": [
736
+ 0.7650681918169819,
737
+ 0.6929633300297324,
738
+ 0.7242628039317124,
739
+ 0.6631222812953117
740
+ ],
741
+ "seconds": 70.1
742
+ },
743
+ {
744
+ "n": 200,
745
+ "rep": 2,
746
+ "n_shots": 800,
747
+ "full": 0.7180229359779726,
748
+ "full_n_scored": 25192,
749
+ "full_per_class": [
750
+ 0.770847835166365,
751
+ 0.7027319773739319,
752
+ 0.7245050871519836,
753
+ 0.6740068442196102
754
+ ],
755
+ "sub2000": 0.7149225707742534,
756
+ "sub2000_n_scored": 1935,
757
+ "sub2000_per_class": [
758
+ 0.7636022514071295,
759
+ 0.7097767513471901,
760
+ 0.7099476439790576,
761
+ 0.6763636363636364
762
+ ],
763
+ "sub8000": 0.7154840310541581,
764
+ "sub8000_n_scored": 7774,
765
+ "sub8000_per_class": [
766
+ 0.770725096001807,
767
+ 0.7087155963302753,
768
+ 0.7244053774560496,
769
+ 0.6580900544285008
770
+ ],
771
+ "seconds": 70.4
772
+ },
773
+ {
774
+ "n": 200,
775
+ "rep": 3,
776
+ "n_shots": 800,
777
+ "full": 0.7184116256432109,
778
+ "full_n_scored": 25192,
779
+ "full_per_class": [
780
+ 0.7688070929607738,
781
+ 0.6962648041299727,
782
+ 0.7291430422809007,
783
+ 0.6794315632011967
784
+ ],
785
+ "sub2000": 0.7239447557025716,
786
+ "sub2000_n_scored": 1951,
787
+ "sub2000_per_class": [
788
+ 0.7715555555555556,
789
+ 0.7079783113865221,
790
+ 0.7228144989339019,
791
+ 0.6934306569343066
792
+ ],
793
+ "sub8000": 0.7175247037326138,
794
+ "sub8000_n_scored": 7752,
795
+ "sub8000_per_class": [
796
+ 0.7662366061666303,
797
+ 0.6992627085758634,
798
+ 0.7320331285065456,
799
+ 0.672566371681416
800
+ ],
801
+ "seconds": 75.8
802
+ },
803
+ {
804
+ "n": 200,
805
+ "rep": 4,
806
+ "n_shots": 800,
807
+ "full": 0.7265424335350622,
808
+ "full_n_scored": 25192,
809
+ "full_per_class": [
810
+ 0.7739442825188098,
811
+ 0.7075471698113207,
812
+ 0.7328964194373402,
813
+ 0.6917818623727784
814
+ ],
815
+ "sub2000": 0.7326547339766687,
816
+ "sub2000_n_scored": 1929,
817
+ "sub2000_per_class": [
818
+ 0.777676120768527,
819
+ 0.7307398932112891,
820
+ 0.7348242811501597,
821
+ 0.6873786407766991
822
+ ],
823
+ "sub8000": 0.7220171449149205,
824
+ "sub8000_n_scored": 7770,
825
+ "sub8000_per_class": [
826
+ 0.7695019832525342,
827
+ 0.7113421914115156,
828
+ 0.738880167451596,
829
+ 0.6683442375440363
830
+ ],
831
+ "seconds": 78.8
832
+ }
833
+ ]
834
+ }
results/llm_protocol_results_colab.md ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Notebook 5 results (Qwen3-4B LoRA, Amsterdam), transcribed from Colab
2
+
3
+ Run: A100 High-RAM, 2026-11-10, finished about 18:50 AST. The runtime then
4
+ disconnected, so `/content/hack4dev/data/processed/llm_results.pkl` and
5
+ `/content/hack4dev/reports/protocol_results.csv` were written only to the
6
+ Colab VM's disk, not to Drive. They are recovered from the downloaded notebook:
7
+
8
+ - `reports/05_evaluate_and_adapt_colab_run.ipynb`: the executed notebook with
9
+ every cell's output (no errors).
10
+ - `reports/protocol_results.csv`: rebuilt from cell 7's per-draw log
11
+ (30 rows: 5 values of n x 6 repeats).
12
+ - Values are rounded to 4 decimals as printed.
13
+ - The means and stds match the notebook's summary table to within 1e-4.
14
+ - `reports/f1_vs_log2_samples.png`: the plot, taken from the notebook output.
15
+
16
+ `llm_results.pkl` (the zero-shot probabilities) is not recoverable without a
17
+ re-run. Nothing downstream needs it.
18
+
19
+ ## Zero-shot (n = 0, no Amsterdam labels)
20
+
21
+ - Macro F1 **0.5537** on 6,000 random Amsterdam pixels.
22
+ - Bootstrap std 0.0067, 95% interval [0.5411, 0.5669].
23
+ - That band is sampling noise on a single prediction set, not variation
24
+ across labelled draws.
25
+
26
+ ## Protocol: macro F1 mean (std) on a fixed 2,000-pixel evaluation set
27
+
28
+ | n per class | retrieval + Madrid rule | + bias recalibration | + tuned decision weights | ens LLM 0.3 / LGBM 0.7 | ens 0.5 / 0.5 | ens 0.7 / 0.3 |
29
+ |---|---|---|---|---|---|---|
30
+ | 5 | 0.5140 (0.0111) | n/a | n/a | 0.5600 (0.0042) | 0.5490 (0.0497) | 0.5471 (0.0359) |
31
+ | 25 | 0.5174 (0.0165) | n/a | n/a | 0.5647 (0.0011) | 0.4756 (0.0053) | 0.4732 (0.0053) |
32
+ | 50 | 0.5378 (0.0258) | 0.5044 (0.0443) | n/a | 0.5770 (0.0048) | 0.4753 (0.0072) | 0.4754 (0.0057) |
33
+ | 100 | 0.5492 (0.0303) | 0.5298 (0.0590) | n/a | 0.5800 (0.0039) | 0.4825 (0.0030) | 0.4804 (0.0042) |
34
+ | 200 | 0.5677 (0.0223) | 0.5993 (0.0149) | **0.6222** (0.0256) | 0.5847 (0.0039) | 0.4904 (0.0054) | 0.4908 (0.0039) |
35
+
36
+ The "+ bias recalibration" step needs n >= 50 and the "+ tuned decision
37
+ weights" step needs n >= 200, so the other cells are empty by design.
38
+
39
+ Notebook summary for the deliverable rule (as specified: richest step
40
+ available at each n):
41
+
42
+ - 5: 0.5140, 25: 0.5174, 50: 0.5044, 100: 0.5298, 200: 0.6222
43
+ - Average across n: 0.5376. Peak: 0.6222. Low-data (n=25): 0.5174 +- 0.0165.
44
+ - Best single fixed variant: ensemble LLM 0.3 / LightGBM 0.7, with average
45
+ 0.5733, peak 0.5847 and 0.5647 at n=25.
46
+
47
+ ## Against the classical models (notebook 3)
48
+
49
+ Classical best, scored on all of Amsterdam:
50
+ - Average 0.6563 and peak 0.7204, from the upweighted-shots LightGBM.
51
+ - Best at n=25: 0.6361, from the within-class Mahalanobis prototype.
52
+ - Zero-shot LightGBM trained on Madrid only, with no normalisation, scores
53
+ 0.5928 at every n.
54
+
55
+ The LLM zero-shot score of 0.5537 has a 95% interval of [0.541, 0.567], which
56
+ sits clearly below that.
57
+
58
+ The LLM track does not beat them at any n.
59
+
60
+ ## Things to look into
61
+
62
+ 1. **Retrieval hurts.** The zero-shot model (0.554) beats the
63
+ retrieval-from-Amsterdam-shots variant at every n up to 100. The model was
64
+ trained with a large Madrid retrieval pool, and a handful of Amsterdam shots
65
+ gives far, sparse neighbours.
66
+ 2. **The ensemble uses the Madrid-only LightGBM.** `amsterdam_proba_lgbm`
67
+ comes from notebook 3's `final` model and is the same for every n; it never
68
+ sees the Amsterdam shots. That explains why its std is tiny and why it stays
69
+ below the classical track, which retrains with the shots.
70
+ 3. **The ensemble is not monotone in the weight.** At n >= 25, the 0.5/0.5 and
71
+ 0.7/0.3 mixes (about 0.48) score below both of their parts. That needs
72
+ checking:
73
+ - whether the class order of `Pc` and the LightGBM probabilities match;
74
+ - how `macro_f1_plugin`'s prior interacts with the mixed probabilities.
results/logs/01_build_features.log ADDED
@@ -0,0 +1,129 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ platform: local
2
+ looking for raw parquet under: C:\Users\Lenovo\Desktop\hack4dev\data\raw
3
+ Madrid : C:\Users\Lenovo\Desktop\hack4dev\data\raw\madrid_train.parquet
4
+ Amsterdam: C:\Users\Lenovo\Desktop\hack4dev\data\raw\amsterdam_data.parquet
5
+ 3,172,765 rows x 48 columns
6
+
7
+ columns:
8
+ ['pixel_id', 'city', 'year', 'px_key', 'py_key', 'pixel_x', 'pixel_y', 'row', 'col', 'coverage', 'usable_label', 'weighted_mean_year', 'Blue_1', 'Green_1', 'Red_1', 'NIR_1', 'SWIR1_1', 'SWIR2_1', 'QA_PIXEL_1', 'QA_RADSAT_1', 'doy_1', 'scene_id_1', 'dataset_1', 'qa_valid_1', 'Blue_2', 'Green_2', 'Red_2', 'NIR_2', 'SWIR1_2', 'SWIR2_2', 'QA_PIXEL_2', 'QA_RADSAT_2', 'doy_2', 'scene_id_2', 'dataset_2', 'qa_valid_2', 'Blue_3', 'Green_3', 'Red_3', 'NIR_3', 'SWIR1_3', 'SWIR2_3', 'QA_PIXEL_3', 'QA_RADSAT_3', 'doy_3', 'scene_id_3', 'dataset_3', 'qa_valid_3']
9
+
10
+ years: 1984 - 2025 (42 distinct)
11
+ unique pixels: 76,263
12
+
13
+ first row:
14
+ pixel_id Madrid_14194_149671_1984
15
+ city Madrid
16
+ year 1984
17
+ px_key 14194
18
+ py_key 149671
19
+ pixel_x 425820.0
20
+ pixel_y 4490130.0
21
+ row 5
22
+ col 20
23
+ coverage 0.15
24
+ usable_label True
25
+ weighted_mean_year 2014.0
26
+ Blue_1 9184.207031
27
+ Green_1 9821.585938
28
+ Red_1 10537.317383
29
+ NIR_1 15736.041016
30
+ SWIR1_1 16615.345703
31
+ SWIR2_1 13306.133789
32
+ QA_PIXEL_1 5440
33
+ QA_RADSAT_1 0
34
+ doy_1 199
35
+ scene_id_1 LT05_L2SP_201032_19840717_20200918_02_T1
36
+ dataset_1 landsat_tm_c2_l2
37
+ qa_valid_1 True
38
+ Blue_2 9511.026367
39
+ Green_2 10161.28125
40
+ Red_2 10884.139648
41
+ NIR_2 15506.376953
42
+ SWIR1_2 16767.910156
43
+ SWIR2_2 13043.585938
44
+ QA_PIXEL_2 5440
45
+ QA_RADSAT_2 0
46
+ doy_2 231
47
+ scene_id_2 LT05_L2SP_201032_19840818_20200918_02_T1
48
+ dataset_2 landsat_tm_c2_l2
49
+ qa_valid_2 True
50
+ Blue_3 9323.154297
51
+ Green_3 10433.227539
52
+ Red_3 11603.330078
53
+ NIR_3 16555.982422
54
+ SWIR1_3 18056.576172
55
+ SWIR2_3 13991.788086
56
+ QA_PIXEL_3 5440
57
+ QA_RADSAT_3 0
58
+ doy_3 247
59
+ scene_id_3 LT05_L2SP_201032_19840903_20201008_02_T2
60
+ dataset_3 landsat_tm_c2_l2
61
+ qa_valid_3 True
62
+ Red_1 digital numbers : {0.0: 0.0, 0.01: 0.0, 0.5: 12419.0, 0.99: 18727.0, 1.0: 65454.0}
63
+ Red_1 as reflectance : {0.0: -0.2, 0.01: -0.2, 0.5: 0.142, 0.99: 0.315, 1.0: 1.6}
64
+ qa_valid_1: 95.0% valid
65
+ qa_valid_2: 95.7% valid
66
+ qa_valid_3: 90.1% valid
67
+ reading C:\Users\Lenovo\Desktop\hack4dev\data\raw\madrid_train.parquet
68
+ 3,172,765 rows x 48 cols | years 1984-2025
69
+ removed 36,017 haze rows (Blue > 15,000)
70
+ -> 76,263 labelled pixels | 60 official + 316 extended features
71
+ reading C:\Users\Lenovo\Desktop\hack4dev\data\raw\amsterdam_data.parquet
72
+ 1,062,877 rows x 48 cols | years 1984-2025
73
+ removed 6,340 haze rows (Blue > 15,000)
74
+ -> 25,992 labelled pixels | 60 official + 316 extended features
75
+ preprocessed_data.pkl not found - skipping the cross-check.
76
+
77
+ Madrid: 76,263 labelled pixels
78
+ class 1 (before 1960): 13,794 18.1%
79
+ class 2 ( 1960-1984): 27,152 35.6%
80
+ class 3 ( 1984-2004): 18,672 24.5%
81
+ class 4 ( 2004-2024): 16,645 21.8%
82
+
83
+ Amsterdam: 25,992 labelled pixels
84
+ class 1 (before 1945): 7,568 29.1%
85
+ class 2 ( 1945-1984): 8,802 33.9%
86
+ class 3 ( 1984-2004): 6,745 26.0%
87
+ class 4 ( 2004-2024): 2,877 11.1%
88
+ madrid_mean amsterdam_mean madrid_sd gap_in_madrid_sd
89
+ sr_Red_mean 0.151 0.083 0.035 -1.971
90
+ sr_brightness_mean 0.173 0.113 0.033 -1.857
91
+ sr_SWIR1_mean 0.245 0.162 0.050 -1.648
92
+ sr_NIR_mean 0.225 0.175 0.038 -1.307
93
+ sr_BSI_mean 0.103 0.022 0.066 -1.218
94
+ sr_UI_mean -0.055 -0.163 0.116 -0.926
95
+ sr_NDBI_mean 0.034 -0.034 0.079 -0.861
96
+ sr_brightness_std 0.028 0.021 0.014 -0.530
97
+ sr_MNDWI_mean -0.347 -0.381 0.072 -0.470
98
+ coverage 0.505 0.421 0.251 -0.334
99
+ step_brightness_njumps 9.831 9.781 4.004 -0.012
100
+ step_brightness_mag -0.003 0.001 0.057 0.079
101
+ sr_brightness_late_minus_early -0.019 -0.008 0.030 0.364
102
+ sr_NDVI_mean 0.204 0.355 0.099 1.534
103
+ C:\Users\Lenovo\Desktop\hack4dev\notebooks\src\01_build_features.py:220: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
104
+ plt.show()
105
+
106
+ Madrid
107
+ step_brightness_year vs true year (classes 3+4): r=+0.206 median |error| = 6.1 years within 5 y: 44.6%
108
+ step_SWIR1_year vs true year (classes 3+4): r=+0.240 median |error| = 6.0 years within 5 y: 47.6%
109
+ step_NDVI_year vs true year (classes 3+4): r=+0.130 median |error| = 7.0 years within 5 y: 41.1%
110
+ mean |step magnitude| by class: {1: 0.0356, 2: 0.0399, 3: 0.054, 4: 0.0687}
111
+
112
+ Amsterdam
113
+ step_brightness_year vs true year (classes 3+4): r=+0.481 median |error| = 3.0 years within 5 y: 69.4%
114
+ step_SWIR1_year vs true year (classes 3+4): r=+0.473 median |error| = 2.7 years within 5 y: 70.0%
115
+ step_NDVI_year vs true year (classes 3+4): r=+0.321 median |error| = 3.0 years within 5 y: 64.5%
116
+ mean |step magnitude| by class: {1: 0.0216, 2: 0.0264, 3: 0.0505, 4: 0.0647}
117
+ C:\Users\Lenovo\Desktop\hack4dev\notebooks\src\01_build_features.py:256: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
118
+ plt.show()
119
+ added 15 neighbourhood features
120
+ nbr_sr_brightness_mean nbr_sr_brightness_mean_diff nbr_sr_NDVI_mean nbr_sr_NDVI_mean_diff nbr_sr_SWIR1_mean nbr_sr_SWIR1_mean_diff
121
+ age_class
122
+ 1 0.1625 -0.0003 0.1808 0.0044 0.2340 0.0006
123
+ 2 0.1658 -0.0011 0.2074 0.0015 0.2335 -0.0017
124
+ 3 0.1747 -0.0001 0.2051 0.0007 0.2440 -0.0008
125
+ 4 0.1925 0.0015 0.2087 -0.0000 0.2747 0.0023
126
+ C:\Users\Lenovo\Desktop\hack4dev\data\processed\madrid_features.parquet 165.5 MB (76263, 400)
127
+ C:\Users\Lenovo\Desktop\hack4dev\data\processed\amsterdam_features.parquet 56.2 MB (25992, 400)
128
+
129
+ feature groups: 60 official, 276 reflectance-based, 15 change-event, 15 neighbourhood
results/logs/03_gbdt_models.log ADDED
@@ -0,0 +1,305 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ feature source: notebook 1 (official 60 + extended)
3
+ (76263, 400) (25992, 400)
4
+ dead (zero variance in at least one city): ['has_early_data', 'sat_share_oli_early']
5
+ official60 59 features
6
+ reflectance 276 features
7
+ events 15 features
8
+ neighbourhood 15 features
9
+ everything 393 features
10
+ block sizes: [3933 4437 785 2982 3078 3441 2820 3521 2363 3020 2929 2330 3738 2660
11
+ 3573 2154 3367 3580 2947 3250 2674 2376 3575 4257 2473]
12
+ C:\Users\Lenovo\Desktop\hack4dev\notebooks\src\03_gbdt_models.py:115: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
13
+ plt.show()
14
+ city-vs-city AUC: 1.0000 (1.0 = the two cities are trivially told apart)
15
+ feature city_importance
16
+ sat_share_oli_late 0.036333
17
+ sat_share_tm_early 0.035556
18
+ sat_share_etm_late 0.034222
19
+ sat_switches 0.032667
20
+ nbr_sr_brightness_mean 0.032000
21
+ sr_NDVI_early_std 0.030556
22
+ sat_share_etm_early 0.028778
23
+ sr_BSI_dec84_93_mean 0.026222
24
+ sat_share_oli 0.025778
25
+ sat_share_etm 0.024333
26
+ sr_NDVI_dec84_93_mean 0.023444
27
+ nbr_sr_SWIR1_mean 0.020111
28
+ radial_frac 0.019778
29
+ nbr_sr_NDVI_mean 0.016556
30
+ radial_px 0.016111
31
+ sr_Red_dec84_93_mean 0.014667
32
+ sr_NDVI_max 0.014444
33
+ n_observed_years 0.013444
34
+ dist_to_water_px 0.013000
35
+ sat_share_tm 0.011333
36
+ worst offenders, raw vs after per-city rank normalisation:
37
+ feature auc_raw auc_after_rank
38
+ sat_share_etm_late 0.990 0.971
39
+ sat_share_oli_late 0.989 0.593
40
+ sat_share_oli 0.988 0.579
41
+ sat_share_etm 0.972 0.607
42
+ sr_Red_dec84_93_mean 0.969 0.500
43
+ nbr_sr_brightness_mean 0.966 0.500
44
+ sr_Red_early_mean 0.963 0.500
45
+ Red_early_mean 0.963 0.500
46
+ sr_Red_tm_mean 0.957 0.500
47
+ nbr_sr_SWIR1_mean 0.957 0.500
48
+ sr_SWIR2_dec84_93_mean 0.955 0.500
49
+ sr_brightness_dec84_93_mean 0.951 0.500
50
+ sr_Red_min 0.951 0.500
51
+ sr_Red_mean 0.951 0.500
52
+ Red_mean 0.951 0.500
53
+ sr_Green_dec84_93_mean 0.948 0.500
54
+ sr_brightness_early_mean 0.947 0.500
55
+ SWIR2_early_mean 0.945 0.500
56
+ sr_SWIR2_early_mean 0.945 0.500
57
+ Green_early_mean 0.944 0.500
58
+ sr_Green_early_mean 0.944 0.500
59
+ sr_brightness_mean 0.941 0.500
60
+ sat_switches 0.941 0.906
61
+ sr_brightness_tm_mean 0.940 0.500
62
+ sr_TCB_early_mean 0.939 0.500
63
+
64
+ median domain AUC: raw 0.750 -> after rank 0.500
65
+ features still above 0.90 after normalisation: 5 of 393
66
+ C:\Users\Lenovo\Desktop\hack4dev\notebooks\src\03_gbdt_models.py:183: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
67
+ plt.show()
68
+ AUC raw 1.0000 | robust-z 1.0000 | rank 1.0000
69
+
70
+ still city-specific after rank normalisation:
71
+ feature city_importance
72
+ sat_share_tm 0.109133
73
+ n_observed_years 0.094531
74
+ sat_share_tm_late 0.083899
75
+ sat_share_etm_late 0.072243
76
+ sat_switches 0.053670
77
+ sat_share_etm 0.043294
78
+ sat_share_oli 0.043294
79
+ sat_share_oli_late 0.041629
80
+ sat_share_tm_early 0.033944
81
+ sat_share_etm_early 0.017805
82
+ step_SWIR1_njumps 0.015883
83
+ last_observed_year 0.014090
84
+ features still above AUC 0.9 after normalisation: 5
85
+ dropping 4: ['sat_share_etm_late', 'sat_switches', 'sat_share_tm_late', 'n_observed_years']
86
+ pruned set: 389 features
87
+
88
+ most city-specific features that we KEPT because they carry age signal:
89
+ feature auc_raw auc_after_rank age_importance
90
+ n_observed_years 0.6849 1.0000 0.0049
91
+ sat_share_tm_late 0.7312 0.9943 0.0016
92
+ sat_share_tm 0.5642 0.9912 0.0064
93
+ sat_share_etm_late 0.9901 0.9715 0.0011
94
+ sat_switches 0.9410 0.9063 0.0013
95
+ pre_state 0.7931 0.7109 0.0001
96
+ sat_share_etm 0.9718 0.6065 0.0037
97
+ sat_share_oli_late 0.9890 0.5930 0.0015
98
+ official60 lgbm 0.5557 +- 0.0540 (4.1 min)
99
+ official60 lgbm_unweighted 0.5504 +- 0.0633 (4.2 min)
100
+ official60 lgbm_ordinal 0.5462 +- 0.0637 (3.2 min)
101
+ official60 catboost 0.5446 +- 0.0480 (7.5 min)
102
+ everything lgbm 0.5702 +- 0.0703 (12.4 min)
103
+ everything lgbm_unweighted 0.5690 +- 0.0830 (11.1 min)
104
+ everything lgbm_ordinal 0.5623 +- 0.0912 (11.0 min)
105
+ pruned lgbm 0.5693 +- 0.0730 (12.9 min)
106
+ pruned lgbm_unweighted 0.5688 +- 0.0853 (12.1 min)
107
+ pruned lgbm_ordinal 0.5631 +- 0.0897 (10.2 min)
108
+
109
+ features model n_feat macro_f1 std minutes
110
+ everything lgbm 393 0.5702 0.0703 12.4381
111
+ pruned lgbm 389 0.5693 0.0730 12.8702
112
+ everything lgbm_unweighted 393 0.5690 0.0830 11.1390
113
+ pruned lgbm_unweighted 389 0.5688 0.0853 12.1380
114
+ pruned lgbm_ordinal 389 0.5631 0.0897 10.1615
115
+ everything lgbm_ordinal 393 0.5623 0.0912 10.9793
116
+ official60 lgbm 59 0.5557 0.0540 4.0525
117
+ official60 lgbm_unweighted 59 0.5504 0.0633 4.1540
118
+ official60 lgbm_ordinal 59 0.5462 0.0637 3.2285
119
+ official60 catboost 59 0.5446 0.0480 7.5037
120
+ normalise=None 0.5989 +- 0.0230
121
+ normalise=robustz 0.5690 +- 0.0830
122
+ normalise=rank 0.5697 +- 0.0789
123
+
124
+ normalise macro_f1 std
125
+ None 0.5989 0.0230
126
+ rank 0.5697 0.0789
127
+ robustz 0.5690 0.0830
128
+
129
+ normalisation chosen on Madrid: None
130
+ Whatever Amsterdam says later, this is the choice the rules allow us to
131
+ make. If Amsterdam disagrees, that is a finding to report, not a reason
132
+ to switch.
133
+ baseline argmax, Madrid block-CV: 0.5761
134
+ tuned class weights: [1.058 1.333 0.943 0.666]
135
+ after weighting: 0.5821
136
+ spatial smoothing r= 2.0: 0.5906
137
+ spatial smoothing r= 4.0: 0.5868
138
+ spatial smoothing r= 8.0: 0.5835
139
+ pred 1 pred 2 pred 3 pred 4
140
+ true 1 0.446 0.427 0.083 0.044
141
+ true 2 0.170 0.681 0.091 0.058
142
+ true 3 0.069 0.221 0.587 0.123
143
+ true 4 0.056 0.172 0.183 0.589
144
+ regime both clean covariate label delta_vs_argmax_mean delta_vs_argmax_worst
145
+ strategy
146
+ argmax 0.4766 0.6056 0.5329 0.5392 0.0000 0.0000
147
+ em_free 0.3589 0.5312 0.3768 0.4900 -0.0993 -0.1079
148
+ em_pinned_shrunk 0.4406 0.5838 0.4792 0.5267 -0.0310 -0.0333
149
+ em_shrunk 0.4563 0.5895 0.4965 0.5314 -0.0201 -0.0063
150
+ logit_adj_tau0.25 0.4803 0.6093 0.5369 0.5438 0.0040 0.0010
151
+ logit_adj_tau0.5 0.4819 0.6102 0.5392 0.5469 0.0060 0.0014
152
+ logit_adj_tau1.0 0.4806 0.6068 0.5351 0.5477 0.0040 0.0156
153
+ plugin_srcprior 0.4885 0.6143 0.5524 0.5460 0.0117 0.0211
154
+ tau0.5_em_plugin 0.4693 0.6030 0.5212 0.5428 -0.0045 0.0159
155
+ tau0.5_then_plugin 0.4889 0.6134 0.5531 0.5472 0.0121 0.0267
156
+ rules that never lose to plain argmax in any regime:
157
+ regime both clean covariate label delta_vs_argmax_mean delta_vs_argmax_worst
158
+ strategy
159
+ argmax 0.4766 0.6056 0.5329 0.5392 0.0000 0.0000
160
+ logit_adj_tau0.25 0.4803 0.6093 0.5369 0.5438 0.0040 0.0010
161
+ logit_adj_tau0.5 0.4819 0.6102 0.5392 0.5469 0.0060 0.0014
162
+ logit_adj_tau1.0 0.4806 0.6068 0.5351 0.5477 0.0040 0.0156
163
+ plugin_srcprior 0.4885 0.6143 0.5524 0.5460 0.0117 0.0211
164
+ tau0.5_em_plugin 0.4693 0.6030 0.5212 0.5428 -0.0045 0.0159
165
+ tau0.5_then_plugin 0.4889 0.6134 0.5531 0.5472 0.0121 0.0267
166
+
167
+ rules to avoid:
168
+ regime both clean covariate label delta_vs_argmax_mean delta_vs_argmax_worst
169
+ strategy
170
+ em_free 0.3589 0.5312 0.3768 0.4900 -0.0993 -0.1079
171
+ em_pinned_shrunk 0.4406 0.5838 0.4792 0.5267 -0.0310 -0.0333
172
+
173
+ rule chosen on Madrid alone: tau0.5_then_plugin
174
+ Madrid class prior: [0.181 0.356 0.245 0.218]
175
+
176
+ organisers' prototype (Euclidean, Madrid scaler)
177
+ n=5 macro F1 = 0.5133 +- 0.0277 per-class [0.554 0.493 0.51 0.496]
178
+ n=25 macro F1 = 0.5805 +- 0.0161 per-class [0.627 0.563 0.568 0.565]
179
+ n=50 macro F1 = 0.5914 +- 0.0150 per-class [0.65 0.562 0.578 0.576]
180
+ n=100 macro F1 = 0.6000 +- 0.0069 per-class [0.648 0.599 0.581 0.572]
181
+ n=200 macro F1 = 0.5921 +- 0.0052 per-class [0.63 0.593 0.576 0.569]
182
+ (12 draws per n, 0.6 min)
183
+
184
+ pure retrieval k-NN (no source model)
185
+ n=5 macro F1 = 0.3834 +- 0.0652 per-class [0.286 0.446 0.385 0.417]
186
+ n=25 macro F1 = 0.6165 +- 0.0110 per-class [0.673 0.594 0.641 0.558]
187
+ n=50 macro F1 = 0.6301 +- 0.0098 per-class [0.693 0.609 0.653 0.566]
188
+ n=100 macro F1 = 0.6496 +- 0.0060 per-class [0.714 0.632 0.679 0.573]
189
+ n=200 macro F1 = 0.6646 +- 0.0044 per-class [0.73 0.653 0.686 0.589]
190
+ (12 draws per n, 4.7 min)
191
+
192
+ zero-shot, no normalisation
193
+ n=5 macro F1 = 0.5871 +- 0.0002 per-class [0.583 0.55 0.621 0.594]
194
+ n=25 macro F1 = 0.5868 +- 0.0004 per-class [0.584 0.549 0.621 0.593]
195
+ n=50 macro F1 = 0.5867 +- 0.0005 per-class [0.583 0.55 0.621 0.593]
196
+ n=100 macro F1 = 0.5858 +- 0.0004 per-class [0.583 0.55 0.62 0.59 ]
197
+ n=200 macro F1 = 0.5852 +- 0.0013 per-class [0.584 0.55 0.621 0.587]
198
+ (12 draws per n, 9.7 min)
199
+
200
+ zero-shot, robust z
201
+ n=5 macro F1 = 0.5245 +- 0.0002 per-class [0.513 0.532 0.541 0.513]
202
+ n=25 macro F1 = 0.5243 +- 0.0004 per-class [0.513 0.532 0.54 0.512]
203
+ n=50 macro F1 = 0.5242 +- 0.0006 per-class [0.513 0.532 0.54 0.512]
204
+ n=100 macro F1 = 0.5229 +- 0.0005 per-class [0.513 0.532 0.54 0.507]
205
+ n=200 macro F1 = 0.5222 +- 0.0013 per-class [0.513 0.531 0.54 0.504]
206
+ (12 draws per n, 10.9 min)
207
+
208
+ zero-shot, rank
209
+ n=5 macro F1 = 0.5542 +- 0.0001 per-class [0.563 0.571 0.562 0.522]
210
+ n=25 macro F1 = 0.5541 +- 0.0003 per-class [0.563 0.571 0.562 0.521]
211
+ n=50 macro F1 = 0.5540 +- 0.0007 per-class [0.563 0.571 0.562 0.521]
212
+ n=100 macro F1 = 0.5527 +- 0.0006 per-class [0.563 0.571 0.561 0.516]
213
+ n=200 macro F1 = 0.5523 +- 0.0013 per-class [0.563 0.571 0.563 0.513]
214
+ (12 draws per n, 13.7 min)
215
+
216
+ zero-shot, robust z + CORAL
217
+ n=5 macro F1 = 0.5377 +- 0.0001 per-class [0.544 0.516 0.564 0.527]
218
+ n=25 macro F1 = 0.5375 +- 0.0004 per-class [0.544 0.516 0.564 0.526]
219
+ n=50 macro F1 = 0.5375 +- 0.0006 per-class [0.544 0.516 0.564 0.526]
220
+ n=100 macro F1 = 0.5363 +- 0.0007 per-class [0.544 0.516 0.563 0.522]
221
+ n=200 macro F1 = 0.5355 +- 0.0012 per-class [0.544 0.516 0.564 0.518]
222
+ (12 draws per n, 10.9 min)
223
+
224
+ prototype, robust z + euclidean
225
+ n=5 macro F1 = 0.4515 +- 0.0671 per-class [0.416 0.435 0.466 0.489]
226
+ n=25 macro F1 = 0.4148 +- 0.1209 per-class [0.42 0.342 0.475 0.422]
227
+ n=50 macro F1 = 0.4335 +- 0.1161 per-class [0.497 0.328 0.491 0.419]
228
+ n=100 macro F1 = 0.4524 +- 0.0918 per-class [0.433 0.308 0.554 0.514]
229
+ n=200 macro F1 = 0.2526 +- 0.1068 per-class [0.113 0.199 0.316 0.383]
230
+ (12 draws per n, 14.7 min)
231
+
232
+ prototype, rank + euclidean
233
+ n=5 macro F1 = 0.5181 +- 0.0402 per-class [0.587 0.503 0.517 0.466]
234
+ n=25 macro F1 = 0.5939 +- 0.0157 per-class [0.643 0.574 0.634 0.524]
235
+ n=50 macro F1 = 0.6041 +- 0.0146 per-class [0.656 0.576 0.651 0.534]
236
+ n=100 macro F1 = 0.6192 +- 0.0068 per-class [0.668 0.602 0.653 0.554]
237
+ n=200 macro F1 = 0.6189 +- 0.0055 per-class [0.665 0.609 0.651 0.551]
238
+ (12 draws per n, 19.8 min)
239
+
240
+ prototype, robust z + total-cov Mahalanobis
241
+ n=5 macro F1 = 0.4181 +- 0.0458 per-class [0.492 0.451 0.327 0.403]
242
+ n=25 macro F1 = 0.6097 +- 0.0205 per-class [0.622 0.564 0.626 0.626]
243
+ n=50 macro F1 = 0.6460 +- 0.0513 per-class [0.676 0.581 0.673 0.654]
244
+ n=100 macro F1 = 0.6865 +- 0.0102 per-class [0.71 0.661 0.699 0.676]
245
+ n=200 macro F1 = 0.6992 +- 0.0031 per-class [0.716 0.679 0.717 0.685]
246
+ (12 draws per n, 17.1 min)
247
+
248
+ prototype, robust z + within-class Mahalanobis
249
+ n=5 macro F1 = 0.5025 +- 0.0406 per-class [0.55 0.491 0.482 0.487]
250
+ n=25 macro F1 = 0.6334 +- 0.0192 per-class [0.64 0.594 0.663 0.636]
251
+ n=50 macro F1 = 0.6695 +- 0.0132 per-class [0.69 0.636 0.692 0.66 ]
252
+ n=100 macro F1 = 0.6967 +- 0.0060 per-class [0.718 0.673 0.71 0.687]
253
+ n=200 macro F1 = 0.7085 +- 0.0024 per-class [0.73 0.686 0.724 0.694]
254
+ (12 draws per n, 17.3 min)
255
+
256
+ recalibrated (stratification-corrected)
257
+ n=5 macro F1 = 0.5309 +- 0.0083 per-class [0.499 0.546 0.549 0.529]
258
+ n=25 macro F1 = 0.5349 +- 0.0042 per-class [0.517 0.545 0.55 0.528]
259
+ n=50 macro F1 = 0.5452 +- 0.0081 per-class [0.53 0.547 0.561 0.542]
260
+ n=100 macro F1 = 0.5514 +- 0.0167 per-class [0.542 0.539 0.573 0.551]
261
+ n=200 macro F1 = 0.5591 +- 0.0054 per-class [0.55 0.546 0.583 0.557]
262
+ (12 draws per n, 25.8 min)
263
+
264
+ refit leaf values (structure frozen)
265
+ n=5 macro F1 = 0.5702 +- 0.0056 per-class [0.571 0.584 0.582 0.545]
266
+ n=25 macro F1 = 0.5999 +- 0.0070 per-class [0.633 0.596 0.607 0.564]
267
+ n=50 macro F1 = 0.6155 +- 0.0063 per-class [0.654 0.61 0.622 0.577]
268
+ n=100 macro F1 = 0.6314 +- 0.0040 per-class [0.673 0.629 0.638 0.586]
269
+ n=200 macro F1 = 0.6437 +- 0.0019 per-class [0.687 0.64 0.653 0.594]
270
+ (12 draws per n, 11.1 min)
271
+
272
+ upweighted shots, retrained
273
+ n=5 macro F1 = 0.5609 +- 0.0105 per-class [0.585 0.565 0.56 0.533]
274
+ n=25 macro F1 = 0.6259 +- 0.0126 per-class [0.676 0.605 0.635 0.588]
275
+ n=50 macro F1 = 0.6664 +- 0.0103 per-class [0.719 0.654 0.675 0.616]
276
+ n=100 macro F1 = 0.6949 +- 0.0029 per-class [0.748 0.68 0.705 0.646]
277
+ n=200 macro F1 = 0.7182 +- 0.0052 per-class [0.77 0.707 0.731 0.664]
278
+ (5 draws per n, 56.7 min)
279
+
280
+ catboost recalibrated
281
+ n=5 macro F1 = 0.5453 +- 0.0052 per-class [0.586 0.523 0.554 0.519]
282
+ n=25 macro F1 = 0.5477 +- 0.0036 per-class [0.592 0.523 0.555 0.521]
283
+ n=50 macro F1 = 0.5487 +- 0.0061 per-class [0.597 0.518 0.557 0.523]
284
+ n=100 macro F1 = 0.5550 +- 0.0068 per-class [0.597 0.51 0.574 0.539]
285
+ n=200 macro F1 = 0.5541 +- 0.0070 per-class [0.598 0.509 0.574 0.535]
286
+ (12 draws per n, 24.8 min)
287
+ 5 25 50 100 200 average peak
288
+ upweighted shots, retrained 0.5609 0.6259 0.6664 0.6949 0.7182 0.6533 0.7182
289
+ prototype, robust z + within-class Mahalanobis 0.5025 0.6334 0.6695 0.6967 0.7085 0.6421 0.7085
290
+ refit leaf values (structure frozen) 0.5702 0.5999 0.6155 0.6314 0.6437 0.6121 0.6437
291
+ prototype, robust z + total-cov Mahalanobis 0.4181 0.6097 0.6460 0.6865 0.6992 0.6119 0.6992
292
+ prototype, rank + euclidean 0.5181 0.5939 0.6041 0.6192 0.6189 0.5908 0.6192
293
+ pure retrieval k-NN (no source model) 0.3834 0.6165 0.6301 0.6496 0.6646 0.5889 0.6646
294
+ zero-shot, no normalisation 0.5871 0.5868 0.5867 0.5858 0.5852 0.5863 0.5871
295
+ organisers' prototype (Euclidean, Madrid scaler) 0.5133 0.5805 0.5914 0.6000 0.5921 0.5754 0.6000
296
+ zero-shot, rank 0.5542 0.5541 0.5540 0.5527 0.5523 0.5535 0.5542
297
+ catboost recalibrated 0.5453 0.5477 0.5487 0.5550 0.5541 0.5502 0.5550
298
+ recalibrated (stratification-corrected) 0.5309 0.5349 0.5452 0.5514 0.5591 0.5443 0.5591
299
+ zero-shot, robust z + CORAL 0.5377 0.5375 0.5375 0.5363 0.5355 0.5369 0.5377
300
+ zero-shot, robust z 0.5245 0.5243 0.5242 0.5229 0.5222 0.5236 0.5245
301
+ prototype, robust z + euclidean 0.4515 0.4148 0.4335 0.4524 0.2526 0.4010 0.4524
302
+ C:\Users\Lenovo\Desktop\hack4dev\notebooks\src\03_gbdt_models.py:517: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
303
+ plt.show()
304
+ best classical variant on the protocol: upweighted shots, retrained
305
+ saved: C:\Users\Lenovo\Desktop\hack4dev\data\processed\gbdt_outputs.pkl
results/logs/03_gbdt_models_fullscore_run.log ADDED
@@ -0,0 +1,192 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ feature source: notebook 1 (official 60 + extended)
3
+ (76263, 400) (25992, 400)
4
+ dead (zero variance in at least one city): ['has_early_data', 'sat_share_oli_early']
5
+ official60 59 features
6
+ reflectance 276 features
7
+ events 15 features
8
+ neighbourhood 15 features
9
+ everything 393 features
10
+ block sizes: [3933 4437 785 2982 3078 3441 2820 3521 2363 3020 2929 2330 3738 2660
11
+ 3573 2154 3367 3580 2947 3250 2674 2376 3575 4257 2473]
12
+ C:\Users\Lenovo\Desktop\hack4dev\notebooks\src\03_gbdt_models.py:115: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
13
+ plt.show()
14
+ city-vs-city AUC: 1.0000 (1.0 = the two cities are trivially told apart)
15
+ feature city_importance
16
+ sat_share_oli_late 0.036333
17
+ sat_share_tm_early 0.035556
18
+ sat_share_etm_late 0.034222
19
+ sat_switches 0.032667
20
+ nbr_sr_brightness_mean 0.032000
21
+ sr_NDVI_early_std 0.030556
22
+ sat_share_etm_early 0.028778
23
+ sr_BSI_dec84_93_mean 0.026222
24
+ sat_share_oli 0.025778
25
+ sat_share_etm 0.024333
26
+ sr_NDVI_dec84_93_mean 0.023444
27
+ nbr_sr_SWIR1_mean 0.020111
28
+ radial_frac 0.019778
29
+ nbr_sr_NDVI_mean 0.016556
30
+ radial_px 0.016111
31
+ sr_Red_dec84_93_mean 0.014667
32
+ sr_NDVI_max 0.014444
33
+ n_observed_years 0.013444
34
+ dist_to_water_px 0.013000
35
+ sat_share_tm 0.011333
36
+ worst offenders, raw vs after per-city rank normalisation:
37
+ feature auc_raw auc_after_rank
38
+ sat_share_etm_late 0.990 0.971
39
+ sat_share_oli_late 0.989 0.593
40
+ sat_share_oli 0.988 0.579
41
+ sat_share_etm 0.972 0.607
42
+ sr_Red_dec84_93_mean 0.969 0.500
43
+ nbr_sr_brightness_mean 0.966 0.500
44
+ sr_Red_early_mean 0.963 0.500
45
+ Red_early_mean 0.963 0.500
46
+ sr_Red_tm_mean 0.957 0.500
47
+ nbr_sr_SWIR1_mean 0.957 0.500
48
+ sr_SWIR2_dec84_93_mean 0.955 0.500
49
+ sr_brightness_dec84_93_mean 0.951 0.500
50
+ sr_Red_min 0.951 0.500
51
+ sr_Red_mean 0.951 0.500
52
+ Red_mean 0.951 0.500
53
+ sr_Green_dec84_93_mean 0.948 0.500
54
+ sr_brightness_early_mean 0.947 0.500
55
+ SWIR2_early_mean 0.945 0.500
56
+ sr_SWIR2_early_mean 0.945 0.500
57
+ Green_early_mean 0.944 0.500
58
+ sr_Green_early_mean 0.944 0.500
59
+ sr_brightness_mean 0.941 0.500
60
+ sat_switches 0.941 0.906
61
+ sr_brightness_tm_mean 0.940 0.500
62
+ sr_TCB_early_mean 0.939 0.500
63
+
64
+ median domain AUC: raw 0.750 -> after rank 0.500
65
+ features still above 0.90 after normalisation: 5 of 393
66
+ C:\Users\Lenovo\Desktop\hack4dev\notebooks\src\03_gbdt_models.py:183: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
67
+ plt.show()
68
+ AUC raw 1.0000 | robust-z 1.0000 | rank 1.0000
69
+
70
+ still city-specific after rank normalisation:
71
+ feature city_importance
72
+ sat_share_tm 0.109133
73
+ n_observed_years 0.094531
74
+ sat_share_tm_late 0.083899
75
+ sat_share_etm_late 0.072243
76
+ sat_switches 0.053670
77
+ sat_share_etm 0.043294
78
+ sat_share_oli 0.043294
79
+ sat_share_oli_late 0.041629
80
+ sat_share_tm_early 0.033944
81
+ sat_share_etm_early 0.017805
82
+ step_SWIR1_njumps 0.015883
83
+ last_observed_year 0.014090
84
+ features still above AUC 0.9 after normalisation: 5
85
+ dropping 4: ['sat_share_etm_late', 'sat_switches', 'sat_share_tm_late', 'n_observed_years']
86
+ pruned set: 389 features
87
+
88
+ most city-specific features that we KEPT because they carry age signal:
89
+ feature auc_raw auc_after_rank age_importance
90
+ n_observed_years 0.6849 1.0000 0.0049
91
+ sat_share_tm_late 0.7312 0.9943 0.0016
92
+ sat_share_tm 0.5642 0.9912 0.0064
93
+ sat_share_etm_late 0.9901 0.9715 0.0011
94
+ sat_switches 0.9410 0.9063 0.0013
95
+ pre_state 0.7931 0.7109 0.0001
96
+ sat_share_etm 0.9718 0.6065 0.0037
97
+ sat_share_oli_late 0.9890 0.5930 0.0015
98
+ official60 lgbm 0.5557 +- 0.0540 (2.2 min)
99
+ official60 lgbm_unweighted 0.5504 +- 0.0633 (2.2 min)
100
+ official60 catboost 0.5446 +- 0.0480 (6.6 min)
101
+ official60 lgbm_ordinal 0.5462 +- 0.0637 (1.8 min)
102
+ everything lgbm 0.5702 +- 0.0703 (11.7 min)
103
+ everything lgbm_unweighted 0.5690 +- 0.0830 (9.6 min)
104
+ everything catboost 0.5621 +- 0.0671 (34.8 min)
105
+ everything lgbm_ordinal 0.5623 +- 0.0912 (7.8 min)
106
+ pruned lgbm 0.5693 +- 0.0730 (9.3 min)
107
+ pruned lgbm_unweighted 0.5688 +- 0.0853 (8.9 min)
108
+ pruned catboost 0.5627 +- 0.0692 (35.2 min)
109
+ pruned lgbm_ordinal 0.5631 +- 0.0897 (7.6 min)
110
+
111
+ features model n_feat macro_f1 std minutes
112
+ everything lgbm 393 0.5702 0.0703 11.7390
113
+ pruned lgbm 389 0.5693 0.0730 9.2962
114
+ everything lgbm_unweighted 393 0.5690 0.0830 9.5889
115
+ pruned lgbm_unweighted 389 0.5688 0.0853 8.8673
116
+ pruned lgbm_ordinal 389 0.5631 0.0897 7.5890
117
+ pruned catboost 389 0.5627 0.0692 35.2418
118
+ everything lgbm_ordinal 393 0.5623 0.0912 7.7758
119
+ everything catboost 393 0.5621 0.0671 34.8359
120
+ official60 lgbm 59 0.5557 0.0540 2.1759
121
+ official60 lgbm_unweighted 59 0.5504 0.0633 2.2104
122
+ official60 lgbm_ordinal 59 0.5462 0.0637 1.8472
123
+ official60 catboost 59 0.5446 0.0480 6.5561
124
+ baseline argmax, Madrid block-CV: 0.5761
125
+ tuned class weights: [1.058 1.333 0.943 0.666]
126
+ after weighting: 0.5821
127
+ spatial smoothing r= 2.0: 0.5906
128
+ spatial smoothing r= 4.0: 0.5868
129
+ spatial smoothing r= 8.0: 0.5835
130
+ pred 1 pred 2 pred 3 pred 4
131
+ true 1 0.446 0.427 0.083 0.044
132
+ true 2 0.170 0.681 0.091 0.058
133
+ true 3 0.069 0.221 0.587 0.123
134
+ true 4 0.056 0.172 0.183 0.589
135
+ regime both clean covariate label delta_vs_argmax_mean delta_vs_argmax_worst
136
+ strategy
137
+ argmax 0.4766 0.6056 0.5329 0.5392 0.0000 0.0000
138
+ em_free 0.3589 0.5312 0.3768 0.4900 -0.0993 -0.1079
139
+ em_pinned_shrunk 0.4406 0.5838 0.4792 0.5267 -0.0310 -0.0333
140
+ em_shrunk 0.4563 0.5895 0.4965 0.5314 -0.0201 -0.0063
141
+ logit_adj_tau0.25 0.4803 0.6093 0.5369 0.5438 0.0040 0.0010
142
+ logit_adj_tau0.5 0.4819 0.6102 0.5392 0.5469 0.0060 0.0014
143
+ logit_adj_tau1.0 0.4806 0.6068 0.5351 0.5477 0.0040 0.0156
144
+ plugin_srcprior 0.4885 0.6143 0.5524 0.5460 0.0117 0.0211
145
+ tau0.5_em_plugin 0.4693 0.6030 0.5212 0.5428 -0.0045 0.0159
146
+ tau0.5_then_plugin 0.4889 0.6134 0.5531 0.5472 0.0121 0.0267
147
+ rules that never lose to plain argmax in any regime:
148
+ regime both clean covariate label delta_vs_argmax_mean delta_vs_argmax_worst
149
+ strategy
150
+ argmax 0.4766 0.6056 0.5329 0.5392 0.0000 0.0000
151
+ logit_adj_tau0.25 0.4803 0.6093 0.5369 0.5438 0.0040 0.0010
152
+ logit_adj_tau0.5 0.4819 0.6102 0.5392 0.5469 0.0060 0.0014
153
+ logit_adj_tau1.0 0.4806 0.6068 0.5351 0.5477 0.0040 0.0156
154
+ plugin_srcprior 0.4885 0.6143 0.5524 0.5460 0.0117 0.0211
155
+ tau0.5_em_plugin 0.4693 0.6030 0.5212 0.5428 -0.0045 0.0159
156
+ tau0.5_then_plugin 0.4889 0.6134 0.5531 0.5472 0.0121 0.0267
157
+
158
+ rules to avoid:
159
+ regime both clean covariate label delta_vs_argmax_mean delta_vs_argmax_worst
160
+ strategy
161
+ em_free 0.3589 0.5312 0.3768 0.4900 -0.0993 -0.1079
162
+ em_pinned_shrunk 0.4406 0.5838 0.4792 0.5267 -0.0310 -0.0333
163
+
164
+ rule chosen on Madrid alone: tau0.5_then_plugin
165
+ Madrid class prior: [0.181 0.356 0.245 0.218]
166
+
167
+ organisers' prototype (Euclidean, Madrid scaler)
168
+ n=5 macro F1 = 0.5066 +- 0.0358 per-class [0.552 0.471 0.504 0.499]
169
+ n=25 macro F1 = 0.5821 +- 0.0166 per-class [0.626 0.555 0.573 0.574]
170
+ n=50 macro F1 = 0.5965 +- 0.0142 per-class [0.649 0.571 0.583 0.583]
171
+ n=100 macro F1 = 0.6034 +- 0.0086 per-class [0.661 0.591 0.583 0.579]
172
+ n=200 macro F1 = 0.5972 +- 0.0057 per-class [0.643 0.591 0.579 0.577]
173
+ (20 draws per n, 2.0 min)
174
+
175
+ pure retrieval k-NN (no source model)
176
+ n=5 macro F1 = 0.3775 +- 0.0758 per-class [0.292 0.43 0.366 0.422]
177
+ n=25 macro F1 = 0.6139 +- 0.0155 per-class [0.67 0.583 0.638 0.563]
178
+ n=50 macro F1 = 0.6375 +- 0.0112 per-class [0.697 0.613 0.659 0.581]
179
+ n=100 macro F1 = 0.6533 +- 0.0063 per-class [0.717 0.632 0.676 0.588]
180
+ n=200 macro F1 = 0.6699 +- 0.0037 per-class [0.734 0.654 0.687 0.605]
181
+ (20 draws per n, 12.6 min)
182
+
183
+ zero-shot, no normalisation
184
+ n=5 macro F1 = 0.5928 +- 0.0001 per-class [0.587 0.547 0.631 0.605]
185
+ n=25 macro F1 = 0.5925 +- 0.0002 per-class [0.588 0.548 0.631 0.604]
186
+ n=50 macro F1 = 0.5922 +- 0.0003 per-class [0.587 0.548 0.631 0.603]
187
+ n=100 macro F1 = 0.5918 +- 0.0004 per-class [0.587 0.548 0.631 0.601]
188
+ n=200 macro F1 = 0.5903 +- 0.0006 per-class [0.587 0.548 0.63 0.595]
189
+ (20 draws per n, 32.0 min)
190
+
191
+ zero-shot, robust z
192
+ n=5 macro F1 = 0.5268 +- 0.0001 per-class [0.512 0.521 0.543 0.531]
results/logs/full_score_prototype.log ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ prototype, robust z + within-class Mahalanobis: 393 features, dead ['has_early_data', 'sat_share_oli_early'], Madrid (76263, 393), Amsterdam (25992, 393)
2
+ n=5 rep=0 full=0.5154 sub2000=0.5258 sub8000=0.5134 (39.8s)
3
+ n=5 rep=1 full=0.5326 sub2000=0.5267 sub8000=0.5270 (35.2s)
4
+ n=5 rep=2 full=0.5138 sub2000=0.4877 sub8000=0.5131 (27.8s)
5
+ n=5 rep=3 full=0.5247 sub2000=0.5102 sub8000=0.5209 (27.2s)
6
+ n=5 rep=4 full=0.5269 sub2000=0.5214 sub8000=0.5204 (27.6s)
7
+ n=5 rep=5 full=0.5471 sub2000=0.5534 sub8000=0.5485 (27.6s)
8
+ n=5 rep=6 full=0.4699 sub2000=0.4804 sub8000=0.4765 (34.9s)
9
+ n=5 rep=7 full=0.4917 sub2000=0.4968 sub8000=0.4965 (45.0s)
10
+ n=5 rep=8 full=0.4952 sub2000=0.5003 sub8000=0.4941 (61.9s)
11
+ n=5 rep=9 full=0.5105 sub2000=0.4913 sub8000=0.5075 (55.9s)
12
+ n=5 rep=10 full=0.3944 sub2000=0.4027 sub8000=0.3876 (68.2s)
13
+ n=5 rep=11 full=0.5260 sub2000=0.5280 sub8000=0.5241 (59.9s)
14
+ n=25 rep=0 full=0.6409 sub2000=0.6401 sub8000=0.6376 (68.6s)
15
+ n=25 rep=1 full=0.6462 sub2000=0.6543 sub8000=0.6498 (61.2s)
16
+ n=25 rep=2 full=0.6050 sub2000=0.6000 sub8000=0.6003 (91.6s)
17
+ n=25 rep=3 full=0.6219 sub2000=0.6287 sub8000=0.6204 (64.1s)
18
+ n=25 rep=4 full=0.6077 sub2000=0.6013 sub8000=0.5991 (52.0s)
19
+ n=25 rep=5 full=0.6520 sub2000=0.6490 sub8000=0.6513 (23.6s)
20
+ n=25 rep=6 full=0.6535 sub2000=0.6631 sub8000=0.6554 (24.5s)
21
+ n=25 rep=7 full=0.6382 sub2000=0.6300 sub8000=0.6365 (27.0s)
22
+ n=25 rep=8 full=0.6443 sub2000=0.6365 sub8000=0.6375 (25.4s)
23
+ n=25 rep=9 full=0.6337 sub2000=0.6472 sub8000=0.6351 (28.9s)
24
+ n=25 rep=10 full=0.6329 sub2000=0.6127 sub8000=0.6243 (28.2s)
25
+ n=25 rep=11 full=0.6569 sub2000=0.6490 sub8000=0.6537 (28.2s)
26
+ n=50 rep=0 full=0.6880 sub2000=0.6817 sub8000=0.6822 (31.1s)
27
+ n=50 rep=1 full=0.6869 sub2000=0.6816 sub8000=0.6806 (38.5s)
28
+ n=50 rep=2 full=0.6762 sub2000=0.6843 sub8000=0.6777 (46.1s)
29
+ n=50 rep=3 full=0.6752 sub2000=0.6593 sub8000=0.6707 (45.4s)
30
+ n=50 rep=4 full=0.6800 sub2000=0.6797 sub8000=0.6781 (45.5s)
31
+ n=50 rep=5 full=0.6745 sub2000=0.6656 sub8000=0.6650 (46.3s)
32
+ n=50 rep=6 full=0.6507 sub2000=0.6473 sub8000=0.6467 (45.0s)
33
+ n=50 rep=7 full=0.6476 sub2000=0.6530 sub8000=0.6467 (45.8s)
34
+ n=50 rep=8 full=0.6857 sub2000=0.6787 sub8000=0.6853 (43.8s)
35
+ n=50 rep=9 full=0.6659 sub2000=0.6645 sub8000=0.6624 (44.9s)
36
+ n=50 rep=10 full=0.6831 sub2000=0.6853 sub8000=0.6775 (45.5s)
37
+ n=50 rep=11 full=0.6655 sub2000=0.6668 sub8000=0.6609 (45.4s)
38
+ n=100 rep=0 full=0.6957 sub2000=0.6912 sub8000=0.6886 (45.2s)
39
+ n=100 rep=1 full=0.7032 sub2000=0.6992 sub8000=0.6970 (44.2s)
40
+ n=100 rep=2 full=0.6996 sub2000=0.6893 sub8000=0.6922 (44.7s)
41
+ n=100 rep=3 full=0.7157 sub2000=0.7004 sub8000=0.7070 (44.9s)
42
+ n=100 rep=4 full=0.7103 sub2000=0.7030 sub8000=0.7040 (44.4s)
43
+ n=100 rep=5 full=0.6970 sub2000=0.6889 sub8000=0.6917 (43.7s)
44
+ n=100 rep=6 full=0.7016 sub2000=0.6977 sub8000=0.6958 (45.0s)
45
+ n=100 rep=7 full=0.6994 sub2000=0.6928 sub8000=0.6972 (43.7s)
46
+ n=100 rep=8 full=0.6992 sub2000=0.6888 sub8000=0.6941 (44.4s)
47
+ n=100 rep=9 full=0.6925 sub2000=0.6901 sub8000=0.6895 (44.8s)
48
+ n=100 rep=10 full=0.7088 sub2000=0.6897 sub8000=0.7042 (43.5s)
49
+ n=100 rep=11 full=0.7054 sub2000=0.6914 sub8000=0.6997 (45.0s)
50
+ n=200 rep=0 full=0.7126 sub2000=0.7085 sub8000=0.7088 (48.2s)
51
+ n=200 rep=1 full=0.7123 sub2000=0.7040 sub8000=0.7098 (41.2s)
52
+ n=200 rep=2 full=0.7101 sub2000=0.7072 sub8000=0.7028 (41.1s)
53
+ n=200 rep=3 full=0.7139 sub2000=0.7087 sub8000=0.7079 (41.4s)
54
+ n=200 rep=4 full=0.7186 sub2000=0.7081 sub8000=0.7117 (41.4s)
55
+ n=200 rep=5 full=0.7159 sub2000=0.7095 sub8000=0.7095 (40.9s)
56
+ n=200 rep=6 full=0.7208 sub2000=0.7009 sub8000=0.7113 (41.2s)
57
+ n=200 rep=7 full=0.7118 sub2000=0.7100 sub8000=0.7066 (43.7s)
58
+ n=200 rep=8 full=0.7150 sub2000=0.7057 sub8000=0.7080 (42.9s)
59
+ n=200 rep=9 full=0.7125 sub2000=0.6946 sub8000=0.7067 (42.9s)
60
+ prototype, robust z + within-class Mahalanobis: 393 features, dead ['has_early_data', 'sat_share_oli_early'], Madrid (76263, 393), Amsterdam (25992, 393)
61
+ resuming: 58 draws already saved
62
+ n=200 rep=10 full=0.7141 sub2000=0.7105 sub8000=0.7106 (28.6s)
63
+ n=200 rep=11 full=0.7113 sub2000=0.7121 sub8000=0.7086 (32.9s)
64
+
65
+ full sub2000 sub8000
66
+ 5 0.5040 0.5021 0.5025
67
+ 25 0.6361 0.6343 0.6334
68
+ 50 0.6733 0.6706 0.6695
69
+ 100 0.7023 0.6935 0.6967
70
+ 200 0.7141 0.7067 0.7085
71
+ full average 0.6460 peak 0.7141
72
+ sub2000 average 0.6414 peak 0.7067
73
+ sub8000 average 0.6421 peak 0.7085
74
+ done in 1.0 min
results/logs/full_score_upweighted.log ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ upweighted shots, retrained: 393 features, dead ['has_early_data', 'sat_share_oli_early'], Madrid (76263, 393), Amsterdam (25992, 393)
2
+ n=5 rep=0 full=0.5658 sub2000=0.5750 sub8000=0.5632 (74.1s)
3
+ n=5 rep=1 full=0.5755 sub2000=0.5756 sub8000=0.5699 (51.6s)
4
+ n=5 rep=2 full=0.5476 sub2000=0.5483 sub8000=0.5430 (50.4s)
5
+ n=5 rep=3 full=0.5685 sub2000=0.5612 sub8000=0.5622 (116.9s)
6
+ n=5 rep=4 full=0.5714 sub2000=0.5698 sub8000=0.5664 (179.8s)
7
+ n=25 rep=0 full=0.6303 sub2000=0.6323 sub8000=0.6322 (161.0s)
8
+ n=25 rep=1 full=0.6185 sub2000=0.6252 sub8000=0.6161 (252.8s)
9
+ n=25 rep=2 full=0.6265 sub2000=0.6143 sub8000=0.6197 (46.6s)
10
+ n=25 rep=3 full=0.6478 sub2000=0.6529 sub8000=0.6452 (46.6s)
11
+ n=25 rep=4 full=0.6188 sub2000=0.6208 sub8000=0.6165 (49.6s)
12
+ n=50 rep=0 full=0.6658 sub2000=0.6693 sub8000=0.6587 (50.7s)
13
+ n=50 rep=1 full=0.6741 sub2000=0.6748 sub8000=0.6708 (75.8s)
14
+ n=50 rep=2 full=0.6786 sub2000=0.6779 sub8000=0.6807 (79.3s)
15
+ n=50 rep=3 full=0.6734 sub2000=0.6654 sub8000=0.6671 (75.8s)
16
+ n=50 rep=4 full=0.6565 sub2000=0.6573 sub8000=0.6545 (75.8s)
17
+ n=100 rep=0 full=0.7016 sub2000=0.7054 sub8000=0.6992 (75.6s)
18
+ n=100 rep=1 full=0.6985 sub2000=0.6901 sub8000=0.6933 (71.3s)
19
+ n=100 rep=2 full=0.6914 sub2000=0.6965 sub8000=0.6913 (69.3s)
20
+ n=100 rep=3 full=0.6990 sub2000=0.6979 sub8000=0.6952 (70.0s)
21
+ n=100 rep=4 full=0.6961 sub2000=0.6907 sub8000=0.6952 (69.8s)
22
+ n=200 rep=0 full=0.7265 sub2000=0.7356 sub8000=0.7245 (71.0s)
23
+ n=200 rep=1 full=0.7125 sub2000=0.7080 sub8000=0.7114 (70.1s)
24
+ n=200 rep=2 full=0.7180 sub2000=0.7149 sub8000=0.7155 (70.4s)
25
+ n=200 rep=3 full=0.7184 sub2000=0.7239 sub8000=0.7175 (75.8s)
26
+ n=200 rep=4 full=0.7265 sub2000=0.7327 sub8000=0.7220 (78.8s)
27
+
28
+ full sub2000 sub8000
29
+ 5 0.5658 0.5660 0.5609
30
+ 25 0.6284 0.6291 0.6259
31
+ 50 0.6697 0.6690 0.6664
32
+ 100 0.6973 0.6961 0.6949
33
+ 200 0.7204 0.7230 0.7182
34
+ full average 0.6563 peak 0.7204
35
+ sub2000 average 0.6566 peak 0.7230
36
+ sub8000 average 0.6533 peak 0.7182
37
+ done in 35.2 min
results/protocol_results.csv ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ retrieval + Madrid rule,+ bias recalibration,+ tuned decision weights,ensemble LLM 0.3 / LightGBM 0.7,ensemble LLM 0.5 / LightGBM 0.5,ensemble LLM 0.7 / LightGBM 0.3,n,rep
2
+ 0.5202,,,0.5588,0.4828,0.4765,5,0
3
+ 0.5224,,,0.5606,0.5846,0.5711,5,1
4
+ 0.4924,,,0.5525,0.4872,0.5468,5,2
5
+ 0.5195,,,0.5633,0.5763,0.5649,5,3
6
+ 0.5161,,,0.5639,0.5792,0.5531,5,4
7
+ 0.5133,,,0.5611,0.584,0.5702,5,5
8
+ 0.5376,,,0.5646,0.4759,0.4775,25,0
9
+ 0.5318,,,0.566,0.4687,0.4682,25,1
10
+ 0.512,,,0.5646,0.4715,0.467,25,2
11
+ 0.5243,,,0.5629,0.4745,0.4715,25,3
12
+ 0.5027,,,0.5657,0.4795,0.4802,25,4
13
+ 0.4961,,,0.5642,0.4832,0.4751,25,5
14
+ 0.5797,0.4727,,0.5759,0.4745,0.4818,50,0
15
+ 0.5193,0.497,,0.5684,0.4745,0.4682,50,1
16
+ 0.528,0.5064,,0.5782,0.4817,0.4803,50,2
17
+ 0.5238,0.4917,,0.5763,0.4689,0.4769,50,3
18
+ 0.5595,0.5899,,0.5818,0.4852,0.4762,50,4
19
+ 0.5162,0.4688,,0.5811,0.4666,0.4687,50,5
20
+ 0.4982,0.4817,,0.5749,0.4791,0.478,100,0
21
+ 0.5687,0.5728,,0.586,0.4863,0.4847,100,1
22
+ 0.5367,0.4953,,0.5796,0.4813,0.4767,100,2
23
+ 0.573,0.4902,,0.5801,0.4827,0.4762,100,3
24
+ 0.541,0.5091,,0.5771,0.4797,0.4809,100,4
25
+ 0.5777,0.63,,0.5826,0.4857,0.486,100,5
26
+ 0.59,0.5981,0.6099,0.5831,0.4952,0.4947,200,0
27
+ 0.5413,0.6088,0.6506,0.5849,0.4907,0.4939,200,1
28
+ 0.5607,0.6105,0.594,0.5924,0.4812,0.4918,200,2
29
+ 0.5432,0.571,0.6483,0.5828,0.4961,0.4915,200,3
30
+ 0.5822,0.6088,0.5958,0.5815,0.4882,0.4841,200,4
31
+ 0.5889,0.5985,0.6346,0.5836,0.4907,0.4887,200,5
results/shift_harness.csv ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ strategy,both,clean,covariate,label,delta_vs_argmax_mean,delta_vs_argmax_worst
2
+ argmax,0.4766277348013709,0.6055506844084575,0.5328959332264072,0.5391639540527374,0.0,0.0
3
+ em_free,0.3589397742522877,0.5312464491381202,0.37684760869683986,0.48997794293710095,-0.09930663286615615,-0.10786078556270207
4
+ em_pinned_shrunk,0.4406480520203647,0.5837780673295816,0.47917846504948114,0.5266926677475531,-0.030985263585498135,-0.033297506683739375
5
+ em_shrunk,0.45634006963730467,0.5894954166977777,0.4964833569867393,0.5314349516494004,-0.02012112787943776,-0.006301657202695876
6
+ logit_adj_tau0.25,0.4802725787074079,0.6092793116373957,0.5368932017851271,0.5438140379788996,0.004005205904964226,0.0010334968999556327
7
+ logit_adj_tau0.5,0.48188297075609177,0.6101659145541751,0.5392044403543397,0.54686169194163,0.005969177779315826,0.0014135450449920817
8
+ logit_adj_tau1.0,0.48058168514340693,0.6067736712706344,0.5350619082752373,0.5477426130512237,0.003980392812882316,0.01562759837422195
9
+ plugin_srcprior,0.48850202153623157,0.6143242551861514,0.5523741536954043,0.5459945788738292,0.011739175700660853,0.021085184321292028
10
+ tau0.5_em_plugin,0.46927968168607403,0.6029738936298857,0.5211543503419801,0.5427893252057969,-0.004510263906309153,0.015860548689450027
11
+ tau0.5_then_plugin,0.4888571435470125,0.6133924481265582,0.5530570356897002,0.547176443156529,0.01206119100770664,0.02671852147426451
src/config.py ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Shared configuration for the Building-Age-Classification hackathon
3
+ (Madrid -> Amsterdam transfer).
4
+
5
+ Everything that both the GBDT track and the LLM track need lives here so the
6
+ two never disagree about class boundaries, feature names or file paths.
7
+ """
8
+ from __future__ import annotations
9
+
10
+ import os
11
+ from pathlib import Path
12
+
13
+ # --------------------------------------------------------------------------- #
14
+ # 1. Paths (work on a laptop, on Kaggle and on Colab without edits)
15
+ # --------------------------------------------------------------------------- #
16
+
17
+ def _detect_platform() -> str:
18
+ if os.path.exists("/kaggle/working"):
19
+ return "kaggle"
20
+ try: # noqa: SIM105
21
+ import google.colab # noqa: F401
22
+ return "colab"
23
+ except Exception:
24
+ return "local"
25
+
26
+
27
+ PLATFORM = _detect_platform()
28
+
29
+ if PLATFORM == "kaggle":
30
+ PROJECT_ROOT = Path("/kaggle/working")
31
+ DATA_DIR = Path("/kaggle/input") # the attached dataset lives here
32
+ elif PLATFORM == "colab":
33
+ PROJECT_ROOT = Path("/content/hack4dev")
34
+ DATA_DIR = PROJECT_ROOT / "data"
35
+ else:
36
+ PROJECT_ROOT = Path(__file__).resolve().parent.parent
37
+ DATA_DIR = PROJECT_ROOT / "data"
38
+
39
+ RAW_DIR = DATA_DIR / "raw"
40
+ PROC_DIR = DATA_DIR / "processed"
41
+ REPORT_DIR = PROJECT_ROOT / "results" # "reports" in the original project
42
+ for _d in (PROC_DIR, REPORT_DIR):
43
+ try:
44
+ _d.mkdir(parents=True, exist_ok=True)
45
+ except Exception:
46
+ pass
47
+
48
+ RANDOM_SEED = 42
49
+
50
+ # --------------------------------------------------------------------------- #
51
+ # Hugging Face settings for the LLM track (notebooks 2, 4, 5). That track is
52
+ # not part of this release; nothing here uses these names.
53
+ # --------------------------------------------------------------------------- #
54
+ HF_USER = os.environ.get("HF_USER", "Karar-git")
55
+ DATASET_REPO = f"{HF_USER}/building-age-prompts"
56
+ MODEL_REPO_PREFIX = f"{HF_USER}/"
57
+
58
+
59
+ def hf_ready() -> bool:
60
+ return bool(HF_USER) and HF_USER != "YOUR_HF_USERNAME"
61
+
62
+ # --------------------------------------------------------------------------- #
63
+ # 2. The task definition (from the organisers' Notebook 1 / 3)
64
+ # --------------------------------------------------------------------------- #
65
+
66
+ BASE_BANDS = ["Blue", "Green", "Red", "NIR", "SWIR1", "SWIR2"]
67
+ INDEX_COLS = ["NDVI", "NDBI", "UI", "MNDWI", "BSI"]
68
+
69
+ # Landsat Collection-2 Level-2 surface-reflectance scaling: SR = DN*2.75e-5 - 0.2
70
+ SR_SCALE, SR_OFFSET = 2.75e-5, -0.2
71
+
72
+ BLUE_MAX = 15_000 # haze threshold from the organisers' Notebook 2
73
+ EARLY_CUTOFF = 2003 # early period = 1984..2003
74
+ LATE_START = 2004 # late period = 2004..
75
+ PID = ["px_key", "py_key"]
76
+
77
+ # City-specific boundary between class 1 and class 2
78
+ CLASS_EVENTS = {"Amsterdam": 1945, "Madrid": 1960}
79
+
80
+ CLASS_LABELS = {
81
+ "Amsterdam": {1: "before 1945", 2: "1945-1984", 3: "1984-2004", 4: "2004-2024"},
82
+ "Madrid": {1: "before 1960", 2: "1960-1984", 3: "1984-2004", 4: "2004-2024"},
83
+ }
84
+
85
+ # Published class shares (organisers' guide) - used only for sanity checks,
86
+ # never to fit anything.
87
+ PUBLISHED_SHARES = {"Madrid": {2: 0.356}, "Amsterdam": {4: 0.111}}
88
+
89
+ # --------------------------------------------------------------------------- #
90
+ # 3. The official 60 features, in the official order
91
+ # --------------------------------------------------------------------------- #
92
+
93
+ FEAT_OVERALL = [f"{b}_mean" for b in BASE_BANDS] + [f"{b}_std" for b in BASE_BANDS]
94
+ FEAT_INDICES = [f"{i}_mean" for i in INDEX_COLS] + [f"{i}_std" for i in INDEX_COLS]
95
+ FEAT_EARLY = [f"{b}_early_mean" for b in BASE_BANDS] + [f"{b}_early_std" for b in BASE_BANDS]
96
+ FEAT_LATE = [f"{b}_late_mean" for b in BASE_BANDS] + [f"{b}_late_std" for b in BASE_BANDS]
97
+ FEAT_YOY = [f"d_{b}_mean" for b in BASE_BANDS] + [f"d_{b}_std" for b in BASE_BANDS]
98
+ FEAT_IND = ["has_early_data", "has_late_data"]
99
+
100
+ FEATURE_COLS_60 = FEAT_OVERALL + FEAT_INDICES + FEAT_EARLY + FEAT_LATE + FEAT_YOY + FEAT_IND
101
+
102
+ FEAT_GROUPS_60 = {
103
+ "overall": FEAT_OVERALL,
104
+ "indices": FEAT_INDICES,
105
+ "early": FEAT_EARLY,
106
+ "late": FEAT_LATE,
107
+ "yoy": FEAT_YOY,
108
+ "indicators": FEAT_IND,
109
+ }
110
+
111
+ assert len(FEATURE_COLS_60) == 60, len(FEATURE_COLS_60)
src/evaluation.py ADDED
@@ -0,0 +1,519 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ The official evaluation protocol, plus the decision-rule and prior-shift tools
3
+ that every model in this project shares.
4
+
5
+ Protocol (organisers' Notebook 1, section 1.5):
6
+ 1. train a Stage-1 model on Madrid only;
7
+ 2. adapt it with n labelled Amsterdam pixels per class, n in {5,25,50,100,200};
8
+ 3. score macro F1 on the *rest* of Amsterdam;
9
+ 4. repeat over many random draws and report mean +- std for every n.
10
+
11
+ `modelCreate` is any callable with this signature:
12
+
13
+ predict_proba_fn = model_create(
14
+ X_src, y_src, # Madrid, labelled
15
+ X_tgt_unlabelled, # all target features, NO labels
16
+ X_shot, y_shot, # the n-per-class labelled target sample
17
+ )
18
+
19
+ and must return a function mapping X -> (n, 4) probabilities. Keeping it a
20
+ closure means every adaptation trick (recalibration, prototypes, continued
21
+ boosting, retrieval prompts) fits the same harness.
22
+ """
23
+ from __future__ import annotations
24
+
25
+ import numpy as np
26
+ from sklearn.metrics import confusion_matrix, f1_score
27
+
28
+ CLASSES = np.array([1, 2, 3, 4])
29
+
30
+
31
+ # --------------------------------------------------------------------------- #
32
+ # Metrics
33
+ # --------------------------------------------------------------------------- #
34
+ def macro_f1(y_true, y_pred) -> float:
35
+ return float(f1_score(y_true, y_pred, labels=CLASSES, average="macro", zero_division=0))
36
+
37
+
38
+ def per_class_f1(y_true, y_pred) -> np.ndarray:
39
+ return f1_score(y_true, y_pred, labels=CLASSES, average=None, zero_division=0)
40
+
41
+
42
+ def confusion_proportions(y_true, y_pred) -> np.ndarray:
43
+ cm = confusion_matrix(y_true, y_pred, labels=CLASSES).astype(float)
44
+ return cm / np.clip(cm.sum(axis=1, keepdims=True), 1, None)
45
+
46
+
47
+ # --------------------------------------------------------------------------- #
48
+ # Decision rules. argmax is NOT the macro-F1 optimum.
49
+ # --------------------------------------------------------------------------- #
50
+ def predict_weighted(proba: np.ndarray, weights: np.ndarray) -> np.ndarray:
51
+ """argmax of w_k * p_k -> class label."""
52
+ return CLASSES[np.argmax(proba * np.asarray(weights, dtype=float), axis=1)]
53
+
54
+
55
+ def macro_f1_weighted(y_true, y_pred, sample_weight=None) -> float:
56
+ """
57
+ Macro F1 computed from a *weighted* confusion matrix.
58
+
59
+ This exists because the few-shot sample is drawn n-per-class: on it, a rare
60
+ class looks far more precise than it really is, so tuning a decision rule
61
+ on the raw sample over-predicts rare classes. Feeding
62
+ `gbdt.stratification_weights(...)` here makes the estimate behave like macro
63
+ F1 on a random sample of the city, using only legal information.
64
+ """
65
+ y_true = np.asarray(y_true)
66
+ y_pred = np.asarray(y_pred)
67
+ w = np.ones(len(y_true)) if sample_weight is None else np.asarray(sample_weight, float)
68
+ f1s = []
69
+ for c in CLASSES:
70
+ tp = w[(y_true == c) & (y_pred == c)].sum()
71
+ fp = w[(y_true != c) & (y_pred == c)].sum()
72
+ fn = w[(y_true == c) & (y_pred != c)].sum()
73
+ denom = 2 * tp + fp + fn
74
+ f1s.append(0.0 if denom <= 0 else 2 * tp / denom)
75
+ return float(np.mean(f1s))
76
+
77
+
78
+ def tune_class_weights(proba: np.ndarray, y_true, n_rounds: int = 6, grid=None,
79
+ sample_weight=None, shrink: float = 0.0) -> np.ndarray:
80
+ """
81
+ Coordinate ascent on per-class multipliers to maximise macro F1.
82
+
83
+ Fit this on a validation split of the TRAINING city, or on the labelled
84
+ few-shot sample - never on the full target labels.
85
+
86
+ shrink: pull the result back towards 1 (no adjustment). With only 20
87
+ labelled rows (n=5 per class) the search overfits, so shrink hard; with 800
88
+ rows it can be left at 0.
89
+ """
90
+ if grid is None:
91
+ grid = np.exp(np.linspace(np.log(0.25), np.log(4.0), 25))
92
+ w = np.ones(4)
93
+ score = lambda ww: macro_f1_weighted(y_true, predict_weighted(proba, ww), # noqa: E731
94
+ sample_weight)
95
+ best = score(w)
96
+ for _ in range(n_rounds):
97
+ improved = False
98
+ for k in range(4):
99
+ cur = w[k]
100
+ for g in grid:
101
+ w[k] = g
102
+ s = score(w)
103
+ if s > best + 1e-6:
104
+ best, cur, improved = s, g, True
105
+ w[k] = cur
106
+ if not improved:
107
+ break
108
+ w = w / w.mean()
109
+ if shrink > 0:
110
+ w = np.exp((1 - shrink) * np.log(w))
111
+ return w / w.mean()
112
+
113
+
114
+ def prior_correct(proba: np.ndarray, train_prior: np.ndarray,
115
+ target_prior: np.ndarray) -> np.ndarray:
116
+ """Re-weight probabilities from one class prior to another (label shift)."""
117
+ adj = proba * (np.asarray(target_prior) / np.asarray(train_prior))[None, :]
118
+ return adj / np.clip(adj.sum(axis=1, keepdims=True), 1e-12, None)
119
+
120
+
121
+ def em_target_prior(proba: np.ndarray, train_prior: np.ndarray,
122
+ n_iter: int = 200, tol: float = 1e-7,
123
+ pin: dict | None = None, shrink: float = 0.0) -> np.ndarray:
124
+ """
125
+ Saerens-Latinne-Decaestecker EM estimate of the target class prior from
126
+ UNLABELLED target probabilities. Uses X_amsterdam only - no labels - so it
127
+ stays inside the rules, but it only works if the model is roughly calibrated.
128
+
129
+ pin: fix some class shares to known values and renormalise the rest, e.g.
130
+ {4: 0.111} - the organisers publish Amsterdam's class-4 share in the
131
+ competitors' guide, so pinning it is citable, not leakage. Pinning
132
+ also bounds the failure mode where EM drives a class to zero.
133
+ shrink: pull the estimate back towards the training prior,
134
+ q = (1-shrink)*q_EM + shrink*train_prior. EM is confident and
135
+ frequently wrong under covariate shift; 0.5-0.7 is a sane default
136
+ and should be chosen on the simulated-shift harness.
137
+ """
138
+ train_prior = np.asarray(train_prior, dtype=float)
139
+ train_prior = train_prior / train_prior.sum()
140
+
141
+ def apply_pin(p):
142
+ if not pin:
143
+ return p / p.sum()
144
+ p = p.copy()
145
+ free = [k for k in range(4) if (k + 1) not in pin]
146
+ fixed_mass = sum(pin.values())
147
+ for c, v in pin.items():
148
+ p[c - 1] = v
149
+ rest = p[free].sum()
150
+ if rest > 0:
151
+ p[free] *= (1.0 - fixed_mass) / rest
152
+ else:
153
+ p[free] = (1.0 - fixed_mass) / max(len(free), 1)
154
+ return p
155
+
156
+ prior = apply_pin(train_prior.copy())
157
+ for _ in range(n_iter):
158
+ post = proba * (prior / train_prior)[None, :]
159
+ post /= np.clip(post.sum(axis=1, keepdims=True), 1e-12, None)
160
+ new = apply_pin(post.mean(axis=0))
161
+ if np.max(np.abs(new - prior)) < tol:
162
+ prior = new
163
+ break
164
+ prior = new
165
+ if shrink > 0:
166
+ prior = apply_pin((1 - shrink) * prior + shrink * train_prior)
167
+ return prior
168
+
169
+
170
+ def logit_adjust(proba: np.ndarray, train_prior: np.ndarray,
171
+ tau: float = 0.4) -> np.ndarray:
172
+ """
173
+ tau-scaled logit adjustment (Menon et al. 2021): divide by the training
174
+ prior raised to tau.
175
+
176
+ tau = 1 is Fisher-consistent for *balanced error rate*, i.e. macro RECALL.
177
+ Macro F1 is a different metric and punishes over-predicting a rare class,
178
+ so its optimum sits lower - tune tau on training-city CV and expect
179
+ 0.25-0.5. Leaving tau at 1.0 reliably loses macro F1 here.
180
+ """
181
+ p = np.asarray(proba, dtype=float)
182
+ tp = np.asarray(train_prior, dtype=float)
183
+ adj = p / np.clip(tp, 1e-12, None) ** float(tau)
184
+ return adj / np.clip(adj.sum(axis=1, keepdims=True), 1e-12, None)
185
+
186
+
187
+ def macro_f1_plugin(proba: np.ndarray, prior: np.ndarray, n_iter: int = 50):
188
+ """
189
+ The plug-in decision rule for macro F1 (Ramaswamy et al., "Consistent
190
+ algorithms for multiclass F-measures").
191
+
192
+ argmax of a calibrated posterior maximises accuracy, not macro F1. The
193
+ macro-F1-optimal rule is
194
+ predict argmax_k a_k * (p_k(x) - F1_k / 2), a_k = 2 / (pi_k + q_k)
195
+ where q_k is the predicted share of class k and F1_k the resulting per-class
196
+ F1 - so it is a fixed point, solved by iteration. It needs the probabilities
197
+ and a prior estimate, and NO target labels, which is why it is the right
198
+ tool here.
199
+
200
+ Returns (labels, diagnostics).
201
+ """
202
+ P = np.asarray(proba, dtype=float)
203
+ pi = np.asarray(prior, dtype=float)
204
+ pi = pi / pi.sum()
205
+ f1 = np.zeros(4)
206
+ best = None
207
+ for it in range(n_iter):
208
+ q = np.clip(np.bincount(np.argmax(P, axis=1), minlength=4) / len(P), 1e-6, None) \
209
+ if it == 0 else q_new
210
+ a = 2.0 / np.clip(pi + q, 1e-9, None)
211
+ score = a[None, :] * (P - f1[None, :] / 2.0)
212
+ lab_idx = np.argmax(score, axis=1)
213
+ q_new = np.clip(np.bincount(lab_idx, minlength=4) / len(P), 1e-6, None)
214
+ # soft per-class F1 implied by the current assignment and the posterior
215
+ tp = np.array([P[lab_idx == k, k].sum() for k in range(4)]) / len(P)
216
+ f1_new = 2 * tp / np.clip(pi + q_new, 1e-9, None)
217
+ obj = float(np.mean(f1_new))
218
+ if best is None or obj > best[0]:
219
+ best = (obj, CLASSES[lab_idx].copy(), q_new.copy(), f1_new.copy())
220
+ if np.max(np.abs(f1_new - f1)) < 1e-6 and it > 0:
221
+ f1 = f1_new
222
+ break
223
+ f1 = f1_new
224
+ obj, labels, q, f1 = best
225
+ return labels, dict(expected_macro_f1=obj, predicted_share=q, per_class_f1=f1)
226
+
227
+
228
+ def adapt_pipeline(proba, src_prior, pin=None, tau: float = 0.5,
229
+ shrink: float = 0.7, use_em: bool = False,
230
+ use_plugin: bool = True):
231
+ """
232
+ The label-free adaptation chain: tau-logit-adjust -> (optional prior
233
+ re-estimation) -> macro-F1 plug-in.
234
+
235
+ `use_em` defaults to FALSE on purpose. On the simulated-shift harness, free
236
+ SLD-EM was the worst thing tested: under covariate shift it drove the prior
237
+ to [0.85, 0.10, 0.04, 0.02] against a truth of [0.34, 0.32, 0.22, 0.11] and
238
+ cost 0.19 macro F1 on average, 0.35 at worst. Pinning class 4 at the
239
+ published 0.111 and shrinking hard towards the source prior bounds the
240
+ damage but does not repair it: EM assumes pure label shift, and this problem
241
+ has a large covariate shift on top. Re-run `shift_harness` on the real
242
+ Madrid features before turning it on.
243
+
244
+ What survived the harness: tau-logit-adjustment at tau = 0.25-0.5 (small but
245
+ positive in every regime) and the macro-F1 plug-in rule driven by the
246
+ SOURCE prior (+0.033 mean, +0.038 worst case).
247
+ """
248
+ src_prior = np.asarray(src_prior, dtype=float)
249
+ src_prior = src_prior / src_prior.sum()
250
+ p = logit_adjust(proba, src_prior, tau=tau) if tau else np.asarray(proba, float)
251
+ prior = src_prior
252
+ if use_em:
253
+ prior = em_target_prior(p, src_prior, pin=pin, shrink=shrink)
254
+ p = prior_correct(p, src_prior, prior)
255
+ if use_plugin:
256
+ labels, diag = macro_f1_plugin(p, prior)
257
+ else:
258
+ labels, diag = CLASSES[np.argmax(p, axis=1)], {}
259
+ diag["prior_used"] = prior
260
+ return labels, p, diag
261
+
262
+
263
+ # --------------------------------------------------------------------------- #
264
+ # Spatial smoothing (stage 2): neighbouring pixels share a construction era
265
+ # --------------------------------------------------------------------------- #
266
+ def spatial_smooth(proba: np.ndarray, coords: np.ndarray, radius: float = 4.0,
267
+ alpha: float = 0.5, n_iter: int = 1) -> np.ndarray:
268
+ """
269
+ Blend each pixel's probabilities with a Gaussian-weighted average of its
270
+ neighbours: p <- (1-alpha) * p + alpha * mean_neighbours(p).
271
+ `radius` is in pixel units (1 unit = 30 m). Choose it on Madrid only.
272
+ """
273
+ from sklearn.neighbors import NearestNeighbors
274
+
275
+ coords = np.asarray(coords, dtype=float)
276
+ nn = NearestNeighbors(radius=radius * 2.0).fit(coords)
277
+ dist, idx = nn.radius_neighbors(coords, radius=radius * 2.0)
278
+ out = proba.copy()
279
+ for _ in range(n_iter):
280
+ nxt = out.copy()
281
+ for i, (d, j) in enumerate(zip(dist, idx)):
282
+ if len(j) <= 1:
283
+ continue
284
+ w = np.exp(-0.5 * (d / radius) ** 2)
285
+ keep = j != i
286
+ if not keep.any():
287
+ continue
288
+ avg = np.average(out[j[keep]], axis=0, weights=w[keep])
289
+ nxt[i] = (1 - alpha) * out[i] + alpha * avg
290
+ out = nxt / np.clip(nxt.sum(axis=1, keepdims=True), 1e-12, None)
291
+ return out
292
+
293
+
294
+ # --------------------------------------------------------------------------- #
295
+ # The protocol harness
296
+ # --------------------------------------------------------------------------- #
297
+ def draw_shots(y_tgt, n_per_class: int, rng) -> np.ndarray:
298
+ """Indices of a stratified n-per-class sample of the target city."""
299
+ picks = []
300
+ for c in CLASSES:
301
+ pool = np.flatnonzero(np.asarray(y_tgt) == c)
302
+ if len(pool) == 0:
303
+ continue
304
+ take = min(n_per_class, len(pool))
305
+ picks.append(rng.choice(pool, size=take, replace=False))
306
+ return np.concatenate(picks)
307
+
308
+
309
+ def evaluate_protocol(model_create, X_src, y_src, X_tgt, y_tgt,
310
+ n_list=(5, 25, 50, 100, 200), n_repeats: int = 20,
311
+ seed: int = 42, decision="argmax", verbose: bool = True,
312
+ eval_subsample: int | None = None):
313
+ """
314
+ Run the official few-shot protocol and return
315
+ {n: {"scores": [...], "mean": float, "std": float, "per_class": array}}.
316
+
317
+ `y_tgt` is touched ONLY here: to draw the n-per-class sample that the rules
318
+ allow, and to score the held-out remainder. No tuning may read it.
319
+ """
320
+ X_src, y_src = np.asarray(X_src), np.asarray(y_src)
321
+ X_tgt, y_tgt = np.asarray(X_tgt), np.asarray(y_tgt)
322
+
323
+ # One FIXED evaluation subsample, shared by every n and every repeat.
324
+ #
325
+ # Scoring all 25,992 target rows on every one of 100 draws costs about 19
326
+ # seconds a draw for a 1200-tree LightGBM over ~390 features - 32 minutes
327
+ # per variant, hours for a table. A fixed subsample cuts that and is also a
328
+ # better experiment: the +- then reflects only the draw of the labelled
329
+ # sample, and every variant is compared on identical pixels. It is a plain
330
+ # random sample, so it keeps the target city's real class proportions,
331
+ # which is what macro F1 must be measured against.
332
+ eval_pool = np.arange(len(y_tgt))
333
+ if eval_subsample and eval_subsample < len(y_tgt):
334
+ eval_pool = np.sort(np.random.default_rng(seed).choice(
335
+ len(y_tgt), eval_subsample, replace=False))
336
+
337
+ results = {}
338
+ pool_sizes = {int(c): int((y_tgt == c).sum()) for c in CLASSES}
339
+ for n in n_list:
340
+ short = {c: k for c, k in pool_sizes.items() if k <= n}
341
+ if short:
342
+ print(f" n={n}: only {short} pixels available in those classes - "
343
+ "the draw is capped and the held-out remainder shrinks")
344
+ scores, per_class = [], []
345
+ for r in range(n_repeats):
346
+ rng = np.random.default_rng(seed + 1000 * n + r)
347
+ shot_idx = draw_shots(y_tgt, n, rng)
348
+ keep = np.setdiff1d(eval_pool, shot_idx, assume_unique=False)
349
+ if len(keep) < 4 or len(np.unique(y_tgt[keep])) < 2:
350
+ print(f" n={n} rep={r}: nothing left to score after drawing "
351
+ "the sample - skipped")
352
+ continue
353
+
354
+ fn = model_create(X_src, y_src, X_tgt,
355
+ X_tgt[shot_idx], y_tgt[shot_idx])
356
+ proba = np.asarray(fn(X_tgt[keep]), dtype=float)
357
+ mask = keep
358
+ if callable(decision):
359
+ pred = decision(proba)
360
+ else:
361
+ pred = CLASSES[np.argmax(proba, axis=1)]
362
+ scores.append(macro_f1(y_tgt[mask], pred))
363
+ per_class.append(per_class_f1(y_tgt[mask], pred))
364
+ if not scores:
365
+ results[n] = {"scores": np.array([]), "mean": float("nan"),
366
+ "std": float("nan"), "per_class": np.full(4, np.nan)}
367
+ continue
368
+ results[n] = {
369
+ "scores": np.array(scores),
370
+ "mean": float(np.mean(scores)),
371
+ "std": float(np.std(scores, ddof=1)) if len(scores) > 1 else 0.0,
372
+ "per_class": np.mean(per_class, axis=0),
373
+ }
374
+ if verbose:
375
+ print(f" n={n:<4d} macro F1 = {results[n]['mean']:.4f} "
376
+ f"+- {results[n]['std']:.4f} per-class "
377
+ f"{np.round(results[n]['per_class'], 3)}")
378
+ return results
379
+
380
+
381
+ # --------------------------------------------------------------------------- #
382
+ # The simulated-shift harness: choose every adaptation knob on MADRID only.
383
+ #
384
+ # Nothing above can be tuned honestly without this. A step that helps under
385
+ # pure label shift can lose 0.06-0.10 macro F1 under covariate shift, and
386
+ # without a simulator there is no way to know which regime you are in.
387
+ # --------------------------------------------------------------------------- #
388
+ def resample_to_prior(y, prior, rng, n: int | None = None) -> np.ndarray:
389
+ """Indices of a subsample of `y` whose class shares match `prior`."""
390
+ y = np.asarray(y)
391
+ prior = np.asarray(prior, dtype=float)
392
+ prior = prior / prior.sum()
393
+ pools = {c: np.flatnonzero(y == c) for c in CLASSES}
394
+ if n is None:
395
+ n = int(min(len(pools[c]) / prior[i] for i, c in enumerate(CLASSES)
396
+ if prior[i] > 0 and len(pools[c])))
397
+ picks = []
398
+ for i, c in enumerate(CLASSES):
399
+ k = int(round(prior[i] * n))
400
+ if k and len(pools[c]):
401
+ picks.append(rng.choice(pools[c], size=min(k, len(pools[c])), replace=False))
402
+ return np.concatenate(picks) if picks else np.array([], dtype=int)
403
+
404
+
405
+ def match_marginals(X_fold, X_src, X_tgt, strength: float = 1.0) -> np.ndarray:
406
+ """
407
+ Quantile-map each column of `X_fold` from the source city's marginal onto
408
+ the target city's marginal. Uses X_amsterdam features only, so it is legal,
409
+ and it turns a held-out Madrid fold into a stand-in for Amsterdam that keeps
410
+ its true labels.
411
+ """
412
+ X_fold = np.asarray(X_fold, dtype=float)
413
+ qs = np.linspace(0, 100, 201)
414
+ gs = np.percentile(np.asarray(X_src, float), qs, axis=0)
415
+ gt = np.percentile(np.asarray(X_tgt, float), qs, axis=0)
416
+ out = np.empty_like(X_fold)
417
+ for j in range(X_fold.shape[1]):
418
+ r = np.interp(X_fold[:, j], gs[:, j], qs, left=0.0, right=100.0)
419
+ out[:, j] = np.interp(r, qs, gt[:, j])
420
+ return X_fold + strength * (out - X_fold)
421
+
422
+
423
+ def shift_harness(fit_fn, X, y, blocks, X_tgt, strategies: dict,
424
+ n_folds: int = 5, n_rep: int = 4, seed: int = 0,
425
+ verbose: bool = True):
426
+ """
427
+ For each held-out spatial block, build three versions of it:
428
+ "clean" - as-is (no shift),
429
+ "label" - resampled to a random Dirichlet class prior,
430
+ "covariate" - feature marginals mapped onto the target city,
431
+ "both" - label shift and covariate shift together,
432
+ then score every decision strategy on each. A strategy is
433
+ `fn(proba, src_prior) -> labels`.
434
+
435
+ Returns a DataFrame of mean and worst-case macro F1 per (strategy, regime).
436
+ Ship only the strategies that are non-negative in the WORST case.
437
+ """
438
+ import pandas as pd
439
+
440
+ X, y = np.asarray(X, float), np.asarray(y)
441
+ uniq = np.unique(blocks)
442
+ rng0 = np.random.default_rng(seed)
443
+ folds = np.array_split(rng0.permutation(uniq), n_folds)
444
+ rows = []
445
+ for fi, f in enumerate(folds):
446
+ te = np.isin(blocks, f)
447
+ tr = ~te
448
+ model = fit_fn(X[tr], y[tr])
449
+ src_prior = np.array([(y[tr] == c).mean() for c in CLASSES])
450
+ Xte_clean = X[te]
451
+ Xte_cov = match_marginals(Xte_clean, X[tr], X_tgt)
452
+ for rep in range(n_rep):
453
+ rng = np.random.default_rng(seed + 100 * fi + rep)
454
+ alpha = rng.dirichlet(np.ones(4) * 1.5)
455
+ sel_lab = resample_to_prior(y[te], alpha, rng)
456
+ regimes = {
457
+ "clean": (Xte_clean, y[te]),
458
+ "label": (Xte_clean[sel_lab], y[te][sel_lab]),
459
+ "covariate": (Xte_cov, y[te]),
460
+ "both": (Xte_cov[sel_lab], y[te][sel_lab]),
461
+ }
462
+ for rname, (Xr, yr) in regimes.items():
463
+ if len(yr) == 0:
464
+ continue
465
+ proba = model.predict_proba(Xr)
466
+ for sname, fn in strategies.items():
467
+ rows.append(dict(fold=fi, rep=rep, regime=rname,
468
+ strategy=sname,
469
+ macro_f1=macro_f1(yr, fn(proba, src_prior))))
470
+ df = pd.DataFrame(rows)
471
+ piv = df.pivot_table(index="strategy", columns="regime", values="macro_f1",
472
+ aggfunc="mean")
473
+ worst = df.pivot_table(index="strategy", columns="regime", values="macro_f1",
474
+ aggfunc="min")
475
+ if "argmax" in piv.index:
476
+ regimes_cols = list(piv.columns)
477
+ piv["delta_vs_argmax_mean"] = (piv[regimes_cols].mean(axis=1)
478
+ - piv.loc["argmax", regimes_cols].mean())
479
+ # PER-REGIME deltas, then the worst of them. The previous version took
480
+ # each strategy's own worst cell anywhere and subtracted argmax's own
481
+ # worst cell anywhere, which compares two different regimes - and two
482
+ # different folds, since `worst` is a min over folds. A rule can then
483
+ # look safe because its bad regime happens to be argmax's bad regime
484
+ # too. What the decision actually needs is "in the regime where this
485
+ # rule does worst RELATIVE to argmax, how much does it lose".
486
+ dmean = piv[regimes_cols].sub(piv.loc["argmax", regimes_cols], axis=1)
487
+ piv["delta_vs_argmax_worst_regime"] = dmean.min(axis=1)
488
+ dworst = worst[regimes_cols].sub(worst.loc["argmax", regimes_cols], axis=1)
489
+ piv["delta_vs_argmax_worst_fold"] = dworst.min(axis=1)
490
+ if verbose:
491
+ print(piv.round(4).to_string())
492
+ return df, piv
493
+
494
+
495
+ DEFAULT_STRATEGIES = {
496
+ "argmax": lambda p, sp: CLASSES[np.argmax(p, axis=1)],
497
+ "logit_adj_tau0.25": lambda p, sp: CLASSES[np.argmax(logit_adjust(p, sp, 0.25), 1)],
498
+ "logit_adj_tau0.5": lambda p, sp: CLASSES[np.argmax(logit_adjust(p, sp, 0.5), 1)],
499
+ "logit_adj_tau1.0": lambda p, sp: CLASSES[np.argmax(logit_adjust(p, sp, 1.0), 1)],
500
+ "em_free": lambda p, sp: CLASSES[np.argmax(
501
+ prior_correct(p, sp, em_target_prior(p, sp)), 1)],
502
+ "em_shrunk": lambda p, sp: CLASSES[np.argmax(
503
+ prior_correct(p, sp, em_target_prior(p, sp, shrink=0.7)), 1)],
504
+ "em_pinned_shrunk": lambda p, sp: CLASSES[np.argmax(
505
+ prior_correct(p, sp, em_target_prior(p, sp, pin={4: 0.111}, shrink=0.7)), 1)],
506
+ "plugin_srcprior": lambda p, sp: macro_f1_plugin(p, sp)[0],
507
+ "tau0.5_then_plugin": lambda p, sp: adapt_pipeline(p, sp, tau=0.5)[0],
508
+ "tau0.5_em_plugin": lambda p, sp: adapt_pipeline(p, sp, tau=0.5, use_em=True,
509
+ pin={4: 0.111}, shrink=0.7)[0],
510
+ }
511
+
512
+
513
+ def summarise(results: dict) -> str:
514
+ lines = ["| n per class | macro F1 | std |", "|---|---|---|"]
515
+ for n, r in results.items():
516
+ lines.append(f"| {n} | {r['mean']:.4f} | {r['std']:.4f} |")
517
+ means = [r["mean"] for r in results.values()]
518
+ lines.append(f"\nAverage across n: {np.mean(means):.4f} peak: {np.max(means):.4f}")
519
+ return "\n".join(lines)
src/features.py ADDED
@@ -0,0 +1,634 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Raw wide-format Landsat parquet -> one feature row per 30 m pixel.
3
+
4
+ Two feature sets come out of here:
5
+
6
+ * the OFFICIAL 60 - a faithful re-implementation of the organisers'
7
+ Notebook 3, so our numbers stay comparable with the published baseline;
8
+ * the EXTENDED set - physically scaled surface reflectance, correct spectral
9
+ indices, construction-event ("step") detection, per-decade means, trends
10
+ and early/late contrasts. This is what the LLM verbaliser and the strong
11
+ GBDT models actually use.
12
+
13
+ Everything is vectorised over a (n_pixels x n_years) matrix per band, so a whole
14
+ city is processed in a couple of minutes on a laptop CPU.
15
+ """
16
+ from __future__ import annotations
17
+
18
+ import warnings
19
+
20
+ import numpy as np
21
+ import pandas as pd
22
+
23
+ from config import (BASE_BANDS, BLUE_MAX, CLASS_EVENTS, EARLY_CUTOFF,
24
+ FEATURE_COLS_60, INDEX_COLS, LATE_START, PID, SR_OFFSET,
25
+ SR_SCALE)
26
+
27
+ EPS = 1e-6
28
+
29
+
30
+ # --------------------------------------------------------------------------- #
31
+ # Stage 1 - load, label, flatten, gap-fill (official logic)
32
+ # --------------------------------------------------------------------------- #
33
+ def assign_age_class(df: pd.DataFrame) -> pd.DataFrame:
34
+ """Bin weighted_mean_year into classes 1-4 with city-specific boundaries."""
35
+ out = df
36
+ out["age_class"] = pd.Series(pd.NA, index=out.index, dtype="Int64")
37
+ for city, ev in CLASS_EVENTS.items():
38
+ mask = (out["city"] == city).to_numpy()
39
+ if mask.any():
40
+ binned = pd.cut(out.loc[mask, "weighted_mean_year"],
41
+ bins=[-np.inf, ev, 1984, 2004, np.inf],
42
+ labels=[1, 2, 3, 4])
43
+ out.loc[mask, "age_class"] = binned.astype("Int64")
44
+ return out
45
+
46
+
47
+ def make_flat_view(df: pd.DataFrame, agg: str = "first") -> pd.DataFrame:
48
+ """
49
+ One value per (pixel, year) per band; drop haze-contaminated rows.
50
+
51
+ agg="first" : the first QA-valid observation, exactly as the organisers do.
52
+ agg="median" : the median of all QA-valid observations in that year. With up
53
+ to three summer scenes per year, a single undetected thin
54
+ cloud moves a yearly value by more than any real surface
55
+ change, so the median is the more robust choice - use it for
56
+ our own features and keep "first" to reproduce the official
57
+ 60 exactly.
58
+ """
59
+ keep = [c for c in ["city", "year", "px_key", "py_key", "coverage",
60
+ "weighted_mean_year", "age_class"] if c in df.columns]
61
+ out = df[keep].copy()
62
+ n = len(df)
63
+
64
+ # Which satellite supplied the observation we end up using.
65
+ #
66
+ # This matters more than it looks. Reading scene_id (the dataset_* string
67
+ # column disagrees with it and is not to be trusted), the satellite chosen
68
+ # for a given year DIFFERS BETWEEN THE TWO CITIES in 11 of the 25 years
69
+ # from 2001 on: Madrid's late record is largely Landsat 7 ETM+, which has
70
+ # been SLC-off and striped since 2003, while Amsterdam's is largely
71
+ # Landsat 8/9 OLI. So part of any early-vs-late or year-to-year feature
72
+ # measures which instrument flew over, not what the building did - and that
73
+ # part is city-specific, which is the worst kind of nuisance here. Carrying
74
+ # the sensor through lets a model condition on it instead of confounding
75
+ # through it, and lets us measure how much it costs.
76
+ sat_code = np.zeros(n, dtype=np.int8) # 0 unknown, 1 TM, 2 ETM+, 3 OLI
77
+ _MAP = {"LT04": 1, "LT05": 1, "LE07": 2, "LC08": 3, "LC09": 3}
78
+ assigned = np.zeros(n, dtype=bool)
79
+ for slot in (1, 2, 3):
80
+ sc, qc = f"scene_id_{slot}", f"qa_valid_{slot}"
81
+ if sc not in df.columns or qc not in df.columns:
82
+ continue
83
+ valid = df[qc].fillna(False).to_numpy(dtype=bool)
84
+ codes = (df[sc].astype("string").str.slice(0, 4)
85
+ .map(_MAP).fillna(0).to_numpy(dtype=np.int8))
86
+ take = valid & ~assigned & (codes > 0)
87
+ sat_code[take] = codes[take]
88
+ assigned |= take
89
+ out["sat_code"] = sat_code
90
+
91
+ for band in BASE_BANDS:
92
+ if agg == "median":
93
+ stack = []
94
+ for slot in (1, 2, 3):
95
+ bc, qc = f"{band}_{slot}", f"qa_valid_{slot}"
96
+ if bc not in df.columns or qc not in df.columns:
97
+ continue
98
+ v = np.asarray(df[bc], dtype=np.float32).copy()
99
+ ok = df[qc].fillna(False).to_numpy(dtype=bool)
100
+ v[~ok] = np.nan
101
+ stack.append(v)
102
+ if stack:
103
+ with warnings.catch_warnings(): # all-NaN rows are expected
104
+ warnings.simplefilter("ignore", RuntimeWarning)
105
+ col = np.nanmedian(np.stack(stack), axis=0).astype(np.float32)
106
+ else:
107
+ col = np.full(n, np.nan, dtype=np.float32)
108
+ else:
109
+ col = np.full(n, np.nan, dtype=np.float32)
110
+ for slot in (1, 2, 3):
111
+ bc, qc = f"{band}_{slot}", f"qa_valid_{slot}"
112
+ if bc not in df.columns or qc not in df.columns:
113
+ continue
114
+ valid = df[qc].fillna(False).to_numpy(dtype=bool)
115
+ vals = np.asarray(df[bc], dtype=np.float32)
116
+ take = np.isnan(col) & valid & ~np.isnan(vals)
117
+ col[take] = vals[take]
118
+ out[band] = col
119
+ out = out.dropna(subset=BASE_BANDS)
120
+ before = len(out)
121
+ out = out[out["Blue"] <= BLUE_MAX]
122
+ removed = before - len(out)
123
+ if removed:
124
+ print(f" removed {removed:,} haze rows (Blue > {BLUE_MAX:,})")
125
+ return out
126
+
127
+
128
+ def period_coverage(flat: pd.DataFrame) -> pd.DataFrame:
129
+ """has_early_data / has_late_data, computed on OBSERVED years (pre gap-fill)."""
130
+ pixels = flat[PID].drop_duplicates()
131
+ early = flat.loc[flat["year"] <= EARLY_CUTOFF, PID].drop_duplicates().assign(has_early_data=1.0)
132
+ late = flat.loc[flat["year"] >= LATE_START, PID].drop_duplicates().assign(has_late_data=1.0)
133
+ return (pixels.merge(early, on=PID, how="left")
134
+ .merge(late, on=PID, how="left")
135
+ .fillna({"has_early_data": 0.0, "has_late_data": 0.0}))
136
+
137
+
138
+ def to_year_matrices(flat: pd.DataFrame):
139
+ """
140
+ Pivot the observed rows into one (n_pixels x n_years) matrix per band and
141
+ gap-fill exactly as the organisers do: linear interior, nearest at the edges.
142
+
143
+ Returns (pixel index DataFrame, years array, {band: matrix}, observed mask)
144
+ """
145
+ years = np.arange(int(flat["year"].min()), int(flat["year"].max()) + 1)
146
+ pix = flat[PID].drop_duplicates().sort_values(PID).reset_index(drop=True)
147
+ pix_pos = pd.Series(np.arange(len(pix)), index=pd.MultiIndex.from_frame(pix))
148
+
149
+ ri = pix_pos.loc[pd.MultiIndex.from_frame(flat[PID])].to_numpy()
150
+ ci = (flat["year"].to_numpy() - years[0]).astype(np.int32)
151
+
152
+ mats = {}
153
+ observed = np.zeros((len(pix), len(years)), dtype=bool)
154
+ observed[ri, ci] = True
155
+ sat = np.zeros((len(pix), len(years)), dtype=np.int8)
156
+ if "sat_code" in flat.columns:
157
+ sat[ri, ci] = flat["sat_code"].to_numpy(dtype=np.int8)
158
+ for band in BASE_BANDS:
159
+ M = np.full((len(pix), len(years)), np.nan, dtype=np.float32)
160
+ M[ri, ci] = flat[band].to_numpy(dtype=np.float32)
161
+ M = (pd.DataFrame(M).interpolate(axis=1, method="linear")
162
+ .bfill(axis=1).ffill(axis=1).to_numpy(dtype=np.float32))
163
+ mats[band] = M
164
+ return pix, years, mats, observed, sat
165
+
166
+
167
+ # --------------------------------------------------------------------------- #
168
+ # Stage 2 - the official 60 features, from DN values
169
+ # --------------------------------------------------------------------------- #
170
+ def _indices_from(mats: dict) -> dict:
171
+ """The organisers' five indices, exactly as their Notebook 3 defines them."""
172
+ B, G, R = mats["Blue"], mats["Green"], mats["Red"]
173
+ N, S1, S2 = mats["NIR"], mats["SWIR1"], mats["SWIR2"]
174
+ return {
175
+ "NDVI": (N - R) / (N + R + EPS),
176
+ "NDBI": (S1 - N) / (S1 + N + EPS),
177
+ "UI": (S2 - N) / (S2 + N + EPS),
178
+ "MNDWI": (G - S1) / (G + S1 + EPS),
179
+ "BSI": ((S1 + R) - (N + B)) / ((S1 + R) + (N + B) + EPS),
180
+ }
181
+
182
+
183
+ # Tasseled-cap coefficients for OLI surface reflectance (Baig et al. 2014).
184
+ # Wetness in particular separates impervious surfaces from soil better than
185
+ # any of the five indices above.
186
+ TC_COEF = {
187
+ "TCB": (0.3029, 0.2786, 0.4733, 0.5599, 0.5080, 0.1872),
188
+ "TCG": (-0.2941, -0.2430, -0.5424, 0.7276, 0.0713, -0.1608),
189
+ "TCW": (0.1511, 0.1973, 0.3283, 0.3407, -0.7117, -0.4559),
190
+ }
191
+
192
+
193
+ def _extra_indices_from(sr: dict) -> dict:
194
+ """NBR, tasseled cap, and chromaticity (band / total brightness).
195
+
196
+ Chromaticity matters for transfer: dividing each band by the pixel's own
197
+ 6-band sum removes albedo, illumination and the city-level brightness
198
+ offset entirely, and keeps only the spectral SHAPE.
199
+ """
200
+ order = [sr[b] for b in BASE_BANDS]
201
+ total = sum(order) + EPS
202
+ out = {"NBR": (sr["NIR"] - sr["SWIR2"]) / (sr["NIR"] + sr["SWIR2"] + EPS)}
203
+ for name, coef in TC_COEF.items():
204
+ out[name] = sum(c * m for c, m in zip(coef, order))
205
+ for b in BASE_BANDS:
206
+ out[f"chrom{b}"] = sr[b] / total
207
+ return out
208
+
209
+
210
+ def _median_filter3(M: np.ndarray) -> np.ndarray:
211
+ """
212
+ 3-point running median along the time axis.
213
+
214
+ One undetected thin cloud moves a yearly value by more than any real
215
+ surface change, so the change detector runs on the filtered series while
216
+ the reported statistics stay on the raw one.
217
+ """
218
+ a = np.pad(M, ((0, 0), (1, 1)), mode="edge")
219
+ return np.median(np.stack([a[:, :-2], a[:, 1:-1], a[:, 2:]]), axis=0)
220
+
221
+
222
+ def build_official_60(pix, years, mats, indicators) -> pd.DataFrame:
223
+ """Reproduce the organisers' 60 features (DN units, ddof=1 std)."""
224
+ idx = _indices_from(mats)
225
+ early = years <= EARLY_CUTOFF
226
+ late = years >= LATE_START
227
+ out = {}
228
+ for b in BASE_BANDS:
229
+ M = mats[b]
230
+ out[f"{b}_mean"] = M.mean(1)
231
+ out[f"{b}_std"] = M.std(1, ddof=1)
232
+ out[f"{b}_early_mean"] = M[:, early].mean(1)
233
+ out[f"{b}_early_std"] = M[:, early].std(1, ddof=1)
234
+ out[f"{b}_late_mean"] = M[:, late].mean(1)
235
+ out[f"{b}_late_std"] = M[:, late].std(1, ddof=1)
236
+ D = np.diff(M, axis=1)
237
+ out[f"d_{b}_mean"] = D.mean(1)
238
+ out[f"d_{b}_std"] = D.std(1, ddof=1)
239
+ for i in INDEX_COLS:
240
+ out[f"{i}_mean"] = idx[i].mean(1)
241
+ out[f"{i}_std"] = idx[i].std(1, ddof=1)
242
+ df = pd.concat([pix, pd.DataFrame(out, index=pix.index)], axis=1)
243
+ df = df.merge(indicators, on=PID, how="left")
244
+ return df[PID + FEATURE_COLS_60]
245
+
246
+
247
+ # --------------------------------------------------------------------------- #
248
+ # Stage 3 - extended, physically-scaled features
249
+ # --------------------------------------------------------------------------- #
250
+ DECADES = [(1984, 1993), (1994, 2003), (2004, 2013), (2014, 2024)]
251
+ DECADE_TAGS = ["84_93", "94_03", "04_13", "14_24"]
252
+ STEP_SIGNALS = ["brightness", "SWIR1", "NDVI"]
253
+
254
+
255
+ def _slope_per_decade(M: np.ndarray, years: np.ndarray) -> np.ndarray:
256
+ """OLS slope of each row against year, expressed per decade."""
257
+ t = (years - years.mean()).astype(np.float32)
258
+ denom = float((t * t).sum())
259
+ return (M * t).sum(1) / denom * 10.0
260
+
261
+
262
+ def _step_stats(M: np.ndarray, years: np.ndarray, win: int = 3):
263
+ """
264
+ Construction-event detector. For every candidate break year, compare the
265
+ mean of the `win` years before with the mean of the `win` years after.
266
+ Returns (year of the largest absolute jump, signed size of that jump,
267
+ number of year-on-year jumps above 3 x MAD).
268
+ """
269
+ n, T = M.shape
270
+ cs = np.concatenate([np.zeros((n, 1), np.float32),
271
+ np.cumsum(M, axis=1, dtype=np.float32)], axis=1)
272
+
273
+ def win_mean(a, b):
274
+ return (cs[:, b] - cs[:, a]) / np.float32(b - a)
275
+
276
+ cuts = np.arange(win, T - win + 1)
277
+ diffs = np.stack([win_mean(c, c + win) - win_mean(c - win, c) for c in cuts], axis=1)
278
+
279
+ # A two-sample t statistic, not just the raw difference.
280
+ #
281
+ # argmax over candidate years ALWAYS returns something, so a pixel that was
282
+ # never rebuilt still gets a "biggest step" - and inspection caught the
283
+ # verbaliser telling the model that a building from 1924 changed abruptly in
284
+ # 1998. What distinguishes a real construction event is the size of the step
285
+ # RELATIVE TO the scatter within the two windows being compared, which is
286
+ # exactly a t statistic. The earlier version divided by the scatter about
287
+ # the whole-series mean, which the step itself inflates, so it could not
288
+ # separate the two cases.
289
+ cs2 = np.concatenate([np.zeros((n, 1), np.float32),
290
+ np.cumsum(M.astype(np.float32) ** 2, axis=1, dtype=np.float32)],
291
+ axis=1)
292
+
293
+ def win_var(a, b):
294
+ k = np.float32(b - a)
295
+ mu = (cs[:, b] - cs[:, a]) / k
296
+ ex2 = (cs2[:, b] - cs2[:, a]) / k
297
+ return np.maximum(ex2 - mu * mu, 0.0) * (k / max(k - 1.0, 1.0))
298
+
299
+ tstats = np.empty_like(diffs)
300
+ for m_i, c in enumerate(cuts):
301
+ vb, va = win_var(c - win, c), win_var(c, c + win)
302
+ pooled = np.sqrt(np.maximum((vb + va) / 2.0, 0.0) * (2.0 / win)) + EPS
303
+ tstats[:, m_i] = diffs[:, m_i] / pooled
304
+
305
+ # The break year is chosen by the largest MAGNITUDE, not the largest t.
306
+ #
307
+ # Selecting on t was tried and was a disaster: Amsterdam's agreement with
308
+ # the true construction year fell from 69.4% within five years to 31.5%,
309
+ # and Madrid's from 44.6% to 28.3%. The t statistic is maximised wherever
310
+ # the pooled variance of a 3-year window happens to be near zero, which is
311
+ # noise, not a construction event. The magnitude is the signal.
312
+ #
313
+ # The t value at that same break is still reported as a feature, where it
314
+ # means "how large is this shift relative to the local scatter" and the
315
+ # trees can use it or ignore it. It is NOT used for any wording in the
316
+ # prompt - see the note in verbalize.render_prompt.
317
+ k = np.argmax(np.abs(diffs), axis=1)
318
+ rows = np.arange(n)
319
+ step_mag = diffs[rows, k]
320
+ step_t = np.abs(tstats[rows, k])
321
+ step_year = years[cuts[k]].astype(np.float32)
322
+
323
+ D = np.diff(M, axis=1)
324
+ mad = np.median(np.abs(D - np.median(D, axis=1, keepdims=True)), axis=1) + EPS
325
+ n_jumps = (np.abs(D) > (3.0 * mad[:, None])).sum(1).astype(np.float32)
326
+ return step_year, step_mag, n_jumps, step_t
327
+
328
+
329
+ SPIKE_SIGNALS = ["BSI", "SWIR1", "brightness"]
330
+ # Sensor eras. The early/late split at 2004 also straddles a change of
331
+ # atmospheric-correction algorithm (LEDAPS for TM/ETM+, LaSRC for OLI), and the
332
+ # artefact is latitude-dependent, so part of any early/late contrast measures
333
+ # which algorithm ran rather than what the building did. Splitting at the
334
+ # sensor boundary as well makes that confound explicit instead of hidden.
335
+ TM_ERA = (1984, 2011)
336
+ OLI_ERA = (2013, 2024)
337
+
338
+
339
+ SAT_NAMES = {1: "tm", 2: "etm", 3: "oli"}
340
+
341
+
342
+ def sensor_features(pix, years, observed, sat) -> dict:
343
+ """
344
+ Per-pixel sensor composition: which instrument actually supplied each year.
345
+
346
+ Not a building property at all - which is the point. These columns let a
347
+ model absorb the sensor nuisance explicitly instead of attributing it to
348
+ building age, and they let us measure how much of the Madrid/Amsterdam gap
349
+ is instrument rather than architecture.
350
+ """
351
+ early = years <= EARLY_CUTOFF
352
+ late = years >= LATE_START
353
+ out = {}
354
+ n_obs = np.clip(observed.sum(1), 1, None).astype(np.float32)
355
+ for code, nm in SAT_NAMES.items():
356
+ hit = (sat == code)
357
+ out[f"sat_share_{nm}"] = hit.sum(1) / n_obs
358
+ out[f"sat_share_{nm}_early"] = (hit & early[None, :]).sum(1) / np.clip(
359
+ (observed & early[None, :]).sum(1), 1, None)
360
+ out[f"sat_share_{nm}_late"] = (hit & late[None, :]).sum(1) / np.clip(
361
+ (observed & late[None, :]).sum(1), 1, None)
362
+ # how often the instrument changes from one observed year to the next
363
+ switches = np.zeros(len(sat), dtype=np.float32)
364
+ for i in range(sat.shape[0]):
365
+ s = sat[i][observed[i]]
366
+ s = s[s > 0]
367
+ if len(s) > 1:
368
+ switches[i] = float((s[1:] != s[:-1]).sum())
369
+ out["sat_switches"] = switches
370
+ return {k: np.asarray(v, dtype=np.float32) for k, v in out.items()}
371
+
372
+
373
+ def build_extended(pix, years, mats_dn, observed, sat=None) -> pd.DataFrame:
374
+ """Surface-reflectance features: correct indices, trends, steps, decades."""
375
+ sr = {b: mats_dn[b] * np.float32(SR_SCALE) + np.float32(SR_OFFSET)
376
+ for b in BASE_BANDS}
377
+ idx = _indices_from(sr)
378
+ extra = _extra_indices_from(sr)
379
+ bright = np.mean(np.stack([sr[b] for b in BASE_BANDS]), axis=0, dtype=np.float32)
380
+ signals = {"brightness": bright}
381
+ signals.update(sr)
382
+ signals.update(idx)
383
+ signals.update(extra)
384
+
385
+ early = years <= EARLY_CUTOFF
386
+ late = years >= LATE_START
387
+ tm = (years >= TM_ERA[0]) & (years <= TM_ERA[1])
388
+ oli = (years >= OLI_ERA[0]) & (years <= OLI_ERA[1])
389
+ out = {}
390
+
391
+ # Full statistics for the signals worth the columns; a short set for the
392
+ # chromaticity and tasseled-cap extras, to keep the table manageable.
393
+ rich = {"brightness", *BASE_BANDS, *INDEX_COLS}
394
+ for name, M in signals.items():
395
+ out[f"sr_{name}_mean"] = M.mean(1)
396
+ out[f"sr_{name}_std"] = M.std(1, ddof=1)
397
+ out[f"sr_{name}_early_mean"] = M[:, early].mean(1)
398
+ out[f"sr_{name}_late_mean"] = M[:, late].mean(1)
399
+ out[f"sr_{name}_late_minus_early"] = (out[f"sr_{name}_late_mean"]
400
+ - out[f"sr_{name}_early_mean"])
401
+ # std scales with brightness, so a raw std is partly a city marker;
402
+ # the coefficient of variation is not.
403
+ out[f"sr_{name}_cv"] = out[f"sr_{name}_std"] / (np.abs(out[f"sr_{name}_mean"]) + EPS)
404
+ if name not in rich:
405
+ continue
406
+ out[f"sr_{name}_early_std"] = M[:, early].std(1, ddof=1)
407
+ out[f"sr_{name}_late_std"] = M[:, late].std(1, ddof=1)
408
+ out[f"sr_{name}_slope_decade"] = _slope_per_decade(M, years)
409
+ out[f"sr_{name}_min"] = M.min(1)
410
+ out[f"sr_{name}_max"] = M.max(1)
411
+ out[f"sr_{name}_tm_mean"] = M[:, tm].mean(1)
412
+ out[f"sr_{name}_oli_mean"] = M[:, oli].mean(1)
413
+ out[f"sr_{name}_oli_minus_tm"] = out[f"sr_{name}_oli_mean"] - out[f"sr_{name}_tm_mean"]
414
+ for (y0, y1), tag in zip(DECADES, DECADE_TAGS):
415
+ sel = (years >= y0) & (years <= y1)
416
+ out[f"sr_{name}_dec{tag}_mean"] = M[:, sel].mean(1)
417
+
418
+ # ---- construction events, on the cloud-filtered series -----------------
419
+ smoothed = {k: _median_filter3(signals[k]) for k in set(STEP_SIGNALS) | set(SPIKE_SIGNALS)}
420
+ for name in STEP_SIGNALS:
421
+ sy, sm, nj, st = _step_stats(smoothed[name], years)
422
+ out[f"step_{name}_year"] = sy
423
+ out[f"step_{name}_mag"] = sm
424
+ out[f"step_{name}_njumps"] = nj
425
+ out[f"step_{name}_t"] = st
426
+ resid = signals[name] - signals[name].mean(1, keepdims=True)
427
+ out[f"step_{name}_snr"] = np.abs(sm) / (resid.std(1, ddof=1) + EPS)
428
+
429
+ # ---- transient bare-soil spike: a building site is bare for 1-3 years --
430
+ for name in SPIKE_SIGNALS:
431
+ M = smoothed[name]
432
+ med = np.median(M, axis=1)
433
+ mad = np.median(np.abs(M - med[:, None]), axis=1) + EPS
434
+ k = np.argmax(M, axis=1)
435
+ out[f"spike_{name}_year"] = years[k].astype(np.float32)
436
+ out[f"spike_{name}_z"] = (M[np.arange(len(M)), k] - med) / mad
437
+ # does it come back down? a permanent rise is a new roof, a spike that
438
+ # returns to baseline is a construction site
439
+ after = np.clip(k + 3, 0, M.shape[1] - 1)
440
+ out[f"spike_{name}_returns"] = ((M[np.arange(len(M)), after] - med) / mad)
441
+
442
+ # ---- post-construction greening: street trees take decades to mature ---
443
+ ndvi = signals["NDVI"]
444
+ sy_b = out["step_brightness_year"]
445
+ slope_after = np.zeros(len(ndvi), dtype=np.float32)
446
+ years_since = np.zeros(len(ndvi), dtype=np.float32)
447
+ for i in range(len(ndvi)):
448
+ k = int(np.searchsorted(years, sy_b[i]))
449
+ seg = ndvi[i, k:]
450
+ years_since[i] = float(years[-1] - sy_b[i])
451
+ if len(seg) >= 5:
452
+ t = np.arange(len(seg), dtype=np.float32)
453
+ t -= t.mean()
454
+ slope_after[i] = float((seg * t).sum() / (t * t).sum() * 10.0)
455
+ out["post_step_NDVI_slope_decade"] = slope_after
456
+ out["years_since_step"] = years_since
457
+
458
+ # ---- what was there before the change ----------------------------------
459
+ # 0 vegetated, 1 bare, 2 already built, 3 water. For a pixel that changed
460
+ # after 2004 this separates greenfield expansion from brownfield infill and
461
+ # from land reclamation, which have completely different trajectories but
462
+ # land in the same class.
463
+ pre_ndvi = out["sr_NDVI_early_mean"]
464
+ pre_mndwi = out["sr_MNDWI_early_mean"]
465
+ pre_ndbi = out["sr_NDBI_early_mean"]
466
+ pre = np.full(len(pre_ndvi), 2.0, dtype=np.float32)
467
+ pre[pre_ndvi > 0.30] = 0.0
468
+ pre[(pre_ndvi <= 0.30) & (pre_ndbi < 0.0) & (out["sr_BSI_early_mean"] > 0.05)] = 1.0
469
+ pre[pre_mndwi > 0.10] = 3.0
470
+ out["pre_state"] = pre
471
+
472
+ if sat is not None:
473
+ out.update(sensor_features(pix, years, observed, sat))
474
+
475
+ out["n_observed_years"] = observed.sum(1).astype(np.float32)
476
+ out["first_observed_year"] = years[np.argmax(observed, axis=1)].astype(np.float32)
477
+ last = observed.shape[1] - 1 - np.argmax(observed[:, ::-1], axis=1)
478
+ out["last_observed_year"] = years[last].astype(np.float32)
479
+
480
+ return pd.concat([pix, pd.DataFrame(out, index=pix.index)], axis=1)
481
+
482
+
483
+ PRE_STATE_WORDS = {0: "vegetated - farmland, scrub or garden",
484
+ 1: "bare soil or a cleared site",
485
+ 2: "already built on or paved",
486
+ 3: "open water (later reclaimed)"}
487
+
488
+
489
+ def add_city_geometry(df: pd.DataFrame) -> pd.DataFrame:
490
+ """
491
+ Two purely geometric, label-free features.
492
+
493
+ `radial_frac`: distance from the city centroid, divided by the city's own
494
+ 95th percentile, so it is comparable between cities. European cities share a
495
+ morphological gradient - oldest and densest in the historic core - and this
496
+ is the only transferable handle on the class 1 / class 2 split, whose
497
+ boundary year (1960 vs 1945) has no spectral correlate at all.
498
+
499
+ `dist_to_water_px`: distance to the nearest pixel that looks like open
500
+ water, which is what separates the Amsterdam canal belt from everything
501
+ else.
502
+ """
503
+ from scipy.spatial import cKDTree
504
+
505
+ out = df.copy()
506
+ xy = out[PID].to_numpy(float)
507
+ centre = np.median(xy, axis=0)
508
+ r = np.linalg.norm(xy - centre, axis=1)
509
+ out["radial_px"] = r
510
+ out["radial_frac"] = r / (np.percentile(r, 95) + EPS)
511
+
512
+ far = float(np.percentile(r, 99) * 2.0) # sentinel: "no water anywhere near"
513
+ water = out["sr_MNDWI_max"].to_numpy(float) > 0.0 if "sr_MNDWI_max" in out else None
514
+ if water is not None and water.sum() > 0:
515
+ tree = cKDTree(xy[water])
516
+ d, _ = tree.query(xy, k=1)
517
+ out["dist_to_water_px"] = np.minimum(d, far)
518
+ else:
519
+ out["dist_to_water_px"] = far
520
+ return out
521
+
522
+
523
+ # --------------------------------------------------------------------------- #
524
+ # Fallback: start from the organisers' preprocessed_data.pkl
525
+ # --------------------------------------------------------------------------- #
526
+ def from_official_pkl(path):
527
+ """
528
+ Load the organisers' `preprocessed_data.pkl` into the same shape
529
+ `build_city` produces, so the GBDT notebook can run without the 3 GB of raw
530
+ parquet.
531
+
532
+ What you get: the official 60 features, the labels, and the pixel
533
+ coordinates - enough for every classical model, the spatial validation, the
534
+ shift harness and the full few-shot protocol.
535
+
536
+ What you do NOT get: the extended features (true surface reflectance,
537
+ construction-event detection, per-decade means, tasseled cap,
538
+ chromaticity, building coverage) and therefore not the physical values the
539
+ verbaliser's word bands need. The pkl's X is already z-scored with Madrid's
540
+ scaler, so the numbers are unitless. Within-city RANKS still work, which is
541
+ the main transfer mechanism - so a ranks-only prompt is possible from this
542
+ file alone, just a thinner one.
543
+
544
+ Returns (madrid_df, amsterdam_df).
545
+ """
546
+ import pickle
547
+
548
+ with open(path, "rb") as fh:
549
+ d = pickle.load(fh)
550
+ names = list(d["feature_names"])
551
+ out = []
552
+ for city in ("madrid", "amsterdam"):
553
+ X = np.asarray(d[f"X_{city}"], dtype=np.float32)
554
+ y = np.asarray(d[f"y_{city}"], dtype=int)
555
+ ids = np.asarray(d[f"pixel_ids_{city}"])
556
+ df = pd.DataFrame(X, columns=names)
557
+ df.insert(0, "px_key", ids[:, 0])
558
+ df.insert(1, "py_key", ids[:, 1])
559
+ df["age_class"] = y
560
+ df["city"] = city.capitalize()
561
+ out.append(df)
562
+ print(f"loaded {path}")
563
+ for name, df in zip(("Madrid", "Amsterdam"), out):
564
+ print(f" {name}: {len(df):,} pixels x {len(names)} official features "
565
+ f"| classes {df.age_class.value_counts().sort_index().to_dict()}")
566
+ print(" note: features are Madrid-scaled z-scores, not physical units; "
567
+ "the extended feature set is absent")
568
+
569
+ # Guard against loading the wrong file - a truncated download, a synthetic
570
+ # test fixture, or another city's export - and silently reporting numbers
571
+ # from it. The organisers' pixel counts are published in the competitors'
572
+ # guide, so they are a free integrity check.
573
+ import os
574
+ if os.environ.get("ALLOW_SYNTHETIC_PKL") == "1":
575
+ print(" WARNING: integrity check disabled by "
576
+ "ALLOW_SYNTHETIC_PKL - numbers are NOT real")
577
+ return tuple(out)
578
+ expected = {"Madrid": 76_263, "Amsterdam": 25_992}
579
+ problems = []
580
+ for name, df in zip(("Madrid", "Amsterdam"), out):
581
+ if len(df) != expected[name]:
582
+ problems.append(f"{name} has {len(df):,} pixels, expected "
583
+ f"{expected[name]:,}")
584
+ if len(names) != 60:
585
+ problems.append(f"{len(names)} feature names, expected 60")
586
+ if problems:
587
+ raise ValueError(
588
+ "This does not look like the organisers' preprocessed_data.pkl:\n "
589
+ + "\n ".join(problems)
590
+ + f"\nFile: {path}\nIf you meant to use a test fixture, load it "
591
+ "explicitly rather than through this function.")
592
+ print(" integrity check passed (pixel counts match the published figures)")
593
+ return tuple(out)
594
+
595
+
596
+ # --------------------------------------------------------------------------- #
597
+ # Driver
598
+ # --------------------------------------------------------------------------- #
599
+ def build_city(parquet_path, city_hint: str | None = None, verbose: bool = True,
600
+ agg: str = "median", geometry: bool = True) -> pd.DataFrame:
601
+ if verbose:
602
+ print(f"reading {parquet_path}")
603
+ df = pd.read_parquet(parquet_path)
604
+ if "city" not in df.columns and city_hint:
605
+ df["city"] = city_hint
606
+ if verbose:
607
+ print(f" {len(df):,} rows x {df.shape[1]} cols | "
608
+ f"years {df['year'].min()}-{df['year'].max()}")
609
+
610
+ df = assign_age_class(df)
611
+ flat = make_flat_view(df, agg=agg)
612
+ del df
613
+ indicators = period_coverage(flat)
614
+
615
+ meta_cols = [c for c in ["city", "coverage", "weighted_mean_year", "age_class"]
616
+ if c in flat.columns]
617
+ meta = (flat.sort_values(PID + ["year"])
618
+ .groupby(PID, as_index=False)[meta_cols].first())
619
+
620
+ pix, years, mats, observed, sat = to_year_matrices(flat)
621
+ del flat
622
+
623
+ off = build_official_60(pix, years, mats, indicators)
624
+ ext = build_extended(pix, years, mats, observed, sat)
625
+
626
+ feats = off.merge(ext, on=PID, how="left").merge(meta, on=PID, how="left")
627
+ feats = feats[feats["age_class"].notna()].reset_index(drop=True)
628
+ feats["age_class"] = feats["age_class"].astype(int)
629
+ if geometry:
630
+ feats = add_city_geometry(feats)
631
+ if verbose:
632
+ n_ext = ext.shape[1] - len(PID)
633
+ print(f" -> {len(feats):,} labelled pixels | 60 official + {n_ext} extended features")
634
+ return feats
src/gbdt.py ADDED
@@ -0,0 +1,776 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Model 1 and Model 2 of the submission: LightGBM and CatBoost, plus the
3
+ cross-city plumbing they share with the LLM track.
4
+
5
+ Contents
6
+ * per-city normalisation (rank / robust z) - unsupervised, no labels
7
+ * CORAL covariance alignment - unsupervised, no labels
8
+ * adversarial validation - finds city-specific features
9
+ * macro-F1 eval metrics for both libraries
10
+ * ordinal (cumulative-binary) decomposition - the classes are ordered
11
+ * several `modelCreate` adapters for the few-shot protocol
12
+ """
13
+ from __future__ import annotations
14
+
15
+ import numpy as np
16
+ from sklearn.metrics import f1_score
17
+
18
+ CLASSES = np.array([1, 2, 3, 4])
19
+ SEED = 42
20
+
21
+
22
+ # --------------------------------------------------------------------------- #
23
+ # Per-city normalisation. Fit one of these per city on its own features; it
24
+ # needs no labels, so it is legal for Amsterdam.
25
+ # --------------------------------------------------------------------------- #
26
+ def rankable_mask(X, min_unique: int = 10, max_tie_share: float = 0.5):
27
+ """
28
+ Which columns are safe to rank-transform, judged on the DESIGN city only.
29
+
30
+ A column is unsafe when one value dominates it. The test that matters is
31
+ the size of the biggest tie, not the number of distinct values:
32
+ `first_observed_year` has 20 distinct values but is 1984 for 98.4% of
33
+ pixels, and ranking it turned a domain AUC of 0.507 into 0.997.
34
+ """
35
+ X = np.asarray(X, dtype=np.float64)
36
+ out = np.empty(X.shape[1], dtype=bool)
37
+ for j in range(X.shape[1]):
38
+ col = X[:, j]
39
+ vals, counts = np.unique(col, return_counts=True)
40
+ out[j] = (len(vals) >= min_unique
41
+ and counts.max() / len(col) < max_tie_share)
42
+ return out
43
+
44
+
45
+ class CityNormaliser:
46
+ """
47
+ mode="rank" : map each feature to its within-city percentile (0-1).
48
+ Kills every city-level offset and gain, including ones no
49
+ linear method would catch. Cost: it also erases genuine
50
+ between-city differences in level, and it is distorted
51
+ when the two cities have different class proportions.
52
+ mode="robustz" : (x - median) / IQR per city. Gentler, keeps shape.
53
+ mode=None : identity. The right answer on this data (see
54
+ TREE_NORMALISE), so it has to be a real, explicit mode.
55
+
56
+ `mode=None` used to fall through `if self.mode != "rank"` in `transform`
57
+ and apply robust z anyway. Every caller that asked for no normalisation got
58
+ robust z instead, silently - `mc_refit`, whose default is `TREE_NORMALISE`
59
+ (None), was measured that way. An unrecognised mode now raises rather than
60
+ quietly becoming one of the other two.
61
+ """
62
+
63
+ MODES = (None, "none", "robustz", "rank")
64
+
65
+ def __init__(self, mode: str | None = "robustz", n_grid: int = 1001,
66
+ min_unique: int = 10, max_tie_share: float = 0.5,
67
+ rankable=None):
68
+ if mode not in self.MODES:
69
+ raise ValueError(f"unknown normalise mode {mode!r}; "
70
+ f"expected one of {self.MODES}")
71
+ self.mode = None if mode == "none" else mode
72
+ self.n_grid = n_grid
73
+ self.min_unique = min_unique
74
+ self.max_tie_share = max_tie_share
75
+ # Which columns may be rank-transformed. This is a STRUCTURAL decision
76
+ # about the features and it MUST be identical in every city: deciding
77
+ # it per city let a column be ranked in one and z-scored in the other,
78
+ # putting them on different scales and making the two cities perfectly
79
+ # separable (domain AUC 1.000 on `n_observed_years`). So it is computed
80
+ # once on the design city and passed in here for the target.
81
+ self.rankable = rankable
82
+
83
+ def fit(self, X):
84
+ X = np.asarray(X, dtype=np.float64)
85
+ if self.mode is None:
86
+ return self
87
+ self.med_ = np.median(X, axis=0)
88
+ q75, q25 = np.percentile(X, [75, 25], axis=0)
89
+ self.iqr_ = np.where((q75 - q25) > 1e-12, q75 - q25, 1.0)
90
+ if self.mode == "rank":
91
+ self.q_ = np.linspace(0, 100, self.n_grid)
92
+ self.grid_ = np.percentile(X, self.q_, axis=0) # (n_grid, d)
93
+ # Heavily tied columns must NOT be rank-transformed.
94
+ #
95
+ # Measured on the real data: `first_observed_year` is 1984 for
96
+ # almost every pixel, and its domain AUC went from 0.507 raw to
97
+ # 0.997 after ranking - the transform CREATED a perfect city marker
98
+ # where there was none. A dominant tie makes the ECDF one long
99
+ # plateau, and the plateau sits at a slightly different quantile in
100
+ # each city, so every pixel lands on a different side of it.
101
+ #
102
+ # The test that matters is the size of the biggest tie, not the
103
+ # number of distinct values: that column has 20 distinct values and
104
+ # still breaks. Offending columns fall back to robust z.
105
+ if self.rankable is not None:
106
+ self.rankable_ = np.asarray(self.rankable, dtype=bool)
107
+ else:
108
+ self.rankable_ = rankable_mask(
109
+ X, min_unique=self.min_unique,
110
+ max_tie_share=self.max_tie_share)
111
+ return self
112
+
113
+ def transform(self, X):
114
+ X = np.asarray(X, dtype=np.float64)
115
+ if self.mode is None:
116
+ return X
117
+ if self.mode != "rank":
118
+ return (X - self.med_) / self.iqr_
119
+ out = (X - self.med_) / self.iqr_
120
+ for j in np.flatnonzero(self.rankable_):
121
+ out[:, j] = np.interp(X[:, j], self.grid_[:, j], self.q_ / 100.0,
122
+ left=0.0, right=1.0)
123
+ return out
124
+
125
+ def fit_transform(self, X):
126
+ return self.fit(X).transform(X)
127
+
128
+
129
+ def coral(X_src, X_tgt, eps: float = 1e-6):
130
+ """
131
+ CORAL: whiten the source with its own covariance, then re-colour it with
132
+ the target covariance, so the two feature clouds have the same second-order
133
+ statistics. Uses target FEATURES only - no labels.
134
+ Returns the transformed source.
135
+ """
136
+ Xs = np.asarray(X_src, dtype=np.float64)
137
+ Xt = np.asarray(X_tgt, dtype=np.float64)
138
+ ms, mt = Xs.mean(0), Xt.mean(0)
139
+ Cs = np.cov(Xs - ms, rowvar=False) + eps * np.eye(Xs.shape[1])
140
+ Ct = np.cov(Xt - mt, rowvar=False) + eps * np.eye(Xt.shape[1])
141
+
142
+ def _half(C, power):
143
+ w, V = np.linalg.eigh(C)
144
+ w = np.clip(w, eps, None)
145
+ return V @ np.diag(w ** power) @ V.T
146
+
147
+ A = _half(Cs, -0.5) @ _half(Ct, 0.5)
148
+ return (Xs - ms) @ A + mt
149
+
150
+
151
+ # --------------------------------------------------------------------------- #
152
+ # Adversarial validation: which features simply say "this is Madrid"?
153
+ # --------------------------------------------------------------------------- #
154
+ def adversarial_validation(X_src, X_tgt, feature_names, n_estimators: int = 300,
155
+ seed: int = SEED, max_rows: int = 30_000):
156
+ """
157
+ Train a classifier to tell the two cities apart. An AUC near 1.0 means the
158
+ domains are trivially separable; the top-importance features are the ones
159
+ carrying the city signature rather than the age signal.
160
+
161
+ LightGBM rather than a random forest, and capped at `max_rows` per city:
162
+ this is called several times over ~380 features, and the AUC it reports
163
+ saturates long before 30k rows per city, so a forest here would dominate
164
+ the notebook's runtime for no extra information.
165
+
166
+ Returns (auc, DataFrame of importances sorted descending).
167
+ """
168
+ import lightgbm as lgb
169
+ import pandas as pd
170
+ from sklearn.model_selection import cross_val_predict
171
+ from sklearn.metrics import roc_auc_score
172
+
173
+ rng = np.random.default_rng(seed)
174
+
175
+ def cap(A):
176
+ A = np.asarray(A, dtype=np.float32)
177
+ if len(A) > max_rows:
178
+ A = A[rng.choice(len(A), max_rows, replace=False)]
179
+ return A
180
+
181
+ Xs, Xt = cap(X_src), cap(X_tgt)
182
+ X = np.vstack([Xs, Xt])
183
+ y = np.r_[np.zeros(len(Xs)), np.ones(len(Xt))]
184
+ clf = lgb.LGBMClassifier(objective="binary", n_estimators=n_estimators,
185
+ learning_rate=0.05, num_leaves=31,
186
+ min_child_samples=50, colsample_bytree=0.7,
187
+ subsample=0.8, subsample_freq=1, max_bin=127,
188
+ n_jobs=-1, random_state=seed, verbosity=-1)
189
+ p = cross_val_predict(clf, X, y, cv=3, method="predict_proba", n_jobs=1)[:, 1]
190
+ auc = roc_auc_score(y, p)
191
+ clf.fit(X, y)
192
+ gain = clf.feature_importances_.astype(float)
193
+ imp = (pd.DataFrame({"feature": list(feature_names),
194
+ "city_importance": gain / max(gain.sum(), 1.0)})
195
+ .sort_values("city_importance", ascending=False)
196
+ .reset_index(drop=True))
197
+ return float(auc), imp
198
+
199
+
200
+ # --------------------------------------------------------------------------- #
201
+ # Macro-F1 eval metrics
202
+ # --------------------------------------------------------------------------- #
203
+ def lgb_macro_f1(y_true, y_pred):
204
+ """lightgbm 4.x feval. y_pred arrives as (n*4,) or (n,4) depending on version."""
205
+ p = np.asarray(y_pred)
206
+ if p.ndim == 1:
207
+ p = p.reshape(len(y_true), -1, order="F") if p.size != len(y_true) else p
208
+ lab = np.argmax(p, axis=1) if p.ndim == 2 else p
209
+ return "macro_f1", float(f1_score(y_true, lab, average="macro", zero_division=0)), True
210
+
211
+
212
+ class CatMacroF1:
213
+ """CatBoost custom eval metric (higher is better)."""
214
+
215
+ def is_max_optimal(self):
216
+ return True
217
+
218
+ def evaluate(self, approxes, target, weight):
219
+ p = np.array(approxes).T
220
+ lab = np.argmax(p, axis=1)
221
+ return float(f1_score(np.asarray(target, dtype=int), lab,
222
+ average="macro", zero_division=0)), 1.0
223
+
224
+ def get_final_error(self, error, weight):
225
+ return error
226
+
227
+
228
+ # --------------------------------------------------------------------------- #
229
+ # Default parameter sets. Tuned toward robustness, not toward the last
230
+ # 0.002 of Madrid CV: the test set is a different city.
231
+ # --------------------------------------------------------------------------- #
232
+ LGBM_PARAMS = dict(
233
+ objective="multiclass", num_class=4, learning_rate=0.03, n_estimators=1200,
234
+ num_leaves=31, max_depth=-1, min_child_samples=80, subsample=0.8,
235
+ subsample_freq=1, colsample_bytree=0.6, reg_alpha=0.5, reg_lambda=5.0,
236
+ max_bin=127, n_jobs=-1, random_state=SEED, verbosity=-1,
237
+ )
238
+
239
+ CATBOOST_PARAMS = dict(
240
+ loss_function="MultiClass", classes_count=4, iterations=2000,
241
+ learning_rate=0.03, depth=6, l2_leaf_reg=6.0, random_strength=1.5,
242
+ bootstrap_type="Bernoulli", subsample=0.8, rsm=0.7,
243
+ boosting_type="Plain", random_seed=SEED, verbose=False,
244
+ allow_writing_files=False,
245
+ )
246
+
247
+
248
+ def fit_lgbm(X, y, params=None, class_weight=None, sample_weight=None):
249
+ import lightgbm as lgb
250
+ p = dict(LGBM_PARAMS)
251
+ if params:
252
+ p.update(params)
253
+ if class_weight is not None:
254
+ p["class_weight"] = class_weight
255
+ m = lgb.LGBMClassifier(**p)
256
+ m.fit(np.asarray(X), np.asarray(y) - 1, sample_weight=sample_weight)
257
+ return m
258
+
259
+
260
+ def fit_catboost(X, y, params=None, class_weights=None, sample_weight=None):
261
+ from catboost import CatBoostClassifier
262
+ p = dict(CATBOOST_PARAMS)
263
+ if params:
264
+ p.update(params)
265
+ if class_weights is not None:
266
+ p["class_weights"] = list(class_weights)
267
+ m = CatBoostClassifier(**p)
268
+ m.fit(np.asarray(X), np.asarray(y) - 1, sample_weight=sample_weight)
269
+ return m
270
+
271
+
272
+ def balanced_class_weights(y):
273
+ y = np.asarray(y)
274
+ counts = np.array([(y == c).sum() for c in CLASSES], dtype=float)
275
+ w = len(y) / (4.0 * np.clip(counts, 1, None))
276
+ return w
277
+
278
+
279
+ # --------------------------------------------------------------------------- #
280
+ # Ordinal decomposition: the four classes are ordered age bins, so three
281
+ # cumulative binary problems ("older than X?") use that structure.
282
+ # --------------------------------------------------------------------------- #
283
+ class OrdinalGBDT:
284
+ """P(y > k) for k = 1,2,3, then differenced into class probabilities."""
285
+
286
+ def __init__(self, fit_fn=fit_lgbm, params=None):
287
+ self.fit_fn = fit_fn
288
+ self.params = params
289
+
290
+ def fit(self, X, y, sample_weight=None):
291
+ import lightgbm as lgb
292
+ X, y = np.asarray(X), np.asarray(y)
293
+ params = dict(LGBM_PARAMS)
294
+ params.update(self.params or {})
295
+ params.pop("num_class", None) # binary problems, not 4-way
296
+ params["objective"] = "binary"
297
+ self.models_ = []
298
+ for k in (1, 2, 3):
299
+ m = lgb.LGBMClassifier(**params)
300
+ m.fit(X, (y > k).astype(int), sample_weight=sample_weight)
301
+ self.models_.append(m)
302
+ return self
303
+
304
+ def predict_proba(self, X):
305
+ X = np.asarray(X)
306
+ s = np.column_stack([m.predict_proba(X)[:, 1] for m in self.models_])
307
+ s = np.minimum.accumulate(s, axis=1) # enforce monotonicity
308
+ p1 = 1.0 - s[:, 0]
309
+ p2 = s[:, 0] - s[:, 1]
310
+ p3 = s[:, 1] - s[:, 2]
311
+ p4 = s[:, 2]
312
+ P = np.clip(np.column_stack([p1, p2, p3, p4]), 1e-9, None)
313
+ return P / P.sum(axis=1, keepdims=True)
314
+
315
+
316
+ # --------------------------------------------------------------------------- #
317
+ # Calibration on the few-shot sample
318
+ # --------------------------------------------------------------------------- #
319
+ def stratification_weights(y_shot, target_prior):
320
+ """
321
+ The organisers hand us n labelled pixels *per class*, so the labelled sample
322
+ is balanced while the city is not. Anything fitted on it therefore inherits
323
+ a uniform class prior - which, in testing, cost class 4 almost 0.17 F1.
324
+
325
+ These weights undo the stratification: each row is weighted by
326
+ target_prior[c] / (1/4), so the weighted sample behaves like a random
327
+ sample of the city. `target_prior` must come from somewhere legal - the EM
328
+ estimate on unlabelled Amsterdam probabilities, or the organisers' own
329
+ published class shares.
330
+ """
331
+ y = np.asarray(y_shot, dtype=int)
332
+ tp = np.asarray(target_prior, dtype=float)
333
+ tp = tp / tp.sum()
334
+ counts = np.array([(y == c).sum() for c in CLASSES], dtype=float)
335
+ share = counts / counts.sum()
336
+ w = (tp / np.clip(share, 1e-9, None))[y - 1]
337
+ return w * len(y) / w.sum()
338
+
339
+
340
+ def vector_scaling(proba_shot, y_shot, l2: float = 1.0, n_iter: int = 400,
341
+ lr: float = 0.1, sample_weight=None):
342
+ """
343
+ Learn per-class scale a_k and bias b_k on the log-probabilities using only
344
+ the n-per-class labelled target sample: q = softmax(a * log p + b).
345
+ Eight parameters, so it survives n=5 per class.
346
+
347
+ Pass `sample_weight=stratification_weights(...)` unless you really want the
348
+ balanced prior of the shot sample baked into the output.
349
+ Returns a callable proba -> recalibrated proba.
350
+ """
351
+ P = np.log(np.clip(np.asarray(proba_shot, dtype=np.float64), 1e-9, 1))
352
+ y = np.asarray(y_shot, dtype=int) - 1
353
+ Y = np.eye(4)[y]
354
+ sw = (np.ones(len(y)) if sample_weight is None
355
+ else np.asarray(sample_weight, dtype=float))
356
+ sw = sw / sw.sum()
357
+ a, b = np.ones(4), np.zeros(4)
358
+ for _ in range(n_iter):
359
+ z = P * a + b
360
+ z -= z.max(axis=1, keepdims=True)
361
+ q = np.exp(z)
362
+ q /= q.sum(axis=1, keepdims=True)
363
+ d = (q - Y) * sw[:, None]
364
+ ga = (d * P).sum(0) + l2 * (a - 1.0)
365
+ gb = d.sum(0) + l2 * b
366
+ a -= lr * ga
367
+ b -= lr * gb
368
+
369
+ def apply(proba):
370
+ lp = np.log(np.clip(np.asarray(proba, dtype=np.float64), 1e-9, 1))
371
+ z = lp * a + b
372
+ z -= z.max(axis=1, keepdims=True)
373
+ e = np.exp(z)
374
+ return e / e.sum(axis=1, keepdims=True)
375
+
376
+ apply.params = (a, b)
377
+ return apply
378
+
379
+
380
+ # --------------------------------------------------------------------------- #
381
+ # modelCreate adapters for evaluation.evaluate_protocol
382
+ # --------------------------------------------------------------------------- #
383
+ #: What `modelCreate` is allowed to see of the target city.
384
+ #
385
+ #: Prof. Thron's specification says Amsterdam data "may only be used to
386
+ #: EVALUATE modelCreate", and that every design decision must be made without
387
+ #: looking at Amsterdam. That rules out picking features or hyperparameters by
388
+ #: Amsterdam's score - which we never do. It leaves one genuine question open:
389
+ #: may modelCreate, as a FUNCTION applied to a new city, read that city's
390
+ #: unlabelled feature matrix at adaptation time?
391
+ #:
392
+ #: Both readings are defensible, so both are supported and both get reported.
393
+ #:
394
+ #: "transductive" - modelCreate(D^(N)_city, X_city). Per-city statistics come
395
+ #: from all the city's unlabelled pixels. This is standard unsupervised
396
+ #: domain adaptation and matches the deployment story: pointing this at
397
+ #: Baghdad, you have every Landsat pixel and merely lack labels. The
398
+ #: organisers' own Notebook 3 is transductive in spirit, though it
399
+ #: normalises Amsterdam with Madrid's scaler.
400
+ #:
401
+ #: "inductive" - modelCreate(D^(N)_city) only. Per-city statistics are
402
+ #: estimated from the N-per-class labelled sample alone. Strictly
403
+ #: compliant on the tightest reading. At N=200 that is 800 pixels, which
404
+ #: estimates a percentile well; at N=5 it is 20 and noticeably noisier.
405
+ #:
406
+ #: Ask the organisers which they intend. Until then, lead with whichever is
407
+ #: weaker and quote the other alongside it.
408
+ TARGET_ACCESS = ("transductive", "inductive")
409
+
410
+
411
+ def _target_reference(X_tgt_all, X_shot, access: str):
412
+ """The rows a per-city statistic may be estimated from."""
413
+ if access == "inductive":
414
+ return np.asarray(X_shot, dtype=float)
415
+ return np.asarray(X_tgt_all, dtype=float)
416
+
417
+
418
+ #: Default normalisation for the TREE models: none.
419
+ #:
420
+ #: Decided on Madrid, with blocked CV in which each held-out block is
421
+ #: normalised with its own statistics exactly as a new city would be:
422
+ #: none 0.5989 +- 0.0230 | rank 0.5697 +- 0.0789 | robust z 0.5690 +- 0.0830
423
+ #: and the same ordering then held on Amsterdam (0.5928 against 0.5268).
424
+ #:
425
+ #: The reason is that the absolute level is signal, not only nuisance: building
426
+ #: era is spatially clustered, so a district's average brightness says something
427
+ #: real about when it was built, and normalising a region to its own median
428
+ #: subtracts that along with the city offset.
429
+ #:
430
+ #: DISTANCE-based methods are different and keep their normalisation. A nearest
431
+ #: neighbour over unscaled features is dominated by whichever column has the
432
+ #: largest units, which is an artefact of units rather than a statement about
433
+ #: transfer.
434
+ TREE_NORMALISE = None
435
+ DISTANCE_NORMALISE = "robustz"
436
+
437
+
438
+ def mc_zero_shot(fit=fit_lgbm, normalise: str | None = TREE_NORMALISE,
439
+ use_coral: bool = False, balanced: bool = True,
440
+ access: str = "transductive"):
441
+ """
442
+ Train on the source city. Under "transductive" access the few-shot sample
443
+ is ignored entirely; under "inductive" it is still used, but only to
444
+ estimate the target city's feature distribution, never its labels.
445
+ """
446
+ cache = {}
447
+
448
+ def model_create(X_src, y_src, X_tgt_all, X_shot, y_shot):
449
+ ref = _target_reference(X_tgt_all, X_shot, access)
450
+ rk = rankable_mask(X_src) if normalise == "rank" else None
451
+ nt = CityNormaliser(normalise, rankable=rk).fit(ref) if normalise else None
452
+ key = (normalise, use_coral, balanced, access)
453
+ if key not in cache:
454
+ ns = (CityNormaliser(normalise, rankable=rk).fit(X_src)
455
+ if normalise else None)
456
+ Xs = ns.transform(X_src) if ns else np.asarray(X_src, float)
457
+ if use_coral:
458
+ Xs = coral(Xs, nt.transform(ref) if nt else ref)
459
+ cw = balanced_class_weights(y_src) if balanced else None
460
+ model = (fit(Xs, y_src, class_weight=dict(zip(range(4), cw)) if cw is not None else None)
461
+ if fit is fit_lgbm else fit(Xs, y_src, class_weights=cw))
462
+ cache[key] = model
463
+ model = cache[key]
464
+
465
+ def predict(X):
466
+ return model.predict_proba(nt.transform(X) if nt else X)
467
+ return predict
468
+ return model_create
469
+
470
+
471
+ def mc_recalibrated(fit=fit_lgbm, normalise: str | None = TREE_NORMALISE,
472
+ use_coral: bool = False, balanced: bool = True,
473
+ l2: float = 1.0, correct_stratification: bool = True,
474
+ tune_decision: bool = True):
475
+ """
476
+ Zero-shot model + vector scaling learned on the n-per-class sample, then a
477
+ macro-F1 decision rule tuned on the same sample.
478
+
479
+ `correct_stratification` reweights the shot sample towards the class prior
480
+ that EM estimates from the unlabelled target features, so neither the
481
+ calibration nor the decision rule inherits the artificial 25/25/25/25 prior
482
+ of the sample.
483
+
484
+ `tune_decision` matters more than the calibration: a well-calibrated model
485
+ maximises log-likelihood, and argmax of a calibrated posterior is NOT the
486
+ macro-F1 optimum. The shrinkage is tied to the sample size, so n=5 barely
487
+ moves the rule and n=200 moves it freely.
488
+ """
489
+ base = mc_zero_shot(fit, normalise, use_coral, balanced)
490
+
491
+ def model_create(X_src, y_src, X_tgt_all, X_shot, y_shot):
492
+ from evaluation import em_target_prior, tune_class_weights
493
+ predict0 = base(X_src, y_src, X_tgt_all, X_shot, y_shot)
494
+ sw = None
495
+ if correct_stratification:
496
+ src_prior = np.array([(np.asarray(y_src) == c).mean() for c in CLASSES])
497
+ tgt_prior = em_target_prior(predict0(X_tgt_all), src_prior)
498
+ sw = stratification_weights(y_shot, tgt_prior)
499
+ p_shot = predict0(X_shot)
500
+ cal = vector_scaling(p_shot, y_shot, l2=l2, sample_weight=sw)
501
+ w = np.ones(4)
502
+ if tune_decision:
503
+ n_shot = len(y_shot)
504
+ shrink = float(np.clip(1.0 - n_shot / 400.0, 0.0, 0.9))
505
+ w = tune_class_weights(cal(p_shot), y_shot, sample_weight=sw,
506
+ shrink=shrink)
507
+
508
+ def predict(X):
509
+ p = cal(predict0(X)) * w[None, :]
510
+ return p / np.clip(p.sum(axis=1, keepdims=True), 1e-12, None)
511
+ return predict
512
+ return model_create
513
+
514
+
515
+ def mc_shot_weighted(fit=fit_lgbm, normalise: str | None = TREE_NORMALISE,
516
+ shot_weight: float = 30.0, balanced: bool = True):
517
+ """
518
+ Retrain on source + the few target rows, with the target rows upweighted.
519
+ `shot_weight` must be chosen on the source city (e.g. by simulating a
520
+ held-out Madrid district as a pseudo-target), never on the target score.
521
+ """
522
+ def model_create(X_src, y_src, X_tgt_all, X_shot, y_shot):
523
+ rk = rankable_mask(X_src) if normalise == "rank" else None
524
+ ns = CityNormaliser(normalise, rankable=rk).fit(X_src) if normalise else None
525
+ nt = (CityNormaliser(normalise, rankable=rk).fit(X_tgt_all)
526
+ if normalise else None)
527
+ Xs = ns.transform(X_src) if ns else np.asarray(X_src)
528
+ Xsh = nt.transform(X_shot) if nt else np.asarray(X_shot)
529
+ X = np.vstack([Xs, Xsh])
530
+ y = np.r_[np.asarray(y_src), np.asarray(y_shot)]
531
+ w = np.r_[np.ones(len(y_src)), np.full(len(y_shot), shot_weight)]
532
+ if balanced:
533
+ cw = balanced_class_weights(y)
534
+ w = w * cw[np.asarray(y) - 1]
535
+ model = fit(X, y, sample_weight=w)
536
+
537
+ def predict(X_):
538
+ return model.predict_proba(nt.transform(X_) if nt else X_)
539
+ return predict
540
+ return model_create
541
+
542
+
543
+ def mc_knn(normalise: str = DISTANCE_NORMALISE, k: int | None = None,
544
+ columns=None):
545
+ """
546
+ The pure-retrieval classifier: distance-weighted k-NN over the n labelled
547
+ target pixels alone, ignoring the source city completely.
548
+
549
+ Measure this FIRST. It costs seconds and it bounds everything else: if
550
+ 50-per-class k-NN already beats the reported 0.34 zero-shot Random Forest,
551
+ then the retrieval layer is the story and the source model is support, not
552
+ the other way round. It is also the baseline the retrieval features in the
553
+ LLM prompt have to beat to justify their complexity.
554
+ """
555
+ def model_create(X_src, y_src, X_tgt_all, X_shot, y_shot):
556
+ rk = rankable_mask(X_src) if normalise == "rank" else None
557
+ nt = CityNormaliser(normalise, rankable=rk).fit(X_tgt_all)
558
+ Xs = nt.transform(X_shot)
559
+ y_shot = np.asarray(y_shot)
560
+ kk = k if k is not None else int(min(15, max(3, 3 * len(y_shot) // 4)))
561
+
562
+ # Squared distances via the (a-b)^2 = a^2 - 2ab + b^2 identity, in row
563
+ # chunks. The naive Xn[:, None, :] - Xs[None, :, :] would allocate
564
+ # n_query x n_pool x n_features floats - 63 GB for 26k queries against
565
+ # 800 labelled pixels over 380 features.
566
+ pool_sq = (Xs ** 2).sum(1)
567
+ CHUNK = 2048
568
+
569
+ def predict(X):
570
+ Xn = nt.transform(X)
571
+ P = np.zeros((len(Xn), 4))
572
+ for s in range(0, len(Xn), CHUNK):
573
+ Q = Xn[s:s + CHUNK]
574
+ d2 = (Q ** 2).sum(1)[:, None] - 2.0 * (Q @ Xs.T) + pool_sq[None, :]
575
+ d = np.sqrt(np.maximum(d2, 0.0))
576
+ idx = np.argpartition(d, kk - 1, axis=1)[:, :kk]
577
+ dd = np.take_along_axis(d, idx, axis=1)
578
+ tau = np.median(dd, axis=1, keepdims=True) + 1e-9
579
+ w = np.exp(-(dd / tau) ** 2)
580
+ lab = y_shot[idx]
581
+ for j, c in enumerate(CLASSES):
582
+ P[s:s + CHUNK, j] = (w * (lab == c)).sum(axis=1)
583
+ P = np.clip(P, 1e-9, None)
584
+ return P / P.sum(axis=1, keepdims=True)
585
+ return predict
586
+ return model_create
587
+
588
+
589
+ def mc_refit(normalise: str | None = TREE_NORMALISE, decay: float = 0.9,
590
+ balanced: bool = False):
591
+ """
592
+ LightGBM `Booster.refit`: keep the tree STRUCTURE learned on the source city
593
+ and move only the leaf values towards the target, blended by `decay`.
594
+
595
+ This is the right shape for the problem. Tree structure is what needs tens
596
+ of thousands of rows; leaf values are a handful of numbers per tree and can
597
+ be nudged by a few hundred. Recommended from n=25 upwards; below that there
598
+ is nothing estimable and plain recalibration is safer.
599
+ """
600
+ cache = {}
601
+
602
+ def model_create(X_src, y_src, X_tgt_all, X_shot, y_shot):
603
+ key = (normalise, balanced)
604
+ if key not in cache:
605
+ rk = rankable_mask(X_src) if normalise == "rank" else None
606
+ ns = CityNormaliser(normalise, rankable=rk).fit(X_src)
607
+ nt = CityNormaliser(normalise, rankable=rk).fit(X_tgt_all)
608
+ cw = (dict(zip(range(4), balanced_class_weights(y_src)))
609
+ if balanced else None)
610
+ cache[key] = (fit_lgbm(ns.transform(X_src), y_src, class_weight=cw), nt)
611
+ model, nt = cache[key]
612
+ Xsh = nt.transform(X_shot)
613
+ booster = model.booster_.refit(Xsh, np.asarray(y_shot) - 1,
614
+ decay_rate=decay)
615
+
616
+ def predict(X):
617
+ P = booster.predict(nt.transform(X))
618
+ P = np.asarray(P, dtype=float)
619
+ if P.ndim == 1:
620
+ P = P.reshape(-1, 4)
621
+ return P / np.clip(P.sum(axis=1, keepdims=True), 1e-12, None)
622
+ return predict
623
+ return model_create
624
+
625
+
626
+ def single_feature_domain_auc(X_src, X_tgt, feature_names):
627
+ """
628
+ One AUC per feature for the question "which city is this?".
629
+
630
+ 0.5 means the feature is domain-invariant; 1.0 means it is a pure city
631
+ marker and a Madrid-trained split on it is meaningless in Amsterdam. Run it
632
+ before and after per-city normalisation: the table of the two columns is the
633
+ most convincing single figure for the write-up, and it is the honest basis
634
+ for dropping features. Prune conservatively - drop only a feature that is
635
+ still above ~0.90 after normalisation AND carries little Madrid importance.
636
+ """
637
+ import pandas as pd
638
+ from sklearn.metrics import roc_auc_score
639
+
640
+ Xs = np.asarray(X_src, dtype=float)
641
+ Xt = np.asarray(X_tgt, dtype=float)
642
+ y = np.r_[np.zeros(len(Xs)), np.ones(len(Xt))]
643
+ rows = []
644
+ for j, name in enumerate(feature_names):
645
+ v = np.r_[Xs[:, j], Xt[:, j]]
646
+ ok = np.isfinite(v)
647
+ if ok.sum() < 10 or np.std(v[ok]) == 0:
648
+ rows.append(dict(feature=name, domain_auc=0.5))
649
+ continue
650
+ a = roc_auc_score(y[ok], v[ok])
651
+ rows.append(dict(feature=name, domain_auc=max(a, 1 - a)))
652
+ return (pd.DataFrame(rows).sort_values("domain_auc", ascending=False)
653
+ .reset_index(drop=True))
654
+
655
+
656
+ def mc_prototype_official():
657
+ """
658
+ The organisers' own few-shot baseline, reproduced exactly: class means of
659
+ the labelled target sample, nearest prototype by plain Euclidean distance,
660
+ on features standardised with the SOURCE city's scaler.
661
+
662
+ This is the row every other row in the table has to beat, and it is a
663
+ stronger baseline than it looks. Both the prototypes and the queries come
664
+ from the target city and receive the same transform, so any affine
665
+ difference between the cities cancels exactly in a nearest-prototype
666
+ comparison. The source scaler therefore does no harm here; it only sets the
667
+ relative weighting of features. In synthetic testing this plain version beat
668
+ a rank-normalised Mahalanobis version by 0.05 macro F1 at n=25, because the
669
+ rank transform distorts the cluster geometry that prototypes depend on.
670
+ Beat it on measurements, not on sophistication.
671
+ """
672
+ def model_create(X_src, y_src, X_tgt_all, X_shot, y_shot):
673
+ mu_s = np.mean(X_src, axis=0)
674
+ sd_s = np.std(X_src, axis=0)
675
+ sd_s = np.where(sd_s > 1e-12, sd_s, 1.0)
676
+ Xsh = (np.asarray(X_shot, float) - mu_s) / sd_s
677
+ y_shot = np.asarray(y_shot)
678
+ present = [c for c in CLASSES if (y_shot == c).sum() > 0]
679
+ proto = np.stack([Xsh[y_shot == c].mean(0) for c in present])
680
+
681
+ def predict(X):
682
+ Xn = (np.asarray(X, float) - mu_s) / sd_s
683
+ d = np.linalg.norm(Xn[:, None, :] - proto[None, :, :], axis=2)
684
+ P = np.zeros((len(Xn), 4))
685
+ # hard nearest-prototype assignment, as in their code
686
+ for row, j in enumerate(np.argmin(d, axis=1)):
687
+ P[row, present[j] - 1] = 1.0
688
+ return np.clip(P, 1e-9, None) / np.clip(P, 1e-9, None).sum(1, keepdims=True)
689
+ return predict
690
+ return model_create
691
+
692
+
693
+ def mc_prototype(normalise: str | None = DISTANCE_NORMALISE,
694
+ metric: str = "euclidean",
695
+ shrink: float = 0.5, temperature: float = 1.0):
696
+ """
697
+ Prototype classifier: one class mean from the n labelled target pixels,
698
+ nearest prototype wins. Needs no source model, so it is the honest floor.
699
+
700
+ `normalise` defaults to per-city robust z rather than rank. Rank mapping
701
+ forces each marginal to be uniform, which is excellent for tree splits and
702
+ bad here - it distorts the cluster shape that a distance to a class mean
703
+ depends on. Measured: rank + Mahalanobis lost 0.05 macro F1 at n=25 against
704
+ the plain version on synthetic data.
705
+
706
+ `metric`:
707
+ "euclidean" - isotropic distance.
708
+ "within" - Mahalanobis on the POOLED WITHIN-CLASS covariance of the
709
+ shots (shrunk towards diagonal). This is the statistically
710
+ right metric for prototypes, and it is not the same thing
711
+ as using the target city's total covariance, which is
712
+ dominated by the BETWEEN-class spread and therefore shrinks
713
+ exactly the directions that separate the classes.
714
+ "total" - Mahalanobis on the target city's total covariance; kept
715
+ only so the ablation table can show why it is worse.
716
+ """
717
+ def model_create(X_src, y_src, X_tgt_all, X_shot, y_shot):
718
+ rk = rankable_mask(X_src) if normalise == "rank" else None
719
+ nt = (CityNormaliser(normalise, rankable=rk).fit(X_tgt_all)
720
+ if normalise else None)
721
+ Xt = nt.transform(X_tgt_all) if nt else np.asarray(X_tgt_all, float)
722
+ Xsh = nt.transform(X_shot) if nt else np.asarray(X_shot, float)
723
+ y_shot = np.asarray(y_shot)
724
+ present = [c for c in CLASSES if (y_shot == c).sum() > 0]
725
+ mu = np.stack([Xsh[y_shot == c].mean(0) for c in present])
726
+ d_ = Xt.shape[1]
727
+
728
+ if metric == "within":
729
+ # Pooled within-class scatter of the shots. Macro F1 on synthetic
730
+ # anisotropic data against the organisers' Euclidean prototype:
731
+ # n=5 0.719 vs 0.649, n=25 0.967 vs 0.821, n=100 0.988 vs 0.833 -
732
+ # the largest single gain of any adaptation step tried.
733
+ #
734
+ # Without the blending below it LOSES at n=5, where 20 rows cannot
735
+ # fill a 20x20 covariance, so the estimate is pulled towards the
736
+ # target city's total covariance and towards its own diagonal by how
737
+ # much data there actually is.
738
+ #
739
+ # Honest caveat: those synthetic classes are Gaussian sharing one
740
+ # anisotropic within-class covariance, i.e. exactly the model this
741
+ # metric assumes. Expect less on real Landsat features; trust the
742
+ # ORDERING of the four variants, not the absolute numbers.
743
+ parts = [Xsh[y_shot == c] - Xsh[y_shot == c].mean(0)
744
+ for c in present if (y_shot == c).sum() > 1]
745
+ resid = np.vstack(parts) if parts else np.zeros((0, d_))
746
+ n_res = len(resid)
747
+ Cw = (np.cov(resid, rowvar=False) if n_res > d_
748
+ else (np.diag(np.var(resid, axis=0) + 1e-6) if n_res
749
+ else np.eye(d_)))
750
+ Ct = np.cov(Xt, rowvar=False)
751
+ a = float(np.clip(d_ / (d_ + max(n_res, 1)), 0.0, 0.8))
752
+ C = (1 - a) * Cw + a * Ct
753
+ s = float(np.clip(shrink * d_ / (d_ + max(n_res, 1)) * 2.0, 0.1, 0.9))
754
+ C = (1 - s) * C + s * np.diag(np.diag(C) + 1e-6) + 1e-6 * np.eye(d_)
755
+ A = np.linalg.pinv(C)
756
+ elif metric == "total":
757
+ Ct = np.cov(Xt, rowvar=False)
758
+ Ct = (1 - shrink) * Ct + shrink * np.diag(np.diag(Ct)) + 1e-6 * np.eye(d_)
759
+ A = np.linalg.pinv(Ct)
760
+ else:
761
+ A = np.eye(d_)
762
+
763
+ def predict(X):
764
+ Xn = nt.transform(X) if nt else np.asarray(X, float)
765
+ d = np.stack([np.einsum("ij,jk,ik->i", Xn - m, A, Xn - m) for m in mu],
766
+ axis=1)
767
+ z = -0.5 * d / max(temperature, 1e-6)
768
+ z -= z.max(axis=1, keepdims=True)
769
+ e = np.exp(z)
770
+ e = e / e.sum(axis=1, keepdims=True)
771
+ P = np.full((len(Xn), 4), 1e-9)
772
+ for j, c in enumerate(present):
773
+ P[:, c - 1] = e[:, j]
774
+ return P / P.sum(axis=1, keepdims=True)
775
+ return predict
776
+ return model_create
src/retrieval.py ADDED
@@ -0,0 +1,420 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ The adaptation layer: retrieval over whatever labelled target pixels we are
3
+ given, plus label-free neighbourhood features.
4
+
5
+ Three distinct channels, deliberately kept separate so their contributions can
6
+ be measured one at a time:
7
+
8
+ A. NEIGHBOURHOOD FEATURES - no labels at all. The mean feature vector of the
9
+ surrounding pixels. Buildings are built in blocks, so a pixel's
10
+ neighbourhood says a great deal about its era, and this works at n=0.
11
+ B. SPECTRAL RETRIEVAL - the k labelled pixels in this city whose features are
12
+ closest to the query. Summarised as a class histogram, and/or rendered as
13
+ in-context demonstrations.
14
+ C. SPATIAL RETRIEVAL - the labelled pixels physically nearest the query.
15
+ Weak when only 20 pixels are labelled, strong at 800.
16
+
17
+ Leakage discipline: when these fields are built for MADRID training rows, the
18
+ labelled pool is a RANDOM n-per-class sample drawn from spatial blocks that
19
+ exclude the query pixel. That reproduces exactly the situation the model meets
20
+ in Amsterdam, and a pixel can never retrieve itself or its own neighbours.
21
+ """
22
+ from __future__ import annotations
23
+
24
+ import numpy as np
25
+ from sklearn.neighbors import NearestNeighbors
26
+
27
+ CLASSES = np.array([1, 2, 3, 4])
28
+
29
+
30
+ # --------------------------------------------------------------------------- #
31
+ # A. Label-free neighbourhood features
32
+ # --------------------------------------------------------------------------- #
33
+ def add_neighbourhood_features(df, coord_cols=("px_key", "py_key"),
34
+ feature_cols=None, k: int = 8,
35
+ max_radius: float = 10.0, prefix: str = "nbr_"):
36
+ """
37
+ For each pixel, average a handful of feature columns over its k nearest
38
+ neighbours (excluding itself). Uses no labels, so it is legal in both
39
+ cities and available at n=0.
40
+
41
+ `max_radius` is in pixel units (1 = 30 m); neighbours further away are
42
+ ignored and the pixel keeps its own value instead.
43
+ """
44
+ import pandas as pd
45
+
46
+ if feature_cols is None:
47
+ feature_cols = [c for c in ("sr_brightness_mean", "sr_NDVI_mean",
48
+ "sr_SWIR1_mean", "sr_brightness_late_minus_early",
49
+ "step_brightness_year", "step_brightness_mag",
50
+ "sr_brightness_std") if c in df.columns]
51
+ coords = df[list(coord_cols)].to_numpy(dtype=float)
52
+ V = df[list(feature_cols)].to_numpy(dtype=float)
53
+
54
+ nn = NearestNeighbors(n_neighbors=min(k + 1, len(df))).fit(coords)
55
+ dist, idx = nn.kneighbors(coords)
56
+ dist, idx = dist[:, 1:], idx[:, 1:] # drop self
57
+ ok = dist <= max_radius
58
+ w = ok.astype(float)
59
+ denom = np.clip(w.sum(axis=1, keepdims=True), 1e-9, None)
60
+
61
+ out = {}
62
+ for j, c in enumerate(feature_cols):
63
+ vals = V[idx, j]
64
+ mean = (vals * w).sum(axis=1, keepdims=True) / denom
65
+ mean = np.where(w.sum(axis=1, keepdims=True) > 0, mean, V[:, [j]])
66
+ out[f"{prefix}{c}"] = mean.ravel()
67
+ out[f"{prefix}{c}_diff"] = V[:, j] - mean.ravel()
68
+ out[f"{prefix}n_within_radius"] = ok.sum(axis=1).astype(float)
69
+ return pd.concat([df, pd.DataFrame(out, index=df.index)], axis=1)
70
+
71
+
72
+ # --------------------------------------------------------------------------- #
73
+ # B + C. Retrieval over a labelled pool
74
+ # --------------------------------------------------------------------------- #
75
+ class LabelledPool:
76
+ """
77
+ A set of labelled pixels in one city (in Amsterdam: the n-per-class sample
78
+ the organisers hand us; in Madrid during training: a simulated sample).
79
+
80
+ sklearn only - no FAISS - because the pool is at most a few thousand rows
81
+ and the notebook must run with the internet switched off on Kaggle.
82
+ """
83
+
84
+ def __init__(self, X, y, coords, rows=None):
85
+ self.X = np.asarray(X, dtype=np.float64)
86
+ self.y = np.asarray(y, dtype=int)
87
+ self.coords = np.asarray(coords, dtype=float)
88
+ self.rows = rows # for rendering shots
89
+ self._nn_spec = NearestNeighbors(n_neighbors=min(16, len(self.X))).fit(self.X)
90
+ self._nn_spat = NearestNeighbors(n_neighbors=min(16, len(self.X))).fit(self.coords)
91
+
92
+ def __len__(self):
93
+ return len(self.X)
94
+
95
+ def spectral(self, Xq, k: int = 8):
96
+ k = min(k, len(self.X))
97
+ d, i = self._nn_spec.kneighbors(np.atleast_2d(Xq), n_neighbors=k)
98
+ return d, i
99
+
100
+ def spatial(self, coords_q, k: int = 4):
101
+ k = min(k, len(self.coords))
102
+ d, i = self._nn_spat.kneighbors(np.atleast_2d(coords_q), n_neighbors=k)
103
+ return d, i
104
+
105
+ def stratified_spectral(self, Xq, per_class: int = 1):
106
+ """Closest `per_class` pool members of each class - a class-balanced
107
+ set of demonstrations, so the prompt cannot be swamped by class 2."""
108
+ picks = []
109
+ for c in CLASSES:
110
+ sel = np.flatnonzero(self.y == c)
111
+ if len(sel) == 0:
112
+ continue
113
+ d = np.linalg.norm(self.X[sel] - np.asarray(Xq)[None, :], axis=1)
114
+ order = np.argsort(d)[:per_class]
115
+ picks += [(float(d[o]), int(sel[o])) for o in order]
116
+ picks.sort()
117
+ return picks
118
+
119
+
120
+ LETTERS = ["A", "B", "C", "D"]
121
+
122
+
123
+ def rag_block(pool: LabelledPool, Xq, coords_q, k_spec: int = 16, k_spat: int = 4,
124
+ style: str = "compact", pixel_m: float = 30.0,
125
+ max_spatial_px: float = 30.0, exclude: int | None = None) -> str:
126
+ """
127
+ Turn retrieval into text.
128
+
129
+ exclude : index INTO THE POOL to leave out, for when the query pixel is
130
+ itself a pool member. Without it the block states the answer: the
131
+ identical row gives spectral distance 0.00 to its own class,
132
+ makes itself "the single most similar known building", and shows
133
+ up as "class X at 0 m". That is how the Amsterdam calibration
134
+ shots were being rendered, so the per-class biases were fitted on
135
+ a prompt distribution that never occurs at evaluation time. The
136
+ Madrid training side avoids it structurally, by drawing each
137
+ block's pool from the other blocks (`simulate_pools`,
138
+ `exclude_block=`), but nothing stopped it here.
139
+
140
+ The field that carries the information is the DISTANCE TO THE NEAREST
141
+ LABELLED EXAMPLE OF EACH CLASS - not a class histogram. The organisers hand
142
+ out n labelled pixels *per class*, so the pool is balanced by construction
143
+ and a histogram of the k nearest neighbours carries almost no signal while
144
+ actively inviting the model to copy the majority. Per-class distances remove
145
+ that failure mode by construction instead of patching it afterwards.
146
+
147
+ A second, independent channel reports the physically nearest labelled
148
+ pixels in metres. The two are deliberately NOT fused into one tuned
149
+ distance: there are no target labels to tune the weighting with, so the
150
+ model is left to learn it.
151
+ """
152
+ if len(pool) == 0:
153
+ return ""
154
+ Xq = np.asarray(Xq, dtype=float).ravel()
155
+
156
+ keep = np.ones(len(pool), dtype=bool)
157
+ if exclude is not None:
158
+ keep[int(exclude)] = False
159
+ if not keep.any():
160
+ return ""
161
+ n_kept = int(keep.sum())
162
+ lines = [f"Labelled buildings elsewhere in this city ({n_kept} of them, "
163
+ f"{n_kept // 4} per age class)"]
164
+
165
+ # --- channel 1: spectral distance to the nearest example of each class ---
166
+ per_class = {}
167
+ for c in CLASSES:
168
+ sel = np.flatnonzero((pool.y == c) & keep)
169
+ if len(sel) == 0:
170
+ continue
171
+ d = np.linalg.norm(pool.X[sel] - Xq[None, :], axis=1)
172
+ per_class[int(c)] = float(d.min())
173
+ if per_class:
174
+ lines.append("- how far this pixel is from the most similar known "
175
+ "building of each class (smaller = more similar): "
176
+ + ", ".join(f"{LETTERS[c - 1]} {v:.2f}"
177
+ for c, v in sorted(per_class.items())))
178
+ best = min(per_class, key=per_class.get)
179
+ lines.append(f"- the single most similar known building is class "
180
+ f"{LETTERS[best - 1]}")
181
+
182
+ # --- a soft, distance-weighted vote over the k nearest, per-class normalised
183
+ # Ask for one extra neighbour when a row is excluded, so dropping it still
184
+ # leaves k_spec of them.
185
+ d, i = pool.spectral(Xq, k_spec + (exclude is not None))
186
+ d, i = d[0], i[0]
187
+ if exclude is not None:
188
+ m_keep = i != int(exclude)
189
+ d, i = d[m_keep][:k_spec], i[m_keep][:k_spec]
190
+ tau = float(np.median(d)) + 1e-9
191
+ w = np.exp(-(d / tau) ** 2)
192
+ score = np.zeros(4)
193
+ for c in CLASSES:
194
+ m = pool.y[i] == c
195
+ n_c = max(int(((pool.y == c) & keep).sum()), 1)
196
+ score[c - 1] = w[m].sum() / n_c
197
+ if score.sum() > 0:
198
+ score = score / score.sum()
199
+ lines.append("- similarity-weighted vote of the "
200
+ f"{len(d)} most similar known buildings: "
201
+ + ", ".join(f"{LETTERS[k]} {100 * score[k]:.0f}%"
202
+ for k in range(4)))
203
+
204
+ # --- channel 2: physical distance ---
205
+ ds, isp = pool.spatial(coords_q, k_spat + (exclude is not None))
206
+ ds, isp = ds[0], isp[0]
207
+ if exclude is not None:
208
+ m_keep = isp != int(exclude)
209
+ ds, isp = ds[m_keep][:k_spat], isp[m_keep][:k_spat]
210
+ near = ds <= max_spatial_px
211
+ if near.any():
212
+ lines.append("- nearest known buildings on the ground: "
213
+ + ", ".join(f"class {LETTERS[int(pool.y[j]) - 1]} at {m * pixel_m:.0f} m"
214
+ for j, m in zip(isp[near], ds[near])))
215
+ else:
216
+ lines.append(f"- no known building within "
217
+ f"{max_spatial_px * pixel_m:.0f} m of this pixel")
218
+ return "\n".join(lines)
219
+
220
+
221
+ # --------------------------------------------------------------------------- #
222
+ # Metric space for retrieval
223
+ # --------------------------------------------------------------------------- #
224
+ def transfer_safe_columns(columns) -> list[str]:
225
+ """
226
+ Columns whose *level* does not encode which city we are in.
227
+
228
+ Keep: spectral indices (a multiplicative gain cancels in a normalised
229
+ ratio), every spread/std feature (an additive city offset cancels exactly in
230
+ a standard deviation - the point most teams miss), year-on-year deltas,
231
+ early-minus-late contrasts, slopes, chromaticity, and the change-event
232
+ fields. Drop: raw band means and raw era means, which are precisely the
233
+ "Madrid is brighter than Amsterdam" channel, and the data-availability
234
+ flags, which encode that Amsterdam is cloudier.
235
+ """
236
+ keep, drop_exact = [], {"has_early_data", "has_late_data", "n_observed_years",
237
+ "first_observed_year", "last_observed_year"}
238
+ # Anything that encodes WHICH INSTRUMENT took the picture. The satellite
239
+ # selected for a given year differs between the two cities in 11 of the 25
240
+ # years from 2001 on, and OLI reads about 0.015 darker than TM/ETM+, so
241
+ # these are city markers dressed up as physics. They stay in the feature
242
+ # table so a model can absorb the nuisance, and stay out of any distance.
243
+ sensor_marks = ("sat_share", "sat_switches", "_oli_minus_tm",
244
+ "_oli_mean", "_tm_mean", "sat_code")
245
+
246
+ # Spectral INDEX names, matched case-sensitively on purpose. "NBR" the
247
+ # index and "nbr_" the neighbourhood prefix are not the same thing, and
248
+ # matching "nbr" case-insensitively conflated them: every single one of the
249
+ # 15 `nbr_*` neighbourhood columns was admitted to the metric, including
250
+ # `nbr_sr_brightness_mean` (city AUC 0.966), `nbr_sr_SWIR1_mean` (0.957)
251
+ # and `nbr_n_within_radius`, a pure urban-density marker. That is exactly
252
+ # the "Madrid is brighter than Amsterdam" channel this function exists to
253
+ # remove, let in through the one clause meant to keep a burn ratio.
254
+ INDEX_NAMES = ("NDVI", "NDBI", "_UI_", "MNDWI", "BSI", "NBR", "_TCG",
255
+ "_TCW", "chrom")
256
+
257
+ def judge(name: str) -> bool:
258
+ """Is this column's LEVEL free of city identity?"""
259
+ lc = name.lower()
260
+ is_index = any(ix in name for ix in INDEX_NAMES) or "chrom" in lc
261
+ # Offset- and gain-cancelling shapes. `_diff` is a pixel minus its own
262
+ # neighbourhood, so any city-wide offset cancels in it.
263
+ safe = ("_std" in lc or lc.startswith("d_") or "_minus_" in lc
264
+ or "_slope" in lc or "chrom" in lc or lc.startswith("step_")
265
+ or "_cv" in lc or lc.endswith("_diff") or is_index)
266
+ # A raw level dressed up as a mean. A difference of two levels is fine.
267
+ unsafe = ("_mean" in lc
268
+ and not ("_minus_" in lc or "_cv" in lc or lc.endswith("_diff"))
269
+ and not is_index)
270
+ return safe and not unsafe
271
+
272
+ for c in columns:
273
+ if c in drop_exact or any(s in c for s in sensor_marks):
274
+ continue
275
+ # A neighbourhood aggregate inherits the transfer safety of whatever it
276
+ # aggregates: the mean of a raw band over a 7x7 window is still a raw
277
+ # band level, and the mean of NDVI is still a ratio.
278
+ inner = c[4:] if c.lower().startswith("nbr_") else c
279
+ if judge(inner):
280
+ keep.append(c)
281
+ return keep
282
+
283
+
284
+ class WhitenedMetric:
285
+ """
286
+ PCA-whitening fitted on the TARGET city's own unlabelled features.
287
+
288
+ Plain Euclidean distance over these features triple-counts absolute
289
+ brightness, because `Blue_mean`, `Blue_early_mean` and `Blue_late_mean` are
290
+ near-duplicates. Whitening on the target city removes that redundancy and
291
+ is the cheap stand-in for a learned metric.
292
+ """
293
+
294
+ def __init__(self, n_components: int = 24):
295
+ self.n_components = n_components
296
+
297
+ def fit(self, X):
298
+ from sklearn.decomposition import PCA
299
+ from sklearn.preprocessing import QuantileTransformer
300
+ X = np.asarray(X, dtype=float)
301
+ self.qt_ = QuantileTransformer(output_distribution="normal",
302
+ n_quantiles=min(1000, len(X)),
303
+ random_state=0).fit(X)
304
+ k = min(self.n_components, X.shape[1], len(X))
305
+ self.pca_ = PCA(n_components=k, whiten=True, random_state=0)
306
+ self.pca_.fit(self.qt_.transform(X))
307
+ return self
308
+
309
+ def transform(self, X):
310
+ return self.pca_.transform(self.qt_.transform(np.asarray(X, dtype=float)))
311
+
312
+ def fit_transform(self, X):
313
+ return self.fit(X).transform(X)
314
+
315
+
316
+ def window_stats(df, coord_cols=("px_key", "py_key"), feature_cols=None,
317
+ windows=(1, 3), prefix="win"):
318
+ """
319
+ Mean / std of a few features over square pixel windows around each pixel
320
+ (radius 1 = 3x3, radius 3 = 7x7), plus the pixel-minus-window contrast.
321
+
322
+ Label-free, so it works at n = 0, and it is a local contrast rather than an
323
+ absolute level, which is exactly the kind of quantity that survives the jump
324
+ to another city.
325
+ """
326
+ import pandas as pd
327
+ from scipy.spatial import cKDTree
328
+
329
+ if feature_cols is None:
330
+ feature_cols = [c for c in ("sr_brightness_mean", "sr_NDVI_mean",
331
+ "sr_SWIR1_mean", "step_brightness_year",
332
+ "sr_brightness_late_minus_early")
333
+ if c in df.columns]
334
+ coords = df[list(coord_cols)].to_numpy(float)
335
+ V = df[list(feature_cols)].to_numpy(float)
336
+ n, F = V.shape
337
+ tree = cKDTree(coords)
338
+ out = {}
339
+ for r in windows:
340
+ groups = tree.query_ball_point(coords, r=r * 1.4143, p=2)
341
+ # One pass over pixels, all features at once. Looping per feature as
342
+ # well would be F times as many Python iterations - 760k instead of
343
+ # 152k on the real 76k-pixel city.
344
+ mu = np.empty((n, F))
345
+ sd = np.empty((n, F))
346
+ cnt = np.empty(n)
347
+ for i, g in enumerate(groups):
348
+ block = V[g, :]
349
+ mu[i] = block.mean(axis=0)
350
+ sd[i] = block.std(axis=0, ddof=0)
351
+ cnt[i] = len(g)
352
+ w = 2 * r + 1
353
+ for j, c in enumerate(feature_cols):
354
+ out[f"{prefix}{w}_{c}_mean"] = mu[:, j]
355
+ out[f"{prefix}{w}_{c}_std"] = sd[:, j]
356
+ out[f"{prefix}{w}_{c}_contrast"] = V[:, j] - mu[:, j]
357
+ out[f"{prefix}{w}_n"] = cnt
358
+ return pd.concat([df, pd.DataFrame(out, index=df.index)], axis=1)
359
+
360
+
361
+ def shots_block(pool: LabelledPool, Xq, render_fn, per_class: int = 1,
362
+ max_shots: int = 4) -> str:
363
+ """
364
+ In-context demonstrations: the nearest labelled pixel of each class,
365
+ rendered with the same verbaliser and ending in its true answer.
366
+ Class-balanced on purpose; majority-label bias is a real failure mode.
367
+ """
368
+ if len(pool) == 0 or pool.rows is None:
369
+ return ""
370
+ picks = pool.stratified_spectral(Xq, per_class=per_class)[:max_shots]
371
+ parts = ["Worked examples from this city:"]
372
+ for _, j in picks:
373
+ parts.append(render_fn(pool.rows.iloc[j], int(pool.y[j])))
374
+ return "\n\n".join(parts)
375
+
376
+
377
+ # --------------------------------------------------------------------------- #
378
+ # Simulating the adaptation regime inside the training city
379
+ # --------------------------------------------------------------------------- #
380
+ def spatial_blocks(coords, n_blocks: int = 20) -> np.ndarray:
381
+ """
382
+ Tile the city into roughly equal-area blocks by coordinate quantiles.
383
+ Blocks, not random folds: neighbouring pixels share an era, so a random
384
+ split would let a pixel retrieve its own back garden.
385
+ """
386
+ coords = np.asarray(coords, dtype=float)
387
+ side = int(np.ceil(np.sqrt(n_blocks)))
388
+ bx = np.digitize(coords[:, 0], np.quantile(coords[:, 0], np.linspace(0, 1, side + 1)[1:-1]))
389
+ by = np.digitize(coords[:, 1], np.quantile(coords[:, 1], np.linspace(0, 1, side + 1)[1:-1]))
390
+ return bx * side + by
391
+
392
+
393
+ def simulate_pools(y, blocks, n_per_class: int, rng, exclude_block,
394
+ allowed_blocks=None):
395
+ """
396
+ Draw an n-per-class labelled pool from blocks other than the query's own.
397
+
398
+ exclude_block : one block id, or any iterable of them.
399
+ allowed_blocks : if given, the pool may ONLY come from these blocks. Used
400
+ to keep the spatial validation holdout out of the pool:
401
+ without it, `exclude_block=b` alone still let a training
402
+ block's retrieval text quote the labels of validation-block
403
+ pixels, which is the one thing the holdout exists to
404
+ prevent.
405
+ """
406
+ y = np.asarray(y)
407
+ blocks = np.asarray(blocks)
408
+ excl = ({int(exclude_block)} if np.isscalar(exclude_block)
409
+ else {int(b) for b in exclude_block})
410
+ ok = ~np.isin(blocks, list(excl))
411
+ if allowed_blocks is not None:
412
+ ok &= np.isin(blocks, list({int(b) for b in allowed_blocks}))
413
+ avail = np.flatnonzero(ok)
414
+ picks = []
415
+ for c in CLASSES:
416
+ pool_c = avail[y[avail] == c]
417
+ if len(pool_c) == 0:
418
+ continue
419
+ picks.append(rng.choice(pool_c, size=min(n_per_class, len(pool_c)), replace=False))
420
+ return np.concatenate(picks) if picks else np.array([], dtype=int)
tools/full_score.py ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Score notebook 3's two headline variants on ALL of Amsterdam.
3
+
4
+ Notebook 3 scored every variant on a fixed random subsample of Amsterdam to keep
5
+ the table affordable. This re-runs the two variants that win a prize category
6
+ with the same labelled-sample draws, and scores every fitted model three ways:
7
+
8
+ full all 25,992 pixels minus the drawn shots <- the number to report
9
+ sub2000 the fixed 2,000-pixel subsample <- reproduces the table
10
+ sub8000 the fixed 8,000-pixel subsample <- what nb 3 now uses
11
+
12
+ The subsample columns are a fidelity check: if they reproduce the published
13
+ table, the full column was produced by the same models on the same draws.
14
+
15
+ Everything mirrors `evaluation.evaluate_protocol` as notebook 3 calls it:
16
+ seed 42, shots drawn with rng(42 + 1000 n + r), shots removed from every
17
+ evaluation set, and the decision rule (tau 0.5 then the macro-F1 plug-in, with
18
+ the Madrid prior) applied to each evaluation set on its own, because the
19
+ plug-in rule depends on the batch it is given.
20
+
21
+ `y_amsterdam` is read for exactly two things, as in the protocol: drawing the
22
+ permitted n-per-class sample and scoring.
23
+
24
+ python -I tools/full_score.py prototype results/full_score_prototype.json
25
+ python -I tools/full_score.py upweighted results/full_score_upweighted.json
26
+ """
27
+ from __future__ import annotations
28
+
29
+ import json
30
+ import os
31
+ import sys
32
+ import time
33
+ from pathlib import Path
34
+
35
+ ROOT = Path(__file__).resolve().parent.parent
36
+ sys.path.insert(0, str(ROOT / "src"))
37
+ os.chdir(ROOT)
38
+
39
+ import numpy as np
40
+ import pandas as pd
41
+
42
+ import config as C
43
+ import evaluation as E
44
+ import gbdt as G
45
+
46
+ SEED = 42 # evaluate_protocol's default; notebook 3 keeps it
47
+ N_LIST = (5, 25, 50, 100, 200)
48
+ CHEAP_LGB = dict(n_estimators=400, learning_rate=0.06) # notebook 3, verbatim
49
+
50
+ VARIANTS = {
51
+ "prototype": ("prototype, robust z + within-class Mahalanobis",
52
+ lambda: G.mc_prototype(metric="within"), 12),
53
+ "upweighted": ("upweighted shots, retrained",
54
+ lambda: G.mc_shot_weighted(
55
+ normalise="robustz",
56
+ fit=lambda X, y, **kw: G.fit_lgbm(X, y, params=CHEAP_LGB, **kw)),
57
+ 5),
58
+ }
59
+
60
+
61
+ def load():
62
+ madrid = pd.read_parquet(C.PROC_DIR / "madrid_features.parquet")
63
+ amsterdam = pd.read_parquet(C.PROC_DIR / "amsterdam_features.parquet")
64
+ drop = {"px_key", "py_key", "age_class", "weighted_mean_year", "city"}
65
+ feats = [c for c in madrid.columns
66
+ if c not in drop and madrid[c].dtype.kind in "fiu"]
67
+ dead = [c for c in feats if madrid[c].std() == 0 or amsterdam[c].std() == 0]
68
+ feats = [c for c in feats if c not in dead] # notebook 3's "everything"
69
+ return (madrid[feats].to_numpy(float), madrid["age_class"].to_numpy(),
70
+ amsterdam[feats].to_numpy(float), amsterdam["age_class"].to_numpy(),
71
+ feats, dead)
72
+
73
+
74
+ def main():
75
+ key, out_path = sys.argv[1], Path(sys.argv[2])
76
+ name, make, n_repeats = VARIANTS[key]
77
+ X_mad, y_mad, X_ams, y_ams, feats, dead = load()
78
+ print(f"{name}: {len(feats)} features, dead {dead}, "
79
+ f"Madrid {X_mad.shape}, Amsterdam {X_ams.shape}", flush=True)
80
+ assert len(feats) == 393, len(feats)
81
+
82
+ src_prior = np.array([(y_mad == c).mean() for c in E.CLASSES])
83
+
84
+ def decide(p):
85
+ return E.adapt_pipeline(p, src_prior, tau=0.5)[0] # tau0.5_then_plugin
86
+
87
+ pools = {"full": np.arange(len(y_ams))}
88
+ for k in (2000, 8000):
89
+ pools[f"sub{k}"] = np.sort(np.random.default_rng(SEED).choice(
90
+ len(y_ams), k, replace=False))
91
+
92
+ mc = make()
93
+ draws = []
94
+ # Resume: every draw is fixed by (n, rep), so draws saved by an interrupted
95
+ # run are exactly what a fresh run would recompute.
96
+ if out_path.exists():
97
+ prev = json.loads(out_path.read_text())
98
+ if prev.get("key") == key and prev.get("seed") == SEED:
99
+ draws = prev["draws"]
100
+ print(f"resuming: {len(draws)} draws already saved", flush=True)
101
+ done = {(d["n"], d["rep"]) for d in draws}
102
+ t_all = time.time()
103
+ for n in N_LIST:
104
+ for r in range(n_repeats):
105
+ if (n, r) in done:
106
+ continue
107
+ t0 = time.time()
108
+ rng = np.random.default_rng(SEED + 1000 * n + r)
109
+ shot_idx = E.draw_shots(y_ams, n, rng)
110
+ fn = mc(X_mad, y_mad, X_ams, X_ams[shot_idx], y_ams[shot_idx])
111
+ P_all = np.asarray(fn(X_ams), dtype=float)
112
+
113
+ rec = dict(n=n, rep=r, n_shots=int(len(shot_idx)))
114
+ for pname, pool in pools.items():
115
+ keep = np.setdiff1d(pool, shot_idx)
116
+ if pname == "sub2000" and not draws:
117
+ # Predicting all rows then indexing must equal predicting
118
+ # the subset directly, as evaluate_protocol does.
119
+ direct = np.asarray(fn(X_ams[keep]), dtype=float)
120
+ assert np.allclose(direct, P_all[keep], atol=1e-10), \
121
+ "predictions are not row-wise"
122
+ pred = decide(P_all[keep])
123
+ rec[pname] = float(E.macro_f1(y_ams[keep], pred))
124
+ rec[f"{pname}_n_scored"] = int(len(keep))
125
+ rec[f"{pname}_per_class"] = [
126
+ float(x) for x in E.per_class_f1(y_ams[keep], pred)]
127
+ rec["seconds"] = round(time.time() - t0, 1)
128
+ draws.append(rec)
129
+ print(f"n={n:<4d} rep={r:<2d} full={rec['full']:.4f} "
130
+ f"sub2000={rec['sub2000']:.4f} sub8000={rec['sub8000']:.4f} "
131
+ f"({rec['seconds']}s)", flush=True)
132
+ out_path.write_text(json.dumps(
133
+ dict(variant=name, key=key, seed=SEED, n_repeats=n_repeats,
134
+ complete=False, draws=draws), indent=1))
135
+
136
+ df = pd.DataFrame(draws)
137
+ summary = {}
138
+ for pname in pools:
139
+ g = df.groupby("n")[pname]
140
+ summary[pname] = {int(n): dict(mean=float(g.mean()[n]),
141
+ std=float(g.std(ddof=1)[n]))
142
+ for n in N_LIST}
143
+ means = [summary[pname][n]["mean"] for n in N_LIST]
144
+ summary[pname]["average"] = float(np.mean(means))
145
+ summary[pname]["peak"] = float(np.max(means))
146
+ out_path.write_text(json.dumps(
147
+ dict(variant=name, key=key, seed=SEED, n_repeats=n_repeats,
148
+ complete=True, minutes=round((time.time() - t_all) / 60, 1),
149
+ summary=summary, draws=draws), indent=1))
150
+ print("\n" + pd.DataFrame({p: {n: summary[p][n]["mean"] for n in N_LIST}
151
+ for p in pools}).round(4).to_string())
152
+ for p in pools:
153
+ print(f"{p:8s} average {summary[p]['average']:.4f} "
154
+ f"peak {summary[p]['peak']:.4f}")
155
+ print(f"done in {(time.time() - t_all) / 60:.1f} min", flush=True)
156
+
157
+
158
+ if __name__ == "__main__":
159
+ main()
winners.py ADDED
@@ -0,0 +1,277 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ The two winning models from our Hack4Dev building-age entry, self-contained.
3
+
4
+ Needs only numpy, pandas, pyarrow, scikit-learn and lightgbm - none of the
5
+ project's src/ code. Every function here is a line-for-line port of the code
6
+ that produced the reported numbers (src/gbdt.py, src/evaluation.py); the
7
+ export script checked that the ports give identical results.
8
+
9
+ ZERO-SHOT LightGBM trained on Madrid only. Never sees an Amsterdam label.
10
+ Load it from models/zero_shot_lgbm.txt, or retrain it.
11
+ RETRAINED A fresh LightGBM on all of Madrid plus the n labelled Amsterdam
12
+ pixels per class that the protocol hands you, those pixels
13
+ weighted 30x. Refit for every draw of labelled pixels, so there
14
+ is no single saved model - models/retrained_*.txt are two
15
+ example draws, saved so you can inspect one.
16
+
17
+ Both use the same decision rule: tau = 0.5 logit adjustment by the Madrid class
18
+ prior, then the macro-F1 plug-in rule. Chosen on Madrid alone.
19
+
20
+ Command line (run from this folder):
21
+
22
+ python winners.py zero-shot # score the saved model
23
+ python winners.py retrained --n 25 --rep 0 # one draw, as reported
24
+ python winners.py protocol --model retrained --reps 5
25
+ python winners.py protocol --model zero-shot --reps 3
26
+
27
+ Python:
28
+
29
+ import winners as W
30
+ d = W.load_data()
31
+ model = W.load_zero_shot()
32
+ labels = W.decide(W.predict(model, d["X_ams"]), d["madrid_prior"])
33
+ """
34
+ from __future__ import annotations
35
+
36
+ import argparse
37
+ import json
38
+ from pathlib import Path
39
+
40
+ import numpy as np
41
+ import pandas as pd
42
+
43
+ HERE = Path(__file__).resolve().parent
44
+ DATA = HERE / "data" / "processed"
45
+ MODELS = HERE / "models"
46
+
47
+ CLASSES = np.array([1, 2, 3, 4]) # 1 = oldest ... 4 = newest; see README
48
+ SEED = 42
49
+
50
+ # Full-strength LightGBM (src/gbdt.py LGBM_PARAMS) - used by the zero-shot model.
51
+ LGBM_PARAMS = dict(
52
+ objective="multiclass", num_class=4, learning_rate=0.03, n_estimators=1200,
53
+ num_leaves=31, max_depth=-1, min_child_samples=80, subsample=0.8,
54
+ subsample_freq=1, colsample_bytree=0.6, reg_alpha=0.5, reg_lambda=5.0,
55
+ max_bin=127, n_jobs=-1, random_state=SEED, verbosity=-1,
56
+ )
57
+ # The retrained variant was measured with notebook 3's cheaper settings,
58
+ # because it refits on every draw. Same params otherwise.
59
+ RETRAIN_OVERRIDES = dict(n_estimators=400, learning_rate=0.06)
60
+ SHOT_WEIGHT = 30.0
61
+
62
+
63
+ # --------------------------------------------------------------------------- #
64
+ # Data
65
+ # --------------------------------------------------------------------------- #
66
+ def feature_names() -> list[str]:
67
+ return json.loads((MODELS / "meta.json").read_text())["feature_names"]
68
+
69
+
70
+ def load_data(data_dir: Path | str = DATA) -> dict:
71
+ """Madrid and Amsterdam feature matrices in the exact training column order."""
72
+ data_dir = Path(data_dir)
73
+ feats = feature_names()
74
+ mad = pd.read_parquet(data_dir / "madrid_features.parquet")
75
+ ams = pd.read_parquet(data_dir / "amsterdam_features.parquet")
76
+ y_mad = mad["age_class"].to_numpy()
77
+ return dict(
78
+ X_mad=mad[feats].to_numpy(float), y_mad=y_mad,
79
+ X_ams=ams[feats].to_numpy(float), y_ams=ams["age_class"].to_numpy(),
80
+ coords_ams=ams[["px_key", "py_key"]].to_numpy(float),
81
+ madrid_prior=np.array([(y_mad == c).mean() for c in CLASSES]),
82
+ feature_names=feats,
83
+ )
84
+
85
+
86
+ # --------------------------------------------------------------------------- #
87
+ # Decision rule: tau-logit adjustment, then the macro-F1 plug-in
88
+ # --------------------------------------------------------------------------- #
89
+ def logit_adjust(proba, train_prior, tau=0.5):
90
+ p = np.asarray(proba, dtype=float)
91
+ adj = p / np.clip(np.asarray(train_prior, float), 1e-12, None) ** float(tau)
92
+ return adj / np.clip(adj.sum(axis=1, keepdims=True), 1e-12, None)
93
+
94
+
95
+ def macro_f1_plugin(proba, prior, n_iter=50):
96
+ """Ramaswamy et al.'s plug-in rule for macro F1. Uses no target labels, but
97
+ it does look at the predicted class shares of the batch it is given, so
98
+ call it on the whole set you want labelled, not row by row."""
99
+ P = np.asarray(proba, dtype=float)
100
+ pi = np.asarray(prior, dtype=float)
101
+ pi = pi / pi.sum()
102
+ f1 = np.zeros(4)
103
+ best = None
104
+ for it in range(n_iter):
105
+ q = np.clip(np.bincount(np.argmax(P, axis=1), minlength=4) / len(P), 1e-6, None) \
106
+ if it == 0 else q_new
107
+ a = 2.0 / np.clip(pi + q, 1e-9, None)
108
+ lab_idx = np.argmax(a[None, :] * (P - f1[None, :] / 2.0), axis=1)
109
+ q_new = np.clip(np.bincount(lab_idx, minlength=4) / len(P), 1e-6, None)
110
+ tp = np.array([P[lab_idx == k, k].sum() for k in range(4)]) / len(P)
111
+ f1_new = 2 * tp / np.clip(pi + q_new, 1e-9, None)
112
+ obj = float(np.mean(f1_new))
113
+ if best is None or obj > best[0]:
114
+ best = (obj, CLASSES[lab_idx].copy())
115
+ if np.max(np.abs(f1_new - f1)) < 1e-6 and it > 0:
116
+ break
117
+ f1 = f1_new
118
+ return best[1]
119
+
120
+
121
+ def decide(proba, madrid_prior, tau=0.5):
122
+ """Class labels (1-4) from probabilities. Plain argmax would be
123
+ CLASSES[proba.argmax(1)]; this rule was worth about +0.01 on Madrid."""
124
+ src = np.asarray(madrid_prior, float)
125
+ src = src / src.sum()
126
+ return macro_f1_plugin(logit_adjust(proba, src, tau), src)
127
+
128
+
129
+ def macro_f1(y_true, y_pred) -> float:
130
+ from sklearn.metrics import f1_score
131
+ return float(f1_score(y_true, y_pred, labels=CLASSES, average="macro",
132
+ zero_division=0))
133
+
134
+
135
+ # --------------------------------------------------------------------------- #
136
+ # Models
137
+ # --------------------------------------------------------------------------- #
138
+ def balanced_class_weights(y):
139
+ y = np.asarray(y)
140
+ counts = np.array([(y == c).sum() for c in CLASSES], dtype=float)
141
+ return len(y) / (4.0 * np.clip(counts, 1, None))
142
+
143
+
144
+ def _fit_lgbm(X, y, params, class_weight=None, sample_weight=None):
145
+ import lightgbm as lgb
146
+ p = dict(params)
147
+ if class_weight is not None:
148
+ p["class_weight"] = class_weight
149
+ m = lgb.LGBMClassifier(**p)
150
+ m.fit(np.asarray(X), np.asarray(y) - 1, sample_weight=sample_weight)
151
+ return m
152
+
153
+
154
+ def train_zero_shot(X_mad, y_mad):
155
+ """Madrid only, class-balanced, raw features (no per-city scaling)."""
156
+ cw = balanced_class_weights(y_mad)
157
+ return _fit_lgbm(X_mad, y_mad, LGBM_PARAMS,
158
+ class_weight=dict(zip(range(4), cw)))
159
+
160
+
161
+ def robustz_fit(X):
162
+ X = np.asarray(X, dtype=np.float64)
163
+ med = np.median(X, axis=0)
164
+ q75, q25 = np.percentile(X, [75, 25], axis=0)
165
+ iqr = np.where((q75 - q25) > 1e-12, q75 - q25, 1.0)
166
+ return med, iqr
167
+
168
+
169
+ def robustz(X, stats):
170
+ med, iqr = stats
171
+ return (np.asarray(X, dtype=np.float64) - med) / iqr
172
+
173
+
174
+ def train_retrained(X_mad, y_mad, X_ams_all, X_shot, y_shot,
175
+ shot_weight=SHOT_WEIGHT):
176
+ """
177
+ Retrain on Madrid + the labelled Amsterdam shots, shots upweighted.
178
+
179
+ Each city is robust-z scaled with its OWN median/IQR. Amsterdam's are taken
180
+ from its whole unlabelled feature matrix (features only, no labels) - the
181
+ protocol passes that in, and the organisers' baseline does the same.
182
+ Returns (model, amsterdam_stats); predict with predict_retrained().
183
+ """
184
+ mad_stats = robustz_fit(X_mad)
185
+ ams_stats = robustz_fit(X_ams_all)
186
+ X = np.vstack([robustz(X_mad, mad_stats), robustz(X_shot, ams_stats)])
187
+ y = np.r_[np.asarray(y_mad), np.asarray(y_shot)]
188
+ w = np.r_[np.ones(len(y_mad)), np.full(len(y_shot), shot_weight)]
189
+ w = w * balanced_class_weights(y)[y - 1]
190
+ model = _fit_lgbm(X, y, {**LGBM_PARAMS, **RETRAIN_OVERRIDES}, sample_weight=w)
191
+ return model, ams_stats
192
+
193
+
194
+ def predict(model, X):
195
+ """Class probabilities, columns = classes 1..4. Accepts a fitted
196
+ LGBMClassifier or a lightgbm.Booster loaded from models/*.txt."""
197
+ if hasattr(model, "predict_proba"):
198
+ return model.predict_proba(np.asarray(X))
199
+ return model.predict(np.asarray(X))
200
+
201
+
202
+ def predict_retrained(model, ams_stats, X):
203
+ return predict(model, robustz(X, ams_stats))
204
+
205
+
206
+ def load_zero_shot():
207
+ import lightgbm as lgb
208
+ return lgb.Booster(model_file=str(MODELS / "zero_shot_lgbm.txt"))
209
+
210
+
211
+ def load_retrained_example(n: int):
212
+ """One saved retrained model (draw rep 0 at this n) and its Amsterdam stats."""
213
+ import lightgbm as lgb
214
+ booster = lgb.Booster(model_file=str(MODELS / f"retrained_n{n}_rep0_lgbm.txt"))
215
+ z = np.load(MODELS / "amsterdam_robustz_stats.npz")
216
+ return booster, (z["median"], z["iqr"])
217
+
218
+
219
+ # --------------------------------------------------------------------------- #
220
+ # The official few-shot protocol
221
+ # --------------------------------------------------------------------------- #
222
+ def draw_shots(y_tgt, n_per_class, rng):
223
+ """Stratified n-per-class sample - identical to src/evaluation.draw_shots."""
224
+ picks = []
225
+ for c in CLASSES:
226
+ pool = np.flatnonzero(np.asarray(y_tgt) == c)
227
+ if len(pool):
228
+ picks.append(rng.choice(pool, size=min(n_per_class, len(pool)),
229
+ replace=False))
230
+ return np.concatenate(picks)
231
+
232
+
233
+ def shots_for(y_ams, n, rep):
234
+ """The exact draw we reported for (n, rep)."""
235
+ return draw_shots(y_ams, n, np.random.default_rng(SEED + 1000 * n + rep))
236
+
237
+
238
+ def score_draw(d, model_name, n, rep, zero_shot_model=None):
239
+ """Macro F1 on all of Amsterdam minus this draw's labelled pixels."""
240
+ shot = shots_for(d["y_ams"], n, rep)
241
+ keep = np.setdiff1d(np.arange(len(d["y_ams"])), shot)
242
+ if model_name == "zero-shot":
243
+ m = zero_shot_model or load_zero_shot()
244
+ P = predict(m, d["X_ams"][keep])
245
+ else:
246
+ m, st = train_retrained(d["X_mad"], d["y_mad"], d["X_ams"],
247
+ d["X_ams"][shot], d["y_ams"][shot])
248
+ P = predict_retrained(m, st, d["X_ams"][keep])
249
+ return macro_f1(d["y_ams"][keep], decide(P, d["madrid_prior"]))
250
+
251
+
252
+ def main():
253
+ ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
254
+ ap.add_argument("what", choices=["zero-shot", "retrained", "protocol"])
255
+ ap.add_argument("--model", choices=["zero-shot", "retrained"], default="retrained")
256
+ ap.add_argument("--n", type=int, default=25)
257
+ ap.add_argument("--rep", type=int, default=0)
258
+ ap.add_argument("--reps", type=int, default=5)
259
+ a = ap.parse_args()
260
+ d = load_data()
261
+ if a.what == "zero-shot":
262
+ P = predict(load_zero_shot(), d["X_ams"])
263
+ print(f"zero-shot, all {len(P):,} Amsterdam pixels: macro F1 "
264
+ f"{macro_f1(d['y_ams'], decide(P, d['madrid_prior'])):.4f}")
265
+ elif a.what == "retrained":
266
+ print(f"retrained, n={a.n} rep={a.rep}: macro F1 "
267
+ f"{score_draw(d, 'retrained', a.n, a.rep):.4f}")
268
+ else:
269
+ zs = load_zero_shot() if a.model == "zero-shot" else None
270
+ for n in (5, 25, 50, 100, 200):
271
+ s = [score_draw(d, a.model, n, r, zs) for r in range(a.reps)]
272
+ print(f"{a.model} n={n:<4d} macro F1 {np.mean(s):.4f} "
273
+ f"+- {np.std(s, ddof=1) if len(s) > 1 else 0:.4f}", flush=True)
274
+
275
+
276
+ if __name__ == "__main__":
277
+ main()