README.md

metadata

tags:
  - tabular-classification
  - lightgbm
  - remote-sensing
  - landsat
  - domain-adaptation
  - few-shot
library_name: lightgbm

Building age from Landsat: Madrid → Amsterdam (Hack4Dev 2026)

This repo holds the models from our Iraq Hackathon 2026 / Hack4Dev entry "Building Age Classification from Satellite Imagery", together with the code, data and notebooks needed to reproduce them or compare against them. It has our best pipeline (FINAL-2, added on 11 November 2026) and the two LightGBM models of the first release.

The task. Predict a building-age class (1–4) for each 30 m Landsat pixel. Train on Madrid and transfer to Amsterdam. Scoring is macro F1 on Amsterdam after you are given n = 5, 25, 50, 100 or 200 labelled Amsterdam pixels per class (the few-shot protocol).

class	Amsterdam	Madrid
1	before 1945	before 1960
2	1945–1984	1960–1984
3	1984–2004	1984–2004
4	2004–2024	2004–2024

Update, 11 November 2026: the best pipeline (FINAL-2)

After the first release we scored about 130 more model variants in one night on a laptop CPU: neural networks with a frozen backbone, metric learning, SVMs and other classical models, and ideas from the domain-adaptation literature. The best result combines two models with a fixed weighting and a fixed decision rule:

1. Zero-shot anchor. A teammate's XGBoost + DANN design on the organisers' 60 features, re-implemented by us on CPU. It is trained on Madrid labels and unlabelled Amsterdam features, and never sees an Amsterdam label.
2. LightGBM with domain copies. The retrained LightGBM from the first release with one change, Daumé's "frustratingly easy" domain adaptation. The 120 most important features each get two extra copies: one filled only on Madrid rows and one filled only on Amsterdam rows.
3. Combine. Average the two models' log-probabilities. The zero-shot model gets weight 10/(n+10) and the LightGBM gets the rest. No weight is fitted.
4. Decide. Smooth each pixel's probabilities with its neighbours (radius 6 pixels, the teammate's setting, which they report choosing on Madrid). Then apply our decision rule: tau = 0.5 logit adjustment with the Madrid class prior, then the macro-F1 plug-in. The rule was chosen on Madrid.

The XGBoost + DANN anchor and the neighbour smoothing are a teammate's design. Their functions are in tools/compare_teammate.py, published with their agreement.

Macro F1 on all of Amsterdam, on draws 5–11 (seven per n). These draws were never used to choose anything.

pipeline	n=5	n=25	n=50	n=100	n=200	average
FINAL-2, with neighbour smoothing	0.672	0.705	0.716	0.732	0.744	0.714
FINAL-2 without coordinates (no smoothing)	0.652	0.694	0.702	0.720	0.734	0.700
LightGBM with domain copies alone, with smoothing	0.623	0.699	0.712	0.730	0.744	0.702
LightGBM with domain copies alone, no smoothing	0.607	0.687	0.698	0.718	0.734	0.689
Round-1 five-model ensemble + label propagation, with smoothing	0.677	0.705	0.714	0.725	0.728	0.710
Retrained LightGBM (first release), with smoothing	0.587	0.652	0.680	0.712	0.732	0.673
Retrained LightGBM (first release), no smoothing	0.559	0.629	0.660	0.697	0.721	0.653
Teammate's zero-shot XGBoost + DANN (our re-implementation), with smoothing and our decision rule	0.654	0.653	0.653	0.652	0.651	0.653

• Compare like with like. With smoothing on both sides FINAL-2 averages 0.714 against 0.673 for the retrained LightGBM. Without coordinates on both sides it averages 0.700 against 0.653.
• No pixel coordinates? Quote the second row, 0.700.
• What the anchor adds. Against the LightGBM with domain copies alone, both smoothed, FINAL-2 gains about 0.05 at n=5 and nothing at n=200 (0.744 for both).
• The margin over the round-1 ensemble is small. FINAL-2 is 0.004 ahead on average. At n=5 the ensemble is 0.004 ahead, which is inside the draw noise. We prefer FINAL-2 because it is simpler: two models and no network fitted per draw.
• The draws differ from the first-release table. The retrained LightGBM averages 0.653 here and 0.656 in the table further down, which uses draws 0–4.
• Standard deviation over the seven draws for FINAL-2: 0.007 / 0.012 / 0.015 / 0.004 / 0.003 at n = 5 / 25 / 50 / 100 / 200.

Run it for one draw. It takes one to two minutes. Set NNLAB_THREADS to your core count, because the default is 3.

python tools/nnlab/model_create2.py --n 25 --rep 6
python tools/nnlab/model_create2.py --n 200 --rep 3 --no-spatial

Expected output, from a run inside a clean copy of this repo:

FINAL-2 n=25 rep=6 spatial=True: macro F1 0.696921
FINAL-2 n=200 rep=3 spatial=False: macro F1 0.735816

From Python: model_create(shot_idx, shot_labels).predict() in tools/nnlab/model_create2.py. The full write-up is reports/nnlab/OVERNIGHT_RESULTS.md.

Caveats for FINAL-2

• The choice of models was made on Amsterdam. About 130 variants were scored on all of Amsterdam on draws 0–4. The LightGBM with domain copies was picked from those full-city scores, and the two-model combination was picked on half of the city. FINAL-2 is the second of two rounds of picks. A round-1 rule (switch pipelines at n=100) failed its confirmation and was dropped. Only draws 5–11 were never used for choosing. FINAL-2 reads 0.005 lower on them than on draws 0–4, and the review estimated 0.005 to 0.01 of optimism for the top picks.
• It uses more of the unlabelled target city. The DANN inside the anchor is trained against unlabelled Amsterdam features, each city is scaled with its own statistics, and the smoothing needs pixel coordinates.
• Batch-level. The decision rule and the smoothing label the whole city at once.
• The anchor weight 10/(n+10) was fixed in advance and was not validated on Madrid.
• The anchor's settings are the teammate's. Their XGBoost and DANN hyper-parameters and the smoothing radius are copied, not re-tuned here, and this repo cannot show how they were chosen.
• The anchor ships as cached Amsterdam probabilities (data/processed/zeroshot_xgb_dann_ams.npz). python tools/full_score_combo.py zeroshot rebuilds it in six to thirteen minutes and needs xgboost and torch. On our CPU the rebuild is identical to the shipped file.
• Fresh draws are not a fresh city. They are new labelled pixels in the same Amsterdam.
• The neural networks use one seed.
• The checks were automated. verification*.json and script_reviews.json come from separate AI review agents, not from human or external review.

What else was tried overnight

Selected variants of each kind. Draws 0–4, all of Amsterdam, our decision rule, no smoothing.

model	script	n=5	n=25	n=50	n=100	n=200	average
LightGBM with domain copies	lgbm_var.py	0.634	0.678	0.712	0.719	0.736	0.696
DANN network, last layer fine-tuned, then the whole network	lpft.py	0.601	0.642	0.674	0.690	0.703	0.662
Retrained LightGBM (first release)	baselines.py	0.566	0.628	0.670	0.697	0.720	0.656
Metric learning: cosine encoder, Mahalanobis head	metric.py	0.583	0.638	0.678	0.688	0.691	0.656
Metric learning: prototypical net + CORAL, Mahalanobis head	metric.py	0.605	0.640	0.671	0.679	0.682	0.655
Shrinkage LDA on 64 PCA components	classical.py	0.600	0.625	0.648	0.673	0.688	0.647
Metric learning: prototypical net + CORAL, projection layer fine-tuned on the shots	metric.py	0.597	0.627	0.656	0.673	0.677	0.646
Logistic regression	classical.py	0.564	0.608	0.658	0.679	0.694	0.640
DANN network, backbone frozen, last layer fine-tuned	tab_nn.py	0.605	0.628	0.643	0.650	0.660	0.637
RBF SVM, domain-augmented features	classical.py	0.556	0.611	0.651	0.661	0.674	0.631
ResNet, backbone frozen, last layer fine-tuned	tab_nn.py	0.552	0.603	0.642	0.662	0.673	0.626
1D CNN + DANN on the raw yearly series, last layer fine-tuned	ts_nn.py	0.516	0.601	0.620	0.649	0.658	0.609
DANN network, same backbone, last layer refit on Madrid + shots	tab_nn.py	0.567	0.572	0.578	0.587	0.603	0.581

• Frozen backbone, last layer fine-tuned. Starting the last layer from its Madrid weights and fine-tuning it on the shots beats refitting it on Madrid plus shots: 0.637 against 0.581 for the same DANN backbone.
• Unfreezing afterwards helps. Fine-tuning the whole network for 60 small steps after the last layer lifts the DANN to 0.662. That is level with the retrained LightGBM. A variant with an entropy term scores 0.663.
• Metric learning matches the retrained LightGBM on average. It is ahead at n=5 and n=25 and about 0.03 behind at n=200. The Mahalanobis head does better than fine-tuning the projection layer on three of the four encoders.
• None of them beats the LightGBM with domain copies, and adding them to the combination lowered it.
• Did not help: prior re-estimation by EM, neural embeddings as LightGBM inputs, networks on the raw time series.
• Not recommended: self-training (+0.006 on one ensemble, but below its teacher on its own) and label propagation (+0.006 on Amsterdam, but worse on a Madrid stand-in).

All 134 scored methods are in reports/nnlab/leaderboard.csv.

The two winners of the first release

	Zero-shot LightGBM	Retrained LightGBM
What it is	LightGBM trained on Madrid only. It never sees an Amsterdam label.	A fresh LightGBM trained on all of Madrid plus the n labelled Amsterdam pixels per class, with those pixels weighted 30×.
When to use it	You may not use any Amsterdam labels	You are given a few labelled target pixels, which is the official protocol
Amsterdam macro F1	0.590–0.593 at every n (it ignores the labels; it drifts only because each draw's labelled pixels leave the scored set)	0.656 average over n, 0.720 at n=200
Saved model	models/zero_shot_lgbm.txt	Refit for every draw, so there is no single model. models/retrained_n{25,200}_rep0_lgbm.txt are two example draws.
Training cost	A few minutes on a 12-thread CPU	~1–4 min per draw (84 s mean on a 12-thread CPU)

Both models use the same 393 features and the same decision rule. All numbers are on all of Amsterdam (25,992 pixels), minus the labelled pixels of each draw.

Results on all of Amsterdam

model	n=5	n=25	n=50	n=100	n=200	average	peak
Retrained LightGBM (5 draws per n)	0.5658 ± 0.0108	0.6284 ± 0.0120	0.6697 ± 0.0087	0.6973 ± 0.0038	0.7204 ± 0.0060	0.6563	0.7204
Prototype, within-class Mahalanobis (12 draws)	0.5040 ± 0.0402	0.6361 ± 0.0170	0.6733 ± 0.0135	0.7023 ± 0.0067	0.7141 ± 0.0031	0.6460	0.7141
Zero-shot LightGBM (20 draws)	0.5928	0.5925	0.5922	0.5918	0.5903	0.5919	0.5928
Organisers' prototype method, re-run by us on our 393 features (20 draws)	0.5066	0.5821	0.5965	0.6034	0.5972	0.5772	0.6034
Organisers' prototype, as reported in their 4-Modelling.ipynb (their 60 features, their draws)	0.5437	0.5986	0.6079	0.6129	0.6150	0.5956	0.6150
Organisers' reported zero-shot Random Forest (60 features)	0.3427

Mean ± std over draws. Each draw is a fresh random sample of labelled pixels, so the ± is the spread you get from which pixels you happen to be handed.

The two italic rows are the organisers' own figures, copied from their notebook. Their draws and evaluation set are not ours, so compare them loosely. Their prototype scores higher on their 60 features than on our 393. Both of our winners still beat it at every n.

Best by prize category, within the first release. FINAL-2 above is higher than each of these: 0.714 average, 0.744 peak, 0.705 at n=25, 0.672 at n=5. Those figures use smoothing, which needs pixel coordinates, and draws 5–11. Without coordinates they are 0.700 / 0.734 / 0.694 / 0.652. The draws and the amount of unlabelled Amsterdam information differ from the rows below, so this is not a like-for-like ranking.

• Average: retrained LightGBM, 0.6563.
• Peak: retrained LightGBM at n=200, 0.7204.
• n=25: the prototype, 0.6361. It is implemented in src/gbdt.py (mc_prototype(metric="within")) but not packaged as a model here.
• n=5: zero-shot LightGBM, 0.5928.

The fine-tuned LLM we also tried, for comparison

We also fine-tuned Qwen3-4B-Base with LoRA on text prompts that describe each Madrid pixel. It took about 2 h on an A100: 1,513 steps at ~0.2 it/s.

• Held-out Madrid areas: it went from 0.305 untrained to 0.537 macro F1. These come from the fine-tuning notebook's validation output, which is not in this repo.
• Amsterdam: 0.554 with no labels, on a 6,000-pixel random subset. 0.622 at best (n=200), on a fixed 2,000-pixel subset with 6 draws. These are not full-city numbers like the table above.

It does not beat the zero-shot LightGBM below n=200, so it is not one of the winners. Details are in results/llm_protocol_results_colab.md.

Quick start

pip install numpy pandas pyarrow scikit-learn scipy lightgbm matplotlib
python winners.py zero-shot                  # ~15-40 s
python winners.py retrained --n 25 --rep 0   # ~2-4 min
python tools/nnlab/model_create2.py --n 25 --rep 6   # FINAL-2, 1-2 min

The neural and metric-learning scripts in tools/nnlab/ also need torch. Rebuilding the zero-shot anchor needs torch and xgboost.

Expected output, from the verification run on our machine:

zero-shot, all 25,992 Amsterdam pixels: macro F1 0.5928
retrained, n=25 rep=0: macro F1 0.6303
FINAL-2 n=25 rep=6 spatial=True: macro F1 0.696921

To re-run the whole protocol: python winners.py protocol --model retrained --reps 5 (25 refits: ~35 min on our machine, up to ~1.5 h on a slower one). Or open notebooks/winners_demo.ipynb, which is already executed, for a walk-through of both models with outputs and the chart.

LightGBM is multi-threaded, so on a different machine the numbers can differ in the 4th decimal.

What we did, exactly

1. Features (notebooks/01_build_features.ipynb, src/features.py)

We started from the organisers' raw per-pixel Landsat time series and built one row per pixel with 393 usable features:

group	count	what
official	59	The organisers' 60 features, reproduced exactly. One has zero variance in a city and was dropped.
reflectance	276	Surface reflectance per band; correct spectral indices; tasseled cap; chromaticity; per-decade and early/late-period means
events	15	Construction-event detection: when and how sharply the pixel's spectrum stepped
neighbourhood	15	The same signals averaged over surrounding pixels
other	28	Disturbance spikes and post-construction trends (12), Landsat sensor shares (9), observation coverage (4), position in the city: radial distance and distance to water (3)

Two features with zero variance in one city were dropped, leaving 393 of 395. The pixel coordinates and the label are never features. The resulting tables are in data/processed/ (Madrid 76,263 pixels, Amsterdam 25,992). Notebook 1 needs the raw hackathon data, which is not in this repo; the processed tables are.

2. Model selection, on Madrid only (notebooks/03_gbdt_models.ipynb)

The rule we followed was that Amsterdam labels are used only to draw the permitted n-per-class sample and to score. Every design choice was made on Madrid.

Spatial cross-validation. Madrid was split into 25 geographic blocks, with 5-fold cross-validation over whole blocks. Random-pixel folds leak, because neighbouring pixels share a construction era. That is why the published 0.618 Madrid baseline looks better than it transfers.

model	59 official	393 all features
LightGBM, class-balanced	0.556	0.570
LightGBM, unweighted	0.550	0.569
LightGBM, ordinal	0.546	0.562
CatBoost, class-balanced	0.545	0.562 *

This comparison used cheap settings (400 rounds, learning rate 0.06) and robust-z scaling per block. We were reading the ranking here, not the final scores. The normalisation test below then dropped the scaling. * From an earlier run of the same notebook (results/logs/03_gbdt_models_fullscore_run.log).

Per-city normalisation, decided on Madrid. Each held-out block was scaled with its own statistics, the same way a new city would be.

• No normalisation: 0.599.
• Robust z (median/IQR): 0.569.

So the zero-shot model uses raw features. This went against our starting intuition, and Amsterdam agreed: robust z scored 0.527 there against 0.593 with no normalisation.

Decision rule. Plain argmax maximises accuracy, not macro F1. A simulated shift harness makes held-out Madrid blocks look like a new city, by resampling the class mix and mapping the features onto Amsterdam's unlabelled distribution. In it, the most robust rule was:

1. Divide the probabilities by the Madrid class prior raised to 0.5 (tau-logit adjustment).
2. Apply the macro-F1 plug-in rule (Ramaswamy et al.).

It never lost to argmax in any simulated regime: +0.012 on average. EM-based prior estimation was the worst option tested (−0.099) and is not used.

3. The two winners

Zero-shot LightGBM (winners.train_zero_shot)

• 1,200 rounds, learning rate 0.03, 31 leaves, min_child_samples 80, subsample 0.8, colsample_bytree 0.6, L1 0.5, L2 5.0, seed 42.
• Class-balanced weights, raw features.
• Prediction is the decision rule above, applied to all of Amsterdam's probabilities at once. The rule uses the batch's predicted class shares, never its labels.

Retrained LightGBM (winners.train_retrained). This follows the organisers' modelCreate spec. The deck Building-Age-from-Space-A-Transferable-Classifier.pptx is in their hackathon.zip download, and its slide 4 says:

"We are NOT building a classifier. We are building a METHOD that creates classifiers."

Slide 5 says that modelCreate takes N labelled pixels per class from the new city and outputs "a fully functional classifier for that city". There is one liberty beyond the tightest reading of slides 5 and 7, which give modelCreate only the labelled sample: step 1 below also reads Amsterdam's unlabelled feature table to compute scaling statistics (see Caveats). For each draw of n labelled Amsterdam pixels per class:

1. Robust-z scale Madrid with Madrid's median/IQR. Scale Amsterdam with Amsterdam's median/IQR, computed from its whole unlabelled feature table.
2. Stack the 76,263 Madrid rows and the 4n Amsterdam rows. The Amsterdam rows get weight 30, then everything is multiplied by class-balanced weights.
3. Fit LightGBM with the same parameters as the zero-shot model, but 400 rounds at learning rate 0.06. This is the cheaper setting the protocol was run with, because it refits on every draw.
4. Predict every other Amsterdam pixel and apply the same decision rule.

4. The protocol, i.e. how every number was measured

• Labelled sample: for each n and each repeat r, n pixels per class are drawn with numpy.random.default_rng(42 + 1000·n + r). Use winners.shots_for(y_ams, n, r) to get the identical pixels.
• Scoring: those pixels are removed and macro F1 is computed on every remaining Amsterdam pixel.
• Repeats: 5 for the retrained model, since each draw is a full refit. 12 for the prototype, 20 for the zero-shot and the organisers' baseline.
• Exploration subsample: notebook 3 explored 14 variants on a fixed 8,000-pixel subsample.
• Full-city re-scoring: tools/full_score.py re-scored the retrained LightGBM and the within-class prototype on all of Amsterdam. In the same run it re-scored the 8,000-pixel subsample, and that reproduced notebook 3's table to the 4th decimal, so the full-city numbers come from identical models. The full city reads 0.002–0.006 higher per n, and 0.003–0.004 on the average.
• The other full-city rows: the zero-shot and organisers' baseline rows come from an earlier full-city run of notebook 3 (results/logs/03_gbdt_models_fullscore_run.log, 20 draws).

5. The other 12 Amsterdam variants (8,000-pixel subsample, average over n)

variant	average
refit leaf values, tree structure frozen †	0.612
prototype, total-covariance Mahalanobis	0.612
prototype, rank + Euclidean	0.591
pure k-NN on the labelled pixels only	0.589
zero-shot, no normalisation (the zero-shot winner, before full-city scoring)	0.586
organisers' Euclidean prototype (re-run on our 393 features)	0.575
zero-shot, per-city rank / robust z / robust z + CORAL	0.553 / 0.524 / 0.537
recalibrated LightGBM / CatBoost	0.544 / 0.550
prototype, robust z + Euclidean	0.401

† Measured with a normalisation bug: it applied robust z when its label says none.

The models that actually learn from the labelled Amsterdam pixels win. The ones that only rescale or recalibrate Madrid's output score below the organisers' baseline (0.524–0.553 against 0.575). The exception is the plain zero-shot model with no scaling at all.

Caveats

• Transductive scaling. The retrained model scales Amsterdam with the median/IQR of Amsterdam's whole unlabelled feature table. That uses features only, never labels, and the protocol passes the table in. The organisers' baseline does not do this: it scales Amsterdam with Madrid's scaler. On a strict reading, where modelCreate gets only the labelled sample, you would compute those statistics from the shots alone. On a new test set, compute them on that set's features. The zero-shot model is unaffected, because it uses no scaling.
• Batch-level decision rule. The rule looks at the predicted class shares of the batch it labels. With plain argmax the zero-shot model scores 0.5680 on all of Amsterdam instead of 0.5928.
• Few repeats. The retrained model's ± comes from 5 draws per n.
• Hidden test sets. The organisers score on held-out Amsterdam and Madrid sets we never saw. Expect those numbers to be lower.

Repo layout

winners.py                     self-contained: both models, the decision rule, the protocol
models/
  zero_shot_lgbm.txt           the zero-shot model (LightGBM text format)
  retrained_n25_rep0_lgbm.txt  example retrained models, draw rep 0
  retrained_n200_rep0_lgbm.txt
  amsterdam_robustz_stats.npz  Amsterdam median/IQR the retrained models expect
  retrained_example_shot_indices.json   which Amsterdam rows they were trained on
  meta.json                    feature order, Madrid prior, verification results
data/processed/                madrid_features.parquet, amsterdam_features.parquet
  *_official60_first.parquet   the organisers' 60 features on the same pixels
  zeroshot_xgb_dann_ams.npz    cached Amsterdam probabilities of the zero-shot anchor
notebooks/
  winners_demo.ipynb           executed walk-through of both winners (start here)
  03_gbdt_models.ipynb         full model study: CV, normalisation, decision rule, all 14 variants
  01_build_features.ipynb      raw data -> features (needs the hackathon raw data)
  src/                         the same notebooks as plain .py
src/                           project code the notebooks import (gbdt.py, evaluation.py, ...)
tools/full_score.py            full-city re-scoring of the two winners
tools/nnlab/                   the overnight search (11 November)
  model_create2.py             FINAL-2 from a set of shots (start here)
  lab.py                       shared harness: draws, prediction store, scoring
  lgbm_var.py                  LightGBM variants, including the domain copies
  tab_nn.py, lpft.py           tabular networks: frozen backbone, last-layer and whole-network fine-tuning
  metric.py                    metric-learning encoders and heads
  classical.py                 SVM, logistic regression, kNN, LDA
  ts_nn.py                     networks on the raw yearly series (needs the hackathon raw data)
  transductive.py, ensemble.py, baselines.py, confirm.py, plot_confirm.py, model_create.py, lpft_steps.py
tools/compare_teammate.py      the teammate's XGBoost, DANN and smoothing functions
tools/full_score_combo.py      `zeroshot` mode rebuilds the anchor cache
tools/build_official60_first.py   builds the 60-feature tables (needs the hackathon raw data)
results/                       per-draw JSON, logs of every run, the LLM results (first release)
reports/nnlab/                 results of the overnight search
  OVERNIGHT_RESULTS.md         the full write-up
  confirmation.csv             the fresh-draw tables
  leaderboard.csv              every scored method (leaderboard_A.csv: half A of the city only)
  f1_vs_n_overnight.png        the chart above
  research_shortlist.json      the literature sweep behind the ideas tried
  scores/                      per-draw scores of every method
  PREREGISTRATION.md           the picks, written before their confirmation
  verification*.json, script_reviews.json, verify/   checks by separate AI review agents
  cache/                       the knobs chosen on Madrid, with their evidence
  logs/                        logs of every run

The per-draw prediction files and the trained network weights of the overnight search are not in this repo (about 3 GB). The scripts regenerate them into reports/nnlab/preds/ and reports/nnlab/cache/. The scripts in reports/nnlab/verify/ are the record of the checks as they were run; they read those prediction files.

models/meta.json → verification records the checks run when these files were exported:

• the ported code in winners.py gives the same probabilities as the project code (max abs diff 0);
• the saved models reload identically;
• the two example draws reproduce the reported per-draw scores exactly.