| [ |
| { |
| "verdict": "holds-with-caveats", |
| "summary": "I traced every route by which an Amsterdam label could reach a scored prediction and found no leak. Amsterdam labels are read in exactly two places: lab.shot_labels(shots_for(n, rep)) for the same draw that is being written, and the scorer. The transductive clamps (__lp, __lapshot, __clamp) seed only the current draw's shots, and the scorer drops those rows with setdiff. No shot-dependent method has an n0_r0 file it could fall back to. The graph and the zero-shot XGBoost+DANN cache use no labels. My own macro F1, using my own shot draws, matches the stored argmax score to 5 decimals. The headline numbers reproduce exactly. The +0.12 jump at n=5 has two legitimate sources: (a) the zero-shot anchor zs_xgb_dann already scores rule_smooth 0.654 at every n, and (b) the headline compares the new method's rule_smooth column against the old method's rule column. Read on the same column, the gain is +0.099 at n=5 and +0.038 on average, not +0.12 / +0.055. The caveats are about how the comparison is reported and one hyper-parameter origin I could not check; neither is a leak.", |
| "findings": [ |
| { |
| "severity": "note", |
| "claim": "Amsterdam labels enter only through lab.shot_labels(lab.shots_for(d,n,rep)) for the draw being written. The scorer is the only other reader.", |
| "evidence": "grep across tools/nnlab: _ya appears only at lab.py:88 (load), :105 (shots_for), :110 (shot_labels), :288 (score_method), plus an equality assert at baselines.py:17. Every family calls shots_for and shot_labels with the loop's own (n, rep) right before save_pred(name, n, rep): baselines.py:67-69, metric.py:460-475, tab_nn.py:466-479, transductive.py:339-350 and :375-381, classical.py:245-246. ensemble.py reads no labels at all (lines 27-34)." |
| }, |
| { |
| "severity": "note", |
| "claim": "__lp, __lapshot and __clamp clamp only the current draw's shot rows, and the scorer excludes exactly those rows.", |
| "evidence": "transductive.py:339-348: shot = shots_for(d, n, rep), then r_lp, r_lapshot or r_clamp is applied to the base file of the same draw (base_pred(B, n, rep), :308-315), then saved as (n, rep) at :350. r_lp :147-148 and r_lapshot :160/:163 write one-hot values only at index `shot`. Scorer lab.py:295-297: keep = setdiff1d(arange, shots_for(d, n, rep)). In the scorer's smoothing (lab.py:274-275) shots are also not neighbours: A[keep][:, keep]." |
| }, |
| { |
| "severity": "note", |
| "claim": "No draw can silently read another draw's or another method's file. The load_pred fallback has no guard, but it never fires for any shot-dependent method in the store.", |
| "evidence": "load_pred (lab.py:242-246) falls back only to the SAME method's n0_r0.npy. I listed all preds/* dirs: n0_r0.npy exists only in the shot-free *_zs, zs_* and zs_*__em dirs. Every shot-dependent dir (ens_*, lgbm_upw, met_*, tab_*_ft/joint/proto, ts_*_ft/joint, cls_*) has only per-draw files. ensemble.py:30 requires a per-draw file for every --shot member. An md5 check over the 25 draws gives 25 distinct files for each of ens_e4__lp, ens_e2__lapshot, lgbm_upw, met_protoda_maha, met_cos_maha and tab_dann_rz_ft. Latent hazard only: if a shot-dependent method were ever given an n0_r0 file, or passed via --zs, the fallback would hide a missing draw. A guard in load_pred would close this." |
| }, |
| { |
| "severity": "note", |
| "claim": "The transductive graph uses no labels.", |
| "evidence": "transductive.py:108-123: PCA-32 of lab.robustz(Xa), a cosine kNN graph (k=10) union lab.adjacency(pa, r=3); only features and coordinates go in. Graph() at :126-134 is also label-free. The em variant estimates its prior on the non-shot rows from predictions only (:176-181)." |
| }, |
| { |
| "severity": "note", |
| "claim": "The zero-shot cache data/processed/zeroshot_xgb_dann_ams.npz was trained without Amsterdam labels.", |
| "evidence": "tools/full_score_combo.py:55-70 build_cache: XGBoost is fit on (Xm60, ym) with balanced weights and predicts Xa60; DANN is CT.train_dann(region_norm(Xm60), ym, region_norm(Xa60)), and compare_teammate.py:109-137 uses Xt only as unlabelled domain-classifier input (bce against ones). d['ya'] is loaded by CT.load (compare_teammate.py:196) but build_cache never uses it. In CT.load's 'first' path, age_class_o is used only in an alignment assert (:187). The 393 + 60 feature names contain no label-like column (my name scan flagged only 'coverage')." |
| }, |
| { |
| "severity": "minor", |
| "claim": "I could not verify where the teammate's zero-shot hyper-parameters came from. If they were tuned on Amsterdam, that would be an indirect (selection) leak into the strongest ensemble member.", |
| "evidence": "compare_teammate.py:57-60: XGB_PARAMS are commented \"their best_config.json\", and DANN_EPOCHS=40 is fixed. No best_config.json exists in the repo outside held_out (I did not open those). This is a provenance question, not a leak I found.", |
| "suggested_fix": "Ask the teammate whether best_config.json was tuned on Madrid CV or on Amsterdam. If on Amsterdam, re-tune on lab.madrid_split or say so in PREREGISTRATION.md." |
| }, |
| { |
| "severity": "minor", |
| "claim": "The headline gain mixes decision columns: the new method's rule_smooth against the old method's rule. Read on the same column, the gain is smaller.", |
| "evidence": "scores/*.json, n=5/25/50/100/200, then the average. lgbm_upw rule = 0.566/0.628/0.670/0.697/0.720 (0.656), but lgbm_upw rule_smooth = 0.595/0.654/0.689/0.712/0.731 (0.676). ens_e1__lp rule_smooth = 0.689/0.699/0.720/0.723/0.726 (0.711) and ens_e4__lp rule_smooth = 0.694/0.703/0.720/0.725/0.727 (0.714). On the same column the gain at n=5 is 0.689-0.595 = +0.094 (e1) / +0.099 (e4), not +0.12. On average it is +0.035 / +0.038, not +0.055. At n=200 it is about -0.005 (0.726/0.727 vs 0.731).", |
| "suggested_fix": "Report both methods on the same column (rule_smooth vs rule_smooth)." |
| }, |
| { |
| "severity": "note", |
| "claim": "The jump at n=5 comes mostly from the zero-shot anchor, which has weight 10/15 = 0.67 at n=5. That is legitimate, not leakage.", |
| "evidence": "zs_xgb_dann rule_smooth = 0.654/0.653/0.653/0.652/0.651 (it uses no shots; the values differ slightly only because the keep set changes). Steps at n=5, rule_smooth: ens_e2 = 0.669 (zs + lgbm_upw), ens_e4 = 0.682 (adds the metric and tab members), ens_e4__lp = 0.694 (LP adds +0.012)." |
| }, |
| { |
| "severity": "note", |
| "claim": "Empirical test: shot rows are fit perfectly as expected, and my independent macro F1 on non-shot rows equals the scorer's argmax value exactly, so the keep set excludes the shots.", |
| "evidence": "Script reports/nnlab/verify/leak_check.py re-implements the shot draw (rng 42+1000n+rep) and reads labels from amsterdam_features.parquet. ens_e4__lp n5 r0: acc_shot=1.000 (20 rows), acc_nonshot=0.6706, F1 mine=0.66695, json argmax=0.66695, F1 with shots included=0.66723. n5 r1: 1.000 / 0.6976 / 0.68851 = json 0.68851. n200 r0: 1.000 / 0.7373 / 0.73287 = json 0.73287 (with shots included it would be 0.74254, so excluding them matters at n=200). ens_e2__lapshot n5 r0: 1.000 / 0.6479 / 0.64479 = json. ens_e4, before LP: acc_shot 0.95 / 0.90 / 0.892, F1 matches json in every case." |
| }, |
| { |
| "severity": "note", |
| "claim": "half_A and the scorer's decision rule use no labels.", |
| "evidence": "lab.half_A (lab.py:280-284) is a coordinate checkerboard. W.decide (winners.py:121-126) applies tau-logit adjustment with the Madrid prior, then the plug-in rule (:95-118), which uses only the predicted class shares of the batch it is given, here the keep rows. Labels are used only in W.macro_f1 for scoring. Metric encoder length selection (metric.py:274-291) and tab_nn selection (tab_nn.py:317+) use lab.madrid_split on Madrid only." |
| } |
| ], |
| "numbers": "My own numpy/sklearn scorer (reports/nnlab/verify/leak_check.py, 2 threads, about 1 min) re-draws the shots with rng(42+1000n+rep) and reads labels from amsterdam_features.parquet. On non-shot rows its argmax macro F1 equals scores/<method>.json \"argmax\" to 5 decimals for ens_e4__lp, ens_e2__lapshot, ens_e1__lp, ens_e4 and lgbm_upw at draws (5,0), (5,1) and (200,0); for example ens_e4__lp n5 r0 gives 0.66695 in both. Shot-row accuracy is 1.000 for clamped methods and lgbm_upw, and 0.95 / 0.90 / 0.892 for ens_e4 before LP. All 25 draw files are distinct by md5 for the six methods checked. Leaderboard means from scores/*.json (n=5/25/50/100/200, then the average): ens_e1__lp rule_smooth 0.689/0.699/0.720/0.723/0.726 (0.711) and lgbm_upw rule 0.566/0.628/0.670/0.697/0.720 (0.656), both reproducing the headline. Also: ens_e4__lp rule_smooth 0.694/0.703/0.720/0.725/0.727 (0.714); lgbm_upw rule_smooth 0.595/0.654/0.689/0.712/0.731 (0.676); zs_xgb_dann rule_smooth 0.654 flat; ens_e2 rule_smooth n5 0.669; ens_e4 rule_smooth n5 0.682." |
| }, |
| { |
| "verdict": "holds-with-caveats", |
| "summary": "I rebuilt everything independently and it matches. The ensemble, the label propagation, the graph, the shot draws and the scorer all reproduce the stored files to float32 precision, and the macro F1 values match to 4 decimals. The reference numbers also reproduce: lgbm_upw per-draw scores differ from the originals by exactly 0, and zs_lgbm is 0.5928 at n=5. The headline numbers are correct: ens_e1__lp averages 0.7113 (0.6888/0.6986/0.7200/0.7232/0.7256) and ens_e4__lp averages 0.7137. I found no leakage or computation bug. The only caveat is how the +0.128 at n=5 is framed. About +0.088 of it does not come from tonight's work. It comes from the teammate's zero-shot XGB+DANN model (+0.053, Madrid labels plus unlabelled Amsterdam only) and from neighbour smoothing (+0.035). The headline also compares a smoothed score with an unsmoothed one. Compared fairly against the best pipeline that existed before tonight (zs_xgb_dann, rule_smooth 0.6536), the net gain at n=5 is +0.040. That gain is consistent across all 5 draws and also shows up on half B, which was not used for choosing.", |
| "findings": [ |
| { |
| "severity": "note", |
| "claim": "(1) My own re-implementation of the ens_e4 rule matches the stored ens_e4 file for all three draws. The rule is w0=10/(n+10), mean of clipped log-probs of zero-shot members vs shot-using members, then softmax.", |
| "evidence": "tools/nnlab/ensemble.py:26-33 rule. max|own - stored| = 2.98e-08 for n5_r0, n25_r1 and n200_r2 (float32 storage). Script: reports/nnlab/verify/recompute.py" |
| }, |
| { |
| "severity": "note", |
| "claim": "(2) I rebuilt the union graph from scratch and wrote my own label propagation. Both match the stored ens_e4__lp file. The graph is robust-z with clip 10, PCA-32 (random_state=0), cosine 10-NN made symmetric, union with the r=3 spatial graph, no self-loops, 0/1 weights. The propagation is S=D^-1/2 A D^-1/2, Y0 = probabilities with shot rows set to 10 x one-hot, 20 iterations of F = 0.8 S F + 0.2 Y0, then row-normalised.", |
| "evidence": "Own graph: nnz=583310, identical to cache/transd_graph_ams_r3_k10_pca32.npz (0 differing entries), mean degree 22.44. LP max|own - stored| = 3.45e-08 (n5_r0), 3.38e-08 (n25_r1), 3.32e-08 (n200_r2). argmax agreement 1.000000. Code: transductive.py:96-123, 146-152" |
| }, |
| { |
| "severity": "note", |
| "claim": "(3) My own macro F1 matches the stored score file on every draw and every column. Shots are re-derived with default_rng(42+1000n+rep) and removed before scoring. They equal lab.shots_for (20/100/800 unique rows).", |
| "evidence": "ens_e4__lp. n5_r0: argmax 0.6669, rule 0.6815, rule_smooth 0.6919. n25_r1: 0.6814 / 0.6906 / 0.6961. n200_r2: 0.7208 / 0.7156 / 0.7163. ens_e4. n5_r0: 0.6432 / 0.6504 / 0.6748. n25_r1: 0.6576 / 0.6629 / 0.6794. n200_r2: 0.7120 / 0.7083 / 0.7171. Own and stored are equal to 4 decimals in every cell." |
| }, |
| { |
| "severity": "note", |
| "claim": "(4) The reference numbers reproduce.", |
| "evidence": "lgbm_upw stored 'rule' per draw vs reports/full_score_upweighted.json 'full': max abs diff 0.0 over all 25 draws. Means 0.5658 / 0.6284 / 0.6697 / 0.6973 / 0.7204, avg 0.6563. zs_lgbm rule at n=5 with my scorer: 0.5928 on every rep 0-4." |
| }, |
| { |
| "severity": "minor", |
| "claim": "(5) The '+0.12 at n=5' is real, but most of it comes from work done before tonight. The headline also compares rule_smooth with an unsmoothed 'rule'. Against lgbm_upw with the same smoothing (rule_smooth 0.5950) the gain is +0.099. Against the teammate's zero-shot model with smoothing (0.6536), which already beat lgbm_upw at n=5 (reports/teammate_comparison.md lines 84-88), the gain is +0.040.", |
| "evidence": "Means over n=5 draws 0-4, with per-rep values. Each step is plausible. (a) lgbm_upw rule 0.5658 to zs_xgb_dann rule 0.6188 (+0.053): a zero-shot model trained on Madrid labels plus unlabelled Amsterdam (tools/full_score_combo.py:55-69), constant across reps (0.6186-0.6189). (b) Smoothing to 0.6536 (+0.035): consistent with the earlier +0.02 to +0.05 reported for smoothing. (c) Ensemble ens_e4 rule 0.6595 (+0.041 over the zero-shot rule). It is consistent across reps (0.6504 / 0.6687 / 0.6520 / 0.6598 / 0.6664) even though every shot-using member is weaker alone (protoda_maha 0.605, cos_maha 0.583, tab_dann_rz_ft 0.605, lgbm_upw 0.566) and the zero-shot weight is 0.667. This is ordinary ensemble diversity and I found no leakage path (by contrast, ens_e2 alone is 0.6397). (d) ens_e4 rule_smooth 0.6823 (+0.023). (e) ens_e4__lp rule 0.6863 and rule_smooth 0.6941: LP plus smoothing adds +0.012 over (d)." |
| }, |
| { |
| "severity": "minor", |
| "claim": "Variants were chosen on half A and the full-city headline includes half A, so it is mildly optimistic. The ranking still holds on half B.", |
| "evidence": "PREREGISTRATION.md:5-12 (e1-e3 full-city scores were seen earlier). Half-B rule_smooth at n=5: ens_e4__lp 0.6825, zs_xgb_dann 0.6354, lgbm_upw 0.5767 (lgbm_upw rule_B 0.5512). Half-B average over n: ens_e4__lp 0.7020 vs lgbm_upw rule 0.6390." |
| }, |
| { |
| "severity": "note", |
| "claim": "Small mismatch: the teammate table gives the zero-shot FINAL as rule 0.6142 / rule+smooth 0.6506. nnlab gives 0.6188 / 0.6536. The likely causes are the 3-seed DANN averaging (full_score_combo.py:20) and nnlab's smoothing excluding shots. This does not affect the conclusions.", |
| "evidence": "reports/teammate_comparison.md:63 vs reports/nnlab/scores/zs_xgb_dann.json mean at n=5" |
| } |
| ], |
| "numbers": "Recomputed in reports/nnlab/verify/recompute.py (my own ensemble, my own graph, my own LP, my own F1; lab.W.decide and lab.smooth used for the rule columns) and reports/nnlab/verify/decomp.py (n=5 decomposition from the score files, plus my own scoring of zs_lgbm, lgbm_upw and zs_xgb_dann at n=5). Rows below are n = 5/25/50/100/200, means over draws 0-4. ens_e4__lp rule_smooth: 0.6941/0.7027/0.7201/0.7249/0.7269, avg 0.7137. ens_e1__lp rule_smooth: 0.6888/0.6986/0.7200/0.7232/0.7256, avg 0.7113 (matches the claim). lgbm_upw rule: 0.5658/0.6284/0.6697/0.6973/0.7204, avg 0.6563. lgbm_upw rule_smooth: 0.5950/0.6540/0.6887/0.7116/0.7310, avg 0.6761. zs_xgb_dann rule_smooth: 0.6536 at n=5. Only the n=5 decomposition steps are recomputed from the score files: +0.0530 (zero-shot model), +0.0348 (smoothing), +0.0059 (ens_e4 rule_smooth vs zs rule_smooth: +0.0059 = 0.6823 - 0.6536 - ... namely ens_e4 rule 0.6595, rule_smooth 0.6823), +0.0118 (LP plus smoothing over ens_e4 rule_smooth). Total from lgbm_upw rule to ens_e4__lp rule_smooth: +0.1283." |
| }, |
| { |
| "verdict": "holds-with-caveats", |
| "summary": "The stored scores reproduce the headline numbers. The 5 numbers are per-n means (n = 5, 25, 50, 100, 200) over reps 0-4, not per-draw values. The large gain at n=5 is real: it shows up on half B (+0.131) at least as strongly as on half A (+0.122), and it adds up from parts that make sense. Going from lgbm_upw \"rule\" at 0.566 to ens_e4__lp \"rule_smooth\" at 0.694: the zero-shot XGB+DANN anchor alone adds +0.053 (0.619 rule), mixing it with the shot-using members adds +0.041 (ens_e4 rule 0.660), neighbour smoothing adds +0.023 (0.682) and label propagation adds +0.012 (0.694). So the \"+0.12\" is mostly an apples-to-oranges comparison, not leakage. The reference ignores both the zero-shot model, which already beat it at n<=25, and the smoothing step that the pick gets. Against lgbm_upw read with the same rule_smooth column, the gain is +0.099 / +0.049 / +0.031 / +0.013 / -0.004. At n=200 the pick is slightly WORSE than lgbm_upw with smoothing, and worse than lgbm_upw__lp. Selection optimism is small next to the gain: the top picks on half A lose about 0.005-0.008 more from A to B than the typical variant does. The checkerboard guard is weak, though: only 69 blocks; A and B share the same shot draws; part of B was effectively seen. One preregistered choice does not hold up on B: the n>=100 switch to FINAL-PEAK. Several pipeline components were chosen on Amsterdam: the member list, the zero-shot anchor, the post-processor, the decision column and the switch point. The draws 5-11 that the preregistration needs for confirmation do not exist yet: every score file has 25 draws. The whole gain on top of non-spatial methods also depends on having coordinates for all test pixels and predicting them all at once.", |
| "findings": [ |
| { |
| "severity": "major", |
| "claim": "The +0.12-0.13 jump at n=5 is real (it holds on half B), but it comes from comparing different column choices and a reference that leaves out the zero-shot model. It is not evidence that the few-shot method is better. The fair 'previous best' at n=5 was the teammate's zero-shot FINAL (0.639 argmax+smooth as built; 0.654 rule_smooth). Against that, ens_e4__lp gains about +0.04-0.055 at n=5.", |
| "evidence": "From scores/*.json, n=5 full city: lgbm_upw rule 0.5658; zs_xgb_dann rule 0.6188; ens_e4 rule 0.6595; ens_e4 rule_smooth 0.6823; ens_e4__lp rule_smooth 0.6941. Paired difference ens_e4__lp rule_smooth minus lgbm_upw rule at n=5: all +0.1284+-0.0037, A +0.1224, B +0.1313. Minus lgbm_upw rule_smooth: +0.0991 / +0.0487 / +0.0314 / +0.0132 / -0.0041 for n = 5..200. Minus zs_xgb_dann rule_smooth at n=5: +0.0406. reports/teammate_comparison.md lists their FINAL zero-shot at 0.639 for every n.", |
| "suggested_fix": "Report the gain against the strongest earlier pipeline at each n, read with the same decision column (lgbm_upw rule_smooth, and zs_xgb_dann rule_smooth / teammate FINAL 0.639). Do not report it against lgbm_upw 'rule'." |
| }, |
| { |
| "severity": "major", |
| "claim": "The FINAL-SWITCH threshold (move to FINAL-PEAK at n>=100) was chosen on half A and does not hold on half B at n=100. On B it would cost about 0.013 at n=100. Only n=200 holds up.", |
| "evidence": "ens_e2__lapshot minus ens_e4__lp (both rule_smooth), per n = 5/25/50/100/200. Half A: -0.0177 -0.0157 -0.0118 +0.0020 +0.0132. Half B: -0.0224 -0.0241 -0.0218 -0.0128 +0.0063. Full: ... -0.0055 +0.0095. At n=100 the top 3 on A (lgbm_upw__lp, ens_e2__lp, ens_e2__lapshot) rank 23, 22 and 33 on B.", |
| "suggested_fix": "Use FINAL-AVG for every n, or switch only at n=200. Say that the switch point was picked on Amsterdam. Draws 5-11 can then confirm it without having been used to choose it." |
| }, |
| { |
| "severity": "major", |
| "claim": "The preregistered confirmation has not been run yet. Every score file contains only draws 0-4 (25 draws). The protection given by half B is also weaker than it looks: full-city scores (which include B) were printed during selection (as PREREGISTRATION.md admits). A and B share the same 25 shot draws, so a lucky draw helps both halves. This verification has now also tabulated B for draws 0-4.", |
| "evidence": "All 114 scores/*.json have n_draws = 25. PREREGISTRATION.md, 'Honest caveat' paragraph. lab.py score_method computes rule_A / rule_B from the same P_all per (n, rep).", |
| "suggested_fix": "Run NNLAB_REPS=12 and report all pixels on draws 5-11 for ens_e4__lp, ens_e2__lapshot and lgbm_upw (both rule and rule_smooth), as the preregistration lists. Treat these as the only clean numbers." |
| }, |
| { |
| "severity": "minor", |
| "claim": "Selection bias from scoring about 110 methods (221 method x decision variants) on Amsterdam is measurable but small, about 0.005-0.01 on the average score. That is far below the average gain of about +0.057 over lgbm_upw rule, or +0.038 over lgbm_upw rule_smooth. Choosing between e4, e6 and e7 is pure noise: they differ by under 0.001 on A, they reorder on B, and the paired standard errors are about 0.002-0.004.", |
| "evidence": "Across 221 variants: Spearman(A avg, B avg) = 0.983; the A-B gap has median 0.0179 and SD 0.0142. Excess of the mean gap over the median gap for the top k on A: top1 +0.0076, top3 +0.0056, top10 +0.0050, top20 +0.0051. ens_e4__lp: A 0.7236, B 0.7020, B rank 2 of 221. ens_e6__lp, ranked 1st on A, is 8th on B. The best on B is ens_e4__lapshot (A rank 13). Per-draw std of ens_e4__lp rule_smooth by n: .0074 .0069 .0036 .0039 .0070, so the SE of a 5-rep mean is .0015-.0033. The gain over the reference on B is larger than on A at every n (e.g. vs lgbm_upw rule: A avg +0.0505, B +0.0630), so the gain is not an artefact of selecting on A.", |
| "suggested_fix": "Quote the result as about 0.71 +-0.01 and not to three decimals. Name e4 as a representative of a group of equivalent ensembles, not as the winner." |
| }, |
| { |
| "severity": "minor", |
| "claim": "The checkerboard is a guard against pixel-level label noise. It is not an independent test. There are only 69 blocks of 2 km (about 35 per half), and the halves interleave the same districts. The spatial operators cross the A/B border, but that leaks no labels, because they carry predictions and allowed shots, not B labels. Half B is also systematically harder for every method, so absolute levels on another subset of pixels can easily move by about 0.02.", |
| "evidence": "Computed from amsterdam_features.parquet coordinates only (no labels): 69 blocks; A share 0.481. r=3: 11.3% of pixels have a neighbour in the other half; on average 3.1% of a pixel's neighbours are in the other half. r=6: 26.2% and 6.8%. The kNN feature graph spans the whole city anyway. The A-B gap is positive for nearly all variants (median +0.018; lgbm_upw rule +0.034).", |
| "suggested_fix": "Describe A/B as a sanity split. Claim robustness from draws 5-11, and if possible from a blocked hold-out (whole contiguous regions), not from B." |
| }, |
| { |
| "severity": "minor", |
| "claim": "Knob provenance. Fixed in advance from Madrid or the literature: K=10 in w0=K/(n+K) (research_shortlist.json item 6, a heuristic rather than a published value); smoothing r=6, alpha=0.5 (teammate, Madrid; lab.py:61); the tau-0.5 rule (Madrid); the within-family knobs (metric.py:16-18 training steps, tab_nn.py:34-45 epochs and lam, both on lab.madrid_split). Fixed ad hoc and not Madrid-validated, but not searched on Amsterdam either: the LP knobs alpha 0.8, seed weight x10, k=10 cosine 0/1, spatial r=3, 20 iterations (transductive.py:67-69). They depart from shortlist item 4 (alpha 0.9, seed x5, k=15 Gaussian, spatial kNN 8, choose on the Madrid harness). The Madrid graph cache exists, but no madrid-check output is in the logs. Selected on Amsterdam: the anchor zs_xgb_dann (best zero-shot in the teammate_comparison Amsterdam table); the shot members met_protoda_maha, met_cos_maha, tab_dann_rz_ft (out of about 100 family variants); the e1-e7 compositions; the post-processor per base (lp / lapshot / clamp / em / st); reading rule_smooth rather than rule for __lp; and the n>=100 switch.", |
| "evidence": "ensemble.py docstring; transductive.py:67-69 and its DEVIATIONS block; research_shortlist.json items 4 and 6; preds/ens_e*/meta.json member lists; lab.py:61 'teammate's choice, made on Madrid'; no 'madrid-check' string anywhere in reports/nnlab/logs/.", |
| "suggested_fix": "List the knobs selected on Amsterdam explicitly in the write-up. Run transductive.py --madrid-check (LP knobs) and log the output, so the LP setting gets a Madrid justification after the fact." |
| }, |
| { |
| "severity": "major", |
| "claim": "Protocol fairness: the spatial and transductive parts need things a standard harness may not provide. They need coordinates of the test pixels (smoothing, the LP spatial graph). They need ALL unlabelled test features at fit time (the DANN inside zs_xgb_dann, tab_dann, protoda CORAL, the PCA-32 kNN graph over 25,992 rows, per-city normalisation). And they need batch prediction (smoothing and LP couple the test pixels). Scored one pixel at a time, smoothing reduces to the identity and LP to the base, so the result falls to about ens_e4 'rule'. If held-out pixels are not among the 25,992 rows, the graph and the DANN would have to be refit with them included. Spatial distance from the shots matters little at small n and somewhat more at n=200.", |
| "evidence": "Without spatial steps, ens_e4 rule = 0.6595 / 0.6736 / 0.6988 / 0.7067 / 0.7135 (avg 0.6904), still +0.034 avg over lgbm_upw rule (0.6563). Smoothing is worth +0.017 avg for ens_e4 (0.7071 vs 0.6904), and LP a further +0.007 (0.7137). Median distance to the nearest shot, using random-placement proxies: 20 shots 1342 m, 200 shots 356 m, 800 shots 155 m. At n=5 the LP spatial channel (r=3, 90 m) hardly reaches any test pixel. LP gain at n=200: e4 0.7253 to e4__lp 0.7269.", |
| "suggested_fix": "Confirm the organisers' interface: does modelCreate receive the unlabelled pool plus coordinates, and does prediction run as one batch? Prepare a fallback (ens_e4 + rule, no spatial steps) and report its numbers next to the main ones." |
| }, |
| { |
| "severity": "note", |
| "claim": "Reading rule_smooth on a __lp output does not reuse any labels. The scorer's smooth() drops shot rows as neighbours (lab.py:268-277, keep=). It does apply spatial structure twice (LP r=3 graph, then r=6 averaging), which amounts to a wider kernel, and the column was chosen on A. The effect is small and holds on B, so it does not invalidate the claim. It should be described as one combined post-processing step and not as 'label propagation' alone.", |
| "evidence": "ens_e4__lp rule to rule_smooth: full 0.7084 to 0.7137 (+0.0053); A +0.0050; B 0.6966 to 0.7020 (+0.0054). The transductive.py docstring forbids double smoothing only for __clamp.", |
| "suggested_fix": "Describe the method as 'ensemble, then LP (r=3 + feature kNN), then r=6 smoothing, then rule', and say the final smoothing was kept after looking at A." |
| }, |
| { |
| "severity": "note", |
| "claim": "The headline quotes ens_e1__lp, which is not the preregistered pick. The FINAL-AVG pick ens_e4__lp is slightly higher. Its numbers should be the ones quoted.", |
| "evidence": "ens_e1__lp rule_smooth: 0.6888 0.6986 0.7200 0.7232 0.7256 (avg 0.7113), matching the headline. ens_e4__lp rule_smooth: 0.6941 0.7027 0.7201 0.7249 0.7269 (avg 0.7137). lgbm_upw rule: 0.5658 0.6284 0.6697 0.6973 0.7204 (avg 0.6563), matching.", |
| "suggested_fix": "Quote ens_e4__lp." |
| } |
| ], |
| "numbers": "Scripts: reports/nnlab/verify/meth_skeptic.py, meth_skeptic2.py and meth_skeptic3.py. They read only scores/*.json, plus coordinate columns of amsterdam_features.parquet for the border and shot-distance statistics; no labels were read.\n\n(a) Reproduction: ens_e1__lp, ens_e4__lp and lgbm_upw match the headline exactly (see the last finding).\n\n(b) n=5 decomposition on all pixels: 0.5658 (lgbm_upw rule), then 0.6188 (+0.053, zs_xgb_dann rule), then 0.6595 (+0.041, ens_e4 rule), then 0.6823 (+0.023, smoothing), then 0.6941 (+0.012, LP). Total +0.128.\n\n(c) Paired difference per draw, mean +- SE over 5 reps, ens_e4__lp rule_smooth minus lgbm_upw rule_smooth:\n- all pixels: +.0991+-.0027, +.0487+-.0053, +.0314+-.0032, +.0132+-.0015, -.0041+-.0021\n- half B: +.1058, +.0572, +.0379, +.0218, -.0010\n\n(d) A vs B across 221 variants: Spearman 0.983. Median A-B gap 0.0179 (SD 0.0142). The top 1/3/10/20 on A have a gap larger than the median by 0.0076 / 0.0056 / 0.0050 / 0.0051, which estimates the winner's-curse optimism. ens_e4__lp: A 0.7236, B 0.7020 (B rank 2). ens_e2__lapshot: A 0.7176, B 0.6870 (B rank 26 on the average; rank 3 at n=200).\n\n(e) Switch check, ens_e2__lapshot minus ens_e4__lp for n = 5/25/50/100/200:\n- half A: -.0177, -.0157, -.0118, +.0020, +.0132\n- half B: -.0224, -.0241, -.0218, -.0128, +.0063\n\n(f) Fallbacks without spatial steps: ens_e4 rule averages 0.6904; ens_e2 rule averages 0.6780.\n\n(g) Geometry: 69 blocks of 2 km. Pixels with a neighbour in the other half: 11.3% at r=3, 26.2% at r=6. Median distance to the nearest of k random pixels: 1342 m for k=20, 526 m for k=100, 356 m for k=200, 155 m for k=800.\n\n(h) Draws: n_draws = 25 in all 114 score files, so draws 5-11 have not been scored.\n\nWording I would accept: \"On Amsterdam (all pixels except the shots; 5 shot draws per n, not yet confirmed on the preregistered draws 5-11), a fixed-rule ensemble anchored on a zero-shot domain-adapted model, followed by graph label propagation and neighbour smoothing, reaches macro F1 0.694 / 0.703 / 0.720 / 0.725 / 0.727 for n = 5 / 25 / 50 / 100 / 200 (average 0.714). The strongest earlier pipelines, read with the same smoothing and decision rule, reach 0.654 (zero-shot) at n=5 and 0.654 / 0.676 / 0.689 / 0.712 / 0.731 (retrained LightGBM). The gain is about +0.04 to +0.10 for n <= 50, about +0.01 at n=100 and none at n=200. Members, post-processing and the decision column were selected on half of Amsterdam; we estimate the resulting optimism at about 0.005-0.01. The method is transductive: it needs the coordinates and features of all test pixels at once. Without the spatial steps it scores 0.660-0.714 (average 0.690).\"" |
| } |
| ] |
