| [ |
| { |
| "verdict": "holds-with-caveats", |
| "summary": "I recomputed every cell of both tables from reports/nnlab/scores/*.json; my script is reports/nnlab/verify/factcheck_tables.py. The fresh view uses draws 5-11, the single-models table uses draws 0-4 with the \"rule\" column, and half B uses draws 0-11. No cell is off by more than 0.0006. Three cells are rounded up when they should round down: two in the results table and one in the single-models table. The half-B figure for the five-model ensemble in caveat 3 is also rounded wrong. These checks all match the sources: the gains in the bullets, the standard deviations, the count of about 130 variants (134 score files), the half-A selection numbers, the Madrid domain-copies check (8 of 9 draws; +0.005, +0.018 and +0.020), the shuffle test, and the reviewers' statements. I re-ran model_create2.py --n 5 --rep 0 and it reproduced the stored ens_e12 result: F1 0.688960 against 0.688960 stored, max probability difference 3e-8. Every command in the Reproducing section exists with the arguments shown. The substantive problems are wording that goes beyond the evidence. \"Best in peak\" is a tie with lgbm_feda alone. The supercon encoder breaks the Mahalanobis-head claim. EM helped on this night's Madrid stand-in, which contradicts two sentences. Caveat 1 puts the anchor weight among the things chosen on Amsterdam, but the scripts say it was fixed in advance. The line-by-line review claim has no record. The self-training and \"one step where both agree\" statements are too strong.", |
| "findings": [ |
| { |
| "severity": "minor", |
| "claim": "Results table, five-model ensemble with smoothing, n=200: the report says 0.726. The source value is 0.7255, which rounds to 0.725.", |
| "evidence": "confirmation.csv ens_e4 rule_smooth fresh n200 = 0.725464; the per-draw mean from scores/ens_e4.json for reps 5-11 is 0.7255. The difference is 0.0005, inside the 0.0006 tolerance, but the rounding is wrong.", |
| "suggested_fix": "0.725" |
| }, |
| { |
| "severity": "minor", |
| "claim": "Results table, five-model ensemble plus label propagation, n=200: the report says 0.729. The source value is 0.7285, which rounds to 0.728.", |
| "evidence": "ens_e4__lp rule_smooth fresh n200 = 0.728490. The difference is 0.0005; the rounding is wrong.", |
| "suggested_fix": "0.728" |
| }, |
| { |
| "severity": "minor", |
| "claim": "Single-models table, metric learning with the cosine encoder and Mahalanobis head, n=100: the report says 0.689. The source value is 0.68846, which rounds to 0.688.", |
| "evidence": "scores/met_cos_maha.json, rule, reps 0-4, n=100 gives 0.68846. Every other cell in the 14 single-model rows matches to within 0.0005, and all averages are correct.", |
| "suggested_fix": "0.688" |
| }, |
| { |
| "severity": "minor", |
| "claim": "Caveat 3 says the five-model ensemble averages 0.697 on half B. No row gives 0.697: ens_e4 with smoothing is 0.696, and ens_e4 plus label propagation is 0.699.", |
| "evidence": "confirmation.csv view B: ens_e4 rule_smooth avg 0.696466 and ens_e4__lp rule_smooth avg 0.699328. FINAL-2 at 0.7005 (0.700) and lgbm_upw at 0.6560 (0.656) are correct.", |
| "suggested_fix": "'0.696 for the five-model ensemble (0.699 with label propagation)'" |
| }, |
| { |
| "severity": "minor", |
| "claim": "'FINAL-2 is best on the fresh draws in ... the peak (0.744 at n=200)' is effectively a tie: lgbm_feda alone with smoothing reaches the same peak.", |
| "evidence": "Fresh view, n=200: ens_e12 rule_smooth 0.744281 against lgbm_feda rule_smooth 0.744275, a difference of 0.000006. FINAL-2's peak gain comes entirely from the LightGBM with domain copies. The average claim (0.714 against 0.710 for the next best) is correct, as are the n=25 tie (0.70499 against 0.70465) and the n=5 deficit of 0.004 (paired difference -0.0043, SE 0.0039, ens_e4__lp ahead in 5 of 7 draws, so inside the noise).", |
| "suggested_fix": "'... and ties the LightGBM with domain copies alone at the peak (0.744 at n=200).'" |
| }, |
| { |
| "severity": "minor", |
| "claim": "'Against the round-1 five-model ensemble the margin ... comes from n >= 100' holds against the version with label propagation. Against the plain ensemble FINAL-2 is ahead at every n, and n=50 contributes too.", |
| "evidence": "Fresh, smoothing, FINAL-2 minus ens_e4: +0.004 / +0.006 / +0.010 / +0.011 / +0.019. FINAL-2 minus ens_e4__lp: -0.004 / +0.000 / +0.002 / +0.007 / +0.016. The averages +0.010 and +0.004 are correct.", |
| "suggested_fix": "Say the n>=100 statement refers to the ensemble with label propagation, or say the gap widens with n." |
| }, |
| { |
| "severity": "minor", |
| "claim": "Finding 'a within-class Mahalanobis head is best on all four encoders' is false for the supcon encoder.", |
| "evidence": "Draws 0-4, rule: met_supcon_maha 0.6304 < met_supcon_ftproj 0.6407. On cos, proto and protoda, Mahalanobis is best (0.656, 0.648, 0.655). The ranges quoted for prototypes (0.56-0.60), soft k-means (0.54-0.60) and the projection fine-tune (about 0.64) are correct.", |
| "suggested_fix": "'best on three of four encoders (supcon: projection fine-tune 0.641 against Mahalanobis 0.630)'." |
| }, |
| { |
| "severity": "minor", |
| "claim": "'Prior re-estimation by EM hurt on every base, as it did on the Madrid shift harness earlier' contradicts this night's Madrid stand-in. There EM improved the score.", |
| "evidence": "logs/lp_madrid_check.log: base avg 0.4981 against em 0.5080 (+0.010). On Amsterdam, EM did hurt on all four bases: lgbm_upw 0.656 to 0.646, zs_dann 0.576 to 0.571, zs_lgbm 0.592 to 0.583, zs_xgb_dann 0.618 to 0.600. Caveat 4 ('Twice a Madrid stand-in and Amsterdam disagreed') therefore leaves out a third disagreement, EM. Clamp was also positive on Madrid (0.5057).", |
| "suggested_fix": "Drop 'as it did on the Madrid shift harness', or cite which harness, and add EM to the disagreements in caveat 4." |
| }, |
| { |
| "severity": "minor", |
| "claim": "Caveat 4 says 'the domain copies are the one adaptation step where both agree'. That is not the only one: LP-FT against LP also agreed on Madrid and Amsterdam.", |
| "evidence": "cache/lpft/madrid_check.json: lp 0.484/0.528/0.527 against lpft 0.491/0.550/0.574 at n=5/50/200. On Amsterdam, draws 0-4: tab_dann_rz_ft 0.637 against tab_dann_rz_lpft 0.662. The fine-tune strength from tab_nn selection.json (lam=10) is also chosen on Madrid.", |
| "suggested_fix": "'one of the adaptation steps where both agree'." |
| }, |
| { |
| "severity": "minor", |
| "claim": "Caveat 1 lists the anchor weight 10/(n+10) under 'Chosen on Amsterdam, not Madrid'. The scripts and the reviewer say it was fixed in advance from the literature shortlist, not chosen on Amsterdam. Separately, the claim that 'every knob inside the models' was fixed on Madrid or in advance overstates it.", |
| "evidence": "ensemble.py docstring: 'Rule, fixed before any ensemble was scored (from the literature shortlist, item 6)'. The verification.json methodology agent lists K=10 as 'Fixed in advance ... a heuristic rather than a published value'. On the knobs: the leakage agent could not establish where the teammate's XGBoost and DANN hyper-parameters came from ('best_config.json' is not in the repo). The transductive LP knobs were ad hoc and never checked on Madrid. TOPK=120 and shot weight 30 were fixed in advance (lgbm_var.py docstring) but not tuned on Madrid. They also depart from shortlist item 10, which specifies 60 DANN + top-60 features. Separately, the Amsterdam-chosen list correctly includes the member list, i.e. picking lgbm_feda over w100/emb/fedaw100 and e12 over e10/e11 on half A; that part is consistent with PREREGISTRATION.md.", |
| "suggested_fix": "Move the anchor weight to 'fixed in advance (heuristic, not Madrid-validated)'. Add the teammate's zero-shot hyper-parameters as 'provenance unknown'." |
| }, |
| { |
| "severity": "minor", |
| "claim": "'Self-training ... added nothing' is only true once label propagation follows. Without it, adding the self-trained LightGBM raised the ensemble, and on its own the self-trained model scored below its pseudo-label source.", |
| "evidence": "Half A, draws 0-4, rule_smooth: ens_e6 (e4 + ens_e1__lp__st) 0.7199 against ens_e4 0.7139 (+0.006). Draws 0-4 all pixels, rule: 0.6971 against 0.6904. With LP: ens_e6__lp 0.7243 against ens_e4__lp 0.7236 (+0.0007), which matches the pre-registration's 'under 0.001'. ens_e1__lp__st alone scores 0.690 against its base ens_e1__lp at 0.706.", |
| "suggested_fix": "'added nothing once label propagation is applied (+0.001); +0.006 without it'." |
| }, |
| { |
| "severity": "minor", |
| "claim": "In the finding on LP-FT, '300 steps ... was worse on Amsterdam (0.640)' quotes the fresh-draw value inside a paragraph whose other numbers are draws 0-4.", |
| "evidence": "tab_dann_rz_lpft_s300 rule: draws 0-4 0.644, fresh 0.640. The comparison values 0.637 and 0.662 are draws 0-4, and 0.658 is fresh (0.6575). Madrid preferred 300 (steps_selection.json mean 0.5459 against 0.5381 for 60), but its margin over 150 and 600 is under 0.001. STEPS=60 is the a-priori value from shortlist item 12, so the reported 60-step model was not chosen on Madrid.", |
| "suggested_fix": "'(0.644 on draws 0-4, 0.640 fresh)'. Also note that 60 steps was the a-priori value and was kept against the Madrid choice." |
| }, |
| { |
| "severity": "minor", |
| "claim": "'The round-2 scripts each had an independent line-by-line review before running' has no supporting record in the repository.", |
| "evidence": "verification.json holds only the three round-1 agents (leakage, recomputation, methodology). The only review log is logs/ts_nn_smoke_review.log (00:36, a round-1 sequence-network smoke test). Nothing records reviews of lgbm_var.py, lpft.py or the e10-e12 ensembles.", |
| "suggested_fix": "Remove it, or point to where the reviews are recorded." |
| }, |
| { |
| "severity": "minor", |
| "claim": "Selection optimism (bullet and caveat 2): 0.696 to 0.689 for lgbm_feda and '0.004 to 0.007' are correct. But reference models that were never selected also drop by about 0.003 from draws 0-4 to fresh, so not all of the gap is selection.", |
| "evidence": "First 5 minus fresh: lgbm_feda rule 0.6957-0.6887 = 0.007; FINAL-2 smooth 0.7189-0.7138 = 0.005; ens_e4__lp 0.7137-0.7097 = 0.004. The never-selected lgbm_upw rule 0.6563-0.6532 = 0.003 and rule_smooth 0.6761-0.6727 = 0.003. The reviewers' 0.005-0.01 estimate comes from verification.json (top-k excess A-B gap +0.005 to +0.008).", |
| "suggested_fix": "Mention that the unselected reference drops by 0.003, so net selection optimism is about 0.001-0.004." |
| }, |
| { |
| "severity": "note", |
| "claim": "'Shuffled labels ... The gain comes from the shot labels, not from a leak' slightly overstates what the test shows. The numbers are correct.", |
| "evidence": "Shuffled 0.3152 (n=5) and 0.2285 (n=200) against true 0.6154 and 0.7380 (scores/lgbm_feda.json rep 0 rule: 0.6154, 0.7380). The test shows the model depends on the shot labels. Ruling out leaks rests on the code audit (round 1, verification.json), and that audit predates lgbm_feda.", |
| "suggested_fix": "'... so the gain depends on the shot labels' and cite the leakage audit separately." |
| }, |
| { |
| "severity": "note", |
| "claim": "Several context claims check out: the Madrid domain-copies check, its caveat, the step-by-step description, the half-A selection numbers and the verification quotes.", |
| "evidence": "feda_madrid_check.json: feda wins 8/9 (loses only n=5 rep 2: 0.5003 against 0.5110), means +0.0049 / +0.0181 / +0.0204. The top-120 list comes from zero_shot_lgbm.txt trained on all Madrid (lgbm_var.top_gain_idx), as the caveat says. Half A in PREREGISTRATION.md: ens_e12 0.7052/0.7161/0.7410/0.7481/0.7591 (avg 0.734), e10 0.721, e11 0.724, ens_e4 0.7139, lgbm_feda 0.7236, all reproduced. Feature layout [x, x_top, NaN] / [x, NaN, x_top], weight 30, balanced classes and per-city robust-z all appear in fit_predict. SMOOTH_R=6, alpha 0.5, 'teammate's choice, made on Madrid' is at lab.py:61. There are 69 2-km blocks (computed from px_key/py_key). lgbm_upw draws 0-4 = 0.5658/0.6284/0.6697/0.6973/0.7204 and reproduces with zero difference according to verification.json. Variant count: 134 score files. Shot weight 100 on top of the copies: 0.6943 against 0.6957. The tabular-network row lists lpft (0.662) as the best variant, but lpft_ent is marginally higher (0.663)." |
| }, |
| { |
| "severity": "note", |
| "claim": "Most of the Reproducing section is correct, but it leaves out the steps that build several compared rows. The 'cache/*/selection*.json' glob also misses most of the Madrid selection files.", |
| "evidence": "All named scripts exist and their arguments parse: full_score_combo.py zeroshot (inserts its own sys.path, so it works under -I), baselines.py all, ensemble.py NAME --zs --shot [--k], tab_nn.py --seeds, lab.py score, model_create2.py --n --rep [--no-spatial]. model_create.py exists with --n/--rep. Missing are tools/nnlab/transductive.py (it builds ens_e4__lp, lgbm_upw__lp and __st) and the ensemble.py calls for e4 (round 1). The Madrid choices are in lpft/steps_selection.json, metric/*_select.json and ts_nn/select_*.json, which 'selection*' does not match; only tab_nn/selection.json does.", |
| "suggested_fix": "Add 'python tools/nnlab/transductive.py' and the e4 ensemble command, and change the glob to cache/*/*select*.json." |
| }, |
| { |
| "severity": "note", |
| "claim": "The headline numbers and the remaining derived figures are correct.", |
| "evidence": "Fresh: FINAL-2 0.7138 (0.7003), lgbm_upw 0.6727 (0.6532). FINAL-2 SDs 0.0074/0.0116/0.0146/0.0037/0.0033 match '0.007 / 0.012 / 0.015 / 0.004 / 0.003'. Like-for-like gains +0.085/+0.053/+0.035/+0.020/+0.012, no-coordinates +0.047, LightGBM with copies against the old LightGBM +0.036 (rule; +0.029 with smoothing). LP +0.006 on fresh smoothed draws, though +0.018 in the draws 0-4 rule view. LP on Madrid 0.475 against 0.498. Finding numbers 0.637/0.582, 0.662, 0.658, DANN/ResNet/MLP/CORAL 0.637/0.626/0.603/0.578, sequence-network zero-shot 0.32-0.53, embeddings 0.628 against 0.656. The fine-tune-beats-refit pattern holds on every backbone, including the sequence networks." |
| } |
| ] |
| }, |
| { |
| "verdict": "holds-with-caveats", |
| "summary": "I found no leakage in lgbm_feda or ens_e12 (FINAL-2), and an independent refit reproduces the stored numbers. The only Amsterdam labels that reach the fit are the current draw's shots. The top-120 feature list comes from a Madrid-only booster. The train and predict matrices have the same layout, and shot and non-shot rows are built the same way at predict time. I refit the draw n=50, rep=7 with my own matrix construction, using 3 threads instead of the original 6. It matches the stored lgbm_feda to within 3e-8, with 100% argmax agreement. Rebuilding ens_e12 from the stored members also matches to within 3e-8. My own scorer, with the shots re-derived, gives argmax macro F1 0.671204 and rule_smooth 0.699202, the same as scores/ens_e12.json to every printed digit. The gain is not an artefact of the scorer: on all 35 fresh draws (reps 5-11) lgbm_feda beats lgbm_upw under all three decision rules and on half B. The gain is largest under plain argmax (+0.039, against +0.036 rule and +0.029 rule_smooth), so the decision rule shrinks it rather than creating it. The caveat is the mechanism: \"Amsterdam pixels get branches of their own, fitted on the shots\" does not describe what happens. Most of the split gain sits on the Madrid-only copies, and an ablation shows that almost any way of letting the trees tell the two cities apart recovers most of the gain.", |
| "findings": [ |
| { |
| "severity": "note", |
| "claim": "No Amsterdam label other than the current draw's shots reaches the lgbm_feda fit.", |
| "evidence": "In tools/nnlab/lgbm_var.py main(), Amsterdam is read with columns=feats+PID only, without age_class. The labels come only from lab.shot_labels(d, shot) for lab.shots_for(d, n, rep). Amsterdam robust-z uses W.robustz_fit over all Amsterdam rows' features, which needs no labels, does not depend on the draw, and is transductive use the protocol allows. model_create2.py calls the same fit_predict and reads non-shot labels only in its CLI scoring line. In my refit I re-derived the shots with default_rng(42+1000n+rep) and per-class rng.choice; they are identical to W.shots_for." |
| }, |
| { |
| "severity": "note", |
| "claim": "The top-120 feature list uses no Amsterdam labels: it comes from a Madrid-only zero-shot booster.", |
| "evidence": "tools/export_winners.py calls W.train_zero_shot(d['X_mad'], d['y_mad']), which trains on Madrid only with class-balanced weights and raw features, and saves models/zero_shot_lgbm.txt. In the saved booster, tree 0 has a root internal_count of 60929, about 0.8 (bagging) times the 76263 Madrid rows; a Madrid+Amsterdam fit would show about 82k. The feature list drops columns whose std is 0 in either city, which also needs no labels. top_gain_idx() equals the feda_idx stored in preds/lgbm_feda/meta.json (checked: True)." |
| }, |
| { |
| "severity": "note", |
| "claim": "Train and predict matrices have the same column layout, and shot rows are not treated differently from other Amsterdam rows at predict time.", |
| "evidence": "Training rows are Madrid [x, x_top, NaN] and shots [x, NaN, x_top]. Prediction rows are all 25,992 Amsterdam rows as [x, NaN, x_top], so the shots get exactly the same row format as every other Amsterdam row. Shots are removed by keep before scoring and before smoothing (lab.smooth with keep slices the adjacency matrix, so shots are never neighbours). The refit model classifies the shots 100% correctly against 66% on non-shot rows, but the shots are never scored, so this does not inflate the result." |
| }, |
| { |
| "severity": "note", |
| "claim": "An independent refit of lgbm_feda for n=50, rep=7 reproduces the stored predictions, and the ens_e12 rebuild reproduces the stored file and the score file.", |
| "evidence": "Refit with my own matrix code, {**LGBM_PARAMS, **RETRAIN_OVERRIDES, n_jobs=3} (the stored run used n_jobs=6), took 66 s. Against the stored file: max abs diff 2.98e-08, mean 4.6e-09, argmax agreement 1.0. The zs_xgb_dann store equals (P_xgb + mean DANN seeds)/2 from the npz exactly. My ens_e12 rebuild with w0=10/60 differs from the stored file by at most 2.98e-08. My macro F1 on non-shot rows is 0.671204 (argmax) and 0.699202 (rule_smooth), equal to scores/ens_e12.json (argmax 0.6712044, rule_smooth 0.6992015). Outputs are in reports/nnlab/verify/leak_recompute_e12_n50_r7.json." |
| }, |
| { |
| "severity": "minor", |
| "claim": "The report's explanation of the mechanism is not supported: most of the gain comes from the Madrid-only copies, and simply letting the trees separate the two domains reproduces most of it.", |
| "evidence": "Split-gain shares in the refit model (n=50, rep=7): shared block 0.670, Madrid-only copies 0.285, Amsterdam-only copies 0.045. Split-count shares: Madrid copies 0.297, Amsterdam copies 0.019. Ablation on the same draw (argmax macro F1, non-shot rows): lgbm_upw 0.6410; a single 0/1 domain-indicator column 0.6471; Madrid copies only 0.6616; Amsterdam copies only 0.6590; full feda 0.6637. Outputs are in reports/nnlab/verify/leak_feda_ablation_n50_r7.json. Either copy on its own gives the trees missing-value routing that separates the cities: in a Madrid-copy split, Madrid rows split on the value and all Amsterdam rows go to the missing-value branch. This also fits the structure of the hyper-parameters. min_child_samples=80 counts rows, so an Amsterdam-only copy can hardly be split with 20 shots (n=5) or 100 shots (n=25), yet the largest gains are at n=5 (+0.055) and n=25 (+0.062). That contradicts 'branches of their own, fitted on the shots'. The gain is real and label-driven (the shuffled-label check collapses it), so this is a wording problem, not a leak.", |
| "suggested_fix": "Change step 2 of OVERNIGHT_RESULTS.md to something like: 'the copies let trees learn Madrid-specific splits that Amsterdam rows bypass through the missing-value branch; one domain indicator column alone gives about a quarter of the gain, and Madrid-only or Amsterdam-only copies alone give most of it (one draw, n=50 rep 7)'. Mark this as a single-draw observation." |
| }, |
| { |
| "severity": "note", |
| "claim": "The gain is not an artefact of the decision rule or of the class-balanced shots: it holds under plain argmax, on every fresh draw, and on half B.", |
| "evidence": "lgbm_feda minus lgbm_upw, mean over draws 5-11 (35 draws), from the scores/*.json files: argmax +0.0388, rule +0.0355, rule_smooth +0.0288. Half B: +0.0402 / +0.0372 / +0.0315. By n (argmax): n5 +0.055, n25 +0.062, n50 +0.041, n100 +0.022, n200 +0.014. lgbm_feda is better on 35 of 35 draws under every rule and on both halves. The rule and smoothing shrink the gain, so they are not producing it. Draws 0-4 show the same pattern (argmax +0.078 at n=5, down to +0.016 at n=200)." |
| }, |
| { |
| "severity": "note", |
| "claim": "A minor point about data provenance: lgbm_var and model_create2 read features from release/hack4dev-building-age-winners/data/processed (W.DATA), while lab.load reads data/processed (lab.PROC).", |
| "evidence": "lgbm_var.main asserts that the row order of the two tables is the same (px/py keys and Madrid labels). My refit used W.DATA, and its labels reproduce the scores exactly, so the two copies agree as far as this check reaches. This is not a correctness problem." |
| } |
| ] |
| } |
| ] |
