| [ |
| { |
| "verdict": "clean", |
| "leakage_findings": [ |
| "None found. A grep of classical.py for _ya, age_class, weighted_mean_year, usable_label and held_out finds nothing. Amsterdam labels are read only in the main loop, through lab.shot_labels(d, lab.shots_for(d, n, rep)), and are used only to fit that draw's model.", |
| "Labels from one draw never reach another draw. Every cache (cls_pca64, cls_nys500, cls_knn_madrid) holds only label-free Madrid and Amsterdam features or Madrid neighbour indices, so no shot labels are stored in them.", |
| "No hyper-parameter was chosen from an Amsterdam score. The two departures from the spec (SVC subsample 6000, LR max_iter=100 with tol=1e-3) are compute-budget cuts based on timing only. Both are recorded in the meta CFG." |
| ], |
| "correctness_findings": [ |
| "robustz is applied to Madrid and to Amsterdam separately (Zm and Za). PCA(64) and Nystroem are fitted label-free on vstack(Zm, Za), as the spec asks.", |
| "Class indexing is right. Labels 1..4 index balanced_class_weights with y-1. LR, LDA and SVC column order comes from the sorted classes_ [1,2,3,4]. kNN builds column c from votes where lbl == c+1.", |
| "Every unit of work predicts all 25,992 Amsterdam rows in table order. The kNN shot-label matrix broadcast_to(ys) lines up column by column with S = Pa[idx]. Merging the cached Madrid top-25 with the distances to the shots and then taking argpartition gives exactly the joint top-25 search.", |
| "Shot rows predict themselves through a self-distance of 0 (kNN) or by being in the training set. This does no harm because the scorer drops shot rows.", |
| "Resume works per (method, n, rep) through lab.have_pred. Smoke outputs carry the _SMOKE suffix, so they cannot mix with full outputs. Zero-shot methods are saved once, at n=0, rep=0.", |
| "The caches are keyed well enough for this script. The PCA and kNN settings are the same in smoke and full runs, and the Nystroem cache name includes the component count (smoke 60 vs full 500). One edge case: the kNN cache depends on the PCA cache, so if cls_pca64.npz is deleted, cls_knn_madrid.npz must be deleted with it. Today PCA is deterministic (random_state=0), so this is harmless.", |
| "Minor, in lab.py and not a bug: save_pred is not atomic. A run killed in the middle of np.save could leave a truncated .npy that have_pred then treats as done.", |
| "There is no backbone or seed averaging to check in this family. Seeds are fixed (PCA, Nystroem and the SVC subsample all use seed 0), so runs are deterministic." |
| ], |
| "fixes_applied": [], |
| "full_run_command": "cd && NNLAB_THREADS=3 python tools/nnlab/classical.py", |
| "est_full_minutes": 45, |
| "smoke_rerun_passed": true, |
| "notes": "I changed nothing, but I re-ran the smoke test to check RAM and the code paths. It passed with NNLAB_THREADS=2: all 9 methods ran without error in 61 s. Peak working set was 1.68 GB, measured with psutil peak_wset; the largest step is the 1179-dimensional FEDA LR. Memory for that step does not depend on the iteration count, so the full run should also stay under 2.5 GB.\n\nI scored cls_lda_SMOKE and cls_svm_SMOKE (rule: lda 0.597 / 0.686, svm 0.524 / 0.689 at n=5 / n=200). These are smoke configs, not results. Afterwards I deleted every _SMOKE prediction folder, the _SMOKE score files and the smoke-only cache cls_nys60.npz. The valid full-run caches are kept under reports/nnlab/cache: cls_pca64.npz, cls_nys500.npz and cls_knn_madrid.npz.\n\nMethods the full run produces: cls_lr_zs, cls_svm_zs, cls_lr, cls_lr_feda, cls_svm, cls_svm_feda, cls_nys_lr, cls_knn, cls_lda.\n\nRun time: the author estimates about 42 min on 3 threads, from per-draw timings at n=200 (lr 16 s, lr_feda 25 s, nys 12 s, svm about 20-24 s, svm_feda about 40-47 s). That estimate is plausible, but it sits close to the 45-minute budget, and the shared laptop could push it to about 50 min. I report 45 as a realistic central figure. I did not re-time the full SVC at 6000 myself.\n\nDisclosed deviations from the spec, all acceptable:\n- SVC subsample is 6000, not 12000 or the fallback of 8000.\n- LR runs max_iter=100 with tol=1e-3, so it is not fully converged.\n- kNN has no class balancing.\n- LDA uses empirical priors.\n- SVC probabilities are a plain softmax over the ovr decision function.\n\nI found no bug in lab.py." |
| }, |
| { |
| "verdict": "fixed", |
| "leakage_findings": [ |
| "None found. grep for _ya, age_class, weighted_mean_year, usable_label and held_out in transductive.py matches only a docstring line that explains why W.load_data is avoided.", |
| "The script reads Amsterdam labels only as lab.shot_labels(d, lab.shots_for(d, n, rep)), and only inside that same draw's refinement or st refit. No labels carry over from one draw to another.", |
| "Every knob is a module constant that matches the spec. The lapshot row-normalisation is a structural deviation. The author justifies it a priori with the graph degree and checks it on the Madrid pseudo-target only (base 0.485, lapshot_row 0.493, lapshot_raw 0.464, from a 40-tree smoke run). It is recorded in the meta as graph_norm=row. No Amsterdam score feeds into any choice.", |
| "--madrid-check draws its pseudo-shots from Madrid labels on the madrid_split pseudo-target, with a different rng seed (7+1000n+rep). It robust-z scales source and pseudo-target separately." |
| ], |
| "correctness_findings": [ |
| "The per-city transforms are right. The graph uses PCA-32 of lab.robustz(Xa) on Amsterdam alone. st scales Madrid with its own stats and Amsterdam with stats from the whole unlabelled Amsterdam matrix, which matches winners.train_retrained exactly (same LGBM_PARAMS+RETRAIN_OVERRIDES, shots x30, then balanced_class_weights over the union).", |
| "Class indexing is right. onehot uses y-1, pseudo-label classes are k+1, the weight lookup is [y-1], and _fit_lgbm subtracts 1 internally.", |
| "Every save is (25992,4) in table order. save_pred asserts the shape. Clamp, lp and lapshot work on all rows. EM and st predict all rows.", |
| "LP iterates F=aSF+(1-a)Y0 from Y0 for 20 steps with S=D^-1/2 A D^-1/2, which is correct. A has its self-loops removed after the union with lab.adjacency, which includes self.", |
| "EM (Saerens) is right. It reweights by pi/pi_s and renormalises, sets pi to the mean posterior, shrinks 50% to the Madrid prior and reweights by pi_hat/pi_s.", |
| "Resume is safe. Smoke outputs carry the _SMOKE suffix, so smoke and full runs never mix. The zero-shot EM is saved once at n=0. Per-draw bases are checked with have_pred before load_pred (lab.load_pred silently falls back to n0_r0). In the full run, missing lgbm_upw draws are skipped, not substituted.", |
| "Caches are keyed correctly. The graph cache name carries r, k and pca and does not depend on the draw. The Madrid base cache is keyed SMOKE/full.", |
| "Minor issue, fixed: the docstring said the union graph had about 35-45 neighbours. The measured mean degree is 22.4 (min 10). Code was correct.", |
| "Nit, not fixed: knn_graph assumes the first neighbour returned is the point itself. With exact duplicate PCA vectors one row loses one neighbour. The effect is negligible, and any self-loop is removed later anyway.", |
| "Race to watch, not a code bug: lgbm_upw is still being written by baselines.py (only n5_r0..r2 existed at review time). save_pred is not atomic, so start the full run only after baselines.py finishes." |
| ], |
| "fixes_applied": [ |
| "tools/nnlab/transductive.py docstring (DEVIATIONS, lapshot): changed the neighbour count from '~35-45, logit up to ~40' to the measured '~22 on average (min 10), logit up to ~22'. Documentation only; no code changed." |
| ], |
| "full_run_command": "cd && NNLAB_THREADS=3 python tools/nnlab/transductive.py --base lgbm_upw,zs_xgb_dann,zs_lgbm,zs_dann (start only after baselines.py has written all 25 lgbm_upw draws)", |
| "est_full_minutes": 100, |
| "smoke_rerun_passed": true, |
| "notes": "Reviewed all of lab.py, transductive.py and the winners.train_retrained / robustz / LGBM params code it reuses. The change was documentation only. I re-ran a light smoke on the fast path: NNLAB_THREADS=2, --base zs_xgb_dann --smoke --only lapshot. It used the cached graph (583,310 edges, mean degree 22.4) and took 0.1-0.2 s per draw. lab.py score ran on it (rule avg 0.637 for 2 draws; smoke number only, not used for anything). I then deleted its _SMOKE preds and score JSON. I did not re-run the st smoke because 5 other python jobs were already running on the shared machine, and a 40-tree LightGBM fit would have been a second heavy process; the st code did not change. Budget: the fast methods for all 4 bases take about 1-2 min. st costs one 400-tree LightGBM fit per draw on about 76k Madrid rows, plus the shots, plus about 15% of Amsterdam (about 3.9k pseudo-labelled rows), x393 features. Observed lgbm_upw fits under contention took 2-7 min per draw (file times 00:08, 00:10, 00:17), so 25 draws come to about 75-125 min. The 100 min estimate is plausible but could reach about 150 min if the machine stays contended. Peak RAM estimate is about 1.1 GB (Madrid and Amsterdam float64 plus their robust-z copies plus the vstack), under 2.5 GB. Methods produced: lgbm_upw__{clamp,lp,lapshot,em,st}, and for each of zs_xgb_dann, zs_lgbm and zs_dann the suffixes __clamp, __lp, __lapshot and __em (the zs __em ones are saved once at n=0). For __clamp read the rule column. lab.py bug-risk (not modified): load_pred silently falls back to n0_r0; the script guards against it with have_pred. st reads features as float32 values cast to float64, so they can differ from lgbm_upw's float64 W.load_data features at rounding level (documented deviation)." |
| }, |
| { |
| "verdict": "fixed", |
| "leakage_findings": [ |
| "No leakage found. ts_nn.py never mentions _ya, age_class, weighted_mean_year, usable_label or held_out (grep is empty). Amsterdam labels are read only at main() lines ~480-481, via lab.shots_for(d, n, rep) followed by lab.shot_labels(d, shot). They are used only to fit that draw's ft and joint heads, and are not shared between draws.", |
| "The raw-parquet reader asks only for px_key, py_key, year, the 18 band columns and qa_valid_1..3, so it never reads a label column. Madrid labels come from lab.load()['ym'].", |
| "Epoch count and lam are both chosen on Madrid only. The backbone trains on the madrid_split source part. Epochs are scored by argmax macro F1 on the pseudo-target. lam is scored with simulated n=5 and n=50 shots drawn from the pseudo-target, which the backbone never saw. No Amsterdam score is computed anywhere in the script. The choices are written to select_*.json and copied into meta['selection']." |
| ], |
| "correctness_findings": [ |
| "Row alignment and observation selection are verified by experiment. For both cities, interp_time(T)*SR_SCALE+SR_OFFSET averaged over years matches the feature table's sr_{band}_mean columns to within 3e-7 absolute (correlation 1.0) for all 6 bands. So the tensor rows follow the feature-table order, and the selection matches make_flat_view(agg='median') plus to_year_matrices: per-slot qa_valid, nanmedian, dropping years with any band NaN or Blue > BLUE_MAX, linear interpolation inside the series and nearest values at the edges.", |
| "make_inputs is called separately for each city, so the mean/std standardisation and the ry per-year median use that city's own label-free statistics. The observed-mask channel is left out of the ry subtraction.", |
| "Class indices are correct. Training uses ys-1, the class weights come from bincount[1:], and finetune_last_layer and joint_head take labels 1..4. Columns 0..3 map to classes 1..4: the softmax head is ordered that way, and joint_head's predict_proba is too, because all 4 classes appear in Madrid.", |
| "P has shape (25992, 4) and covers every Amsterdam row in table order (proba(Za...) / predict_proba(Sa)). Probabilities are averaged over seeds with np.mean over embs. The backbone stays frozen: ft starts from the backbone's (W0, b0) and only touches the last layer on the cached raw embeddings, and joint is an LR on the frozen embeddings, standardised with the Madrid embedding mean and std (label-free).", |
| "Smoke and full outputs cannot mix: method names carry the _SMOKE suffix and cache files carry the _SMOKE tag. The tensor cache is shared, but it is deterministic and its shape is asserted. Resume works per (method, n, rep).", |
| "BUG (fixed). When select_*.json exists but the embedding npz does not (a crash between the two writes, or any seed beyond the first), backbone() retrains for sel['epochs'] epochs. For tcnn_dann the GRL lambda schedule was then stretched over sel['epochs'] instead of max(epoch_grid), so the rebuilt backbone was not the model that was selected. I added train(..., sched_epochs=...), and the retrain branch now passes max(sel['epoch_grid']). Verified: after deleting the DANN npz, the retrain branch reproduces the zs predictions with a max abs diff of 0.0.", |
| "Minor, not changed. The cache files are keyed by backbone, variant, seed and smoke tag, not by hyper-parameters such as EPOCH_GRID or LR. If those constants are edited later, the reports/nnlab/cache/ts_nn/*.npz and select_*.json files must be deleted by hand.", |
| "Documented deviations that I accept as honest: 42 years (1984-2025, as the feature pipeline uses), epoch grid {5,10,15}, 1 seed, the selected checkpoint used directly with no retrain on all Madrid, and gru and tcnn_dann_ry only with --extended. All of these are recorded in meta. Re-running the smoke test gave the same selection scores as the author's run, so it is deterministic." |
| ], |
| "fixes_applied": [ |
| "tools/nnlab/ts_nn.py: train() now takes sched_epochs (the length of the DANN lambda schedule; the default is unchanged). The retrain-from-selection branch in backbone() passes sched_epochs=max(sel['epoch_grid']), so it reproduces the selected checkpoint exactly. The normal path is unchanged." |
| ], |
| "full_run_command": "cd && NNLAB_THREADS=3 python tools/nnlab/ts_nn.py > reports/nnlab/logs/ts_nn.log 2>&1", |
| "est_full_minutes": 66, |
| "smoke_rerun_passed": true, |
| "notes": "Methods the full run produces: ts_tcnn_st_{zs,ft,joint}, ts_tcnn_ry_{zs,ft,joint}, ts_tcnn_dann_st_{zs,ft,joint}. --extended adds tcnn_dann_ry, gru_st and gru_ry, which takes about 2.5 h more and is beyond budget.\n\nSmoke re-run with NNLAB_THREADS=2: 188 s, peak RSS 1.55 GB. The --only tcnn_dann retrain-branch check took 75 s. Scoring ts_tcnn_ry_ft_SMOKE and ts_tcnn_dann_st_joint_SMOKE worked and gave the same numbers as the author's report. Afterwards I deleted every ts_*_SMOKE prediction folder, every score json and cache/ts_nn/*SMOKE*. The tab_mlp_*_SMOKE folders belong to another job and were left alone. The smoke log is at reports/nnlab/logs/ts_nn_smoke_review.log.\n\nBudget, measured at 2 threads under contention: tcnn about 47-50 s per full source epoch, so 15 epochs plus evals, embedding and lam selection come to about 13 min per variant. tcnn_dann is about 90-107 s per epoch, about 25-28 min. Heads are about 7-10 s per draw for tcnn and 12-17 s for dann, about 12 min over 75 draw-combos. With loading, the total is about 60-70 min on 3 threads, so it should fit 70 min but with little margin if the machine stays contended. Peak RAM is about 1.6 GB, under 2.5 GB.\n\nNo bug found in lab.py." |
| }, |
| { |
| "verdict": "fixed", |
| "leakage_findings": [ |
| "None found. The script never mentions _ya, age_class, weighted_mean_year, usable_label, held_out or read_parquet. It reads Amsterdam labels only through lab.shot_labels(d, lab.shots_for(d, n, rep)), and only to fit that one draw's ft/joint/proto head; nothing carries over between draws.", |
| "Epochs and ft lam are chosen only on lab.madrid_split. Training uses the source part, scoring uses macro F1 of W.decide on the Madrid pseudo-target, the prior comes from the source part, and the shots are left out of the pseudo-target score. Unlabelled Amsterdam rows are used only as the DANN/CORAL adversary target in the final backbones, which the rules allow.", |
| "Fixed-in-advance knobs (shot_weight 30, C=1, proto k=10 and T=20, CORAL weight 1, the optimiser settings) and the selection scores all go into every meta.json." |
| ], |
| "correctness_findings": [ |
| "RAM (real problem): with the first smoke run (your version), peak RSS of the process was 2710 MB, over the 2.5 GB cap. Cause: lab.rankgauss on the full 76k x 393 Madrid matrix creates about 1.5 GB of float64/rank temporaries (lab.robustz adds about 0.5 GB), on top of about 1 GB already resident. Fixed by running the transforms in column blocks.", |
| "Cache key gap (minor, fixed): emb_{bb}_{inp}_s{seed}.npz is keyed without the epoch count. If selection.json were regenerated with a different choice, a stale net would be reused silently while meta reported the new count. The cached 'epochs' field is now checked against selection.json and the net is retrained if they differ.", |
| "Efficiency (minor, fixed): head_proto used dict.setdefault(key, _unit(...)), which runs the full Madrid and Amsterdam normalisation on every call.", |
| "Verified correct: each city is transformed separately (main run Madrid and Amsterdam; selection source part and pseudo-target). Row order is kept, because embed works in order and Xm60/Xa60 are aligned by lab's merge. Labels 1..4 map to columns 0..3 consistently (Y = ys - 1, predict_proba classes 1..4, head columns go to W.decide). Every save is (25992, 4) for all rows. The 3 seeds' probabilities are averaged before saving. The backbone stays frozen: the heads only use the cached embeddings and W0/b0. zs is saved once at n=0, rep=0. Smoke and full runs use separate _SMOKE names and a separate cache dir, so they cannot mix. Resume skip works (a re-run of the smoke skipped everything in 4 s). The DANN loop matches tools/compare_teammate.py train_dann. The '45-epoch run evaluated at 15/30/45' shortcut really equals separate runs (constant lr, same generator, eval uses no RNG). sklearn 1.9 LogisticRegression is multinomial, and newton-cholesky warm start from coef_ works. Runs are deterministic: smoke scores after my fixes are identical to the author's.", |
| "Documented deviation, kept as is: the joint head does not call lab.joint_head. It standardises the embeddings with Madrid mean/sd and uses newton-cholesky with a warm start. Because the C=1 L2 penalty then acts on standardised weights, the objective is not literally the same as lab.joint_head on raw embeddings, despite the docstring's claim. It is still a legitimate, fixed-in-advance, label-free choice, and the author's convergence argument for it holds.", |
| "lab.py: no bug found." |
| ], |
| "fixes_applied": [ |
| "tools/nnlab/tab_nn.py: new _by_columns() wrapper; TRANSFORM rz and rg now run lab.robustz / lab.rankgauss in 48-column blocks. Output is bit-identical (np.array_equal checked on Xa for both), and smoke peak RSS fell from 2710 MB to 1546 MB. The selection inputs use the same wrapper.", |
| "tools/nnlab/tab_nn.py main(): a cached embedding file whose stored 'epochs' differs from selection.json is deleted and retrained, not reused.", |
| "tools/nnlab/tab_nn.py head_proto: the unit-normalised embeddings are computed once per seed, not on every call (setdefault ran its argument eagerly)." |
| ], |
| "full_run_command": "cd && NNLAB_THREADS=3 python tools/nnlab/tab_nn.py", |
| "est_full_minutes": 240, |
| "smoke_rerun_passed": true, |
| "notes": "The smoke rerun after the fixes (NNLAB_THREADS=2, from scratch) took 182 s with peak RSS 1546 MB. A re-run skipped every saved unit in 4 s. lab.py score gave the same numbers as the author's smoke: tab_mlp_rz_ft_SMOKE avg 0.635 rule / 0.662 smooth, tab_dann_rz_joint_SMOKE 0.607 / 0.611, tab_coral_o60_proto_SMOKE 0.524 / 0.542. I then deleted all _SMOKE preds, scores and cache/tab_nn_SMOKE. The full run was not started.\n\nBudget: the full run will not fit the brief's 60-minute target. With rg already dropped for res and coral it is 40 methods (the author's list is correct). The author's estimate of about 240 min (range 150-330) is consistent with the smoke timings: selection is about 40 min, final training about 90/180/275 min if the chosen epoch count is 15/30/45, and heads about 40-50 min (joint is about 2.5-6 s per seed per draw). The fallback `--seeds 1` gives about 100-120 min but changes the spec, since method names stay the same and the seed count is recorded only in meta. Resident RAM in the full run should stay around 1.1-1.6 GB, under the cap.\n\nShared-machine note: three other Python processes (about 0.9-1.3 GB each) were running during my test, so wall times will vary.\n\nMethod list: tab_{mlp,dann}_{rz,rg,o60}_{zs,ft,joint,proto} and tab_{res,coral}_{rz,o60}_{zs,ft,joint,proto}." |
| }, |
| { |
| "verdict": "clean", |
| "leakage_findings": [ |
| "No leakage found. metric.py never mentions _ya, age_class, weighted_mean_year or held_out. Amsterdam labels are read only at line 461, as lab.shot_labels(d, lab.shots_for(d, n, r)).", |
| "tpn (head_tpn) uses only the shot labels plus soft assignments of the non-shot Amsterdam rows. No hidden labels go in, and the cap w = min(1, 5*n_c/mass) is implemented as specified.", |
| "ftproj (head_ftproj) trains only on the shots' cached activations plus Madrid episodes, which use Madrid labels. Nothing carries from one draw to the next: Wt/bt start from torch.tensor(W0), a copy of the cached source weights that is never mutated. The shared Madrid sampler gets a fresh rng each draw (seed+10*rep+1000*n), so results do not depend on resume order.", |
| "The only data-driven knob is training length. It is picked from the proto F1 on the Madrid pseudo-target (madrid_split, rng 7, Madrid labels) and saved to {enc}_select.json and into meta. protoda's CORAL target in the selection run is the unlabelled pseudo-target, not Amsterdam. Every head knob is a fixed constant. Nothing is chosen from an Amsterdam score." |
| ], |
| "correctness_findings": [ |
| "robustz is applied separately per city: Xm and Xa in the main run, and src and pseudo-target separately in selection. CL2N centres each city on its own mean: cl2n is called separately on Em and Ea, and on the pseudo-target in selection.", |
| "Class indexing is consistent. Labels 1..4 map to 0..3 via y-1, ysh-1 and the lab.CLASSES order in the Sampler, episodes, means and kNN. Every head returns (25992, 4) in table order, with rows summing to 1 (checked).", |
| "The encoder stays frozen where it should. embed_H calls model.eval(), so BN uses running stats and dropout is off. ftproj trains only the 256x64 W and b on those cached eval-mode activations.", |
| "Cross-block episodes are disjoint. One random block half is drawn per episode and shared by all classes; support comes from one half, queries from the other. The fallback to any pixel (which breaks disjointness) never triggers on Madrid: I measured 0 of 8000 class-episodes, since each class covers 90-122 blocks.", |
| "Seeds are averaged with np.mean(Ps, 0) for pn, maha, tpn and knn. ftproj uses seed 0 only, and meta.seeds says so.", |
| "Resume keeps smoke and full runs apart: pred names, select json and encoder npz all carry the _SMOKE suffix. The encoder cache is keyed by (encoder, seed, chosen length, smoke).", |
| "Minor (not fixed): select json and npz are not keyed by CAND or other encoder config, so they go stale if the script is edited. Clear reports/nnlab/cache/metric after any config change.", |
| "Minor (by design, not a bug): in protoda the separate train-mode forward of the Amsterdam batch also updates the BN running stats, so the eval-time BN stats are a Madrid/Amsterdam blend. This uses only unlabelled data, so it is allowed.", |
| "No bug found in lab.py." |
| ], |
| "fixes_applied": [], |
| "full_run_command": "cd && NNLAB_THREADS=3 python tools/nnlab/metric.py 2>&1 | tee reports/nnlab/logs/metric.log ; python tools/nnlab/lab.py score met_supcon_pn met_supcon_maha met_supcon_tpn met_supcon_knn met_supcon_ftproj met_proto_pn met_proto_maha met_proto_tpn met_proto_knn met_proto_ftproj met_protoda_pn met_protoda_maha met_protoda_tpn met_protoda_knn met_protoda_ftproj met_cos_pn met_cos_maha met_cos_tpn met_cos_knn met_cos_ftproj", |
| "est_full_minutes": 58, |
| "notes": "I changed nothing. I re-ran the smoke test anyway (NNLAB_THREADS=2, 131 s, on a CPU shared with 4 other python jobs) and it passed on every code path. Scoring met_cos_ftproj_SMOKE and met_protoda_tpn_SMOKE with lab.py score also worked. I then deleted all met_*_SMOKE preds and scores and the smoke caches in reports/nnlab/cache/metric, which is now empty.\n\nFine-tuning the metric is covered, as the user asked: the ftproj head fine-tunes the encoder's projection layer on the shots, with a Madrid episode and an L2-SP penalty.\n\nHead cost per seed, measured: pn 0.03 s, maha 0.2-1.3 s, tpn 0.4 s, knn up to 0.9 s, ftproj about 2.7 s once warm (9.7 s on the very first call). That is about 5-6 s per draw per encoder, so about 9-10 min for heads over 100 draw-encoder units.\n\nEncoder training measured 0.10-0.13 s/step for proto, protoda and cos, and 0.13-0.25 s/step for supcon. If every encoder picks its longest candidate, the selection run plus 2 retrains cost about 4.5-4.8k steps each, roughly 35-45 min. My estimate for the full run is 45-65 min at 3 threads, central about 58, so it may go slightly over 60 when the machine is loaded.\n\nRAM: peak about 1 GB in my estimate. Raw plus robust-z Madrid is about 240 MB; the robustz float64 temporaries are short-lived; 2 seeds of cached H are about 210 MB; the kNN similarity matrix is up to 166 MB. That is well under 2.5 GB.\n\nReminder: --only 'proto' also matches protoda." |
| }, |
| { |
| "verdict": "fixed", |
| "leakage_findings": [ |
| "None found. Amsterdam labels are read only in main(), as lab.shot_labels(d, lab.shots_for(d, n, rep)) for the current draw. d['_ya'] is never touched. The Madrid check uses only Madrid labels (d['ym']).", |
| "No state carries over between draws. lpft() rebuilds the network each draw from the cached state_dict with make_net(state), deep-copies it per variant, and builds new optimisers and generators. The base state dict is never modified in place.", |
| "No knob is chosen from Amsterdam scores. STEPS, the learning rates, replay, L2-SP and entropy weights are fixed by the brief. epochs=15 and lam=10 come from tab_nn's Madrid-only selection.json. The Madrid check is logged and stored but not used to pick anything." |
| ], |
| "correctness_findings": [ |
| "Variant confound, now fixed: both variants drew from one torch.Generator. In the _ent variant the extra Amsterdam-batch draws shifted the replay batch indices on every later step, so lpft and lpft_ent differed by more than the entropy term. The entropy batch now has its own generator (seed + 7777777), so both variants see the same replay batches.", |
| "Smoke-mode threads, now fixed: in --smoke the base nets were trained with NNLAB_BASE_THREADS=3, which breaks the NNLAB_THREADS=2 smoke rule. The thread override only exists to reproduce tab_nn's cache, and the smoke base is never compared with it. In smoke mode BASE_THREADS now equals lab.THREADS. The full run is unchanged.", |
| "I checked the cached base myself. Loading reports/nnlab/cache/lpft/dann_rz_s0.pt and embedding gives Za max abs diff 1.43e-06 and Zm 3.3e-06 against emb_dann_rz_s0.npz, and b0 matches exactly, so it is the right net and the right epoch count. Every module is in eval mode.", |
| "BatchNorm: net.eval() is set on the copy before fine-tuning and net.train() is never called, so running stats stay frozen and dropout stays off. logits_of uses no_grad on the eval-mode net. BN affine parameters are trained and included in L2-SP, which matches the spec.", |
| "Labels and columns line up: Ys = ys - 1, the replay targets are arange(4) for CLASSES 1..4, and finetune_last_layer gets labels 1..4. X_pred = Xa in full, so P is (25992, 4) in feature-table order. Scaling is per city via tab_nn.city_inputs (robustz on Madrid and Amsterdam separately). The Madrid check transforms the source and pseudo-target parts separately, as tab_nn's selection does.", |
| "The L2-SP anchor is theta_LP: it is cloned after the LP head is copied in. The domain head is frozen and receives zero gradient (GradReverse with lam=0). The entropy term is the information-maximisation loss as specified: mean entropy minus entropy of the mean prediction.", |
| "Resume works per variant via lab.have_pred. Each variant has its own deterministic generators, so running the variants separately with --only gives the same draws.", |
| "Minor, left as is: the shot CE is a weighted mean, not a sum. With n shots per class every class weight is equal, so this is plain mean CE. The spec does not say sum." |
| ], |
| "fixes_applied": [ |
| "tools/nnlab/lpft.py lpft(): added g_ent = torch.Generator().manual_seed(seed + 7_777_777) and switched the entropy-batch randint to it, so the replay batches are identical across variants.", |
| "tools/nnlab/lpft.py main(): under --smoke, set the global BASE_THREADS = lab.THREADS." |
| ], |
| "full_run_command": "cd && NNLAB_THREADS=3 python tools/nnlab/lpft.py", |
| "est_full_minutes": 3, |
| "notes": "I re-ran the smoke test after the fixes with NNLAB_THREADS=2: 24 s wall time in total. Bases were trained at 2 threads, and each draw took 2.0 s at n=5 and 2.9 s at n=200 for both variants together. lab.py score worked for tab_dann_rz_lpft_SMOKE and tab_dann_rz_lpft_ent_SMOKE. I then deleted their preds, their scores and cache/lpft_SMOKE. The pre-existing lgbm_*_SMOKE dirs are not mine and were left alone.\n\nMethods are tab_dann_rz_lpft and tab_dann_rz_lpft_ent. The Amsterdam base (reports/nnlab/cache/lpft/dann_rz_s0.pt) is already cached and verified. The full run trains only the Madrid-check source base, about 50 s at 3 threads, then runs 9 check draws and 25 protocol draws at about 2.5-3 s each, so about 3 min in total at 3 threads. There are no predictions under reports/nnlab/preds yet, so the full run starts clean.\n\nScript: tools/nnlab/lpft.py" |
| }, |
| { |
| "verdict": "fixed", |
| "leakage_findings": [ |
| "No Amsterdam score is used to pick any setting. Shot labels are read only through lab.shot_labels(d, lab.shots_for(d, n, rep)) for the current draw. A fresh LGBMClassifier is built for every draw (random_state fixed), so no state carries over between draws. TOPK=120 and shot weight 100 are fixed by the spec. The protoda length L400 comes from protoda_select.json, which was chosen on the Madrid pseudo-target.", |
| "Minor rule-1 issue (fixed): the script loaded data with W.load_data(), which reads Amsterdam's age_class column into wd['y_ams']. That column was never used, but the spec says never to read Amsterdam age_class. The script now reads the same parquet files with columns= set to the features (plus px/py keys for Amsterdam, plus age_class for Madrid only). Values, dtype and order are unchanged: the base check still gives a 2.98e-08 max diff.", |
| "Process issue in the author's cleanup (not in the script): they deleted with the broad glob reports/nnlab/preds/*_SMOKE, which could have removed another agent's smoke outputs. I deleted only lgbm_{feda,emb,w100}_SMOKE preds and scores." |
| ], |
| "correctness_findings": [ |
| "Baseline match: the robust-z per city, Amsterdam stats from all 25,992 rows, weights of 1 for Madrid and 30 for shots times balanced_class_weights(y)[y-1], params {**LGBM_PARAMS, **RETRAIN_OVERRIDES} with n_jobs=lab.THREADS, and W.predict on the scaled matrix all match W.train_retrained and baselines.py. lgbm_base_check at 400 rounds on draw (5,0) with 2 threads: max abs diff 2.98e-08, argmax agreement 100%. PASS.", |
| "lgbm_w100 changes only the shot weight. lgbm_emb appends unscaled [DANN Zm|Za (64), l2n(H @ W.T + b) of protoda_s0_L400 (64)] for 521 columns. The formula matches metric.py's Enc.forward (proj(body)) with l2n. The cache arrays are (76263,64)/(25992,64) and (.,256) in feature-table order, and the fit uses Ea[shot] for the shot rows.", |
| "lgbm_feda: the column order [X(393), Madrid copy(120), Amsterdam copy(120)] and the NaN placement match at fit time (Madrid rows [x,x,NaN], shots [x,NaN,x]) and at predict time (all Amsterdam rows [x,NaN,x]). The top-120 come from zero_shot_lgbm.txt gain importance (393 features asserted) and are sorted by index.", |
| "Labels and columns: y-1 goes into LightGBM and predict_proba columns 0..3 map to classes 1..4. Predictions cover all 25,992 rows in table order. I added an assert that the Amsterdam px/py keys equal lab's d['pa']. Resume uses lab.have_pred per draw.", |
| "BUDGET (fixed): the smoke run's peak RAM measured 2.68 GB, above the 2.5 GB limit. I computed the per-city robust-z once in main instead of on every draw; it is label-free and doesn't depend on the draw, so results are bit-identical. I also built the feda matrices in one preallocated array instead of nested hstack/vstack. Peak is now 2.28 GB. The author's '~100 s smoke' and '1.5-2 GB' claims were unmeasured or wrong: their smoke took 167 s and peaked at 2.68 GB." |
| ], |
| "fixes_applied": [ |
| "tools/nnlab/lgbm_var.py: replaced W.load_data() with pd.read_parquet(W.DATA/..., columns=feats(+PID for Amsterdam, +age_class for Madrid)), so Amsterdam labels are never loaded. Added an assert that the Amsterdam row order equals lab's pa.", |
| "tools/nnlab/lgbm_var.py: the per-city robust-z (W.robustz_fit/W.robustz) is now done once in main. fit_predict takes the scaled matrices. Peak RAM went from 2.68 GB to 2.28 GB, about 3 s are saved per fit, and the output is identical (base check still 2.98e-08).", |
| "tools/nnlab/lgbm_var.py: the feda train and predict matrices are built in one preallocated NaN array. The layout is the same and there are fewer temporaries.", |
| "Re-ran the smoke test with NNLAB_THREADS=2: 150 s wall, base check PASS. Scored all three _SMOKE methods with lab.py score (rule n5/n200: feda 0.640/0.722, emb 0.571/0.666, w100 0.593/0.724 at 40 trees). Then deleted only these three methods' _SMOKE preds dirs and score files." |
| ], |
| "full_run_command": "cd && NNLAB_THREADS=2 python tools/nnlab/lgbm_var.py (methods: lgbm_feda, lgbm_emb, lgbm_w100; 25 draws each; resumable; optional --only lgbm_feda|lgbm_emb|lgbm_w100 to split across sessions)", |
| "est_full_minutes": 100, |
| "notes": "I recommend NNLAB_THREADS=2, because that is the setting I measured. Full 400-round fit plus predict at 2 threads: lgbm_feda 96 s, lgbm_emb 81 s, lgbm_w100 about 64 s (the base-check fit). Over 25 draws that is about 6,000 s, or roughly 100 min for all 75 fits. I did not measure 3 threads; I would guess about 70-80 min. Peak RAM at 2 threads is 2.28 GB, measured with psutil peak_wset over the whole smoke run. The full run was not started. My timing script is scratchpad/time_full.py; it saves nothing. The only file modified is tools/nnlab/lgbm_var.py." |
| }, |
| { |
| "verdict": "reproduces-after-fix", |
| "evidence": "Both untested draws reproduce exactly, and so do all the members. Shot order is the one real problem: it changes the LightGBM member by much more than floating-point noise.\n(2) Draws the author did not test, with NNLAB_THREADS=2, about 174 s each:\n- n=50, rep=3, spatial: macro F1 rule_smooth = 0.725269. The stored ens_e4__lp.json entry is 0.7252687, a difference of +0.00e+00. max |proba - stored ens_e4__lp| = 2.98e-08, the float32 rounding of the stored file. Every member (zs_xgb_dann, lgbm_upw, met_protoda_maha, met_cos_maha, tab_dann_rz_ft) and the pre-LP ensemble differ by 0.0.\n- n=200, rep=4, --no-spatial: macro F1 rule = 0.719744. The stored ens_e4.json entry is 0.7197439, a difference of +0.00e+00. max |proba - stored ens_e4| = 0.0, and all members differ by 0.0.\n(3) Shuffled shot order (n=25, rep=2, permutation from rng 7, spatial): macro F1 = 0.705238 against 0.705882 stored, a drop of 0.0006. max |proba - stored ens_e4__lp| = 0.0209. Per member: lgbm_upw 0.331, met_protoda_maha 1.3e-06, met_cos_maha 2.1e-06, tab_dann_rz_ft 0.0, zs 0.0. So the metric heads, tab head and LP only show rounding noise, but LightGBM depends on order. The cause is LGBM_PARAMS subsample=0.8 with subsample_freq=1, which bags rows by position, so the same shots in a different order give a different fit. lab.shots_for does not return sorted indices (draw_shots joins rng.choice output class by class), so no canonical order can match the stored files.\nAfter the fix: two different permutations of the n=5, rep=0 shots with sort_shots=True and spatial=False give max |dproba| = 0.0 and identical F1 (0.650188 for both).", |
| "leakage_findings": [ |
| "No leakage found. The only Amsterdam labels model_create() sees are the shot_labels passed in. lab.load does hold d['_ya'] in memory, but the only code that reads it is the CLI, through lab.shots_for / lab.shot_labels, and score_only(). The float64 LightGBM features are read from parquet as feature columns only, without age_class.", |
| "Label-free transductive inputs, as allowed by the protocol: the Amsterdam robust-z statistics inside W.train_retrained, the CL2N centring of the metric embeddings, the cached union graph (spatial r=3 plus feature kNN; union_graph builds it from features and coordinates only), and lab.smooth over the non-shot rows.", |
| "metric.py is imported with sys.argv swapped out. Its import-time work (lab.load, robustz) uses no Amsterdam labels. select_length and encoder_cache only read caches here, because model_create asserts that the json and npz files exist first. SFX is '' because no --smoke is passed, so the paths checked are the right ones.", |
| "reports/nnlab/preds is read only inside main()'s comparison block, through lab.pred_path. reports/nnlab/scores is only read to print the stored F1.", |
| "keep comes from the shot indices alone (setdiff1d). predict() smooths and decides on every non-shot row at once, so the only thing that depends on which rows are scored is the batch-level W.decide plug-in, which is allowed." |
| ], |
| "fixes_applied": [ |
| "tools/nnlab/model_create.py: model_create() takes a new argument sort_shots=False. With True it sorts (shot, label) pairs by row index (stable argsort) before fitting, so the method no longer depends on input order. The default False keeps the caller's order, so the stored per-draw files are still reproduced bit for bit. The docstring now describes the order dependence with the measured numbers.", |
| "The same function now asserts that there are no duplicate shot indices; a duplicate would silently double-weight that shot in LightGBM and in the heads.", |
| "Nothing else was changed. No other tools/nnlab file was touched, and nothing was written to reports/nnlab/preds or reports/nnlab/scores. The test scripts are in the scratchpad: scratchpad/shuf.py and scratchpad/sort.py." |
| ], |
| "notes": "Recommendation for the organiser-facing modelCreate wrapper: call model_create(..., sort_shots=True) so the output does not depend on the order the samples arrive in. The cost is that lgbm_upw no longer matches the stored files bit for bit, and the expected F1 change is of LightGBM-seed size (about 1e-3). Keep the default when reproducing stored numbers. The stored-file reproduction checks above all used the default path; with sort_shots=True I only checked permutation invariance, on n=5, rep=0. Not checked: whether sort_shots=True changes the averaged F1 over all draws. The whole order dependence comes from winners.train_retrained bagging rows by position. The stored lab results have the same property, because they were made in lab.shots_for order. The author's other claims hold: about 175 s per call with 2 threads, LightGBM with n_jobs=2 gives exactly the stored lgbm_upw, and no encoder or backbone was retrained." |
| } |
| ] |
