| # PANDA — reproducibility recipe | |
| **PANDA** (Pan-tissue Adversarial Normalized Domain-invariant Anchored MLP) is a | |
| compact prototype-anchored MLP classifier for scRNA-seq cell identity across skin, | |
| hematopoietic, and pancreatic tissues, trained under a composite loss (supervised- | |
| contrastive + VICReg + sub-center angular prototype-InfoNCE + gradient-reversal | |
| dataset/depth adversary + HSIC depth-decorrelation). Two | |
| input variants ship out of the box: **PANDA-PCA** (`PCA(50) -> trunk`) and | |
| **PANDA-Marker** (`[PCA(50) || marker_expr] -> trunk`); the marker channel's gain | |
| is system-dependent, largest on pancreas (+4.2% CV accuracy). | |
| This README is a complete recipe to reproduce every result in `PAPER.tex` from a | |
| clean clone. All commands are copy-pasteable and run from the repo root; scripts | |
| resolve the repo root automatically (see `PANDA_ROOT` below). | |
| Repository layout: | |
| ``` | |
| panda/ model + losses + panda/markers.yaml | |
| scripts/pan_skin/ skin pipeline: download -> corpus -> train -> CV -> zero-shot | |
| scripts/pancreas/ pancreas pipeline (same shape) | |
| scripts/hematopoiesis/ HSC pipeline (same shape) | |
| scripts/common/ system-agnostic train / CV / zero-shot drivers | |
| scripts/analysis/ downstream discovery + interpretability | |
| scripts/figures/ paper + supplement figure builders | |
| data/corpus/{sys}/ downloaded + harmonized data | |
| data/raw/ per-dataset raw counts (not in git) | |
| checkpoints/{sys}/{variant}/panda_final.pt | |
| discovery/{sys}/{variant}/*.json,*.csv all quantitative artefacts | |
| figures/ fig1_..fig6, PANDA_supplement.pdf, biology/*.pdf | |
| ``` | |
| --- | |
| ## 1. Requirements | |
| - **Python**: 3.9+ (project is tested on 3.10). | |
| - **CUDA**: PyTorch 2.6.0 wheels — CUDA 12.1/12.4 runtime works. | |
| - **GPU**: any single CUDA GPU works — the trainer is single-device (batch 256, | |
| a compact MLP; a consumer GPU trains a checkpoint in well under an hour). | |
| - **Disk**: ~400 GB (raw GEO tars + harmonized corpora + checkpoints). | |
| - **RAM**: ~64 GB (the pancreas HVG builder peaks near ~40 GB). | |
| Pinned runtime dependencies (from `pyproject.toml` / `requirements.txt`): | |
| ``` | |
| torch==2.6.0 scanpy==1.11.5 anndata>=0.11.4 | |
| harmonypy==0.2.0 numpy>=1.24,<3.0 scipy>=1.10 | |
| scikit-learn>=1.2 pandas>=1.5 matplotlib>=3.7 | |
| seaborn>=0.12 umap-learn>=0.5 pyyaml>=6.0 | |
| tqdm>=4.65 leidenalg>=0.10 pynndescent>=0.5 | |
| scikit-misc>=0.5 | |
| ``` | |
| Optional extras: `viz` (plotly), `dev` (pytest, ruff, mypy). | |
| Install: | |
| ```bash | |
| git clone <this-repo> panda && cd panda | |
| pip install -e . | |
| # or, editable dev install: | |
| pip install -e ".[dev]" | |
| ``` | |
| ### Repository root resolution (`PANDA_ROOT`) | |
| Every script resolves the repository root as | |
| `os.environ.get("PANDA_ROOT", <two directories above the script>)`, so a clone at | |
| any path works out of the box. Set `PANDA_ROOT` explicitly only if you run scripts | |
| from a copied/vendored location or want data/checkpoints rooted elsewhere: | |
| ```bash | |
| export PANDA_ROOT=/path/to/clone # optional | |
| ``` | |
| Data and checkpoints are expected at `$PANDA_ROOT/data` and `$PANDA_ROOT/checkpoints` | |
| (symlink/junction these to the Hugging Face mirror download if you keep it separate). | |
| ### 1.1 LD_LIBRARY_PATH prefix (required for every PyTorch invocation) | |
| PyTorch 2.6 sparse ops load `libcusparseLt.so.0` which sits under the pip-installed | |
| `nvidia-cusparselt-cu12` package, and `scanpy` needs a modern `libstdc++`. Both | |
| paths must be exported **at the shell level, before Python starts**: | |
| ```bash | |
| export LD_LIBRARY_PATH="$(python -c "import site,os; print(os.path.join(site.getsitepackages()[0],'nvidia','cusparselt','lib'))"):$LD_LIBRARY_PATH" | |
| ``` | |
| If you also have a conda env that ships a newer `libstdc++`, prepend it: | |
| ```bash | |
| # example — path is machine-specific; drop it if your system libstdc++ is >= 3.4.30 | |
| export LD_LIBRARY_PATH="/home/bcheng/.conda/pkgs/libstdcxx-15.2.0-h39759b7_7/lib:$LD_LIBRARY_PATH" | |
| ``` | |
| Every `bash scripts/*/run_all.sh` driver applies the same export automatically. | |
| --- | |
| ## 2. Data acquisition | |
| Every URL below is a public GEO/ArrayExpress FTP link. Raw data is **not** | |
| committed — it must be re-downloaded before anything else runs. Corpus builders | |
| expect files at `data/corpus/{system}/tier_{a,b,c,v2}/`. | |
| ### Pan-skin (6 studies, 45,387 cells) | |
| | Study | GEO | Role | | |
| |---|---|---| | |
| | Sulic 2023 (E14.5 dorsal) | GSE212673 | anchor + held-out zero-shot | | |
| | Dingwall 2024 (En1-cKO) | GSE220977 | discovery target (paired with Aldrich GSE214695) | | |
| | Belote 2021 (human melanocyte) | GSE151091 | melanocyte anchor + held-out zero-shot | | |
| | Haensel/Annusver 2020 | GSE142471 | adult homeostasis + wound | | |
| | Joost 2016 | GSE67602 | Smart-seq2 platform anchor | | |
| | Sennett 2015 (bulk RNA) | GSE70288 | placode/dermal-condensate marker reference | | |
| | Han MCA 2018 (neonatal skin) | GSE108097 | Microwell-seq low-depth anchor | | |
| | Merkel 2022 | GSE201447 | touch dome / volar biology | | |
| | Aldrich 2023 (paired with Dingwall) | GSE214695 | En1-cKO snRNA-seq | | |
| ```bash | |
| bash scripts/pan_skin/01_download_tier_a.sh # Aldrich, Ge/Gupta, Joost, Haensel | |
| bash scripts/pan_skin/02_download_tier_b.sh # MCA, WIHN, Ge/Fuchs, Merkel | |
| bash scripts/pan_skin/03_download_tier_c.sh # Sennett, Tie, Wiedemann (bulk + human) | |
| # Dingwall / Sulic / Belote must be placed in data/raw/ manually — see repo notes | |
| ``` | |
| ### Pan-hematopoietic (3 studies used in the paper, 192,833 cells) | |
| | Study | GEO | Role | | |
| |---|---|---| | |
| | Weinreb LARRY 2020 | GSE140802 | corpus anchor | | |
| | Baccin whole-BM 2020 | GSE122465 | corpus (stromal + hematopoietic) | | |
| | Tabula Muris Senis BM 2020 | GSE132042 | corpus (paper-labeled) | | |
| | Nestorowa 2016 | GSE81682 | held-out zero-shot (Smart-seq2) | | |
| | Dahlin 2018 | GSE107727 | discovery target (Kit-W41 mutant) | | |
| | Paul 2015 (auxiliary) | GSE72857 | myeloid branch reference | | |
| | Tusi 2018 (auxiliary) | GSE89754 | erythroid trajectory | | |
| ```bash | |
| bash scripts/hematopoiesis/01_download.sh # Paul, Nestorowa, Tusi, Dahlin | |
| bash scripts/hematopoiesis/02_download.sh # Baccin whole-BM, TMS bone marrow | |
| ``` | |
| ### Pan-pancreatic (6 studies, 120,611 cells) | |
| | Study | GEO | Role | | |
| |---|---|---| | |
| | Baron 2016 | GSE84133 | corpus mouse-train half + held-out mouse-test half | | |
| | Bastidas-Ponce 2019 (E15.5) | GSE132188 | corpus (endocrine progenitor time course) | | |
| | Byrnes 2018 | GSE101099 | corpus (paper-labeled subset) | | |
| | Yu 2021 | GSE139627 | corpus (paper-labeled Ngn3 lineage) | | |
| | Hrovatin MIA 2023 | GSE211796 | corpus (adult islet, paper-labeled) | | |
| | Veres 2019 | GSE114412 | 57,297 corpus + 12,297 held-out slice | | |
| ```bash | |
| bash scripts/pancreas/01_download.sh # Baron, Muraro, Grun, Byrnes, Veres | |
| bash scripts/pancreas/09_download.sh # Yu Ngn3 seq-EP, MIA 4-month adult islet | |
| ``` | |
| Wall-clock: 2-6 h depending on bandwidth (GSE108097 MCA tar is ~9 GB, GSE140802 | |
| Weinreb is ~14 GB, GSE114412 Veres is ~4 GB). | |
| --- | |
| ## 3. Corpus build (per system) | |
| Each system builds a `data/corpus/{system}/harmonized/corpus.h5ad` plus a shared | |
| HVG list, per-HVG mean/std, and a fitted PCA basis. Corpus is 100% paper-labeled; | |
| every cell carries a label from its source paper's supplementary table. | |
| ### Pan-skin | |
| ```bash | |
| python scripts/pan_skin/06_build_per_dataset_h5ads.py | |
| python scripts/pan_skin/07_build_shared_hvgs_and_pca.py | |
| python scripts/pan_skin/08_assign_labels.py | |
| python scripts/pan_skin/08b_curated_label_override.py | |
| python scripts/pan_skin/10_build_corpus.py # canonical corpus.h5ad | |
| python scripts/pan_skin/93_add_belote_anchor.py # +Belote melanocyte anchor | |
| ``` | |
| Wall-clock ~10-20 min (HVG + PCA is the expensive step). | |
| ### Pan-hematopoietic | |
| ```bash | |
| python scripts/hematopoiesis/02_build_per_dataset.py | |
| python scripts/hematopoiesis/03_shared_hvgs_and_pca.py | |
| python scripts/hematopoiesis/10_build_corpus.py # canonical corpus.h5ad | |
| python scripts/hematopoiesis/11_filter_paper_only.py # enforce paper-labeled subset | |
| python scripts/hematopoiesis/09_retrain_with_nestorowa_anchor.py # optional anchor | |
| ``` | |
| Wall-clock ~15-30 min. | |
| ### Pan-pancreatic | |
| ```bash | |
| python scripts/pancreas/02_build_per_dataset.py | |
| python scripts/pancreas/03_shared_hvgs_and_pca.py | |
| python scripts/pancreas/04_assign_labels.py | |
| python scripts/pancreas/11_build_corpus.py # canonical corpus.h5ad | |
| python scripts/pancreas/08_add_baron_split.py # 943-cell Baron test-half | |
| ``` | |
| Wall-clock ~30-60 min (peak ~40 GB RAM on the union HVG step). | |
| Also generate the held-out labeled slices used for zero-shot: | |
| ```bash | |
| python scripts/common/generate_missing_holdouts.py | |
| ``` | |
| writes `data/corpus/hematopoiesis/held_out_labeled/nestorowa_GSE81682_test.h5ad` | |
| and `data/corpus/pan_skin/held_out_labeled/sulic_GSE212673_test.h5ad`. | |
| --- | |
| ## 4. Training | |
| The **canonical trainer** is system-agnostic. It reads | |
| `data/corpus/{system}/harmonized/corpus.h5ad` and writes | |
| `checkpoints/{system}/{variant}/panda_final.pt`. | |
| ```bash | |
| # 6 checkpoints total (3 systems x 2 variants). ~30-60 min each on 1x A100. | |
| python -m scripts.common.train_panda pan_skin --variant pca --epochs 8 | |
| python -m scripts.common.train_panda pan_skin --variant marker --epochs 8 | |
| python -m scripts.common.train_panda hematopoiesis --variant pca --epochs 8 | |
| python -m scripts.common.train_panda hematopoiesis --variant marker --epochs 8 | |
| python -m scripts.common.train_panda pancreas --variant pca --epochs 8 | |
| python -m scripts.common.train_panda pancreas --variant marker --epochs 8 | |
| ``` | |
| Legacy per-system entry points also exist and are functionally equivalent for | |
| skin/HSC/pancreas single-variant training: | |
| `scripts/pan_skin/20_train_panda.py`, `scripts/hematopoiesis/05_train_panda.py`, | |
| `scripts/pancreas/05_train_panda.py`. Prefer `scripts.common.train_panda`. | |
| --- | |
| ## 5. Held-out 5-fold cross-validation (Table 1) | |
| The paper's Table 1 CV block reads | |
| `discovery/{system}/{variant}/cv_5fold.json`. Two drivers exist: | |
| - **`scripts/common/run_cv.py`** — canonical, 5 epochs per fold, matches | |
| paper numbers (mean acc / F1 / AUROC + per-class report). | |
| - `scripts/common/cv_holdout.py` — same architecture but supports GroupKFold | |
| by dataset and a fuller 6-8 epoch curriculum; slower. | |
| Both accept `--systems` and `--variants`: | |
| ```bash | |
| # canonical 5-fold CV for all 3 systems x 2 variants | |
| python -m scripts.common.run_cv --folds 5 --epochs 5 | |
| ``` | |
| Per-system CV drivers also exist (`scripts/pan_skin/40_heldout_5fold_cv.py`, | |
| `scripts/hematopoiesis/07_heldout_5fold_cv.py`, | |
| `scripts/pancreas/07_heldout_5fold_cv.py`); they are single-variant, single- | |
| system alternatives. | |
| ### Multi-seed rigor | |
| Seed replicates use the canonical driver on the canonical corpora; non-zero seeds | |
| write `cv_5fold_seed{N}.json` next to the seed-0 `cv_5fold.json`: | |
| ```bash | |
| python -m scripts.common.run_cv --folds 5 --epochs 5 # seed 0 (canonical) | |
| python -m scripts.common.run_cv --folds 5 --epochs 5 --seed 1 | |
| python -m scripts.common.run_cv --folds 5 --epochs 5 --seed 2 | |
| ``` | |
| > **Correction (2026-08):** older `cv_5fold_seed{1,2}.json` files in the repo history | |
| > were produced on different corpus builds (different cell counts and label | |
| > vocabularies) by `cv_holdout.py` with a different curriculum; the "33/35 | |
| > fold-comparisons" claim previously derived from them was invalid and has been | |
| > withdrawn. The seeds committed now are regenerated with the commands above on | |
| > the canonical corpora (verified same n/K per configuration). Honest result: | |
| > Marker beats PCA on **34/45** fold-comparisons — pancreas 15/15 (+0.04 acc | |
| > every seed), skin 11/15 (gap ≤0.003), hematopoiesis 8/15 (parity). | |
| --- | |
| ## 6. Held-out labeled zero-shot targets (Section 5) | |
| One driver runs every zero-shot target for both variants: | |
| ```bash | |
| python -m scripts.common.run_all_zero_shot \ | |
| --systems pan_skin hematopoiesis pancreas \ | |
| --variants pca marker | |
| ``` | |
| writes `discovery/{system}/{variant}/{target}_predictions.csv` and | |
| `{target}_summary.json`. Individual per-target scripts exist for finer-grained | |
| control: | |
| | Target | Script | Populates | | |
| |---|---|---| | |
| | Baron test-half (pancreas, 943 cells) | `scripts/analysis/93_true_zero_shot_baron.py` | Table 1 Baron row + Sec 5.1 | | |
| | Nestorowa Smart-seq2 (HSC, 66 LT-HSC gated) | `scripts/analysis/94_true_zero_shot_nestorowa.py` | Sec 5.3 | | |
| | Sulic E14.5 dorsal skin (4,183 cells) | `scripts/common/run_all_zero_shot.py` (target `sulic`) | Sec 5.5 | | |
| | Belote melanocyte (6,088 cells) | `scripts/common/run_all_zero_shot.py` (target `belote`) | Sec 5.4 | | |
| | Veres held-out slice (12,297 pancreas) | `scripts/common/run_all_zero_shot.py` (target `veres`) | Sec 5.2 | | |
| | Dingwall (25,344 skin, discovery) | `scripts/pan_skin/30_zero_shot_aldrich.py` | Sec 6 | | |
| | Dahlin (61,122 HSC, discovery) | `scripts/common/run_all_zero_shot.py` (target `dahlin`) | Sec 7 | | |
| Adult-beta canonical panel validation on Veres: | |
| ```bash | |
| python scripts/analysis/95_adult_beta_validation.py | |
| # -> discovery/pancreas/marker/95_adult_beta_validation.json | |
| ``` | |
| --- | |
| ## 7. Discovery analyses | |
| Grouped by paper section. All write to `discovery/{system}/{variant}/`. | |
| ### 7.1 Section 5 (held-out labeled) marker deep-dives | |
| ```bash | |
| python scripts/analysis/90_dingwall_marker_deep_dive.py # skin -> 90_..._marker_deep_dive.csv | |
| python scripts/analysis/91_veres_marker_deep_dive.py # pancreas | |
| python scripts/analysis/92_dahlin_marker_deep_dive.py # HSC | |
| ``` | |
| ### 7.2 Section 6 — Dingwall En1-cKO (skin) | |
| ```bash | |
| python scripts/analysis/44_en1_cko_contrast.py # class-level cKO vs WT contrast | |
| python scripts/analysis/45_marker_refinement.py # per-class marker refinement | |
| python scripts/analysis/49_melanocyte_deep_dive.py # melanocyte 2x expansion | |
| python scripts/analysis/57_multiclass_pathway_analysis.py # Dingwall pathway table | |
| python scripts/analysis/57_pathway_analysis.py # symmetric 25+ module scoring, all systems | |
| python scripts/analysis/99_en1_dual_role_analysis.py # spatial repressor / local activator | |
| python scripts/analysis/106_melanoblast_neural_crest.py # Sec 6.5 MITF-axis vs NC reversion | |
| python scripts/analysis/107_dingwall_class_deg_count.py # Sec 6.7 HF-placode DEG rank | |
| # EDEN validation — three complementary lines of evidence (Sec 6.4) | |
| python scripts/analysis/98_eden_posthoc_detection.py # Line A (null) | |
| python scripts/analysis/103_replicate_dingwall_seurat_pipeline.py # Line B (Derm10 4.32x) | |
| python scripts/analysis/104_train_on_dingwall_derm_labels.py # Line C (Derm10 5.20x) | |
| # Primary EDEN (Derm2) discovery — Sec 6.4.1 | |
| python scripts/analysis/100_primary_eden_discovery.py # sub-cluster fibro predictions | |
| python scripts/analysis/101_primary_eden_derm_scoring.py # score vs Data S1C panels | |
| python scripts/analysis/105_primary_eden_full_dermal.py # on full dermal denominator | |
| # Auxiliary: variant-A Dingwall training used for scope comparison | |
| python scripts/analysis/102_train_on_dingwall_variantA.py | |
| ``` | |
| ### 7.3 Section 7 — Dahlin Kit-mutant (hematopoiesis) | |
| ```bash | |
| python scripts/analysis/66_dahlin_kit_mutant.py # class enrichment WT vs Kit-W41 | |
| python scripts/analysis/67_dahlin_within_class.py # within-class Wilcoxon DE | |
| python scripts/analysis/73_novel_populations_dahlin.py # abstain-gated novel pops | |
| python scripts/analysis/108_dahlin_lineage_metabolism.py # per-lineage OXPHOS/glycolysis | |
| ``` | |
| ### 7.4 Section 7.4 — Veres held-out (pancreas) | |
| ```bash | |
| python scripts/analysis/62_time_course_analysis.py # class fractions across LARRY days (HSC time-course template) | |
| python scripts/analysis/109_veres_mature_beta.py # Sec 5.2 adult MAFA/UCN3 quadrant | |
| python scripts/analysis/110_veres_polyhormonal_alpha.py # Sec 5.2 polyhormonal alpha cluster | |
| ``` | |
| ### 7.5 Section 8 — cross-system prototype geometry + interpretability | |
| ```bash | |
| python scripts/analysis/70_prototype_geometry.py # intra + cross-system cosine + eff-dim | |
| python scripts/analysis/72_emergent_axes.py # within-class PCA of 128-d z | |
| python scripts/analysis/80_prototype_gene_attribution.py # integrated gradients per prototype | |
| python scripts/analysis/81_counterfactual_knockouts.py # per-gene KO delta on cosine | |
| python scripts/analysis/82_gene_coattribution_modules.py # gene co-attribution modules | |
| python scripts/analysis/83_prototype_training_trajectory.py # prototype drift across curriculum | |
| python scripts/analysis/84_adversary_purification.py # test GRL adversary is at chance | |
| python scripts/analysis/85_hessian_gene_interactions.py # second-order gene pair Hessian | |
| python scripts/analysis/63_nestorowa_zero_shot.py # Nestorowa unlabeled discovery | |
| ``` | |
| --- | |
| ## 8. Figures + supplement | |
| ### Main-text figures (`figures/fig{1,2,3,4}_*.pdf`) | |
| ```bash | |
| python scripts/figures/generate_paper_figures.py | |
| # fig1_perclass_f1.pdf 3-panel per-class held-out F1 bars | |
| # fig3_dahlin_heatmap.pdf within-class module-score heatmap | |
| # fig4_veres_stage_stack.pdf Veres per-stage class fractions | |
| # (there is no fig2 in the current paper; numbering is historical) | |
| ``` | |
| Figures 5/6 (En1-cKO + Kit-W41 recap) are built by the biology page pipeline | |
| below — the standalone `regen_fig5_fig6.py` referenced in older notes is not in | |
| the current tree; use the biology pipeline instead. | |
| ### Supplement (`figures/PANDA_supplement.pdf`) | |
| ```bash | |
| python scripts/figures/build_pca_vs_marker_umaps.py # PCA vs Marker UMAPs per target | |
| python scripts/figures/build_figure_supplement.py # combined supplement PDF | |
| ``` | |
| ### Biology deep-dive supplement pages | |
| Cache UMAPs once, then build per-topic pages, then merge into the supplement: | |
| ```bash | |
| python scripts/figures/biology_00_umap_cache.py | |
| python scripts/figures/biology_01_dingwall_umap.py | |
| python scripts/figures/biology_02_primary_eden.py | |
| python scripts/figures/biology_03_melanoblast_mitf.py | |
| python scripts/figures/biology_04_dahlin_metabolism.py | |
| python scripts/figures/biology_05_dahlin_composition.py | |
| python scripts/figures/biology_06_veres_beta_quadrant.py | |
| python scripts/figures/biology_07_veres_polyhormonal.py | |
| python scripts/figures/biology_08_prototype_geometry.py | |
| python scripts/figures/biology_99_merge_supplement.py # appends into PANDA_supplement.pdf | |
| ``` | |
| Wall-clock: 20-40 min end-to-end (UMAPs dominate). | |
| --- | |
| ## 9. PDF build | |
| The paper is a self-contained LaTeX document referencing PDFs in `figures/`: | |
| ```bash | |
| cd <repo-root> | |
| pdflatex -interaction=nonstopmode PAPER.tex # first pass (writes .aux) | |
| pdflatex -interaction=nonstopmode PAPER.tex # second pass (resolves refs) | |
| ``` | |
| `bibtex` is not required — the paper uses an embedded `thebibliography`. | |
| --- | |
| ## 10. End-to-end make target | |
| The provided `Makefile` covers the canonical skin pipeline end-to-end: | |
| ```bash | |
| make install # pip install -e . | |
| make run # bash scripts/pan_skin/run_all.sh — full skin pipeline | |
| make test-heldout # scripts/pan_skin/40_heldout_5fold_cv.py | |
| make clean # remove __pycache__ + *.pyc | |
| ``` | |
| For a full three-system reproduction, chain the per-section commands above. | |
| A minimal "everything" recipe: | |
| ```bash | |
| # 1) data | |
| bash scripts/pan_skin/01_download_tier_a.sh | |
| bash scripts/pan_skin/02_download_tier_b.sh | |
| bash scripts/pan_skin/03_download_tier_c.sh | |
| bash scripts/hematopoiesis/01_download.sh | |
| bash scripts/hematopoiesis/02_download.sh | |
| bash scripts/pancreas/01_download.sh | |
| bash scripts/pancreas/09_download.sh | |
| # 2) corpora | |
| bash scripts/pan_skin/run_all.sh # includes build + train + skin CV | |
| python scripts/hematopoiesis/02_build_per_dataset.py | |
| python scripts/hematopoiesis/03_shared_hvgs_and_pca.py | |
| python scripts/hematopoiesis/10_build_corpus.py | |
| python scripts/hematopoiesis/11_filter_paper_only.py | |
| python scripts/pancreas/02_build_per_dataset.py | |
| python scripts/pancreas/03_shared_hvgs_and_pca.py | |
| python scripts/pancreas/04_assign_labels.py | |
| python scripts/pancreas/11_build_corpus.py | |
| python scripts/pancreas/08_add_baron_split.py | |
| python scripts/common/generate_missing_holdouts.py | |
| python scripts/pan_skin/93_add_belote_anchor.py | |
| # 3) all 6 training runs | |
| for sys in pan_skin hematopoiesis pancreas; do | |
| for v in pca marker; do | |
| python -m scripts.common.train_panda $sys --variant $v --epochs 8 | |
| done | |
| done | |
| # 4) CV + zero-shot | |
| python -m scripts.common.run_cv --folds 5 --epochs 5 | |
| python -m scripts.common.run_all_zero_shot | |
| # 5) discovery | |
| bash scripts/common/rerun_all_discovery.sh # drives scripts/analysis/* end-to-end | |
| python scripts/analysis/95_adult_beta_validation.py | |
| # 6) figures + PDF | |
| python scripts/figures/generate_paper_figures.py | |
| python scripts/figures/build_pca_vs_marker_umaps.py | |
| python scripts/figures/build_figure_supplement.py | |
| python scripts/figures/biology_00_umap_cache.py | |
| for i in 01 02 03 04 05 06 07 08; do | |
| python scripts/figures/biology_${i}_*.py | |
| done | |
| python scripts/figures/biology_99_merge_supplement.py | |
| pdflatex -interaction=nonstopmode PAPER.tex && pdflatex -interaction=nonstopmode PAPER.tex | |
| ``` | |
| --- | |
| ## 11. Trouble-shooting | |
| - **`libcusparseLt.so.0: cannot open shared object file`** — you forgot to | |
| export `LD_LIBRARY_PATH` **before** Python started. The pip-installed | |
| `nvidia-cusparselt-cu12` provides the library; PyTorch does not add its | |
| path to the loader search. See section 1.1. | |
| - **`GLIBCXX_3.4.30 not found`** — your system `libstdc++` is too old; prepend | |
| a newer `libstdc++.so.6`'s directory to `LD_LIBRARY_PATH`. | |
| - **Dingwall GSM -> genotype mapping** (frequent bug source): the correct map is | |
| `WT = {GSM6833478, GSM6833479, GSM6833480, GSM6833481}`, | |
| `cKO = {GSM6833482, GSM6833483}`. | |
| GSM6833480/481 are `rttaControl` (Cre-negative WT), **not** cKO. Getting this | |
| wrong flips every En1-cKO enrichment sign. | |
| - **Pancreas HVG builder OOM** — `scripts/pancreas/03_shared_hvgs_and_pca.py` | |
| peaks near 40 GB RAM on the 6-study union. Run on a node with >= 64 GB. | |
| - **`data/raw` not in git** — it is git-ignored (14+ GB of GEO tars). Rerun | |
| section 2 to repopulate. | |
| - **`stratified split failure` on rare classes** — small-support classes | |
| (< 2 members per fold) are merged into the parent `canonical_label`. If a | |
| fold still errors, check that `corpus.h5ad`'s `canonical_label` column has | |
| the expected vocabulary; the paper vocab is the union enumerated in | |
| `PAPER.tex` Sec 3. | |
| - **`n_conditions` mismatch** — `PANDAEncoder` reads it from | |
| `len(datasets)` in the checkpoint; regenerate the checkpoint if you have | |
| added/removed a dataset. | |
| - **`ContrastiveSampler` in 1-condition data** — auto-disabled when there is | |
| only one condition; no config change needed. | |
| - **DataParallel batch-size** — default `bs=256` is calibrated for 4x A100 | |
| 40 GB. Drop to `bs=64` for a single GPU or you will OOM inside the | |
| sub-center prototype attention. | |
| - **Embeddings not persisted** — after training writes the checkpoint, the 128-d | |
| projection is re-computed on demand in every downstream analysis; if you want it | |
| cached, re-save the AnnData explicitly via `adata.write_h5ad()`. | |
| - **Sulic is NOT held out of the standard corpus** — all 4,683 Sulic cells | |
| (including the 4,183-cell `held_out_labeled` slice) are inside | |
| `data/corpus/pan_skin/harmonized/corpus.h5ad`. Scoring the standard checkpoints on | |
| the Sulic slice is a train-set evaluation. Use | |
| `scripts/pan_skin/92_retrain_with_sulic_anchor.py` (500-cell anchor corpus, | |
| artefacts `97_sulic_anchor_*`) for the honest number, as the paper now does. | |
| - **Corpus rebuild from raw GEO data is currently not possible from this repo** — | |
| the per-dataset loader modules (`panda/data/*_loaders.py`) that | |
| `06_build_per_dataset_h5ads.py` / `02_build_per_dataset.py` import were never | |
| committed (only stale v2 copies exist under `archive/`). Everything from | |
| training onward is reproducible from the harmonized corpora on the Hugging Face | |
| mirror; re-deriving the corpora from GEO requires reconstructing the loaders. | |
| --- | |
| ## 12. Artefact index (from PAPER.tex Section 11) | |
| Every quantitative claim traces to one of: | |
| - **Model checkpoints**: `checkpoints/{system}/{variant}/panda_final.pt` | |
| - **Marker gene lists**: `panda/markers.yaml` | |
| - **Corpus builders**: `scripts/{pan_skin,hematopoiesis,pancreas}/` | |
| - **Unified trainer**: `scripts/common/train_panda.py` | |
| - **Unified 5-fold CV**: `scripts/common/run_cv.py` (or `cv_holdout.py`) | |
| - **Zero-shot driver**: `scripts/common/run_all_zero_shot.py` | |
| - **External label supplements**: | |
| `data/external_labels/{dingwall_supp,haensel,joost2016,mca,mia,byrnes,yu,baccin,melanocyte_anchor}/` | |
| - **CV outputs**: `discovery/{system}/{variant}/cv_5fold{,_seed1,_seed2}.json` | |
| - **Zero-shot summaries**: | |
| - `discovery/pancreas/{pca,marker}/baron_summary.json` | |
| - `discovery/pancreas/{pca,marker}/veres_summary.json` | |
| - `discovery/hematopoiesis/{pca,marker}/nestorowa_summary.json` | |
| - `discovery/pan_skin/{pca,marker}/sulic_summary.json` | |
| - `discovery/pan_skin/{pca,marker}/belote_summary.json` | |
| - **Adult-beta panel**: `discovery/pancreas/marker/95_adult_beta_validation.json` | |
| - **Pathway modules**: `discovery/{system}/marker/57_pathway_class_by_module_{padj,delta}.tsv` | |
| - **Dingwall EDEN validation**: | |
| Line A `discovery/pan_skin/marker/98_eden_summary.json`; | |
| Line B `data/processed/dingwall_replica/dingwall_replica.h5ad`, | |
| `replica_cluster_20_qc.json`, `replica_marker_matches.csv`; | |
| Line C `discovery/pan_skin/marker/104_dingwall_derm_summary.json` + prediction CSVs. | |
| - **Primary EDEN (Derm2)**: | |
| `discovery/pan_skin/marker/100_primary_eden_discovery.csv`, | |
| `100_primary_eden_summary.json`, | |
| `101_derm_identity_summary.json`, | |
| `101_derm_subcluster_scores.csv`. | |
| - **Dingwall other**: | |
| `discovery/pan_skin/marker/57_pathway_analysis.csv`, | |
| `90_dingwall_marker_deep_dive.csv`. | |
| - **Dahlin Kit-mutant**: | |
| `discovery/hematopoiesis/marker/92_dahlin_marker_deep_dive.csv`, `dahlin_summary.json`. | |
| - **Veres deep-dive**: `discovery/pancreas/marker/91_veres_marker_deep_dive.csv`. | |
| Central architecture: `panda/model.py`. Composite loss lives in the same file | |
| (`supcon_loss`, `vicreg_loss`, `hsic_biased`, `subcenter_angular_infonce`, | |
| `prototype_repulsion`) and is imported as `from panda import PANDAEncoder, ...`. | |
| ## Data mirror | |
| Full data (~195 GB corpus + raw + processed + external labels) is mirrored to Hugging Face at [bryan7264/PANDA](https://huggingface.co/bryan7264/PANDA). Fetch with: | |
| ```bash | |
| huggingface-cli download bryan7264/PANDA --local-dir . --include "data/corpus/pan_skin/**" | |
| ``` | |
| Priority folders (fetch these first for the minimum-reproducible pipeline): | |
| - `data/corpus/{pan_skin,hematopoiesis,pancreas}/harmonized/` — training corpora | |
| - `data/external_labels/` — paper-supplement label files | |
| - `checkpoints/{pan_skin,hematopoiesis,pancreas}/marker/` — trained weights | |
| Bulk (only needed to reproduce corpus builds from scratch): | |
| - `data/raw/` — GEO downloads (regenerable from `scripts/*/03_download*.sh`) | |
| - `data/corpus/tier{1,2,3}/` — pretraining tier data | |
| - `data/processed/` — intermediate build artefacts | |