GNN4Colliders / docs /usage.md
ho22joshua's picture
feat: evaluation subsample, Slurm wrappers for the study and label-efficiency plots
e71731f
|
Raw History Blame
13.5 kB

Using GNN4Colliders

GNN4Colliders trains event classifiers on reconstructed collider objects:

ROOT files --prepare--> event cache (one row per object) --> graph or token batches --> model

Install

Python 3.14 and PyTorch 2.14 (CUDA 13), managed by uv:

uv sync --dev          # includes pytest, ruff and the ONNX packages
uv sync --dev --extra tabpfn   # also TabPFN (configs/gamgam_tabpfn.yaml)
uv sync --dev --extra tabicl   # also TabICL (configs/gamgam_tabicl.yaml)
uv sync --dev --extra tabdpt   # also TabDPT (configs/gamgam_tabdpt.yaml)
uv sync --dev --extra plots    # matplotlib, for scripts/plot_label_efficiency.py
uv run pytest          # about 2 minutes; GPU tests run when CUDA is available
uv run pytest --device mps   # the fixture runs on an Apple GPU

Commands

Every command takes an experiment file and optional dotted.key=value overrides (values are YAML):

uv run gnn4colliders prepare  configs/gamgam.yaml   # ROOT -> data.cache
uv run gnn4colliders train    configs/gamgam.yaml   # -> run_dir/{best,last}.pt
uv run gnn4colliders evaluate configs/gamgam.yaml   # -> run_dir/test_metrics.json
uv run gnn4colliders predict  configs/gamgam.yaml   # -> run_dir/test_predictions.npz
uv run gnn4colliders export   configs/gamgam.yaml   # -> run_dir/model.onnx
uv run gnn4colliders probe    configs/gamgam_jepa_finetune.yaml \
    model.pretrained=outputs/delphes_jepa/last.pt data.train_events=1000  # -> run_dir/test_probe.json

train writes, in run_dir (default outputs/<name>):

  • config.yaml: the resolved configuration;
  • history.jsonl: one line per epoch with the loss and metrics of the training and validation splits, the learning rate and the epoch time;
  • best.pt: the epoch with the best validation train.best_metric (AUC by default), last.pt: the latest epoch (resume with train.resume=<run_dir>/last.pt), and epoch_NNNN.pt every train.save_every epochs.

probe fits k-NN and linear classifiers to the frozen event embeddings of the encoder of model.pretrained (JEPA, contrastive or supervised; random without) on data.train_events training events and reports their metrics on split, to compare pretraining without fine-tuning.

data.eval_events=20000 evaluates (and validates during training, predicts and probes) only that many random validation or test events, seeded by data.seed: AUCs to about 0.003, in minutes, for quick runs.

evaluate, predict and export use checkpoint (default best.pt, else last.pt) on split (default test); output overrides the output file. Predictions hold the logits, scores, predicted class, label, weight, fold, source file and entry of every event; gnn4colliders.inference.write_root_scores adds scores to a copy of a ROOT tree.

Experiments

File Task
configs/gamgam.yaml ATLAS Open Data GamGam, 5 Higgs production modes, from scratch (the paper baseline and main benchmark)
configs/gamgam_finetune.yaml the same task fine-tuned from the pretrained multiclass model
configs/gamgam_transformer.yaml the same task with a transformer with pair attention bias (model.pair_bias=false for a plain transformer; same event cache)
configs/gamgam_deep_sets.yaml the same task with Deep Sets, the baseline without object interactions (same event cache)
configs/tth_cp_even_odd.yaml CP-even versus CP-odd ttH (Delphes, binary; local files)
configs/delphes_pretraining.yaml 12-class pretraining on the Delphes samples
configs/delphes_jepa.yaml JEPA pretraining of the transformer encoder on the same samples, without labels (same event cache)
configs/delphes_jepa_edge.yaml the same for the edge network (fine-tune with configs/gamgam_finetune.yaml model.pretrained=outputs/delphes_jepa_edge/last.pt)
configs/delphes_contrastive.yaml contrastive (SimCLR) pretraining of the transformer encoder on the same samples (model.backbone=edge_network for the edge network)
configs/delphes_masked.yaml masked object modelling: JEPA predicting hidden objects' kinematics
configs/gamgam_jepa_finetune.yaml GamGam with the JEPA-pretrained encoder (model.pretrained=null: from scratch); also takes contrastive and masked-object encoders, and probe
configs/gamgam_tabpfn.yaml GamGam classified in context by TabPFN-3.5, no training (evaluate only)
configs/gamgam_tabicl.yaml the same by TabICLv2 (BSD-3 weights, downloaded from Hugging Face on first use)
configs/gamgam_tabdpt.yaml the same by TabDPT (Apache-2.0 weights; model.tabdpt_context_size retrieves each event's context)
configs/delphes_pfn.yaml meta-training of our own PFN on tasks drawn from the Delphes samples (same event cache)
configs/gamgam_pfn.yaml GamGam classified in context by that PFN, no training (evaluate only)
configs/delphes_tabpfn.yaml meta-training of TabPFN-3.5 (LoRA by default) on the same tasks; evaluate with configs/gamgam_tabpfn.yaml checkpoint=outputs/delphes_tabpfn/best.pt
configs/delphes_tabicl.yaml the same for TabICL; evaluate with configs/gamgam_tabicl.yaml checkpoint=outputs/delphes_tabicl/best.pt

Each file documents its keys; src/gnn4colliders/config.py lists every key with its default. Common overrides:

train.fast=true                 # torch.compile + CUDA graphs (edge network) + bfloat16 + TF32
train.seed=3 data.seed=3        # initialization and shuffling seeds
train.max_epochs=100
model.pretrained=Pretrained_GNN/multiclass_pretrained_model_12/model_epoch_71.pt
model.freeze_backbone=true      # train only the new classifier
data.files=[{path: my.root, label: 0}] data.hf_repo=null   # local inputs

Data

Large inputs belong on scratch storage, not in the repository: set HF_HOME (where Hugging Face downloads go) and pass data.cache=..., e.g. on Perlmutter HF_HOME=$SCRATCH/.cache/huggingface and data.cache=$SCRATCH/gnn4colliders/delphes/events.pt (the 105M Delphes events take 52 GB of ROOT files and a 21 GB cache).

data.files lists the ROOT files and their class labels (or a label branch name). With data.hf_repo the paths are inside that Hugging Face dataset at data.hf_revision and are downloaded once and checked against sha256 and size_bytes.

Each object becomes a row of pt, eta, phi, energy, btag, charge, node_type built from feature_branches (one branch or constant per object type, CALC_E = pt cosh(eta), NODE_TYPE = object-type index) and multiplied by scales. derived_branches select objects or threshold tagger scores before that (the GamGam file splits lep_* into electrons and muons). The fold of an event is fold_var, modulo fold_modulus when set, and train_folds/validation_folds/test_folds select the splits. Events are unweighted unless weight_var names a weight branch.

In-context models and label efficiency

data.train_events=N keeps N training events, an equal number per class (smaller N give subsets of larger ones). Supervised models train on them; the in-context models (model.name: tabpfn or pfn) use them as the labelled context and are only evaluated:

uv run gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation data.train_events=1000
uv run gnn4colliders train configs/delphes_pfn.yaml          # meta-training
uv run gnn4colliders evaluate configs/gamgam_pfn.yaml split=validation data.train_events=1000
uv run gnn4colliders train configs/gamgam.yaml data.train_events=1000   # supervised reference

scripts/label_efficiency.py runs such a sweep over several N and configs (restartable) and prints validation AUC and seconds per run; the metrics JSON records seconds (setup, inference) and, on CUDA, peak_memory_gib:

uv run python scripts/label_efficiency.py configs/gamgam_tabpfn.yaml configs/gamgam.yaml \
    --events 100 1000 10000 50000

TabPFN-2.5 and later weights are non-commercial (research and internal benchmarking are allowed). tabpfn downloads them after a one-time login at https://ux.priorlabs.ai (token in ~/.cache/tabpfn/auth_token or TABPFN_TOKEN), but a checkpoint already in its cache directory loads without the login, so on compute nodes download it from Hugging Face:

export TABPFN_MODEL_CACHE_DIR=$SCRATCH/.cache/tabpfn   # default ~/Library/Caches/tabpfn on macOS
uvx --from huggingface_hub hf download Prior-Labs/tabpfn_3_5 config.json \
    tabpfn-v3.5-20260909.safetensors tabpfn-v3.5-20260909_multiclass.safetensors \
    --local-dir $TABPFN_MODEL_CACHE_DIR

model.tabpfn_checkpoint=tabpfn-v3.5-20260909_multiclass.safetensors selects the multiclass checkpoint. model.tabpfn_version=v2 needs no login.

TabDPT retrieval (model.tabdpt_context_size) on macOS needs OMP_NUM_THREADS=1: faiss and torch load separate OpenMP runtimes.

Tabular models can use a pretrained encoder's event embeddings instead of or beside the rows: model.tabular_inputs=embeddings (or both) with model.pretrained=outputs/delphes_jepa/last.pt and the encoder's sizes (those of configs/delphes_jepa.yaml: model.hid_size=128 model.n_layers=4 model.n_heads=8 model.ffn_size=512 model.pair_bias=true).

TabPFN and TabICL classify the same table rows (model.slots objects of each type). Two options add physics they cannot learn from a few events: model.relative_phi=true measures every azimuth from the leading object's, and model.pair_objects=4 appends the invariant mass and Delta R of every pair of the four leading objects (12 columns). Compare with and without, one run name per variant:

uv run python scripts/label_efficiency.py configs/gamgam_tabicl.yaml --events 100 1000 \
    --override name=gamgam_tabicl_physics model.relative_phi=true model.pair_objects=4

TabPFN-3.5 can also be meta-trained on the Delphes tasks, like our PFN. By default only LoRA updates of its in-context blocks train (model.lora_rank, 8; model.lora_rank=0 trains all 219M weights), and checkpoints hold only those (33 MB with the optimizer state, against 2.6 GB for all weights). With a checkpoint, TabPFN is evaluated with its own forward pass, not the classifier:

uv run gnn4colliders train configs/delphes_tabpfn.yaml
uv run gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation \
    data.train_events=1000 checkpoint=outputs/delphes_tabpfn/best.pt

Activations are recomputed in the backward pass. On a 32 GB Apple M2 Max one task of 2048 context events and 256 queries takes about 4.5 s and 16 GiB per step with all weights trained, so the Mac is for development; meta-train on a GPU node.

Fine-tuning

Pretrained_GNN/ holds the pretrained models of the paper, the final epoch of each: multiclass_pretrained_model_12/model_epoch_71.pt and multilabel_pretrained_model_41/model_epoch_44.pt (with their training configuration and log). With model.pretrained the edge network is loaded from that checkpoint and gets a new model.out_size classifier; model.hid_size, n_layers and n_proc_steps must match it (64, 4, 4). A JEPA checkpoint (configs/delphes_jepa*.yaml) loads the same way: its context encoder initializes the edge network or the transformer encoder.

GPUs, Perlmutter and DDP

train.device: auto uses CUDA, else an Apple GPU (MPS), else the CPU. MPS is for development on fixtures and small runs (no float64: metrics are computed on the CPU; no DDP, CUDA graphs or TF32).

On a single GPU, train.fast=true is about 3x faster than float32 for the GamGam model (see design.md). Set OMP_NUM_THREADS to the CPU allocation and, on Perlmutter compute nodes, UV_CACHE_DIR=$SCRATCH/.cache/uv (uv cannot lock its home-directory cache there).

# One training per seed, each followed by test-split evaluation:
for seed in 1 2 3 4 5; do
  sbatch -A <account> --export=ALL,SEED=$seed scripts/train.sh train.fast=true
done
uv run python scripts/summarize_ensemble.py outputs/gamgam/seed*   # mean +- std

The study of in-context models and pretraining has one wrapper per kind of job, each on one GPU (the header of each script lists its variables):

# Label-efficiency sweep of released in-context models and supervised references:
sbatch -A <account> --export=ALL,CONFIGS="configs/gamgam_tabpfn.yaml configs/gamgam_tabicl.yaml configs/gamgam.yaml" \
  scripts/label_efficiency.sh
# Meta-training, then GamGam in context with the meta-trained model:
sbatch -A <account> --export=ALL,CONFIG=configs/delphes_tabicl.yaml,EVAL_CONFIG=configs/gamgam_tabicl.yaml \
  scripts/meta_train.sh task.prior_classes=[1,6,7,8,9,10,11]
# Label-free pretraining, then k-NN and linear probes on GamGam:
sbatch -A <account> --export=ALL,CONFIG=configs/delphes_contrastive.yaml scripts/pretrain.sh
# Curves and tables of every finished run:
uv run --extra plots python scripts/plot_label_efficiency.py --sweeps outputs/label_efficiency \
  --probes outputs/delphes_jepa outputs/delphes_contrastive outputs/delphes_masked

Under torchrun with more than one process, training is data parallel: data.batch_size is per process, training events are sharded over ranks (padded to equal length) and metrics are computed over all ranks' events. CUDA graphs need a single process (so train.fast=true does too; DDP is tested in float32 eager). scripts/train_ddp.sh launches all GPUs of one or more nodes.