GNN4Colliders / docs /usage.md
ho22joshua's picture
Claude Opus 5.5
feat: linear CKA between object encoders
0821800
|
Raw History Blame Contribute Delete
13.8 kB
# Using GNN4Colliders
GNN4Colliders trains event classifiers on reconstructed collider objects:
```text
ROOT files --prepare--> event cache (one row per object) --> graph or token batches --> model
```
## Install
Python 3.14 and PyTorch 2.14 (CUDA 13), managed by [uv](https://docs.astral.sh/uv/):
```bash
uv sync --dev # includes pytest, ruff and the ONNX packages
uv sync --dev --extra tabpfn # also TabPFN (configs/gamgam_tabpfn.yaml)
uv sync --dev --extra tabicl # also TabICL (configs/gamgam_tabicl.yaml)
uv sync --dev --extra tabdpt # also TabDPT (configs/gamgam_tabdpt.yaml)
uv sync --dev --extra plots # matplotlib, for scripts/plot_label_efficiency.py
uv run pytest # about 2 minutes; GPU tests run when CUDA is available
uv run pytest --device mps # the fixture runs on an Apple GPU
```
## Commands
Every command takes an experiment file and optional `dotted.key=value`
overrides (values are YAML):
```bash
uv run gnn4colliders prepare configs/gamgam.yaml # ROOT -> data.cache
uv run gnn4colliders train configs/gamgam.yaml # -> run_dir/{best,last}.pt
uv run gnn4colliders evaluate configs/gamgam.yaml # -> run_dir/test_metrics.json
uv run gnn4colliders predict configs/gamgam.yaml # -> run_dir/test_predictions.npz
uv run gnn4colliders export configs/gamgam.yaml # -> run_dir/model.onnx
uv run gnn4colliders probe configs/gamgam_jepa_finetune.yaml \
model.pretrained=outputs/delphes_jepa/last.pt data.train_events=1000 # -> run_dir/test_probe.json
```
`train` writes, in `run_dir` (default `outputs/<name>`):
* `config.yaml`: the resolved configuration;
* `history.jsonl`: one line per epoch with the loss and metrics of the
training and validation splits, the learning rate and the epoch time;
* `best.pt`: the epoch with the best validation `train.best_metric` (AUC by
default), `last.pt`: the latest epoch (resume with
`train.resume=<run_dir>/last.pt`), and `epoch_NNNN.pt` every
`train.save_every` epochs.
`probe` fits k-NN and linear classifiers to the frozen event embeddings of
the encoder of `model.pretrained` (JEPA, contrastive or supervised; random
without) on `data.train_events` training events and reports their metrics
on `split`, to compare pretraining without fine-tuning.
`data.eval_events=20000` evaluates (and validates during training,
predicts and probes) only that many random validation or test events,
seeded by `data.seed`: AUCs to about 0.003, in minutes, for quick runs.
`evaluate`, `predict` and `export` use `checkpoint` (default `best.pt`, else
`last.pt`) on `split` (default `test`); `output` overrides the output file.
Predictions hold the logits, scores, predicted class, label, weight, fold,
source file and entry of every event; `gnn4colliders.inference.write_root_scores`
adds scores to a copy of a ROOT tree.
## Experiments
| File | Task |
| --- | --- |
| `configs/gamgam.yaml` | ATLAS Open Data GamGam, 5 Higgs production modes, from scratch (the paper baseline and main benchmark) |
| `configs/gamgam_finetune.yaml` | the same task fine-tuned from the pretrained multiclass model |
| `configs/gamgam_transformer.yaml` | the same task with a transformer with pair attention bias (`model.pair_bias=false` for a plain transformer; same event cache) |
| `configs/gamgam_deep_sets.yaml` | the same task with Deep Sets, the baseline without object interactions (same event cache) |
| `configs/tth_cp_even_odd.yaml` | CP-even versus CP-odd ttH (Delphes, binary; local files) |
| `configs/delphes_pretraining.yaml` | 12-class pretraining on the Delphes samples |
| `configs/delphes_jepa.yaml` | JEPA pretraining of the transformer encoder on the same samples, without labels (same event cache) |
| `configs/delphes_jepa_edge.yaml` | the same for the edge network (fine-tune with `configs/gamgam_finetune.yaml model.pretrained=outputs/delphes_jepa_edge/last.pt`) |
| `configs/delphes_contrastive.yaml` | contrastive (SimCLR) pretraining of the transformer encoder on the same samples (`model.backbone=edge_network` for the edge network) |
| `configs/delphes_masked.yaml` | masked object modelling: JEPA predicting hidden objects' kinematics |
| `configs/gamgam_jepa_finetune.yaml` | GamGam with the JEPA-pretrained encoder (`model.pretrained=null`: from scratch); also takes contrastive and masked-object encoders, and `probe` |
| `configs/gamgam_tabpfn.yaml` | GamGam classified in context by TabPFN-3.5, no training (`evaluate` only) |
| `configs/gamgam_tabicl.yaml` | the same by TabICLv2 (BSD-3 weights, downloaded from Hugging Face on first use) |
| `configs/gamgam_tabdpt.yaml` | the same by TabDPT (Apache-2.0 weights; `model.tabdpt_context_size` retrieves each event's context) |
| `configs/delphes_pfn.yaml` | meta-training of our own PFN on tasks drawn from the Delphes samples (same event cache) |
| `configs/gamgam_pfn.yaml` | GamGam classified in context by that PFN, no training (`evaluate` only) |
| `configs/delphes_tabpfn.yaml` | meta-training of TabPFN-3.5 (LoRA by default) on the same tasks; evaluate with `configs/gamgam_tabpfn.yaml checkpoint=outputs/delphes_tabpfn/best.pt` |
| `configs/delphes_tabicl.yaml` | the same for TabICL; evaluate with `configs/gamgam_tabicl.yaml checkpoint=outputs/delphes_tabicl/best.pt` |
Each file documents its keys; `src/gnn4colliders/config.py` lists every key
with its default. Common overrides:
```bash
train.fast=true # torch.compile + CUDA graphs (edge network) + bfloat16 + TF32
train.seed=3 data.seed=3 # initialization and shuffling seeds
train.max_epochs=100
model.pretrained=Pretrained_GNN/multiclass_pretrained_model_12/model_epoch_71.pt
model.freeze_backbone=true # train only the new classifier
data.files=[{path: my.root, label: 0}] data.hf_repo=null # local inputs
```
### Data
Large inputs belong on scratch storage, not in the repository: set
`HF_HOME` (where Hugging Face downloads go) and pass `data.cache=...`, e.g.
on Perlmutter `HF_HOME=$SCRATCH/.cache/huggingface` and
`data.cache=$SCRATCH/gnn4colliders/delphes/events.pt` (the 105M Delphes
events take 52 GB of ROOT files and a 21 GB cache).
`data.files` lists the ROOT files and their class labels (or a label branch
name). With `data.hf_repo` the paths are inside that Hugging Face dataset at
`data.hf_revision` and are downloaded once and checked against `sha256` and
`size_bytes`.
Each object becomes a row of `pt, eta, phi, energy, btag, charge, node_type`
built from `feature_branches` (one branch or constant per object type,
`CALC_E` = pt cosh(eta), `NODE_TYPE` = object-type index) and multiplied by
`scales`. `derived_branches` select objects or threshold tagger scores before
that (the GamGam file splits `lep_*` into electrons and muons). The fold of an
event is `fold_var`, modulo `fold_modulus` when set, and
`train_folds`/`validation_folds`/`test_folds` select the splits. Events are
unweighted unless `weight_var` names a weight branch.
### In-context models and label efficiency
`data.train_events=N` keeps N training events, an equal number per class
(smaller N give subsets of larger ones). Supervised models train on them;
the in-context models (`model.name: tabpfn` or `pfn`) use them as the
labelled context and are only evaluated:
```bash
uv run gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation data.train_events=1000
uv run gnn4colliders train configs/delphes_pfn.yaml # meta-training
uv run gnn4colliders evaluate configs/gamgam_pfn.yaml split=validation data.train_events=1000
uv run gnn4colliders train configs/gamgam.yaml data.train_events=1000 # supervised reference
```
`scripts/label_efficiency.py` runs such a sweep over several `N` and
configs (restartable) and prints validation AUC and seconds per run; the
metrics JSON records `seconds` (setup, inference) and, on CUDA,
`peak_memory_gib`:
```bash
uv run python scripts/label_efficiency.py configs/gamgam_tabpfn.yaml configs/gamgam.yaml \
--events 100 1000 10000 50000
```
TabPFN-2.5 and later weights are non-commercial (research and internal
benchmarking are allowed). tabpfn downloads them after a one-time login at
https://ux.priorlabs.ai (token in `~/.cache/tabpfn/auth_token` or
`TABPFN_TOKEN`), but a checkpoint already in its cache directory loads
without the login, so on compute nodes download it from Hugging Face:
```bash
export TABPFN_MODEL_CACHE_DIR=$SCRATCH/.cache/tabpfn # default ~/Library/Caches/tabpfn on macOS
uvx --from huggingface_hub hf download Prior-Labs/tabpfn_3_5 config.json \
tabpfn-v3.5-20260909.safetensors tabpfn-v3.5-20260909_multiclass.safetensors \
--local-dir $TABPFN_MODEL_CACHE_DIR
```
`model.tabpfn_checkpoint=tabpfn-v3.5-20260909_multiclass.safetensors`
selects the multiclass checkpoint. `model.tabpfn_version=v2` needs no login.
TabDPT retrieval (`model.tabdpt_context_size`) on macOS needs
`OMP_NUM_THREADS=1`: faiss and torch load separate OpenMP runtimes.
Tabular models can use a pretrained encoder's event embeddings instead of
or beside the rows: `model.tabular_inputs=embeddings` (or `both`) with
`model.pretrained=outputs/delphes_jepa/last.pt` and the encoder's sizes
(those of `configs/delphes_jepa.yaml`: `model.hid_size=128 model.n_layers=4
model.n_heads=8 model.ffn_size=512 model.pair_bias=true`).
TabPFN and TabICL classify the same table rows (`model.slots` objects of
each type). Two options add physics they cannot learn from a few events:
`model.relative_phi=true` measures every azimuth from the leading object's,
and `model.pair_objects=4` appends the invariant mass and Delta R of every
pair of the four leading objects (12 columns). Compare with and without,
one run name per variant:
```bash
uv run python scripts/label_efficiency.py configs/gamgam_tabicl.yaml --events 100 1000 \
--override name=gamgam_tabicl_physics model.relative_phi=true model.pair_objects=4
```
TabPFN-3.5 can also be meta-trained on the Delphes tasks, like our PFN.
By default only LoRA updates of its in-context blocks train
(`model.lora_rank`, 8; `model.lora_rank=0` trains all 219M weights), and
checkpoints hold only those (33 MB with the optimizer state, against
2.6 GB for all weights). With a `checkpoint`, TabPFN is
evaluated with its own forward pass, not the classifier:
```bash
uv run gnn4colliders train configs/delphes_tabpfn.yaml
uv run gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation \
data.train_events=1000 checkpoint=outputs/delphes_tabpfn/best.pt
```
Activations are recomputed in the backward pass. On a 32 GB Apple M2 Max
one task of 2048 context events and 256 queries takes about 4.5 s and
16 GiB per step with all weights trained, so the Mac is for development;
meta-train on a GPU node.
### Fine-tuning
`Pretrained_GNN/` holds the pretrained models of the paper, the final
epoch of each: `multiclass_pretrained_model_12/model_epoch_71.pt` and
`multilabel_pretrained_model_41/model_epoch_44.pt` (with their training
configuration and log). With `model.pretrained` the edge network is loaded from
that checkpoint and gets a new `model.out_size` classifier; `model.hid_size`,
`n_layers` and `n_proc_steps` must match it (64, 4, 4). A JEPA checkpoint
(`configs/delphes_jepa*.yaml`) loads the same way: its context encoder
initializes the edge network or the transformer encoder.
## GPUs, Perlmutter and DDP
`train.device: auto` uses CUDA, else an Apple GPU (MPS), else the CPU. MPS
is for development on fixtures and small runs (no float64: metrics are
computed on the CPU; no DDP, CUDA graphs or TF32).
On a single GPU, `train.fast=true` is about 3x faster than float32 for the
GamGam model (see [design.md](design.md)). Set `OMP_NUM_THREADS` to the CPU
allocation and, on Perlmutter compute nodes, `UV_CACHE_DIR=$SCRATCH/.cache/uv`
(uv cannot lock its home-directory cache there).
```bash
# One training per seed, each followed by test-split evaluation:
for seed in 1 2 3 4 5; do
sbatch -A <account> --export=ALL,SEED=$seed scripts/train.sh train.fast=true
done
uv run python scripts/summarize_ensemble.py outputs/gamgam/seed* # mean +- std
```
The study of in-context models and pretraining has one wrapper per kind of
job, each on one GPU (the header of each script lists its variables):
```bash
# Label-efficiency sweep of released in-context models and supervised references:
sbatch -A <account> --export=ALL,CONFIGS="configs/gamgam_tabpfn.yaml configs/gamgam_tabicl.yaml configs/gamgam.yaml" \
scripts/label_efficiency.sh
# Meta-training, then GamGam in context with the meta-trained model:
sbatch -A <account> --export=ALL,CONFIG=configs/delphes_tabicl.yaml,EVAL_CONFIG=configs/gamgam_tabicl.yaml \
scripts/meta_train.sh task.prior_classes=[1,6,7,8,9,10,11]
# Label-free pretraining, then k-NN and linear probes on GamGam:
sbatch -A <account> --export=ALL,CONFIG=configs/delphes_contrastive.yaml scripts/pretrain.sh
# Linear CKA between the pretrained encoders, layer by layer (a login node is enough):
uv run python scripts/compare_encoders.py configs/gamgam_jepa_finetune.yaml --events 5000 \
--encoders random=random jepa=outputs/delphes_jepa/last.pt \
contrastive=outputs/delphes_contrastive/last.pt masked=outputs/delphes_masked/last.pt
# Curves and tables of every finished run:
uv run --extra plots python scripts/plot_label_efficiency.py --sweeps outputs/label_efficiency \
--probes outputs/delphes_jepa outputs/delphes_contrastive outputs/delphes_masked
```
Under `torchrun` with more than one process, training is data parallel:
`data.batch_size` is per process, training events are sharded over ranks
(padded to equal length) and metrics are computed over all ranks' events.
CUDA graphs need a single process (so `train.fast=true` does too; DDP is
tested in float32 eager). `scripts/train_ddp.sh` launches all GPUs of
one or more nodes.