File size: 13,793 Bytes
0cef53c
 
 
 
 
 
 
 
 
 
 
 
 
 
9402518
8dff74c
 
e71731f
0cef53c
8dff74c
0cef53c
 
 
 
 
 
 
 
 
 
 
 
 
8dff74c
 
0cef53c
 
 
 
 
 
 
 
 
 
 
 
8dff74c
 
 
 
 
e71731f
 
 
 
0cef53c
 
 
 
 
 
 
 
 
 
 
 
df3eb75
 
0cef53c
 
9b40da9
722850b
8dff74c
 
 
9402518
8dff74c
 
9402518
 
8dff74c
 
0cef53c
 
 
 
 
f43b1e5
0cef53c
 
 
 
 
 
 
 
 
9b40da9
 
 
 
 
 
0cef53c
 
 
 
 
 
 
 
 
 
 
 
 
 
9402518
 
 
 
 
 
 
 
 
 
 
 
 
 
8dff74c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9402518
0cef53c
 
973bea8
 
 
 
0cef53c
722850b
 
 
0cef53c
 
 
8dff74c
 
 
 
0cef53c
 
 
 
 
 
 
 
2f04f23
0cef53c
 
 
 
e71731f
 
 
 
 
 
 
 
 
 
 
 
0821800
 
 
 
e71731f
 
 
 
 
0cef53c
 
 
 
2f04f23
0cef53c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
# Using GNN4Colliders

GNN4Colliders trains event classifiers on reconstructed collider objects:

```text
ROOT files --prepare--> event cache (one row per object) --> graph or token batches --> model
```

## Install

Python 3.14 and PyTorch 2.14 (CUDA 13), managed by [uv](https://docs.astral.sh/uv/):

```bash
uv sync --dev          # includes pytest, ruff and the ONNX packages
uv sync --dev --extra tabpfn   # also TabPFN (configs/gamgam_tabpfn.yaml)
uv sync --dev --extra tabicl   # also TabICL (configs/gamgam_tabicl.yaml)
uv sync --dev --extra tabdpt   # also TabDPT (configs/gamgam_tabdpt.yaml)
uv sync --dev --extra plots    # matplotlib, for scripts/plot_label_efficiency.py
uv run pytest          # about 2 minutes; GPU tests run when CUDA is available
uv run pytest --device mps   # the fixture runs on an Apple GPU
```

## Commands

Every command takes an experiment file and optional `dotted.key=value`
overrides (values are YAML):

```bash
uv run gnn4colliders prepare  configs/gamgam.yaml   # ROOT -> data.cache
uv run gnn4colliders train    configs/gamgam.yaml   # -> run_dir/{best,last}.pt
uv run gnn4colliders evaluate configs/gamgam.yaml   # -> run_dir/test_metrics.json
uv run gnn4colliders predict  configs/gamgam.yaml   # -> run_dir/test_predictions.npz
uv run gnn4colliders export   configs/gamgam.yaml   # -> run_dir/model.onnx
uv run gnn4colliders probe    configs/gamgam_jepa_finetune.yaml \
    model.pretrained=outputs/delphes_jepa/last.pt data.train_events=1000  # -> run_dir/test_probe.json
```

`train` writes, in `run_dir` (default `outputs/<name>`):

* `config.yaml`: the resolved configuration;
* `history.jsonl`: one line per epoch with the loss and metrics of the
  training and validation splits, the learning rate and the epoch time;
* `best.pt`: the epoch with the best validation `train.best_metric` (AUC by
  default), `last.pt`: the latest epoch (resume with
  `train.resume=<run_dir>/last.pt`), and `epoch_NNNN.pt` every
  `train.save_every` epochs.

`probe` fits k-NN and linear classifiers to the frozen event embeddings of
the encoder of `model.pretrained` (JEPA, contrastive or supervised; random
without) on `data.train_events` training events and reports their metrics
on `split`, to compare pretraining without fine-tuning.

`data.eval_events=20000` evaluates (and validates during training,
predicts and probes) only that many random validation or test events,
seeded by `data.seed`: AUCs to about 0.003, in minutes, for quick runs.

`evaluate`, `predict` and `export` use `checkpoint` (default `best.pt`, else
`last.pt`) on `split` (default `test`); `output` overrides the output file.
Predictions hold the logits, scores, predicted class, label, weight, fold,
source file and entry of every event; `gnn4colliders.inference.write_root_scores`
adds scores to a copy of a ROOT tree.

## Experiments

| File | Task |
| --- | --- |
| `configs/gamgam.yaml` | ATLAS Open Data GamGam, 5 Higgs production modes, from scratch (the paper baseline and main benchmark) |
| `configs/gamgam_finetune.yaml` | the same task fine-tuned from the pretrained multiclass model |
| `configs/gamgam_transformer.yaml` | the same task with a transformer with pair attention bias (`model.pair_bias=false` for a plain transformer; same event cache) |
| `configs/gamgam_deep_sets.yaml` | the same task with Deep Sets, the baseline without object interactions (same event cache) |
| `configs/tth_cp_even_odd.yaml` | CP-even versus CP-odd ttH (Delphes, binary; local files) |
| `configs/delphes_pretraining.yaml` | 12-class pretraining on the Delphes samples |
| `configs/delphes_jepa.yaml` | JEPA pretraining of the transformer encoder on the same samples, without labels (same event cache) |
| `configs/delphes_jepa_edge.yaml` | the same for the edge network (fine-tune with `configs/gamgam_finetune.yaml model.pretrained=outputs/delphes_jepa_edge/last.pt`) |
| `configs/delphes_contrastive.yaml` | contrastive (SimCLR) pretraining of the transformer encoder on the same samples (`model.backbone=edge_network` for the edge network) |
| `configs/delphes_masked.yaml` | masked object modelling: JEPA predicting hidden objects' kinematics |
| `configs/gamgam_jepa_finetune.yaml` | GamGam with the JEPA-pretrained encoder (`model.pretrained=null`: from scratch); also takes contrastive and masked-object encoders, and `probe` |
| `configs/gamgam_tabpfn.yaml` | GamGam classified in context by TabPFN-3.5, no training (`evaluate` only) |
| `configs/gamgam_tabicl.yaml` | the same by TabICLv2 (BSD-3 weights, downloaded from Hugging Face on first use) |
| `configs/gamgam_tabdpt.yaml` | the same by TabDPT (Apache-2.0 weights; `model.tabdpt_context_size` retrieves each event's context) |
| `configs/delphes_pfn.yaml` | meta-training of our own PFN on tasks drawn from the Delphes samples (same event cache) |
| `configs/gamgam_pfn.yaml` | GamGam classified in context by that PFN, no training (`evaluate` only) |
| `configs/delphes_tabpfn.yaml` | meta-training of TabPFN-3.5 (LoRA by default) on the same tasks; evaluate with `configs/gamgam_tabpfn.yaml checkpoint=outputs/delphes_tabpfn/best.pt` |
| `configs/delphes_tabicl.yaml` | the same for TabICL; evaluate with `configs/gamgam_tabicl.yaml checkpoint=outputs/delphes_tabicl/best.pt` |

Each file documents its keys; `src/gnn4colliders/config.py` lists every key
with its default. Common overrides:

```bash
train.fast=true                 # torch.compile + CUDA graphs (edge network) + bfloat16 + TF32
train.seed=3 data.seed=3        # initialization and shuffling seeds
train.max_epochs=100
model.pretrained=Pretrained_GNN/multiclass_pretrained_model_12/model_epoch_71.pt
model.freeze_backbone=true      # train only the new classifier
data.files=[{path: my.root, label: 0}] data.hf_repo=null   # local inputs
```

### Data

Large inputs belong on scratch storage, not in the repository: set
`HF_HOME` (where Hugging Face downloads go) and pass `data.cache=...`, e.g.
on Perlmutter `HF_HOME=$SCRATCH/.cache/huggingface` and
`data.cache=$SCRATCH/gnn4colliders/delphes/events.pt` (the 105M Delphes
events take 52 GB of ROOT files and a 21 GB cache).

`data.files` lists the ROOT files and their class labels (or a label branch
name). With `data.hf_repo` the paths are inside that Hugging Face dataset at
`data.hf_revision` and are downloaded once and checked against `sha256` and
`size_bytes`.

Each object becomes a row of `pt, eta, phi, energy, btag, charge, node_type`
built from `feature_branches` (one branch or constant per object type,
`CALC_E` = pt cosh(eta), `NODE_TYPE` = object-type index) and multiplied by
`scales`. `derived_branches` select objects or threshold tagger scores before
that (the GamGam file splits `lep_*` into electrons and muons). The fold of an
event is `fold_var`, modulo `fold_modulus` when set, and
`train_folds`/`validation_folds`/`test_folds` select the splits. Events are
unweighted unless `weight_var` names a weight branch.

### In-context models and label efficiency

`data.train_events=N` keeps N training events, an equal number per class
(smaller N give subsets of larger ones). Supervised models train on them;
the in-context models (`model.name: tabpfn` or `pfn`) use them as the
labelled context and are only evaluated:

```bash
uv run gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation data.train_events=1000
uv run gnn4colliders train configs/delphes_pfn.yaml          # meta-training
uv run gnn4colliders evaluate configs/gamgam_pfn.yaml split=validation data.train_events=1000
uv run gnn4colliders train configs/gamgam.yaml data.train_events=1000   # supervised reference
```

`scripts/label_efficiency.py` runs such a sweep over several `N` and
configs (restartable) and prints validation AUC and seconds per run; the
metrics JSON records `seconds` (setup, inference) and, on CUDA,
`peak_memory_gib`:

```bash
uv run python scripts/label_efficiency.py configs/gamgam_tabpfn.yaml configs/gamgam.yaml \
    --events 100 1000 10000 50000
```

TabPFN-2.5 and later weights are non-commercial (research and internal
benchmarking are allowed). tabpfn downloads them after a one-time login at
https://ux.priorlabs.ai (token in `~/.cache/tabpfn/auth_token` or
`TABPFN_TOKEN`), but a checkpoint already in its cache directory loads
without the login, so on compute nodes download it from Hugging Face:

```bash
export TABPFN_MODEL_CACHE_DIR=$SCRATCH/.cache/tabpfn   # default ~/Library/Caches/tabpfn on macOS
uvx --from huggingface_hub hf download Prior-Labs/tabpfn_3_5 config.json \
    tabpfn-v3.5-20260909.safetensors tabpfn-v3.5-20260909_multiclass.safetensors \
    --local-dir $TABPFN_MODEL_CACHE_DIR
```

`model.tabpfn_checkpoint=tabpfn-v3.5-20260909_multiclass.safetensors`
selects the multiclass checkpoint. `model.tabpfn_version=v2` needs no login.

TabDPT retrieval (`model.tabdpt_context_size`) on macOS needs
`OMP_NUM_THREADS=1`: faiss and torch load separate OpenMP runtimes.

Tabular models can use a pretrained encoder's event embeddings instead of
or beside the rows: `model.tabular_inputs=embeddings` (or `both`) with
`model.pretrained=outputs/delphes_jepa/last.pt` and the encoder's sizes
(those of `configs/delphes_jepa.yaml`: `model.hid_size=128 model.n_layers=4
model.n_heads=8 model.ffn_size=512 model.pair_bias=true`).

TabPFN and TabICL classify the same table rows (`model.slots` objects of
each type). Two options add physics they cannot learn from a few events:
`model.relative_phi=true` measures every azimuth from the leading object's,
and `model.pair_objects=4` appends the invariant mass and Delta R of every
pair of the four leading objects (12 columns). Compare with and without,
one run name per variant:

```bash
uv run python scripts/label_efficiency.py configs/gamgam_tabicl.yaml --events 100 1000 \
    --override name=gamgam_tabicl_physics model.relative_phi=true model.pair_objects=4
```

TabPFN-3.5 can also be meta-trained on the Delphes tasks, like our PFN.
By default only LoRA updates of its in-context blocks train
(`model.lora_rank`, 8; `model.lora_rank=0` trains all 219M weights), and
checkpoints hold only those (33 MB with the optimizer state, against
2.6 GB for all weights). With a `checkpoint`, TabPFN is
evaluated with its own forward pass, not the classifier:

```bash
uv run gnn4colliders train configs/delphes_tabpfn.yaml
uv run gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation \
    data.train_events=1000 checkpoint=outputs/delphes_tabpfn/best.pt
```

Activations are recomputed in the backward pass. On a 32 GB Apple M2 Max
one task of 2048 context events and 256 queries takes about 4.5 s and
16 GiB per step with all weights trained, so the Mac is for development;
meta-train on a GPU node.

### Fine-tuning

`Pretrained_GNN/` holds the pretrained models of the paper, the final
epoch of each: `multiclass_pretrained_model_12/model_epoch_71.pt` and
`multilabel_pretrained_model_41/model_epoch_44.pt` (with their training
configuration and log). With `model.pretrained` the edge network is loaded from
that checkpoint and gets a new `model.out_size` classifier; `model.hid_size`,
`n_layers` and `n_proc_steps` must match it (64, 4, 4). A JEPA checkpoint
(`configs/delphes_jepa*.yaml`) loads the same way: its context encoder
initializes the edge network or the transformer encoder.

## GPUs, Perlmutter and DDP

`train.device: auto` uses CUDA, else an Apple GPU (MPS), else the CPU. MPS
is for development on fixtures and small runs (no float64: metrics are
computed on the CPU; no DDP, CUDA graphs or TF32).

On a single GPU, `train.fast=true` is about 3x faster than float32 for the
GamGam model (see [design.md](design.md)). Set `OMP_NUM_THREADS` to the CPU
allocation and, on Perlmutter compute nodes, `UV_CACHE_DIR=$SCRATCH/.cache/uv`
(uv cannot lock its home-directory cache there).

```bash
# One training per seed, each followed by test-split evaluation:
for seed in 1 2 3 4 5; do
  sbatch -A <account> --export=ALL,SEED=$seed scripts/train.sh train.fast=true
done
uv run python scripts/summarize_ensemble.py outputs/gamgam/seed*   # mean +- std
```

The study of in-context models and pretraining has one wrapper per kind of
job, each on one GPU (the header of each script lists its variables):

```bash
# Label-efficiency sweep of released in-context models and supervised references:
sbatch -A <account> --export=ALL,CONFIGS="configs/gamgam_tabpfn.yaml configs/gamgam_tabicl.yaml configs/gamgam.yaml" \
  scripts/label_efficiency.sh
# Meta-training, then GamGam in context with the meta-trained model:
sbatch -A <account> --export=ALL,CONFIG=configs/delphes_tabicl.yaml,EVAL_CONFIG=configs/gamgam_tabicl.yaml \
  scripts/meta_train.sh task.prior_classes=[1,6,7,8,9,10,11]
# Label-free pretraining, then k-NN and linear probes on GamGam:
sbatch -A <account> --export=ALL,CONFIG=configs/delphes_contrastive.yaml scripts/pretrain.sh
# Linear CKA between the pretrained encoders, layer by layer (a login node is enough):
uv run python scripts/compare_encoders.py configs/gamgam_jepa_finetune.yaml --events 5000 \
  --encoders random=random jepa=outputs/delphes_jepa/last.pt \
  contrastive=outputs/delphes_contrastive/last.pt masked=outputs/delphes_masked/last.pt
# Curves and tables of every finished run:
uv run --extra plots python scripts/plot_label_efficiency.py --sweeps outputs/label_efficiency \
  --probes outputs/delphes_jepa outputs/delphes_contrastive outputs/delphes_masked
```

Under `torchrun` with more than one process, training is data parallel:
`data.batch_size` is per process, training events are sharded over ranks
(padded to equal length) and metrics are computed over all ranks' events.
CUDA graphs need a single process (so `train.fast=true` does too; DDP is
tested in float32 eager). `scripts/train_ddp.sh` launches all GPUs of
one or more nodes.