GNN4Colliders / docs /design.md
ho22joshua's picture
Claude Opus 5.5
feat: TabPFN-3.5 options for the PFN, a cosine schedule and faster meta-training
71aefe8
|
Raw History Blame Contribute Delete
31 kB

Design

Modules

Module Contents
config.py the experiment dataclasses, YAML loading and overrides
data.py ROOT reading, Hugging Face download, derived branches, object features, PackedEvents and its cache
loader.py EventLoader: deterministic shuffling, DDP sharding, collation workers
graphs.py fully connected TensorGraph batches, edge features, CUDA-graph padding buffers
sequences.py padded token batches for the transformer and Deep Sets, event augmentations
models.py EdgeNetwork, FineTunedEdgeNetwork, ObjectTransformer, TransformerClassifier, DeepSets, checkpoint loading
jepa.py JEPA pretraining of the transformer or edge network encoder (encoding or kinematic targets), event embeddings
contrastive.py contrastive (SimCLR) pretraining of the same encoders
probe.py k-NN and linear probes of frozen encoders, linear CKA between them
pfn.py our prior-fitted network over events and the task prior of meta-training
tabular.py event rows and encoder embeddings for TabPFN, TabICL and TabDPT; LoRA meta-training of TabPFN and TabICL
tasks.py multiclass and binary losses and metrics, the pretraining and PFN losses
train.py Trainer, early stopping, checkpoints
cuda_graphs.py captured training steps
distributed.py torchrun process group and collectives
inference.py model outputs over a loader, prediction files
export.py ONNX export
cli.py the six commands

Data flow

prepare reads each ROOT file in chunks of 100,000 entries (in parallel with data.prepare_workers) and stores every event's objects as rows of one [objects, 7] tensor with offsets, plus per-event labels, folds, weights and source file and entry. Nothing in the cache depends on the model: graphs and token sequences are built per batch, in the loader workers.

A batch of the edge network is a set of fully connected graphs in flat edge-index form (edge_src, edge_dst, node_batch, edge_batch). Edges are directed, source-major and without self-loops except for one-node graphs, with features [deta, dphi, dR]. These conventions, the seven node columns, the MLP layout and the parameter names are those of the original ROOT-GNN, so the pretrained checkpoints in Pretrained_GNN/ load without conversion.

The transformer and Deep Sets take each event's objects as tokens [events, objects, 7], zero-padded to the largest event of the batch, with a mask of real objects. A token is a linear map of the kinematic columns (phi as its cosine and sine) plus a learned embedding of node_type. The objects are stored by type (jets, electrons, muons, photons, MET), so a position embedding would mostly relearn the type; without one, neither model depends on object order, as with the edge network.

  • The transformer is a stack of pre-norm attention blocks, a mean over real objects and a linear classifier. With model.pair_bias (as in the Particle Transformer, Qu et al. 2022), an MLP of the pair features [deta, dphi, dR, log(1 + m^2)] (the edge features plus the invariant mass of the pair, objects taken as massless) adds a bias per head to the attention logits, so the transformer sees the same pairwise angles as the edge network.
  • Deep Sets (Zaheer et al. 2017; the particle flow network of Komiske et al. 2019) applies an MLP to each object, sums over real objects and applies an MLP to the sum: the baseline without object interactions.

JEPA pretraining

model.name: jepa pretrains an object encoder without labels, as in I-JEPA (Assran et al. 2023), on the 12 Delphes samples of the supervised pretraining. model.backbone selects the encoder: the transformer's (configs/delphes_jepa.yaml) or the edge network, of the size of the supervised pretrained models (configs/delphes_jepa_edge.yaml), whose object encodings are its node states after the last processing step and whose context is the fully connected graph of the visible objects only.

  • a random model.mask_fraction of each event's objects (rounded, at least one, and at least one left visible) is hidden;
  • the context encoder sees the visible objects; the target encoder, an exponential moving average of it (momentum rising linearly from model.ema_momentum to 1 over training) without gradients, sees all objects;
  • a predictor (model.predictor_layers attention blocks over the visible objects' encodings and one query per hidden object) predicts the hidden objects' layer-normalized target encodings, with a mean squared error. A query is told only the hidden object's type and its index among the objects of that type (their pt order), not its kinematics; pair features are used only between visible objects. For the edge network, its decoded global state (what its classifier reads) is one more predictor input, so the whole network is pretrained.

The history reports target_spread, the standard deviation of the target encodings over a batch's objects averaged over dimensions, which falls towards zero if the representation collapses. The validation loss compares predictions with a moving target, so it does not select epochs: fine-tuning starts from last.pt (configs/gamgam_jepa_finetune.yaml for the transformer, configs/gamgam_finetune.yaml with model.pretrained for the edge network; encoder sizes must match). With model.freeze_backbone=true only the classifier is trained (a linear probe); model.pretrained=null trains the same architecture from scratch.

model.variance_weight adds the VICReg variance term (Bardes et al. 2022): the mean shortfall below 1 of each dimension's standard deviation over the batch's context encodings (and the edge network's global states).

First checks (one A100, train.fast=true, 2 pretraining epochs on Delphes fold 2, then 2 GamGam epochs, validation AUC, seed 1; not converged):

GamGam model Validation AUC
transformer from scratch / from JEPA 0.898 / 0.904
edge network from scratch 0.895
edge network from the supervised pretrained model 0.907
edge network from JEPA, last node states only 0.845
edge network from JEPA, mean node states 0.869
the same with variance_weight 0.1 / 1 0.874 / 0.810

The edge network's JEPA encodings collapse without the variance term (target_spread 0.09 against about 1 with it), and none of the edge network variants yet beats training from scratch.

Contrastive and masked-object pretraining, and probes

Two more label-free objectives pretrain the same encoders on the same samples, and their encoders load the same way (context_encoder):

  • contrastive.py (configs/delphes_contrastive.yaml): SimCLR, as JetCLR and RS3L for jets. Two views of each event (random phi rotation, eta flip, momentum scale within model.scale_jitter, each object but the first dropped with model.object_dropout) are embedded and projected; the NT-Xent loss at model.temperature contrasts them with the other events of the batch. embedding_spread watches for collapse.
  • model.jepa_target: features (configs/delphes_masked.yaml): masked object modelling (Golling et al. 2024). JEPA's masking and predictor predict the hidden objects' kinematics instead of their target encodings, with no target encoder: the comparison with JEPA isolates predicting in embedding space.

gnn4colliders probe compares encoders without fine-tuning (as jBOT): k-NN (cosine, model.probe_neighbours) and linear (L-BFGS logistic regression, model.probe_l2) classifiers of frozen event embeddings, trained on data.train_events events. A randomly initialized encoder (model.pretrained=null) is the reference; supervised checkpoints probe too. Results on GamGam: "Label efficiency on GamGam" below.

scripts/compare_encoders.py compares encoders of one architecture by linear CKA (Kornblith et al. 2019), as the paper does: of their event embeddings (a matrix), and layer by layer against the first one (the mean over objects after each transformer block, or the edge network's mean node states after each step, then the event embedding). With a random encoder first, it shows where each objective moves the representation.

In-context models: TabPFN, TabICL, TabDPT and a collider PFN

Both classify a dataset's events given data.train_events labelled training events (an equal number per class) as context, without training on that dataset; cli.in_context_forward builds them for evaluate and predict.

  • tabular.py: TabPFN (Prior Labs; model.tabpfn_version, default 3.5) on fixed-width rows: the leading model.slots objects of each type (6 jets, 2 electrons, 2 muons, 3 photons, MET) with their six kinematic columns, NaN when absent, and the number of objects of each type.

  • pfn.py: our own prior-fitted network. The object transformer (which can start from the JEPA encoder) embeds each event; the context events get a label embedding; model.pfn_layers attention blocks run over a task's events, every event attending to the context events only, so queries are independent and the context's keys and values are computed once. It is meta-trained (task.name: pfn) on tasks drawn from the Delphes samples: 2-10 classes of task.prior_classes in random order, Dirichlet class proportions, 16-2048 context events (log-uniform), 256 queries, and per event a random phi rotation and eta flip, per task a momentum and energy scale within task.energy_jitter. The GamGam processes are among the Delphes samples; task.prior_classes=[1,6,7,8,9,10,11] leaves them out. configs/opendata_pfn.yaml (written by scripts/opendata_prior.py) meta-trains it on the ATLAS Open Data 2025 MC instead: 925 classes of 331 processes, H -> gamma gamma left out, a tenth of the processes held out for validation, classes mixing 1 to 3 processes.

  • Options that bring the PFN closer to TabPFN, all off by default (the default model's outputs match the earlier code to 2e-7): model.pfn_summary_tokens learned tokens carry each event through the encoder, with a context event's label added at the input instead of after the encoder; model.pfn_alternating adds attention across the task's events (summary tokens, context only) after every encoder block, as TabPFN alternates attention over features and over rows, so an event's encoding depends on the task; model.pfn_normalize scales each task's momenta so that the context's geometric-mean object pT is 50 GeV (as TabPFN standardizes features by the context); model.pfn_ensemble averages the class probabilities of passes with the events rotated in phi, flipped in eta and the class labels permuted (as TabPFN ensembles over class orders and preprocessings); and task.synthetic_tasks makes a fraction of the training tasks synthetic, as TabPFN's prior draws random functions: events of 1 to 3 processes ranked by a random network (16 tanh units) of 1 to 4 of 21 event observables (per object type the count, scalar sum pT, leading pT and mass; the b-tag count), cut into classes of Dirichlet fractions. A large context is encoded in chunks, one block at a time, but the alternating PFN keeps every context event's tokens and the keys and values of every block on the GPU: 100k events fit, all 1.98M GamGam training events do not (out of memory on a 40 GB A100).

  • Options from TabPFN-3.5 (219M parameters, 79 times ours; read from its checkpoint and code), all off by default: model.pfn_retrieval makes the logits the log of an attention-weighted vote of the context's labels (TabPFN's many-class decoder) instead of a linear classifier; model.pfn_qk_norm RMS-norms queries and keys per head and scales the queries by a learned factor times the log of the number of keys (TabPFN-3's scalable softmax), in all attention across events; model.pfn_query_kv lets the queries read the context through one key/value head of their own, so the memory of a large context is n_heads times smaller; model.pfn_concat joins the summary tokens into one token pfn_summary_tokens * hid_size wide for the blocks after the encoder (TabPFN-3.5 joins 8 x 128 to 1024), with the label embedded again; model.pfn_fourier learned frequencies of each object's log pT, eta and phi and model.pfn_ranks (each object's pT and energy ranked among the context's objects of its type, as TabPFN adds ECDF ranks) are added to the object embeddings, through maps that start at zero. train.lr_schedule: cosine warms up over train.warmup_epochs.

  • Meta-training speed: training batches are encoded in 4 parts of events with similar numbers of objects (only 37% of the padded object tokens were real), tasks are sampled in the loader workers rather than all at the start of an epoch, and under torchrun all ranks draw the same context size for a step, so that they wait less for each other. Steps are drawn from a generator per seed, epoch, rank and step, so the tasks differ from those of runs before this change. In-context inference sorts the context by number of objects and attends in chunks of data.batch_size events.

  • tabular.py also runs TabICLv2 (Qu et al., ICML 2026; BSD-3, open pretraining code) on the same rows, as a second tabular in-context model. Rows can measure azimuths from the leading object's (model.relative_phi) and end with the invariant mass and Delta R of every pair of the model.pair_objects leading objects by pt (energies are pt cosh(eta), so objects are massless; a pair with MET gives a transverse-like mass). Both are off by default, so the default rows are unchanged.

  • tabular.py also runs TabDPT (Ma et al. 2024; Apache-2.0), trained on real tables rather than a synthetic prior; with model.tabdpt_context_size each event's context is its nearest context events (faiss retrieval), beyond what fits in one pass. It computes in float64 in places (on a Mac it runs on the CPU), and on macOS faiss and torch crash together unless OMP_NUM_THREADS=1.

  • Tabular models can see, instead of or beside the rows, the event embeddings of a pretrained object encoder (model.tabular_inputs: embeddings or both, the encoder of model.backbone and model.pretrained): physics-aware embeddings under a generic tabular prior. In meta-training the encoder trains with the LoRA updates unless model.freeze_backbone.

  • tabular.py also meta-trains the TabPFN-3.5 network itself on the PFN's tasks (model.name: tabpfn, task.name: pfn), on the rows above (built after the augmentation), starting from Prior Labs' weights. Calling the network directly skips TabPFN's preprocessing and ensembling, which on the GamGam fixtures cost nothing (validation AUC 0.852 against 0.843 with predict_proba, 61 events). By default the network is frozen and the four attention projections and two MLP layers of each of its 24 in-context blocks get rank-8 LoRA updates (2.8M parameters, zero at the start, so training starts from TabPFN as released); model.lora_rank=0 trains all weights. In context, the context's keys and values are cached (the same logits as one pass over context and queries, to 2e-5). It separates what Prior Labs' prior and what collider tasks contribute: TabPFN as released, TabPFN meta-trained here, and our PFN. TabICL (configs/delphes_tabicl.yaml; BSD-3, so meta-trained weights can be shared) meta-trains the same way. Its attention reads fused projection weights directly, so LoRA is a parametrization of the weights (12 blocks, 0.59M of 28M parameters at rank 8), for TabPFN too. TabICL's network takes no NaN: rows are standardized by the context's statistics, absent values at the mean, clipped at 8 sigma. On the fixtures this gives AUC 0.827 against 0.820 with its classifier, and the cached in-context forward equals the training forward only in float32 (with mixed precision on the CPU, AUC 0.59).

Label efficiency on GamGam (first study, 2026-10-22)

GamGam validation AUC (%) against the number of labelled training events N (data.train_events, equal per class), one seed, on one 4-GPU A100 node in an interactive allocation (3.5 h). Each entry is evaluated on 20,000 random validation events (data.eval_events=20000, about +-0.3); on all 246,922, evaluated for the supervised runs and the leading TabPFN runs, each is 0.05 to 0.41 lower, in the same order (TabPFN-3.5: 80.28, 86.92, 89.01, 0.08 to 0.13 lower, matching the M2 Max's 80.26 and 86.93).

model N = 100 1000 10000 50000
edge network, from scratch 78.00 82.74 87.66
edge network, from Pretrained_GNN 84.95 87.48 89.98
transformer, from scratch 72.33 83.47 87.03
transformer, from JEPA 73.25 85.10 88.60
transformer, from masked objects 75.92 86.12 88.73
TabPFN-3.5 (rows) 80.41 87.05 89.09 89.56
TabPFN-3.5 (rows, relative_phi, pair_objects: 4) 80.61 87.78 89.56
TabPFN-3.5 (rows + JEPA embeddings) 80.97 87.43 89.44
TabPFN-3.5 (rows + masked-object embeddings) 82.99 88.22 89.94
TabPFN-3.5 (rows + Pretrained_GNN embeddings) 87.32 88.96 90.07
TabPFN-3.5, released network, direct forward 80.74 86.64 89.03
TabPFN-3.5, meta-trained on Delphes (LoRA) 80.52 84.77 88.34
TabICLv2 80.53 85.68 88.41 89.12
TabDPT 77.48 84.71 88.42
our PFN, meta-trained on Delphes 78.13 76.18 76.51

Probes (k-NN / linear) of the frozen encoders, the same N:

encoder N = 100 1000 10000
random 73.4 / 72.6 76.6 / 83.1 79.5 / 85.4
JEPA 73.0 / 73.4 78.9 / 83.4 81.8 / 87.9
masked objects 75.0 / 77.5 81.7 / 85.7 84.4 / 88.5
contrastive 61.6 / 67.9 70.0 / 80.3 76.4 / 85.9

Setup (every run and its overrides: scripts/label_efficiency_study.sh): the pretraining runs saw the same 4M class-balanced Delphes events for 8 epochs (data.train_events=4000000, 50 to 70 min each); the PFN meta-trained for 25 epochs of 1000 steps (25,000 steps, 95 min) and TabPFN for 6 epochs of 500 (3,000 steps, 48 min). Supervised runs train for a fixed number of steps with early stopping off (train.early_stopping=null; batch size, epochs: 100, 1000 for N = 100; 128, 400 for N = 1000; 256, 100 for N = 10000; 20 to 35 min at N = 100, 2 to 4 min at N = 10000) and keep the epoch of best validation AUC, chosen on the same 20,000 events that are reported. That choice uses validation labels the in-context models do not get, and from N = 1000 most supervised runs overfit: their last epoch is far below the kept one. On all validation events (92% of them not used for the choice) the kept epochs lose 0.05 to 0.41, no more than the in-context runs lose, so picking the maximum over noisy evaluations inflates them little:

supervised run (N = 100 / 1000 / 10000) kept epoch, 20k kept epoch, all last epoch, 20k
edge network, from scratch 78.00 / 82.74 / 87.66 77.80 / 82.69 / 87.42 68.88 / 78.88 / 87.65
edge network, from Pretrained_GNN 84.95 / 87.48 / 89.98 84.54 / 87.09 / 89.74 80.75 / 81.58 / 86.85
transformer, from scratch 72.33 / 83.47 / 87.03 71.99 / 83.35 / 86.95 72.33 / 83.44 / 87.03
transformer, from JEPA 73.25 / 85.10 / 88.60 73.07 / 84.96 / 88.45 72.34 / 78.82 / 81.46
transformer, from masked objects 75.92 / 86.12 / 88.73 75.73 / 85.84 / 88.60 75.92 / 79.73 / 85.62

With only N labelled events there is no large labelled set to choose the epoch on; without the choice (last epoch) every supervised model is below TabPFN-3.5 as released at every N except Pretrained_GNN at N = 100.

With the configs' settings instead (batch size 1024, at most 100 epochs, early stopping after 5 epochs without training-loss improvement) N = 1000 is one step per epoch, at most 100 steps: the transformer then reaches 70.04 at N = 1000 instead of 83.47. The in-context models take 10 to 30 s for the 20,000 queries at N = 1000 and 10000 (TabPFN 109 s at N = 50000; the first run of each, N = 100, also loads the weights).

Findings:

  • TabPFN-3.5 as released, without training, comes within 0.4 (N = 1000) and 0.9 (N = 10000) of fine-tuning the supervised Pretrained_GNN (0.2 and 0.7 on all validation events), with its epoch chosen on validation labels, and above it from N = 1000 without that choice. It beats every model trained from scratch at every N. TabICL is close behind; TabDPT is behind at small N. The physics row features add up to 0.7.
  • Encoder embeddings help TabPFN most at small N. With the label-free masked-object encoder: +2.6 at N = 100. With Pretrained_GNN (trained with labels, on Delphes samples that include the GamGam processes): the best result at every N run (100 to 10000; not run at 50000), +6.9 at N = 100.
  • Meta-training on Delphes tasks does not help. TabPFN with LoRA loses 1.9 at N = 1000 against the same network with zero updates (its direct forward costs at most 0.4 against the classifier's preprocessing and ensembling). Our PFN is below TabPFN and gets worse with more context (76 at N = 1000, inside the meta-training context sizes 16 to 2048): to investigate before the held-out-processes run, which was not done.
  • Of the label-free pretraining objectives, masked-object modelling is the best, in probes and fine-tuned; JEPA's prediction in embedding space adds nothing over it here. Both improve fine-tuning the transformer over training it from scratch (+0.9 to +3.6) but stay below fine-tuning the edge network from Pretrained_GNN. That gap mixes the architecture (from scratch, the edge network is 5.7 above the transformer at N = 100) with the objective (label-free or supervised); the like-for-like comparison, JEPA of the edge network (configs/delphes_jepa_edge.yaml), was not run.
  • Contrastive pretraining is below a random encoder in the k-NN probe at every N and in the linear probe at N = 100 and 1000 (67.9 against 72.6, 80.3 against 83.1); at N = 10000 its linear probe is 0.5 above (85.9 against 85.4). Without the momentum-scale augmentation (model.scale_jitter=0, 4 epochs) it is no better (72.4 / 84.3 at N = 10000).

Not done:

  • the held-out-processes meta-training (task.prior_classes without the five GamGam processes) of the PFN, gated on it beating TabPFN, and of TabPFN and TabICL;
  • TabICL meta-training and model.lora_rank=0;
  • fine-tuning from contrastive pretraining (only probed), and JEPA of the edge network;
  • more seeds, and the test split.

The loss is a weighted mean per class present in the batch, averaged over those classes, which balances the classes without explicit class weights. Metrics (accuracy, one-vs-rest AUC averaged over classes, macro precision, recall and F1) use events with positive weight.

The TabPFN-like PFN on GamGam (2026-10-27)

GamGam test AUC (%) in context, against the number of labelled context events N (equal per class), on 50,000 test events, one seed. Every PFN is meta-trained on the Open Data prior (configs/opendata_pfn.yaml). The earlier PFN (pooled events, contexts of 16 to 4096, random encoder, fp32) is the checkpoint of epoch 38. The new ones train 48 epochs of 1000 steps (2.3 h on one A100 each, bf16) with 2 summary tokens, alternating attention, the per-task momentum scale, the masked-object encoder (delphes_masked), half synthetic tasks, contexts of 16 to 16384 and batches of 16384 events; each ablation changes one of them.

PFN N = 100 1000 10000 100000
earlier 80.63 82.47 83.54 83.53
earlier, ensemble of 8 81.92 83.87 84.67 84.77
new 79.12 80.39 83.79 84.18
new, ensemble of 8 79.67 82.26 84.46 84.77
new, no alternating attention 77.14 81.21 84.43 84.68
new, random encoder 76.47 81.76 83.42 83.56
new, no synthetic tasks 81.28 83.14 84.72 85.45
new, no synthetic tasks, ensemble of 8 83.75 86.11 86.54 86.52

Validation on held-out Open Data processes after 48 epochs (loss / accuracy): new 0.772 / 0.722, no alternating 0.800 / 0.712, random encoder 0.809 / 0.711, no synthetic tasks 0.665 / 0.764; all still falling. One pass takes 1 to 3 s for the 50,000 queries at N = 1000 and 10000 and 35 to 58 s at N = 100000; the ensemble of 8 takes 460 s at N = 100000.

Findings:

  • Synthetic tasks hurt at every N (0.9 to 2.8 against the same PFN without them; 1.8 to 4.1 ensembled) and in validation. Without them the PFN beats the earlier one at every N, by 1.8 to 2.2 with both ensembled.
  • The ensemble of 8 adds 1.1 to 3.0 to the PFN without synthetic tasks.
  • The masked-object encoder adds 2.7 at N = 100 and 0.6 at N = 100000.
  • Alternating attention adds 2.0 at N = 100 and costs 0.5 to 0.8 from N = 1000; it makes training 20% slower.
  • Against the validation AUCs of the first study (on other events, so only roughly comparable): the best PFN is above TabPFN-3.5 on rows (80.41) and every supervised transformer at N = 100, below the edge network from Pretrained_GNN (84.95) and TabPFN-3.5 with its embeddings (87.32). At N = 1000 it equals the transformer from masked objects (86.12) and is 0.9 and 1.4 below TabPFN-3.5 and Pretrained_GNN; at 10000 it is 2.2 to 3.4 below all three, and it gains nothing from 10000 to 100000.

Not done: all 1.98M training events as context (out of memory; see the PFN options above), longer meta-training, and the held-out-processes prior.

TabPFN-3.5 options and a larger PFN on GamGam (2026-10-28)

Both PFNs start from the best one above (no synthetic tasks) and add the TabPFN-3.5 options pfn_retrieval, pfn_qk_norm, pfn_query_kv, pfn_fourier=16 and pfn_ranks, the cosine schedule (2 warm-up epochs, learning rate 1e-4) and the faster meta-training. "Small" keeps its size (2.9M parameters) and trains 40 epochs on one A100 (168 s each). "Large" has 4 summary tokens joined to one 512-wide token and 12 blocks after the encoder (42M parameters) and trains 26 epochs on three A100s under torchrun (280 s each, three times the tasks per step). Same test events and protocol as above, one seed.

PFN N = 100 1000 10000 100000
best above, ensemble of 8 83.75 86.11 86.54 86.52
small 81.73 84.65 86.76 87.66
small, ensemble of 8 83.15 86.20 87.63 88.20
large 82.90 85.83 87.72 88.43
large, ensemble of 8 84.06 86.80 88.24 88.62

Validation on held-out Open Data processes in the last epoch (loss / accuracy): small 0.599 / 0.789, large 0.561 / 0.799 (best above 0.665 / 0.764). At N = 100000 one pass takes 21 s (small) and 87 s (large), the ensemble of 8 169 s and 698 s; the large ensemble peaks at 13 GiB.

Findings:

  • The options, schedule and faster training add 0.5 to 2.2 to one pass of the best PFN above (81.28, 83.14, 84.72, 85.45), and the AUC now rises from 10000 to 100000 (by 0.4 to 0.9) instead of staying flat.
  • The larger PFN adds 0.8 to 1.2 more. One pass of it is 0.9 and 0.3 below the earlier ensemble at N = 100 and 1000 and 1.2 and 1.9 above it from 10000; ensembled, it gains 0.3 to 2.1 on the earlier ensemble.
  • The ensemble adds less than before: 0.2 to 1.6.
  • The options were not ablated one by one, and the two runs differ in epochs and GPUs as well as size.
  • Against the validation AUCs of the first study (only roughly comparable): at N = 100 the large ensemble is 3.7 above TabPFN-3.5 on rows and 0.9 below Pretrained_GNN; at 1000 it is 0.3 and 0.7 below them, at 10000 0.9 and 1.7 below.

Not done: all 1.98M training events as context, one ablation per option, more seeds.

Performance

Graphs differ in size, which suits neither torch.compile nor CUDA graphs out of the box:

  • torch.compile(dynamic=True) treats node and edge counts as symbolic sizes, so one compiled graph serves all batches without padding.
  • CUDA graph replay needs fixed shapes. Batches are copied into buffers rounded up to multiples of 512 nodes and 4096 edges; the remainder forms 64 padding graphs connected only to themselves, so the real graphs' outputs do not change. A step is captured once per bucket, a few per epoch. The optimizer runs in Adam's capturable form with device-tensor learning rates, which the scheduler updates in place.
  • bfloat16 autocast covers the forward pass; logits, loss, graph sums (which divide by edge counts) and the optimizer stay float32.

GamGam training step, 1024-event batches (about 5,600 nodes and 30,000 edges), one A100-SXM4-80GB, OMP_NUM_THREADS=8, four loader workers (step times measured before the move to flat modules, epoch times after):

Mode ms per batch Epoch with validation
float32 eager 19.2 37 s
CUDA graphs + TF32 8.5
torch.compile 13.0
compile + CUDA graphs + TF32 6.1
compile + CUDA graphs + bfloat16 (train.fast) 5.9 9 s (first: 59 s, compiling)

Eager training is bound by kernel launches (about a thousand small kernels per step), not by arithmetic; bfloat16 alone is slower because of the extra cast kernels, which torch.compile fuses. Results of the fast modes agree with float32 to rounding, not bitwise (validation AUC 0.895 against 0.897 after three epochs). Preparing the 2.5M GamGam events with 8 workers takes about 9 s on a login node (40 s was seen on a compute node, likely a cold file cache).

PyTorch sizes its CPU thread pool to every visible core; on a 128-core node that makes batch construction several times slower and CPU reductions non-reproducible, so set OMP_NUM_THREADS. Single-threaded CPU training is bitwise reproducible.

ATLAS Open Data GamGam versus Ho et al. (2026)

The main benchmark reproduces the from-scratch baseline of Table 3 (5 classes, 2.5M events, 80:10:10 split, five seeds). Choices the paper does not specify:

  • electrons and muons are split from lep_* by lep_type (11, 13);
  • b-tagging is jet_MV2c10 > 0.8244273, the 70% working point;
  • momenta and energies are scaled by 1e-4 (MeV to the pretraining unit of 10 GeV);
  • the fold is eventNumber % 10: folds 0-7 train, 8 validates, 9 tests;
  • training is unweighted (23% of ttH events have mcWeight = -1).

The paper's stopping rule (5 epochs without training-loss improvement) never fires within 200 epochs on this task; validation AUC peaks earlier (epoch 67 for seed 1) and best.pt keeps that epoch.

Test split Accuracy AUC F1 Precision Recall
Paper (5 seeds) 71.8 ± 0.2 91.2 63.0 62.2 66.3
This package, seed 1 (CUDA graphs + TF32) 71.1 91.3 62.6 62.4 66.7