Download docs/design.md from HWresearch/GNN4Colliders: direct link, hf CLI and curl.
- Browser
- Download file 31 kB
-
https://huggingface.co/HWresearch/GNN4Colliders/resolve/main/docs/design.md
- Command line
-
hf download hf://HWresearch/GNN4Colliders/docs/design.md
-
curl -L -o design.md https://huggingface.co/HWresearch/GNN4Colliders/resolve/main/docs/design.md
Design
Modules
| Module | Contents |
|---|---|
config.py |
the experiment dataclasses, YAML loading and overrides |
data.py |
ROOT reading, Hugging Face download, derived branches, object features, PackedEvents and its cache |
loader.py |
EventLoader: deterministic shuffling, DDP sharding, collation workers |
graphs.py |
fully connected TensorGraph batches, edge features, CUDA-graph padding buffers |
sequences.py |
padded token batches for the transformer and Deep Sets, event augmentations |
models.py |
EdgeNetwork, FineTunedEdgeNetwork, ObjectTransformer, TransformerClassifier, DeepSets, checkpoint loading |
jepa.py |
JEPA pretraining of the transformer or edge network encoder (encoding or kinematic targets), event embeddings |
contrastive.py |
contrastive (SimCLR) pretraining of the same encoders |
probe.py |
k-NN and linear probes of frozen encoders, linear CKA between them |
pfn.py |
our prior-fitted network over events and the task prior of meta-training |
tabular.py |
event rows and encoder embeddings for TabPFN, TabICL and TabDPT; LoRA meta-training of TabPFN and TabICL |
tasks.py |
multiclass and binary losses and metrics, the pretraining and PFN losses |
train.py |
Trainer, early stopping, checkpoints |
cuda_graphs.py |
captured training steps |
distributed.py |
torchrun process group and collectives |
inference.py |
model outputs over a loader, prediction files |
export.py |
ONNX export |
cli.py |
the six commands |
Data flow
prepare reads each ROOT file in chunks of 100,000 entries (in parallel with
data.prepare_workers) and stores every event's objects as rows of one
[objects, 7] tensor with offsets, plus per-event labels, folds, weights and
source file and entry. Nothing in the cache depends on the model: graphs and
token sequences are built per batch, in the loader workers.
A batch of the edge network is a set of fully connected graphs in flat
edge-index form (edge_src, edge_dst, node_batch, edge_batch). Edges
are directed, source-major and without self-loops except for one-node graphs,
with features [deta, dphi, dR]. These conventions, the seven node columns,
the MLP layout and the parameter names are those of the original ROOT-GNN, so
the pretrained checkpoints in Pretrained_GNN/ load without conversion.
The transformer and Deep Sets take each event's objects as tokens
[events, objects, 7], zero-padded to the largest event of the batch, with
a mask of real objects. A token is a linear map of the kinematic columns
(phi as its cosine and sine) plus a learned embedding of node_type. The
objects are stored by type (jets, electrons, muons, photons, MET), so a
position embedding would mostly relearn the type; without one, neither
model depends on object order, as with the edge network.
- The transformer is a stack of pre-norm attention blocks, a mean over real
objects and a linear classifier. With
model.pair_bias(as in the Particle Transformer, Qu et al. 2022), an MLP of the pair features[deta, dphi, dR, log(1 + m^2)](the edge features plus the invariant mass of the pair, objects taken as massless) adds a bias per head to the attention logits, so the transformer sees the same pairwise angles as the edge network. - Deep Sets (Zaheer et al. 2017; the particle flow network of Komiske et al. 2019) applies an MLP to each object, sums over real objects and applies an MLP to the sum: the baseline without object interactions.
JEPA pretraining
model.name: jepa pretrains an object encoder without labels, as in
I-JEPA (Assran et al. 2023), on the 12 Delphes samples of the supervised
pretraining. model.backbone selects the encoder: the transformer's
(configs/delphes_jepa.yaml) or the edge network, of the size of the
supervised pretrained models (configs/delphes_jepa_edge.yaml), whose
object encodings are its node states after the last processing step and
whose context is the fully connected graph of the visible objects only.
- a random
model.mask_fractionof each event's objects (rounded, at least one, and at least one left visible) is hidden; - the context encoder sees the visible objects; the target encoder, an
exponential moving average of it (momentum rising linearly from
model.ema_momentumto 1 over training) without gradients, sees all objects; - a predictor (
model.predictor_layersattention blocks over the visible objects' encodings and one query per hidden object) predicts the hidden objects' layer-normalized target encodings, with a mean squared error. A query is told only the hidden object's type and its index among the objects of that type (their pt order), not its kinematics; pair features are used only between visible objects. For the edge network, its decoded global state (what its classifier reads) is one more predictor input, so the whole network is pretrained.
The history reports target_spread, the standard deviation of the target
encodings over a batch's objects averaged over dimensions, which falls
towards zero if the representation collapses. The validation loss compares
predictions with a moving target, so it does not select epochs: fine-tuning
starts from last.pt (configs/gamgam_jepa_finetune.yaml for the
transformer, configs/gamgam_finetune.yaml with model.pretrained for the
edge network; encoder sizes must match). With model.freeze_backbone=true only the classifier is
trained (a linear probe); model.pretrained=null trains the same
architecture from scratch.
model.variance_weight adds the VICReg variance term (Bardes et al. 2022):
the mean shortfall below 1 of each dimension's standard deviation over the
batch's context encodings (and the edge network's global states).
First checks (one A100, train.fast=true, 2 pretraining epochs on Delphes
fold 2, then 2 GamGam epochs, validation AUC, seed 1; not converged):
| GamGam model | Validation AUC |
|---|---|
| transformer from scratch / from JEPA | 0.898 / 0.904 |
| edge network from scratch | 0.895 |
| edge network from the supervised pretrained model | 0.907 |
| edge network from JEPA, last node states only | 0.845 |
| edge network from JEPA, mean node states | 0.869 |
the same with variance_weight 0.1 / 1 |
0.874 / 0.810 |
The edge network's JEPA encodings collapse without the variance term
(target_spread 0.09 against about 1 with it), and none of the edge
network variants yet beats training from scratch.
Contrastive and masked-object pretraining, and probes
Two more label-free objectives pretrain the same encoders on the same
samples, and their encoders load the same way (context_encoder):
contrastive.py(configs/delphes_contrastive.yaml): SimCLR, as JetCLR and RS3L for jets. Two views of each event (random phi rotation, eta flip, momentum scale withinmodel.scale_jitter, each object but the first dropped withmodel.object_dropout) are embedded and projected; the NT-Xent loss atmodel.temperaturecontrasts them with the other events of the batch.embedding_spreadwatches for collapse.model.jepa_target: features(configs/delphes_masked.yaml): masked object modelling (Golling et al. 2024). JEPA's masking and predictor predict the hidden objects' kinematics instead of their target encodings, with no target encoder: the comparison with JEPA isolates predicting in embedding space.
gnn4colliders probe compares encoders without fine-tuning (as jBOT):
k-NN (cosine, model.probe_neighbours) and linear (L-BFGS logistic
regression, model.probe_l2) classifiers of frozen event embeddings,
trained on data.train_events events. A randomly initialized encoder
(model.pretrained=null) is the reference; supervised checkpoints probe
too. Results on GamGam: "Label efficiency on GamGam" below.
scripts/compare_encoders.py compares encoders of one architecture by
linear CKA (Kornblith et al. 2019), as the paper does: of their event
embeddings (a matrix), and layer by layer against the first one (the mean
over objects after each transformer block, or the edge network's mean node
states after each step, then the event embedding). With a random encoder
first, it shows where each objective moves the representation.
In-context models: TabPFN, TabICL, TabDPT and a collider PFN
Both classify a dataset's events given data.train_events labelled
training events (an equal number per class) as context, without training on
that dataset; cli.in_context_forward builds them for evaluate and
predict.
tabular.py: TabPFN (Prior Labs;model.tabpfn_version, default 3.5) on fixed-width rows: the leadingmodel.slotsobjects of each type (6 jets, 2 electrons, 2 muons, 3 photons, MET) with their six kinematic columns, NaN when absent, and the number of objects of each type.pfn.py: our own prior-fitted network. The object transformer (which can start from the JEPA encoder) embeds each event; the context events get a label embedding;model.pfn_layersattention blocks run over a task's events, every event attending to the context events only, so queries are independent and the context's keys and values are computed once. It is meta-trained (task.name: pfn) on tasks drawn from the Delphes samples: 2-10 classes oftask.prior_classesin random order, Dirichlet class proportions, 16-2048 context events (log-uniform), 256 queries, and per event a random phi rotation and eta flip, per task a momentum and energy scale withintask.energy_jitter. The GamGam processes are among the Delphes samples;task.prior_classes=[1,6,7,8,9,10,11]leaves them out.configs/opendata_pfn.yaml(written byscripts/opendata_prior.py) meta-trains it on the ATLAS Open Data 2025 MC instead: 925 classes of 331 processes, H -> gamma gamma left out, a tenth of the processes held out for validation, classes mixing 1 to 3 processes.Options that bring the PFN closer to TabPFN, all off by default (the default model's outputs match the earlier code to 2e-7):
model.pfn_summary_tokenslearned tokens carry each event through the encoder, with a context event's label added at the input instead of after the encoder;model.pfn_alternatingadds attention across the task's events (summary tokens, context only) after every encoder block, as TabPFN alternates attention over features and over rows, so an event's encoding depends on the task;model.pfn_normalizescales each task's momenta so that the context's geometric-mean object pT is 50 GeV (as TabPFN standardizes features by the context);model.pfn_ensembleaverages the class probabilities of passes with the events rotated in phi, flipped in eta and the class labels permuted (as TabPFN ensembles over class orders and preprocessings); andtask.synthetic_tasksmakes a fraction of the training tasks synthetic, as TabPFN's prior draws random functions: events of 1 to 3 processes ranked by a random network (16 tanh units) of 1 to 4 of 21 event observables (per object type the count, scalar sum pT, leading pT and mass; the b-tag count), cut into classes of Dirichlet fractions. A large context is encoded in chunks, one block at a time, but the alternating PFN keeps every context event's tokens and the keys and values of every block on the GPU: 100k events fit, all 1.98M GamGam training events do not (out of memory on a 40 GB A100).Options from TabPFN-3.5 (219M parameters, 79 times ours; read from its checkpoint and code), all off by default:
model.pfn_retrievalmakes the logits the log of an attention-weighted vote of the context's labels (TabPFN's many-class decoder) instead of a linear classifier;model.pfn_qk_normRMS-norms queries and keys per head and scales the queries by a learned factor times the log of the number of keys (TabPFN-3's scalable softmax), in all attention across events;model.pfn_query_kvlets the queries read the context through one key/value head of their own, so the memory of a large context isn_headstimes smaller;model.pfn_concatjoins the summary tokens into one tokenpfn_summary_tokens * hid_sizewide for the blocks after the encoder (TabPFN-3.5 joins 8 x 128 to 1024), with the label embedded again;model.pfn_fourierlearned frequencies of each object's log pT, eta and phi andmodel.pfn_ranks(each object's pT and energy ranked among the context's objects of its type, as TabPFN adds ECDF ranks) are added to the object embeddings, through maps that start at zero.train.lr_schedule: cosinewarms up overtrain.warmup_epochs.Meta-training speed: training batches are encoded in 4 parts of events with similar numbers of objects (only 37% of the padded object tokens were real), tasks are sampled in the loader workers rather than all at the start of an epoch, and under
torchrunall ranks draw the same context size for a step, so that they wait less for each other. Steps are drawn from a generator per seed, epoch, rank and step, so the tasks differ from those of runs before this change. In-context inference sorts the context by number of objects and attends in chunks ofdata.batch_sizeevents.tabular.pyalso runs TabICLv2 (Qu et al., ICML 2026; BSD-3, open pretraining code) on the same rows, as a second tabular in-context model. Rows can measure azimuths from the leading object's (model.relative_phi) and end with the invariant mass and Delta R of every pair of themodel.pair_objectsleading objects by pt (energies arept cosh(eta), so objects are massless; a pair with MET gives a transverse-like mass). Both are off by default, so the default rows are unchanged.tabular.pyalso runs TabDPT (Ma et al. 2024; Apache-2.0), trained on real tables rather than a synthetic prior; withmodel.tabdpt_context_sizeeach event's context is its nearest context events (faiss retrieval), beyond what fits in one pass. It computes in float64 in places (on a Mac it runs on the CPU), and on macOS faiss and torch crash together unlessOMP_NUM_THREADS=1.Tabular models can see, instead of or beside the rows, the event embeddings of a pretrained object encoder (
model.tabular_inputs: embeddingsorboth, the encoder ofmodel.backboneandmodel.pretrained): physics-aware embeddings under a generic tabular prior. In meta-training the encoder trains with the LoRA updates unlessmodel.freeze_backbone.tabular.pyalso meta-trains the TabPFN-3.5 network itself on the PFN's tasks (model.name: tabpfn,task.name: pfn), on the rows above (built after the augmentation), starting from Prior Labs' weights. Calling the network directly skips TabPFN's preprocessing and ensembling, which on the GamGam fixtures cost nothing (validation AUC 0.852 against 0.843 withpredict_proba, 61 events). By default the network is frozen and the four attention projections and two MLP layers of each of its 24 in-context blocks get rank-8 LoRA updates (2.8M parameters, zero at the start, so training starts from TabPFN as released);model.lora_rank=0trains all weights. In context, the context's keys and values are cached (the same logits as one pass over context and queries, to 2e-5). It separates what Prior Labs' prior and what collider tasks contribute: TabPFN as released, TabPFN meta-trained here, and our PFN. TabICL (configs/delphes_tabicl.yaml; BSD-3, so meta-trained weights can be shared) meta-trains the same way. Its attention reads fused projection weights directly, so LoRA is a parametrization of the weights (12 blocks, 0.59M of 28M parameters at rank 8), for TabPFN too. TabICL's network takes no NaN: rows are standardized by the context's statistics, absent values at the mean, clipped at 8 sigma. On the fixtures this gives AUC 0.827 against 0.820 with its classifier, and the cached in-context forward equals the training forward only in float32 (with mixed precision on the CPU, AUC 0.59).
Label efficiency on GamGam (first study, 2026-10-22)
GamGam validation AUC (%) against the number of labelled training events N
(data.train_events, equal per class), one seed, on one 4-GPU A100 node in
an interactive allocation (3.5 h). Each entry is evaluated on 20,000 random
validation events (data.eval_events=20000, about +-0.3); on all 246,922,
evaluated for the supervised runs and the leading TabPFN runs, each is 0.05
to 0.41 lower, in the same order (TabPFN-3.5: 80.28, 86.92, 89.01, 0.08 to
0.13 lower, matching the M2 Max's 80.26 and 86.93).
| model | N = 100 | 1000 | 10000 | 50000 |
|---|---|---|---|---|
| edge network, from scratch | 78.00 | 82.74 | 87.66 | |
edge network, from Pretrained_GNN |
84.95 | 87.48 | 89.98 | |
| transformer, from scratch | 72.33 | 83.47 | 87.03 | |
| transformer, from JEPA | 73.25 | 85.10 | 88.60 | |
| transformer, from masked objects | 75.92 | 86.12 | 88.73 | |
| TabPFN-3.5 (rows) | 80.41 | 87.05 | 89.09 | 89.56 |
TabPFN-3.5 (rows, relative_phi, pair_objects: 4) |
80.61 | 87.78 | 89.56 | |
| TabPFN-3.5 (rows + JEPA embeddings) | 80.97 | 87.43 | 89.44 | |
| TabPFN-3.5 (rows + masked-object embeddings) | 82.99 | 88.22 | 89.94 | |
TabPFN-3.5 (rows + Pretrained_GNN embeddings) |
87.32 | 88.96 | 90.07 | |
| TabPFN-3.5, released network, direct forward | 80.74 | 86.64 | 89.03 | |
| TabPFN-3.5, meta-trained on Delphes (LoRA) | 80.52 | 84.77 | 88.34 | |
| TabICLv2 | 80.53 | 85.68 | 88.41 | 89.12 |
| TabDPT | 77.48 | 84.71 | 88.42 | |
| our PFN, meta-trained on Delphes | 78.13 | 76.18 | 76.51 |
Probes (k-NN / linear) of the frozen encoders, the same N:
| encoder | N = 100 | 1000 | 10000 |
|---|---|---|---|
| random | 73.4 / 72.6 | 76.6 / 83.1 | 79.5 / 85.4 |
| JEPA | 73.0 / 73.4 | 78.9 / 83.4 | 81.8 / 87.9 |
| masked objects | 75.0 / 77.5 | 81.7 / 85.7 | 84.4 / 88.5 |
| contrastive | 61.6 / 67.9 | 70.0 / 80.3 | 76.4 / 85.9 |
Setup (every run and its overrides: scripts/label_efficiency_study.sh):
the pretraining runs saw the same 4M class-balanced Delphes events for 8
epochs (data.train_events=4000000, 50 to 70 min each); the PFN
meta-trained for 25 epochs of 1000 steps (25,000 steps, 95 min) and TabPFN
for 6 epochs of 500 (3,000 steps, 48 min). Supervised runs train for a
fixed number of steps with early stopping off (train.early_stopping=null;
batch size, epochs: 100, 1000 for N = 100; 128, 400 for N = 1000; 256, 100
for N = 10000; 20 to 35 min at N = 100, 2 to 4 min at N = 10000) and keep
the epoch of best validation AUC, chosen on the same 20,000 events that are
reported. That choice uses validation labels the in-context models do not
get, and from N = 1000 most supervised runs overfit: their last epoch is far
below the kept one. On all validation events (92% of them not used for the
choice) the kept epochs lose 0.05 to 0.41, no more than the in-context runs
lose, so picking the maximum over noisy evaluations inflates them little:
| supervised run (N = 100 / 1000 / 10000) | kept epoch, 20k | kept epoch, all | last epoch, 20k |
|---|---|---|---|
| edge network, from scratch | 78.00 / 82.74 / 87.66 | 77.80 / 82.69 / 87.42 | 68.88 / 78.88 / 87.65 |
edge network, from Pretrained_GNN |
84.95 / 87.48 / 89.98 | 84.54 / 87.09 / 89.74 | 80.75 / 81.58 / 86.85 |
| transformer, from scratch | 72.33 / 83.47 / 87.03 | 71.99 / 83.35 / 86.95 | 72.33 / 83.44 / 87.03 |
| transformer, from JEPA | 73.25 / 85.10 / 88.60 | 73.07 / 84.96 / 88.45 | 72.34 / 78.82 / 81.46 |
| transformer, from masked objects | 75.92 / 86.12 / 88.73 | 75.73 / 85.84 / 88.60 | 75.92 / 79.73 / 85.62 |
With only N labelled events there is no large labelled set to choose the
epoch on; without the choice (last epoch) every supervised model is below
TabPFN-3.5 as released at every N except Pretrained_GNN at N = 100.
With the configs' settings instead (batch size 1024, at most 100 epochs, early stopping after 5 epochs without training-loss improvement) N = 1000 is one step per epoch, at most 100 steps: the transformer then reaches 70.04 at N = 1000 instead of 83.47. The in-context models take 10 to 30 s for the 20,000 queries at N = 1000 and 10000 (TabPFN 109 s at N = 50000; the first run of each, N = 100, also loads the weights).
Findings:
- TabPFN-3.5 as released, without training, comes within 0.4 (N = 1000)
and 0.9 (N = 10000) of fine-tuning the supervised
Pretrained_GNN(0.2 and 0.7 on all validation events), with its epoch chosen on validation labels, and above it from N = 1000 without that choice. It beats every model trained from scratch at every N. TabICL is close behind; TabDPT is behind at small N. The physics row features add up to 0.7. - Encoder embeddings help TabPFN most at small N. With the label-free
masked-object encoder: +2.6 at N = 100. With
Pretrained_GNN(trained with labels, on Delphes samples that include the GamGam processes): the best result at every N run (100 to 10000; not run at 50000), +6.9 at N = 100. - Meta-training on Delphes tasks does not help. TabPFN with LoRA loses 1.9 at N = 1000 against the same network with zero updates (its direct forward costs at most 0.4 against the classifier's preprocessing and ensembling). Our PFN is below TabPFN and gets worse with more context (76 at N = 1000, inside the meta-training context sizes 16 to 2048): to investigate before the held-out-processes run, which was not done.
- Of the label-free pretraining objectives, masked-object modelling is the
best, in probes and fine-tuned; JEPA's prediction in embedding space adds
nothing over it here. Both improve fine-tuning the transformer over
training it from scratch (+0.9 to +3.6) but stay below fine-tuning the
edge network from
Pretrained_GNN. That gap mixes the architecture (from scratch, the edge network is 5.7 above the transformer at N = 100) with the objective (label-free or supervised); the like-for-like comparison, JEPA of the edge network (configs/delphes_jepa_edge.yaml), was not run. - Contrastive pretraining is below a random encoder in the k-NN probe at
every N and in the linear probe at N = 100 and 1000 (67.9 against 72.6,
80.3 against 83.1); at N = 10000 its linear probe is 0.5 above (85.9
against 85.4). Without the momentum-scale augmentation
(
model.scale_jitter=0, 4 epochs) it is no better (72.4 / 84.3 at N = 10000).
Not done:
- the held-out-processes meta-training (
task.prior_classeswithout the five GamGam processes) of the PFN, gated on it beating TabPFN, and of TabPFN and TabICL; - TabICL meta-training and
model.lora_rank=0; - fine-tuning from contrastive pretraining (only probed), and JEPA of the edge network;
- more seeds, and the test split.
The loss is a weighted mean per class present in the batch, averaged over those classes, which balances the classes without explicit class weights. Metrics (accuracy, one-vs-rest AUC averaged over classes, macro precision, recall and F1) use events with positive weight.
The TabPFN-like PFN on GamGam (2026-10-27)
GamGam test AUC (%) in context, against the number of labelled context
events N (equal per class), on 50,000 test events, one seed. Every PFN is
meta-trained on the Open Data prior (configs/opendata_pfn.yaml). The
earlier PFN (pooled events, contexts of 16 to 4096, random encoder, fp32)
is the checkpoint of epoch 38. The new ones train 48 epochs of 1000 steps
(2.3 h on one A100 each, bf16) with 2 summary tokens, alternating
attention, the per-task momentum scale, the masked-object encoder
(delphes_masked), half synthetic tasks, contexts of 16 to 16384 and
batches of 16384 events; each ablation changes one of them.
| PFN | N = 100 | 1000 | 10000 | 100000 |
|---|---|---|---|---|
| earlier | 80.63 | 82.47 | 83.54 | 83.53 |
| earlier, ensemble of 8 | 81.92 | 83.87 | 84.67 | 84.77 |
| new | 79.12 | 80.39 | 83.79 | 84.18 |
| new, ensemble of 8 | 79.67 | 82.26 | 84.46 | 84.77 |
| new, no alternating attention | 77.14 | 81.21 | 84.43 | 84.68 |
| new, random encoder | 76.47 | 81.76 | 83.42 | 83.56 |
| new, no synthetic tasks | 81.28 | 83.14 | 84.72 | 85.45 |
| new, no synthetic tasks, ensemble of 8 | 83.75 | 86.11 | 86.54 | 86.52 |
Validation on held-out Open Data processes after 48 epochs (loss / accuracy): new 0.772 / 0.722, no alternating 0.800 / 0.712, random encoder 0.809 / 0.711, no synthetic tasks 0.665 / 0.764; all still falling. One pass takes 1 to 3 s for the 50,000 queries at N = 1000 and 10000 and 35 to 58 s at N = 100000; the ensemble of 8 takes 460 s at N = 100000.
Findings:
- Synthetic tasks hurt at every N (0.9 to 2.8 against the same PFN without them; 1.8 to 4.1 ensembled) and in validation. Without them the PFN beats the earlier one at every N, by 1.8 to 2.2 with both ensembled.
- The ensemble of 8 adds 1.1 to 3.0 to the PFN without synthetic tasks.
- The masked-object encoder adds 2.7 at N = 100 and 0.6 at N = 100000.
- Alternating attention adds 2.0 at N = 100 and costs 0.5 to 0.8 from N = 1000; it makes training 20% slower.
- Against the validation AUCs of the first study (on other events, so
only roughly comparable): the best PFN is above TabPFN-3.5 on rows
(80.41) and every supervised transformer at N = 100, below the edge
network from
Pretrained_GNN(84.95) and TabPFN-3.5 with its embeddings (87.32). At N = 1000 it equals the transformer from masked objects (86.12) and is 0.9 and 1.4 below TabPFN-3.5 andPretrained_GNN; at 10000 it is 2.2 to 3.4 below all three, and it gains nothing from 10000 to 100000.
Not done: all 1.98M training events as context (out of memory; see the PFN options above), longer meta-training, and the held-out-processes prior.
TabPFN-3.5 options and a larger PFN on GamGam (2026-10-28)
Both PFNs start from the best one above (no synthetic tasks) and add the
TabPFN-3.5 options pfn_retrieval, pfn_qk_norm, pfn_query_kv,
pfn_fourier=16 and pfn_ranks, the cosine schedule (2 warm-up epochs,
learning rate 1e-4) and the faster meta-training. "Small" keeps its size
(2.9M parameters) and trains 40 epochs on one A100 (168 s each). "Large"
has 4 summary tokens joined to one 512-wide token and 12 blocks after the
encoder (42M parameters) and trains 26 epochs on three A100s under
torchrun (280 s each, three times the tasks per step). Same test events
and protocol as above, one seed.
| PFN | N = 100 | 1000 | 10000 | 100000 |
|---|---|---|---|---|
| best above, ensemble of 8 | 83.75 | 86.11 | 86.54 | 86.52 |
| small | 81.73 | 84.65 | 86.76 | 87.66 |
| small, ensemble of 8 | 83.15 | 86.20 | 87.63 | 88.20 |
| large | 82.90 | 85.83 | 87.72 | 88.43 |
| large, ensemble of 8 | 84.06 | 86.80 | 88.24 | 88.62 |
Validation on held-out Open Data processes in the last epoch (loss / accuracy): small 0.599 / 0.789, large 0.561 / 0.799 (best above 0.665 / 0.764). At N = 100000 one pass takes 21 s (small) and 87 s (large), the ensemble of 8 169 s and 698 s; the large ensemble peaks at 13 GiB.
Findings:
- The options, schedule and faster training add 0.5 to 2.2 to one pass of the best PFN above (81.28, 83.14, 84.72, 85.45), and the AUC now rises from 10000 to 100000 (by 0.4 to 0.9) instead of staying flat.
- The larger PFN adds 0.8 to 1.2 more. One pass of it is 0.9 and 0.3 below the earlier ensemble at N = 100 and 1000 and 1.2 and 1.9 above it from 10000; ensembled, it gains 0.3 to 2.1 on the earlier ensemble.
- The ensemble adds less than before: 0.2 to 1.6.
- The options were not ablated one by one, and the two runs differ in epochs and GPUs as well as size.
- Against the validation AUCs of the first study (only roughly
comparable): at N = 100 the large ensemble is 3.7 above TabPFN-3.5 on
rows and 0.9 below
Pretrained_GNN; at 1000 it is 0.3 and 0.7 below them, at 10000 0.9 and 1.7 below.
Not done: all 1.98M training events as context, one ablation per option, more seeds.
Performance
Graphs differ in size, which suits neither torch.compile nor CUDA graphs
out of the box:
torch.compile(dynamic=True)treats node and edge counts as symbolic sizes, so one compiled graph serves all batches without padding.- CUDA graph replay needs fixed shapes. Batches are copied into buffers
rounded up to multiples of 512 nodes and 4096 edges; the remainder forms 64
padding graphs connected only to themselves, so the real graphs' outputs do
not change. A step is captured once per bucket, a few per epoch. The
optimizer runs in Adam's
capturableform with device-tensor learning rates, which the scheduler updates in place. - bfloat16 autocast covers the forward pass; logits, loss, graph sums (which divide by edge counts) and the optimizer stay float32.
GamGam training step, 1024-event batches (about 5,600 nodes and 30,000
edges), one A100-SXM4-80GB, OMP_NUM_THREADS=8, four loader workers (step
times measured before the move to flat modules, epoch times after):
| Mode | ms per batch | Epoch with validation |
|---|---|---|
| float32 eager | 19.2 | 37 s |
| CUDA graphs + TF32 | 8.5 | |
torch.compile |
13.0 | |
| compile + CUDA graphs + TF32 | 6.1 | |
compile + CUDA graphs + bfloat16 (train.fast) |
5.9 | 9 s (first: 59 s, compiling) |
Eager training is bound by kernel launches (about a thousand small kernels
per step), not by arithmetic; bfloat16 alone is slower because of the extra
cast kernels, which torch.compile fuses. Results of the fast modes agree
with float32 to rounding, not bitwise (validation AUC 0.895 against 0.897
after three epochs). Preparing the 2.5M GamGam events with 8 workers takes
about 9 s on a login node (40 s was seen on a compute node, likely a cold
file cache).
PyTorch sizes its CPU thread pool to every visible core; on a 128-core node
that makes batch construction several times slower and CPU reductions
non-reproducible, so set OMP_NUM_THREADS. Single-threaded CPU training is
bitwise reproducible.
ATLAS Open Data GamGam versus Ho et al. (2026)
The main benchmark reproduces the from-scratch baseline of Table 3 (5 classes, 2.5M events, 80:10:10 split, five seeds). Choices the paper does not specify:
- electrons and muons are split from
lep_*bylep_type(11, 13); - b-tagging is
jet_MV2c10 > 0.8244273, the 70% working point; - momenta and energies are scaled by 1e-4 (MeV to the pretraining unit of 10 GeV);
- the fold is
eventNumber % 10: folds 0-7 train, 8 validates, 9 tests; - training is unweighted (23% of ttH events have
mcWeight = -1).
The paper's stopping rule (5 epochs without training-loss improvement) never
fires within 200 epochs on this task; validation AUC peaks earlier (epoch 67
for seed 1) and best.pt keeps that epoch.
| Test split | Accuracy | AUC | F1 | Precision | Recall |
|---|---|---|---|---|---|
| Paper (5 seeds) | 71.8 ± 0.2 | 91.2 | 63.0 | 62.2 | 66.3 |
| This package, seed 1 (CUDA graphs + TF32) | 71.1 | 91.3 | 62.6 | 62.4 | 66.7 |