GNN4Colliders / docs /design.md
ho22joshua's picture
Claude Opus 5.5
feat: TabPFN-3.5 options for the PFN, a cosine schedule and faster meta-training
71aefe8
|
Raw History Blame Contribute Delete
31 kB
# Design
## Modules
| Module | Contents |
| --- | --- |
| `config.py` | the experiment dataclasses, YAML loading and overrides |
| `data.py` | ROOT reading, Hugging Face download, derived branches, object features, `PackedEvents` and its cache |
| `loader.py` | `EventLoader`: deterministic shuffling, DDP sharding, collation workers |
| `graphs.py` | fully connected `TensorGraph` batches, edge features, CUDA-graph padding buffers |
| `sequences.py` | padded token batches for the transformer and Deep Sets, event augmentations |
| `models.py` | `EdgeNetwork`, `FineTunedEdgeNetwork`, `ObjectTransformer`, `TransformerClassifier`, `DeepSets`, checkpoint loading |
| `jepa.py` | JEPA pretraining of the transformer or edge network encoder (encoding or kinematic targets), event embeddings |
| `contrastive.py` | contrastive (SimCLR) pretraining of the same encoders |
| `probe.py` | k-NN and linear probes of frozen encoders, linear CKA between them |
| `pfn.py` | our prior-fitted network over events and the task prior of meta-training |
| `tabular.py` | event rows and encoder embeddings for TabPFN, TabICL and TabDPT; LoRA meta-training of TabPFN and TabICL |
| `tasks.py` | multiclass and binary losses and metrics, the pretraining and PFN losses |
| `train.py` | `Trainer`, early stopping, checkpoints |
| `cuda_graphs.py` | captured training steps |
| `distributed.py` | `torchrun` process group and collectives |
| `inference.py` | model outputs over a loader, prediction files |
| `export.py` | ONNX export |
| `cli.py` | the six commands |
## Data flow
`prepare` reads each ROOT file in chunks of 100,000 entries (in parallel with
`data.prepare_workers`) and stores every event's objects as rows of one
`[objects, 7]` tensor with offsets, plus per-event labels, folds, weights and
source file and entry. Nothing in the cache depends on the model: graphs and
token sequences are built per batch, in the loader workers.
A batch of the edge network is a set of fully connected graphs in flat
edge-index form (`edge_src`, `edge_dst`, `node_batch`, `edge_batch`). Edges
are directed, source-major and without self-loops except for one-node graphs,
with features `[deta, dphi, dR]`. These conventions, the seven node columns,
the MLP layout and the parameter names are those of the original ROOT-GNN, so
the pretrained checkpoints in `Pretrained_GNN/` load without conversion.
The transformer and Deep Sets take each event's objects as tokens
`[events, objects, 7]`, zero-padded to the largest event of the batch, with
a mask of real objects. A token is a linear map of the kinematic columns
(`phi` as its cosine and sine) plus a learned embedding of `node_type`. The
objects are stored by type (jets, electrons, muons, photons, MET), so a
position embedding would mostly relearn the type; without one, neither
model depends on object order, as with the edge network.
* The transformer is a stack of pre-norm attention blocks, a mean over real
objects and a linear classifier. With `model.pair_bias` (as in the
Particle Transformer, Qu et al. 2022), an MLP of the pair features
`[deta, dphi, dR, log(1 + m^2)]` (the edge features plus the invariant
mass of the pair, objects taken as massless) adds a bias per head to the
attention logits, so the transformer sees the same pairwise angles as the
edge network.
* Deep Sets (Zaheer et al. 2017; the particle flow network of Komiske et
al. 2019) applies an MLP to each object, sums over real objects and
applies an MLP to the sum: the baseline without object interactions.
### JEPA pretraining
`model.name: jepa` pretrains an object encoder without labels, as in
I-JEPA (Assran et al. 2023), on the 12 Delphes samples of the supervised
pretraining. `model.backbone` selects the encoder: the transformer's
(`configs/delphes_jepa.yaml`) or the edge network, of the size of the
supervised pretrained models (`configs/delphes_jepa_edge.yaml`), whose
object encodings are its node states after the last processing step and
whose context is the fully connected graph of the visible objects only.
* a random `model.mask_fraction` of each event's objects (rounded, at least
one, and at least one left visible) is hidden;
* the context encoder sees the visible objects; the target encoder, an
exponential moving average of it (momentum rising linearly from
`model.ema_momentum` to 1 over training) without gradients, sees all
objects;
* a predictor (`model.predictor_layers` attention blocks over the visible
objects' encodings and one query per hidden object) predicts the hidden
objects' layer-normalized target encodings, with a mean squared error.
A query is told only the hidden object's type and its index among the
objects of that type (their pt order), not its kinematics; pair features
are used only between visible objects. For the edge network, its decoded
global state (what its classifier reads) is one more predictor input, so
the whole network is pretrained.
The history reports `target_spread`, the standard deviation of the target
encodings over a batch's objects averaged over dimensions, which falls
towards zero if the representation collapses. The validation loss compares
predictions with a moving target, so it does not select epochs: fine-tuning
starts from `last.pt` (`configs/gamgam_jepa_finetune.yaml` for the
transformer, `configs/gamgam_finetune.yaml` with `model.pretrained` for the
edge network; encoder sizes must match). With `model.freeze_backbone=true` only the classifier is
trained (a linear probe); `model.pretrained=null` trains the same
architecture from scratch.
`model.variance_weight` adds the VICReg variance term (Bardes et al. 2022):
the mean shortfall below 1 of each dimension's standard deviation over the
batch's context encodings (and the edge network's global states).
First checks (one A100, `train.fast=true`, 2 pretraining epochs on Delphes
fold 2, then 2 GamGam epochs, validation AUC, seed 1; not converged):
| GamGam model | Validation AUC |
| --- | --- |
| transformer from scratch / from JEPA | 0.898 / 0.904 |
| edge network from scratch | 0.895 |
| edge network from the supervised pretrained model | 0.907 |
| edge network from JEPA, last node states only | 0.845 |
| edge network from JEPA, mean node states | 0.869 |
| the same with `variance_weight` 0.1 / 1 | 0.874 / 0.810 |
The edge network's JEPA encodings collapse without the variance term
(`target_spread` 0.09 against about 1 with it), and none of the edge
network variants yet beats training from scratch.
### Contrastive and masked-object pretraining, and probes
Two more label-free objectives pretrain the same encoders on the same
samples, and their encoders load the same way (`context_encoder`):
* `contrastive.py` (`configs/delphes_contrastive.yaml`): SimCLR, as JetCLR
and RS3L for jets. Two views of each event (random phi rotation, eta
flip, momentum scale within `model.scale_jitter`, each object but the
first dropped with `model.object_dropout`) are embedded and projected;
the NT-Xent loss at `model.temperature` contrasts them with the other
events of the batch. `embedding_spread` watches for collapse.
* `model.jepa_target: features` (`configs/delphes_masked.yaml`): masked
object modelling (Golling et al. 2024). JEPA's masking and predictor
predict the hidden objects' kinematics instead of their target
encodings, with no target encoder: the comparison with JEPA isolates
predicting in embedding space.
`gnn4colliders probe` compares encoders without fine-tuning (as jBOT):
k-NN (cosine, `model.probe_neighbours`) and linear (L-BFGS logistic
regression, `model.probe_l2`) classifiers of frozen event embeddings,
trained on `data.train_events` events. A randomly initialized encoder
(`model.pretrained=null`) is the reference; supervised checkpoints probe
too. Results on GamGam: "Label efficiency on GamGam" below.
`scripts/compare_encoders.py` compares encoders of one architecture by
linear CKA (Kornblith et al. 2019), as the paper does: of their event
embeddings (a matrix), and layer by layer against the first one (the mean
over objects after each transformer block, or the edge network's mean node
states after each step, then the event embedding). With a random encoder
first, it shows where each objective moves the representation.
### In-context models: TabPFN, TabICL, TabDPT and a collider PFN
Both classify a dataset's events given `data.train_events` labelled
training events (an equal number per class) as context, without training on
that dataset; `cli.in_context_forward` builds them for `evaluate` and
`predict`.
* `tabular.py`: TabPFN (Prior Labs; `model.tabpfn_version`, default 3.5)
on fixed-width rows: the leading `model.slots` objects of each type (6
jets, 2 electrons, 2 muons, 3 photons, MET) with their six kinematic
columns, NaN when absent, and the number of objects of each type.
* `pfn.py`: our own prior-fitted network. The object transformer (which
can start from the JEPA encoder) embeds each event; the context events get
a label embedding; `model.pfn_layers` attention blocks run over a task's
events, every event attending to the context events only, so queries are
independent and the context's keys and values are computed once. It is
meta-trained (`task.name: pfn`) on tasks drawn from the Delphes samples:
2-10 classes of `task.prior_classes` in random order, Dirichlet class
proportions, 16-2048 context events (log-uniform), 256 queries, and per
event a random phi rotation and eta flip, per task a momentum and energy
scale within `task.energy_jitter`. The GamGam processes are among the
Delphes samples; `task.prior_classes=[1,6,7,8,9,10,11]` leaves them out.
`configs/opendata_pfn.yaml` (written by `scripts/opendata_prior.py`)
meta-trains it on the ATLAS Open Data 2025 MC instead: 925 classes of 331
processes, H -> gamma gamma left out, a tenth of the processes held out
for validation, classes mixing 1 to 3 processes.
* Options that bring the PFN closer to TabPFN, all off by default (the
default model's outputs match the earlier code to 2e-7):
`model.pfn_summary_tokens` learned tokens carry each event through the
encoder, with a context event's label added at the input instead of after
the encoder; `model.pfn_alternating` adds attention across the task's
events (summary tokens, context only) after every encoder block, as
TabPFN alternates attention over features and over rows, so an event's
encoding depends on the task; `model.pfn_normalize` scales each task's
momenta so that the context's geometric-mean object pT is 50 GeV (as
TabPFN standardizes features by the context); `model.pfn_ensemble`
averages the class probabilities of passes with the events rotated in
phi, flipped in eta and the class labels permuted (as TabPFN ensembles
over class orders and preprocessings); and `task.synthetic_tasks` makes
a fraction of the training tasks synthetic, as TabPFN's prior draws
random functions: events of 1 to 3 processes ranked by a random network
(16 tanh units) of 1 to 4 of 21 event observables (per object type the
count, scalar sum pT, leading pT and mass; the b-tag count), cut into
classes of Dirichlet fractions. A large context is encoded in chunks, one
block at a time, but the alternating PFN keeps every context event's
tokens and the keys and values of every block on the GPU: 100k events
fit, all 1.98M GamGam training events do not (out of memory on a 40 GB
A100).
* Options from TabPFN-3.5 (219M parameters, 79 times ours; read from
its checkpoint and code), all off by default: `model.pfn_retrieval` makes
the logits the log of an attention-weighted vote of the context's labels
(TabPFN's many-class decoder) instead of a linear classifier;
`model.pfn_qk_norm` RMS-norms queries and keys per head and scales the
queries by a learned factor times the log of the number of keys
(TabPFN-3's scalable softmax), in all attention across events;
`model.pfn_query_kv` lets the queries read the context through one
key/value head of their own, so the memory of a large context is
`n_heads` times smaller; `model.pfn_concat` joins the summary tokens into
one token `pfn_summary_tokens * hid_size` wide for the blocks after the
encoder (TabPFN-3.5 joins 8 x 128 to 1024), with the label embedded
again; `model.pfn_fourier` learned frequencies of each object's log pT,
eta and phi and `model.pfn_ranks` (each object's pT and energy ranked
among the context's objects of its type, as TabPFN adds ECDF ranks) are
added to the object embeddings, through maps that start at zero.
`train.lr_schedule: cosine` warms up over `train.warmup_epochs`.
* Meta-training speed: training batches are encoded in 4 parts of events
with similar numbers of objects (only 37% of the padded object tokens
were real), tasks are sampled in the loader workers rather than all at
the start of an epoch, and under `torchrun` all ranks draw the same
context size for a step, so that they wait less for each other. Steps are
drawn from a generator per seed, epoch, rank and step, so the tasks
differ from those of runs before this change. In-context inference sorts
the context by number of objects and attends in chunks of
`data.batch_size` events.
* `tabular.py` also runs TabICLv2 (Qu et al., ICML 2026; BSD-3, open
pretraining code) on the same rows, as a second tabular in-context model.
Rows can measure azimuths from the leading object's (`model.relative_phi`)
and end with the invariant mass and Delta R of every pair of the
`model.pair_objects` leading objects by pt (energies are `pt cosh(eta)`,
so objects are massless; a pair with MET gives a transverse-like mass).
Both are off by default, so the default rows are unchanged.
* `tabular.py` also runs TabDPT (Ma et al. 2024; Apache-2.0), trained on
real tables rather than a synthetic prior; with
`model.tabdpt_context_size` each event's context is its nearest context
events (faiss retrieval), beyond what fits in one pass. It computes in
float64 in places (on a Mac it runs on the CPU), and on macOS faiss and
torch crash together unless `OMP_NUM_THREADS=1`.
* Tabular models can see, instead of or beside the rows, the event
embeddings of a pretrained object encoder (`model.tabular_inputs:
embeddings` or `both`, the encoder of `model.backbone` and
`model.pretrained`): physics-aware embeddings under a generic tabular
prior. In meta-training the encoder trains with the LoRA updates unless
`model.freeze_backbone`.
* `tabular.py` also meta-trains the TabPFN-3.5 network itself on the
PFN's tasks (`model.name: tabpfn`, `task.name: pfn`), on the rows above
(built after the augmentation), starting from Prior Labs' weights. Calling
the network directly skips TabPFN's preprocessing and ensembling, which
on the GamGam fixtures cost nothing (validation AUC 0.852 against 0.843
with `predict_proba`, 61 events). By default the network is frozen and
the four attention projections and two MLP layers of each of its 24
in-context blocks get rank-8 LoRA updates (2.8M parameters, zero at the
start, so training starts from TabPFN as released); `model.lora_rank=0`
trains all weights. In context, the context's keys and values are cached
(the same logits as one pass over context and queries, to 2e-5). It
separates what Prior Labs' prior and what collider tasks contribute:
TabPFN as released, TabPFN meta-trained here, and our PFN. TabICL
(`configs/delphes_tabicl.yaml`; BSD-3, so meta-trained weights can be
shared) meta-trains the same way. Its attention reads fused projection
weights directly, so LoRA is a parametrization of the weights (12 blocks,
0.59M of 28M parameters at rank 8), for TabPFN too. TabICL's network takes
no NaN: rows are standardized by the context's statistics, absent values
at the mean, clipped at 8 sigma. On the fixtures this gives AUC 0.827
against 0.820 with its classifier, and the cached in-context forward
equals the training forward only in float32 (with mixed precision on the
CPU, AUC 0.59).
### Label efficiency on GamGam (first study, 2026-10-22)
GamGam validation AUC (%) against the number of labelled training events N
(`data.train_events`, equal per class), one seed, on one 4-GPU A100 node in
an interactive allocation (3.5 h). Each entry is evaluated on 20,000 random
validation events (`data.eval_events=20000`, about +-0.3); on all 246,922,
evaluated for the supervised runs and the leading TabPFN runs, each is 0.05
to 0.41 lower, in the same order (TabPFN-3.5: 80.28, 86.92, 89.01, 0.08 to
0.13 lower, matching the M2 Max's 80.26 and 86.93).
| model | N = 100 | 1000 | 10000 | 50000 |
|---|---|---|---|---|
| edge network, from scratch | 78.00 | 82.74 | 87.66 | |
| edge network, from `Pretrained_GNN` | 84.95 | 87.48 | 89.98 | |
| transformer, from scratch | 72.33 | 83.47 | 87.03 | |
| transformer, from JEPA | 73.25 | 85.10 | 88.60 | |
| transformer, from masked objects | 75.92 | 86.12 | 88.73 | |
| TabPFN-3.5 (rows) | 80.41 | 87.05 | 89.09 | 89.56 |
| TabPFN-3.5 (rows, `relative_phi`, `pair_objects: 4`) | 80.61 | 87.78 | 89.56 | |
| TabPFN-3.5 (rows + JEPA embeddings) | 80.97 | 87.43 | 89.44 | |
| TabPFN-3.5 (rows + masked-object embeddings) | 82.99 | 88.22 | 89.94 | |
| TabPFN-3.5 (rows + `Pretrained_GNN` embeddings) | 87.32 | 88.96 | 90.07 | |
| TabPFN-3.5, released network, direct forward | 80.74 | 86.64 | 89.03 | |
| TabPFN-3.5, meta-trained on Delphes (LoRA) | 80.52 | 84.77 | 88.34 | |
| TabICLv2 | 80.53 | 85.68 | 88.41 | 89.12 |
| TabDPT | 77.48 | 84.71 | 88.42 | |
| our PFN, meta-trained on Delphes | 78.13 | 76.18 | 76.51 | |
Probes (k-NN / linear) of the frozen encoders, the same N:
| encoder | N = 100 | 1000 | 10000 |
|---|---|---|---|
| random | 73.4 / 72.6 | 76.6 / 83.1 | 79.5 / 85.4 |
| JEPA | 73.0 / 73.4 | 78.9 / 83.4 | 81.8 / 87.9 |
| masked objects | 75.0 / 77.5 | 81.7 / 85.7 | 84.4 / 88.5 |
| contrastive | 61.6 / 67.9 | 70.0 / 80.3 | 76.4 / 85.9 |
Setup (every run and its overrides: `scripts/label_efficiency_study.sh`):
the pretraining runs saw the same 4M class-balanced Delphes events for 8
epochs (`data.train_events=4000000`, 50 to 70 min each); the PFN
meta-trained for 25 epochs of 1000 steps (25,000 steps, 95 min) and TabPFN
for 6 epochs of 500 (3,000 steps, 48 min). Supervised runs train for a
fixed number of steps with early stopping off (`train.early_stopping=null`;
batch size, epochs: 100, 1000 for N = 100; 128, 400 for N = 1000; 256, 100
for N = 10000; 20 to 35 min at N = 100, 2 to 4 min at N = 10000) and keep
the epoch of best validation AUC, chosen on the same 20,000 events that are
reported. That choice uses validation labels the in-context models do not
get, and from N = 1000 most supervised runs overfit: their last epoch is far
below the kept one. On all validation events (92% of them not used for the
choice) the kept epochs lose 0.05 to 0.41, no more than the in-context runs
lose, so picking the maximum over noisy evaluations inflates them little:
| supervised run (N = 100 / 1000 / 10000) | kept epoch, 20k | kept epoch, all | last epoch, 20k |
|---|---|---|---|
| edge network, from scratch | 78.00 / 82.74 / 87.66 | 77.80 / 82.69 / 87.42 | 68.88 / 78.88 / 87.65 |
| edge network, from `Pretrained_GNN` | 84.95 / 87.48 / 89.98 | 84.54 / 87.09 / 89.74 | 80.75 / 81.58 / 86.85 |
| transformer, from scratch | 72.33 / 83.47 / 87.03 | 71.99 / 83.35 / 86.95 | 72.33 / 83.44 / 87.03 |
| transformer, from JEPA | 73.25 / 85.10 / 88.60 | 73.07 / 84.96 / 88.45 | 72.34 / 78.82 / 81.46 |
| transformer, from masked objects | 75.92 / 86.12 / 88.73 | 75.73 / 85.84 / 88.60 | 75.92 / 79.73 / 85.62 |
With only N labelled events there is no large labelled set to choose the
epoch on; without the choice (last epoch) every supervised model is below
TabPFN-3.5 as released at every N except `Pretrained_GNN` at N = 100.
With the configs' settings instead (batch size 1024, at most 100 epochs,
early stopping after 5 epochs without training-loss improvement) N = 1000
is one step per epoch, at most 100 steps: the transformer then reaches
70.04 at N = 1000 instead of 83.47. The in-context models take 10 to 30 s
for the 20,000 queries at N = 1000 and 10000 (TabPFN 109 s at N = 50000;
the first run of each, N = 100, also loads the weights).
Findings:
* TabPFN-3.5 as released, without training, comes within 0.4 (N = 1000)
and 0.9 (N = 10000) of fine-tuning the supervised `Pretrained_GNN` (0.2
and 0.7 on all validation events), with its epoch chosen on validation
labels, and above it from N = 1000 without that choice. It beats every
model trained from scratch at every N. TabICL is close behind; TabDPT is
behind at small N. The physics row features add up to 0.7.
* Encoder embeddings help TabPFN most at small N. With the label-free
masked-object encoder: +2.6 at N = 100. With `Pretrained_GNN` (trained
with labels, on Delphes samples that include the GamGam processes): the
best result at every N run (100 to 10000; not run at 50000), +6.9 at
N = 100.
* Meta-training on Delphes tasks does not help. TabPFN with LoRA loses 1.9
at N = 1000 against the same network with zero updates (its direct
forward costs at most 0.4 against the classifier's preprocessing and
ensembling). Our PFN is below TabPFN and gets worse with more context
(76 at N = 1000, inside the meta-training context sizes 16 to 2048): to
investigate before the held-out-processes run, which was not done.
* Of the label-free pretraining objectives, masked-object modelling is the
best, in probes and fine-tuned; JEPA's prediction in embedding space adds
nothing over it here. Both improve fine-tuning the transformer over
training it from scratch (+0.9 to +3.6) but stay below fine-tuning the
edge network from `Pretrained_GNN`. That gap mixes the architecture (from
scratch, the edge network is 5.7 above the transformer at N = 100) with
the objective (label-free or supervised); the like-for-like comparison,
JEPA of the edge network (`configs/delphes_jepa_edge.yaml`), was not run.
* Contrastive pretraining is below a random encoder in the k-NN probe at
every N and in the linear probe at N = 100 and 1000 (67.9 against 72.6,
80.3 against 83.1); at N = 10000 its linear probe is 0.5 above (85.9
against 85.4). Without the momentum-scale augmentation
(`model.scale_jitter=0`, 4 epochs) it is no better (72.4 / 84.3 at
N = 10000).
Not done:
* the held-out-processes meta-training (`task.prior_classes` without the
five GamGam processes) of the PFN, gated on it beating TabPFN, and of
TabPFN and TabICL;
* TabICL meta-training and `model.lora_rank=0`;
* fine-tuning from contrastive pretraining (only probed), and JEPA of the
edge network;
* more seeds, and the test split.
The loss is a weighted mean per class present in the batch, averaged over
those classes, which balances the classes without explicit class weights.
Metrics (accuracy, one-vs-rest AUC averaged over classes, macro precision,
recall and F1) use events with positive weight.
### The TabPFN-like PFN on GamGam (2026-10-27)
GamGam test AUC (%) in context, against the number of labelled context
events N (equal per class), on 50,000 test events, one seed. Every PFN is
meta-trained on the Open Data prior (`configs/opendata_pfn.yaml`). The
earlier PFN (pooled events, contexts of 16 to 4096, random encoder, fp32)
is the checkpoint of epoch 38. The new ones train 48 epochs of 1000 steps
(2.3 h on one A100 each, bf16) with 2 summary tokens, alternating
attention, the per-task momentum scale, the masked-object encoder
(`delphes_masked`), half synthetic tasks, contexts of 16 to 16384 and
batches of 16384 events; each ablation changes one of them.
| PFN | N = 100 | 1000 | 10000 | 100000 |
|---|---|---|---|---|
| earlier | 80.63 | 82.47 | 83.54 | 83.53 |
| earlier, ensemble of 8 | 81.92 | 83.87 | 84.67 | 84.77 |
| new | 79.12 | 80.39 | 83.79 | 84.18 |
| new, ensemble of 8 | 79.67 | 82.26 | 84.46 | 84.77 |
| new, no alternating attention | 77.14 | 81.21 | 84.43 | 84.68 |
| new, random encoder | 76.47 | 81.76 | 83.42 | 83.56 |
| new, no synthetic tasks | 81.28 | 83.14 | 84.72 | 85.45 |
| new, no synthetic tasks, ensemble of 8 | **83.75** | **86.11** | **86.54** | **86.52** |
Validation on held-out Open Data processes after 48 epochs (loss /
accuracy): new 0.772 / 0.722, no alternating 0.800 / 0.712, random encoder
0.809 / 0.711, no synthetic tasks 0.665 / 0.764; all still falling. One
pass takes 1 to 3 s for the 50,000 queries at N = 1000 and 10000 and 35 to
58 s at N = 100000; the ensemble of 8 takes 460 s at N = 100000.
Findings:
* Synthetic tasks hurt at every N (0.9 to 2.8 against the same PFN
without them; 1.8 to 4.1 ensembled) and in validation. Without them the
PFN beats the earlier one at every N, by 1.8 to 2.2 with both
ensembled.
* The ensemble of 8 adds 1.1 to 3.0 to the PFN without synthetic tasks.
* The masked-object encoder adds 2.7 at N = 100 and 0.6 at N = 100000.
* Alternating attention adds 2.0 at N = 100 and costs 0.5 to 0.8 from
N = 1000; it makes training 20% slower.
* Against the validation AUCs of the first study (on other events, so
only roughly comparable): the best PFN is above TabPFN-3.5 on rows
(80.41) and every supervised transformer at N = 100, below the edge
network from `Pretrained_GNN` (84.95) and TabPFN-3.5 with its embeddings
(87.32). At N = 1000 it equals the transformer from masked objects
(86.12) and is 0.9 and 1.4 below TabPFN-3.5 and `Pretrained_GNN`; at
10000 it is 2.2 to 3.4 below all three, and it gains nothing from 10000
to 100000.
Not done: all 1.98M training events as context (out of memory; see the
PFN options above), longer meta-training, and the held-out-processes
prior.
### TabPFN-3.5 options and a larger PFN on GamGam (2026-10-28)
Both PFNs start from the best one above (no synthetic tasks) and add the
TabPFN-3.5 options `pfn_retrieval`, `pfn_qk_norm`, `pfn_query_kv`,
`pfn_fourier=16` and `pfn_ranks`, the cosine schedule (2 warm-up epochs,
learning rate 1e-4) and the faster meta-training. "Small" keeps its size
(2.9M parameters) and trains 40 epochs on one A100 (168 s each). "Large"
has 4 summary tokens joined to one 512-wide token and 12 blocks after the
encoder (42M parameters) and trains 26 epochs on three A100s under
`torchrun` (280 s each, three times the tasks per step). Same test events
and protocol as above, one seed.
| PFN | N = 100 | 1000 | 10000 | 100000 |
|---|---|---|---|---|
| best above, ensemble of 8 | 83.75 | 86.11 | 86.54 | 86.52 |
| small | 81.73 | 84.65 | 86.76 | 87.66 |
| small, ensemble of 8 | 83.15 | 86.20 | 87.63 | 88.20 |
| large | 82.90 | 85.83 | 87.72 | 88.43 |
| large, ensemble of 8 | **84.06** | **86.80** | **88.24** | **88.62** |
Validation on held-out Open Data processes in the last epoch (loss /
accuracy): small 0.599 / 0.789, large 0.561 / 0.799 (best above 0.665 /
0.764). At N = 100000 one pass takes 21 s (small) and 87 s (large), the
ensemble of 8 169 s and 698 s; the large ensemble peaks at 13 GiB.
Findings:
* The options, schedule and faster training add 0.5 to 2.2 to one pass of
the best PFN above (81.28, 83.14, 84.72, 85.45), and the AUC now rises
from 10000 to 100000 (by 0.4 to 0.9) instead of staying flat.
* The larger PFN adds 0.8 to 1.2 more. One pass of it is 0.9 and 0.3
below the earlier ensemble at N = 100 and 1000 and 1.2 and 1.9 above it
from 10000; ensembled, it gains 0.3 to 2.1 on the earlier ensemble.
* The ensemble adds less than before: 0.2 to 1.6.
* The options were not ablated one by one, and the two runs differ in
epochs and GPUs as well as size.
* Against the validation AUCs of the first study (only roughly
comparable): at N = 100 the large ensemble is 3.7 above TabPFN-3.5 on
rows and 0.9 below `Pretrained_GNN`; at 1000 it is 0.3 and 0.7 below
them, at 10000 0.9 and 1.7 below.
Not done: all 1.98M training events as context, one ablation per option,
more seeds.
## Performance
Graphs differ in size, which suits neither `torch.compile` nor CUDA graphs
out of the box:
* `torch.compile(dynamic=True)` treats node and edge counts as symbolic
sizes, so one compiled graph serves all batches without padding.
* CUDA graph replay needs fixed shapes. Batches are copied into buffers
rounded up to multiples of 512 nodes and 4096 edges; the remainder forms 64
padding graphs connected only to themselves, so the real graphs' outputs do
not change. A step is captured once per bucket, a few per epoch. The
optimizer runs in Adam's `capturable` form with device-tensor learning
rates, which the scheduler updates in place.
* bfloat16 autocast covers the forward pass; logits, loss, graph sums (which
divide by edge counts) and the optimizer stay float32.
GamGam training step, 1024-event batches (about 5,600 nodes and 30,000
edges), one A100-SXM4-80GB, `OMP_NUM_THREADS=8`, four loader workers (step
times measured before the move to flat modules, epoch times after):
| Mode | ms per batch | Epoch with validation |
| --- | ---: | ---: |
| float32 eager | 19.2 | 37 s |
| CUDA graphs + TF32 | 8.5 | |
| `torch.compile` | 13.0 | |
| compile + CUDA graphs + TF32 | 6.1 | |
| compile + CUDA graphs + bfloat16 (`train.fast`) | 5.9 | 9 s (first: 59 s, compiling) |
Eager training is bound by kernel launches (about a thousand small kernels
per step), not by arithmetic; bfloat16 alone is slower because of the extra
cast kernels, which `torch.compile` fuses. Results of the fast modes agree
with float32 to rounding, not bitwise (validation AUC 0.895 against 0.897
after three epochs). Preparing the 2.5M GamGam events with 8 workers takes
about 9 s on a login node (40 s was seen on a compute node, likely a cold
file cache).
PyTorch sizes its CPU thread pool to every visible core; on a 128-core node
that makes batch construction several times slower and CPU reductions
non-reproducible, so set `OMP_NUM_THREADS`. Single-threaded CPU training is
bitwise reproducible.
## ATLAS Open Data GamGam versus Ho et al. (2026)
The main benchmark reproduces the from-scratch baseline of Table 3 (5
classes, 2.5M events, 80:10:10 split, five seeds). Choices the paper does not
specify:
* electrons and muons are split from `lep_*` by `lep_type` (11, 13);
* b-tagging is `jet_MV2c10 > 0.8244273`, the 70% working point;
* momenta and energies are scaled by 1e-4 (MeV to the pretraining unit of
10 GeV);
* the fold is `eventNumber % 10`: folds 0-7 train, 8 validates, 9 tests;
* training is unweighted (23% of ttH events have `mcWeight = -1`).
The paper's stopping rule (5 epochs without training-loss improvement) never
fires within 200 epochs on this task; validation AUC peaks earlier (epoch 67
for seed 1) and `best.pt` keeps that epoch.
| Test split | Accuracy | AUC | F1 | Precision | Recall |
| --- | ---: | ---: | ---: | ---: | ---: |
| Paper (5 seeds) | 71.8 ± 0.2 | 91.2 | 63.0 | 62.2 | 66.3 |
| This package, seed 1 (CUDA graphs + TF32) | 71.1 | 91.3 | 62.6 | 62.4 | 66.7 |