# Design ## Modules | Module | Contents | | --- | --- | | `config.py` | the experiment dataclasses, YAML loading and overrides | | `data.py` | ROOT reading, Hugging Face download, derived branches, object features, `PackedEvents` and its cache | | `loader.py` | `EventLoader`: deterministic shuffling, DDP sharding, collation workers | | `graphs.py` | fully connected `TensorGraph` batches, edge features, CUDA-graph padding buffers | | `sequences.py` | padded token batches for the transformer and Deep Sets, event augmentations | | `models.py` | `EdgeNetwork`, `FineTunedEdgeNetwork`, `ObjectTransformer`, `TransformerClassifier`, `DeepSets`, checkpoint loading | | `jepa.py` | JEPA pretraining of the transformer or edge network encoder (encoding or kinematic targets), event embeddings | | `contrastive.py` | contrastive (SimCLR) pretraining of the same encoders | | `probe.py` | k-NN and linear probes of frozen encoders, linear CKA between them | | `pfn.py` | our prior-fitted network over events and the task prior of meta-training | | `tabular.py` | event rows and encoder embeddings for TabPFN, TabICL and TabDPT; LoRA meta-training of TabPFN and TabICL | | `tasks.py` | multiclass and binary losses and metrics, the pretraining and PFN losses | | `train.py` | `Trainer`, early stopping, checkpoints | | `cuda_graphs.py` | captured training steps | | `distributed.py` | `torchrun` process group and collectives | | `inference.py` | model outputs over a loader, prediction files | | `export.py` | ONNX export | | `cli.py` | the six commands | ## Data flow `prepare` reads each ROOT file in chunks of 100,000 entries (in parallel with `data.prepare_workers`) and stores every event's objects as rows of one `[objects, 7]` tensor with offsets, plus per-event labels, folds, weights and source file and entry. Nothing in the cache depends on the model: graphs and token sequences are built per batch, in the loader workers. A batch of the edge network is a set of fully connected graphs in flat edge-index form (`edge_src`, `edge_dst`, `node_batch`, `edge_batch`). Edges are directed, source-major and without self-loops except for one-node graphs, with features `[deta, dphi, dR]`. These conventions, the seven node columns, the MLP layout and the parameter names are those of the original ROOT-GNN, so the pretrained checkpoints in `Pretrained_GNN/` load without conversion. The transformer and Deep Sets take each event's objects as tokens `[events, objects, 7]`, zero-padded to the largest event of the batch, with a mask of real objects. A token is a linear map of the kinematic columns (`phi` as its cosine and sine) plus a learned embedding of `node_type`. The objects are stored by type (jets, electrons, muons, photons, MET), so a position embedding would mostly relearn the type; without one, neither model depends on object order, as with the edge network. * The transformer is a stack of pre-norm attention blocks, a mean over real objects and a linear classifier. With `model.pair_bias` (as in the Particle Transformer, Qu et al. 2022), an MLP of the pair features `[deta, dphi, dR, log(1 + m^2)]` (the edge features plus the invariant mass of the pair, objects taken as massless) adds a bias per head to the attention logits, so the transformer sees the same pairwise angles as the edge network. * Deep Sets (Zaheer et al. 2017; the particle flow network of Komiske et al. 2019) applies an MLP to each object, sums over real objects and applies an MLP to the sum: the baseline without object interactions. ### JEPA pretraining `model.name: jepa` pretrains an object encoder without labels, as in I-JEPA (Assran et al. 2023), on the 12 Delphes samples of the supervised pretraining. `model.backbone` selects the encoder: the transformer's (`configs/delphes_jepa.yaml`) or the edge network, of the size of the supervised pretrained models (`configs/delphes_jepa_edge.yaml`), whose object encodings are its node states after the last processing step and whose context is the fully connected graph of the visible objects only. * a random `model.mask_fraction` of each event's objects (rounded, at least one, and at least one left visible) is hidden; * the context encoder sees the visible objects; the target encoder, an exponential moving average of it (momentum rising linearly from `model.ema_momentum` to 1 over training) without gradients, sees all objects; * a predictor (`model.predictor_layers` attention blocks over the visible objects' encodings and one query per hidden object) predicts the hidden objects' layer-normalized target encodings, with a mean squared error. A query is told only the hidden object's type and its index among the objects of that type (their pt order), not its kinematics; pair features are used only between visible objects. For the edge network, its decoded global state (what its classifier reads) is one more predictor input, so the whole network is pretrained. The history reports `target_spread`, the standard deviation of the target encodings over a batch's objects averaged over dimensions, which falls towards zero if the representation collapses. The validation loss compares predictions with a moving target, so it does not select epochs: fine-tuning starts from `last.pt` (`configs/gamgam_jepa_finetune.yaml` for the transformer, `configs/gamgam_finetune.yaml` with `model.pretrained` for the edge network; encoder sizes must match). With `model.freeze_backbone=true` only the classifier is trained (a linear probe); `model.pretrained=null` trains the same architecture from scratch. `model.variance_weight` adds the VICReg variance term (Bardes et al. 2022): the mean shortfall below 1 of each dimension's standard deviation over the batch's context encodings (and the edge network's global states). First checks (one A100, `train.fast=true`, 2 pretraining epochs on Delphes fold 2, then 2 GamGam epochs, validation AUC, seed 1; not converged): | GamGam model | Validation AUC | | --- | --- | | transformer from scratch / from JEPA | 0.898 / 0.904 | | edge network from scratch | 0.895 | | edge network from the supervised pretrained model | 0.907 | | edge network from JEPA, last node states only | 0.845 | | edge network from JEPA, mean node states | 0.869 | | the same with `variance_weight` 0.1 / 1 | 0.874 / 0.810 | The edge network's JEPA encodings collapse without the variance term (`target_spread` 0.09 against about 1 with it), and none of the edge network variants yet beats training from scratch. ### Contrastive and masked-object pretraining, and probes Two more label-free objectives pretrain the same encoders on the same samples, and their encoders load the same way (`context_encoder`): * `contrastive.py` (`configs/delphes_contrastive.yaml`): SimCLR, as JetCLR and RS3L for jets. Two views of each event (random phi rotation, eta flip, momentum scale within `model.scale_jitter`, each object but the first dropped with `model.object_dropout`) are embedded and projected; the NT-Xent loss at `model.temperature` contrasts them with the other events of the batch. `embedding_spread` watches for collapse. * `model.jepa_target: features` (`configs/delphes_masked.yaml`): masked object modelling (Golling et al. 2024). JEPA's masking and predictor predict the hidden objects' kinematics instead of their target encodings, with no target encoder: the comparison with JEPA isolates predicting in embedding space. `gnn4colliders probe` compares encoders without fine-tuning (as jBOT): k-NN (cosine, `model.probe_neighbours`) and linear (L-BFGS logistic regression, `model.probe_l2`) classifiers of frozen event embeddings, trained on `data.train_events` events. A randomly initialized encoder (`model.pretrained=null`) is the reference; supervised checkpoints probe too. Results on GamGam: "Label efficiency on GamGam" below. `scripts/compare_encoders.py` compares encoders of one architecture by linear CKA (Kornblith et al. 2019), as the paper does: of their event embeddings (a matrix), and layer by layer against the first one (the mean over objects after each transformer block, or the edge network's mean node states after each step, then the event embedding). With a random encoder first, it shows where each objective moves the representation. ### In-context models: TabPFN, TabICL, TabDPT and a collider PFN Both classify a dataset's events given `data.train_events` labelled training events (an equal number per class) as context, without training on that dataset; `cli.in_context_forward` builds them for `evaluate` and `predict`. * `tabular.py`: TabPFN (Prior Labs; `model.tabpfn_version`, default 3.5) on fixed-width rows: the leading `model.slots` objects of each type (6 jets, 2 electrons, 2 muons, 3 photons, MET) with their six kinematic columns, NaN when absent, and the number of objects of each type. * `pfn.py`: our own prior-fitted network. The object transformer (which can start from the JEPA encoder) embeds each event; the context events get a label embedding; `model.pfn_layers` attention blocks run over a task's events, every event attending to the context events only, so queries are independent and the context's keys and values are computed once. It is meta-trained (`task.name: pfn`) on tasks drawn from the Delphes samples: 2-10 classes of `task.prior_classes` in random order, Dirichlet class proportions, 16-2048 context events (log-uniform), 256 queries, and per event a random phi rotation and eta flip, per task a momentum and energy scale within `task.energy_jitter`. The GamGam processes are among the Delphes samples; `task.prior_classes=[1,6,7,8,9,10,11]` leaves them out. `configs/opendata_pfn.yaml` (written by `scripts/opendata_prior.py`) meta-trains it on the ATLAS Open Data 2025 MC instead: 925 classes of 331 processes, H -> gamma gamma left out, a tenth of the processes held out for validation, classes mixing 1 to 3 processes. * Options that bring the PFN closer to TabPFN, all off by default (the default model's outputs match the earlier code to 2e-7): `model.pfn_summary_tokens` learned tokens carry each event through the encoder, with a context event's label added at the input instead of after the encoder; `model.pfn_alternating` adds attention across the task's events (summary tokens, context only) after every encoder block, as TabPFN alternates attention over features and over rows, so an event's encoding depends on the task; `model.pfn_normalize` scales each task's momenta so that the context's geometric-mean object pT is 50 GeV (as TabPFN standardizes features by the context); `model.pfn_ensemble` averages the class probabilities of passes with the events rotated in phi, flipped in eta and the class labels permuted (as TabPFN ensembles over class orders and preprocessings); and `task.synthetic_tasks` makes a fraction of the training tasks synthetic, as TabPFN's prior draws random functions: events of 1 to 3 processes ranked by a random network (16 tanh units) of 1 to 4 of 21 event observables (per object type the count, scalar sum pT, leading pT and mass; the b-tag count), cut into classes of Dirichlet fractions. A large context is encoded in chunks, one block at a time, but the alternating PFN keeps every context event's tokens and the keys and values of every block on the GPU: 100k events fit, all 1.98M GamGam training events do not (out of memory on a 40 GB A100). * Options from TabPFN-3.5 (219M parameters, 79 times ours; read from its checkpoint and code), all off by default: `model.pfn_retrieval` makes the logits the log of an attention-weighted vote of the context's labels (TabPFN's many-class decoder) instead of a linear classifier; `model.pfn_qk_norm` RMS-norms queries and keys per head and scales the queries by a learned factor times the log of the number of keys (TabPFN-3's scalable softmax), in all attention across events; `model.pfn_query_kv` lets the queries read the context through one key/value head of their own, so the memory of a large context is `n_heads` times smaller; `model.pfn_concat` joins the summary tokens into one token `pfn_summary_tokens * hid_size` wide for the blocks after the encoder (TabPFN-3.5 joins 8 x 128 to 1024), with the label embedded again; `model.pfn_fourier` learned frequencies of each object's log pT, eta and phi and `model.pfn_ranks` (each object's pT and energy ranked among the context's objects of its type, as TabPFN adds ECDF ranks) are added to the object embeddings, through maps that start at zero. `train.lr_schedule: cosine` warms up over `train.warmup_epochs`. * Meta-training speed: training batches are encoded in 4 parts of events with similar numbers of objects (only 37% of the padded object tokens were real), tasks are sampled in the loader workers rather than all at the start of an epoch, and under `torchrun` all ranks draw the same context size for a step, so that they wait less for each other. Steps are drawn from a generator per seed, epoch, rank and step, so the tasks differ from those of runs before this change. In-context inference sorts the context by number of objects and attends in chunks of `data.batch_size` events. * `tabular.py` also runs TabICLv2 (Qu et al., ICML 2026; BSD-3, open pretraining code) on the same rows, as a second tabular in-context model. Rows can measure azimuths from the leading object's (`model.relative_phi`) and end with the invariant mass and Delta R of every pair of the `model.pair_objects` leading objects by pt (energies are `pt cosh(eta)`, so objects are massless; a pair with MET gives a transverse-like mass). Both are off by default, so the default rows are unchanged. * `tabular.py` also runs TabDPT (Ma et al. 2024; Apache-2.0), trained on real tables rather than a synthetic prior; with `model.tabdpt_context_size` each event's context is its nearest context events (faiss retrieval), beyond what fits in one pass. It computes in float64 in places (on a Mac it runs on the CPU), and on macOS faiss and torch crash together unless `OMP_NUM_THREADS=1`. * Tabular models can see, instead of or beside the rows, the event embeddings of a pretrained object encoder (`model.tabular_inputs: embeddings` or `both`, the encoder of `model.backbone` and `model.pretrained`): physics-aware embeddings under a generic tabular prior. In meta-training the encoder trains with the LoRA updates unless `model.freeze_backbone`. * `tabular.py` also meta-trains the TabPFN-3.5 network itself on the PFN's tasks (`model.name: tabpfn`, `task.name: pfn`), on the rows above (built after the augmentation), starting from Prior Labs' weights. Calling the network directly skips TabPFN's preprocessing and ensembling, which on the GamGam fixtures cost nothing (validation AUC 0.852 against 0.843 with `predict_proba`, 61 events). By default the network is frozen and the four attention projections and two MLP layers of each of its 24 in-context blocks get rank-8 LoRA updates (2.8M parameters, zero at the start, so training starts from TabPFN as released); `model.lora_rank=0` trains all weights. In context, the context's keys and values are cached (the same logits as one pass over context and queries, to 2e-5). It separates what Prior Labs' prior and what collider tasks contribute: TabPFN as released, TabPFN meta-trained here, and our PFN. TabICL (`configs/delphes_tabicl.yaml`; BSD-3, so meta-trained weights can be shared) meta-trains the same way. Its attention reads fused projection weights directly, so LoRA is a parametrization of the weights (12 blocks, 0.59M of 28M parameters at rank 8), for TabPFN too. TabICL's network takes no NaN: rows are standardized by the context's statistics, absent values at the mean, clipped at 8 sigma. On the fixtures this gives AUC 0.827 against 0.820 with its classifier, and the cached in-context forward equals the training forward only in float32 (with mixed precision on the CPU, AUC 0.59). ### Label efficiency on GamGam (first study, 2026-10-22) GamGam validation AUC (%) against the number of labelled training events N (`data.train_events`, equal per class), one seed, on one 4-GPU A100 node in an interactive allocation (3.5 h). Each entry is evaluated on 20,000 random validation events (`data.eval_events=20000`, about +-0.3); on all 246,922, evaluated for the supervised runs and the leading TabPFN runs, each is 0.05 to 0.41 lower, in the same order (TabPFN-3.5: 80.28, 86.92, 89.01, 0.08 to 0.13 lower, matching the M2 Max's 80.26 and 86.93). | model | N = 100 | 1000 | 10000 | 50000 | |---|---|---|---|---| | edge network, from scratch | 78.00 | 82.74 | 87.66 | | | edge network, from `Pretrained_GNN` | 84.95 | 87.48 | 89.98 | | | transformer, from scratch | 72.33 | 83.47 | 87.03 | | | transformer, from JEPA | 73.25 | 85.10 | 88.60 | | | transformer, from masked objects | 75.92 | 86.12 | 88.73 | | | TabPFN-3.5 (rows) | 80.41 | 87.05 | 89.09 | 89.56 | | TabPFN-3.5 (rows, `relative_phi`, `pair_objects: 4`) | 80.61 | 87.78 | 89.56 | | | TabPFN-3.5 (rows + JEPA embeddings) | 80.97 | 87.43 | 89.44 | | | TabPFN-3.5 (rows + masked-object embeddings) | 82.99 | 88.22 | 89.94 | | | TabPFN-3.5 (rows + `Pretrained_GNN` embeddings) | 87.32 | 88.96 | 90.07 | | | TabPFN-3.5, released network, direct forward | 80.74 | 86.64 | 89.03 | | | TabPFN-3.5, meta-trained on Delphes (LoRA) | 80.52 | 84.77 | 88.34 | | | TabICLv2 | 80.53 | 85.68 | 88.41 | 89.12 | | TabDPT | 77.48 | 84.71 | 88.42 | | | our PFN, meta-trained on Delphes | 78.13 | 76.18 | 76.51 | | Probes (k-NN / linear) of the frozen encoders, the same N: | encoder | N = 100 | 1000 | 10000 | |---|---|---|---| | random | 73.4 / 72.6 | 76.6 / 83.1 | 79.5 / 85.4 | | JEPA | 73.0 / 73.4 | 78.9 / 83.4 | 81.8 / 87.9 | | masked objects | 75.0 / 77.5 | 81.7 / 85.7 | 84.4 / 88.5 | | contrastive | 61.6 / 67.9 | 70.0 / 80.3 | 76.4 / 85.9 | Setup (every run and its overrides: `scripts/label_efficiency_study.sh`): the pretraining runs saw the same 4M class-balanced Delphes events for 8 epochs (`data.train_events=4000000`, 50 to 70 min each); the PFN meta-trained for 25 epochs of 1000 steps (25,000 steps, 95 min) and TabPFN for 6 epochs of 500 (3,000 steps, 48 min). Supervised runs train for a fixed number of steps with early stopping off (`train.early_stopping=null`; batch size, epochs: 100, 1000 for N = 100; 128, 400 for N = 1000; 256, 100 for N = 10000; 20 to 35 min at N = 100, 2 to 4 min at N = 10000) and keep the epoch of best validation AUC, chosen on the same 20,000 events that are reported. That choice uses validation labels the in-context models do not get, and from N = 1000 most supervised runs overfit: their last epoch is far below the kept one. On all validation events (92% of them not used for the choice) the kept epochs lose 0.05 to 0.41, no more than the in-context runs lose, so picking the maximum over noisy evaluations inflates them little: | supervised run (N = 100 / 1000 / 10000) | kept epoch, 20k | kept epoch, all | last epoch, 20k | |---|---|---|---| | edge network, from scratch | 78.00 / 82.74 / 87.66 | 77.80 / 82.69 / 87.42 | 68.88 / 78.88 / 87.65 | | edge network, from `Pretrained_GNN` | 84.95 / 87.48 / 89.98 | 84.54 / 87.09 / 89.74 | 80.75 / 81.58 / 86.85 | | transformer, from scratch | 72.33 / 83.47 / 87.03 | 71.99 / 83.35 / 86.95 | 72.33 / 83.44 / 87.03 | | transformer, from JEPA | 73.25 / 85.10 / 88.60 | 73.07 / 84.96 / 88.45 | 72.34 / 78.82 / 81.46 | | transformer, from masked objects | 75.92 / 86.12 / 88.73 | 75.73 / 85.84 / 88.60 | 75.92 / 79.73 / 85.62 | With only N labelled events there is no large labelled set to choose the epoch on; without the choice (last epoch) every supervised model is below TabPFN-3.5 as released at every N except `Pretrained_GNN` at N = 100. With the configs' settings instead (batch size 1024, at most 100 epochs, early stopping after 5 epochs without training-loss improvement) N = 1000 is one step per epoch, at most 100 steps: the transformer then reaches 70.04 at N = 1000 instead of 83.47. The in-context models take 10 to 30 s for the 20,000 queries at N = 1000 and 10000 (TabPFN 109 s at N = 50000; the first run of each, N = 100, also loads the weights). Findings: * TabPFN-3.5 as released, without training, comes within 0.4 (N = 1000) and 0.9 (N = 10000) of fine-tuning the supervised `Pretrained_GNN` (0.2 and 0.7 on all validation events), with its epoch chosen on validation labels, and above it from N = 1000 without that choice. It beats every model trained from scratch at every N. TabICL is close behind; TabDPT is behind at small N. The physics row features add up to 0.7. * Encoder embeddings help TabPFN most at small N. With the label-free masked-object encoder: +2.6 at N = 100. With `Pretrained_GNN` (trained with labels, on Delphes samples that include the GamGam processes): the best result at every N run (100 to 10000; not run at 50000), +6.9 at N = 100. * Meta-training on Delphes tasks does not help. TabPFN with LoRA loses 1.9 at N = 1000 against the same network with zero updates (its direct forward costs at most 0.4 against the classifier's preprocessing and ensembling). Our PFN is below TabPFN and gets worse with more context (76 at N = 1000, inside the meta-training context sizes 16 to 2048): to investigate before the held-out-processes run, which was not done. * Of the label-free pretraining objectives, masked-object modelling is the best, in probes and fine-tuned; JEPA's prediction in embedding space adds nothing over it here. Both improve fine-tuning the transformer over training it from scratch (+0.9 to +3.6) but stay below fine-tuning the edge network from `Pretrained_GNN`. That gap mixes the architecture (from scratch, the edge network is 5.7 above the transformer at N = 100) with the objective (label-free or supervised); the like-for-like comparison, JEPA of the edge network (`configs/delphes_jepa_edge.yaml`), was not run. * Contrastive pretraining is below a random encoder in the k-NN probe at every N and in the linear probe at N = 100 and 1000 (67.9 against 72.6, 80.3 against 83.1); at N = 10000 its linear probe is 0.5 above (85.9 against 85.4). Without the momentum-scale augmentation (`model.scale_jitter=0`, 4 epochs) it is no better (72.4 / 84.3 at N = 10000). Not done: * the held-out-processes meta-training (`task.prior_classes` without the five GamGam processes) of the PFN, gated on it beating TabPFN, and of TabPFN and TabICL; * TabICL meta-training and `model.lora_rank=0`; * fine-tuning from contrastive pretraining (only probed), and JEPA of the edge network; * more seeds, and the test split. The loss is a weighted mean per class present in the batch, averaged over those classes, which balances the classes without explicit class weights. Metrics (accuracy, one-vs-rest AUC averaged over classes, macro precision, recall and F1) use events with positive weight. ### The TabPFN-like PFN on GamGam (2026-10-27) GamGam test AUC (%) in context, against the number of labelled context events N (equal per class), on 50,000 test events, one seed. Every PFN is meta-trained on the Open Data prior (`configs/opendata_pfn.yaml`). The earlier PFN (pooled events, contexts of 16 to 4096, random encoder, fp32) is the checkpoint of epoch 38. The new ones train 48 epochs of 1000 steps (2.3 h on one A100 each, bf16) with 2 summary tokens, alternating attention, the per-task momentum scale, the masked-object encoder (`delphes_masked`), half synthetic tasks, contexts of 16 to 16384 and batches of 16384 events; each ablation changes one of them. | PFN | N = 100 | 1000 | 10000 | 100000 | |---|---|---|---|---| | earlier | 80.63 | 82.47 | 83.54 | 83.53 | | earlier, ensemble of 8 | 81.92 | 83.87 | 84.67 | 84.77 | | new | 79.12 | 80.39 | 83.79 | 84.18 | | new, ensemble of 8 | 79.67 | 82.26 | 84.46 | 84.77 | | new, no alternating attention | 77.14 | 81.21 | 84.43 | 84.68 | | new, random encoder | 76.47 | 81.76 | 83.42 | 83.56 | | new, no synthetic tasks | 81.28 | 83.14 | 84.72 | 85.45 | | new, no synthetic tasks, ensemble of 8 | **83.75** | **86.11** | **86.54** | **86.52** | Validation on held-out Open Data processes after 48 epochs (loss / accuracy): new 0.772 / 0.722, no alternating 0.800 / 0.712, random encoder 0.809 / 0.711, no synthetic tasks 0.665 / 0.764; all still falling. One pass takes 1 to 3 s for the 50,000 queries at N = 1000 and 10000 and 35 to 58 s at N = 100000; the ensemble of 8 takes 460 s at N = 100000. Findings: * Synthetic tasks hurt at every N (0.9 to 2.8 against the same PFN without them; 1.8 to 4.1 ensembled) and in validation. Without them the PFN beats the earlier one at every N, by 1.8 to 2.2 with both ensembled. * The ensemble of 8 adds 1.1 to 3.0 to the PFN without synthetic tasks. * The masked-object encoder adds 2.7 at N = 100 and 0.6 at N = 100000. * Alternating attention adds 2.0 at N = 100 and costs 0.5 to 0.8 from N = 1000; it makes training 20% slower. * Against the validation AUCs of the first study (on other events, so only roughly comparable): the best PFN is above TabPFN-3.5 on rows (80.41) and every supervised transformer at N = 100, below the edge network from `Pretrained_GNN` (84.95) and TabPFN-3.5 with its embeddings (87.32). At N = 1000 it equals the transformer from masked objects (86.12) and is 0.9 and 1.4 below TabPFN-3.5 and `Pretrained_GNN`; at 10000 it is 2.2 to 3.4 below all three, and it gains nothing from 10000 to 100000. Not done: all 1.98M training events as context (out of memory; see the PFN options above), longer meta-training, and the held-out-processes prior. ### TabPFN-3.5 options and a larger PFN on GamGam (2026-10-28) Both PFNs start from the best one above (no synthetic tasks) and add the TabPFN-3.5 options `pfn_retrieval`, `pfn_qk_norm`, `pfn_query_kv`, `pfn_fourier=16` and `pfn_ranks`, the cosine schedule (2 warm-up epochs, learning rate 1e-4) and the faster meta-training. "Small" keeps its size (2.9M parameters) and trains 40 epochs on one A100 (168 s each). "Large" has 4 summary tokens joined to one 512-wide token and 12 blocks after the encoder (42M parameters) and trains 26 epochs on three A100s under `torchrun` (280 s each, three times the tasks per step). Same test events and protocol as above, one seed. | PFN | N = 100 | 1000 | 10000 | 100000 | |---|---|---|---|---| | best above, ensemble of 8 | 83.75 | 86.11 | 86.54 | 86.52 | | small | 81.73 | 84.65 | 86.76 | 87.66 | | small, ensemble of 8 | 83.15 | 86.20 | 87.63 | 88.20 | | large | 82.90 | 85.83 | 87.72 | 88.43 | | large, ensemble of 8 | **84.06** | **86.80** | **88.24** | **88.62** | Validation on held-out Open Data processes in the last epoch (loss / accuracy): small 0.599 / 0.789, large 0.561 / 0.799 (best above 0.665 / 0.764). At N = 100000 one pass takes 21 s (small) and 87 s (large), the ensemble of 8 169 s and 698 s; the large ensemble peaks at 13 GiB. Findings: * The options, schedule and faster training add 0.5 to 2.2 to one pass of the best PFN above (81.28, 83.14, 84.72, 85.45), and the AUC now rises from 10000 to 100000 (by 0.4 to 0.9) instead of staying flat. * The larger PFN adds 0.8 to 1.2 more. One pass of it is 0.9 and 0.3 below the earlier ensemble at N = 100 and 1000 and 1.2 and 1.9 above it from 10000; ensembled, it gains 0.3 to 2.1 on the earlier ensemble. * The ensemble adds less than before: 0.2 to 1.6. * The options were not ablated one by one, and the two runs differ in epochs and GPUs as well as size. * Against the validation AUCs of the first study (only roughly comparable): at N = 100 the large ensemble is 3.7 above TabPFN-3.5 on rows and 0.9 below `Pretrained_GNN`; at 1000 it is 0.3 and 0.7 below them, at 10000 0.9 and 1.7 below. Not done: all 1.98M training events as context, one ablation per option, more seeds. ## Performance Graphs differ in size, which suits neither `torch.compile` nor CUDA graphs out of the box: * `torch.compile(dynamic=True)` treats node and edge counts as symbolic sizes, so one compiled graph serves all batches without padding. * CUDA graph replay needs fixed shapes. Batches are copied into buffers rounded up to multiples of 512 nodes and 4096 edges; the remainder forms 64 padding graphs connected only to themselves, so the real graphs' outputs do not change. A step is captured once per bucket, a few per epoch. The optimizer runs in Adam's `capturable` form with device-tensor learning rates, which the scheduler updates in place. * bfloat16 autocast covers the forward pass; logits, loss, graph sums (which divide by edge counts) and the optimizer stay float32. GamGam training step, 1024-event batches (about 5,600 nodes and 30,000 edges), one A100-SXM4-80GB, `OMP_NUM_THREADS=8`, four loader workers (step times measured before the move to flat modules, epoch times after): | Mode | ms per batch | Epoch with validation | | --- | ---: | ---: | | float32 eager | 19.2 | 37 s | | CUDA graphs + TF32 | 8.5 | | | `torch.compile` | 13.0 | | | compile + CUDA graphs + TF32 | 6.1 | | | compile + CUDA graphs + bfloat16 (`train.fast`) | 5.9 | 9 s (first: 59 s, compiling) | Eager training is bound by kernel launches (about a thousand small kernels per step), not by arithmetic; bfloat16 alone is slower because of the extra cast kernels, which `torch.compile` fuses. Results of the fast modes agree with float32 to rounding, not bitwise (validation AUC 0.895 against 0.897 after three epochs). Preparing the 2.5M GamGam events with 8 workers takes about 9 s on a login node (40 s was seen on a compute node, likely a cold file cache). PyTorch sizes its CPU thread pool to every visible core; on a 128-core node that makes batch construction several times slower and CPU reductions non-reproducible, so set `OMP_NUM_THREADS`. Single-threaded CPU training is bitwise reproducible. ## ATLAS Open Data GamGam versus Ho et al. (2026) The main benchmark reproduces the from-scratch baseline of Table 3 (5 classes, 2.5M events, 80:10:10 split, five seeds). Choices the paper does not specify: * electrons and muons are split from `lep_*` by `lep_type` (11, 13); * b-tagging is `jet_MV2c10 > 0.8244273`, the 70% working point; * momenta and energies are scaled by 1e-4 (MeV to the pretraining unit of 10 GeV); * the fold is `eventNumber % 10`: folds 0-7 train, 8 validates, 9 tests; * training is unweighted (23% of ttH events have `mcWeight = -1`). The paper's stopping rule (5 epochs without training-loss improvement) never fires within 200 epochs on this task; validation AUC peaks earlier (epoch 67 for seed 1) and `best.pt` keeps that epoch. | Test split | Accuracy | AUC | F1 | Precision | Recall | | --- | ---: | ---: | ---: | ---: | ---: | | Paper (5 seeds) | 71.8 ± 0.2 | 91.2 | 63.0 | 62.2 | 66.3 | | This package, seed 1 (CUDA graphs + TF32) | 71.1 | 91.3 | 62.6 | 62.4 | 66.7 |