Claude Opus 5.5
feat: TabPFN-3.5 options for the PFN, a cosine schedule and faster meta-training
71aefe8 |
Download docs/design.md from HWresearch/GNN4Colliders: direct link, hf CLI and curl.
- Browser
- Download file 31 kB
-
https://huggingface.co/HWresearch/GNN4Colliders/resolve/main/docs/design.md
- Command line
-
hf download hf://HWresearch/GNN4Colliders/docs/design.md
-
curl -L -o design.md https://huggingface.co/HWresearch/GNN4Colliders/resolve/main/docs/design.md
31 kB
| # Design | |
| ## Modules | |
| | Module | Contents | | |
| | --- | --- | | |
| | `config.py` | the experiment dataclasses, YAML loading and overrides | | |
| | `data.py` | ROOT reading, Hugging Face download, derived branches, object features, `PackedEvents` and its cache | | |
| | `loader.py` | `EventLoader`: deterministic shuffling, DDP sharding, collation workers | | |
| | `graphs.py` | fully connected `TensorGraph` batches, edge features, CUDA-graph padding buffers | | |
| | `sequences.py` | padded token batches for the transformer and Deep Sets, event augmentations | | |
| | `models.py` | `EdgeNetwork`, `FineTunedEdgeNetwork`, `ObjectTransformer`, `TransformerClassifier`, `DeepSets`, checkpoint loading | | |
| | `jepa.py` | JEPA pretraining of the transformer or edge network encoder (encoding or kinematic targets), event embeddings | | |
| | `contrastive.py` | contrastive (SimCLR) pretraining of the same encoders | | |
| | `probe.py` | k-NN and linear probes of frozen encoders, linear CKA between them | | |
| | `pfn.py` | our prior-fitted network over events and the task prior of meta-training | | |
| | `tabular.py` | event rows and encoder embeddings for TabPFN, TabICL and TabDPT; LoRA meta-training of TabPFN and TabICL | | |
| | `tasks.py` | multiclass and binary losses and metrics, the pretraining and PFN losses | | |
| | `train.py` | `Trainer`, early stopping, checkpoints | | |
| | `cuda_graphs.py` | captured training steps | | |
| | `distributed.py` | `torchrun` process group and collectives | | |
| | `inference.py` | model outputs over a loader, prediction files | | |
| | `export.py` | ONNX export | | |
| | `cli.py` | the six commands | | |
| ## Data flow | |
| `prepare` reads each ROOT file in chunks of 100,000 entries (in parallel with | |
| `data.prepare_workers`) and stores every event's objects as rows of one | |
| `[objects, 7]` tensor with offsets, plus per-event labels, folds, weights and | |
| source file and entry. Nothing in the cache depends on the model: graphs and | |
| token sequences are built per batch, in the loader workers. | |
| A batch of the edge network is a set of fully connected graphs in flat | |
| edge-index form (`edge_src`, `edge_dst`, `node_batch`, `edge_batch`). Edges | |
| are directed, source-major and without self-loops except for one-node graphs, | |
| with features `[deta, dphi, dR]`. These conventions, the seven node columns, | |
| the MLP layout and the parameter names are those of the original ROOT-GNN, so | |
| the pretrained checkpoints in `Pretrained_GNN/` load without conversion. | |
| The transformer and Deep Sets take each event's objects as tokens | |
| `[events, objects, 7]`, zero-padded to the largest event of the batch, with | |
| a mask of real objects. A token is a linear map of the kinematic columns | |
| (`phi` as its cosine and sine) plus a learned embedding of `node_type`. The | |
| objects are stored by type (jets, electrons, muons, photons, MET), so a | |
| position embedding would mostly relearn the type; without one, neither | |
| model depends on object order, as with the edge network. | |
| * The transformer is a stack of pre-norm attention blocks, a mean over real | |
| objects and a linear classifier. With `model.pair_bias` (as in the | |
| Particle Transformer, Qu et al. 2022), an MLP of the pair features | |
| `[deta, dphi, dR, log(1 + m^2)]` (the edge features plus the invariant | |
| mass of the pair, objects taken as massless) adds a bias per head to the | |
| attention logits, so the transformer sees the same pairwise angles as the | |
| edge network. | |
| * Deep Sets (Zaheer et al. 2017; the particle flow network of Komiske et | |
| al. 2019) applies an MLP to each object, sums over real objects and | |
| applies an MLP to the sum: the baseline without object interactions. | |
| ### JEPA pretraining | |
| `model.name: jepa` pretrains an object encoder without labels, as in | |
| I-JEPA (Assran et al. 2023), on the 12 Delphes samples of the supervised | |
| pretraining. `model.backbone` selects the encoder: the transformer's | |
| (`configs/delphes_jepa.yaml`) or the edge network, of the size of the | |
| supervised pretrained models (`configs/delphes_jepa_edge.yaml`), whose | |
| object encodings are its node states after the last processing step and | |
| whose context is the fully connected graph of the visible objects only. | |
| * a random `model.mask_fraction` of each event's objects (rounded, at least | |
| one, and at least one left visible) is hidden; | |
| * the context encoder sees the visible objects; the target encoder, an | |
| exponential moving average of it (momentum rising linearly from | |
| `model.ema_momentum` to 1 over training) without gradients, sees all | |
| objects; | |
| * a predictor (`model.predictor_layers` attention blocks over the visible | |
| objects' encodings and one query per hidden object) predicts the hidden | |
| objects' layer-normalized target encodings, with a mean squared error. | |
| A query is told only the hidden object's type and its index among the | |
| objects of that type (their pt order), not its kinematics; pair features | |
| are used only between visible objects. For the edge network, its decoded | |
| global state (what its classifier reads) is one more predictor input, so | |
| the whole network is pretrained. | |
| The history reports `target_spread`, the standard deviation of the target | |
| encodings over a batch's objects averaged over dimensions, which falls | |
| towards zero if the representation collapses. The validation loss compares | |
| predictions with a moving target, so it does not select epochs: fine-tuning | |
| starts from `last.pt` (`configs/gamgam_jepa_finetune.yaml` for the | |
| transformer, `configs/gamgam_finetune.yaml` with `model.pretrained` for the | |
| edge network; encoder sizes must match). With `model.freeze_backbone=true` only the classifier is | |
| trained (a linear probe); `model.pretrained=null` trains the same | |
| architecture from scratch. | |
| `model.variance_weight` adds the VICReg variance term (Bardes et al. 2022): | |
| the mean shortfall below 1 of each dimension's standard deviation over the | |
| batch's context encodings (and the edge network's global states). | |
| First checks (one A100, `train.fast=true`, 2 pretraining epochs on Delphes | |
| fold 2, then 2 GamGam epochs, validation AUC, seed 1; not converged): | |
| | GamGam model | Validation AUC | | |
| | --- | --- | | |
| | transformer from scratch / from JEPA | 0.898 / 0.904 | | |
| | edge network from scratch | 0.895 | | |
| | edge network from the supervised pretrained model | 0.907 | | |
| | edge network from JEPA, last node states only | 0.845 | | |
| | edge network from JEPA, mean node states | 0.869 | | |
| | the same with `variance_weight` 0.1 / 1 | 0.874 / 0.810 | | |
| The edge network's JEPA encodings collapse without the variance term | |
| (`target_spread` 0.09 against about 1 with it), and none of the edge | |
| network variants yet beats training from scratch. | |
| ### Contrastive and masked-object pretraining, and probes | |
| Two more label-free objectives pretrain the same encoders on the same | |
| samples, and their encoders load the same way (`context_encoder`): | |
| * `contrastive.py` (`configs/delphes_contrastive.yaml`): SimCLR, as JetCLR | |
| and RS3L for jets. Two views of each event (random phi rotation, eta | |
| flip, momentum scale within `model.scale_jitter`, each object but the | |
| first dropped with `model.object_dropout`) are embedded and projected; | |
| the NT-Xent loss at `model.temperature` contrasts them with the other | |
| events of the batch. `embedding_spread` watches for collapse. | |
| * `model.jepa_target: features` (`configs/delphes_masked.yaml`): masked | |
| object modelling (Golling et al. 2024). JEPA's masking and predictor | |
| predict the hidden objects' kinematics instead of their target | |
| encodings, with no target encoder: the comparison with JEPA isolates | |
| predicting in embedding space. | |
| `gnn4colliders probe` compares encoders without fine-tuning (as jBOT): | |
| k-NN (cosine, `model.probe_neighbours`) and linear (L-BFGS logistic | |
| regression, `model.probe_l2`) classifiers of frozen event embeddings, | |
| trained on `data.train_events` events. A randomly initialized encoder | |
| (`model.pretrained=null`) is the reference; supervised checkpoints probe | |
| too. Results on GamGam: "Label efficiency on GamGam" below. | |
| `scripts/compare_encoders.py` compares encoders of one architecture by | |
| linear CKA (Kornblith et al. 2019), as the paper does: of their event | |
| embeddings (a matrix), and layer by layer against the first one (the mean | |
| over objects after each transformer block, or the edge network's mean node | |
| states after each step, then the event embedding). With a random encoder | |
| first, it shows where each objective moves the representation. | |
| ### In-context models: TabPFN, TabICL, TabDPT and a collider PFN | |
| Both classify a dataset's events given `data.train_events` labelled | |
| training events (an equal number per class) as context, without training on | |
| that dataset; `cli.in_context_forward` builds them for `evaluate` and | |
| `predict`. | |
| * `tabular.py`: TabPFN (Prior Labs; `model.tabpfn_version`, default 3.5) | |
| on fixed-width rows: the leading `model.slots` objects of each type (6 | |
| jets, 2 electrons, 2 muons, 3 photons, MET) with their six kinematic | |
| columns, NaN when absent, and the number of objects of each type. | |
| * `pfn.py`: our own prior-fitted network. The object transformer (which | |
| can start from the JEPA encoder) embeds each event; the context events get | |
| a label embedding; `model.pfn_layers` attention blocks run over a task's | |
| events, every event attending to the context events only, so queries are | |
| independent and the context's keys and values are computed once. It is | |
| meta-trained (`task.name: pfn`) on tasks drawn from the Delphes samples: | |
| 2-10 classes of `task.prior_classes` in random order, Dirichlet class | |
| proportions, 16-2048 context events (log-uniform), 256 queries, and per | |
| event a random phi rotation and eta flip, per task a momentum and energy | |
| scale within `task.energy_jitter`. The GamGam processes are among the | |
| Delphes samples; `task.prior_classes=[1,6,7,8,9,10,11]` leaves them out. | |
| `configs/opendata_pfn.yaml` (written by `scripts/opendata_prior.py`) | |
| meta-trains it on the ATLAS Open Data 2025 MC instead: 925 classes of 331 | |
| processes, H -> gamma gamma left out, a tenth of the processes held out | |
| for validation, classes mixing 1 to 3 processes. | |
| * Options that bring the PFN closer to TabPFN, all off by default (the | |
| default model's outputs match the earlier code to 2e-7): | |
| `model.pfn_summary_tokens` learned tokens carry each event through the | |
| encoder, with a context event's label added at the input instead of after | |
| the encoder; `model.pfn_alternating` adds attention across the task's | |
| events (summary tokens, context only) after every encoder block, as | |
| TabPFN alternates attention over features and over rows, so an event's | |
| encoding depends on the task; `model.pfn_normalize` scales each task's | |
| momenta so that the context's geometric-mean object pT is 50 GeV (as | |
| TabPFN standardizes features by the context); `model.pfn_ensemble` | |
| averages the class probabilities of passes with the events rotated in | |
| phi, flipped in eta and the class labels permuted (as TabPFN ensembles | |
| over class orders and preprocessings); and `task.synthetic_tasks` makes | |
| a fraction of the training tasks synthetic, as TabPFN's prior draws | |
| random functions: events of 1 to 3 processes ranked by a random network | |
| (16 tanh units) of 1 to 4 of 21 event observables (per object type the | |
| count, scalar sum pT, leading pT and mass; the b-tag count), cut into | |
| classes of Dirichlet fractions. A large context is encoded in chunks, one | |
| block at a time, but the alternating PFN keeps every context event's | |
| tokens and the keys and values of every block on the GPU: 100k events | |
| fit, all 1.98M GamGam training events do not (out of memory on a 40 GB | |
| A100). | |
| * Options from TabPFN-3.5 (219M parameters, 79 times ours; read from | |
| its checkpoint and code), all off by default: `model.pfn_retrieval` makes | |
| the logits the log of an attention-weighted vote of the context's labels | |
| (TabPFN's many-class decoder) instead of a linear classifier; | |
| `model.pfn_qk_norm` RMS-norms queries and keys per head and scales the | |
| queries by a learned factor times the log of the number of keys | |
| (TabPFN-3's scalable softmax), in all attention across events; | |
| `model.pfn_query_kv` lets the queries read the context through one | |
| key/value head of their own, so the memory of a large context is | |
| `n_heads` times smaller; `model.pfn_concat` joins the summary tokens into | |
| one token `pfn_summary_tokens * hid_size` wide for the blocks after the | |
| encoder (TabPFN-3.5 joins 8 x 128 to 1024), with the label embedded | |
| again; `model.pfn_fourier` learned frequencies of each object's log pT, | |
| eta and phi and `model.pfn_ranks` (each object's pT and energy ranked | |
| among the context's objects of its type, as TabPFN adds ECDF ranks) are | |
| added to the object embeddings, through maps that start at zero. | |
| `train.lr_schedule: cosine` warms up over `train.warmup_epochs`. | |
| * Meta-training speed: training batches are encoded in 4 parts of events | |
| with similar numbers of objects (only 37% of the padded object tokens | |
| were real), tasks are sampled in the loader workers rather than all at | |
| the start of an epoch, and under `torchrun` all ranks draw the same | |
| context size for a step, so that they wait less for each other. Steps are | |
| drawn from a generator per seed, epoch, rank and step, so the tasks | |
| differ from those of runs before this change. In-context inference sorts | |
| the context by number of objects and attends in chunks of | |
| `data.batch_size` events. | |
| * `tabular.py` also runs TabICLv2 (Qu et al., ICML 2026; BSD-3, open | |
| pretraining code) on the same rows, as a second tabular in-context model. | |
| Rows can measure azimuths from the leading object's (`model.relative_phi`) | |
| and end with the invariant mass and Delta R of every pair of the | |
| `model.pair_objects` leading objects by pt (energies are `pt cosh(eta)`, | |
| so objects are massless; a pair with MET gives a transverse-like mass). | |
| Both are off by default, so the default rows are unchanged. | |
| * `tabular.py` also runs TabDPT (Ma et al. 2024; Apache-2.0), trained on | |
| real tables rather than a synthetic prior; with | |
| `model.tabdpt_context_size` each event's context is its nearest context | |
| events (faiss retrieval), beyond what fits in one pass. It computes in | |
| float64 in places (on a Mac it runs on the CPU), and on macOS faiss and | |
| torch crash together unless `OMP_NUM_THREADS=1`. | |
| * Tabular models can see, instead of or beside the rows, the event | |
| embeddings of a pretrained object encoder (`model.tabular_inputs: | |
| embeddings` or `both`, the encoder of `model.backbone` and | |
| `model.pretrained`): physics-aware embeddings under a generic tabular | |
| prior. In meta-training the encoder trains with the LoRA updates unless | |
| `model.freeze_backbone`. | |
| * `tabular.py` also meta-trains the TabPFN-3.5 network itself on the | |
| PFN's tasks (`model.name: tabpfn`, `task.name: pfn`), on the rows above | |
| (built after the augmentation), starting from Prior Labs' weights. Calling | |
| the network directly skips TabPFN's preprocessing and ensembling, which | |
| on the GamGam fixtures cost nothing (validation AUC 0.852 against 0.843 | |
| with `predict_proba`, 61 events). By default the network is frozen and | |
| the four attention projections and two MLP layers of each of its 24 | |
| in-context blocks get rank-8 LoRA updates (2.8M parameters, zero at the | |
| start, so training starts from TabPFN as released); `model.lora_rank=0` | |
| trains all weights. In context, the context's keys and values are cached | |
| (the same logits as one pass over context and queries, to 2e-5). It | |
| separates what Prior Labs' prior and what collider tasks contribute: | |
| TabPFN as released, TabPFN meta-trained here, and our PFN. TabICL | |
| (`configs/delphes_tabicl.yaml`; BSD-3, so meta-trained weights can be | |
| shared) meta-trains the same way. Its attention reads fused projection | |
| weights directly, so LoRA is a parametrization of the weights (12 blocks, | |
| 0.59M of 28M parameters at rank 8), for TabPFN too. TabICL's network takes | |
| no NaN: rows are standardized by the context's statistics, absent values | |
| at the mean, clipped at 8 sigma. On the fixtures this gives AUC 0.827 | |
| against 0.820 with its classifier, and the cached in-context forward | |
| equals the training forward only in float32 (with mixed precision on the | |
| CPU, AUC 0.59). | |
| ### Label efficiency on GamGam (first study, 2026-10-22) | |
| GamGam validation AUC (%) against the number of labelled training events N | |
| (`data.train_events`, equal per class), one seed, on one 4-GPU A100 node in | |
| an interactive allocation (3.5 h). Each entry is evaluated on 20,000 random | |
| validation events (`data.eval_events=20000`, about +-0.3); on all 246,922, | |
| evaluated for the supervised runs and the leading TabPFN runs, each is 0.05 | |
| to 0.41 lower, in the same order (TabPFN-3.5: 80.28, 86.92, 89.01, 0.08 to | |
| 0.13 lower, matching the M2 Max's 80.26 and 86.93). | |
| | model | N = 100 | 1000 | 10000 | 50000 | | |
| |---|---|---|---|---| | |
| | edge network, from scratch | 78.00 | 82.74 | 87.66 | | | |
| | edge network, from `Pretrained_GNN` | 84.95 | 87.48 | 89.98 | | | |
| | transformer, from scratch | 72.33 | 83.47 | 87.03 | | | |
| | transformer, from JEPA | 73.25 | 85.10 | 88.60 | | | |
| | transformer, from masked objects | 75.92 | 86.12 | 88.73 | | | |
| | TabPFN-3.5 (rows) | 80.41 | 87.05 | 89.09 | 89.56 | | |
| | TabPFN-3.5 (rows, `relative_phi`, `pair_objects: 4`) | 80.61 | 87.78 | 89.56 | | | |
| | TabPFN-3.5 (rows + JEPA embeddings) | 80.97 | 87.43 | 89.44 | | | |
| | TabPFN-3.5 (rows + masked-object embeddings) | 82.99 | 88.22 | 89.94 | | | |
| | TabPFN-3.5 (rows + `Pretrained_GNN` embeddings) | 87.32 | 88.96 | 90.07 | | | |
| | TabPFN-3.5, released network, direct forward | 80.74 | 86.64 | 89.03 | | | |
| | TabPFN-3.5, meta-trained on Delphes (LoRA) | 80.52 | 84.77 | 88.34 | | | |
| | TabICLv2 | 80.53 | 85.68 | 88.41 | 89.12 | | |
| | TabDPT | 77.48 | 84.71 | 88.42 | | | |
| | our PFN, meta-trained on Delphes | 78.13 | 76.18 | 76.51 | | | |
| Probes (k-NN / linear) of the frozen encoders, the same N: | |
| | encoder | N = 100 | 1000 | 10000 | | |
| |---|---|---|---| | |
| | random | 73.4 / 72.6 | 76.6 / 83.1 | 79.5 / 85.4 | | |
| | JEPA | 73.0 / 73.4 | 78.9 / 83.4 | 81.8 / 87.9 | | |
| | masked objects | 75.0 / 77.5 | 81.7 / 85.7 | 84.4 / 88.5 | | |
| | contrastive | 61.6 / 67.9 | 70.0 / 80.3 | 76.4 / 85.9 | | |
| Setup (every run and its overrides: `scripts/label_efficiency_study.sh`): | |
| the pretraining runs saw the same 4M class-balanced Delphes events for 8 | |
| epochs (`data.train_events=4000000`, 50 to 70 min each); the PFN | |
| meta-trained for 25 epochs of 1000 steps (25,000 steps, 95 min) and TabPFN | |
| for 6 epochs of 500 (3,000 steps, 48 min). Supervised runs train for a | |
| fixed number of steps with early stopping off (`train.early_stopping=null`; | |
| batch size, epochs: 100, 1000 for N = 100; 128, 400 for N = 1000; 256, 100 | |
| for N = 10000; 20 to 35 min at N = 100, 2 to 4 min at N = 10000) and keep | |
| the epoch of best validation AUC, chosen on the same 20,000 events that are | |
| reported. That choice uses validation labels the in-context models do not | |
| get, and from N = 1000 most supervised runs overfit: their last epoch is far | |
| below the kept one. On all validation events (92% of them not used for the | |
| choice) the kept epochs lose 0.05 to 0.41, no more than the in-context runs | |
| lose, so picking the maximum over noisy evaluations inflates them little: | |
| | supervised run (N = 100 / 1000 / 10000) | kept epoch, 20k | kept epoch, all | last epoch, 20k | | |
| |---|---|---|---| | |
| | edge network, from scratch | 78.00 / 82.74 / 87.66 | 77.80 / 82.69 / 87.42 | 68.88 / 78.88 / 87.65 | | |
| | edge network, from `Pretrained_GNN` | 84.95 / 87.48 / 89.98 | 84.54 / 87.09 / 89.74 | 80.75 / 81.58 / 86.85 | | |
| | transformer, from scratch | 72.33 / 83.47 / 87.03 | 71.99 / 83.35 / 86.95 | 72.33 / 83.44 / 87.03 | | |
| | transformer, from JEPA | 73.25 / 85.10 / 88.60 | 73.07 / 84.96 / 88.45 | 72.34 / 78.82 / 81.46 | | |
| | transformer, from masked objects | 75.92 / 86.12 / 88.73 | 75.73 / 85.84 / 88.60 | 75.92 / 79.73 / 85.62 | | |
| With only N labelled events there is no large labelled set to choose the | |
| epoch on; without the choice (last epoch) every supervised model is below | |
| TabPFN-3.5 as released at every N except `Pretrained_GNN` at N = 100. | |
| With the configs' settings instead (batch size 1024, at most 100 epochs, | |
| early stopping after 5 epochs without training-loss improvement) N = 1000 | |
| is one step per epoch, at most 100 steps: the transformer then reaches | |
| 70.04 at N = 1000 instead of 83.47. The in-context models take 10 to 30 s | |
| for the 20,000 queries at N = 1000 and 10000 (TabPFN 109 s at N = 50000; | |
| the first run of each, N = 100, also loads the weights). | |
| Findings: | |
| * TabPFN-3.5 as released, without training, comes within 0.4 (N = 1000) | |
| and 0.9 (N = 10000) of fine-tuning the supervised `Pretrained_GNN` (0.2 | |
| and 0.7 on all validation events), with its epoch chosen on validation | |
| labels, and above it from N = 1000 without that choice. It beats every | |
| model trained from scratch at every N. TabICL is close behind; TabDPT is | |
| behind at small N. The physics row features add up to 0.7. | |
| * Encoder embeddings help TabPFN most at small N. With the label-free | |
| masked-object encoder: +2.6 at N = 100. With `Pretrained_GNN` (trained | |
| with labels, on Delphes samples that include the GamGam processes): the | |
| best result at every N run (100 to 10000; not run at 50000), +6.9 at | |
| N = 100. | |
| * Meta-training on Delphes tasks does not help. TabPFN with LoRA loses 1.9 | |
| at N = 1000 against the same network with zero updates (its direct | |
| forward costs at most 0.4 against the classifier's preprocessing and | |
| ensembling). Our PFN is below TabPFN and gets worse with more context | |
| (76 at N = 1000, inside the meta-training context sizes 16 to 2048): to | |
| investigate before the held-out-processes run, which was not done. | |
| * Of the label-free pretraining objectives, masked-object modelling is the | |
| best, in probes and fine-tuned; JEPA's prediction in embedding space adds | |
| nothing over it here. Both improve fine-tuning the transformer over | |
| training it from scratch (+0.9 to +3.6) but stay below fine-tuning the | |
| edge network from `Pretrained_GNN`. That gap mixes the architecture (from | |
| scratch, the edge network is 5.7 above the transformer at N = 100) with | |
| the objective (label-free or supervised); the like-for-like comparison, | |
| JEPA of the edge network (`configs/delphes_jepa_edge.yaml`), was not run. | |
| * Contrastive pretraining is below a random encoder in the k-NN probe at | |
| every N and in the linear probe at N = 100 and 1000 (67.9 against 72.6, | |
| 80.3 against 83.1); at N = 10000 its linear probe is 0.5 above (85.9 | |
| against 85.4). Without the momentum-scale augmentation | |
| (`model.scale_jitter=0`, 4 epochs) it is no better (72.4 / 84.3 at | |
| N = 10000). | |
| Not done: | |
| * the held-out-processes meta-training (`task.prior_classes` without the | |
| five GamGam processes) of the PFN, gated on it beating TabPFN, and of | |
| TabPFN and TabICL; | |
| * TabICL meta-training and `model.lora_rank=0`; | |
| * fine-tuning from contrastive pretraining (only probed), and JEPA of the | |
| edge network; | |
| * more seeds, and the test split. | |
| The loss is a weighted mean per class present in the batch, averaged over | |
| those classes, which balances the classes without explicit class weights. | |
| Metrics (accuracy, one-vs-rest AUC averaged over classes, macro precision, | |
| recall and F1) use events with positive weight. | |
| ### The TabPFN-like PFN on GamGam (2026-10-27) | |
| GamGam test AUC (%) in context, against the number of labelled context | |
| events N (equal per class), on 50,000 test events, one seed. Every PFN is | |
| meta-trained on the Open Data prior (`configs/opendata_pfn.yaml`). The | |
| earlier PFN (pooled events, contexts of 16 to 4096, random encoder, fp32) | |
| is the checkpoint of epoch 38. The new ones train 48 epochs of 1000 steps | |
| (2.3 h on one A100 each, bf16) with 2 summary tokens, alternating | |
| attention, the per-task momentum scale, the masked-object encoder | |
| (`delphes_masked`), half synthetic tasks, contexts of 16 to 16384 and | |
| batches of 16384 events; each ablation changes one of them. | |
| | PFN | N = 100 | 1000 | 10000 | 100000 | | |
| |---|---|---|---|---| | |
| | earlier | 80.63 | 82.47 | 83.54 | 83.53 | | |
| | earlier, ensemble of 8 | 81.92 | 83.87 | 84.67 | 84.77 | | |
| | new | 79.12 | 80.39 | 83.79 | 84.18 | | |
| | new, ensemble of 8 | 79.67 | 82.26 | 84.46 | 84.77 | | |
| | new, no alternating attention | 77.14 | 81.21 | 84.43 | 84.68 | | |
| | new, random encoder | 76.47 | 81.76 | 83.42 | 83.56 | | |
| | new, no synthetic tasks | 81.28 | 83.14 | 84.72 | 85.45 | | |
| | new, no synthetic tasks, ensemble of 8 | **83.75** | **86.11** | **86.54** | **86.52** | | |
| Validation on held-out Open Data processes after 48 epochs (loss / | |
| accuracy): new 0.772 / 0.722, no alternating 0.800 / 0.712, random encoder | |
| 0.809 / 0.711, no synthetic tasks 0.665 / 0.764; all still falling. One | |
| pass takes 1 to 3 s for the 50,000 queries at N = 1000 and 10000 and 35 to | |
| 58 s at N = 100000; the ensemble of 8 takes 460 s at N = 100000. | |
| Findings: | |
| * Synthetic tasks hurt at every N (0.9 to 2.8 against the same PFN | |
| without them; 1.8 to 4.1 ensembled) and in validation. Without them the | |
| PFN beats the earlier one at every N, by 1.8 to 2.2 with both | |
| ensembled. | |
| * The ensemble of 8 adds 1.1 to 3.0 to the PFN without synthetic tasks. | |
| * The masked-object encoder adds 2.7 at N = 100 and 0.6 at N = 100000. | |
| * Alternating attention adds 2.0 at N = 100 and costs 0.5 to 0.8 from | |
| N = 1000; it makes training 20% slower. | |
| * Against the validation AUCs of the first study (on other events, so | |
| only roughly comparable): the best PFN is above TabPFN-3.5 on rows | |
| (80.41) and every supervised transformer at N = 100, below the edge | |
| network from `Pretrained_GNN` (84.95) and TabPFN-3.5 with its embeddings | |
| (87.32). At N = 1000 it equals the transformer from masked objects | |
| (86.12) and is 0.9 and 1.4 below TabPFN-3.5 and `Pretrained_GNN`; at | |
| 10000 it is 2.2 to 3.4 below all three, and it gains nothing from 10000 | |
| to 100000. | |
| Not done: all 1.98M training events as context (out of memory; see the | |
| PFN options above), longer meta-training, and the held-out-processes | |
| prior. | |
| ### TabPFN-3.5 options and a larger PFN on GamGam (2026-10-28) | |
| Both PFNs start from the best one above (no synthetic tasks) and add the | |
| TabPFN-3.5 options `pfn_retrieval`, `pfn_qk_norm`, `pfn_query_kv`, | |
| `pfn_fourier=16` and `pfn_ranks`, the cosine schedule (2 warm-up epochs, | |
| learning rate 1e-4) and the faster meta-training. "Small" keeps its size | |
| (2.9M parameters) and trains 40 epochs on one A100 (168 s each). "Large" | |
| has 4 summary tokens joined to one 512-wide token and 12 blocks after the | |
| encoder (42M parameters) and trains 26 epochs on three A100s under | |
| `torchrun` (280 s each, three times the tasks per step). Same test events | |
| and protocol as above, one seed. | |
| | PFN | N = 100 | 1000 | 10000 | 100000 | | |
| |---|---|---|---|---| | |
| | best above, ensemble of 8 | 83.75 | 86.11 | 86.54 | 86.52 | | |
| | small | 81.73 | 84.65 | 86.76 | 87.66 | | |
| | small, ensemble of 8 | 83.15 | 86.20 | 87.63 | 88.20 | | |
| | large | 82.90 | 85.83 | 87.72 | 88.43 | | |
| | large, ensemble of 8 | **84.06** | **86.80** | **88.24** | **88.62** | | |
| Validation on held-out Open Data processes in the last epoch (loss / | |
| accuracy): small 0.599 / 0.789, large 0.561 / 0.799 (best above 0.665 / | |
| 0.764). At N = 100000 one pass takes 21 s (small) and 87 s (large), the | |
| ensemble of 8 169 s and 698 s; the large ensemble peaks at 13 GiB. | |
| Findings: | |
| * The options, schedule and faster training add 0.5 to 2.2 to one pass of | |
| the best PFN above (81.28, 83.14, 84.72, 85.45), and the AUC now rises | |
| from 10000 to 100000 (by 0.4 to 0.9) instead of staying flat. | |
| * The larger PFN adds 0.8 to 1.2 more. One pass of it is 0.9 and 0.3 | |
| below the earlier ensemble at N = 100 and 1000 and 1.2 and 1.9 above it | |
| from 10000; ensembled, it gains 0.3 to 2.1 on the earlier ensemble. | |
| * The ensemble adds less than before: 0.2 to 1.6. | |
| * The options were not ablated one by one, and the two runs differ in | |
| epochs and GPUs as well as size. | |
| * Against the validation AUCs of the first study (only roughly | |
| comparable): at N = 100 the large ensemble is 3.7 above TabPFN-3.5 on | |
| rows and 0.9 below `Pretrained_GNN`; at 1000 it is 0.3 and 0.7 below | |
| them, at 10000 0.9 and 1.7 below. | |
| Not done: all 1.98M training events as context, one ablation per option, | |
| more seeds. | |
| ## Performance | |
| Graphs differ in size, which suits neither `torch.compile` nor CUDA graphs | |
| out of the box: | |
| * `torch.compile(dynamic=True)` treats node and edge counts as symbolic | |
| sizes, so one compiled graph serves all batches without padding. | |
| * CUDA graph replay needs fixed shapes. Batches are copied into buffers | |
| rounded up to multiples of 512 nodes and 4096 edges; the remainder forms 64 | |
| padding graphs connected only to themselves, so the real graphs' outputs do | |
| not change. A step is captured once per bucket, a few per epoch. The | |
| optimizer runs in Adam's `capturable` form with device-tensor learning | |
| rates, which the scheduler updates in place. | |
| * bfloat16 autocast covers the forward pass; logits, loss, graph sums (which | |
| divide by edge counts) and the optimizer stay float32. | |
| GamGam training step, 1024-event batches (about 5,600 nodes and 30,000 | |
| edges), one A100-SXM4-80GB, `OMP_NUM_THREADS=8`, four loader workers (step | |
| times measured before the move to flat modules, epoch times after): | |
| | Mode | ms per batch | Epoch with validation | | |
| | --- | ---: | ---: | | |
| | float32 eager | 19.2 | 37 s | | |
| | CUDA graphs + TF32 | 8.5 | | | |
| | `torch.compile` | 13.0 | | | |
| | compile + CUDA graphs + TF32 | 6.1 | | | |
| | compile + CUDA graphs + bfloat16 (`train.fast`) | 5.9 | 9 s (first: 59 s, compiling) | | |
| Eager training is bound by kernel launches (about a thousand small kernels | |
| per step), not by arithmetic; bfloat16 alone is slower because of the extra | |
| cast kernels, which `torch.compile` fuses. Results of the fast modes agree | |
| with float32 to rounding, not bitwise (validation AUC 0.895 against 0.897 | |
| after three epochs). Preparing the 2.5M GamGam events with 8 workers takes | |
| about 9 s on a login node (40 s was seen on a compute node, likely a cold | |
| file cache). | |
| PyTorch sizes its CPU thread pool to every visible core; on a 128-core node | |
| that makes batch construction several times slower and CPU reductions | |
| non-reproducible, so set `OMP_NUM_THREADS`. Single-threaded CPU training is | |
| bitwise reproducible. | |
| ## ATLAS Open Data GamGam versus Ho et al. (2026) | |
| The main benchmark reproduces the from-scratch baseline of Table 3 (5 | |
| classes, 2.5M events, 80:10:10 split, five seeds). Choices the paper does not | |
| specify: | |
| * electrons and muons are split from `lep_*` by `lep_type` (11, 13); | |
| * b-tagging is `jet_MV2c10 > 0.8244273`, the 70% working point; | |
| * momenta and energies are scaled by 1e-4 (MeV to the pretraining unit of | |
| 10 GeV); | |
| * the fold is `eventNumber % 10`: folds 0-7 train, 8 validates, 9 tests; | |
| * training is unweighted (23% of ttH events have `mcWeight = -1`). | |
| The paper's stopping rule (5 epochs without training-loss improvement) never | |
| fires within 200 epochs on this task; validation AUC peaks earlier (epoch 67 | |
| for seed 1) and `best.pt` keeps that epoch. | |
| | Test split | Accuracy | AUC | F1 | Precision | Recall | | |
| | --- | ---: | ---: | ---: | ---: | ---: | | |
| | Paper (5 seeds) | 71.8 ± 0.2 | 91.2 | 63.0 | 62.2 | 66.3 | | |
| | This package, seed 1 (CUDA graphs + TF32) | 71.1 | 91.3 | 62.6 | 62.4 | 66.7 | | |