The PFN gains a retrieval head (an attention-weighted vote of the context's
labels), QK-norm with a log-context-size temperature, one key/value head for
queries, joined summary tokens for a wider stage across events, Fourier
input features and context ranks of pT and energy; all off by default.
train.lr_schedule=cosine warms up over train.warmup_epochs. Meta-training
encodes events in object-count buckets, samples tasks in loader workers and
shares each step's context size across DDP ranks; in-context inference
sorts the context and attends in chunks. Task draws differ from earlier
runs (one generator per step). docs/design.md has the GamGam results.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
27 days ago
feat: TabPFN-like PFN options: summary tokens, alternating attention, per-task momentum scale, synthetic tasks and an inference ensemble
- model.pfn_summary_tokens: learned tokens carry each event through the
encoder; a context event's label enters at the input.
- model.pfn_alternating: attention across the task's events after every
encoder block (TabPFN's alternation of feature and row attention).
- model.pfn_normalize: each task's momenta are scaled so the context's
geometric-mean object pT is 50 GeV.
- model.pfn_ensemble: in-context inference averages class probabilities
over passes with rotated, flipped events and permuted class labels.
- task.synthetic_tasks: a fraction of training tasks rank events of 1-3
processes by a random network of event observables.
Defaults keep the previous model and prior: outputs of the pooled PFN
match the previous code to 2.4e-7, and the prior draws the same tasks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix: PFN class proportions when the random draw is exactly 0
-log(rand) is infinite for a draw of 0, which made the proportions NaN
and stopped meta-training on the Open Data prior after about 11 epochs.
Draws are clamped at 1e-12; no other draw, and so no earlier prior,
changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat: PFN prior from the ATLAS Open Data 2025 MC, held-out processes and mixed classes
configs/opendata_pfn.yaml (written by scripts/opendata_prior.py) meta-trains
the PFN on 1299 classes, one per skim and process, of 334 processes of the
2025 Open Data MC; H -> gamma gamma processes are left out so GamGam stays
unseen. New keys: task.holdout_classes (left out of training tasks, the only
labels of validation tasks), task.mixture_labels (a class mixes several
labels in Dirichlet proportions), task.min_class_events and
data.max_events_per_file. Without them the prior and its random stream are
unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix: warm up CUDA graphs again after resuming; test split and all events in label_efficiency
A resumed trainer skipped the eager warmup steps (they counted global
steps), so with train.fast torch.compile compiled during CUDA graph
capture and failed. Count each trainer's own eager steps instead.
scripts/label_efficiency.py takes --events all, --split and
--eval-override.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix: CKA of collapsed encoders, encoder labels and seeding
- probe.linear_cka is NaN (not 0/0) for a representation that is the same
for every event; compare_encoders.py writes it as null in strict JSON and
names the encoders.
- compare_encoders.py rejects --encoders entries without label= and
repeated labels, draws its events with data.seed (no --seed), and sets
model.pretrained on the loaded config instead of re-parsing YAML.
- Layer-wise embeddings come from the encoders: ObjectTransformer
.encode_layers (forward returns its last) and jepa.layer_event_embeddings
(the edge network's mean node states via _mean_by_group), replacing the
forward hooks and copied pooling in probe.py.
- Tests: CKA of constant representations, cross-encoder and layer-wise
values of the script, and its argument errors.
Model outputs and existing results are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
probe.linear_cka and probe.layer_embeddings, and
scripts/compare_encoders.py: CKA of the event embeddings of several
encoders of one architecture (pretrained with JEPA, contrastive, masked
objects or supervision, or random), and layer by layer against the first,
as in Ho et al. (2026). Tested on fixtures for both backbones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat: evaluation subsample, Slurm wrappers for the study and label-efficiency plots
- data.eval_events: evaluate (validation in training, evaluate, predict,
probe) only that many seeded random validation or test events.
- scripts/label_efficiency.sh, meta_train.sh, pretrain.sh: one-GPU Slurm
wrappers for the sweep, meta-training then in-context sweep, and
pretraining then probes.
- scripts/plot_label_efficiency.py: AUC against labelled events from sweeps
and probes, as PDF/PNG and Markdown/LaTeX tables; optional extra `plots`.
Wrappers smoke-tested outside Slurm on fixtures; tests pass on CPU and MPS.
No change to existing models' results.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat: tabular in-context models, LoRA meta-training, pretraining objectives and probes
In-context models (tabular.py):
- TabPFN-3.5 meta-trained on the PFN's Delphes tasks (model.name: tabpfn,
task.name: pfn), LoRA on its 24 in-context blocks by default
(model.lora_rank; 0 trains all weights); checkpoints hold only trained
weights. configs/delphes_tabpfn.yaml.
- TabICLv2 (BSD-3) as released and meta-trained the same way
(configs/gamgam_tabicl.yaml, configs/delphes_tabicl.yaml); LoRA is a
weight parametrization, as TabICL reads fused projections directly.
- TabDPT (Apache-2.0) as released, with optional per-event retrieval of the
context (configs/gamgam_tabdpt.yaml).
- Row options: azimuths from the leading object (model.relative_phi), pair
masses and Delta R (model.pair_objects); encoder embeddings instead of or
beside rows (model.tabular_inputs).
- model.tabpfn_checkpoint; evaluate records setup/inference seconds and
CUDA peak memory.
Pretraining (all label-free, encoders load as JEPA's):
- contrastive.py: SimCLR with physics augmentations
(configs/…
feat: TabPFN and a collider PFN as in-context models
- data.train_events: an equal number of training events per class (nested
subsets), for supervised training and as the in-context models' context.
- tabular.py: TabPFN (default 3.5) on fixed-width event rows; optional
extra `tabpfn`. configs/gamgam_tabpfn.yaml.
- pfn.py: a prior-fitted network over events (object-transformer
embeddings, label embedding, attention to context events only),
meta-trained on tasks drawn from the Delphes samples (task.name: pfn).
configs/delphes_pfn.yaml, configs/gamgam_pfn.yaml.
- evaluate/predict run in-context models without a training step.
- EncoderBlock split into project/attend so context keys are cached.
Tested on CPU fixtures only; GPU study pending (docs/design.md). Existing
models' features, splits, losses and metrics are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
model.backbone selects the JEPA encoder (transformer or edge_network).
The edge network's object encodings are its node states averaged over
the encoder and processing steps (its last states alone are nearly equal
within an event at initialization), its global state is a predictor
token, and model.variance_weight adds a VICReg variance term against
collapse. A JEPA checkpoint fine-tunes through gamgam_finetune.yaml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
model.name: jepa with task.name: jepa pretrains the transformer encoder
without labels (I-JEPA): a predictor told only the type and type index of
hidden objects predicts their encodings by an EMA target encoder.
configs/delphes_jepa.yaml uses the 12 Delphes samples and cache of the
supervised pretraining; configs/gamgam_jepa_finetune.yaml fine-tunes the
context encoder on GamGam (model.pretrained=null: from scratch).
TransformerClassifier now wraps an ObjectTransformer encoder.
The Slurm scripts put Hugging Face downloads on scratch (HF_HOME).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat: transformer pair attention bias and a Deep Sets model
model.pair_bias adds a per-head attention bias from an MLP of the pair
features [deta, dphi, dR, log(1 + m^2)] (Particle Transformer style); it is
on in configs/gamgam_transformer.yaml. configs/gamgam_deep_sets.yaml adds
Deep Sets, the order-independent baseline without object interactions. Both
share an ObjectEmbedding and export to ONNX.
The transformer now uses its own pre-norm blocks on scaled_dot_product_attention:
nn.TransformerEncoder's inference fast path returns NaN with a per-head mask.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(transformer): type embedding instead of positions, pre-norm, ONNX export
Objects are stored by type, so the position embedding mostly relearned the
object type and made the output depend on object order. Tokens are now a
linear map of the kinematics (phi as cos/sin) plus a node-type embedding;
model.max_seq_len is gone. The transformer exports to ONNX with tokens and
token_mask inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
refactor: flat modules, one YAML per experiment, no Hydra
Replace the 14 subpackages with 13 flat modules (6.7k -> 2.7k lines),
the Hydra config groups with typed dataclasses and one self-contained
YAML file per experiment (configs/), and the docs with docs/usage.md and
docs/design.md. All capabilities stay: GamGam, ttH CP, Delphes, binary
and multiclass tasks, transformer, fine-tuning from Pretrained_GNN, DDP,
torch.compile/CUDA graphs/bfloat16, predict and ONNX export.
Behavior: prepared caches, pretrained-model logits and two CPU training
epochs are bit-identical to the previous code on the fixtures and a 60k
event GamGam subset. Energies are now computed with one vectorized cosh
(<= 2.4e-7 relative difference on 969 of the full GamGam objects). The
ttH and Delphes configs now use Number % 10 as the fold (the raw event
index selected only the first 10 events), and use the paper recipe
(64/4/4, Adam 1e-4, decay 0.99). The ROOT score writer handles jagged
trees. Prediction files store source file and entry instead of sample
ids, and checkpo…
feat: upgrade to Python 3.14 and PyTorch 2.14; add torch.compile and bfloat16
- Python 3.14, torch 2.14.1 (cu130 wheels), NumPy 2; frozen parity
references pass unchanged
- trainer.compile (dynamic-shape torch.compile) and trainer.precision
(bfloat16 autocast; graph sums, loss, optimizer in float32)
- CUDA graph steps use device-tensor learning rates, so one capture per
bucket follows the schedule; checkpoints store portable optimizer state
- trainer=paper_fast: compile + CUDA graphs + bfloat16, 5.9 vs 19.2 ms per
GamGam batch for float32 eager on an A100
- loader workers fork explicitly (Python 3.14 defaults to forkserver);
ONNX export keeps the TorchScript exporter
Correction to 5b73bcb: the pre-change float32 eager step was 21.3 ms per
batch (not ~29 ms; that profile used unshuffled ttH-only batches), so the
default path improved by about 1.2x and the large gains are opt-in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat: add the ATLAS Open Data GamGam 5-class task and faster training
Add the Higgs production multiclass task of Ho et al. (2026), Table 3:
ttH, ggF, VBF, WH, ZH from the ATLAS Open Data 13 TeV GamGam MC, pinned to
a Hugging Face dataset revision.
Data:
- derived branches split ``lep_*`` into electrons and muons and apply the
MV2c10 70% working point, so the pretraining node schema applies
- ``fold_modulus`` derives folds from event numbers in exact integers
- 64-event fixtures of the five samples and an end-to-end workflow test
Training:
- paper recipe (Adam 1e-4, 0.99 decay, stop after 5 epochs without
training-loss improvement) and ``best.pt`` selection by validation AUC
- checkpoint save_every_epochs/save_best/save_last are now honored;
per-epoch history.jsonl; evaluate can write metrics JSON
- macro precision/recall/F1 for multiclass tasks
- O(n log n) weighted ROC AUC on the metric device (was O(P*N) pairs)
Speed (A100, GamGam, batch 1024): about 56 s -> 35 s per epoch by default,
18 s with the opt-in paper_fast preset:
- loader worker pr…
feat: replace DGL with shared packed events and native PyTorch graphs
Track the documented Slurm wrappers that an unanchored slurm/ ignore rule
excluded, move graph-cache preparation from config.application into
graphs.preparation (outputs verified identical to main), archive rewrite
task specs under docs/history, move the paper source under docs/paper,
remove empty scaffolding, and refresh stale transformer documentation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- validation: use pinned public Hugging Face fixture (528346c0a2dcf57bc6f47e0e929113ae0f5ec47a)
- data: resolve training sources from Hugging Face (02920c2576f35911f7068c0379c06585ae9c8b57)