The PFN gains a retrieval head (an attention-weighted vote of the context's
labels), QK-norm with a log-context-size temperature, one key/value head for
queries, joined summary tokens for a wider stage across events, Fourier
input features and context ranks of pT and energy; all off by default.
train.lr_schedule=cosine warms up over train.warmup_epochs. Meta-training
encodes events in object-count buckets, samples tasks in loader workers and
shares each step's context size across DDP ranks; in-context inference
sorts the context and attends in chunks. Task draws differ from earlier
runs (one generator per step). docs/design.md has the GamGam results.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs: correct the label-efficiency study and add a script that reruns it
Report the kept supervised epochs on all validation events and their last
epochs, fix the step counts, the subsample offsets and the contrastive
claims, add timings, and list everything the plan did not run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Validation AUC against labelled events (100 to 50000) for TabPFN-3.5,
TabICLv2 and TabDPT (with physics row features and encoder embeddings),
TabPFN and our PFN meta-trained on Delphes, the supervised edge network
and transformer (from scratch, Pretrained_GNN, JEPA and masked-object
pretraining, step-matched), and probes of the pretrained encoders. One
seed, 20k-event validation subsample checked against full validation; one
4-GPU interactive allocation. No code changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
probe.linear_cka and probe.layer_embeddings, and
scripts/compare_encoders.py: CKA of the event embeddings of several
encoders of one architecture (pretrained with JEPA, contrastive, masked
objects or supervision, or random), and layer by layer against the first,
as in Ho et al. (2026). Tested on fixtures for both backbones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat: tabular in-context models, LoRA meta-training, pretraining objectives and probes
In-context models (tabular.py):
- TabPFN-3.5 meta-trained on the PFN's Delphes tasks (model.name: tabpfn,
task.name: pfn), LoRA on its 24 in-context blocks by default
(model.lora_rank; 0 trains all weights); checkpoints hold only trained
weights. configs/delphes_tabpfn.yaml.
- TabICLv2 (BSD-3) as released and meta-trained the same way
(configs/gamgam_tabicl.yaml, configs/delphes_tabicl.yaml); LoRA is a
weight parametrization, as TabICL reads fused projections directly.
- TabDPT (Apache-2.0) as released, with optional per-event retrieval of the
context (configs/gamgam_tabdpt.yaml).
- Row options: azimuths from the leading object (model.relative_phi), pair
masses and Delta R (model.pair_objects); encoder embeddings instead of or
beside rows (model.tabular_inputs).
- model.tabpfn_checkpoint; evaluate records setup/inference seconds and
CUDA peak memory.
Pretraining (all label-free, encoders load as JEPA's):
- contrastive.py: SimCLR with physics augmentations
(configs/…
feat: TabPFN and a collider PFN as in-context models
- data.train_events: an equal number of training events per class (nested
subsets), for supervised training and as the in-context models' context.
- tabular.py: TabPFN (default 3.5) on fixed-width event rows; optional
extra `tabpfn`. configs/gamgam_tabpfn.yaml.
- pfn.py: a prior-fitted network over events (object-transformer
embeddings, label embedding, attention to context events only),
meta-trained on tasks drawn from the Delphes samples (task.name: pfn).
configs/delphes_pfn.yaml, configs/gamgam_pfn.yaml.
- evaluate/predict run in-context models without a training step.
- EncoderBlock split into project/attend so context keys are cached.
Tested on CPU fixtures only; GPU study pending (docs/design.md). Existing
models' features, splits, losses and metrics are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
model.backbone selects the JEPA encoder (transformer or edge_network).
The edge network's object encodings are its node states averaged over
the encoder and processing steps (its last states alone are nearly equal
within an event at initialization), its global state is a predictor
token, and model.variance_weight adds a VICReg variance term against
collapse. A JEPA checkpoint fine-tunes through gamgam_finetune.yaml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
model.name: jepa with task.name: jepa pretrains the transformer encoder
without labels (I-JEPA): a predictor told only the type and type index of
hidden objects predicts their encodings by an EMA target encoder.
configs/delphes_jepa.yaml uses the 12 Delphes samples and cache of the
supervised pretraining; configs/gamgam_jepa_finetune.yaml fine-tunes the
context encoder on GamGam (model.pretrained=null: from scratch).
TransformerClassifier now wraps an ObjectTransformer encoder.
The Slurm scripts put Hugging Face downloads on scratch (HF_HOME).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat: transformer pair attention bias and a Deep Sets model
model.pair_bias adds a per-head attention bias from an MLP of the pair
features [deta, dphi, dR, log(1 + m^2)] (Particle Transformer style); it is
on in configs/gamgam_transformer.yaml. configs/gamgam_deep_sets.yaml adds
Deep Sets, the order-independent baseline without object interactions. Both
share an ObjectEmbedding and export to ONNX.
The transformer now uses its own pre-norm blocks on scaled_dot_product_attention:
nn.TransformerEncoder's inference fast path returns NaN with a per-head mask.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(transformer): type embedding instead of positions, pre-norm, ONNX export
Objects are stored by type, so the position embedding mostly relearned the
object type and made the output depend on object order. Tokens are now a
linear map of the kinematics (phi as cos/sin) plus a node-type embedding;
model.max_seq_len is gone. The transformer exports to ONNX with tokens and
token_mask inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
refactor: flat modules, one YAML per experiment, no Hydra
Replace the 14 subpackages with 13 flat modules (6.7k -> 2.7k lines),
the Hydra config groups with typed dataclasses and one self-contained
YAML file per experiment (configs/), and the docs with docs/usage.md and
docs/design.md. All capabilities stay: GamGam, ttH CP, Delphes, binary
and multiclass tasks, transformer, fine-tuning from Pretrained_GNN, DDP,
torch.compile/CUDA graphs/bfloat16, predict and ONNX export.
Behavior: prepared caches, pretrained-model logits and two CPU training
epochs are bit-identical to the previous code on the fixtures and a 60k
event GamGam subset. Energies are now computed with one vectorized cosh
(<= 2.4e-7 relative difference on 969 of the full GamGam objects). The
ttH and Delphes configs now use Number % 10 as the fold (the raw event
index selected only the first 10 events), and use the paper recipe
(64/4/4, Adam 1e-4, decay 0.99). The ROOT score writer handles jagged
trees. Prediction files store source file and entry instead of sample
ids, and checkpo…