feat: tabular in-context models, LoRA meta-training, pretraining objectives and probes
Browse filesIn-context models (tabular.py):
- TabPFN-3.5 meta-trained on the PFN's Delphes tasks (model.name: tabpfn,
task.name: pfn), LoRA on its 24 in-context blocks by default
(model.lora_rank; 0 trains all weights); checkpoints hold only trained
weights. configs/delphes_tabpfn.yaml.
- TabICLv2 (BSD-3) as released and meta-trained the same way
(configs/gamgam_tabicl.yaml, configs/delphes_tabicl.yaml); LoRA is a
weight parametrization, as TabICL reads fused projections directly.
- TabDPT (Apache-2.0) as released, with optional per-event retrieval of the
context (configs/gamgam_tabdpt.yaml).
- Row options: azimuths from the leading object (model.relative_phi), pair
masses and Delta R (model.pair_objects); encoder embeddings instead of or
beside rows (model.tabular_inputs).
- model.tabpfn_checkpoint; evaluate records setup/inference seconds and
CUDA peak memory.
Pretraining (all label-free, encoders load as JEPA's):
- contrastive.py: SimCLR with physics augmentations
(configs/delphes_contrastive.yaml).
- JEPA with kinematic targets, masked object modelling
(model.jepa_target: features; configs/delphes_masked.yaml).
- probe command (probe.py): k-NN and linear probes of frozen encoders.
Development on Apple GPUs: train.device auto picks MPS; float64 metrics move
to the CPU on MPS (PyTorch 2.14 zeroes a combined device/dtype cast);
pytest --device mps.
scripts/label_efficiency.py runs label-efficiency sweeps.
Tested on fixtures on CPU and MPS (114 passed; CUDA-only tests not run);
first TabPFN-3.5 GamGam numbers in docs/design.md. Existing models'
features, splits, losses and metrics are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- configs/delphes_contrastive.yaml +81 -0
- configs/delphes_masked.yaml +78 -0
- configs/delphes_tabicl.yaml +89 -0
- configs/delphes_tabpfn.yaml +89 -0
- configs/gamgam_tabdpt.yaml +91 -0
- configs/gamgam_tabicl.yaml +95 -0
- configs/gamgam_tabpfn.yaml +8 -0
- docs/design.md +89 -12
- docs/usage.md +85 -6
- pyproject.toml +6 -0
- scripts/label_efficiency.py +83 -0
- src/gnn4colliders/cli.py +107 -8
- src/gnn4colliders/config.py +59 -10
- src/gnn4colliders/contrastive.py +112 -0
- src/gnn4colliders/distributed.py +11 -1
- src/gnn4colliders/jepa.py +68 -6
- src/gnn4colliders/models.py +177 -42
- src/gnn4colliders/pfn.py +7 -11
- src/gnn4colliders/probe.py +161 -0
- src/gnn4colliders/sequences.py +27 -1
- src/gnn4colliders/tabular.py +644 -37
- src/gnn4colliders/tasks.py +28 -12
- tests/conftest.py +13 -2
- tests/test_contrastive.py +123 -0
- tests/test_jepa.py +35 -5
- tests/test_probe.py +152 -0
- tests/test_tabular.py +341 -7
- tests/test_workflows.py +4 -1
- uv.lock +137 -26
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Contrastive (SimCLR) pretraining of the object transformer on the 12
|
| 2 |
+
# Delphes samples of configs/delphes_jepa.yaml, without their labels (same
|
| 3 |
+
# event cache): two random views of every event (rotation, eta flip, energy
|
| 4 |
+
# scale, object dropout) are pulled together (src/gnn4colliders/contrastive.py).
|
| 5 |
+
# Fine-tune with configs/gamgam_jepa_finetune.yaml
|
| 6 |
+
# model.pretrained=outputs/delphes_contrastive/last.pt, or compare encoders
|
| 7 |
+
# without fine-tuning: gnn4colliders probe configs/gamgam_jepa_finetune.yaml
|
| 8 |
+
# data.train_events=1000 split=validation model.pretrained=...
|
| 9 |
+
#
|
| 10 |
+
# gnn4colliders train configs/delphes_contrastive.yaml
|
| 11 |
+
name: delphes_contrastive
|
| 12 |
+
|
| 13 |
+
data:
|
| 14 |
+
hf_repo: HWresearch/Delphes
|
| 15 |
+
hf_revision: 76a6c362bfba8e766ba7255a5c08a55257f78d0e
|
| 16 |
+
files:
|
| 17 |
+
- {path: samples/higgs/top-associated/tth/ttH_NLO_inc.root, label: 0}
|
| 18 |
+
- {path: samples/higgs/top-associated/thjb/tHjb_NLO_inc.root, label: 1}
|
| 19 |
+
- {path: samples/higgs/ggf/ggF_NLO_inc.root, label: 2}
|
| 20 |
+
- {path: samples/higgs/vbf/VBF_NLO_inc.root, label: 3}
|
| 21 |
+
- {path: samples/higgs/vh/wh/WH_NLO_inc.root, label: 4}
|
| 22 |
+
- {path: samples/higgs/vh/zh/ZH_NLO_inc.root, label: 5}
|
| 23 |
+
- {path: samples/photon/ttyy/ttyy.root, label: 6}
|
| 24 |
+
- {path: samples/top/multitop/tttt.root, label: 7}
|
| 25 |
+
- {path: samples/top/single-top/SingleT_schan.root, label: 8}
|
| 26 |
+
- {path: samples/top/ttbar/ttbar.root, label: 9}
|
| 27 |
+
- {path: samples/top/ttv/ttW.root, label: 10}
|
| 28 |
+
- {path: samples/top/multitop/ttt.root, label: 11}
|
| 29 |
+
tree_name: output
|
| 30 |
+
feature_branches:
|
| 31 |
+
- [jet_pt, ele_pt, mu_pt, ph_pt, MET_met]
|
| 32 |
+
- [jet_eta, ele_eta, mu_eta, ph_eta, 0]
|
| 33 |
+
- [jet_phi, ele_phi, mu_phi, ph_phi, MET_phi]
|
| 34 |
+
- CALC_E
|
| 35 |
+
- [jet_btag, 0, 0, 0, 0]
|
| 36 |
+
- [0, ele_charge, mu_charge, 0, 0]
|
| 37 |
+
- NODE_TYPE
|
| 38 |
+
object_types: [vector, vector, vector, vector, single]
|
| 39 |
+
scales: [0.1, 1, 1, 0.1, 1, 1, 1]
|
| 40 |
+
# Number is the event index; Number % 10 is the fold. Pretraining trained
|
| 41 |
+
# on folds 2-9 and tested on 0-1; here fold 0 validates and fold 1 tests.
|
| 42 |
+
fold_var: Number
|
| 43 |
+
fold_modulus: 10
|
| 44 |
+
train_folds: [2, 3, 4, 5, 6, 7, 8, 9]
|
| 45 |
+
validation_folds: [0]
|
| 46 |
+
test_folds: [1]
|
| 47 |
+
# JEPA ignores labels and weights; the cache is that of
|
| 48 |
+
# configs/delphes_pretraining.yaml.
|
| 49 |
+
weight_var: weight
|
| 50 |
+
cache: outputs/delphes_pretraining/events.pt
|
| 51 |
+
batch_size: 1024
|
| 52 |
+
prepare_workers: 8
|
| 53 |
+
loader_workers: 4
|
| 54 |
+
pin_memory: true
|
| 55 |
+
|
| 56 |
+
model:
|
| 57 |
+
name: contrastive
|
| 58 |
+
backbone: transformer
|
| 59 |
+
# The encoder; configs/gamgam_jepa_finetune.yaml must match it.
|
| 60 |
+
hid_size: 128
|
| 61 |
+
n_layers: 4
|
| 62 |
+
n_heads: 8
|
| 63 |
+
ffn_size: 512
|
| 64 |
+
dropout: 0.0
|
| 65 |
+
pair_bias: true
|
| 66 |
+
temperature: 0.1
|
| 67 |
+
scale_jitter: 0.1
|
| 68 |
+
object_dropout: 0.1
|
| 69 |
+
|
| 70 |
+
task:
|
| 71 |
+
name: contrastive
|
| 72 |
+
|
| 73 |
+
train:
|
| 74 |
+
max_epochs: 30
|
| 75 |
+
optimizer: adamw
|
| 76 |
+
learning_rate: 5.0e-4
|
| 77 |
+
weight_decay: 0.05
|
| 78 |
+
lr_decay: 0.95
|
| 79 |
+
early_stopping: null
|
| 80 |
+
# Fine-tune from last.pt.
|
| 81 |
+
best_metric: loss
|
|
@@ -0,0 +1,78 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Masked object modelling of the object transformer on the 12 Delphes
|
| 2 |
+
# samples of configs/delphes_jepa.yaml, without their labels (same event
|
| 3 |
+
# cache): JEPA's masking and predictor, but the predictor predicts the hidden
|
| 4 |
+
# objects' kinematics instead of their target encodings (model.jepa_target:
|
| 5 |
+
# features; src/gnn4colliders/jepa.py). Fine-tune or probe with
|
| 6 |
+
# configs/gamgam_jepa_finetune.yaml model.pretrained=outputs/delphes_masked/last.pt.
|
| 7 |
+
#
|
| 8 |
+
# gnn4colliders train configs/delphes_masked.yaml
|
| 9 |
+
name: delphes_masked
|
| 10 |
+
|
| 11 |
+
data:
|
| 12 |
+
hf_repo: HWresearch/Delphes
|
| 13 |
+
hf_revision: 76a6c362bfba8e766ba7255a5c08a55257f78d0e
|
| 14 |
+
files:
|
| 15 |
+
- {path: samples/higgs/top-associated/tth/ttH_NLO_inc.root, label: 0}
|
| 16 |
+
- {path: samples/higgs/top-associated/thjb/tHjb_NLO_inc.root, label: 1}
|
| 17 |
+
- {path: samples/higgs/ggf/ggF_NLO_inc.root, label: 2}
|
| 18 |
+
- {path: samples/higgs/vbf/VBF_NLO_inc.root, label: 3}
|
| 19 |
+
- {path: samples/higgs/vh/wh/WH_NLO_inc.root, label: 4}
|
| 20 |
+
- {path: samples/higgs/vh/zh/ZH_NLO_inc.root, label: 5}
|
| 21 |
+
- {path: samples/photon/ttyy/ttyy.root, label: 6}
|
| 22 |
+
- {path: samples/top/multitop/tttt.root, label: 7}
|
| 23 |
+
- {path: samples/top/single-top/SingleT_schan.root, label: 8}
|
| 24 |
+
- {path: samples/top/ttbar/ttbar.root, label: 9}
|
| 25 |
+
- {path: samples/top/ttv/ttW.root, label: 10}
|
| 26 |
+
- {path: samples/top/multitop/ttt.root, label: 11}
|
| 27 |
+
tree_name: output
|
| 28 |
+
feature_branches:
|
| 29 |
+
- [jet_pt, ele_pt, mu_pt, ph_pt, MET_met]
|
| 30 |
+
- [jet_eta, ele_eta, mu_eta, ph_eta, 0]
|
| 31 |
+
- [jet_phi, ele_phi, mu_phi, ph_phi, MET_phi]
|
| 32 |
+
- CALC_E
|
| 33 |
+
- [jet_btag, 0, 0, 0, 0]
|
| 34 |
+
- [0, ele_charge, mu_charge, 0, 0]
|
| 35 |
+
- NODE_TYPE
|
| 36 |
+
object_types: [vector, vector, vector, vector, single]
|
| 37 |
+
scales: [0.1, 1, 1, 0.1, 1, 1, 1]
|
| 38 |
+
# Number is the event index; Number % 10 is the fold. Pretraining trained
|
| 39 |
+
# on folds 2-9 and tested on 0-1; here fold 0 validates and fold 1 tests.
|
| 40 |
+
fold_var: Number
|
| 41 |
+
fold_modulus: 10
|
| 42 |
+
train_folds: [2, 3, 4, 5, 6, 7, 8, 9]
|
| 43 |
+
validation_folds: [0]
|
| 44 |
+
test_folds: [1]
|
| 45 |
+
# JEPA ignores labels and weights; the cache is that of
|
| 46 |
+
# configs/delphes_pretraining.yaml.
|
| 47 |
+
weight_var: weight
|
| 48 |
+
cache: outputs/delphes_pretraining/events.pt
|
| 49 |
+
batch_size: 1024
|
| 50 |
+
prepare_workers: 8
|
| 51 |
+
loader_workers: 4
|
| 52 |
+
pin_memory: true
|
| 53 |
+
|
| 54 |
+
model:
|
| 55 |
+
name: jepa
|
| 56 |
+
jepa_target: features
|
| 57 |
+
# The encoder; configs/gamgam_jepa_finetune.yaml must match it.
|
| 58 |
+
hid_size: 128
|
| 59 |
+
n_layers: 4
|
| 60 |
+
n_heads: 8
|
| 61 |
+
ffn_size: 512
|
| 62 |
+
dropout: 0.0
|
| 63 |
+
pair_bias: true
|
| 64 |
+
predictor_layers: 2
|
| 65 |
+
mask_fraction: 0.4
|
| 66 |
+
|
| 67 |
+
task:
|
| 68 |
+
name: jepa
|
| 69 |
+
|
| 70 |
+
train:
|
| 71 |
+
max_epochs: 30
|
| 72 |
+
optimizer: adamw
|
| 73 |
+
learning_rate: 5.0e-4
|
| 74 |
+
weight_decay: 0.05
|
| 75 |
+
lr_decay: 0.95
|
| 76 |
+
early_stopping: null
|
| 77 |
+
# Fine-tune from last.pt.
|
| 78 |
+
best_metric: loss
|
|
@@ -0,0 +1,89 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Meta-training of the TabICL network (src/gnn4colliders/tabular.py; BSD-3,
|
| 2 |
+
# so meta-trained weights can be shared freely) on the classification tasks
|
| 3 |
+
# of configs/delphes_pfn.yaml, on the fixed-width event rows of
|
| 4 |
+
# configs/gamgam_tabicl.yaml. By default TabICL is frozen and its in-context
|
| 5 |
+
# blocks learn LoRA updates (model.lora_rank; 0 trains all 28M weights).
|
| 6 |
+
# task.prior_classes=[1,6,7,8,9,10,11] leaves out the five processes of the
|
| 7 |
+
# GamGam task. Evaluate in context with
|
| 8 |
+
# gnn4colliders evaluate configs/gamgam_tabicl.yaml split=validation \
|
| 9 |
+
# checkpoint=outputs/delphes_tabicl/best.pt
|
| 10 |
+
# Needs `uv sync --extra tabicl`.
|
| 11 |
+
#
|
| 12 |
+
# gnn4colliders train configs/delphes_tabicl.yaml
|
| 13 |
+
name: delphes_tabicl
|
| 14 |
+
|
| 15 |
+
data:
|
| 16 |
+
hf_repo: HWresearch/Delphes
|
| 17 |
+
hf_revision: 76a6c362bfba8e766ba7255a5c08a55257f78d0e
|
| 18 |
+
files:
|
| 19 |
+
- {path: samples/higgs/top-associated/tth/ttH_NLO_inc.root, label: 0}
|
| 20 |
+
- {path: samples/higgs/top-associated/thjb/tHjb_NLO_inc.root, label: 1}
|
| 21 |
+
- {path: samples/higgs/ggf/ggF_NLO_inc.root, label: 2}
|
| 22 |
+
- {path: samples/higgs/vbf/VBF_NLO_inc.root, label: 3}
|
| 23 |
+
- {path: samples/higgs/vh/wh/WH_NLO_inc.root, label: 4}
|
| 24 |
+
- {path: samples/higgs/vh/zh/ZH_NLO_inc.root, label: 5}
|
| 25 |
+
- {path: samples/photon/ttyy/ttyy.root, label: 6}
|
| 26 |
+
- {path: samples/top/multitop/tttt.root, label: 7}
|
| 27 |
+
- {path: samples/top/single-top/SingleT_schan.root, label: 8}
|
| 28 |
+
- {path: samples/top/ttbar/ttbar.root, label: 9}
|
| 29 |
+
- {path: samples/top/ttv/ttW.root, label: 10}
|
| 30 |
+
- {path: samples/top/multitop/ttt.root, label: 11}
|
| 31 |
+
tree_name: output
|
| 32 |
+
feature_branches:
|
| 33 |
+
- [jet_pt, ele_pt, mu_pt, ph_pt, MET_met]
|
| 34 |
+
- [jet_eta, ele_eta, mu_eta, ph_eta, 0]
|
| 35 |
+
- [jet_phi, ele_phi, mu_phi, ph_phi, MET_phi]
|
| 36 |
+
- CALC_E
|
| 37 |
+
- [jet_btag, 0, 0, 0, 0]
|
| 38 |
+
- [0, ele_charge, mu_charge, 0, 0]
|
| 39 |
+
- NODE_TYPE
|
| 40 |
+
object_types: [vector, vector, vector, vector, single]
|
| 41 |
+
scales: [0.1, 1, 1, 0.1, 1, 1, 1]
|
| 42 |
+
# Number is the event index; Number % 10 is the fold. Pretraining trained
|
| 43 |
+
# on folds 2-9 and tested on 0-1; here fold 0 validates and fold 1 tests.
|
| 44 |
+
fold_var: Number
|
| 45 |
+
fold_modulus: 10
|
| 46 |
+
train_folds: [2, 3, 4, 5, 6, 7, 8, 9]
|
| 47 |
+
validation_folds: [0]
|
| 48 |
+
test_folds: [1]
|
| 49 |
+
# Tasks use the labels, not the weights; the cache is that of
|
| 50 |
+
# configs/delphes_pretraining.yaml.
|
| 51 |
+
weight_var: weight
|
| 52 |
+
cache: outputs/delphes_pretraining/events.pt
|
| 53 |
+
# Events per step (tasks of context and query events). TabICL (28M
|
| 54 |
+
# weights) recomputes activations in the backward pass.
|
| 55 |
+
batch_size: 4096
|
| 56 |
+
prepare_workers: 8
|
| 57 |
+
loader_workers: 4
|
| 58 |
+
pin_memory: true
|
| 59 |
+
|
| 60 |
+
model:
|
| 61 |
+
name: tabicl
|
| 62 |
+
tabicl_checkpoint: tabicl-classifier-v2-20260212.ckpt
|
| 63 |
+
# Leading jets, electrons, muons, photons and MET in a table row.
|
| 64 |
+
slots: [6, 2, 2, 3, 1]
|
| 65 |
+
# As in configs/gamgam_tabicl.yaml, which evaluates this model.
|
| 66 |
+
relative_phi: false
|
| 67 |
+
pair_objects: 0
|
| 68 |
+
max_classes: 10
|
| 69 |
+
lora_rank: 8
|
| 70 |
+
|
| 71 |
+
task:
|
| 72 |
+
name: pfn
|
| 73 |
+
# The prior: classes per task, context events (log-uniform) and queries.
|
| 74 |
+
prior_classes: null
|
| 75 |
+
task_classes: [2, 10]
|
| 76 |
+
context_events: [16, 2048]
|
| 77 |
+
query_events: 256
|
| 78 |
+
energy_jitter: 0.2
|
| 79 |
+
steps_per_epoch: 500
|
| 80 |
+
validation_steps: 50
|
| 81 |
+
|
| 82 |
+
train:
|
| 83 |
+
max_epochs: 20
|
| 84 |
+
optimizer: adamw
|
| 85 |
+
learning_rate: 1.0e-4
|
| 86 |
+
weight_decay: 0.01
|
| 87 |
+
lr_decay: 0.95
|
| 88 |
+
early_stopping: null
|
| 89 |
+
best_metric: loss
|
|
@@ -0,0 +1,89 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Meta-training of the TabPFN-3.5 network (src/gnn4colliders/tabular.py) on
|
| 2 |
+
# the classification tasks of configs/delphes_pfn.yaml, on the fixed-width
|
| 3 |
+
# event rows of configs/gamgam_tabpfn.yaml. By default TabPFN is frozen and
|
| 4 |
+
# its in-context blocks learn LoRA updates (model.lora_rank; 0 trains all
|
| 5 |
+
# weights). task.prior_classes=[1,6,7,8,9,10,11] leaves out the five
|
| 6 |
+
# processes of the GamGam task. Evaluate in context with
|
| 7 |
+
# gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation \
|
| 8 |
+
# checkpoint=outputs/delphes_tabpfn/best.pt
|
| 9 |
+
# Needs `uv sync --extra tabpfn` and the TabPFN-3.5 weights (docs/usage.md).
|
| 10 |
+
#
|
| 11 |
+
# gnn4colliders train configs/delphes_tabpfn.yaml
|
| 12 |
+
name: delphes_tabpfn
|
| 13 |
+
|
| 14 |
+
data:
|
| 15 |
+
hf_repo: HWresearch/Delphes
|
| 16 |
+
hf_revision: 76a6c362bfba8e766ba7255a5c08a55257f78d0e
|
| 17 |
+
files:
|
| 18 |
+
- {path: samples/higgs/top-associated/tth/ttH_NLO_inc.root, label: 0}
|
| 19 |
+
- {path: samples/higgs/top-associated/thjb/tHjb_NLO_inc.root, label: 1}
|
| 20 |
+
- {path: samples/higgs/ggf/ggF_NLO_inc.root, label: 2}
|
| 21 |
+
- {path: samples/higgs/vbf/VBF_NLO_inc.root, label: 3}
|
| 22 |
+
- {path: samples/higgs/vh/wh/WH_NLO_inc.root, label: 4}
|
| 23 |
+
- {path: samples/higgs/vh/zh/ZH_NLO_inc.root, label: 5}
|
| 24 |
+
- {path: samples/photon/ttyy/ttyy.root, label: 6}
|
| 25 |
+
- {path: samples/top/multitop/tttt.root, label: 7}
|
| 26 |
+
- {path: samples/top/single-top/SingleT_schan.root, label: 8}
|
| 27 |
+
- {path: samples/top/ttbar/ttbar.root, label: 9}
|
| 28 |
+
- {path: samples/top/ttv/ttW.root, label: 10}
|
| 29 |
+
- {path: samples/top/multitop/ttt.root, label: 11}
|
| 30 |
+
tree_name: output
|
| 31 |
+
feature_branches:
|
| 32 |
+
- [jet_pt, ele_pt, mu_pt, ph_pt, MET_met]
|
| 33 |
+
- [jet_eta, ele_eta, mu_eta, ph_eta, 0]
|
| 34 |
+
- [jet_phi, ele_phi, mu_phi, ph_phi, MET_phi]
|
| 35 |
+
- CALC_E
|
| 36 |
+
- [jet_btag, 0, 0, 0, 0]
|
| 37 |
+
- [0, ele_charge, mu_charge, 0, 0]
|
| 38 |
+
- NODE_TYPE
|
| 39 |
+
object_types: [vector, vector, vector, vector, single]
|
| 40 |
+
scales: [0.1, 1, 1, 0.1, 1, 1, 1]
|
| 41 |
+
# Number is the event index; Number % 10 is the fold. Pretraining trained
|
| 42 |
+
# on folds 2-9 and tested on 0-1; here fold 0 validates and fold 1 tests.
|
| 43 |
+
fold_var: Number
|
| 44 |
+
fold_modulus: 10
|
| 45 |
+
train_folds: [2, 3, 4, 5, 6, 7, 8, 9]
|
| 46 |
+
validation_folds: [0]
|
| 47 |
+
test_folds: [1]
|
| 48 |
+
# Tasks use the labels, not the weights; the cache is that of
|
| 49 |
+
# configs/delphes_pretraining.yaml.
|
| 50 |
+
weight_var: weight
|
| 51 |
+
cache: outputs/delphes_pretraining/events.pt
|
| 52 |
+
# Events per step (tasks of context and query events): one task of the
|
| 53 |
+
# largest context. TabPFN recomputes activations in the backward pass.
|
| 54 |
+
batch_size: 2304
|
| 55 |
+
prepare_workers: 8
|
| 56 |
+
loader_workers: 4
|
| 57 |
+
pin_memory: true
|
| 58 |
+
|
| 59 |
+
model:
|
| 60 |
+
name: tabpfn
|
| 61 |
+
tabpfn_version: v3.5
|
| 62 |
+
tabpfn_checkpoint: null
|
| 63 |
+
# Leading jets, electrons, muons, photons and MET in a table row.
|
| 64 |
+
slots: [6, 2, 2, 3, 1]
|
| 65 |
+
# As in configs/gamgam_tabpfn.yaml, which evaluates this model.
|
| 66 |
+
relative_phi: false
|
| 67 |
+
pair_objects: 0
|
| 68 |
+
max_classes: 10
|
| 69 |
+
lora_rank: 8
|
| 70 |
+
|
| 71 |
+
task:
|
| 72 |
+
name: pfn
|
| 73 |
+
# The prior: classes per task, context events (log-uniform) and queries.
|
| 74 |
+
prior_classes: null
|
| 75 |
+
task_classes: [2, 10]
|
| 76 |
+
context_events: [16, 2048]
|
| 77 |
+
query_events: 256
|
| 78 |
+
energy_jitter: 0.2
|
| 79 |
+
steps_per_epoch: 500
|
| 80 |
+
validation_steps: 50
|
| 81 |
+
|
| 82 |
+
train:
|
| 83 |
+
max_epochs: 20
|
| 84 |
+
optimizer: adamw
|
| 85 |
+
learning_rate: 1.0e-4
|
| 86 |
+
weight_decay: 0.01
|
| 87 |
+
lr_decay: 0.95
|
| 88 |
+
early_stopping: null
|
| 89 |
+
best_metric: loss
|
|
@@ -0,0 +1,91 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# The GamGam 5-class task classified in context by TabDPT (Layer 6;
|
| 2 |
+
# Apache-2.0 weights from Hugging Face), with no training: data.train_events
|
| 3 |
+
# labelled training events, an equal number per class, are the context of
|
| 4 |
+
# every prediction (src/gnn4colliders/tabular.py). TabDPT is trained on real
|
| 5 |
+
# tables; with model.tabdpt_context_size each event's context is its nearest
|
| 6 |
+
# context events (retrieval), which allows contexts beyond what fits at once.
|
| 7 |
+
# Needs `uv sync --extra tabdpt`. Uses the cache of configs/gamgam.yaml.
|
| 8 |
+
#
|
| 9 |
+
# gnn4colliders evaluate configs/gamgam_tabdpt.yaml split=validation
|
| 10 |
+
name: gamgam_tabdpt
|
| 11 |
+
|
| 12 |
+
data:
|
| 13 |
+
hf_repo: ho22joshua/atlas_opendata
|
| 14 |
+
hf_revision: d1bcc9ccaa304ea09e11f7d724342568f72a0d6e
|
| 15 |
+
files:
|
| 16 |
+
- path: GamGam/2020/MC/mc_341081.ttH125_gamgam.GamGam.root
|
| 17 |
+
label: 0
|
| 18 |
+
size_bytes: 217819076
|
| 19 |
+
sha256: c498974926f6c55143d17fc4d3d6fa69bd3718ff92cf7d5cad5c3d67a836c7b7
|
| 20 |
+
- path: GamGam/2020/MC/mc_343981.ggH125_gamgam.GamGam.root
|
| 21 |
+
label: 1
|
| 22 |
+
size_bytes: 218418347
|
| 23 |
+
sha256: 05b3e6709e8836a7b862a7fde09086b3a9d4d494d62a6984d5df39d8daad13eb
|
| 24 |
+
- path: GamGam/2020/MC/mc_345041.VBFH125_gamgam.GamGam.root
|
| 25 |
+
label: 2
|
| 26 |
+
size_bytes: 108616148
|
| 27 |
+
sha256: 92334fce3bffc9eb6e0745dd6ef354d5ce33da764858511b8f9ca49b862b44e2
|
| 28 |
+
- path: GamGam/2020/MC/mc_345318.WpH125J_Wincl_gamgam.GamGam.root
|
| 29 |
+
label: 3
|
| 30 |
+
size_bytes: 29757932
|
| 31 |
+
sha256: f255a9c3c6935f976f788b6f32d9a8f6bf576eb12ecd03a6f3fd0a5dea9d1248
|
| 32 |
+
- path: GamGam/2020/MC/mc_345319.ZH125J_Zincl_gamgam.GamGam.root
|
| 33 |
+
label: 4
|
| 34 |
+
size_bytes: 58701804
|
| 35 |
+
sha256: c85623b04ac0176c64acdb2ecf8e992a63824cfc430a421df742319518ff6748
|
| 36 |
+
class_names: [ttH, ggF, VBF, WH, ZH]
|
| 37 |
+
tree_name: mini
|
| 38 |
+
# Electrons and muons share the lep_* collection; b-tagging uses the
|
| 39 |
+
# MV2c10 70% efficiency working point of the Open Data documentation.
|
| 40 |
+
derived_branches:
|
| 41 |
+
ele_pt: {source: lep_pt, where: {lep_type: 11}}
|
| 42 |
+
ele_eta: {source: lep_eta, where: {lep_type: 11}}
|
| 43 |
+
ele_phi: {source: lep_phi, where: {lep_type: 11}}
|
| 44 |
+
ele_charge: {source: lep_charge, where: {lep_type: 11}}
|
| 45 |
+
mu_pt: {source: lep_pt, where: {lep_type: 13}}
|
| 46 |
+
mu_eta: {source: lep_eta, where: {lep_type: 13}}
|
| 47 |
+
mu_phi: {source: lep_phi, where: {lep_type: 13}}
|
| 48 |
+
mu_charge: {source: lep_charge, where: {lep_type: 13}}
|
| 49 |
+
jet_btag: {source: jet_MV2c10, above: 0.8244273}
|
| 50 |
+
# Object types (node types) as in pretraining: jet, electron, muon, photon, MET.
|
| 51 |
+
feature_branches:
|
| 52 |
+
- [jet_pt, ele_pt, mu_pt, photon_pt, met_et]
|
| 53 |
+
- [jet_eta, ele_eta, mu_eta, photon_eta, 0]
|
| 54 |
+
- [jet_phi, ele_phi, mu_phi, photon_phi, met_phi]
|
| 55 |
+
- CALC_E
|
| 56 |
+
- [jet_btag, 0, 0, 0, 0]
|
| 57 |
+
- [0, ele_charge, mu_charge, 0, 0]
|
| 58 |
+
- NODE_TYPE
|
| 59 |
+
object_types: [vector, vector, vector, vector, single]
|
| 60 |
+
# Momenta are in MeV; 1e-4 gives the pretraining unit of 10 GeV.
|
| 61 |
+
scales: [1.0e-4, 1, 1, 1.0e-4, 1, 1, 1]
|
| 62 |
+
# eventNumber % 10 gives the paper's 80:10:10 train:validation:test split.
|
| 63 |
+
fold_var: eventNumber
|
| 64 |
+
fold_modulus: 10
|
| 65 |
+
train_folds: [0, 1, 2, 3, 4, 5, 6, 7]
|
| 66 |
+
validation_folds: [8]
|
| 67 |
+
test_folds: [9]
|
| 68 |
+
# Labelled events (equal per class) that form the context.
|
| 69 |
+
train_events: 1000
|
| 70 |
+
# Unweighted: each event counts once and classes are balanced by the
|
| 71 |
+
# per-class mean loss (23% of ttH events have mcWeight = -1).
|
| 72 |
+
weight_var: null
|
| 73 |
+
cache: outputs/gamgam/events.pt
|
| 74 |
+
# Events classified per TabDPT call.
|
| 75 |
+
batch_size: 8192
|
| 76 |
+
prepare_workers: 8
|
| 77 |
+
loader_workers: 4
|
| 78 |
+
pin_memory: true
|
| 79 |
+
|
| 80 |
+
model:
|
| 81 |
+
name: tabdpt
|
| 82 |
+
out_size: 5
|
| 83 |
+
tabdpt_estimators: 8
|
| 84 |
+
tabdpt_context_size: null
|
| 85 |
+
# Leading jets, electrons, muons, photons and MET in a table row.
|
| 86 |
+
slots: [6, 2, 2, 3, 1]
|
| 87 |
+
relative_phi: false
|
| 88 |
+
pair_objects: 0
|
| 89 |
+
|
| 90 |
+
task:
|
| 91 |
+
name: multiclass
|
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# The GamGam 5-class task classified in context by TabICLv2 (Inria; BSD-3
|
| 2 |
+
# weights from Hugging Face), with no training: data.train_events labelled
|
| 3 |
+
# training events, an equal number per class, are the context of every
|
| 4 |
+
# prediction (src/gnn4colliders/tabular.py). Needs `uv sync --extra tabicl`.
|
| 5 |
+
# Uses the cache of configs/gamgam.yaml. The rows are those of
|
| 6 |
+
# configs/gamgam_tabpfn.yaml; model.relative_phi=true and
|
| 7 |
+
# model.pair_objects=4 add the physics features.
|
| 8 |
+
#
|
| 9 |
+
# gnn4colliders evaluate configs/gamgam_tabicl.yaml split=validation
|
| 10 |
+
#
|
| 11 |
+
# With checkpoint=outputs/delphes_tabicl/best.pt, the TabICL network
|
| 12 |
+
# meta-trained by configs/delphes_tabicl.yaml (model.lora_rank and
|
| 13 |
+
# model.max_classes must match it) classifies with its own forward pass.
|
| 14 |
+
name: gamgam_tabicl
|
| 15 |
+
|
| 16 |
+
data:
|
| 17 |
+
hf_repo: ho22joshua/atlas_opendata
|
| 18 |
+
hf_revision: d1bcc9ccaa304ea09e11f7d724342568f72a0d6e
|
| 19 |
+
files:
|
| 20 |
+
- path: GamGam/2020/MC/mc_341081.ttH125_gamgam.GamGam.root
|
| 21 |
+
label: 0
|
| 22 |
+
size_bytes: 217819076
|
| 23 |
+
sha256: c498974926f6c55143d17fc4d3d6fa69bd3718ff92cf7d5cad5c3d67a836c7b7
|
| 24 |
+
- path: GamGam/2020/MC/mc_343981.ggH125_gamgam.GamGam.root
|
| 25 |
+
label: 1
|
| 26 |
+
size_bytes: 218418347
|
| 27 |
+
sha256: 05b3e6709e8836a7b862a7fde09086b3a9d4d494d62a6984d5df39d8daad13eb
|
| 28 |
+
- path: GamGam/2020/MC/mc_345041.VBFH125_gamgam.GamGam.root
|
| 29 |
+
label: 2
|
| 30 |
+
size_bytes: 108616148
|
| 31 |
+
sha256: 92334fce3bffc9eb6e0745dd6ef354d5ce33da764858511b8f9ca49b862b44e2
|
| 32 |
+
- path: GamGam/2020/MC/mc_345318.WpH125J_Wincl_gamgam.GamGam.root
|
| 33 |
+
label: 3
|
| 34 |
+
size_bytes: 29757932
|
| 35 |
+
sha256: f255a9c3c6935f976f788b6f32d9a8f6bf576eb12ecd03a6f3fd0a5dea9d1248
|
| 36 |
+
- path: GamGam/2020/MC/mc_345319.ZH125J_Zincl_gamgam.GamGam.root
|
| 37 |
+
label: 4
|
| 38 |
+
size_bytes: 58701804
|
| 39 |
+
sha256: c85623b04ac0176c64acdb2ecf8e992a63824cfc430a421df742319518ff6748
|
| 40 |
+
class_names: [ttH, ggF, VBF, WH, ZH]
|
| 41 |
+
tree_name: mini
|
| 42 |
+
# Electrons and muons share the lep_* collection; b-tagging uses the
|
| 43 |
+
# MV2c10 70% efficiency working point of the Open Data documentation.
|
| 44 |
+
derived_branches:
|
| 45 |
+
ele_pt: {source: lep_pt, where: {lep_type: 11}}
|
| 46 |
+
ele_eta: {source: lep_eta, where: {lep_type: 11}}
|
| 47 |
+
ele_phi: {source: lep_phi, where: {lep_type: 11}}
|
| 48 |
+
ele_charge: {source: lep_charge, where: {lep_type: 11}}
|
| 49 |
+
mu_pt: {source: lep_pt, where: {lep_type: 13}}
|
| 50 |
+
mu_eta: {source: lep_eta, where: {lep_type: 13}}
|
| 51 |
+
mu_phi: {source: lep_phi, where: {lep_type: 13}}
|
| 52 |
+
mu_charge: {source: lep_charge, where: {lep_type: 13}}
|
| 53 |
+
jet_btag: {source: jet_MV2c10, above: 0.8244273}
|
| 54 |
+
# Object types (node types) as in pretraining: jet, electron, muon, photon, MET.
|
| 55 |
+
feature_branches:
|
| 56 |
+
- [jet_pt, ele_pt, mu_pt, photon_pt, met_et]
|
| 57 |
+
- [jet_eta, ele_eta, mu_eta, photon_eta, 0]
|
| 58 |
+
- [jet_phi, ele_phi, mu_phi, photon_phi, met_phi]
|
| 59 |
+
- CALC_E
|
| 60 |
+
- [jet_btag, 0, 0, 0, 0]
|
| 61 |
+
- [0, ele_charge, mu_charge, 0, 0]
|
| 62 |
+
- NODE_TYPE
|
| 63 |
+
object_types: [vector, vector, vector, vector, single]
|
| 64 |
+
# Momenta are in MeV; 1e-4 gives the pretraining unit of 10 GeV.
|
| 65 |
+
scales: [1.0e-4, 1, 1, 1.0e-4, 1, 1, 1]
|
| 66 |
+
# eventNumber % 10 gives the paper's 80:10:10 train:validation:test split.
|
| 67 |
+
fold_var: eventNumber
|
| 68 |
+
fold_modulus: 10
|
| 69 |
+
train_folds: [0, 1, 2, 3, 4, 5, 6, 7]
|
| 70 |
+
validation_folds: [8]
|
| 71 |
+
test_folds: [9]
|
| 72 |
+
# Labelled events (equal per class) that form the context.
|
| 73 |
+
train_events: 1000
|
| 74 |
+
# Unweighted: each event counts once and classes are balanced by the
|
| 75 |
+
# per-class mean loss (23% of ttH events have mcWeight = -1).
|
| 76 |
+
weight_var: null
|
| 77 |
+
cache: outputs/gamgam/events.pt
|
| 78 |
+
# Events classified per TabICL call.
|
| 79 |
+
batch_size: 8192
|
| 80 |
+
prepare_workers: 8
|
| 81 |
+
loader_workers: 4
|
| 82 |
+
pin_memory: true
|
| 83 |
+
|
| 84 |
+
model:
|
| 85 |
+
name: tabicl
|
| 86 |
+
out_size: 5
|
| 87 |
+
tabicl_checkpoint: tabicl-classifier-v2-20260212.ckpt
|
| 88 |
+
tabicl_estimators: 8
|
| 89 |
+
# Leading jets, electrons, muons, photons and MET in a table row.
|
| 90 |
+
slots: [6, 2, 2, 3, 1]
|
| 91 |
+
relative_phi: false
|
| 92 |
+
pair_objects: 0
|
| 93 |
+
|
| 94 |
+
task:
|
| 95 |
+
name: multiclass
|
|
@@ -6,6 +6,10 @@
|
|
| 6 |
# configs/gamgam.yaml.
|
| 7 |
#
|
| 8 |
# gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
name: gamgam_tabpfn
|
| 10 |
|
| 11 |
data:
|
|
@@ -83,6 +87,10 @@ model:
|
|
| 83 |
tabpfn_estimators: 4
|
| 84 |
# Leading jets, electrons, muons, photons and MET in a table row.
|
| 85 |
slots: [6, 2, 2, 3, 1]
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
|
| 87 |
task:
|
| 88 |
name: multiclass
|
|
|
|
| 6 |
# configs/gamgam.yaml.
|
| 7 |
#
|
| 8 |
# gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation
|
| 9 |
+
#
|
| 10 |
+
# With checkpoint=outputs/delphes_tabpfn/best.pt, the TabPFN network
|
| 11 |
+
# meta-trained by configs/delphes_tabpfn.yaml (model.lora_rank and
|
| 12 |
+
# model.max_classes must match it) classifies with its own forward pass.
|
| 13 |
name: gamgam_tabpfn
|
| 14 |
|
| 15 |
data:
|
|
|
|
| 87 |
tabpfn_estimators: 4
|
| 88 |
# Leading jets, electrons, muons, photons and MET in a table row.
|
| 89 |
slots: [6, 2, 2, 3, 1]
|
| 90 |
+
# Physics features: azimuths from the leading object's, and the mass and
|
| 91 |
+
# Delta R of every pair of the leading pair_objects objects.
|
| 92 |
+
relative_phi: false
|
| 93 |
+
pair_objects: 0
|
| 94 |
|
| 95 |
task:
|
| 96 |
name: multiclass
|
|
@@ -8,16 +8,20 @@
|
|
| 8 |
| `data.py` | ROOT reading, Hugging Face download, derived branches, object features, `PackedEvents` and its cache |
|
| 9 |
| `loader.py` | `EventLoader`: deterministic shuffling, DDP sharding, collation workers |
|
| 10 |
| `graphs.py` | fully connected `TensorGraph` batches, edge features, CUDA-graph padding buffers |
|
| 11 |
-
| `sequences.py` | padded token batches for the transformer and Deep Sets |
|
| 12 |
| `models.py` | `EdgeNetwork`, `FineTunedEdgeNetwork`, `ObjectTransformer`, `TransformerClassifier`, `DeepSets`, checkpoint loading |
|
| 13 |
-
| `jepa.py` | JEPA pretraining of the transformer or edge network encoder |
|
| 14 |
-
| `
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
| `train.py` | `Trainer`, early stopping, checkpoints |
|
| 16 |
| `cuda_graphs.py` | captured training steps |
|
| 17 |
| `distributed.py` | `torchrun` process group and collectives |
|
| 18 |
| `inference.py` | model outputs over a loader, prediction files |
|
| 19 |
| `export.py` | ONNX export |
|
| 20 |
-
| `cli.py` | the
|
| 21 |
|
| 22 |
## Data flow
|
| 23 |
|
|
@@ -108,7 +112,31 @@ The edge network's JEPA encodings collapse without the variance term
|
|
| 108 |
(`target_spread` 0.09 against about 1 with it), and none of the edge
|
| 109 |
network variants yet beats training from scratch.
|
| 110 |
|
| 111 |
-
###
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
|
| 113 |
Both classify a dataset's events given `data.train_events` labelled
|
| 114 |
training events (an equal number per class) as context, without training on
|
|
@@ -131,17 +159,66 @@ that dataset; `cli.in_context_forward` builds them for `evaluate` and
|
|
| 131 |
scale within `task.energy_jitter`. The GamGam processes are among the
|
| 132 |
Delphes samples; `task.prior_classes=[1,6,7,8,9,10,11]` leaves them out.
|
| 133 |
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 139 |
2. Meta-train `configs/delphes_pfn.yaml` for about an hour, on all 12
|
| 140 |
processes and with the GamGam processes left out, then GamGam in context
|
| 141 |
-
with the same `train_events`.
|
|
|
|
|
|
|
| 142 |
3. Supervised references with the same `train_events`: the edge network
|
| 143 |
from scratch and from `Pretrained_GNN`, the transformer from scratch and
|
| 144 |
-
from JEPA.
|
| 145 |
4. One table of validation AUC against labelled events here.
|
| 146 |
|
| 147 |
The loss is a weighted mean per class present in the batch, averaged over
|
|
|
|
| 8 |
| `data.py` | ROOT reading, Hugging Face download, derived branches, object features, `PackedEvents` and its cache |
|
| 9 |
| `loader.py` | `EventLoader`: deterministic shuffling, DDP sharding, collation workers |
|
| 10 |
| `graphs.py` | fully connected `TensorGraph` batches, edge features, CUDA-graph padding buffers |
|
| 11 |
+
| `sequences.py` | padded token batches for the transformer and Deep Sets, event augmentations |
|
| 12 |
| `models.py` | `EdgeNetwork`, `FineTunedEdgeNetwork`, `ObjectTransformer`, `TransformerClassifier`, `DeepSets`, checkpoint loading |
|
| 13 |
+
| `jepa.py` | JEPA pretraining of the transformer or edge network encoder (encoding or kinematic targets), event embeddings |
|
| 14 |
+
| `contrastive.py` | contrastive (SimCLR) pretraining of the same encoders |
|
| 15 |
+
| `probe.py` | k-NN and linear probes of frozen encoders |
|
| 16 |
+
| `pfn.py` | our prior-fitted network over events and the task prior of meta-training |
|
| 17 |
+
| `tabular.py` | event rows and encoder embeddings for TabPFN, TabICL and TabDPT; LoRA meta-training of TabPFN and TabICL |
|
| 18 |
+
| `tasks.py` | multiclass and binary losses and metrics, the pretraining and PFN losses |
|
| 19 |
| `train.py` | `Trainer`, early stopping, checkpoints |
|
| 20 |
| `cuda_graphs.py` | captured training steps |
|
| 21 |
| `distributed.py` | `torchrun` process group and collectives |
|
| 22 |
| `inference.py` | model outputs over a loader, prediction files |
|
| 23 |
| `export.py` | ONNX export |
|
| 24 |
+
| `cli.py` | the six commands |
|
| 25 |
|
| 26 |
## Data flow
|
| 27 |
|
|
|
|
| 112 |
(`target_spread` 0.09 against about 1 with it), and none of the edge
|
| 113 |
network variants yet beats training from scratch.
|
| 114 |
|
| 115 |
+
### Contrastive and masked-object pretraining, and probes
|
| 116 |
+
|
| 117 |
+
Two more label-free objectives pretrain the same encoders on the same
|
| 118 |
+
samples, and their encoders load the same way (`context_encoder`):
|
| 119 |
+
|
| 120 |
+
* `contrastive.py` (`configs/delphes_contrastive.yaml`): SimCLR, as JetCLR
|
| 121 |
+
and RS3L for jets. Two views of each event (random phi rotation, eta
|
| 122 |
+
flip, momentum scale within `model.scale_jitter`, each object but the
|
| 123 |
+
first dropped with `model.object_dropout`) are embedded and projected;
|
| 124 |
+
the NT-Xent loss at `model.temperature` contrasts them with the other
|
| 125 |
+
events of the batch. `embedding_spread` watches for collapse.
|
| 126 |
+
* `model.jepa_target: features` (`configs/delphes_masked.yaml`): masked
|
| 127 |
+
object modelling (Golling et al. 2024). JEPA's masking and predictor
|
| 128 |
+
predict the hidden objects' kinematics instead of their target
|
| 129 |
+
encodings, with no target encoder: the comparison with JEPA isolates
|
| 130 |
+
predicting in embedding space.
|
| 131 |
+
|
| 132 |
+
`gnn4colliders probe` compares encoders without fine-tuning (as jBOT):
|
| 133 |
+
k-NN (cosine, `model.probe_neighbours`) and linear (L-BFGS logistic
|
| 134 |
+
regression, `model.probe_l2`) classifiers of frozen event embeddings,
|
| 135 |
+
trained on `data.train_events` events. A randomly initialized encoder
|
| 136 |
+
(`model.pretrained=null`) is the reference; supervised checkpoints probe
|
| 137 |
+
too. Not yet run beyond fixtures.
|
| 138 |
+
|
| 139 |
+
### In-context models: TabPFN, TabICL, TabDPT and a collider PFN
|
| 140 |
|
| 141 |
Both classify a dataset's events given `data.train_events` labelled
|
| 142 |
training events (an equal number per class) as context, without training on
|
|
|
|
| 159 |
scale within `task.energy_jitter`. The GamGam processes are among the
|
| 160 |
Delphes samples; `task.prior_classes=[1,6,7,8,9,10,11]` leaves them out.
|
| 161 |
|
| 162 |
+
* `tabular.py` also runs TabICLv2 (Qu et al., ICML 2026; BSD-3, open
|
| 163 |
+
pretraining code) on the same rows, as a second tabular in-context model.
|
| 164 |
+
Rows can measure azimuths from the leading object's (`model.relative_phi`)
|
| 165 |
+
and end with the invariant mass and Delta R of every pair of the
|
| 166 |
+
`model.pair_objects` leading objects by pt (energies are `pt cosh(eta)`,
|
| 167 |
+
so objects are massless; a pair with MET gives a transverse-like mass).
|
| 168 |
+
Both are off by default, so the default rows are unchanged.
|
| 169 |
+
* `tabular.py` also runs TabDPT (Ma et al. 2024; Apache-2.0), trained on
|
| 170 |
+
real tables rather than a synthetic prior; with
|
| 171 |
+
`model.tabdpt_context_size` each event's context is its nearest context
|
| 172 |
+
events (faiss retrieval), beyond what fits in one pass. It computes in
|
| 173 |
+
float64 in places (on a Mac it runs on the CPU), and on macOS faiss and
|
| 174 |
+
torch crash together unless `OMP_NUM_THREADS=1`.
|
| 175 |
+
* Tabular models can see, instead of or beside the rows, the event
|
| 176 |
+
embeddings of a pretrained object encoder (`model.tabular_inputs:
|
| 177 |
+
embeddings` or `both`, the encoder of `model.backbone` and
|
| 178 |
+
`model.pretrained`): physics-aware embeddings under a generic tabular
|
| 179 |
+
prior. In meta-training the encoder trains with the LoRA updates unless
|
| 180 |
+
`model.freeze_backbone`.
|
| 181 |
+
* `tabular.py` also meta-trains the TabPFN-3.5 network itself on the
|
| 182 |
+
PFN's tasks (`model.name: tabpfn`, `task.name: pfn`), on the rows above
|
| 183 |
+
(built after the augmentation), starting from Prior Labs' weights. Calling
|
| 184 |
+
the network directly skips TabPFN's preprocessing and ensembling, which
|
| 185 |
+
on the GamGam fixtures cost nothing (validation AUC 0.852 against 0.843
|
| 186 |
+
with `predict_proba`, 61 events). By default the network is frozen and
|
| 187 |
+
the four attention projections and two MLP layers of each of its 24
|
| 188 |
+
in-context blocks get rank-8 LoRA updates (2.8M parameters, zero at the
|
| 189 |
+
start, so training starts from TabPFN as released); `model.lora_rank=0`
|
| 190 |
+
trains all weights. In context, the context's keys and values are cached
|
| 191 |
+
(the same logits as one pass over context and queries, to 2e-5). It
|
| 192 |
+
separates what Prior Labs' prior and what collider tasks contribute:
|
| 193 |
+
TabPFN as released, TabPFN meta-trained here, and our PFN. TabICL
|
| 194 |
+
(`configs/delphes_tabicl.yaml`; BSD-3, so meta-trained weights can be
|
| 195 |
+
shared) meta-trains the same way. Its attention reads fused projection
|
| 196 |
+
weights directly, so LoRA is a parametrization of the weights (12 blocks,
|
| 197 |
+
0.59M of 28M parameters at rank 8), for TabPFN too. TabICL's network takes
|
| 198 |
+
no NaN: rows are standardized by the context's statistics, absent values
|
| 199 |
+
at the mean, clipped at 8 sigma. On the fixtures this gives AUC 0.827
|
| 200 |
+
against 0.820 with its classifier, and the cached in-context forward
|
| 201 |
+
equals the training forward only in float32 (with mixed precision on the
|
| 202 |
+
CPU, AUC 0.59).
|
| 203 |
+
|
| 204 |
+
Status: implemented and tested on fixtures on the CPU and on MPS. First
|
| 205 |
+
result, TabPFN-3.5 as released (default rows, 4 estimators) on the 246,922
|
| 206 |
+
GamGam validation events on an M2 Max: AUC 80.26 with 100 labelled events
|
| 207 |
+
(3.1 min) and 86.93 with 1000 (3.9 min). Full-scale runs are for
|
| 208 |
+
Perlmutter. `scripts/label_efficiency.py` runs the sweeps below. Planned
|
| 209 |
+
study, in interactive allocations:
|
| 210 |
+
|
| 211 |
+
1. TabPFN-3.5, TabICLv2 and TabDPT on GamGam validation with
|
| 212 |
+
`train_events` 100, 1000, 10000 and 50000 (AUC and time), with and
|
| 213 |
+
without the physics row features, and with encoder embeddings.
|
| 214 |
2. Meta-train `configs/delphes_pfn.yaml` for about an hour, on all 12
|
| 215 |
processes and with the GamGam processes left out, then GamGam in context
|
| 216 |
+
with the same `train_events`. The same for TabPFN-3.5 and TabICL
|
| 217 |
+
(`configs/delphes_tabpfn.yaml`, `configs/delphes_tabicl.yaml`; LoRA, and
|
| 218 |
+
`model.lora_rank=0`).
|
| 219 |
3. Supervised references with the same `train_events`: the edge network
|
| 220 |
from scratch and from `Pretrained_GNN`, the transformer from scratch and
|
| 221 |
+
from JEPA, contrastive and masked-object pretraining (also probed).
|
| 222 |
4. One table of validation AUC against labelled events here.
|
| 223 |
|
| 224 |
The loss is a weighted mean per class present in the batch, averaged over
|
|
@@ -13,7 +13,10 @@ Python 3.14 and PyTorch 2.14 (CUDA 13), managed by [uv](https://docs.astral.sh/u
|
|
| 13 |
```bash
|
| 14 |
uv sync --dev # includes pytest, ruff and the ONNX packages
|
| 15 |
uv sync --dev --extra tabpfn # also TabPFN (configs/gamgam_tabpfn.yaml)
|
|
|
|
|
|
|
| 16 |
uv run pytest # about 2 minutes; GPU tests run when CUDA is available
|
|
|
|
| 17 |
```
|
| 18 |
|
| 19 |
## Commands
|
|
@@ -27,6 +30,8 @@ uv run gnn4colliders train configs/gamgam.yaml # -> run_dir/{best,last}.pt
|
|
| 27 |
uv run gnn4colliders evaluate configs/gamgam.yaml # -> run_dir/test_metrics.json
|
| 28 |
uv run gnn4colliders predict configs/gamgam.yaml # -> run_dir/test_predictions.npz
|
| 29 |
uv run gnn4colliders export configs/gamgam.yaml # -> run_dir/model.onnx
|
|
|
|
|
|
|
| 30 |
```
|
| 31 |
|
| 32 |
`train` writes, in `run_dir` (default `outputs/<name>`):
|
|
@@ -39,6 +44,11 @@ uv run gnn4colliders export configs/gamgam.yaml # -> run_dir/model.onnx
|
|
| 39 |
`train.resume=<run_dir>/last.pt`), and `epoch_NNNN.pt` every
|
| 40 |
`train.save_every` epochs.
|
| 41 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
`evaluate`, `predict` and `export` use `checkpoint` (default `best.pt`, else
|
| 43 |
`last.pt`) on `split` (default `test`); `output` overrides the output file.
|
| 44 |
Predictions hold the logits, scores, predicted class, label, weight, fold,
|
|
@@ -57,10 +67,16 @@ adds scores to a copy of a ROOT tree.
|
|
| 57 |
| `configs/delphes_pretraining.yaml` | 12-class pretraining on the Delphes samples |
|
| 58 |
| `configs/delphes_jepa.yaml` | JEPA pretraining of the transformer encoder on the same samples, without labels (same event cache) |
|
| 59 |
| `configs/delphes_jepa_edge.yaml` | the same for the edge network (fine-tune with `configs/gamgam_finetune.yaml model.pretrained=outputs/delphes_jepa_edge/last.pt`) |
|
| 60 |
-
| `configs/
|
|
|
|
|
|
|
| 61 |
| `configs/gamgam_tabpfn.yaml` | GamGam classified in context by TabPFN-3.5, no training (`evaluate` only) |
|
|
|
|
|
|
|
| 62 |
| `configs/delphes_pfn.yaml` | meta-training of our own PFN on tasks drawn from the Delphes samples (same event cache) |
|
| 63 |
| `configs/gamgam_pfn.yaml` | GamGam classified in context by that PFN, no training (`evaluate` only) |
|
|
|
|
|
|
|
| 64 |
|
| 65 |
Each file documents its keys; `src/gnn4colliders/config.py` lists every key
|
| 66 |
with its default. Common overrides:
|
|
@@ -110,11 +126,70 @@ uv run gnn4colliders evaluate configs/gamgam_pfn.yaml split=validation data.trai
|
|
| 110 |
uv run gnn4colliders train configs/gamgam.yaml data.train_events=1000 # supervised reference
|
| 111 |
```
|
| 112 |
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
|
| 119 |
### Fine-tuning
|
| 120 |
|
|
@@ -129,6 +204,10 @@ initializes the edge network or the transformer encoder.
|
|
| 129 |
|
| 130 |
## GPUs, Perlmutter and DDP
|
| 131 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
On a single GPU, `train.fast=true` is about 3x faster than float32 for the
|
| 133 |
GamGam model (see [design.md](design.md)). Set `OMP_NUM_THREADS` to the CPU
|
| 134 |
allocation and, on Perlmutter compute nodes, `UV_CACHE_DIR=$SCRATCH/.cache/uv`
|
|
|
|
| 13 |
```bash
|
| 14 |
uv sync --dev # includes pytest, ruff and the ONNX packages
|
| 15 |
uv sync --dev --extra tabpfn # also TabPFN (configs/gamgam_tabpfn.yaml)
|
| 16 |
+
uv sync --dev --extra tabicl # also TabICL (configs/gamgam_tabicl.yaml)
|
| 17 |
+
uv sync --dev --extra tabdpt # also TabDPT (configs/gamgam_tabdpt.yaml)
|
| 18 |
uv run pytest # about 2 minutes; GPU tests run when CUDA is available
|
| 19 |
+
uv run pytest --device mps # the fixture runs on an Apple GPU
|
| 20 |
```
|
| 21 |
|
| 22 |
## Commands
|
|
|
|
| 30 |
uv run gnn4colliders evaluate configs/gamgam.yaml # -> run_dir/test_metrics.json
|
| 31 |
uv run gnn4colliders predict configs/gamgam.yaml # -> run_dir/test_predictions.npz
|
| 32 |
uv run gnn4colliders export configs/gamgam.yaml # -> run_dir/model.onnx
|
| 33 |
+
uv run gnn4colliders probe configs/gamgam_jepa_finetune.yaml \
|
| 34 |
+
model.pretrained=outputs/delphes_jepa/last.pt data.train_events=1000 # -> run_dir/test_probe.json
|
| 35 |
```
|
| 36 |
|
| 37 |
`train` writes, in `run_dir` (default `outputs/<name>`):
|
|
|
|
| 44 |
`train.resume=<run_dir>/last.pt`), and `epoch_NNNN.pt` every
|
| 45 |
`train.save_every` epochs.
|
| 46 |
|
| 47 |
+
`probe` fits k-NN and linear classifiers to the frozen event embeddings of
|
| 48 |
+
the encoder of `model.pretrained` (JEPA, contrastive or supervised; random
|
| 49 |
+
without) on `data.train_events` training events and reports their metrics
|
| 50 |
+
on `split`, to compare pretraining without fine-tuning.
|
| 51 |
+
|
| 52 |
`evaluate`, `predict` and `export` use `checkpoint` (default `best.pt`, else
|
| 53 |
`last.pt`) on `split` (default `test`); `output` overrides the output file.
|
| 54 |
Predictions hold the logits, scores, predicted class, label, weight, fold,
|
|
|
|
| 67 |
| `configs/delphes_pretraining.yaml` | 12-class pretraining on the Delphes samples |
|
| 68 |
| `configs/delphes_jepa.yaml` | JEPA pretraining of the transformer encoder on the same samples, without labels (same event cache) |
|
| 69 |
| `configs/delphes_jepa_edge.yaml` | the same for the edge network (fine-tune with `configs/gamgam_finetune.yaml model.pretrained=outputs/delphes_jepa_edge/last.pt`) |
|
| 70 |
+
| `configs/delphes_contrastive.yaml` | contrastive (SimCLR) pretraining of the transformer encoder on the same samples (`model.backbone=edge_network` for the edge network) |
|
| 71 |
+
| `configs/delphes_masked.yaml` | masked object modelling: JEPA predicting hidden objects' kinematics |
|
| 72 |
+
| `configs/gamgam_jepa_finetune.yaml` | GamGam with the JEPA-pretrained encoder (`model.pretrained=null`: from scratch); also takes contrastive and masked-object encoders, and `probe` |
|
| 73 |
| `configs/gamgam_tabpfn.yaml` | GamGam classified in context by TabPFN-3.5, no training (`evaluate` only) |
|
| 74 |
+
| `configs/gamgam_tabicl.yaml` | the same by TabICLv2 (BSD-3 weights, downloaded from Hugging Face on first use) |
|
| 75 |
+
| `configs/gamgam_tabdpt.yaml` | the same by TabDPT (Apache-2.0 weights; `model.tabdpt_context_size` retrieves each event's context) |
|
| 76 |
| `configs/delphes_pfn.yaml` | meta-training of our own PFN on tasks drawn from the Delphes samples (same event cache) |
|
| 77 |
| `configs/gamgam_pfn.yaml` | GamGam classified in context by that PFN, no training (`evaluate` only) |
|
| 78 |
+
| `configs/delphes_tabpfn.yaml` | meta-training of TabPFN-3.5 (LoRA by default) on the same tasks; evaluate with `configs/gamgam_tabpfn.yaml checkpoint=outputs/delphes_tabpfn/best.pt` |
|
| 79 |
+
| `configs/delphes_tabicl.yaml` | the same for TabICL; evaluate with `configs/gamgam_tabicl.yaml checkpoint=outputs/delphes_tabicl/best.pt` |
|
| 80 |
|
| 81 |
Each file documents its keys; `src/gnn4colliders/config.py` lists every key
|
| 82 |
with its default. Common overrides:
|
|
|
|
| 126 |
uv run gnn4colliders train configs/gamgam.yaml data.train_events=1000 # supervised reference
|
| 127 |
```
|
| 128 |
|
| 129 |
+
`scripts/label_efficiency.py` runs such a sweep over several `N` and
|
| 130 |
+
configs (restartable) and prints validation AUC and seconds per run; the
|
| 131 |
+
metrics JSON records `seconds` (setup, inference) and, on CUDA,
|
| 132 |
+
`peak_memory_gib`:
|
| 133 |
+
|
| 134 |
+
```bash
|
| 135 |
+
uv run python scripts/label_efficiency.py configs/gamgam_tabpfn.yaml configs/gamgam.yaml \
|
| 136 |
+
--events 100 1000 10000 50000
|
| 137 |
+
```
|
| 138 |
+
|
| 139 |
+
TabPFN-2.5 and later weights are non-commercial (research and internal
|
| 140 |
+
benchmarking are allowed). tabpfn downloads them after a one-time login at
|
| 141 |
+
https://ux.priorlabs.ai (token in `~/.cache/tabpfn/auth_token` or
|
| 142 |
+
`TABPFN_TOKEN`), but a checkpoint already in its cache directory loads
|
| 143 |
+
without the login, so on compute nodes download it from Hugging Face:
|
| 144 |
+
|
| 145 |
+
```bash
|
| 146 |
+
export TABPFN_MODEL_CACHE_DIR=$SCRATCH/.cache/tabpfn # default ~/Library/Caches/tabpfn on macOS
|
| 147 |
+
uvx --from huggingface_hub hf download Prior-Labs/tabpfn_3_5 config.json \
|
| 148 |
+
tabpfn-v3.5-20260909.safetensors tabpfn-v3.5-20260909_multiclass.safetensors \
|
| 149 |
+
--local-dir $TABPFN_MODEL_CACHE_DIR
|
| 150 |
+
```
|
| 151 |
+
|
| 152 |
+
`model.tabpfn_checkpoint=tabpfn-v3.5-20260909_multiclass.safetensors`
|
| 153 |
+
selects the multiclass checkpoint. `model.tabpfn_version=v2` needs no login.
|
| 154 |
+
|
| 155 |
+
TabDPT retrieval (`model.tabdpt_context_size`) on macOS needs
|
| 156 |
+
`OMP_NUM_THREADS=1`: faiss and torch load separate OpenMP runtimes.
|
| 157 |
+
|
| 158 |
+
Tabular models can use a pretrained encoder's event embeddings instead of
|
| 159 |
+
or beside the rows: `model.tabular_inputs=embeddings` (or `both`) with
|
| 160 |
+
`model.pretrained=outputs/delphes_jepa/last.pt` and the encoder's sizes
|
| 161 |
+
(those of `configs/delphes_jepa.yaml`: `model.hid_size=128 model.n_layers=4
|
| 162 |
+
model.n_heads=8 model.ffn_size=512 model.pair_bias=true`).
|
| 163 |
+
|
| 164 |
+
TabPFN and TabICL classify the same table rows (`model.slots` objects of
|
| 165 |
+
each type). Two options add physics they cannot learn from a few events:
|
| 166 |
+
`model.relative_phi=true` measures every azimuth from the leading object's,
|
| 167 |
+
and `model.pair_objects=4` appends the invariant mass and Delta R of every
|
| 168 |
+
pair of the four leading objects (12 columns). Compare with and without,
|
| 169 |
+
one run name per variant:
|
| 170 |
+
|
| 171 |
+
```bash
|
| 172 |
+
uv run python scripts/label_efficiency.py configs/gamgam_tabicl.yaml --events 100 1000 \
|
| 173 |
+
--override name=gamgam_tabicl_physics model.relative_phi=true model.pair_objects=4
|
| 174 |
+
```
|
| 175 |
+
|
| 176 |
+
TabPFN-3.5 can also be meta-trained on the Delphes tasks, like our PFN.
|
| 177 |
+
By default only LoRA updates of its in-context blocks train
|
| 178 |
+
(`model.lora_rank`, 8; `model.lora_rank=0` trains all 219M weights), and
|
| 179 |
+
checkpoints hold only those (33 MB with the optimizer state, against
|
| 180 |
+
2.6 GB for all weights). With a `checkpoint`, TabPFN is
|
| 181 |
+
evaluated with its own forward pass, not the classifier:
|
| 182 |
+
|
| 183 |
+
```bash
|
| 184 |
+
uv run gnn4colliders train configs/delphes_tabpfn.yaml
|
| 185 |
+
uv run gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation \
|
| 186 |
+
data.train_events=1000 checkpoint=outputs/delphes_tabpfn/best.pt
|
| 187 |
+
```
|
| 188 |
+
|
| 189 |
+
Activations are recomputed in the backward pass. On a 32 GB Apple M2 Max
|
| 190 |
+
one task of 2048 context events and 256 queries takes about 4.5 s and
|
| 191 |
+
16 GiB per step with all weights trained, so the Mac is for development;
|
| 192 |
+
meta-train on a GPU node.
|
| 193 |
|
| 194 |
### Fine-tuning
|
| 195 |
|
|
|
|
| 204 |
|
| 205 |
## GPUs, Perlmutter and DDP
|
| 206 |
|
| 207 |
+
`train.device: auto` uses CUDA, else an Apple GPU (MPS), else the CPU. MPS
|
| 208 |
+
is for development on fixtures and small runs (no float64: metrics are
|
| 209 |
+
computed on the CPU; no DDP, CUDA graphs or TF32).
|
| 210 |
+
|
| 211 |
On a single GPU, `train.fast=true` is about 3x faster than float32 for the
|
| 212 |
GamGam model (see [design.md](design.md)). Set `OMP_NUM_THREADS` to the CPU
|
| 213 |
allocation and, on Perlmutter compute nodes, `UV_CACHE_DIR=$SCRATCH/.cache/uv`
|
|
@@ -25,6 +25,12 @@ onnx = [
|
|
| 25 |
tabpfn = [
|
| 26 |
"tabpfn>=9.2,<10",
|
| 27 |
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
[project.scripts]
|
| 30 |
gnn4colliders = "gnn4colliders.cli:main"
|
|
|
|
| 25 |
tabpfn = [
|
| 26 |
"tabpfn>=9.2,<10",
|
| 27 |
]
|
| 28 |
+
tabicl = [
|
| 29 |
+
"tabicl>=2.2,<3",
|
| 30 |
+
]
|
| 31 |
+
tabdpt = [
|
| 32 |
+
"tabdpt>=1.3.1,<2",
|
| 33 |
+
]
|
| 34 |
|
| 35 |
[project.scripts]
|
| 36 |
gnn4colliders = "gnn4colliders.cli:main"
|
|
@@ -0,0 +1,83 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Validation AUC against the number of labelled training events.
|
| 2 |
+
|
| 3 |
+
For every config and every ``--events`` count N, runs the model with
|
| 4 |
+
``data.train_events=N`` in ``<out>/<name>/n<N>``: supervised models are
|
| 5 |
+
trained, then evaluated; in-context models (TabPFN, TabICL, PFN) are only evaluated,
|
| 6 |
+
with the N events as context. Runs whose metrics exist are skipped, so an
|
| 7 |
+
interrupted sweep restarts where it stopped. Prints a table of AUC (%) and
|
| 8 |
+
evaluation seconds. The event cache must exist (``gnn4colliders prepare``).
|
| 9 |
+
|
| 10 |
+
uv run python scripts/label_efficiency.py configs/gamgam_tabpfn.yaml \\
|
| 11 |
+
configs/gamgam.yaml --events 100 1000 10000 50000
|
| 12 |
+
|
| 13 |
+
Meta-trained in-context models (the PFN; TabPFN or TabICL after
|
| 14 |
+
meta-training) need ``--override checkpoint=<path>``. Overrides apply to
|
| 15 |
+
every config; ``name=...`` keeps variants apart (run one config at a time
|
| 16 |
+
then, as names must differ):
|
| 17 |
+
|
| 18 |
+
uv run python scripts/label_efficiency.py configs/gamgam_tabpfn.yaml \\
|
| 19 |
+
--override name=gamgam_tabpfn_multiclass \\
|
| 20 |
+
model.tabpfn_checkpoint=tabpfn-v3.5-20260909_multiclass.safetensors
|
| 21 |
+
"""
|
| 22 |
+
|
| 23 |
+
from __future__ import annotations
|
| 24 |
+
|
| 25 |
+
import argparse
|
| 26 |
+
import json
|
| 27 |
+
from pathlib import Path
|
| 28 |
+
|
| 29 |
+
from gnn4colliders.cli import main as gnn4colliders
|
| 30 |
+
from gnn4colliders.config import load_config
|
| 31 |
+
from gnn4colliders.models import IN_CONTEXT_MODELS
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def _run(config: Path, events: int, out: Path, overrides: list[str]) -> dict:
|
| 35 |
+
name = load_config(config, overrides).name
|
| 36 |
+
run_dir = out / name / f"n{events}"
|
| 37 |
+
metrics = run_dir / "validation_metrics.json"
|
| 38 |
+
if not metrics.exists():
|
| 39 |
+
args = [*overrides, f"data.train_events={events}", f"run_dir={run_dir}"]
|
| 40 |
+
resolved = load_config(config, args)
|
| 41 |
+
if resolved.model.name == "pfn" and resolved.checkpoint is None:
|
| 42 |
+
# The run directory is new: the meta-trained weights are elsewhere.
|
| 43 |
+
raise SystemExit("the PFN needs --override checkpoint=<meta-trained>.pt")
|
| 44 |
+
if resolved.model.name not in IN_CONTEXT_MODELS:
|
| 45 |
+
if gnn4colliders(["train", str(config), *args]) != 0:
|
| 46 |
+
raise SystemExit(f"training {name} with {events} events failed")
|
| 47 |
+
if gnn4colliders(["evaluate", str(config), *args, "split=validation"]) != 0:
|
| 48 |
+
raise SystemExit(f"evaluating {name} with {events} events failed")
|
| 49 |
+
return {"name": name, "events": events, **json.loads(metrics.read_text())}
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def main() -> None:
|
| 53 |
+
parser = argparse.ArgumentParser(
|
| 54 |
+
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
|
| 55 |
+
)
|
| 56 |
+
parser.add_argument("configs", nargs="+", type=Path)
|
| 57 |
+
parser.add_argument("--events", nargs="+", type=int, required=True)
|
| 58 |
+
parser.add_argument("--override", nargs="*", default=[], help="key=value")
|
| 59 |
+
parser.add_argument("--out", type=Path, default=Path("outputs/label_efficiency"))
|
| 60 |
+
args = parser.parse_args()
|
| 61 |
+
names = [load_config(config, args.override).name for config in args.configs]
|
| 62 |
+
if len(set(names)) < len(names):
|
| 63 |
+
raise SystemExit(f"configs share a run name ({names}); override name per run")
|
| 64 |
+
runs = [
|
| 65 |
+
_run(config, events, args.out, args.override)
|
| 66 |
+
for config in args.configs
|
| 67 |
+
for events in args.events
|
| 68 |
+
]
|
| 69 |
+
print("| model | " + " | ".join(f"N = {n}" for n in args.events) + " |")
|
| 70 |
+
print("|---" * (len(args.events) + 1) + "|")
|
| 71 |
+
for name in dict.fromkeys(run["name"] for run in runs):
|
| 72 |
+
cells = []
|
| 73 |
+
for events in args.events:
|
| 74 |
+
run = next(r for r in runs if r["name"] == name and r["events"] == events)
|
| 75 |
+
auc = 100 * run["metrics"]["auc"]
|
| 76 |
+
seconds = run.get("seconds")
|
| 77 |
+
total = f", {sum(seconds.values()):.0f} s" if seconds else ""
|
| 78 |
+
cells.append(f"{auc:.2f}{total}")
|
| 79 |
+
print(f"| {name} | " + " | ".join(cells) + " |")
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
if __name__ == "__main__":
|
| 83 |
+
main()
|
|
@@ -8,10 +8,16 @@ Commands:
|
|
| 8 |
* ``evaluate``: metrics of ``checkpoint`` on ``split``, written as JSON.
|
| 9 |
* ``predict``: per-event scores of ``checkpoint`` on ``split``, as ``.npz``.
|
| 10 |
* ``export``: ONNX model of ``checkpoint``.
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
-
The in-context models (``model.name: pfn``
|
|
|
|
| 13 |
without training on the task: their context is ``data.train_events``
|
| 14 |
-
labelled training events. The PFN
|
|
|
|
|
|
|
| 15 |
"""
|
| 16 |
|
| 17 |
from __future__ import annotations
|
|
@@ -29,7 +35,14 @@ import torch
|
|
| 29 |
|
| 30 |
from .config import Config, load_config, save_config
|
| 31 |
from .data import PackedEvents, load_events, prepare
|
| 32 |
-
from .distributed import
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
from .graphs import collate_graph_batch
|
| 34 |
from .inference import collect_outputs, write_predictions
|
| 35 |
from .loader import EventLoader
|
|
@@ -260,6 +273,20 @@ def _checkpoint(config: Config) -> Path:
|
|
| 260 |
return best if best.exists() else config.path / "last.pt"
|
| 261 |
|
| 262 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 263 |
def in_context_forward(
|
| 264 |
config: Config, events: PackedEvents, device: torch.device
|
| 265 |
) -> tuple[Any, str]:
|
|
@@ -272,25 +299,32 @@ def in_context_forward(
|
|
| 272 |
raise ValueError("in-context models classify with task.name: multiclass")
|
| 273 |
train = events.split_indices(data.train_folds)
|
| 274 |
context = events.balanced_subset(train, data.train_events, data.seed)
|
| 275 |
-
if model_config.name
|
| 276 |
-
from .
|
| 277 |
|
| 278 |
-
forward =
|
| 279 |
model_config,
|
| 280 |
events,
|
| 281 |
context,
|
| 282 |
num_classes=model_config.out_size,
|
| 283 |
num_node_types=len(data.object_types),
|
|
|
|
| 284 |
device=device,
|
| 285 |
seed=config.train.seed,
|
| 286 |
)
|
| 287 |
-
return forward,
|
| 288 |
from .pfn import in_context_forward as pfn_forward
|
|
|
|
| 289 |
|
| 290 |
checkpoint = _checkpoint(config)
|
| 291 |
model = make_model(config, events, load_pretrained=False)
|
| 292 |
load_weights(model, load_checkpoint(checkpoint))
|
| 293 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 294 |
model.to(device),
|
| 295 |
events,
|
| 296 |
context,
|
|
@@ -308,6 +342,9 @@ def run_inference(config: Config, *, predict: bool) -> Path:
|
|
| 308 |
torch.backends.cuda.matmul.allow_tf32 = config.train.allow_tf32
|
| 309 |
events = load_events(config.data.cache)
|
| 310 |
loader = make_loaders(config, events, (config.split,), context)[config.split]
|
|
|
|
|
|
|
|
|
|
| 311 |
if config.model.name in IN_CONTEXT_MODELS:
|
| 312 |
forward, checkpoint = in_context_forward(config, events, context.device)
|
| 313 |
else:
|
|
@@ -315,9 +352,20 @@ def run_inference(config: Config, *, predict: bool) -> Path:
|
|
| 315 |
forward = make_model(config, events, load_pretrained=False)
|
| 316 |
load_weights(forward, load_checkpoint(checkpoint))
|
| 317 |
forward = forward.to(context.device).eval()
|
|
|
|
|
|
|
| 318 |
outputs = collect_outputs(
|
| 319 |
forward, loader, device=context.device, context=context
|
| 320 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 321 |
task = make_task(config)
|
| 322 |
suffix = "predictions.npz" if predict else "metrics.json"
|
| 323 |
target = Path(config.output or config.path / f"{config.split}_{suffix}")
|
|
@@ -331,6 +379,8 @@ def run_inference(config: Config, *, predict: bool) -> Path:
|
|
| 331 |
"split": config.split,
|
| 332 |
"num_samples": int(outputs.labels.shape[0]),
|
| 333 |
"metrics": metrics,
|
|
|
|
|
|
|
| 334 |
}
|
| 335 |
target.parent.mkdir(parents=True, exist_ok=True)
|
| 336 |
target.write_text(json.dumps(summary, indent=2) + "\n")
|
|
@@ -359,12 +409,61 @@ def run_export(config: Config) -> Path:
|
|
| 359 |
)
|
| 360 |
|
| 361 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 362 |
COMMANDS = {
|
| 363 |
"prepare": run_prepare,
|
| 364 |
"train": run_train,
|
| 365 |
"evaluate": lambda config: run_inference(config, predict=False),
|
| 366 |
"predict": lambda config: run_inference(config, predict=True),
|
| 367 |
"export": run_export,
|
|
|
|
| 368 |
}
|
| 369 |
|
| 370 |
|
|
|
|
| 8 |
* ``evaluate``: metrics of ``checkpoint`` on ``split``, written as JSON.
|
| 9 |
* ``predict``: per-event scores of ``checkpoint`` on ``split``, as ``.npz``.
|
| 10 |
* ``export``: ONNX model of ``checkpoint``.
|
| 11 |
+
* ``probe``: k-NN and linear classifiers of the frozen encodings of the
|
| 12 |
+
encoder of ``model.pretrained`` (random without), trained on
|
| 13 |
+
``data.train_events`` events; their metrics on ``split``, as JSON.
|
| 14 |
|
| 15 |
+
The in-context models (``model.name: pfn``, ``tabpfn``, ``tabicl`` or
|
| 16 |
+
``tabdpt``) are evaluated
|
| 17 |
without training on the task: their context is ``data.train_events``
|
| 18 |
+
labelled training events. The PFN, TabPFN and TabICL are meta-trained with
|
| 19 |
+
``task.name: pfn``; TabPFN and TabICL without a ``checkpoint`` are the
|
| 20 |
+
classifiers as released, and TabDPT is only used as released.
|
| 21 |
"""
|
| 22 |
|
| 23 |
from __future__ import annotations
|
|
|
|
| 35 |
|
| 36 |
from .config import Config, load_config, save_config
|
| 37 |
from .data import PackedEvents, load_events, prepare
|
| 38 |
+
from .distributed import (
|
| 39 |
+
DistributedContext,
|
| 40 |
+
barrier,
|
| 41 |
+
finalize,
|
| 42 |
+
initialize,
|
| 43 |
+
resolve_device,
|
| 44 |
+
synchronize,
|
| 45 |
+
)
|
| 46 |
from .graphs import collate_graph_batch
|
| 47 |
from .inference import collect_outputs, write_predictions
|
| 48 |
from .loader import EventLoader
|
|
|
|
| 273 |
return best if best.exists() else config.path / "last.pt"
|
| 274 |
|
| 275 |
|
| 276 |
+
#: The tabular in-context models used as released (without a meta-trained
|
| 277 |
+
#: ``checkpoint``), and how the metrics name them.
|
| 278 |
+
RELEASED_MODELS = {
|
| 279 |
+
"tabpfn": lambda model: (
|
| 280 |
+
f"TabPFN {model.tabpfn_version} "
|
| 281 |
+
f"({model.tabpfn_checkpoint or 'default checkpoint'})"
|
| 282 |
+
),
|
| 283 |
+
"tabicl": lambda model: f"TabICL ({model.tabicl_checkpoint})",
|
| 284 |
+
"tabdpt": lambda model: (
|
| 285 |
+
f"TabDPT (context {model.tabdpt_context_size or 'all'} per query)"
|
| 286 |
+
),
|
| 287 |
+
}
|
| 288 |
+
|
| 289 |
+
|
| 290 |
def in_context_forward(
|
| 291 |
config: Config, events: PackedEvents, device: torch.device
|
| 292 |
) -> tuple[Any, str]:
|
|
|
|
| 299 |
raise ValueError("in-context models classify with task.name: multiclass")
|
| 300 |
train = events.split_indices(data.train_folds)
|
| 301 |
context = events.balanced_subset(train, data.train_events, data.seed)
|
| 302 |
+
if model_config.name in RELEASED_MODELS and config.checkpoint is None:
|
| 303 |
+
from . import tabular
|
| 304 |
|
| 305 |
+
forward = getattr(tabular, f"{model_config.name}_forward")(
|
| 306 |
model_config,
|
| 307 |
events,
|
| 308 |
context,
|
| 309 |
num_classes=model_config.out_size,
|
| 310 |
num_node_types=len(data.object_types),
|
| 311 |
+
batch_size=data.batch_size,
|
| 312 |
device=device,
|
| 313 |
seed=config.train.seed,
|
| 314 |
)
|
| 315 |
+
return forward, RELEASED_MODELS[model_config.name](model_config)
|
| 316 |
from .pfn import in_context_forward as pfn_forward
|
| 317 |
+
from .tabular import meta_tabular_forward
|
| 318 |
|
| 319 |
checkpoint = _checkpoint(config)
|
| 320 |
model = make_model(config, events, load_pretrained=False)
|
| 321 |
load_weights(model, load_checkpoint(checkpoint))
|
| 322 |
+
build = (
|
| 323 |
+
meta_tabular_forward
|
| 324 |
+
if model_config.name in ("tabpfn", "tabicl")
|
| 325 |
+
else pfn_forward
|
| 326 |
+
)
|
| 327 |
+
forward = build(
|
| 328 |
model.to(device),
|
| 329 |
events,
|
| 330 |
context,
|
|
|
|
| 342 |
torch.backends.cuda.matmul.allow_tf32 = config.train.allow_tf32
|
| 343 |
events = load_events(config.data.cache)
|
| 344 |
loader = make_loaders(config, events, (config.split,), context)[config.split]
|
| 345 |
+
if context.device.type == "cuda":
|
| 346 |
+
torch.cuda.reset_peak_memory_stats(context.device)
|
| 347 |
+
start = time.perf_counter()
|
| 348 |
if config.model.name in IN_CONTEXT_MODELS:
|
| 349 |
forward, checkpoint = in_context_forward(config, events, context.device)
|
| 350 |
else:
|
|
|
|
| 352 |
forward = make_model(config, events, load_pretrained=False)
|
| 353 |
load_weights(forward, load_checkpoint(checkpoint))
|
| 354 |
forward = forward.to(context.device).eval()
|
| 355 |
+
synchronize(context.device)
|
| 356 |
+
loaded = time.perf_counter()
|
| 357 |
outputs = collect_outputs(
|
| 358 |
forward, loader, device=context.device, context=context
|
| 359 |
)
|
| 360 |
+
synchronize(context.device)
|
| 361 |
+
# Setup: loading the checkpoint, or the in-context model's pass over
|
| 362 |
+
# the context; inference: this rank's events.
|
| 363 |
+
seconds = {"setup": loaded - start, "inference": time.perf_counter() - loaded}
|
| 364 |
+
peak_memory = (
|
| 365 |
+
torch.cuda.max_memory_allocated(context.device) / 2**30
|
| 366 |
+
if context.device.type == "cuda"
|
| 367 |
+
else None
|
| 368 |
+
)
|
| 369 |
task = make_task(config)
|
| 370 |
suffix = "predictions.npz" if predict else "metrics.json"
|
| 371 |
target = Path(config.output or config.path / f"{config.split}_{suffix}")
|
|
|
|
| 379 |
"split": config.split,
|
| 380 |
"num_samples": int(outputs.labels.shape[0]),
|
| 381 |
"metrics": metrics,
|
| 382 |
+
"seconds": seconds,
|
| 383 |
+
"peak_memory_gib": peak_memory,
|
| 384 |
}
|
| 385 |
target.parent.mkdir(parents=True, exist_ok=True)
|
| 386 |
target.write_text(json.dumps(summary, indent=2) + "\n")
|
|
|
|
| 409 |
)
|
| 410 |
|
| 411 |
|
| 412 |
+
def run_probe(config: Config) -> Path:
|
| 413 |
+
from .probe import probe
|
| 414 |
+
|
| 415 |
+
if int(os.environ.get("WORLD_SIZE", "1")) > 1:
|
| 416 |
+
raise ValueError("probe runs in one process")
|
| 417 |
+
data = config.data
|
| 418 |
+
if data.train_events is None:
|
| 419 |
+
raise ValueError("probe needs data.train_events (the probes' training events)")
|
| 420 |
+
events = load_events(data.cache)
|
| 421 |
+
folds = {
|
| 422 |
+
"train": data.train_folds,
|
| 423 |
+
"validation": data.validation_folds,
|
| 424 |
+
"test": data.test_folds,
|
| 425 |
+
}
|
| 426 |
+
train = events.balanced_subset(
|
| 427 |
+
events.split_indices(data.train_folds), data.train_events, data.seed
|
| 428 |
+
)
|
| 429 |
+
queries = events.split_indices(folds[config.split])
|
| 430 |
+
# The probes are not scored on the events they are fitted to.
|
| 431 |
+
queries = queries[~torch.isin(queries, train)]
|
| 432 |
+
device = resolve_device(config.train.device)
|
| 433 |
+
start = time.perf_counter()
|
| 434 |
+
# All the dataset's classes, whatever the config's task (a pretraining
|
| 435 |
+
# config probes too).
|
| 436 |
+
num_classes = int(events.labels.max()) + 1
|
| 437 |
+
logits = probe(
|
| 438 |
+
config, events, train, queries, num_classes=num_classes, device=device
|
| 439 |
+
)
|
| 440 |
+
task = build_task("multiclass")
|
| 441 |
+
labels, weight = events.labels[queries], events.weight[queries]
|
| 442 |
+
summary = {
|
| 443 |
+
"encoder": config.model.pretrained,
|
| 444 |
+
"split": config.split,
|
| 445 |
+
"train_events": int(len(train)),
|
| 446 |
+
"num_samples": int(len(queries)),
|
| 447 |
+
**{
|
| 448 |
+
name: task.metrics(values.cpu(), labels, weight)
|
| 449 |
+
for name, values in logits.items()
|
| 450 |
+
},
|
| 451 |
+
"seconds": time.perf_counter() - start,
|
| 452 |
+
}
|
| 453 |
+
target = Path(config.output or config.path / f"{config.split}_probe.json")
|
| 454 |
+
target.parent.mkdir(parents=True, exist_ok=True)
|
| 455 |
+
target.write_text(json.dumps(summary, indent=2) + "\n")
|
| 456 |
+
print(json.dumps({name: summary[name] for name in logits}, indent=2))
|
| 457 |
+
return target
|
| 458 |
+
|
| 459 |
+
|
| 460 |
COMMANDS = {
|
| 461 |
"prepare": run_prepare,
|
| 462 |
"train": run_train,
|
| 463 |
"evaluate": lambda config: run_inference(config, predict=False),
|
| 464 |
"predict": lambda config: run_inference(config, predict=True),
|
| 465 |
"export": run_export,
|
| 466 |
+
"probe": run_probe,
|
| 467 |
}
|
| 468 |
|
| 469 |
|
|
@@ -65,8 +65,9 @@ class DataConfig:
|
|
| 65 |
|
| 66 |
@dataclass
|
| 67 |
class ModelConfig:
|
| 68 |
-
#: ``edge_network``, ``transformer``, ``deep_sets``, ``jepa``
|
| 69 |
-
#: (pretraining), or the in-context models ``pfn``
|
|
|
|
| 70 |
name: str = "edge_network"
|
| 71 |
hid_size: int = 64
|
| 72 |
out_size: int = 1
|
|
@@ -83,10 +84,14 @@ class ModelConfig:
|
|
| 83 |
ffn_size: int | None = None
|
| 84 |
#: Attention bias from object-pair features (Particle Transformer).
|
| 85 |
pair_bias: bool = False
|
| 86 |
-
# JEPA
|
| 87 |
-
#: The encoder
|
| 88 |
-
#:
|
| 89 |
backbone: str = "transformer"
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
predictor_layers: int = 2
|
| 91 |
#: Fraction of each event's objects to predict (at least one, and at
|
| 92 |
#: least one left visible).
|
|
@@ -96,6 +101,14 @@ class ModelConfig:
|
|
| 96 |
#: Weight of a VICReg variance term on the context encodings (and the
|
| 97 |
#: edge network's global state) that counters collapse.
|
| 98 |
variance_weight: float = 0.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
# PFN only (the encoder is the transformer above).
|
| 100 |
#: Classes the PFN can tell apart in one task.
|
| 101 |
max_classes: int = 10
|
|
@@ -103,16 +116,41 @@ class ModelConfig:
|
|
| 103 |
pfn_layers: int = 6
|
| 104 |
# TabPFN only.
|
| 105 |
tabpfn_version: str = "v3.5"
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
tabpfn_estimators: int = 4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 107 |
#: Objects of each type (in ``data.object_types`` order) in a table row.
|
| 108 |
slots: list[int] = field(default_factory=lambda: [6, 2, 2, 3, 1])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 109 |
|
| 110 |
|
| 111 |
@dataclass
|
| 112 |
class TaskConfig:
|
| 113 |
#: ``multiclass`` (``model.out_size`` classes), ``binary`` (one logit)
|
| 114 |
-
#: ``jepa``
|
| 115 |
-
#:
|
|
|
|
| 116 |
name: str = "multiclass"
|
| 117 |
#: Use ``|weight|`` in the loss (signed NLO weights).
|
| 118 |
absolute_weights: bool = False
|
|
@@ -134,7 +172,7 @@ class TaskConfig:
|
|
| 134 |
|
| 135 |
@dataclass
|
| 136 |
class TrainConfig:
|
| 137 |
-
#: ``auto`` (CUDA
|
| 138 |
device: str = "auto"
|
| 139 |
seed: int = 42
|
| 140 |
max_epochs: int = 100
|
|
@@ -186,8 +224,19 @@ class Config:
|
|
| 186 |
self.run_dir = self.run_dir or f"outputs/{self.name}"
|
| 187 |
if self.task.name == "binary" and self.model.out_size != 1:
|
| 188 |
raise ValueError("binary classification needs model.out_size: 1")
|
| 189 |
-
if self.task.name == "pfn" and self.model.name
|
| 190 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 191 |
if self.split not in ("train", "validation", "test"):
|
| 192 |
raise ValueError("split must be train, validation or test")
|
| 193 |
|
|
|
|
| 65 |
|
| 66 |
@dataclass
|
| 67 |
class ModelConfig:
|
| 68 |
+
#: ``edge_network``, ``transformer``, ``deep_sets``, ``jepa`` or
|
| 69 |
+
#: ``contrastive`` (pretraining), or the in-context models ``pfn``,
|
| 70 |
+
#: ``tabpfn``, ``tabicl`` and ``tabdpt``.
|
| 71 |
name: str = "edge_network"
|
| 72 |
hid_size: int = 64
|
| 73 |
out_size: int = 1
|
|
|
|
| 84 |
ffn_size: int | None = None
|
| 85 |
#: Attention bias from object-pair features (Particle Transformer).
|
| 86 |
pair_bias: bool = False
|
| 87 |
+
# JEPA and contrastive pretraining, probes and tabular embeddings.
|
| 88 |
+
#: The encoder: ``transformer`` or ``edge_network`` (the sizes above
|
| 89 |
+
#: apply to it; JEPA's predictor is a transformer).
|
| 90 |
backbone: str = "transformer"
|
| 91 |
+
# JEPA only.
|
| 92 |
+
#: What the predictor predicts for hidden objects: their ``encoding`` by
|
| 93 |
+
#: the target encoder, or their kinematic ``features`` (masked objects).
|
| 94 |
+
jepa_target: str = "encoding"
|
| 95 |
predictor_layers: int = 2
|
| 96 |
#: Fraction of each event's objects to predict (at least one, and at
|
| 97 |
#: least one left visible).
|
|
|
|
| 101 |
#: Weight of a VICReg variance term on the context encodings (and the
|
| 102 |
#: edge network's global state) that counters collapse.
|
| 103 |
variance_weight: float = 0.0
|
| 104 |
+
# Contrastive only (contrastive.py): the NT-Xent temperature and the
|
| 105 |
+
# views' momentum-scale jitter and object dropout probability.
|
| 106 |
+
temperature: float = 0.1
|
| 107 |
+
scale_jitter: float = 0.1
|
| 108 |
+
object_dropout: float = 0.1
|
| 109 |
+
# Probes of frozen encoders (probe.py).
|
| 110 |
+
probe_neighbours: int = 20
|
| 111 |
+
probe_l2: float = 1e-4
|
| 112 |
# PFN only (the encoder is the transformer above).
|
| 113 |
#: Classes the PFN can tell apart in one task.
|
| 114 |
max_classes: int = 10
|
|
|
|
| 116 |
pfn_layers: int = 6
|
| 117 |
# TabPFN only.
|
| 118 |
tabpfn_version: str = "v3.5"
|
| 119 |
+
#: A checkpoint of that version (a path, or a file name in the TabPFN
|
| 120 |
+
#: cache) instead of its default, e.g.
|
| 121 |
+
#: ``tabpfn-v3.5-20260909_multiclass.safetensors``.
|
| 122 |
+
tabpfn_checkpoint: str | None = None
|
| 123 |
tabpfn_estimators: int = 4
|
| 124 |
+
#: Meta-training of the TabPFN or TabICL network (``task.name: pfn``):
|
| 125 |
+
#: rank of the LoRA updates of its in-context blocks, or 0 to train all
|
| 126 |
+
#: its weights.
|
| 127 |
+
lora_rank: int = 8
|
| 128 |
+
# TabICL only.
|
| 129 |
+
tabicl_checkpoint: str = "tabicl-classifier-v2-20260212.ckpt"
|
| 130 |
+
tabicl_estimators: int = 8
|
| 131 |
+
# TabDPT only.
|
| 132 |
+
tabdpt_estimators: int = 8
|
| 133 |
+
#: Context events per query, its nearest rows (retrieval); null: all.
|
| 134 |
+
tabdpt_context_size: int | None = None
|
| 135 |
+
# The table rows of TabPFN, TabICL and TabDPT (tabular.py).
|
| 136 |
#: Objects of each type (in ``data.object_types`` order) in a table row.
|
| 137 |
slots: list[int] = field(default_factory=lambda: [6, 2, 2, 3, 1])
|
| 138 |
+
#: Azimuths measured from that of the event's leading object.
|
| 139 |
+
relative_phi: bool = False
|
| 140 |
+
#: Invariant mass and Delta R of every pair of this many leading objects.
|
| 141 |
+
pair_objects: int = 0
|
| 142 |
+
#: What a tabular model sees of an event: its ``rows``, the
|
| 143 |
+
#: ``embeddings`` of the encoder of ``model.backbone`` and
|
| 144 |
+
#: ``model.pretrained`` (sizes as above), or ``both``.
|
| 145 |
+
tabular_inputs: str = "rows"
|
| 146 |
|
| 147 |
|
| 148 |
@dataclass
|
| 149 |
class TaskConfig:
|
| 150 |
#: ``multiclass`` (``model.out_size`` classes), ``binary`` (one logit)
|
| 151 |
+
#: ``jepa`` or ``contrastive`` (pretraining, with the same
|
| 152 |
+
#: ``model.name``) or ``pfn``
|
| 153 |
+
#: (meta-training with ``model.name: pfn``, ``tabpfn`` or ``tabicl``).
|
| 154 |
name: str = "multiclass"
|
| 155 |
#: Use ``|weight|`` in the loss (signed NLO weights).
|
| 156 |
absolute_weights: bool = False
|
|
|
|
| 172 |
|
| 173 |
@dataclass
|
| 174 |
class TrainConfig:
|
| 175 |
+
#: ``auto`` (CUDA, else MPS, else CPU), ``cpu``, ``cuda`` or ``mps``.
|
| 176 |
device: str = "auto"
|
| 177 |
seed: int = 42
|
| 178 |
max_epochs: int = 100
|
|
|
|
| 224 |
self.run_dir = self.run_dir or f"outputs/{self.name}"
|
| 225 |
if self.task.name == "binary" and self.model.out_size != 1:
|
| 226 |
raise ValueError("binary classification needs model.out_size: 1")
|
| 227 |
+
if self.task.name == "pfn" and self.model.name not in (
|
| 228 |
+
"pfn",
|
| 229 |
+
"tabpfn",
|
| 230 |
+
"tabicl",
|
| 231 |
+
):
|
| 232 |
+
raise ValueError(
|
| 233 |
+
"task.name: pfn meta-trains model.name: pfn, tabpfn or tabicl"
|
| 234 |
+
)
|
| 235 |
+
for pretraining in ("jepa", "contrastive"):
|
| 236 |
+
if (self.model.name == pretraining) != (self.task.name == pretraining):
|
| 237 |
+
raise ValueError(
|
| 238 |
+
f"model.name: {pretraining} trains task.name: {pretraining}"
|
| 239 |
+
)
|
| 240 |
if self.split not in ("train", "validation", "test"):
|
| 241 |
raise ValueError("split must be train, validation or test")
|
| 242 |
|
|
@@ -0,0 +1,112 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Contrastive pretraining of an object encoder (SimCLR, Chen et al. 2020;
|
| 2 |
+
for jets JetCLR, Dillon et al. 2022, and RS3L, Harris et al. 2024).
|
| 3 |
+
|
| 4 |
+
Every event is seen in two random views. Each view rotates the event in
|
| 5 |
+
phi, flips it in eta (probability 1/2), scales its momenta and energies by
|
| 6 |
+
a factor uniform in ``1 +- model.scale_jitter`` and drops each object
|
| 7 |
+
other than the first with probability ``model.object_dropout``. The encoder
|
| 8 |
+
(``model.backbone``, as for JEPA) embeds each view (the mean of its object
|
| 9 |
+
encodings, or the edge network's decoded global state), and a projection MLP
|
| 10 |
+
maps the embedding to a unit vector. The NT-Xent loss at temperature
|
| 11 |
+
``model.temperature`` pulls the two views of an event together and pushes
|
| 12 |
+
them from the other events of the batch (on each rank) apart; labels are not
|
| 13 |
+
used. The pretrained encoder (``context_encoder``, as JEPA names it)
|
| 14 |
+
initializes a classifier through ``model.pretrained``.
|
| 15 |
+
"""
|
| 16 |
+
|
| 17 |
+
from __future__ import annotations
|
| 18 |
+
|
| 19 |
+
import math
|
| 20 |
+
from typing import Any
|
| 21 |
+
|
| 22 |
+
import torch
|
| 23 |
+
import torch.nn.functional as F
|
| 24 |
+
from torch import nn
|
| 25 |
+
|
| 26 |
+
from .jepa import event_embedding
|
| 27 |
+
from .sequences import augment_tokens
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
# Random augmentations from a generator do not trace; they are cheap.
|
| 31 |
+
@torch.compiler.disable
|
| 32 |
+
def random_view(
|
| 33 |
+
tokens: torch.Tensor,
|
| 34 |
+
mask: torch.Tensor,
|
| 35 |
+
*,
|
| 36 |
+
scale_jitter: float,
|
| 37 |
+
object_dropout: float,
|
| 38 |
+
generator: torch.Generator | None = None,
|
| 39 |
+
) -> tuple[torch.Tensor, torch.Tensor]:
|
| 40 |
+
"""An augmented view ``(tokens, mask)`` of every event (see the module
|
| 41 |
+
documentation)."""
|
| 42 |
+
events, device = tokens.shape[0], tokens.device
|
| 43 |
+
|
| 44 |
+
def uniform(*shape: int) -> torch.Tensor:
|
| 45 |
+
return torch.rand(shape, generator=generator, device=device)
|
| 46 |
+
|
| 47 |
+
scale = 1 + scale_jitter * (2 * uniform(events) - 1)
|
| 48 |
+
rotation = math.pi * (2 * uniform(events) - 1)
|
| 49 |
+
flip = torch.where(uniform(events) < 0.5, -1.0, 1.0)
|
| 50 |
+
view = augment_tokens(tokens, mask, scale=scale, rotation=rotation, flip=flip)
|
| 51 |
+
dropped = uniform(*mask.shape) < object_dropout
|
| 52 |
+
dropped[:, 0] = False # every event keeps an object
|
| 53 |
+
return view, mask & ~dropped
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
class Contrastive(nn.Module):
|
| 57 |
+
"""The encoder and the projection MLP.
|
| 58 |
+
|
| 59 |
+
``forward`` returns ``[events, 3]``: each event's loss, 1 (every event
|
| 60 |
+
counts), and the spread of the embeddings over the batch (their standard
|
| 61 |
+
deviation, averaged over dimensions; it falls towards zero if the
|
| 62 |
+
representation collapses).
|
| 63 |
+
"""
|
| 64 |
+
|
| 65 |
+
def __init__(
|
| 66 |
+
self,
|
| 67 |
+
encoder: nn.Module,
|
| 68 |
+
*,
|
| 69 |
+
temperature: float = 0.1,
|
| 70 |
+
scale_jitter: float = 0.1,
|
| 71 |
+
object_dropout: float = 0.1,
|
| 72 |
+
) -> None:
|
| 73 |
+
super().__init__()
|
| 74 |
+
hid_size = encoder.hid_size
|
| 75 |
+
self.temperature = temperature
|
| 76 |
+
self.scale_jitter, self.object_dropout = scale_jitter, object_dropout
|
| 77 |
+
self.context_encoder = encoder
|
| 78 |
+
self.projector = nn.Sequential(
|
| 79 |
+
nn.Linear(hid_size, hid_size), nn.ReLU(), nn.Linear(hid_size, hid_size)
|
| 80 |
+
)
|
| 81 |
+
|
| 82 |
+
def forward(self, batch: Any) -> torch.Tensor:
|
| 83 |
+
tokens, mask = batch.tokens, batch.token_mask
|
| 84 |
+
# Validation sees the same views every epoch.
|
| 85 |
+
generator = (
|
| 86 |
+
None if self.training else torch.Generator(tokens.device).manual_seed(0)
|
| 87 |
+
)
|
| 88 |
+
embedded = [
|
| 89 |
+
event_embedding(
|
| 90 |
+
self.context_encoder,
|
| 91 |
+
*random_view(
|
| 92 |
+
tokens,
|
| 93 |
+
mask,
|
| 94 |
+
scale_jitter=self.scale_jitter,
|
| 95 |
+
object_dropout=self.object_dropout,
|
| 96 |
+
generator=generator,
|
| 97 |
+
),
|
| 98 |
+
)
|
| 99 |
+
for _ in range(2)
|
| 100 |
+
]
|
| 101 |
+
spread = embedded[0].float().std(dim=0).mean()
|
| 102 |
+
z = F.normalize(self.projector(torch.cat(embedded)).float(), dim=1)
|
| 103 |
+
events = tokens.shape[0]
|
| 104 |
+
similarity = (z @ z.T) / self.temperature
|
| 105 |
+
similarity = similarity.masked_fill(
|
| 106 |
+
torch.eye(2 * events, dtype=torch.bool, device=z.device), float("-inf")
|
| 107 |
+
)
|
| 108 |
+
# The other view of each event is its positive.
|
| 109 |
+
positive = torch.arange(2 * events, device=z.device).roll(events)
|
| 110 |
+
loss = F.cross_entropy(similarity, positive, reduction="none")
|
| 111 |
+
loss = loss.view(2, events).mean(dim=0)
|
| 112 |
+
return torch.stack((loss, torch.ones_like(loss), spread.expand_as(loss)), dim=1)
|
|
@@ -30,10 +30,20 @@ class DistributedContext:
|
|
| 30 |
|
| 31 |
def resolve_device(device: str) -> torch.device:
|
| 32 |
if device == "auto":
|
| 33 |
-
|
|
|
|
|
|
|
| 34 |
return torch.device(device)
|
| 35 |
|
| 36 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
def initialize(device: str = "auto") -> DistributedContext:
|
| 38 |
world_size = int(os.environ.get("WORLD_SIZE", "1"))
|
| 39 |
requested = resolve_device(device)
|
|
|
|
| 30 |
|
| 31 |
def resolve_device(device: str) -> torch.device:
|
| 32 |
if device == "auto":
|
| 33 |
+
if torch.cuda.is_available():
|
| 34 |
+
return torch.device("cuda")
|
| 35 |
+
return torch.device("mps" if torch.backends.mps.is_available() else "cpu")
|
| 36 |
return torch.device(device)
|
| 37 |
|
| 38 |
|
| 39 |
+
def synchronize(device: torch.device) -> None:
|
| 40 |
+
"""Wait for the work queued on ``device``, for timing."""
|
| 41 |
+
if device.type == "cuda":
|
| 42 |
+
torch.cuda.synchronize(device)
|
| 43 |
+
elif device.type == "mps":
|
| 44 |
+
torch.mps.synchronize()
|
| 45 |
+
|
| 46 |
+
|
| 47 |
def initialize(device: str = "auto") -> DistributedContext:
|
| 48 |
world_size = int(os.environ.get("WORLD_SIZE", "1"))
|
| 49 |
requested = resolve_device(device)
|
|
@@ -10,6 +10,12 @@ encoder is an exponential moving average of the context encoder and gets no
|
|
| 10 |
gradient. The loss is the mean squared error between predicted and
|
| 11 |
layer-normalized target encodings; labels are not used.
|
| 12 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
The pretrained context encoder initializes a classifier of the same
|
| 14 |
backbone (``model.name: transformer`` or ``edge_network`` with
|
| 15 |
``model.pretrained``).
|
|
@@ -34,6 +40,7 @@ from torch import nn
|
|
| 34 |
|
| 35 |
from .graphs import build_graph
|
| 36 |
from .models import NODE_TYPE, EdgeNetwork, EncoderBlock
|
|
|
|
| 37 |
|
| 38 |
#: Objects of one type beyond this index share the last index embedding.
|
| 39 |
MAX_TYPE_INDEX = 16
|
|
@@ -93,6 +100,42 @@ def encode(
|
|
| 93 |
return states, encoder.global_decoder(global_h)
|
| 94 |
|
| 95 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 96 |
def _variance_penalty(encodings: torch.Tensor) -> torch.Tensor:
|
| 97 |
"""VICReg variance term (Bardes et al. 2022): the mean shortfall of each
|
| 98 |
dimension's standard deviation over ``encodings [n, dims]`` below 1."""
|
|
@@ -106,7 +149,8 @@ class JEPA(nn.Module):
|
|
| 106 |
``forward`` returns ``[events, 3]``: each event's loss, whether it has
|
| 107 |
hidden objects, and the spread of the target encodings over the batch
|
| 108 |
(their standard deviation over objects, averaged over dimensions; it
|
| 109 |
-
falls towards zero if the representation collapses).
|
|
|
|
| 110 |
"""
|
| 111 |
|
| 112 |
def __init__(
|
|
@@ -121,13 +165,22 @@ class JEPA(nn.Module):
|
|
| 121 |
mask_fraction: float = 0.4,
|
| 122 |
ema_momentum: float = 0.996,
|
| 123 |
variance_weight: float = 0.0,
|
|
|
|
| 124 |
) -> None:
|
| 125 |
super().__init__()
|
|
|
|
|
|
|
|
|
|
|
|
|
| 126 |
hid_size = encoder.hid_size
|
| 127 |
self.mask_fraction, self.ema_momentum = mask_fraction, ema_momentum
|
| 128 |
self.variance_weight = variance_weight
|
| 129 |
self.context_encoder = encoder
|
| 130 |
-
self.target_encoder =
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
self.query_type = nn.Embedding(num_node_types, hid_size)
|
| 132 |
self.query_index = nn.Embedding(MAX_TYPE_INDEX, hid_size)
|
| 133 |
self.predictor = nn.ModuleList(
|
|
@@ -135,7 +188,9 @@ class JEPA(nn.Module):
|
|
| 135 |
for _ in range(predictor_layers)
|
| 136 |
)
|
| 137 |
self.predictor_norm = nn.LayerNorm(hid_size)
|
| 138 |
-
self.predictor_out = nn.Linear(
|
|
|
|
|
|
|
| 139 |
#: Optimizer steps taken, and the steps over which the momentum
|
| 140 |
#: reaches 1 (set before training; see :meth:`update_target`).
|
| 141 |
self.register_buffer("step", torch.zeros((), dtype=torch.long))
|
|
@@ -149,10 +204,15 @@ class JEPA(nn.Module):
|
|
| 149 |
)
|
| 150 |
hidden = sample_targets(mask, self.mask_fraction, generator)
|
| 151 |
with torch.no_grad():
|
| 152 |
-
|
| 153 |
-
|
| 154 |
-
|
|
|
|
|
|
|
|
|
|
| 155 |
predicted, context, event = self.predict(tokens, mask, hidden)
|
|
|
|
|
|
|
| 156 |
error = (predicted - target).square().mean(dim=-1) * hidden
|
| 157 |
count = hidden.sum(dim=1)
|
| 158 |
loss = error.sum(dim=1) / count.clamp_min(1)
|
|
@@ -193,6 +253,8 @@ class JEPA(nn.Module):
|
|
| 193 |
"""Move the target encoder towards the context encoder, with a
|
| 194 |
momentum rising linearly from ``ema_momentum`` to 1 over
|
| 195 |
``total_steps``. Called by the trainer after each optimizer step."""
|
|
|
|
|
|
|
| 196 |
progress = (self.step.float() / self.total_steps).clamp(max=1)
|
| 197 |
momentum = self.ema_momentum + (1 - self.ema_momentum) * progress
|
| 198 |
for target, context in zip(
|
|
|
|
| 10 |
gradient. The loss is the mean squared error between predicted and
|
| 11 |
layer-normalized target encodings; labels are not used.
|
| 12 |
|
| 13 |
+
With ``model.jepa_target: features`` the predictor instead predicts the
|
| 14 |
+
hidden objects' kinematics, ``log(1 + pt)``, eta, ``cos phi``, ``sin phi``,
|
| 15 |
+
``log(1 + energy)``, b-tag and charge, as in masked particle modelling
|
| 16 |
+
(Golling et al. 2024); there is no target encoder. Both differ only in
|
| 17 |
+
whether the target is an encoding or the input.
|
| 18 |
+
|
| 19 |
The pretrained context encoder initializes a classifier of the same
|
| 20 |
backbone (``model.name: transformer`` or ``edge_network`` with
|
| 21 |
``model.pretrained``).
|
|
|
|
| 40 |
|
| 41 |
from .graphs import build_graph
|
| 42 |
from .models import NODE_TYPE, EdgeNetwork, EncoderBlock
|
| 43 |
+
from .sequences import BTAG, CHARGE, ENERGY, ETA, PHI, PT
|
| 44 |
|
| 45 |
#: Objects of one type beyond this index share the last index embedding.
|
| 46 |
MAX_TYPE_INDEX = 16
|
|
|
|
| 100 |
return states, encoder.global_decoder(global_h)
|
| 101 |
|
| 102 |
|
| 103 |
+
def event_embedding(
|
| 104 |
+
encoder: nn.Module, tokens: torch.Tensor, mask: torch.Tensor
|
| 105 |
+
) -> torch.Tensor:
|
| 106 |
+
"""Embeddings ``[events, hid_size]`` of events by an object encoder: the
|
| 107 |
+
mean of the encodings of the objects in ``mask``, or the edge network's
|
| 108 |
+
decoded global state (what its classifier reads)."""
|
| 109 |
+
states, event = encode(encoder, tokens, mask)
|
| 110 |
+
if event is not None:
|
| 111 |
+
return event
|
| 112 |
+
counts = mask.sum(dim=1, keepdim=True).clamp_min(1)
|
| 113 |
+
return (states * mask.unsqueeze(-1)).sum(dim=1) / counts
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
#: Width of the kinematic targets of :func:`object_features`.
|
| 117 |
+
FEATURE_TARGETS = 7
|
| 118 |
+
|
| 119 |
+
|
| 120 |
+
def object_features(tokens: torch.Tensor) -> torch.Tensor:
|
| 121 |
+
"""Kinematic targets ``[events, objects, FEATURE_TARGETS]`` (see the
|
| 122 |
+
module documentation)."""
|
| 123 |
+
pt, energy, phi = tokens[..., PT], tokens[..., ENERGY], tokens[..., PHI]
|
| 124 |
+
btag, charge = tokens[..., BTAG], tokens[..., CHARGE]
|
| 125 |
+
return torch.stack(
|
| 126 |
+
(
|
| 127 |
+
pt.clamp_min(0).log1p(),
|
| 128 |
+
tokens[..., ETA],
|
| 129 |
+
phi.cos(),
|
| 130 |
+
phi.sin(),
|
| 131 |
+
energy.clamp_min(0).log1p(),
|
| 132 |
+
btag,
|
| 133 |
+
charge,
|
| 134 |
+
),
|
| 135 |
+
dim=-1,
|
| 136 |
+
)
|
| 137 |
+
|
| 138 |
+
|
| 139 |
def _variance_penalty(encodings: torch.Tensor) -> torch.Tensor:
|
| 140 |
"""VICReg variance term (Bardes et al. 2022): the mean shortfall of each
|
| 141 |
dimension's standard deviation over ``encodings [n, dims]`` below 1."""
|
|
|
|
| 149 |
``forward`` returns ``[events, 3]``: each event's loss, whether it has
|
| 150 |
hidden objects, and the spread of the target encodings over the batch
|
| 151 |
(their standard deviation over objects, averaged over dimensions; it
|
| 152 |
+
falls towards zero if the representation collapses). With kinematic
|
| 153 |
+
targets, the spread is that of the context encodings.
|
| 154 |
"""
|
| 155 |
|
| 156 |
def __init__(
|
|
|
|
| 165 |
mask_fraction: float = 0.4,
|
| 166 |
ema_momentum: float = 0.996,
|
| 167 |
variance_weight: float = 0.0,
|
| 168 |
+
target: str = "encoding",
|
| 169 |
) -> None:
|
| 170 |
super().__init__()
|
| 171 |
+
if target not in ("encoding", "features"):
|
| 172 |
+
raise ValueError(
|
| 173 |
+
f"unknown jepa_target {target!r}; use encoding or features"
|
| 174 |
+
)
|
| 175 |
hid_size = encoder.hid_size
|
| 176 |
self.mask_fraction, self.ema_momentum = mask_fraction, ema_momentum
|
| 177 |
self.variance_weight = variance_weight
|
| 178 |
self.context_encoder = encoder
|
| 179 |
+
self.target_encoder = (
|
| 180 |
+
copy.deepcopy(encoder).requires_grad_(False)
|
| 181 |
+
if target == "encoding"
|
| 182 |
+
else None
|
| 183 |
+
)
|
| 184 |
self.query_type = nn.Embedding(num_node_types, hid_size)
|
| 185 |
self.query_index = nn.Embedding(MAX_TYPE_INDEX, hid_size)
|
| 186 |
self.predictor = nn.ModuleList(
|
|
|
|
| 188 |
for _ in range(predictor_layers)
|
| 189 |
)
|
| 190 |
self.predictor_norm = nn.LayerNorm(hid_size)
|
| 191 |
+
self.predictor_out = nn.Linear(
|
| 192 |
+
hid_size, hid_size if target == "encoding" else FEATURE_TARGETS
|
| 193 |
+
)
|
| 194 |
#: Optimizer steps taken, and the steps over which the momentum
|
| 195 |
#: reaches 1 (set before training; see :meth:`update_target`).
|
| 196 |
self.register_buffer("step", torch.zeros((), dtype=torch.long))
|
|
|
|
| 204 |
)
|
| 205 |
hidden = sample_targets(mask, self.mask_fraction, generator)
|
| 206 |
with torch.no_grad():
|
| 207 |
+
if self.target_encoder is None:
|
| 208 |
+
target = object_features(tokens).float()
|
| 209 |
+
else:
|
| 210 |
+
target = encode(self.target_encoder, tokens, mask)[0].float()
|
| 211 |
+
spread = target[mask].std(dim=0).mean()
|
| 212 |
+
target = nn.functional.layer_norm(target, target.shape[-1:])
|
| 213 |
predicted, context, event = self.predict(tokens, mask, hidden)
|
| 214 |
+
if self.target_encoder is None:
|
| 215 |
+
spread = context[mask & ~hidden].detach().float().std(dim=0).mean()
|
| 216 |
error = (predicted - target).square().mean(dim=-1) * hidden
|
| 217 |
count = hidden.sum(dim=1)
|
| 218 |
loss = error.sum(dim=1) / count.clamp_min(1)
|
|
|
|
| 253 |
"""Move the target encoder towards the context encoder, with a
|
| 254 |
momentum rising linearly from ``ema_momentum`` to 1 over
|
| 255 |
``total_steps``. Called by the trainer after each optimizer step."""
|
| 256 |
+
if self.target_encoder is None:
|
| 257 |
+
return
|
| 258 |
progress = (self.step.float() / self.total_steps).clamp(max=1)
|
| 259 |
momentum = self.ema_momentum + (1 - self.ema_momentum) * progress
|
| 260 |
for target, context in zip(
|
|
@@ -14,11 +14,11 @@ from torch import nn
|
|
| 14 |
|
| 15 |
from .data import NODE_FEATURE_NAMES
|
| 16 |
from .graphs import EDGE_FEATURE_NAMES, TensorGraph
|
|
|
|
| 17 |
|
| 18 |
if TYPE_CHECKING:
|
| 19 |
from .config import ModelConfig
|
| 20 |
|
| 21 |
-
PT, ETA, PHI = (NODE_FEATURE_NAMES.index(name) for name in ("pt", "eta", "phi"))
|
| 22 |
NODE_TYPE = NODE_FEATURE_NAMES.index("node_type")
|
| 23 |
|
| 24 |
|
|
@@ -413,9 +413,18 @@ class DeepSets(nn.Module):
|
|
| 413 |
|
| 414 |
|
| 415 |
#: Models that take :class:`~gnn4colliders.sequences.SequenceBatch` inputs.
|
| 416 |
-
SEQUENCE_MODELS = (
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 417 |
#: Models that classify events given labelled context events (see cli.py).
|
| 418 |
-
IN_CONTEXT_MODELS = ("pfn", "tabpfn")
|
| 419 |
|
| 420 |
|
| 421 |
def state_dict_of(checkpoint: Mapping[str, Any]) -> dict[str, Any]:
|
|
@@ -440,17 +449,47 @@ def load_weights(model: nn.Module, checkpoint: Mapping[str, Any]) -> None:
|
|
| 440 |
model.load_state_dict(state_dict_of(checkpoint), strict=True)
|
| 441 |
|
| 442 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 443 |
def pretrained_state(path: str) -> dict[str, Any]:
|
| 444 |
"""Weights of a pretrained backbone: those of a supervised checkpoint, or
|
| 445 |
-
the context encoder of a JEPA checkpoint."""
|
| 446 |
-
state =
|
| 447 |
-
|
| 448 |
-
|
| 449 |
-
|
| 450 |
-
|
| 451 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 452 |
}
|
| 453 |
-
return encoder or state
|
| 454 |
|
| 455 |
|
| 456 |
def _load_encoder(encoder: ObjectTransformer, path: str) -> None:
|
|
@@ -460,23 +499,10 @@ def _load_encoder(encoder: ObjectTransformer, path: str) -> None:
|
|
| 460 |
encoder.load_state_dict(state, strict=True)
|
| 461 |
|
| 462 |
|
| 463 |
-
def
|
| 464 |
-
config: ModelConfig,
|
| 465 |
-
|
| 466 |
-
|
| 467 |
-
num_node_types: int,
|
| 468 |
-
num_global_features: int = 0,
|
| 469 |
-
load_pretrained: bool = True,
|
| 470 |
-
) -> nn.Module:
|
| 471 |
-
"""The model of ``config`` for inputs of the given widths.
|
| 472 |
-
|
| 473 |
-
With ``config.pretrained`` the edge network is a
|
| 474 |
-
:class:`FineTunedEdgeNetwork` whose backbone is loaded from a supervised
|
| 475 |
-
or JEPA checkpoint, and the transformer's encoder is loaded from a JEPA
|
| 476 |
-
checkpoint, as is the PFN's (all skipped with ``load_pretrained=False``,
|
| 477 |
-
e.g. when a trained checkpoint is loaded next).
|
| 478 |
-
"""
|
| 479 |
-
transformer = {
|
| 480 |
"input_size": num_features,
|
| 481 |
"num_node_types": num_node_types,
|
| 482 |
"hid_size": config.hid_size,
|
|
@@ -486,7 +512,10 @@ def build_model(
|
|
| 486 |
"dropout": config.dropout,
|
| 487 |
"pair_bias": config.pair_bias,
|
| 488 |
}
|
| 489 |
-
|
|
|
|
|
|
|
|
|
|
| 490 |
"node_size": num_features,
|
| 491 |
"edge_size": len(EDGE_FEATURE_NAMES),
|
| 492 |
"hid_size": config.hid_size,
|
|
@@ -494,19 +523,87 @@ def build_model(
|
|
| 494 |
"n_proc_steps": config.n_proc_steps,
|
| 495 |
"dropout": config.dropout,
|
| 496 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 497 |
if config.name == "jepa":
|
| 498 |
from .jepa import JEPA
|
| 499 |
|
| 500 |
-
|
| 501 |
-
|
| 502 |
-
|
| 503 |
-
|
| 504 |
-
|
| 505 |
-
|
| 506 |
-
|
| 507 |
-
raise ValueError(
|
| 508 |
-
f"unknown backbone {config.backbone!r}; use transformer or edge_network"
|
| 509 |
-
)
|
| 510 |
return JEPA(
|
| 511 |
encoder,
|
| 512 |
num_node_types=num_node_types,
|
|
@@ -517,6 +614,22 @@ def build_model(
|
|
| 517 |
mask_fraction=config.mask_fraction,
|
| 518 |
ema_momentum=config.ema_momentum,
|
| 519 |
variance_weight=config.variance_weight,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 520 |
)
|
| 521 |
if config.name == "transformer":
|
| 522 |
model = TransformerClassifier(
|
|
@@ -542,8 +655,30 @@ def build_model(
|
|
| 542 |
ffn_size=config.ffn_size or 4 * config.hid_size,
|
| 543 |
dropout=config.dropout,
|
| 544 |
)
|
| 545 |
-
if config.name
|
| 546 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 547 |
if config.name == "deep_sets":
|
| 548 |
return DeepSets(
|
| 549 |
input_size=num_features,
|
|
@@ -557,7 +692,7 @@ def build_model(
|
|
| 557 |
if config.name != "edge_network":
|
| 558 |
raise ValueError(
|
| 559 |
f"unknown model {config.name!r}; use edge_network, transformer, "
|
| 560 |
-
"deep_sets, jepa, pfn or
|
| 561 |
)
|
| 562 |
sizes = {"global_size": num_global_features, **edge_network}
|
| 563 |
if not config.pretrained:
|
|
|
|
| 14 |
|
| 15 |
from .data import NODE_FEATURE_NAMES
|
| 16 |
from .graphs import EDGE_FEATURE_NAMES, TensorGraph
|
| 17 |
+
from .sequences import ETA, PHI, PT
|
| 18 |
|
| 19 |
if TYPE_CHECKING:
|
| 20 |
from .config import ModelConfig
|
| 21 |
|
|
|
|
| 22 |
NODE_TYPE = NODE_FEATURE_NAMES.index("node_type")
|
| 23 |
|
| 24 |
|
|
|
|
| 413 |
|
| 414 |
|
| 415 |
#: Models that take :class:`~gnn4colliders.sequences.SequenceBatch` inputs.
|
| 416 |
+
SEQUENCE_MODELS = (
|
| 417 |
+
"transformer",
|
| 418 |
+
"deep_sets",
|
| 419 |
+
"jepa",
|
| 420 |
+
"contrastive",
|
| 421 |
+
"pfn",
|
| 422 |
+
"tabpfn",
|
| 423 |
+
"tabicl",
|
| 424 |
+
"tabdpt",
|
| 425 |
+
)
|
| 426 |
#: Models that classify events given labelled context events (see cli.py).
|
| 427 |
+
IN_CONTEXT_MODELS = ("pfn", "tabpfn", "tabicl", "tabdpt")
|
| 428 |
|
| 429 |
|
| 430 |
def state_dict_of(checkpoint: Mapping[str, Any]) -> dict[str, Any]:
|
|
|
|
| 449 |
model.load_state_dict(state_dict_of(checkpoint), strict=True)
|
| 450 |
|
| 451 |
|
| 452 |
+
def _strip_prefix(
|
| 453 |
+
state: dict[str, Any], prefixes: tuple[str, ...]
|
| 454 |
+
) -> dict[str, Any] | None:
|
| 455 |
+
"""The entries of ``state`` under the first of ``prefixes`` it has, the
|
| 456 |
+
prefix removed, or None."""
|
| 457 |
+
for prefix in prefixes:
|
| 458 |
+
part = {
|
| 459 |
+
key.removeprefix(prefix): value
|
| 460 |
+
for key, value in state.items()
|
| 461 |
+
if key.startswith(prefix)
|
| 462 |
+
}
|
| 463 |
+
if part:
|
| 464 |
+
return part
|
| 465 |
+
return None
|
| 466 |
+
|
| 467 |
+
|
| 468 |
+
def _load_state(path: str) -> dict[str, Any]:
|
| 469 |
+
return state_dict_of(torch.load(path, map_location="cpu", weights_only=False))
|
| 470 |
+
|
| 471 |
+
|
| 472 |
def pretrained_state(path: str) -> dict[str, Any]:
|
| 473 |
"""Weights of a pretrained backbone: those of a supervised checkpoint, or
|
| 474 |
+
the context encoder of a JEPA or contrastive checkpoint."""
|
| 475 |
+
state = _load_state(path)
|
| 476 |
+
return _strip_prefix(state, ("context_encoder.",)) or state
|
| 477 |
+
|
| 478 |
+
|
| 479 |
+
#: Prefixes of an encoder's weights in the checkpoints this package writes:
|
| 480 |
+
#: pretraining (JEPA, contrastive), a supervised transformer, a fine-tuned
|
| 481 |
+
#: edge network and the encoder of a meta-trained tabular model.
|
| 482 |
+
ENCODER_PREFIXES = ("context_encoder.", "encoder.", "backbone.", "inputs.encoder.")
|
| 483 |
+
|
| 484 |
+
|
| 485 |
+
def encoder_state(path: str) -> dict[str, torch.Tensor]:
|
| 486 |
+
"""The encoder weights of a checkpoint (see ``ENCODER_PREFIXES``; else a
|
| 487 |
+
supervised edge network), without a classifier."""
|
| 488 |
+
state = _load_state(path)
|
| 489 |
+
state = _strip_prefix(state, ENCODER_PREFIXES) or state
|
| 490 |
+
return {
|
| 491 |
+
key: value for key, value in state.items() if not key.startswith("classifier.")
|
| 492 |
}
|
|
|
|
| 493 |
|
| 494 |
|
| 495 |
def _load_encoder(encoder: ObjectTransformer, path: str) -> None:
|
|
|
|
| 499 |
encoder.load_state_dict(state, strict=True)
|
| 500 |
|
| 501 |
|
| 502 |
+
def _transformer_sizes(
|
| 503 |
+
config: ModelConfig, num_features: int, num_node_types: int
|
| 504 |
+
) -> dict[str, Any]:
|
| 505 |
+
return {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 506 |
"input_size": num_features,
|
| 507 |
"num_node_types": num_node_types,
|
| 508 |
"hid_size": config.hid_size,
|
|
|
|
| 512 |
"dropout": config.dropout,
|
| 513 |
"pair_bias": config.pair_bias,
|
| 514 |
}
|
| 515 |
+
|
| 516 |
+
|
| 517 |
+
def _edge_network_sizes(config: ModelConfig, num_features: int) -> dict[str, Any]:
|
| 518 |
+
return {
|
| 519 |
"node_size": num_features,
|
| 520 |
"edge_size": len(EDGE_FEATURE_NAMES),
|
| 521 |
"hid_size": config.hid_size,
|
|
|
|
| 523 |
"n_proc_steps": config.n_proc_steps,
|
| 524 |
"dropout": config.dropout,
|
| 525 |
}
|
| 526 |
+
|
| 527 |
+
|
| 528 |
+
def build_encoder(
|
| 529 |
+
config: ModelConfig, backbone: str, *, num_features: int, num_node_types: int
|
| 530 |
+
) -> nn.Module:
|
| 531 |
+
"""The object encoder ``backbone`` (``transformer`` or ``edge_network``)
|
| 532 |
+
with the sizes of ``config``, as pretrained (JEPA, contrastive) and
|
| 533 |
+
probed: the edge network without global features or classifier."""
|
| 534 |
+
if backbone == "transformer":
|
| 535 |
+
return ObjectTransformer(
|
| 536 |
+
**_transformer_sizes(config, num_features, num_node_types)
|
| 537 |
+
)
|
| 538 |
+
if backbone == "edge_network":
|
| 539 |
+
encoder = EdgeNetwork(
|
| 540 |
+
global_size=0, out_size=1, **_edge_network_sizes(config, num_features)
|
| 541 |
+
)
|
| 542 |
+
encoder.classifier = nn.Identity()
|
| 543 |
+
return encoder
|
| 544 |
+
raise ValueError(f"unknown backbone {backbone!r}; use transformer or edge_network")
|
| 545 |
+
|
| 546 |
+
|
| 547 |
+
#: Seed of the initialization of encoders without pretrained weights, so
|
| 548 |
+
#: that a frozen random encoder is rebuilt identically at evaluation and on
|
| 549 |
+
#: every rank.
|
| 550 |
+
RANDOM_ENCODER_SEED = 0
|
| 551 |
+
|
| 552 |
+
|
| 553 |
+
def pretrained_encoder(
|
| 554 |
+
config: ModelConfig,
|
| 555 |
+
*,
|
| 556 |
+
num_features: int,
|
| 557 |
+
num_node_types: int,
|
| 558 |
+
backbone: str | None = None,
|
| 559 |
+
freeze: bool | None = None,
|
| 560 |
+
) -> nn.Module:
|
| 561 |
+
"""The object encoder ``backbone`` (default ``config.backbone``) with
|
| 562 |
+
the weights of ``config.pretrained``, or randomly initialized with
|
| 563 |
+
``RANDOM_ENCODER_SEED`` without; frozen if ``freeze`` (default
|
| 564 |
+
``config.freeze_backbone``)."""
|
| 565 |
+
with torch.random.fork_rng(devices=[]):
|
| 566 |
+
torch.manual_seed(RANDOM_ENCODER_SEED)
|
| 567 |
+
encoder = build_encoder(
|
| 568 |
+
config,
|
| 569 |
+
backbone or config.backbone,
|
| 570 |
+
num_features=num_features,
|
| 571 |
+
num_node_types=num_node_types,
|
| 572 |
+
)
|
| 573 |
+
if config.pretrained:
|
| 574 |
+
encoder.load_state_dict(encoder_state(config.pretrained), strict=True)
|
| 575 |
+
freeze = config.freeze_backbone if freeze is None else freeze
|
| 576 |
+
return encoder.requires_grad_(not freeze)
|
| 577 |
+
|
| 578 |
+
|
| 579 |
+
def build_model(
|
| 580 |
+
config: ModelConfig,
|
| 581 |
+
*,
|
| 582 |
+
num_features: int,
|
| 583 |
+
num_node_types: int,
|
| 584 |
+
num_global_features: int = 0,
|
| 585 |
+
load_pretrained: bool = True,
|
| 586 |
+
) -> nn.Module:
|
| 587 |
+
"""The model of ``config`` for inputs of the given widths.
|
| 588 |
+
|
| 589 |
+
With ``config.pretrained`` the edge network is a
|
| 590 |
+
:class:`FineTunedEdgeNetwork` whose backbone is loaded from a supervised
|
| 591 |
+
or JEPA checkpoint, and the transformer's encoder is loaded from a JEPA
|
| 592 |
+
checkpoint, as is the PFN's (all skipped with ``load_pretrained=False``,
|
| 593 |
+
e.g. when a trained checkpoint is loaded next).
|
| 594 |
+
"""
|
| 595 |
+
transformer = _transformer_sizes(config, num_features, num_node_types)
|
| 596 |
+
edge_network = _edge_network_sizes(config, num_features)
|
| 597 |
if config.name == "jepa":
|
| 598 |
from .jepa import JEPA
|
| 599 |
|
| 600 |
+
# Global features are not used in pretraining.
|
| 601 |
+
encoder = build_encoder(
|
| 602 |
+
config,
|
| 603 |
+
config.backbone,
|
| 604 |
+
num_features=num_features,
|
| 605 |
+
num_node_types=num_node_types,
|
| 606 |
+
)
|
|
|
|
|
|
|
|
|
|
| 607 |
return JEPA(
|
| 608 |
encoder,
|
| 609 |
num_node_types=num_node_types,
|
|
|
|
| 614 |
mask_fraction=config.mask_fraction,
|
| 615 |
ema_momentum=config.ema_momentum,
|
| 616 |
variance_weight=config.variance_weight,
|
| 617 |
+
target=config.jepa_target,
|
| 618 |
+
)
|
| 619 |
+
if config.name == "contrastive":
|
| 620 |
+
from .contrastive import Contrastive
|
| 621 |
+
|
| 622 |
+
encoder = build_encoder(
|
| 623 |
+
config,
|
| 624 |
+
config.backbone,
|
| 625 |
+
num_features=num_features,
|
| 626 |
+
num_node_types=num_node_types,
|
| 627 |
+
)
|
| 628 |
+
return Contrastive(
|
| 629 |
+
encoder,
|
| 630 |
+
temperature=config.temperature,
|
| 631 |
+
scale_jitter=config.scale_jitter,
|
| 632 |
+
object_dropout=config.object_dropout,
|
| 633 |
)
|
| 634 |
if config.name == "transformer":
|
| 635 |
model = TransformerClassifier(
|
|
|
|
| 655 |
ffn_size=config.ffn_size or 4 * config.hid_size,
|
| 656 |
dropout=config.dropout,
|
| 657 |
)
|
| 658 |
+
if config.name in ("tabpfn", "tabicl"):
|
| 659 |
+
from . import tabular
|
| 660 |
+
|
| 661 |
+
# The released weights load whatever load_pretrained says.
|
| 662 |
+
if config.name == "tabpfn":
|
| 663 |
+
network = tabular.load_tabpfn_network(
|
| 664 |
+
config.tabpfn_version, config.tabpfn_checkpoint
|
| 665 |
+
)
|
| 666 |
+
meta = tabular.TabularPFN
|
| 667 |
+
else:
|
| 668 |
+
network = tabular.load_tabicl_network(config.tabicl_checkpoint)
|
| 669 |
+
meta = tabular.TabularICL
|
| 670 |
+
# So do those of a frozen encoder of the inputs.
|
| 671 |
+
inputs = tabular.EventInputs.of(
|
| 672 |
+
config, num_features=num_features, num_node_types=num_node_types
|
| 673 |
+
)
|
| 674 |
+
return meta(
|
| 675 |
+
network,
|
| 676 |
+
inputs=inputs,
|
| 677 |
+
max_classes=config.max_classes,
|
| 678 |
+
lora_rank=config.lora_rank,
|
| 679 |
+
)
|
| 680 |
+
if config.name == "tabdpt":
|
| 681 |
+
raise ValueError("tabdpt is not trained; run evaluate or predict")
|
| 682 |
if config.name == "deep_sets":
|
| 683 |
return DeepSets(
|
| 684 |
input_size=num_features,
|
|
|
|
| 692 |
if config.name != "edge_network":
|
| 693 |
raise ValueError(
|
| 694 |
f"unknown model {config.name!r}; use edge_network, transformer, "
|
| 695 |
+
"deep_sets, jepa, contrastive, pfn, tabpfn, tabicl or tabdpt"
|
| 696 |
)
|
| 697 |
sizes = {"global_size": num_global_features, **edge_network}
|
| 698 |
if not config.pretrained:
|
|
@@ -29,11 +29,10 @@ import torch
|
|
| 29 |
from torch import nn
|
| 30 |
|
| 31 |
from .config import TaskConfig
|
| 32 |
-
from .data import
|
| 33 |
-
from .models import
|
| 34 |
-
from .sequences import SequenceBatch, collate_sequence_batch
|
| 35 |
|
| 36 |
-
ENERGY = NODE_FEATURE_NAMES.index("energy")
|
| 37 |
#: Logit of the classes a task does not have.
|
| 38 |
ABSENT_LOGIT = -1e4
|
| 39 |
|
|
@@ -177,13 +176,10 @@ class _Step:
|
|
| 177 |
|
| 178 |
def _augment(batch: SequenceBatch, step: _Step) -> SequenceBatch:
|
| 179 |
"""Scale momenta and energies, rotate in phi and flip eta, per event."""
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
|
| 183 |
-
|
| 184 |
-
phi = tokens[..., PHI] + step.rotation[:, None]
|
| 185 |
-
tokens[..., PHI] = torch.remainder(phi + math.pi, 2 * math.pi) - math.pi
|
| 186 |
-
tokens = torch.where(mask.unsqueeze(-1), tokens, batch.tokens)
|
| 187 |
return SequenceBatch(
|
| 188 |
tokens=tokens,
|
| 189 |
token_mask=mask,
|
|
|
|
| 29 |
from torch import nn
|
| 30 |
|
| 31 |
from .config import TaskConfig
|
| 32 |
+
from .data import PackedEvents, map_tensors
|
| 33 |
+
from .models import EncoderBlock, ObjectTransformer
|
| 34 |
+
from .sequences import SequenceBatch, augment_tokens, collate_sequence_batch
|
| 35 |
|
|
|
|
| 36 |
#: Logit of the classes a task does not have.
|
| 37 |
ABSENT_LOGIT = -1e4
|
| 38 |
|
|
|
|
| 176 |
|
| 177 |
def _augment(batch: SequenceBatch, step: _Step) -> SequenceBatch:
|
| 178 |
"""Scale momenta and energies, rotate in phi and flip eta, per event."""
|
| 179 |
+
mask = batch.token_mask
|
| 180 |
+
tokens = augment_tokens(
|
| 181 |
+
batch.tokens, mask, scale=step.scale, rotation=step.rotation, flip=step.flip
|
| 182 |
+
)
|
|
|
|
|
|
|
|
|
|
| 183 |
return SequenceBatch(
|
| 184 |
tokens=tokens,
|
| 185 |
token_mask=mask,
|
|
@@ -0,0 +1,161 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""k-NN and linear probes of a frozen object encoder (as for jBOT, Tsoi and
|
| 2 |
+
Rankin 2026), to compare pretraining objectives without fine-tuning.
|
| 3 |
+
|
| 4 |
+
The encoder of ``model.pretrained`` embeds every event: the mean of its
|
| 5 |
+
object encodings, or the edge network's decoded global state (what its
|
| 6 |
+
classifier reads). The checkpoint can be a JEPA, contrastive or masked-object
|
| 7 |
+
one (its context encoder) or a supervised transformer or edge network; with
|
| 8 |
+
none, the encoder is randomly initialized (with a fixed seed), the
|
| 9 |
+
reference every pretraining should beat. The config can be that of the
|
| 10 |
+
pretraining or of a classifier with the encoder's sizes. Two classifiers of
|
| 11 |
+
the frozen embeddings, of all the dataset's classes, are trained on
|
| 12 |
+
``data.train_events`` labelled training events and evaluated on ``split``
|
| 13 |
+
(without those events):
|
| 14 |
+
|
| 15 |
+
* k-NN: the class fractions among the ``model.probe_neighbours`` training
|
| 16 |
+
embeddings of largest cosine similarity;
|
| 17 |
+
* linear: multinomial logistic regression of the standardized embeddings
|
| 18 |
+
(full-batch L-BFGS, L2 penalty ``model.probe_l2``).
|
| 19 |
+
"""
|
| 20 |
+
|
| 21 |
+
from __future__ import annotations
|
| 22 |
+
|
| 23 |
+
from typing import Any
|
| 24 |
+
|
| 25 |
+
import torch
|
| 26 |
+
import torch.nn.functional as F
|
| 27 |
+
from torch import nn
|
| 28 |
+
|
| 29 |
+
from .config import Config
|
| 30 |
+
from .data import PackedEvents
|
| 31 |
+
from .jepa import event_embedding
|
| 32 |
+
from .models import pretrained_encoder
|
| 33 |
+
from .sequences import collate_sequence_batch
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def backbone_of(config: Config) -> str:
|
| 37 |
+
"""The encoder architecture of the model of ``config``."""
|
| 38 |
+
name = config.model.name
|
| 39 |
+
if name in ("jepa", "contrastive"):
|
| 40 |
+
return config.model.backbone
|
| 41 |
+
if name in ("transformer", "edge_network"):
|
| 42 |
+
return name
|
| 43 |
+
raise ValueError(f"cannot probe model.name {name!r}")
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def load_encoder(config: Config, events: PackedEvents) -> nn.Module:
|
| 47 |
+
"""The (pretrained) frozen encoder of ``config``."""
|
| 48 |
+
encoder = pretrained_encoder(
|
| 49 |
+
config.model,
|
| 50 |
+
num_features=events.num_features,
|
| 51 |
+
num_node_types=len(config.data.object_types),
|
| 52 |
+
backbone=backbone_of(config),
|
| 53 |
+
freeze=True,
|
| 54 |
+
)
|
| 55 |
+
return encoder.eval()
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
@torch.inference_mode()
|
| 59 |
+
def embed(
|
| 60 |
+
encoder: nn.Module,
|
| 61 |
+
events: PackedEvents,
|
| 62 |
+
indices: torch.Tensor,
|
| 63 |
+
*,
|
| 64 |
+
batch_size: int,
|
| 65 |
+
device: torch.device,
|
| 66 |
+
) -> torch.Tensor:
|
| 67 |
+
"""Event embeddings ``[events, hid_size]`` of ``indices``
|
| 68 |
+
(:func:`~gnn4colliders.jepa.event_embedding`)."""
|
| 69 |
+
parts = []
|
| 70 |
+
for part in indices.split(batch_size):
|
| 71 |
+
batch = collate_sequence_batch(events, part).to(device)
|
| 72 |
+
parts.append(event_embedding(encoder, batch.tokens, batch.token_mask).float())
|
| 73 |
+
return torch.cat(parts)
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
def knn_logits(
|
| 77 |
+
train: torch.Tensor,
|
| 78 |
+
labels: torch.Tensor,
|
| 79 |
+
queries: torch.Tensor,
|
| 80 |
+
*,
|
| 81 |
+
num_classes: int,
|
| 82 |
+
neighbours: int,
|
| 83 |
+
chunk: int = 4096,
|
| 84 |
+
) -> torch.Tensor:
|
| 85 |
+
"""Log class fractions ``[queries, num_classes]`` among each query's
|
| 86 |
+
``neighbours`` most cosine-similar ``train`` embeddings."""
|
| 87 |
+
train, queries = F.normalize(train, dim=1), F.normalize(queries, dim=1)
|
| 88 |
+
neighbours = min(neighbours, len(train))
|
| 89 |
+
one_hot = F.one_hot(labels, num_classes).float()
|
| 90 |
+
parts = []
|
| 91 |
+
for part in queries.split(chunk):
|
| 92 |
+
nearest = (part @ train.T).topk(neighbours, dim=1).indices
|
| 93 |
+
fractions = one_hot[nearest].mean(dim=1)
|
| 94 |
+
parts.append((fractions + 1e-3).log())
|
| 95 |
+
return torch.cat(parts)
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def linear_logits(
|
| 99 |
+
train: torch.Tensor,
|
| 100 |
+
labels: torch.Tensor,
|
| 101 |
+
queries: torch.Tensor,
|
| 102 |
+
*,
|
| 103 |
+
num_classes: int,
|
| 104 |
+
l2: float,
|
| 105 |
+
steps: int = 200,
|
| 106 |
+
) -> torch.Tensor:
|
| 107 |
+
"""Logits ``[queries, num_classes]`` of a multinomial logistic regression
|
| 108 |
+
fitted on the standardized ``train`` embeddings."""
|
| 109 |
+
mean, std = train.mean(dim=0), train.std(dim=0).clamp_min(1e-6)
|
| 110 |
+
x = (train - mean) / std
|
| 111 |
+
linear = nn.Linear(x.shape[1], num_classes).to(x.device)
|
| 112 |
+
optimizer = torch.optim.LBFGS(
|
| 113 |
+
linear.parameters(), max_iter=steps, line_search_fn="strong_wolfe"
|
| 114 |
+
)
|
| 115 |
+
|
| 116 |
+
def closure() -> torch.Tensor:
|
| 117 |
+
optimizer.zero_grad()
|
| 118 |
+
loss = F.cross_entropy(linear(x), labels) + l2 * linear.weight.square().sum()
|
| 119 |
+
loss.backward()
|
| 120 |
+
return loss
|
| 121 |
+
|
| 122 |
+
with torch.enable_grad():
|
| 123 |
+
optimizer.step(closure)
|
| 124 |
+
with torch.no_grad():
|
| 125 |
+
return linear((queries - mean) / std)
|
| 126 |
+
|
| 127 |
+
|
| 128 |
+
def probe(
|
| 129 |
+
config: Config,
|
| 130 |
+
events: PackedEvents,
|
| 131 |
+
train: torch.Tensor,
|
| 132 |
+
queries: torch.Tensor,
|
| 133 |
+
*,
|
| 134 |
+
num_classes: int,
|
| 135 |
+
device: torch.device,
|
| 136 |
+
) -> dict[str, Any]:
|
| 137 |
+
"""Logits ``[queries, num_classes]`` of the k-NN and linear probes on the
|
| 138 |
+
``queries`` events, given the labelled ``train`` events."""
|
| 139 |
+
encoder = load_encoder(config, events).to(device)
|
| 140 |
+
batch_size = config.data.batch_size
|
| 141 |
+
embedded = {
|
| 142 |
+
name: embed(encoder, events, indices, batch_size=batch_size, device=device)
|
| 143 |
+
for name, indices in (("train", train), ("queries", queries))
|
| 144 |
+
}
|
| 145 |
+
labels = events.labels[train].to(device, torch.long)
|
| 146 |
+
return {
|
| 147 |
+
"knn": knn_logits(
|
| 148 |
+
embedded["train"],
|
| 149 |
+
labels,
|
| 150 |
+
embedded["queries"],
|
| 151 |
+
num_classes=num_classes,
|
| 152 |
+
neighbours=config.model.probe_neighbours,
|
| 153 |
+
),
|
| 154 |
+
"linear": linear_logits(
|
| 155 |
+
embedded["train"],
|
| 156 |
+
labels,
|
| 157 |
+
embedded["queries"],
|
| 158 |
+
num_classes=num_classes,
|
| 159 |
+
l2=config.model.probe_l2,
|
| 160 |
+
),
|
| 161 |
+
}
|
|
@@ -2,11 +2,17 @@
|
|
| 2 |
|
| 3 |
from __future__ import annotations
|
| 4 |
|
|
|
|
| 5 |
from dataclasses import dataclass
|
| 6 |
|
| 7 |
import torch
|
| 8 |
|
| 9 |
-
from .data import PackedEvents, map_tensors
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
|
| 11 |
|
| 12 |
@dataclass(frozen=True)
|
|
@@ -49,3 +55,23 @@ def collate_sequence_batch(
|
|
| 49 |
else events.global_features[indices],
|
| 50 |
event_index=indices,
|
| 51 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
from __future__ import annotations
|
| 4 |
|
| 5 |
+
import math
|
| 6 |
from dataclasses import dataclass
|
| 7 |
|
| 8 |
import torch
|
| 9 |
|
| 10 |
+
from .data import NODE_FEATURE_NAMES, PackedEvents, map_tensors
|
| 11 |
+
|
| 12 |
+
PT, ETA, PHI, ENERGY, BTAG, CHARGE = (
|
| 13 |
+
NODE_FEATURE_NAMES.index(name)
|
| 14 |
+
for name in ("pt", "eta", "phi", "energy", "btag", "charge")
|
| 15 |
+
)
|
| 16 |
|
| 17 |
|
| 18 |
@dataclass(frozen=True)
|
|
|
|
| 55 |
else events.global_features[indices],
|
| 56 |
event_index=indices,
|
| 57 |
)
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def augment_tokens(
|
| 61 |
+
tokens: torch.Tensor,
|
| 62 |
+
mask: torch.Tensor,
|
| 63 |
+
*,
|
| 64 |
+
scale: torch.Tensor,
|
| 65 |
+
rotation: torch.Tensor,
|
| 66 |
+
flip: torch.Tensor,
|
| 67 |
+
) -> torch.Tensor:
|
| 68 |
+
"""``tokens`` with each event's momenta and energies scaled, azimuths
|
| 69 |
+
rotated and pseudorapidities multiplied by ``flip`` (1 or -1), all
|
| 70 |
+
``[events]``; padding is left unchanged."""
|
| 71 |
+
augmented = tokens.clone()
|
| 72 |
+
for column in (PT, ENERGY):
|
| 73 |
+
augmented[..., column] *= scale[:, None]
|
| 74 |
+
augmented[..., ETA] *= flip[:, None]
|
| 75 |
+
phi = augmented[..., PHI] + rotation[:, None]
|
| 76 |
+
augmented[..., PHI] = torch.remainder(phi + math.pi, 2 * math.pi) - math.pi
|
| 77 |
+
return torch.where(mask.unsqueeze(-1), augmented, tokens)
|
|
@@ -1,44 +1,125 @@
|
|
| 1 |
-
"""
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
"""
|
| 15 |
|
| 16 |
from __future__ import annotations
|
| 17 |
|
|
|
|
| 18 |
import math
|
|
|
|
|
|
|
| 19 |
from collections.abc import Callable
|
|
|
|
| 20 |
|
| 21 |
import torch
|
| 22 |
import torch.nn.functional as F
|
|
|
|
|
|
|
| 23 |
|
| 24 |
from .config import ModelConfig
|
| 25 |
from .data import NODE_FEATURE_NAMES, PackedEvents
|
| 26 |
-
from .
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
NODE_TYPE = NODE_FEATURE_NAMES.index("node_type")
|
| 29 |
-
#: Log-probability of the classes
|
| 30 |
MIN_LOG_PROBABILITY = math.log(1e-12)
|
| 31 |
|
| 32 |
|
| 33 |
-
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
) -> torch.Tensor:
|
| 36 |
-
"""Rows ``[events,
|
|
|
|
| 37 |
if NODE_TYPE != len(NODE_FEATURE_NAMES) - 1:
|
| 38 |
-
raise ValueError("
|
| 39 |
-
|
| 40 |
-
count, types = len(
|
| 41 |
-
event = torch.repeat_interleave(
|
|
|
|
|
|
|
| 42 |
node_type = rows[:, NODE_TYPE].long()
|
| 43 |
one_hot = F.one_hot(node_type, types)
|
| 44 |
# Index of every object among its event's objects of the same type.
|
|
@@ -46,13 +127,130 @@ def table_features(
|
|
| 46 |
first = counts.cumsum(0) - counts
|
| 47 |
before = (seen - one_hot)[first[event]]
|
| 48 |
rank = (seen - before).gather(1, node_type[:, None]).squeeze(1) - 1
|
| 49 |
-
capacity = torch.tensor(slots)
|
| 50 |
offset = capacity.cumsum(0) - capacity
|
| 51 |
keep = rank < capacity[node_type]
|
| 52 |
-
table = torch.full((count, sum(slots), NODE_TYPE), math.nan)
|
| 53 |
table[event[keep], (offset[node_type] + rank)[keep]] = rows[keep, :NODE_TYPE]
|
| 54 |
-
|
| 55 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
|
| 57 |
|
| 58 |
def tabpfn_forward(
|
|
@@ -62,6 +260,7 @@ def tabpfn_forward(
|
|
| 62 |
*,
|
| 63 |
num_classes: int,
|
| 64 |
num_node_types: int,
|
|
|
|
| 65 |
device: torch.device,
|
| 66 |
seed: int,
|
| 67 |
) -> Callable[[SequenceBatch], torch.Tensor]:
|
|
@@ -72,8 +271,7 @@ def tabpfn_forward(
|
|
| 72 |
from tabpfn.constants import ModelVersion
|
| 73 |
except ImportError as error:
|
| 74 |
raise ImportError("tabpfn is not installed: uv sync --extra tabpfn") from error
|
| 75 |
-
|
| 76 |
-
raise ValueError("model.slots needs one count per object type")
|
| 77 |
classifier = TabPFNClassifier.create_default_for_version(
|
| 78 |
ModelVersion(config.tabpfn_version),
|
| 79 |
n_estimators=config.tabpfn_estimators,
|
|
@@ -81,18 +279,427 @@ def tabpfn_forward(
|
|
| 81 |
random_state=seed,
|
| 82 |
# The context is processed once for all batches of events.
|
| 83 |
fit_mode="fit_with_cache",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 84 |
)
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
)
|
| 89 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
|
| 91 |
def forward(batch: SequenceBatch) -> torch.Tensor:
|
| 92 |
-
|
| 93 |
-
probability = torch.as_tensor(classifier.predict_proba(rows.numpy()))
|
| 94 |
-
logits = torch.full((len(rows), num_classes), MIN_LOG_PROBABILITY)
|
| 95 |
-
logits[:, classes] = probability.float().clamp_min(1e-12).log()
|
| 96 |
-
return logits.to(device)
|
| 97 |
|
| 98 |
return forward
|
|
|
|
| 1 |
+
"""Tabular in-context models on events as fixed-width rows: TabPFN
|
| 2 |
+
(Hollmann et al. 2025; Prior Labs), TabICL (Qu et al. 2025, 2026; Inria)
|
| 3 |
+
and TabDPT (Ma et al. 2024; Layer 6), which is trained on real tables and
|
| 4 |
+
can retrieve each query's nearest context rows.
|
| 5 |
+
|
| 6 |
+
Both classify the rows of a table in context, given labelled rows, in one
|
| 7 |
+
forward pass and without training. A row of an event (:class:`RowLayout`)
|
| 8 |
+
holds the leading ``model.slots[t]`` objects of each object type ``t`` in
|
| 9 |
+
stored (pt) order, with their six kinematic columns (``pt, eta, phi, energy,
|
| 10 |
+
btag, charge``; NaN for absent objects, which both models handle), then the
|
| 11 |
+
number of objects of each type. Optionally (``model.relative_phi``) every
|
| 12 |
+
azimuth is measured from that of the event's leading object, which removes
|
| 13 |
+
the azimuthal rotation the models cannot learn from a few events, and
|
| 14 |
+
(``model.pair_objects``) the row ends with the invariant mass and Delta R of
|
| 15 |
+
every pair of the event's leading objects. The context is
|
| 16 |
+
``data.train_events`` labelled training events.
|
| 17 |
+
|
| 18 |
+
The ``tabpfn``, ``tabicl`` and ``tabdpt`` packages are optional
|
| 19 |
+
dependencies (``uv sync --extra tabpfn --extra tabicl --extra tabdpt``).
|
| 20 |
+
TabICL's (BSD-3) and TabDPT's (Apache-2.0) weights download from Hugging
|
| 21 |
+
Face (``jingang/TabICL``, ``Layer6/TabDPT``). TabPFN-2.5 and later weights
|
| 22 |
+
are under a non-commercial license and download after a one-time login (a
|
| 23 |
+
token in ``~/.cache/tabpfn`` or ``TABPFN_TOKEN``); ``TABPFN_MODEL_CACHE_DIR``
|
| 24 |
+
sets where they are stored. A checkpoint already in that directory loads
|
| 25 |
+
without the login, so the weights can also be downloaded from Hugging Face
|
| 26 |
+
(``Prior-Labs/tabpfn_3_5``).
|
| 27 |
+
|
| 28 |
+
:class:`TabularPFN` and :class:`TabularICL` meta-train the TabPFN-3.5 and
|
| 29 |
+
TabICL networks on the same rows, like the PFN of :mod:`gnn4colliders.pfn`
|
| 30 |
+
(``task.name: pfn``, the same tasks and augmentations), and classify in
|
| 31 |
+
context with the network's own forward pass instead of the classifier (no
|
| 32 |
+
preprocessing beyond TabICL's standardization, no ensembling). By default
|
| 33 |
+
the network is frozen and its in-context blocks learn low-rank updates
|
| 34 |
+
(``model.lora_rank``).
|
| 35 |
"""
|
| 36 |
|
| 37 |
from __future__ import annotations
|
| 38 |
|
| 39 |
+
import dataclasses
|
| 40 |
import math
|
| 41 |
+
import os
|
| 42 |
+
import sys
|
| 43 |
from collections.abc import Callable
|
| 44 |
+
from typing import Any
|
| 45 |
|
| 46 |
import torch
|
| 47 |
import torch.nn.functional as F
|
| 48 |
+
from torch import nn
|
| 49 |
+
from torch.nn.utils import parametrize
|
| 50 |
|
| 51 |
from .config import ModelConfig
|
| 52 |
from .data import NODE_FEATURE_NAMES, PackedEvents
|
| 53 |
+
from .jepa import event_embedding
|
| 54 |
+
from .models import pretrained_encoder
|
| 55 |
+
from .pfn import ABSENT_LOGIT, TaskBatch
|
| 56 |
+
from .sequences import ENERGY, ETA, PHI, PT, SequenceBatch, collate_sequence_batch
|
| 57 |
|
| 58 |
NODE_TYPE = NODE_FEATURE_NAMES.index("node_type")
|
| 59 |
+
#: Log-probability of the classes a classifier gives no probability.
|
| 60 |
MIN_LOG_PROBABILITY = math.log(1e-12)
|
| 61 |
|
| 62 |
|
| 63 |
+
@dataclasses.dataclass(frozen=True)
|
| 64 |
+
class RowLayout:
|
| 65 |
+
"""What a row of an event holds (see the module documentation)."""
|
| 66 |
+
|
| 67 |
+
slots: tuple[int, ...]
|
| 68 |
+
relative_phi: bool = False
|
| 69 |
+
pair_objects: int = 0
|
| 70 |
+
|
| 71 |
+
def __post_init__(self) -> None:
|
| 72 |
+
if not 0 <= self.pair_objects <= sum(self.slots):
|
| 73 |
+
raise ValueError("model.pair_objects must be between 0 and sum(slots)")
|
| 74 |
+
|
| 75 |
+
@classmethod
|
| 76 |
+
def of(cls, config: ModelConfig) -> RowLayout:
|
| 77 |
+
return cls(tuple(config.slots), config.relative_phi, config.pair_objects)
|
| 78 |
+
|
| 79 |
+
@property
|
| 80 |
+
def width(self) -> int:
|
| 81 |
+
pairs = self.pair_objects * (self.pair_objects - 1) // 2
|
| 82 |
+
return sum(self.slots) * NODE_TYPE + len(self.slots) + 2 * pairs
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
def _wrap(phi: torch.Tensor) -> torch.Tensor:
|
| 86 |
+
return torch.remainder(phi + math.pi, 2 * math.pi) - math.pi
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def _pair_features(table: torch.Tensor, objects: int) -> torch.Tensor:
|
| 90 |
+
"""Invariant mass and Delta R ``[events, 2 * pairs]`` of every pair of
|
| 91 |
+
the ``objects`` leading slots of ``table [events, slots, columns]`` by pt
|
| 92 |
+
(NaN for pairs with an absent object)."""
|
| 93 |
+
order = table[..., PT].nan_to_num(-1.0).argsort(dim=1, descending=True)
|
| 94 |
+
leading = table.gather(1, order[:, :objects, None].expand(-1, -1, NODE_TYPE))
|
| 95 |
+
first, second = torch.triu_indices(objects, objects, 1, device=table.device)
|
| 96 |
+
a, b = leading[:, first], leading[:, second]
|
| 97 |
+
|
| 98 |
+
def momentum(x: torch.Tensor) -> torch.Tensor:
|
| 99 |
+
pt, eta, phi = x[..., PT], x[..., ETA], x[..., PHI]
|
| 100 |
+
return torch.stack(
|
| 101 |
+
(pt * torch.cos(phi), pt * torch.sin(phi), pt * torch.sinh(eta)), dim=-1
|
| 102 |
+
)
|
| 103 |
+
|
| 104 |
+
energy = a[..., ENERGY] + b[..., ENERGY]
|
| 105 |
+
squared = energy.square() - (momentum(a) + momentum(b)).square().sum(dim=-1)
|
| 106 |
+
mass = squared.clamp_min(0).sqrt()
|
| 107 |
+
delta_r = torch.hypot(a[..., ETA] - b[..., ETA], _wrap(a[..., PHI] - b[..., PHI]))
|
| 108 |
+
return torch.cat((mass, delta_r), dim=1)
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def object_table(
|
| 112 |
+
rows: torch.Tensor, counts: torch.Tensor, layout: RowLayout
|
| 113 |
) -> torch.Tensor:
|
| 114 |
+
"""Rows ``[events, layout.width]`` of the events with ``counts``
|
| 115 |
+
consecutive object ``rows`` each."""
|
| 116 |
if NODE_TYPE != len(NODE_FEATURE_NAMES) - 1:
|
| 117 |
+
raise ValueError("object_table expects node_type as the last column")
|
| 118 |
+
slots = layout.slots
|
| 119 |
+
device, count, types = rows.device, len(counts), len(slots)
|
| 120 |
+
event = torch.repeat_interleave(
|
| 121 |
+
torch.arange(count, device=device), counts, output_size=len(rows)
|
| 122 |
+
)
|
| 123 |
node_type = rows[:, NODE_TYPE].long()
|
| 124 |
one_hot = F.one_hot(node_type, types)
|
| 125 |
# Index of every object among its event's objects of the same type.
|
|
|
|
| 127 |
first = counts.cumsum(0) - counts
|
| 128 |
before = (seen - one_hot)[first[event]]
|
| 129 |
rank = (seen - before).gather(1, node_type[:, None]).squeeze(1) - 1
|
| 130 |
+
capacity = torch.tensor(slots, device=device)
|
| 131 |
offset = capacity.cumsum(0) - capacity
|
| 132 |
keep = rank < capacity[node_type]
|
| 133 |
+
table = torch.full((count, sum(slots), NODE_TYPE), math.nan, device=device)
|
| 134 |
table[event[keep], (offset[node_type] + rank)[keep]] = rows[keep, :NODE_TYPE]
|
| 135 |
+
if layout.relative_phi:
|
| 136 |
+
leading = table[..., PT].nan_to_num(-1.0).argmax(dim=1)
|
| 137 |
+
reference = table[torch.arange(count, device=device), leading, PHI]
|
| 138 |
+
table[..., PHI] = _wrap(table[..., PHI] - reference[:, None])
|
| 139 |
+
multiplicity = torch.zeros(count, types, device=device)
|
| 140 |
+
multiplicity.index_add_(0, event, one_hot.float())
|
| 141 |
+
parts = [table.flatten(1), multiplicity]
|
| 142 |
+
if layout.pair_objects > 1:
|
| 143 |
+
parts.append(_pair_features(table, layout.pair_objects))
|
| 144 |
+
return torch.cat(parts, dim=1)
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
def table_features(
|
| 148 |
+
events: PackedEvents, indices: torch.Tensor, layout: RowLayout
|
| 149 |
+
) -> torch.Tensor:
|
| 150 |
+
"""Rows ``[events, layout.width]`` of ``indices``."""
|
| 151 |
+
return object_table(*events.gather_objects(indices), layout)
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
def sequence_table(batch: SequenceBatch, layout: RowLayout) -> torch.Tensor:
|
| 155 |
+
"""The rows of :func:`table_features` of a (possibly augmented) batch."""
|
| 156 |
+
mask = batch.token_mask
|
| 157 |
+
return object_table(batch.tokens[mask], mask.sum(dim=1), layout)
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
class EventInputs(nn.Module):
|
| 161 |
+
"""The inputs ``[events, columns]`` of a tabular model: each event's row
|
| 162 |
+
(:class:`RowLayout`), its embedding by an object ``encoder``
|
| 163 |
+
(:func:`~gnn4colliders.jepa.event_embedding`), or both
|
| 164 |
+
(``model.tabular_inputs``: ``rows``, ``embeddings`` or ``both``). The
|
| 165 |
+
encoder is that of ``model.backbone`` with the weights of
|
| 166 |
+
``model.pretrained`` (randomly initialized without), trained with the
|
| 167 |
+
LoRA updates in meta-training unless ``model.freeze_backbone``."""
|
| 168 |
+
|
| 169 |
+
def __init__(
|
| 170 |
+
self, layout: RowLayout, encoder: nn.Module | None, *, rows: bool
|
| 171 |
+
) -> None:
|
| 172 |
+
super().__init__()
|
| 173 |
+
if not rows and encoder is None:
|
| 174 |
+
raise ValueError("tabular inputs need rows or an encoder")
|
| 175 |
+
self.layout, self.encoder, self.rows = layout, encoder, rows
|
| 176 |
+
|
| 177 |
+
@classmethod
|
| 178 |
+
def of(
|
| 179 |
+
cls, config: ModelConfig, *, num_features: int, num_node_types: int
|
| 180 |
+
) -> EventInputs:
|
| 181 |
+
if len(config.slots) != num_node_types:
|
| 182 |
+
raise ValueError("model.slots needs one count per object type")
|
| 183 |
+
inputs = config.tabular_inputs
|
| 184 |
+
if inputs not in ("rows", "embeddings", "both"):
|
| 185 |
+
raise ValueError(
|
| 186 |
+
f"unknown tabular_inputs {inputs!r}; use rows, embeddings or both"
|
| 187 |
+
)
|
| 188 |
+
encoder = None
|
| 189 |
+
if inputs != "rows":
|
| 190 |
+
encoder = pretrained_encoder(
|
| 191 |
+
config, num_features=num_features, num_node_types=num_node_types
|
| 192 |
+
)
|
| 193 |
+
return cls(RowLayout.of(config), encoder, rows=inputs != "embeddings")
|
| 194 |
+
|
| 195 |
+
def forward(self, batch: SequenceBatch) -> torch.Tensor:
|
| 196 |
+
parts = []
|
| 197 |
+
if self.rows:
|
| 198 |
+
parts.append(sequence_table(batch, self.layout))
|
| 199 |
+
if self.encoder is not None:
|
| 200 |
+
mask = batch.token_mask
|
| 201 |
+
parts.append(event_embedding(self.encoder, batch.tokens, mask).float())
|
| 202 |
+
return torch.cat(parts, dim=1)
|
| 203 |
+
|
| 204 |
+
def of_events(
|
| 205 |
+
self,
|
| 206 |
+
events: PackedEvents,
|
| 207 |
+
indices: torch.Tensor,
|
| 208 |
+
*,
|
| 209 |
+
batch_size: int,
|
| 210 |
+
device: torch.device,
|
| 211 |
+
) -> torch.Tensor:
|
| 212 |
+
"""The inputs of the events ``indices``, on ``device``."""
|
| 213 |
+
return torch.cat(
|
| 214 |
+
[
|
| 215 |
+
self(collate_sequence_batch(events, part).to(device))
|
| 216 |
+
for part in indices.split(batch_size)
|
| 217 |
+
]
|
| 218 |
+
)
|
| 219 |
+
|
| 220 |
+
|
| 221 |
+
def _classifier_forward(
|
| 222 |
+
classifier: Any,
|
| 223 |
+
inputs: EventInputs,
|
| 224 |
+
events: PackedEvents,
|
| 225 |
+
context: torch.Tensor,
|
| 226 |
+
*,
|
| 227 |
+
num_classes: int,
|
| 228 |
+
batch_size: int,
|
| 229 |
+
device: torch.device,
|
| 230 |
+
predict: Callable[[Any], Any] | None = None,
|
| 231 |
+
) -> Callable[[SequenceBatch], torch.Tensor]:
|
| 232 |
+
"""Logits (log-probabilities) ``[events, num_classes]`` of a batch by a
|
| 233 |
+
scikit-learn ``classifier`` fitted on the ``inputs`` of the labelled
|
| 234 |
+
``context``; its probabilities are ``predict(inputs)`` (default
|
| 235 |
+
``predict_proba``)."""
|
| 236 |
+
predict = predict or classifier.predict_proba
|
| 237 |
+
inputs = inputs.to(device).eval()
|
| 238 |
+
# The context's classes, numbered 0, 1, ... for the classifier.
|
| 239 |
+
labels = events.labels[context].long()
|
| 240 |
+
classes = labels.unique()
|
| 241 |
+
with torch.no_grad():
|
| 242 |
+
rows = inputs.of_events(events, context, batch_size=batch_size, device=device)
|
| 243 |
+
classifier.fit(rows.cpu().numpy(), torch.searchsorted(classes, labels).numpy())
|
| 244 |
+
|
| 245 |
+
def forward(batch: SequenceBatch) -> torch.Tensor:
|
| 246 |
+
with torch.no_grad():
|
| 247 |
+
rows = inputs(batch).cpu()
|
| 248 |
+
probability = torch.as_tensor(predict(rows.numpy()))
|
| 249 |
+
logits = torch.full((len(rows), num_classes), MIN_LOG_PROBABILITY)
|
| 250 |
+
logits[:, classes] = probability.float().clamp_min(1e-12).log()
|
| 251 |
+
return logits.to(device)
|
| 252 |
+
|
| 253 |
+
return forward
|
| 254 |
|
| 255 |
|
| 256 |
def tabpfn_forward(
|
|
|
|
| 260 |
*,
|
| 261 |
num_classes: int,
|
| 262 |
num_node_types: int,
|
| 263 |
+
batch_size: int,
|
| 264 |
device: torch.device,
|
| 265 |
seed: int,
|
| 266 |
) -> Callable[[SequenceBatch], torch.Tensor]:
|
|
|
|
| 271 |
from tabpfn.constants import ModelVersion
|
| 272 |
except ImportError as error:
|
| 273 |
raise ImportError("tabpfn is not installed: uv sync --extra tabpfn") from error
|
| 274 |
+
checkpoint = config.tabpfn_checkpoint
|
|
|
|
| 275 |
classifier = TabPFNClassifier.create_default_for_version(
|
| 276 |
ModelVersion(config.tabpfn_version),
|
| 277 |
n_estimators=config.tabpfn_estimators,
|
|
|
|
| 279 |
random_state=seed,
|
| 280 |
# The context is processed once for all batches of events.
|
| 281 |
fit_mode="fit_with_cache",
|
| 282 |
+
**({} if checkpoint is None else {"model_path": checkpoint}),
|
| 283 |
+
)
|
| 284 |
+
return _classifier_forward(
|
| 285 |
+
classifier,
|
| 286 |
+
EventInputs.of(
|
| 287 |
+
config, num_features=events.num_features, num_node_types=num_node_types
|
| 288 |
+
),
|
| 289 |
+
events,
|
| 290 |
+
context,
|
| 291 |
+
num_classes=num_classes,
|
| 292 |
+
batch_size=batch_size,
|
| 293 |
+
device=device,
|
| 294 |
+
)
|
| 295 |
+
|
| 296 |
+
|
| 297 |
+
def tabicl_forward(
|
| 298 |
+
config: ModelConfig,
|
| 299 |
+
events: PackedEvents,
|
| 300 |
+
context: torch.Tensor,
|
| 301 |
+
*,
|
| 302 |
+
num_classes: int,
|
| 303 |
+
num_node_types: int,
|
| 304 |
+
batch_size: int,
|
| 305 |
+
device: torch.device,
|
| 306 |
+
seed: int,
|
| 307 |
+
) -> Callable[[SequenceBatch], torch.Tensor]:
|
| 308 |
+
"""A function from a batch of events to their logits (log-probabilities)
|
| 309 |
+
``[events, num_classes]`` by TabICL given the labelled ``context``."""
|
| 310 |
+
try:
|
| 311 |
+
from tabicl import TabICLClassifier
|
| 312 |
+
except ImportError as error:
|
| 313 |
+
raise ImportError("tabicl is not installed: uv sync --extra tabicl") from error
|
| 314 |
+
classifier = TabICLClassifier(
|
| 315 |
+
n_estimators=config.tabicl_estimators,
|
| 316 |
+
checkpoint_version=config.tabicl_checkpoint,
|
| 317 |
+
device=str(device),
|
| 318 |
+
random_state=seed,
|
| 319 |
+
# The context is processed once for all batches of events.
|
| 320 |
+
kv_cache=True,
|
| 321 |
+
)
|
| 322 |
+
return _classifier_forward(
|
| 323 |
+
classifier,
|
| 324 |
+
EventInputs.of(
|
| 325 |
+
config, num_features=events.num_features, num_node_types=num_node_types
|
| 326 |
+
),
|
| 327 |
+
events,
|
| 328 |
+
context,
|
| 329 |
+
num_classes=num_classes,
|
| 330 |
+
batch_size=batch_size,
|
| 331 |
+
device=device,
|
| 332 |
+
)
|
| 333 |
+
|
| 334 |
+
|
| 335 |
+
def tabdpt_forward(
|
| 336 |
+
config: ModelConfig,
|
| 337 |
+
events: PackedEvents,
|
| 338 |
+
context: torch.Tensor,
|
| 339 |
+
*,
|
| 340 |
+
num_classes: int,
|
| 341 |
+
num_node_types: int,
|
| 342 |
+
batch_size: int,
|
| 343 |
+
device: torch.device,
|
| 344 |
+
seed: int,
|
| 345 |
+
) -> Callable[[SequenceBatch], torch.Tensor]:
|
| 346 |
+
"""A function from a batch of events to their logits (log-probabilities)
|
| 347 |
+
``[events, num_classes]`` by TabDPT given the labelled ``context``; with
|
| 348 |
+
``model.tabdpt_context_size`` each query's context is its nearest rows.
|
| 349 |
+
|
| 350 |
+
TabDPT computes in float64 in places, so on MPS it runs on the CPU. Its
|
| 351 |
+
retrieval (faiss) and PyTorch load separate OpenMP runtimes on macOS,
|
| 352 |
+
which crash together unless ``OMP_NUM_THREADS=1``."""
|
| 353 |
+
try:
|
| 354 |
+
from tabdpt import TabDPTClassifier
|
| 355 |
+
except ImportError as error:
|
| 356 |
+
raise ImportError("tabdpt is not installed: uv sync --extra tabdpt") from error
|
| 357 |
+
size = config.tabdpt_context_size
|
| 358 |
+
if size is not None and sys.platform == "darwin":
|
| 359 |
+
if os.environ.get("OMP_NUM_THREADS") != "1":
|
| 360 |
+
raise ValueError(
|
| 361 |
+
"TabDPT retrieval on macOS needs OMP_NUM_THREADS=1 (faiss and "
|
| 362 |
+
"torch load separate OpenMP runtimes)"
|
| 363 |
+
)
|
| 364 |
+
classifier = TabDPTClassifier(
|
| 365 |
+
# Absent objects are NaN: imputed, with indicator columns.
|
| 366 |
+
missing_indicators=True,
|
| 367 |
+
context_reduction="subsample" if size is None else "retrieval",
|
| 368 |
+
device="cpu" if device.type == "mps" else str(device),
|
| 369 |
+
verbose=False,
|
| 370 |
)
|
| 371 |
+
|
| 372 |
+
def predict(rows: Any) -> Any:
|
| 373 |
+
return classifier.ensemble_predict_proba(
|
| 374 |
+
rows, n_ensembles=config.tabdpt_estimators, context_size=size, seed=seed
|
| 375 |
+
)
|
| 376 |
+
|
| 377 |
+
return _classifier_forward(
|
| 378 |
+
classifier,
|
| 379 |
+
EventInputs.of(
|
| 380 |
+
config, num_features=events.num_features, num_node_types=num_node_types
|
| 381 |
+
),
|
| 382 |
+
events,
|
| 383 |
+
context,
|
| 384 |
+
num_classes=num_classes,
|
| 385 |
+
batch_size=batch_size,
|
| 386 |
+
device=device,
|
| 387 |
+
predict=predict,
|
| 388 |
)
|
| 389 |
+
|
| 390 |
+
|
| 391 |
+
# --- Meta-training of the tabular networks ------------------------------------
|
| 392 |
+
|
| 393 |
+
|
| 394 |
+
def load_tabpfn_network(version: str, checkpoint: str | None) -> nn.Module:
|
| 395 |
+
"""The TabPFN-3.5 network of ``model.tabpfn_version`` and
|
| 396 |
+
``model.tabpfn_checkpoint`` (default: that version's)."""
|
| 397 |
+
try:
|
| 398 |
+
from tabpfn.model_loading import load_model_criterion_config
|
| 399 |
+
except ImportError as error:
|
| 400 |
+
raise ImportError("tabpfn is not installed: uv sync --extra tabpfn") from error
|
| 401 |
+
if version not in ("v3.5", "v3.5-fast"):
|
| 402 |
+
raise ValueError("TabularPFN trains a TabPFN-3.5 network (v3.5, v3.5-fast)")
|
| 403 |
+
models, *_ = load_model_criterion_config(
|
| 404 |
+
checkpoint,
|
| 405 |
+
check_bar_distribution_criterion=False,
|
| 406 |
+
estimator_type="classifier",
|
| 407 |
+
version=version,
|
| 408 |
+
download_if_not_exists=True,
|
| 409 |
+
)
|
| 410 |
+
return models[0]
|
| 411 |
+
|
| 412 |
+
|
| 413 |
+
def load_tabicl_network(checkpoint: str) -> nn.Module:
|
| 414 |
+
"""The TabICL network of ``model.tabicl_checkpoint`` (a path, or a file
|
| 415 |
+
name in the Hugging Face repository ``jingang/TabICL``), recomputing
|
| 416 |
+
activations in the backward pass (gradient checkpointing)."""
|
| 417 |
+
try:
|
| 418 |
+
from huggingface_hub import hf_hub_download
|
| 419 |
+
from tabicl._model.tabicl import TabICL
|
| 420 |
+
except ImportError as error:
|
| 421 |
+
raise ImportError("tabicl is not installed: uv sync --extra tabicl") from error
|
| 422 |
+
path = checkpoint if os.path.exists(checkpoint) else None
|
| 423 |
+
path = path or hf_hub_download(repo_id="jingang/TabICL", filename=checkpoint)
|
| 424 |
+
saved = torch.load(path, map_location="cpu", weights_only=True)
|
| 425 |
+
network = TabICL(**{**saved["config"], "recompute": True})
|
| 426 |
+
network.load_state_dict(saved["state_dict"])
|
| 427 |
+
return network
|
| 428 |
+
|
| 429 |
+
|
| 430 |
+
class LoRA(nn.Module):
|
| 431 |
+
"""A parametrization that adds a trainable low-rank update ``up @ down``
|
| 432 |
+
(LoRA; Hu et al. 2022), zero at initialization, to a frozen weight. As a
|
| 433 |
+
parametrization it also reaches weights the code reads directly, such as
|
| 434 |
+
TabICL's fused attention projections."""
|
| 435 |
+
|
| 436 |
+
def __init__(self, weight: torch.Tensor, rank: int) -> None:
|
| 437 |
+
super().__init__()
|
| 438 |
+
out_features, in_features = weight.shape
|
| 439 |
+
down = torch.randn(rank, in_features) / math.sqrt(in_features)
|
| 440 |
+
self.down = nn.Parameter(down.to(weight))
|
| 441 |
+
self.up = nn.Parameter(weight.new_zeros(out_features, rank))
|
| 442 |
+
|
| 443 |
+
def forward(self, weight: torch.Tensor) -> torch.Tensor:
|
| 444 |
+
return weight + (self.up @ self.down).to(weight.dtype)
|
| 445 |
+
|
| 446 |
+
|
| 447 |
+
def add_lora(network: nn.Module, weights: list[str], rank: int) -> None:
|
| 448 |
+
"""Freeze ``network`` and give its ``weights`` (dotted names) LoRA
|
| 449 |
+
updates of ``rank``."""
|
| 450 |
+
network.requires_grad_(False)
|
| 451 |
+
for name in weights:
|
| 452 |
+
owner_name, _, attribute = name.rpartition(".")
|
| 453 |
+
owner = network.get_submodule(owner_name)
|
| 454 |
+
weight = getattr(owner, attribute).detach()
|
| 455 |
+
parametrize.register_parametrization(owner, attribute, LoRA(weight, rank))
|
| 456 |
+
|
| 457 |
+
|
| 458 |
+
def merge_lora(network: nn.Module) -> None:
|
| 459 |
+
"""Fold the LoRA updates into the weights, for inference (the network
|
| 460 |
+
can no longer be trained or saved as LoRA updates)."""
|
| 461 |
+
for module in list(network.modules()):
|
| 462 |
+
if parametrize.is_parametrized(module):
|
| 463 |
+
for name in list(module.parametrizations):
|
| 464 |
+
parametrize.remove_parametrizations(
|
| 465 |
+
module, name, leave_parametrized=True
|
| 466 |
+
)
|
| 467 |
+
|
| 468 |
+
|
| 469 |
+
class MetaTabular(nn.Module):
|
| 470 |
+
"""Query logits ``[tasks * queries, max_classes]`` of a
|
| 471 |
+
:class:`~gnn4colliders.pfn.TaskBatch` by a tabular in-context
|
| 472 |
+
``network`` on the events' :class:`EventInputs` (interface of
|
| 473 |
+
:class:`~gnn4colliders.pfn.EventPFN`); the logits of classes a task does
|
| 474 |
+
not have are ``ABSENT_LOGIT``.
|
| 475 |
+
|
| 476 |
+
With ``lora_rank > 0`` only the LoRA updates of the network's
|
| 477 |
+
:attr:`lora_weights` train (:func:`add_lora`), else all weights do; an
|
| 478 |
+
encoder of the inputs trains too unless frozen. The state dict holds only
|
| 479 |
+
the trainable weights (and buffers): the rest are those of the released
|
| 480 |
+
network and of ``model.pretrained``.
|
| 481 |
+
"""
|
| 482 |
+
|
| 483 |
+
def __init__(
|
| 484 |
+
self,
|
| 485 |
+
network: nn.Module,
|
| 486 |
+
*,
|
| 487 |
+
inputs: EventInputs,
|
| 488 |
+
max_classes: int,
|
| 489 |
+
lora_rank: int,
|
| 490 |
+
) -> None:
|
| 491 |
+
super().__init__()
|
| 492 |
+
self.network, self.inputs, self.max_classes = network, inputs, max_classes
|
| 493 |
+
if lora_rank > 0:
|
| 494 |
+
add_lora(network, self.lora_weights(), lora_rank)
|
| 495 |
+
self.register_state_dict_post_hook(_drop_frozen)
|
| 496 |
+
self.register_load_state_dict_post_hook(_allow_missing_frozen)
|
| 497 |
+
|
| 498 |
+
def lora_weights(self) -> list[str]:
|
| 499 |
+
"""The weights of the network's in-context blocks that LoRA updates."""
|
| 500 |
+
raise NotImplementedError
|
| 501 |
+
|
| 502 |
+
def task_logits(self, rows: torch.Tensor, labels: torch.Tensor) -> torch.Tensor:
|
| 503 |
+
"""Logits ``[tasks, queries, classes]`` of tasks of ``rows [tasks,
|
| 504 |
+
context + queries, columns]`` given the context ``labels [tasks,
|
| 505 |
+
context]``."""
|
| 506 |
+
raise NotImplementedError
|
| 507 |
+
|
| 508 |
+
def in_context(
|
| 509 |
+
self, rows: torch.Tensor, labels: torch.Tensor
|
| 510 |
+
) -> Callable[[torch.Tensor], torch.Tensor]:
|
| 511 |
+
"""A function from query rows to their logits ``[queries, classes]``
|
| 512 |
+
given the context ``rows`` and ``labels`` (one task, computed once)."""
|
| 513 |
+
raise NotImplementedError
|
| 514 |
+
|
| 515 |
+
def frozen(self) -> set[str]:
|
| 516 |
+
return {
|
| 517 |
+
name for name, value in self.named_parameters() if not value.requires_grad
|
| 518 |
+
}
|
| 519 |
+
|
| 520 |
+
def forward(self, batch: TaskBatch) -> torch.Tensor:
|
| 521 |
+
tasks = batch.context_labels.shape[0]
|
| 522 |
+
rows = self.inputs(batch.events)
|
| 523 |
+
rows = rows.view(tasks, -1, rows.shape[1])
|
| 524 |
+
logits = self.task_logits(rows, batch.context_labels)
|
| 525 |
+
logits = logits[..., : self.max_classes].float()
|
| 526 |
+
classes = torch.arange(self.max_classes, device=logits.device)
|
| 527 |
+
absent = classes >= batch.num_classes.reshape(-1, 1, 1)
|
| 528 |
+
return logits.masked_fill(absent, ABSENT_LOGIT).flatten(0, 1)
|
| 529 |
+
|
| 530 |
+
|
| 531 |
+
class TabularPFN(MetaTabular):
|
| 532 |
+
""":class:`MetaTabular` of the TabPFN-3.5 network: LoRA updates of the
|
| 533 |
+
attention projections and MLP layers of its 24 in-context blocks."""
|
| 534 |
+
|
| 535 |
+
def lora_weights(self) -> list[str]:
|
| 536 |
+
names = []
|
| 537 |
+
for index in range(len(self.network.icl_blocks)):
|
| 538 |
+
block = f"icl_blocks.{index}"
|
| 539 |
+
names += [
|
| 540 |
+
f"{block}.icl_attention.{name}_projection.weight"
|
| 541 |
+
for name in ("q", "k", "v", "out")
|
| 542 |
+
]
|
| 543 |
+
names += [f"{block}.mlp.0.weight", f"{block}.mlp.2.weight"]
|
| 544 |
+
return names
|
| 545 |
+
|
| 546 |
+
def options(self) -> Any:
|
| 547 |
+
"""TabPFN's performance options; with gradients, activations are
|
| 548 |
+
recomputed in the backward pass (gradient checkpointing)."""
|
| 549 |
+
options = self.network.get_default_performance_options()
|
| 550 |
+
recompute = torch.is_grad_enabled()
|
| 551 |
+
return dataclasses.replace(options, force_recompute_layer=recompute)
|
| 552 |
+
|
| 553 |
+
def task_logits(self, rows: torch.Tensor, labels: torch.Tensor) -> torch.Tensor:
|
| 554 |
+
x = rows.transpose(0, 1) # TabPFN: [rows, tasks, columns]
|
| 555 |
+
y = labels.T.to(x.dtype)
|
| 556 |
+
output = self.network(x, y, "multiclass", performance_options=self.options())
|
| 557 |
+
return output.transpose(0, 1)
|
| 558 |
+
|
| 559 |
+
def in_context(
|
| 560 |
+
self, rows: torch.Tensor, labels: torch.Tensor
|
| 561 |
+
) -> Callable[[torch.Tensor], torch.Tensor]:
|
| 562 |
+
y = labels.to(rows.dtype)[:, None]
|
| 563 |
+
options = self.options()
|
| 564 |
+
_, cache = self.network(
|
| 565 |
+
rows[:, None],
|
| 566 |
+
y,
|
| 567 |
+
"multiclass",
|
| 568 |
+
performance_options=options,
|
| 569 |
+
return_kv_cache=True,
|
| 570 |
+
)
|
| 571 |
+
|
| 572 |
+
def forward(queries: torch.Tensor) -> torch.Tensor:
|
| 573 |
+
output = self.network(
|
| 574 |
+
queries[:, None],
|
| 575 |
+
y,
|
| 576 |
+
"multiclass",
|
| 577 |
+
performance_options=options,
|
| 578 |
+
kv_cache=cache,
|
| 579 |
+
x_is_test_only=True,
|
| 580 |
+
)
|
| 581 |
+
return output[:, 0]
|
| 582 |
+
|
| 583 |
+
return forward
|
| 584 |
+
|
| 585 |
+
|
| 586 |
+
#: Standardized TabICL inputs are clipped to this many standard deviations.
|
| 587 |
+
TABICL_CLIP = 8.0
|
| 588 |
+
|
| 589 |
+
|
| 590 |
+
def _standardizer(context: torch.Tensor) -> Callable[[torch.Tensor], torch.Tensor]:
|
| 591 |
+
"""Standardization of rows ``[..., rows, columns]`` by the mean and
|
| 592 |
+
standard deviation of each task's ``context`` rows, absent (NaN) values
|
| 593 |
+
at the mean, clipped at ``TABICL_CLIP``: TabICL's network takes no NaN
|
| 594 |
+
(its classifier preprocesses similarly, without ensembling)."""
|
| 595 |
+
mean = context.nanmean(dim=-2, keepdim=True)
|
| 596 |
+
std = (context - mean).square().nanmean(dim=-2, keepdim=True).sqrt()
|
| 597 |
+
mean, std = mean.nan_to_num(0.0), std.nan_to_num(1.0).clamp_min(1e-6)
|
| 598 |
+
|
| 599 |
+
def standardize(rows: torch.Tensor) -> torch.Tensor:
|
| 600 |
+
z = ((rows - mean) / std).nan_to_num(0.0)
|
| 601 |
+
return z.clamp(-TABICL_CLIP, TABICL_CLIP)
|
| 602 |
+
|
| 603 |
+
return standardize
|
| 604 |
+
|
| 605 |
+
|
| 606 |
+
class TabularICL(MetaTabular):
|
| 607 |
+
""":class:`MetaTabular` of the TabICL network: LoRA updates of the
|
| 608 |
+
attention projections and feed-forward layers of its 12 in-context
|
| 609 |
+
blocks. The rows are standardized by :func:`_standardizer`."""
|
| 610 |
+
|
| 611 |
+
def lora_weights(self) -> list[str]:
|
| 612 |
+
names = []
|
| 613 |
+
for index in range(len(self.network.icl_predictor.tf_icl.blocks)):
|
| 614 |
+
block = f"icl_predictor.tf_icl.blocks.{index}"
|
| 615 |
+
names += [
|
| 616 |
+
f"{block}.attn.in_proj_weight",
|
| 617 |
+
f"{block}.attn.out_proj.weight",
|
| 618 |
+
f"{block}.linear1.weight",
|
| 619 |
+
f"{block}.linear2.weight",
|
| 620 |
+
]
|
| 621 |
+
return names
|
| 622 |
+
|
| 623 |
+
def task_logits(self, rows: torch.Tensor, labels: torch.Tensor) -> torch.Tensor:
|
| 624 |
+
context = labels.shape[1]
|
| 625 |
+
x = _standardizer(rows[:, :context])(rows)
|
| 626 |
+
# The training forward, in validation too: its logits cover
|
| 627 |
+
# max_classes whatever the context's classes. In eval mode TabICL's
|
| 628 |
+
# modules switch to their inference path, which chooses its own
|
| 629 |
+
# device; its dropout is 0, so training mode changes nothing else.
|
| 630 |
+
training = self.network.training
|
| 631 |
+
self.network.train()
|
| 632 |
+
try:
|
| 633 |
+
return self.network._train_forward(x, labels.to(x.dtype))
|
| 634 |
+
finally:
|
| 635 |
+
self.network.train(training)
|
| 636 |
+
|
| 637 |
+
def in_context(
|
| 638 |
+
self, rows: torch.Tensor, labels: torch.Tensor
|
| 639 |
+
) -> Callable[[torch.Tensor], torch.Tensor]:
|
| 640 |
+
from tabicl import InferenceConfig
|
| 641 |
+
|
| 642 |
+
# On the model's device (TabICL picks one itself by default) and in
|
| 643 |
+
# float32, as in meta-training: mixed precision on the CPU changes
|
| 644 |
+
# the predictions substantially.
|
| 645 |
+
inference = InferenceConfig()
|
| 646 |
+
for stage in (inference.COL_CONFIG, inference.ROW_CONFIG, inference.ICL_CONFIG):
|
| 647 |
+
stage.update({"device": rows.device, "use_amp": False})
|
| 648 |
+
standardize = _standardizer(rows)
|
| 649 |
+
self.network.forward_with_cache(
|
| 650 |
+
X_train=standardize(rows)[None],
|
| 651 |
+
y_train=labels.to(rows.dtype)[None],
|
| 652 |
+
store_cache=True,
|
| 653 |
+
inference_config=inference,
|
| 654 |
+
)
|
| 655 |
+
|
| 656 |
+
def forward(queries: torch.Tensor) -> torch.Tensor:
|
| 657 |
+
logits = self.network.forward_with_cache(
|
| 658 |
+
X_test=standardize(queries)[None],
|
| 659 |
+
use_cache=True,
|
| 660 |
+
store_cache=False,
|
| 661 |
+
return_logits=True,
|
| 662 |
+
inference_config=inference,
|
| 663 |
+
)
|
| 664 |
+
return logits[0]
|
| 665 |
+
|
| 666 |
+
return forward
|
| 667 |
+
|
| 668 |
+
|
| 669 |
+
def _drop_frozen(
|
| 670 |
+
module: MetaTabular, state: dict[str, Any], prefix: str, metadata: Any
|
| 671 |
+
) -> None:
|
| 672 |
+
for name in module.frozen():
|
| 673 |
+
del state[prefix + name]
|
| 674 |
+
|
| 675 |
+
|
| 676 |
+
def _allow_missing_frozen(module: MetaTabular, keys: Any) -> None:
|
| 677 |
+
frozen = module.frozen()
|
| 678 |
+
keys.missing_keys[:] = [key for key in keys.missing_keys if key not in frozen]
|
| 679 |
+
|
| 680 |
+
|
| 681 |
+
@torch.inference_mode()
|
| 682 |
+
def meta_tabular_forward(
|
| 683 |
+
model: MetaTabular,
|
| 684 |
+
events: PackedEvents,
|
| 685 |
+
context: torch.Tensor,
|
| 686 |
+
*,
|
| 687 |
+
num_classes: int,
|
| 688 |
+
batch_size: int,
|
| 689 |
+
device: torch.device,
|
| 690 |
+
) -> Callable[[SequenceBatch], torch.Tensor]:
|
| 691 |
+
"""A function from a batch of events to their logits ``[events,
|
| 692 |
+
num_classes]``, given the labelled ``context`` events (one task). The
|
| 693 |
+
context is processed once."""
|
| 694 |
+
if num_classes > model.max_classes:
|
| 695 |
+
raise ValueError(f"the model tells apart at most {model.max_classes} classes")
|
| 696 |
+
model.eval()
|
| 697 |
+
merge_lora(model.network)
|
| 698 |
+
rows = model.inputs.of_events(events, context, batch_size=batch_size, device=device)
|
| 699 |
+
labels = events.labels[context].to(device)
|
| 700 |
+
logits = model.in_context(rows, labels)
|
| 701 |
|
| 702 |
def forward(batch: SequenceBatch) -> torch.Tensor:
|
| 703 |
+
return logits(model.inputs(batch))[:, :num_classes]
|
|
|
|
|
|
|
|
|
|
|
|
|
| 704 |
|
| 705 |
return forward
|
|
@@ -31,6 +31,13 @@ def per_class_weighted_mean(
|
|
| 31 |
return torch.where(present, means, torch.zeros_like(means)).sum() / present.sum()
|
| 32 |
|
| 33 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
def weighted_binary_auc(
|
| 35 |
labels: torch.Tensor, scores: torch.Tensor, weights: torch.Tensor
|
| 36 |
) -> float:
|
|
@@ -39,9 +46,10 @@ def weighted_binary_auc(
|
|
| 39 |
Equals the weighted fraction of (positive, negative) pairs ordered
|
| 40 |
correctly. Returns NaN without positive or negative weight.
|
| 41 |
"""
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
|
|
|
| 45 |
zero = weights.new_zeros(())
|
| 46 |
positive = torch.where(labels, weights, zero)
|
| 47 |
negative = torch.where(labels, zero, weights)
|
|
@@ -72,6 +80,7 @@ def macro_scores(
|
|
| 72 |
minlength=num_classes * num_classes,
|
| 73 |
)
|
| 74 |
.reshape(num_classes, num_classes)
|
|
|
|
| 75 |
.to(torch.float64)
|
| 76 |
)
|
| 77 |
hits, predicted, actual = confusion.diagonal(), confusion.sum(0), confusion.sum(1)
|
|
@@ -172,10 +181,14 @@ class BinaryClassificationTask:
|
|
| 172 |
}
|
| 173 |
|
| 174 |
|
| 175 |
-
class
|
| 176 |
-
"""The loss of JEPA pretraining. The model's outputs per
|
| 177 |
-
``[loss,
|
| 178 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 179 |
|
| 180 |
@staticmethod
|
| 181 |
def loss(
|
|
@@ -184,11 +197,10 @@ class JepaTask:
|
|
| 184 |
has_targets = outputs[:, 1]
|
| 185 |
return (outputs[:, 0] * has_targets).sum() / has_targets.sum().clamp_min(1)
|
| 186 |
|
| 187 |
-
@staticmethod
|
| 188 |
def metrics(
|
| 189 |
-
outputs: torch.Tensor, labels: torch.Tensor, weight: torch.Tensor
|
| 190 |
) -> dict[str, float]:
|
| 191 |
-
return {
|
| 192 |
|
| 193 |
|
| 194 |
class PfnTask(MulticlassClassificationTask):
|
|
@@ -213,7 +225,11 @@ def build_task(name: str, *, absolute_weights: bool = False, threshold: float =
|
|
| 213 |
absolute_weights=absolute_weights, threshold=threshold
|
| 214 |
)
|
| 215 |
if name == "jepa":
|
| 216 |
-
return
|
|
|
|
|
|
|
| 217 |
if name == "pfn":
|
| 218 |
return PfnTask()
|
| 219 |
-
raise ValueError(
|
|
|
|
|
|
|
|
|
| 31 |
return torch.where(present, means, torch.zeros_like(means)).sum() / present.sum()
|
| 32 |
|
| 33 |
|
| 34 |
+
def float64_device(tensor: torch.Tensor) -> torch.device:
|
| 35 |
+
"""Where float64 metrics of ``tensor`` run: the CPU for MPS, which has no
|
| 36 |
+
float64, else its own device. Move, then cast: on PyTorch 2.14 a combined
|
| 37 |
+
``.to(device="cpu", dtype=torch.float64)`` of an MPS tensor gives zeros."""
|
| 38 |
+
return torch.device("cpu") if tensor.device.type == "mps" else tensor.device
|
| 39 |
+
|
| 40 |
+
|
| 41 |
def weighted_binary_auc(
|
| 42 |
labels: torch.Tensor, scores: torch.Tensor, weights: torch.Tensor
|
| 43 |
) -> float:
|
|
|
|
| 46 |
Equals the weighted fraction of (positive, negative) pairs ordered
|
| 47 |
correctly. Returns NaN without positive or negative weight.
|
| 48 |
"""
|
| 49 |
+
device = float64_device(scores)
|
| 50 |
+
labels = labels.detach().to(device).to(torch.bool).reshape(-1)
|
| 51 |
+
scores = scores.detach().to(device).to(torch.float64).reshape(-1)
|
| 52 |
+
weights = weights.detach().to(device).to(torch.float64).reshape(-1)
|
| 53 |
zero = weights.new_zeros(())
|
| 54 |
positive = torch.where(labels, weights, zero)
|
| 55 |
negative = torch.where(labels, zero, weights)
|
|
|
|
| 80 |
minlength=num_classes * num_classes,
|
| 81 |
)
|
| 82 |
.reshape(num_classes, num_classes)
|
| 83 |
+
.to(float64_device(labels))
|
| 84 |
.to(torch.float64)
|
| 85 |
)
|
| 86 |
hits, predicted, actual = confusion.diagonal(), confusion.sum(0), confusion.sum(1)
|
|
|
|
| 181 |
}
|
| 182 |
|
| 183 |
|
| 184 |
+
class PretrainingTask:
|
| 185 |
+
"""The loss of JEPA or contrastive pretraining. The model's outputs per
|
| 186 |
+
event are ``[loss, counts, spread]`` (:class:`~gnn4colliders.jepa.JEPA`,
|
| 187 |
+
:class:`~gnn4colliders.contrastive.Contrastive`), the spread reported as
|
| 188 |
+
``spread_name``; labels and weights are not used."""
|
| 189 |
+
|
| 190 |
+
def __init__(self, spread_name: str) -> None:
|
| 191 |
+
self.spread_name = spread_name
|
| 192 |
|
| 193 |
@staticmethod
|
| 194 |
def loss(
|
|
|
|
| 197 |
has_targets = outputs[:, 1]
|
| 198 |
return (outputs[:, 0] * has_targets).sum() / has_targets.sum().clamp_min(1)
|
| 199 |
|
|
|
|
| 200 |
def metrics(
|
| 201 |
+
self, outputs: torch.Tensor, labels: torch.Tensor, weight: torch.Tensor
|
| 202 |
) -> dict[str, float]:
|
| 203 |
+
return {self.spread_name: float(outputs[:, 2].mean())}
|
| 204 |
|
| 205 |
|
| 206 |
class PfnTask(MulticlassClassificationTask):
|
|
|
|
| 225 |
absolute_weights=absolute_weights, threshold=threshold
|
| 226 |
)
|
| 227 |
if name == "jepa":
|
| 228 |
+
return PretrainingTask("target_spread")
|
| 229 |
+
if name == "contrastive":
|
| 230 |
+
return PretrainingTask("embedding_spread")
|
| 231 |
if name == "pfn":
|
| 232 |
return PfnTask()
|
| 233 |
+
raise ValueError(
|
| 234 |
+
f"unknown task {name!r}; use multiclass, binary, jepa, contrastive or pfn"
|
| 235 |
+
)
|
|
@@ -14,6 +14,17 @@ FIXTURES = ROOT / "tests/fixtures"
|
|
| 14 |
GAMGAM_FIXTURES = sorted((FIXTURES / "atlas_opendata_gamgam").glob("mc_*_64.root"))
|
| 15 |
TTH_FIXTURE = FIXTURES / "ttH_NLO_64.root"
|
| 16 |
PRETRAINED = ROOT / "Pretrained_GNN"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
|
| 19 |
def random_events(count: int = 40, *, seed: int = 0, classes: int = 3) -> PackedEvents:
|
|
@@ -28,7 +39,7 @@ def random_events(count: int = 40, *, seed: int = 0, classes: int = 3) -> Packed
|
|
| 28 |
def fixture_config(
|
| 29 |
name: str, files: list[Path], labels: list[int], tmp_path: Path, *overrides: str
|
| 30 |
) -> Config:
|
| 31 |
-
"""``configs/<name>.yaml`` on local fixture files, on
|
| 32 |
entries = [
|
| 33 |
{"path": str(path), "label": label} for path, label in zip(files, labels)
|
| 34 |
]
|
|
@@ -42,7 +53,7 @@ def fixture_config(
|
|
| 42 |
"data.loader_workers=0",
|
| 43 |
"data.pin_memory=false",
|
| 44 |
"data.batch_size=64",
|
| 45 |
-
"train.device=
|
| 46 |
f"run_dir={tmp_path}/run",
|
| 47 |
*overrides,
|
| 48 |
],
|
|
|
|
| 14 |
GAMGAM_FIXTURES = sorted((FIXTURES / "atlas_opendata_gamgam").glob("mc_*_64.root"))
|
| 15 |
TTH_FIXTURE = FIXTURES / "ttH_NLO_64.root"
|
| 16 |
PRETRAINED = ROOT / "Pretrained_GNN"
|
| 17 |
+
#: ``train.device`` of the fixture runs: ``pytest --device mps`` (default cpu).
|
| 18 |
+
DEVICE = "cpu"
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def pytest_addoption(parser: pytest.Parser) -> None:
|
| 22 |
+
parser.addoption("--device", default="cpu", help="train.device of fixture runs")
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def pytest_configure(config: pytest.Config) -> None:
|
| 26 |
+
global DEVICE
|
| 27 |
+
DEVICE = config.getoption("--device")
|
| 28 |
|
| 29 |
|
| 30 |
def random_events(count: int = 40, *, seed: int = 0, classes: int = 3) -> PackedEvents:
|
|
|
|
| 39 |
def fixture_config(
|
| 40 |
name: str, files: list[Path], labels: list[int], tmp_path: Path, *overrides: str
|
| 41 |
) -> Config:
|
| 42 |
+
"""``configs/<name>.yaml`` on local fixture files, on ``DEVICE``, in tmp_path."""
|
| 43 |
entries = [
|
| 44 |
{"path": str(path), "label": label} for path, label in zip(files, labels)
|
| 45 |
]
|
|
|
|
| 53 |
"data.loader_workers=0",
|
| 54 |
"data.pin_memory=false",
|
| 55 |
"data.batch_size=64",
|
| 56 |
+
f"train.device={DEVICE}",
|
| 57 |
f"run_dir={tmp_path}/run",
|
| 58 |
*overrides,
|
| 59 |
],
|
|
@@ -0,0 +1,123 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Contrastive pretraining of both backbones: the random views, the NT-Xent
|
| 2 |
+
loss, and pretraining then fine-tuning and probing."""
|
| 3 |
+
|
| 4 |
+
import json
|
| 5 |
+
import math
|
| 6 |
+
|
| 7 |
+
import pytest
|
| 8 |
+
import torch
|
| 9 |
+
from conftest import GAMGAM_FIXTURES, TTH_FIXTURE, fixture_config, random_events
|
| 10 |
+
|
| 11 |
+
from gnn4colliders.cli import main
|
| 12 |
+
from gnn4colliders.config import ModelConfig, save_config
|
| 13 |
+
from gnn4colliders.contrastive import random_view
|
| 14 |
+
from gnn4colliders.models import build_model
|
| 15 |
+
from gnn4colliders.sequences import ETA, PHI, PT, collate_sequence_batch
|
| 16 |
+
|
| 17 |
+
BACKBONES = {
|
| 18 |
+
"transformer": {"hid_size": 16, "n_layers": 1, "n_heads": 2, "pair_bias": True},
|
| 19 |
+
"edge_network": {"hid_size": 16, "n_layers": 2, "n_proc_steps": 2, "n_heads": 2},
|
| 20 |
+
}
|
| 21 |
+
backbones = pytest.mark.parametrize("backbone", BACKBONES)
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def _batch():
|
| 25 |
+
events = random_events()
|
| 26 |
+
events.object_features[:, 6] = torch.randint(
|
| 27 |
+
0, 5, (events.object_features.shape[0],)
|
| 28 |
+
).float()
|
| 29 |
+
return collate_sequence_batch(events, torch.arange(16))
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def test_views_rotate_flip_scale_and_drop_objects_but_keep_the_first():
|
| 33 |
+
batch = _batch()
|
| 34 |
+
tokens, mask = batch.tokens, batch.token_mask
|
| 35 |
+
generator = torch.Generator().manual_seed(0)
|
| 36 |
+
view, kept = random_view(
|
| 37 |
+
tokens, mask, scale_jitter=0.2, object_dropout=0.5, generator=generator
|
| 38 |
+
)
|
| 39 |
+
assert kept[:, 0].all() and not (kept & ~mask).any() and (mask & ~kept).any()
|
| 40 |
+
# One factor per event on pt, one sign on eta, azimuths in [-pi, pi).
|
| 41 |
+
ratio = view[..., PT][mask] / tokens[..., PT][mask]
|
| 42 |
+
assert ((ratio > 0.8 - 1e-6) & (ratio < 1.2 + 1e-6)).all()
|
| 43 |
+
sign = view[..., ETA][mask] / tokens[..., ETA][mask]
|
| 44 |
+
torch.testing.assert_close(sign.abs(), torch.ones_like(sign))
|
| 45 |
+
assert (view[..., PHI].abs() <= math.pi).all()
|
| 46 |
+
torch.testing.assert_close(view[~mask], tokens[~mask])
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
@backbones
|
| 50 |
+
def test_loss_is_finite_and_trains_every_encoder_parameter(backbone):
|
| 51 |
+
torch.manual_seed(0)
|
| 52 |
+
config = ModelConfig(name="contrastive", backbone=backbone, **BACKBONES[backbone])
|
| 53 |
+
model = build_model(config, num_features=7, num_node_types=5).train()
|
| 54 |
+
outputs = model(_batch())
|
| 55 |
+
assert outputs.shape == (16, 3) and torch.isfinite(outputs).all()
|
| 56 |
+
# A loss above log(1) and at most about log(2 * 16 - 1) at initialization.
|
| 57 |
+
assert 0 < outputs[:, 0].mean() < math.log(31) + 1
|
| 58 |
+
outputs[:, 0].mean().backward()
|
| 59 |
+
for name, parameter in model.context_encoder.named_parameters():
|
| 60 |
+
assert parameter.grad is not None and parameter.grad.abs().sum() > 0, name
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
@pytest.mark.parametrize(
|
| 64 |
+
("backbone", "pretrain_config", "finetune_config", "overrides"),
|
| 65 |
+
[
|
| 66 |
+
("transformer", "delphes_contrastive", "gamgam_jepa_finetune", []),
|
| 67 |
+
(
|
| 68 |
+
"edge_network",
|
| 69 |
+
"delphes_contrastive",
|
| 70 |
+
"gamgam_finetune",
|
| 71 |
+
["model.backbone=edge_network"],
|
| 72 |
+
),
|
| 73 |
+
("transformer", "delphes_masked", "gamgam_jepa_finetune", []),
|
| 74 |
+
],
|
| 75 |
+
)
|
| 76 |
+
def test_pretrain_then_fine_tune_and_probe_on_gamgam(
|
| 77 |
+
backbone, pretrain_config, finetune_config, overrides, tmp_path
|
| 78 |
+
):
|
| 79 |
+
small = [f"model.{key}={value}" for key, value in BACKBONES[backbone].items()]
|
| 80 |
+
pretrain = fixture_config(
|
| 81 |
+
pretrain_config,
|
| 82 |
+
[TTH_FIXTURE, TTH_FIXTURE],
|
| 83 |
+
[0, 1],
|
| 84 |
+
tmp_path / "pretrain",
|
| 85 |
+
"train.max_epochs=2",
|
| 86 |
+
*small,
|
| 87 |
+
*overrides,
|
| 88 |
+
)
|
| 89 |
+
path = str(save_config(pretrain, tmp_path / "pretrain.yaml"))
|
| 90 |
+
assert main(["prepare", path]) == 0
|
| 91 |
+
assert main(["train", path]) == 0
|
| 92 |
+
history = [
|
| 93 |
+
json.loads(line)
|
| 94 |
+
for line in (pretrain.path / "history.jsonl").read_text().splitlines()
|
| 95 |
+
]
|
| 96 |
+
spread = (
|
| 97 |
+
"embedding_spread" if pretrain.model.name == "contrastive" else "target_spread"
|
| 98 |
+
)
|
| 99 |
+
assert len(history) == 2 and history[1]["validation"][spread] > 0
|
| 100 |
+
# The pretraining config probes its own encoder, on its labels.
|
| 101 |
+
probe = ["data.train_events=40", "model.probe_neighbours=5", "split=validation"]
|
| 102 |
+
probe.append(f"model.pretrained={pretrain.path}/last.pt")
|
| 103 |
+
assert main(["probe", path, *probe]) == 0
|
| 104 |
+
summary = json.loads((pretrain.path / "validation_probe.json").read_text())
|
| 105 |
+
assert math.isfinite(summary["knn"]["accuracy"])
|
| 106 |
+
|
| 107 |
+
finetune = fixture_config(
|
| 108 |
+
finetune_config,
|
| 109 |
+
GAMGAM_FIXTURES,
|
| 110 |
+
list(range(5)),
|
| 111 |
+
tmp_path / "finetune",
|
| 112 |
+
"train.max_epochs=1",
|
| 113 |
+
f"model.pretrained={pretrain.path}/last.pt",
|
| 114 |
+
"data.train_events=100",
|
| 115 |
+
"model.probe_neighbours=5",
|
| 116 |
+
*small,
|
| 117 |
+
)
|
| 118 |
+
path = str(save_config(finetune, tmp_path / "finetune.yaml"))
|
| 119 |
+
assert main(["prepare", path]) == 0
|
| 120 |
+
assert main(["train", path]) == 0
|
| 121 |
+
assert main(["probe", path]) == 0
|
| 122 |
+
summary = json.loads((finetune.path / "test_probe.json").read_text())
|
| 123 |
+
assert math.isfinite(summary["linear"]["auc"])
|
|
@@ -1,7 +1,8 @@
|
|
| 1 |
-
"""JEPA pretraining of both backbones: masking, the predictor, the EMA target
|
| 2 |
-
and fine-tuning."""
|
| 3 |
|
| 4 |
import json
|
|
|
|
| 5 |
|
| 6 |
import pytest
|
| 7 |
import torch
|
|
@@ -9,7 +10,13 @@ from conftest import GAMGAM_FIXTURES, TTH_FIXTURE, fixture_config, random_events
|
|
| 9 |
|
| 10 |
from gnn4colliders.cli import main
|
| 11 |
from gnn4colliders.config import ModelConfig, save_config
|
| 12 |
-
from gnn4colliders.jepa import
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
from gnn4colliders.models import build_model
|
| 14 |
from gnn4colliders.sequences import collate_sequence_batch
|
| 15 |
from gnn4colliders.train import load_checkpoint
|
|
@@ -23,9 +30,11 @@ BACKBONES = {
|
|
| 23 |
backbones = pytest.mark.parametrize("backbone", BACKBONES)
|
| 24 |
|
| 25 |
|
| 26 |
-
def _jepa(backbone="transformer"):
|
| 27 |
torch.manual_seed(0)
|
| 28 |
-
config = ModelConfig(
|
|
|
|
|
|
|
| 29 |
return build_model(config, num_features=7, num_node_types=5)
|
| 30 |
|
| 31 |
|
|
@@ -77,6 +86,27 @@ def test_loss_is_finite_and_trains_every_context_parameter(backbone):
|
|
| 77 |
assert parameter.grad is not None and parameter.grad.abs().sum() > 0, name
|
| 78 |
|
| 79 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
def test_variance_term_penalizes_only_collapsed_dimensions():
|
| 81 |
spread = torch.randn(4096, 8, generator=torch.Generator().manual_seed(0))
|
| 82 |
assert _variance_penalty(spread) < 0.03
|
|
|
|
| 1 |
+
"""JEPA pretraining of both backbones: masking, the predictor, the EMA target,
|
| 2 |
+
kinematic targets (masked objects) and fine-tuning."""
|
| 3 |
|
| 4 |
import json
|
| 5 |
+
import math
|
| 6 |
|
| 7 |
import pytest
|
| 8 |
import torch
|
|
|
|
| 10 |
|
| 11 |
from gnn4colliders.cli import main
|
| 12 |
from gnn4colliders.config import ModelConfig, save_config
|
| 13 |
+
from gnn4colliders.jepa import (
|
| 14 |
+
FEATURE_TARGETS,
|
| 15 |
+
_variance_penalty,
|
| 16 |
+
object_features,
|
| 17 |
+
sample_targets,
|
| 18 |
+
type_index,
|
| 19 |
+
)
|
| 20 |
from gnn4colliders.models import build_model
|
| 21 |
from gnn4colliders.sequences import collate_sequence_batch
|
| 22 |
from gnn4colliders.train import load_checkpoint
|
|
|
|
| 30 |
backbones = pytest.mark.parametrize("backbone", BACKBONES)
|
| 31 |
|
| 32 |
|
| 33 |
+
def _jepa(backbone="transformer", target="encoding"):
|
| 34 |
torch.manual_seed(0)
|
| 35 |
+
config = ModelConfig(
|
| 36 |
+
name="jepa", backbone=backbone, jepa_target=target, **BACKBONES[backbone]
|
| 37 |
+
)
|
| 38 |
return build_model(config, num_features=7, num_node_types=5)
|
| 39 |
|
| 40 |
|
|
|
|
| 86 |
assert parameter.grad is not None and parameter.grad.abs().sum() > 0, name
|
| 87 |
|
| 88 |
|
| 89 |
+
def test_kinematic_targets_of_objects():
|
| 90 |
+
tokens = torch.tensor([[[1.0, 0.5, math.pi / 2, 2.0, 1.0, -1.0, 0.0]]])
|
| 91 |
+
torch.testing.assert_close(
|
| 92 |
+
object_features(tokens),
|
| 93 |
+
torch.tensor([[[math.log(2), 0.5, 0.0, 1.0, math.log(3), 1.0, -1.0]]]),
|
| 94 |
+
)
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
@backbones
|
| 98 |
+
def test_masked_objects_predict_kinematics_without_a_target_encoder(backbone):
|
| 99 |
+
model, batch = _jepa(backbone, "features").train(), _batch()
|
| 100 |
+
assert model.target_encoder is None
|
| 101 |
+
model.update_target() # nothing to update
|
| 102 |
+
outputs = model(batch)
|
| 103 |
+
assert outputs.shape == (16, 3) and torch.isfinite(outputs).all()
|
| 104 |
+
assert model.predictor_out.out_features == FEATURE_TARGETS
|
| 105 |
+
outputs[:, 0].mean().backward()
|
| 106 |
+
for name, parameter in model.context_encoder.named_parameters():
|
| 107 |
+
assert parameter.grad is not None and parameter.grad.abs().sum() > 0, name
|
| 108 |
+
|
| 109 |
+
|
| 110 |
def test_variance_term_penalizes_only_collapsed_dimensions():
|
| 111 |
spread = torch.randn(4096, 8, generator=torch.Generator().manual_seed(0))
|
| 112 |
assert _variance_penalty(spread) < 0.03
|
|
@@ -0,0 +1,152 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""k-NN and linear probes of frozen encoders: the classifiers, the encoder
|
| 2 |
+
weights of each kind of checkpoint, and the probe command."""
|
| 3 |
+
|
| 4 |
+
import json
|
| 5 |
+
import math
|
| 6 |
+
|
| 7 |
+
import pytest
|
| 8 |
+
import torch
|
| 9 |
+
from conftest import GAMGAM_FIXTURES, TTH_FIXTURE, fixture_config
|
| 10 |
+
|
| 11 |
+
from gnn4colliders.cli import main
|
| 12 |
+
from gnn4colliders.config import ModelConfig, save_config
|
| 13 |
+
from gnn4colliders.data import load_events
|
| 14 |
+
from gnn4colliders.models import encoder_state, pretrained_encoder
|
| 15 |
+
from gnn4colliders.probe import knn_logits, linear_logits
|
| 16 |
+
|
| 17 |
+
SMALL = {"hid_size": 16, "n_layers": 1, "n_heads": 2, "pair_bias": True}
|
| 18 |
+
BACKBONES = {
|
| 19 |
+
"transformer": SMALL,
|
| 20 |
+
"edge_network": {"hid_size": 16, "n_layers": 2, "n_proc_steps": 2, "n_heads": 2},
|
| 21 |
+
}
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def _clusters(count=60, seed=0):
|
| 25 |
+
"""Two well separated classes of 8-dimensional embeddings."""
|
| 26 |
+
generator = torch.Generator().manual_seed(seed)
|
| 27 |
+
labels = torch.arange(count) % 2
|
| 28 |
+
centres = torch.tensor([[3.0] + [0.0] * 7, [0.0, 3.0] + [0.0] * 6])
|
| 29 |
+
return centres[labels] + 0.3 * torch.randn(count, 8, generator=generator), labels
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def test_knn_and_linear_probes_separate_clusters():
|
| 33 |
+
train, labels = _clusters(seed=0)
|
| 34 |
+
queries, expected = _clusters(seed=1)
|
| 35 |
+
for logits in (
|
| 36 |
+
knn_logits(train, labels, queries, num_classes=2, neighbours=5),
|
| 37 |
+
linear_logits(train, labels, queries, num_classes=2, l2=1e-4),
|
| 38 |
+
):
|
| 39 |
+
assert logits.shape == (60, 2)
|
| 40 |
+
assert torch.equal(logits.argmax(dim=1), expected)
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def test_encoder_weights_of_each_kind_of_checkpoint(tmp_path):
|
| 44 |
+
weight = torch.ones(2)
|
| 45 |
+
checkpoints = {
|
| 46 |
+
"jepa": {"context_encoder.norm.weight": weight, "target_encoder.x": weight},
|
| 47 |
+
"transformer": {"encoder.norm.weight": weight, "classifier.weight": weight},
|
| 48 |
+
"edge_network": {"node_encoder.0.weight": weight, "classifier.weight": weight},
|
| 49 |
+
"fine_tuned": {"backbone.node_encoder.0.weight": weight, "head.weight": weight},
|
| 50 |
+
"tabular": {"inputs.encoder.norm.weight": weight, "network.x.up": weight},
|
| 51 |
+
}
|
| 52 |
+
expected = {
|
| 53 |
+
"jepa": ["norm.weight"],
|
| 54 |
+
"transformer": ["norm.weight"],
|
| 55 |
+
"edge_network": ["node_encoder.0.weight"],
|
| 56 |
+
"fine_tuned": ["node_encoder.0.weight"],
|
| 57 |
+
"tabular": ["norm.weight"],
|
| 58 |
+
}
|
| 59 |
+
for name, state in checkpoints.items():
|
| 60 |
+
path = tmp_path / f"{name}.pt"
|
| 61 |
+
torch.save({"model_state_dict": state}, path)
|
| 62 |
+
assert list(encoder_state(str(path))) == expected[name]
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
@pytest.mark.parametrize(
|
| 66 |
+
("backbone", "pretrain_config", "probe_config"),
|
| 67 |
+
[
|
| 68 |
+
("transformer", "delphes_jepa", "gamgam_jepa_finetune"),
|
| 69 |
+
("edge_network", "delphes_jepa_edge", "gamgam_finetune"),
|
| 70 |
+
],
|
| 71 |
+
)
|
| 72 |
+
def test_probe_random_and_jepa_encoders_on_gamgam(
|
| 73 |
+
backbone, pretrain_config, probe_config, tmp_path
|
| 74 |
+
):
|
| 75 |
+
small = [f"model.{key}={value}" for key, value in BACKBONES[backbone].items()]
|
| 76 |
+
pretrain = fixture_config(
|
| 77 |
+
pretrain_config,
|
| 78 |
+
[TTH_FIXTURE, TTH_FIXTURE],
|
| 79 |
+
[0, 1],
|
| 80 |
+
tmp_path / "jepa",
|
| 81 |
+
"train.max_epochs=1",
|
| 82 |
+
*small,
|
| 83 |
+
)
|
| 84 |
+
path = str(save_config(pretrain, tmp_path / "jepa.yaml"))
|
| 85 |
+
assert main(["prepare", path]) == 0
|
| 86 |
+
assert main(["train", path]) == 0
|
| 87 |
+
|
| 88 |
+
for name, encoder in (("random", "null"), ("jepa", f"{pretrain.path}/last.pt")):
|
| 89 |
+
config = fixture_config(
|
| 90 |
+
probe_config,
|
| 91 |
+
GAMGAM_FIXTURES,
|
| 92 |
+
list(range(5)),
|
| 93 |
+
tmp_path / name,
|
| 94 |
+
f"model.pretrained={encoder}",
|
| 95 |
+
"data.train_events=100",
|
| 96 |
+
"model.probe_neighbours=5",
|
| 97 |
+
"split=validation",
|
| 98 |
+
*small,
|
| 99 |
+
)
|
| 100 |
+
path = str(save_config(config, tmp_path / f"{name}.yaml"))
|
| 101 |
+
assert main(["prepare", path]) == 0
|
| 102 |
+
assert main(["probe", path]) == 0
|
| 103 |
+
summary = json.loads((config.path / "validation_probe.json").read_text())
|
| 104 |
+
assert summary["train_events"] == 100
|
| 105 |
+
for probe in ("knn", "linear"):
|
| 106 |
+
assert math.isfinite(summary[probe]["auc"])
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
def test_probe_a_supervised_transformer(tmp_path):
|
| 110 |
+
small = [f"model.{key}={value}" for key, value in SMALL.items()]
|
| 111 |
+
trained = fixture_config(
|
| 112 |
+
"gamgam_transformer",
|
| 113 |
+
GAMGAM_FIXTURES,
|
| 114 |
+
list(range(5)),
|
| 115 |
+
tmp_path / "supervised",
|
| 116 |
+
"train.max_epochs=1",
|
| 117 |
+
*small,
|
| 118 |
+
)
|
| 119 |
+
path = str(save_config(trained, tmp_path / "supervised.yaml"))
|
| 120 |
+
assert main(["prepare", path]) == 0
|
| 121 |
+
assert main(["train", path]) == 0
|
| 122 |
+
config = fixture_config(
|
| 123 |
+
"gamgam_transformer",
|
| 124 |
+
GAMGAM_FIXTURES,
|
| 125 |
+
list(range(5)),
|
| 126 |
+
tmp_path / "probe",
|
| 127 |
+
f"model.pretrained={trained.path}/last.pt",
|
| 128 |
+
"data.train_events=50",
|
| 129 |
+
*small,
|
| 130 |
+
)
|
| 131 |
+
path = str(save_config(config, tmp_path / "probe.yaml"))
|
| 132 |
+
assert main(["prepare", path]) == 0
|
| 133 |
+
assert main(["probe", path]) == 0
|
| 134 |
+
summary = json.loads((config.path / "test_probe.json").read_text())
|
| 135 |
+
assert summary["encoder"] == f"{trained.path}/last.pt"
|
| 136 |
+
# On the training split, the probes' own events are not queries.
|
| 137 |
+
assert main(["probe", path, "split=train"]) == 0
|
| 138 |
+
summary = json.loads((config.path / "train_probe.json").read_text())
|
| 139 |
+
events = load_events(config.data.cache)
|
| 140 |
+
train_fold = torch.isin(events.fold, torch.tensor(config.data.train_folds))
|
| 141 |
+
assert summary["num_samples"] == int(train_fold.sum()) - 50
|
| 142 |
+
|
| 143 |
+
|
| 144 |
+
def test_a_random_encoder_is_rebuilt_identically_without_touching_global_state():
|
| 145 |
+
config = ModelConfig(name="transformer", **SMALL)
|
| 146 |
+
state = torch.random.get_rng_state()
|
| 147 |
+
first, second = (
|
| 148 |
+
pretrained_encoder(config, num_features=7, num_node_types=5) for _ in range(2)
|
| 149 |
+
)
|
| 150 |
+
assert torch.equal(torch.random.get_rng_state(), state)
|
| 151 |
+
for a, b in zip(first.parameters(), second.parameters()):
|
| 152 |
+
assert torch.equal(a, b)
|
|
@@ -1,17 +1,31 @@
|
|
| 1 |
-
"""Fixed-width event tables
|
| 2 |
-
context (when installed
|
|
|
|
| 3 |
|
| 4 |
import json
|
| 5 |
import math
|
|
|
|
|
|
|
| 6 |
|
| 7 |
import pytest
|
| 8 |
import torch
|
| 9 |
-
from conftest import GAMGAM_FIXTURES, fixture_config
|
|
|
|
|
|
|
| 10 |
|
| 11 |
from gnn4colliders.cli import main
|
| 12 |
-
from gnn4colliders.config import save_config
|
| 13 |
from gnn4colliders.data import pack_events
|
| 14 |
-
from gnn4colliders.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
|
| 16 |
|
| 17 |
def _objects(types):
|
|
@@ -23,8 +37,9 @@ def _objects(types):
|
|
| 23 |
def test_table_keeps_the_leading_objects_of_each_type():
|
| 24 |
first, second = _objects([0, 0, 0, 3, 3, 4]), _objects([1, 4])
|
| 25 |
events = pack_events([first, second], [0, 1])
|
| 26 |
-
|
| 27 |
-
|
|
|
|
| 28 |
slots = table[:, :36].view(2, 6, 6)
|
| 29 |
# Slots: jet, jet, electron, muon, photon, MET.
|
| 30 |
torch.testing.assert_close(slots[0, :2], first[:2, :6])
|
|
@@ -36,6 +51,67 @@ def test_table_keeps_the_leading_objects_of_each_type():
|
|
| 36 |
assert table[:, 36:].tolist() == [[3, 0, 0, 2, 1], [0, 1, 0, 0, 1]]
|
| 37 |
|
| 38 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
def test_balanced_subsets_are_equal_per_class_and_nested():
|
| 40 |
labels = torch.tensor([0] * 50 + [1] * 30 + [2] * 5)
|
| 41 |
events = pack_events([torch.zeros(1, 7)] * len(labels), labels)
|
|
@@ -67,3 +143,261 @@ def test_tabpfn_classifies_gamgam_in_context(tmp_path):
|
|
| 67 |
assert main(["evaluate", path]) == 0
|
| 68 |
metrics = json.loads((config.path / "validation_metrics.json").read_text())
|
| 69 |
assert not math.isnan(metrics["metrics"]["auc"])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Fixed-width event tables, balanced label subsets, LoRA, TabICL in
|
| 2 |
+
context, and TabPFN in context and meta-trained (when installed, with its
|
| 3 |
+
weights)."""
|
| 4 |
|
| 5 |
import json
|
| 6 |
import math
|
| 7 |
+
import os
|
| 8 |
+
import sys
|
| 9 |
|
| 10 |
import pytest
|
| 11 |
import torch
|
| 12 |
+
from conftest import GAMGAM_FIXTURES, TTH_FIXTURE, fixture_config
|
| 13 |
+
from torch import nn
|
| 14 |
+
from torch.nn.utils import parametrize
|
| 15 |
|
| 16 |
from gnn4colliders.cli import main
|
| 17 |
+
from gnn4colliders.config import ModelConfig, save_config
|
| 18 |
from gnn4colliders.data import pack_events
|
| 19 |
+
from gnn4colliders.sequences import collate_sequence_batch
|
| 20 |
+
from gnn4colliders.tabular import (
|
| 21 |
+
EventInputs,
|
| 22 |
+
RowLayout,
|
| 23 |
+
add_lora,
|
| 24 |
+
merge_lora,
|
| 25 |
+
sequence_table,
|
| 26 |
+
table_features,
|
| 27 |
+
)
|
| 28 |
+
from gnn4colliders.train import load_checkpoint
|
| 29 |
|
| 30 |
|
| 31 |
def _objects(types):
|
|
|
|
| 37 |
def test_table_keeps_the_leading_objects_of_each_type():
|
| 38 |
first, second = _objects([0, 0, 0, 3, 3, 4]), _objects([1, 4])
|
| 39 |
events = pack_events([first, second], [0, 1])
|
| 40 |
+
layout = RowLayout((2, 1, 1, 1, 1))
|
| 41 |
+
table = table_features(events, torch.tensor([0, 1]), layout)
|
| 42 |
+
assert table.shape == (2, layout.width) == (2, 6 * 6 + 5)
|
| 43 |
slots = table[:, :36].view(2, 6, 6)
|
| 44 |
# Slots: jet, jet, electron, muon, photon, MET.
|
| 45 |
torch.testing.assert_close(slots[0, :2], first[:2, :6])
|
|
|
|
| 51 |
assert table[:, 36:].tolist() == [[3, 0, 0, 2, 1], [0, 1, 0, 0, 1]]
|
| 52 |
|
| 53 |
|
| 54 |
+
def test_sequence_batches_give_the_same_table():
|
| 55 |
+
events = pack_events(
|
| 56 |
+
[_objects([0, 0, 0, 3, 3, 4]), _objects([1, 4]), _objects([2, 2, 2, 0])],
|
| 57 |
+
[0, 1, 2],
|
| 58 |
+
)
|
| 59 |
+
indices = torch.tensor([2, 0, 1])
|
| 60 |
+
layout = RowLayout((2, 1, 1, 1, 1), relative_phi=True, pair_objects=3)
|
| 61 |
+
torch.testing.assert_close(
|
| 62 |
+
sequence_table(collate_sequence_batch(events, indices), layout),
|
| 63 |
+
table_features(events, indices, layout),
|
| 64 |
+
equal_nan=True,
|
| 65 |
+
)
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
def _object(pt, eta, phi, node_type):
|
| 69 |
+
return [pt, eta, phi, pt * math.cosh(eta), 0.0, 0.0, node_type]
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def test_relative_phi_and_the_masses_and_delta_r_of_leading_pairs():
|
| 73 |
+
# Two back-to-back photons (mass 2, Delta R pi) and a softer jet; an
|
| 74 |
+
# event with one photon has no pair.
|
| 75 |
+
pair = torch.tensor(
|
| 76 |
+
[_object(1.0, 0.0, 0.5, 3), _object(1.0, 0.0, 0.5 - math.pi, 3)]
|
| 77 |
+
+ [_object(0.5, 1.0, 2.0, 0)]
|
| 78 |
+
)
|
| 79 |
+
single = torch.tensor([_object(1.0, 0.3, 1.0, 3)])
|
| 80 |
+
events = pack_events([pair, single], [0, 1])
|
| 81 |
+
layout = RowLayout((1, 0, 0, 2, 0), relative_phi=True, pair_objects=2)
|
| 82 |
+
table = table_features(events, torch.tensor([0, 1]), layout)
|
| 83 |
+
assert table.shape == (2, layout.width) == (2, 3 * 6 + 5 + 2)
|
| 84 |
+
slots = table[:, :18].view(2, 3, 6)
|
| 85 |
+
# Azimuths from the leading photon's.
|
| 86 |
+
torch.testing.assert_close(slots[0, :, 2], torch.tensor([1.5, 0.0, -math.pi]))
|
| 87 |
+
assert slots[1, 1, 2] == 0
|
| 88 |
+
torch.testing.assert_close(table[0, -2:], torch.tensor([2.0, math.pi]))
|
| 89 |
+
assert table[1, -2:].isnan().all()
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def test_lora_starts_at_the_frozen_weight_and_trains_only_its_update():
|
| 93 |
+
network = nn.Sequential(nn.Linear(6, 4))
|
| 94 |
+
weight, x = network[0].weight.detach().clone(), torch.randn(5, 6)
|
| 95 |
+
expected = network(x)
|
| 96 |
+
add_lora(network, ["0.weight"], rank=2)
|
| 97 |
+
torch.testing.assert_close(network(x), expected)
|
| 98 |
+
trainable = [name for name, p in network.named_parameters() if p.requires_grad]
|
| 99 |
+
assert sorted(trainable) == [
|
| 100 |
+
"0.parametrizations.weight.0.down",
|
| 101 |
+
"0.parametrizations.weight.0.up",
|
| 102 |
+
]
|
| 103 |
+
# Code that reads the weight directly sees the update too.
|
| 104 |
+
with torch.no_grad():
|
| 105 |
+
network[0].parametrizations.weight[0].up.fill_(1.0)
|
| 106 |
+
lora = network[0].parametrizations.weight[0]
|
| 107 |
+
torch.testing.assert_close(network[0].weight, weight + lora.up @ lora.down)
|
| 108 |
+
# Merged for inference, the network computes the same.
|
| 109 |
+
expected = network(x)
|
| 110 |
+
merge_lora(network)
|
| 111 |
+
assert not parametrize.is_parametrized(network[0])
|
| 112 |
+
torch.testing.assert_close(network(x), expected)
|
| 113 |
+
|
| 114 |
+
|
| 115 |
def test_balanced_subsets_are_equal_per_class_and_nested():
|
| 116 |
labels = torch.tensor([0] * 50 + [1] * 30 + [2] * 5)
|
| 117 |
events = pack_events([torch.zeros(1, 7)] * len(labels), labels)
|
|
|
|
| 143 |
assert main(["evaluate", path]) == 0
|
| 144 |
metrics = json.loads((config.path / "validation_metrics.json").read_text())
|
| 145 |
assert not math.isnan(metrics["metrics"]["auc"])
|
| 146 |
+
assert metrics["checkpoint"] == "TabPFN v2 (default checkpoint)"
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def _tabpfn_3_5_weights() -> None:
|
| 150 |
+
pytest.importorskip("tabpfn")
|
| 151 |
+
from tabpfn.model_loading import get_cache_dir
|
| 152 |
+
|
| 153 |
+
if not (get_cache_dir() / "tabpfn-v3.5-20260909.safetensors").exists():
|
| 154 |
+
pytest.skip("TabPFN-3.5 weights are not in the TabPFN cache")
|
| 155 |
+
|
| 156 |
+
|
| 157 |
+
def test_meta_train_tabpfn_with_lora_then_classify_gamgam(tmp_path):
|
| 158 |
+
_tabpfn_3_5_weights()
|
| 159 |
+
meta = fixture_config(
|
| 160 |
+
"delphes_tabpfn",
|
| 161 |
+
[TTH_FIXTURE, TTH_FIXTURE],
|
| 162 |
+
[0, 1],
|
| 163 |
+
tmp_path / "meta",
|
| 164 |
+
"train.max_epochs=1",
|
| 165 |
+
"task.context_events=[4,16]",
|
| 166 |
+
"task.query_events=4",
|
| 167 |
+
"task.steps_per_epoch=2",
|
| 168 |
+
"task.validation_steps=1",
|
| 169 |
+
"model.relative_phi=true",
|
| 170 |
+
"model.pair_objects=4",
|
| 171 |
+
)
|
| 172 |
+
path = str(save_config(meta, tmp_path / "meta.yaml"))
|
| 173 |
+
assert main(["prepare", path]) == 0
|
| 174 |
+
assert main(["train", path]) == 0
|
| 175 |
+
# Of the weights, only the LoRA updates are saved: 24 blocks of 4
|
| 176 |
+
# projections and 2 MLP layers.
|
| 177 |
+
state = load_checkpoint(meta.path / "best.pt")["model_state_dict"]
|
| 178 |
+
weights = [key for key in state if key.endswith(".weight")]
|
| 179 |
+
updates = [key for key in state if key.endswith((".down", ".up"))]
|
| 180 |
+
assert not weights and len(updates) == 24 * 6 * 2
|
| 181 |
+
|
| 182 |
+
gamgam = fixture_config(
|
| 183 |
+
"gamgam_tabpfn",
|
| 184 |
+
GAMGAM_FIXTURES,
|
| 185 |
+
list(range(5)),
|
| 186 |
+
tmp_path / "gamgam",
|
| 187 |
+
f"checkpoint={meta.path}/best.pt",
|
| 188 |
+
"data.train_events=40",
|
| 189 |
+
"split=validation",
|
| 190 |
+
"model.relative_phi=true",
|
| 191 |
+
"model.pair_objects=4",
|
| 192 |
+
)
|
| 193 |
+
path = str(save_config(gamgam, tmp_path / "gamgam.yaml"))
|
| 194 |
+
assert main(["prepare", path]) == 0
|
| 195 |
+
assert main(["evaluate", path]) == 0
|
| 196 |
+
metrics = json.loads((gamgam.path / "validation_metrics.json").read_text())
|
| 197 |
+
assert metrics["checkpoint"] == f"{meta.path}/best.pt"
|
| 198 |
+
assert math.isfinite(metrics["metrics"]["auc"])
|
| 199 |
+
|
| 200 |
+
|
| 201 |
+
def test_tabicl_classifies_gamgam_in_context_with_physics_features(tmp_path):
|
| 202 |
+
pytest.importorskip("tabicl")
|
| 203 |
+
config = fixture_config(
|
| 204 |
+
"gamgam_tabicl",
|
| 205 |
+
GAMGAM_FIXTURES,
|
| 206 |
+
list(range(5)),
|
| 207 |
+
tmp_path,
|
| 208 |
+
"data.train_events=100",
|
| 209 |
+
"model.tabicl_estimators=1",
|
| 210 |
+
"model.relative_phi=true",
|
| 211 |
+
"model.pair_objects=4",
|
| 212 |
+
"split=validation",
|
| 213 |
+
)
|
| 214 |
+
path = str(save_config(config, tmp_path / "tabicl.yaml"))
|
| 215 |
+
assert main(["prepare", path]) == 0
|
| 216 |
+
assert main(["evaluate", path]) == 0
|
| 217 |
+
metrics = json.loads((config.path / "validation_metrics.json").read_text())
|
| 218 |
+
assert metrics["checkpoint"] == "TabICL (tabicl-classifier-v2-20260212.ckpt)"
|
| 219 |
+
assert math.isfinite(metrics["metrics"]["auc"])
|
| 220 |
+
|
| 221 |
+
|
| 222 |
+
#: faiss and torch crash together on macOS unless OpenMP has one thread.
|
| 223 |
+
RETRIEVAL = sys.platform != "darwin" or os.environ.get("OMP_NUM_THREADS") == "1"
|
| 224 |
+
|
| 225 |
+
|
| 226 |
+
@pytest.mark.parametrize(
|
| 227 |
+
"context_size",
|
| 228 |
+
[
|
| 229 |
+
None,
|
| 230 |
+
pytest.param(
|
| 231 |
+
50,
|
| 232 |
+
marks=pytest.mark.skipif(
|
| 233 |
+
not RETRIEVAL, reason="retrieval on macOS needs OMP_NUM_THREADS=1"
|
| 234 |
+
),
|
| 235 |
+
),
|
| 236 |
+
],
|
| 237 |
+
)
|
| 238 |
+
def test_tabdpt_classifies_gamgam_in_context(context_size, tmp_path):
|
| 239 |
+
pytest.importorskip("tabdpt")
|
| 240 |
+
config = fixture_config(
|
| 241 |
+
"gamgam_tabdpt",
|
| 242 |
+
GAMGAM_FIXTURES,
|
| 243 |
+
list(range(5)),
|
| 244 |
+
tmp_path,
|
| 245 |
+
"data.train_events=100",
|
| 246 |
+
"model.tabdpt_estimators=1",
|
| 247 |
+
f"model.tabdpt_context_size={context_size or 'null'}",
|
| 248 |
+
"split=validation",
|
| 249 |
+
)
|
| 250 |
+
path = str(save_config(config, tmp_path / "tabdpt.yaml"))
|
| 251 |
+
assert main(["prepare", path]) == 0
|
| 252 |
+
assert main(["evaluate", path]) == 0
|
| 253 |
+
metrics = json.loads((config.path / "validation_metrics.json").read_text())
|
| 254 |
+
assert (
|
| 255 |
+
metrics["checkpoint"] == f"TabDPT (context {context_size or 'all'} per query)"
|
| 256 |
+
)
|
| 257 |
+
assert math.isfinite(metrics["metrics"]["auc"])
|
| 258 |
+
|
| 259 |
+
|
| 260 |
+
def test_meta_train_tabicl_with_lora_then_classify_gamgam(tmp_path):
|
| 261 |
+
pytest.importorskip("tabicl")
|
| 262 |
+
meta = fixture_config(
|
| 263 |
+
"delphes_tabicl",
|
| 264 |
+
[TTH_FIXTURE, TTH_FIXTURE],
|
| 265 |
+
[0, 1],
|
| 266 |
+
tmp_path / "meta",
|
| 267 |
+
"train.max_epochs=1",
|
| 268 |
+
"task.context_events=[4,16]",
|
| 269 |
+
"task.query_events=4",
|
| 270 |
+
"task.steps_per_epoch=2",
|
| 271 |
+
"task.validation_steps=1",
|
| 272 |
+
"model.pair_objects=4",
|
| 273 |
+
)
|
| 274 |
+
path = str(save_config(meta, tmp_path / "meta.yaml"))
|
| 275 |
+
assert main(["prepare", path]) == 0
|
| 276 |
+
assert main(["train", path]) == 0
|
| 277 |
+
# Only the LoRA updates are saved: 12 blocks of 4 weights.
|
| 278 |
+
state = load_checkpoint(meta.path / "best.pt")["model_state_dict"]
|
| 279 |
+
assert len(state) == 12 * 4 * 2
|
| 280 |
+
assert all(key.endswith((".down", ".up")) for key in state)
|
| 281 |
+
|
| 282 |
+
gamgam = fixture_config(
|
| 283 |
+
"gamgam_tabicl",
|
| 284 |
+
GAMGAM_FIXTURES,
|
| 285 |
+
list(range(5)),
|
| 286 |
+
tmp_path / "gamgam",
|
| 287 |
+
f"checkpoint={meta.path}/best.pt",
|
| 288 |
+
"data.train_events=40",
|
| 289 |
+
"split=validation",
|
| 290 |
+
"model.pair_objects=4",
|
| 291 |
+
)
|
| 292 |
+
path = str(save_config(gamgam, tmp_path / "gamgam.yaml"))
|
| 293 |
+
assert main(["prepare", path]) == 0
|
| 294 |
+
assert main(["evaluate", path]) == 0
|
| 295 |
+
metrics = json.loads((gamgam.path / "validation_metrics.json").read_text())
|
| 296 |
+
assert metrics["checkpoint"] == f"{meta.path}/best.pt"
|
| 297 |
+
assert math.isfinite(metrics["metrics"]["auc"])
|
| 298 |
+
|
| 299 |
+
|
| 300 |
+
#: The encoder of the hybrid tests, as pretrained by JEPA on the fixtures.
|
| 301 |
+
ENCODER = {
|
| 302 |
+
"hid_size": 16,
|
| 303 |
+
"n_layers": 1,
|
| 304 |
+
"n_heads": 2,
|
| 305 |
+
"ffn_size": 64,
|
| 306 |
+
"pair_bias": True,
|
| 307 |
+
}
|
| 308 |
+
|
| 309 |
+
|
| 310 |
+
def test_event_inputs_are_rows_embeddings_or_both():
|
| 311 |
+
events = pack_events([_objects([0, 0, 3, 3, 4]), _objects([1, 3, 4])], [0, 1])
|
| 312 |
+
batch = collate_sequence_batch(events, torch.tensor([0, 1]))
|
| 313 |
+
width = RowLayout((6, 2, 2, 3, 1)).width
|
| 314 |
+
for inputs, expected in (("rows", width), ("embeddings", 16), ("both", width + 16)):
|
| 315 |
+
config = ModelConfig(name="tabicl", tabular_inputs=inputs, **ENCODER)
|
| 316 |
+
model = EventInputs.of(config, num_features=7, num_node_types=5)
|
| 317 |
+
assert model(batch).shape == (2, expected)
|
| 318 |
+
with pytest.raises(ValueError, match="tabular_inputs"):
|
| 319 |
+
EventInputs.of(
|
| 320 |
+
ModelConfig(tabular_inputs="objects"), num_features=7, num_node_types=5
|
| 321 |
+
)
|
| 322 |
+
|
| 323 |
+
|
| 324 |
+
@pytest.fixture
|
| 325 |
+
def jepa_encoder(tmp_path):
|
| 326 |
+
small = [f"model.{key}={value}" for key, value in ENCODER.items()]
|
| 327 |
+
config = fixture_config(
|
| 328 |
+
"delphes_jepa",
|
| 329 |
+
[TTH_FIXTURE, TTH_FIXTURE],
|
| 330 |
+
[0, 1],
|
| 331 |
+
tmp_path / "jepa",
|
| 332 |
+
"train.max_epochs=1",
|
| 333 |
+
*small,
|
| 334 |
+
)
|
| 335 |
+
path = str(save_config(config, tmp_path / "jepa.yaml"))
|
| 336 |
+
assert main(["prepare", path]) == 0
|
| 337 |
+
assert main(["train", path]) == 0
|
| 338 |
+
return f"{config.path}/last.pt"
|
| 339 |
+
|
| 340 |
+
|
| 341 |
+
def _hybrid(name, tmp_path, jepa_encoder, *overrides):
|
| 342 |
+
small = [f"model.{key}={value}" for key, value in ENCODER.items()]
|
| 343 |
+
config = fixture_config(
|
| 344 |
+
name,
|
| 345 |
+
GAMGAM_FIXTURES if name.startswith("gamgam") else [TTH_FIXTURE, TTH_FIXTURE],
|
| 346 |
+
list(range(5)) if name.startswith("gamgam") else [0, 1],
|
| 347 |
+
tmp_path / name,
|
| 348 |
+
"model.tabular_inputs=both",
|
| 349 |
+
f"model.pretrained={jepa_encoder}",
|
| 350 |
+
*small,
|
| 351 |
+
*overrides,
|
| 352 |
+
)
|
| 353 |
+
path = str(save_config(config, tmp_path / f"{name}.yaml"))
|
| 354 |
+
assert main(["prepare", path]) == 0
|
| 355 |
+
return config, path
|
| 356 |
+
|
| 357 |
+
|
| 358 |
+
def test_tabicl_classifies_gamgam_on_rows_and_jepa_embeddings(tmp_path, jepa_encoder):
|
| 359 |
+
pytest.importorskip("tabicl")
|
| 360 |
+
config, path = _hybrid(
|
| 361 |
+
"gamgam_tabicl",
|
| 362 |
+
tmp_path,
|
| 363 |
+
jepa_encoder,
|
| 364 |
+
"data.train_events=100",
|
| 365 |
+
"model.tabicl_estimators=1",
|
| 366 |
+
"split=validation",
|
| 367 |
+
)
|
| 368 |
+
assert main(["evaluate", path]) == 0
|
| 369 |
+
metrics = json.loads((config.path / "validation_metrics.json").read_text())
|
| 370 |
+
assert math.isfinite(metrics["metrics"]["auc"])
|
| 371 |
+
|
| 372 |
+
|
| 373 |
+
@pytest.mark.parametrize("freeze", [False, True])
|
| 374 |
+
def test_meta_train_tabicl_with_its_encoder(tmp_path, jepa_encoder, freeze):
|
| 375 |
+
pytest.importorskip("tabicl")
|
| 376 |
+
meta, path = _hybrid(
|
| 377 |
+
"delphes_tabicl",
|
| 378 |
+
tmp_path,
|
| 379 |
+
jepa_encoder,
|
| 380 |
+
"train.max_epochs=1",
|
| 381 |
+
"task.context_events=[4,16]",
|
| 382 |
+
"task.query_events=4",
|
| 383 |
+
"task.steps_per_epoch=2",
|
| 384 |
+
"task.validation_steps=1",
|
| 385 |
+
f"model.freeze_backbone={str(freeze).lower()}",
|
| 386 |
+
)
|
| 387 |
+
assert main(["train", path]) == 0
|
| 388 |
+
state = load_checkpoint(meta.path / "best.pt")["model_state_dict"]
|
| 389 |
+
encoder = [key for key in state if key.startswith("inputs.encoder.")]
|
| 390 |
+
# A trained encoder is saved; a frozen one comes from model.pretrained.
|
| 391 |
+
assert bool(encoder) != freeze
|
| 392 |
+
gamgam, path = _hybrid(
|
| 393 |
+
"gamgam_tabicl",
|
| 394 |
+
tmp_path,
|
| 395 |
+
jepa_encoder,
|
| 396 |
+
f"checkpoint={meta.path}/best.pt",
|
| 397 |
+
"data.train_events=40",
|
| 398 |
+
"split=validation",
|
| 399 |
+
f"model.freeze_backbone={str(freeze).lower()}",
|
| 400 |
+
)
|
| 401 |
+
assert main(["evaluate", path]) == 0
|
| 402 |
+
metrics = json.loads((gamgam.path / "validation_metrics.json").read_text())
|
| 403 |
+
assert math.isfinite(metrics["metrics"]["auc"])
|
|
@@ -53,6 +53,7 @@ def test_gamgam_train_evaluate_predict_export_and_resume(gamgam, tmp_path):
|
|
| 53 |
events = load_events(gamgam.data.cache)
|
| 54 |
assert summary["num_samples"] == int((events.fold == 9).sum())
|
| 55 |
assert all(math.isfinite(value) for value in summary["metrics"].values())
|
|
|
|
| 56 |
|
| 57 |
assert main(["predict", path, "split=validation"]) == 0
|
| 58 |
predictions = np.load(run / "validation_predictions.npz")
|
|
@@ -167,7 +168,9 @@ def _rank(rank: int, port: int, argv: list[str], queue) -> None:
|
|
| 167 |
|
| 168 |
|
| 169 |
def test_training_runs_on_two_ddp_ranks(gamgam, tmp_path):
|
| 170 |
-
|
|
|
|
|
|
|
| 171 |
context = mp.get_context("spawn")
|
| 172 |
queue = context.Queue()
|
| 173 |
port = _port()
|
|
|
|
| 53 |
events = load_events(gamgam.data.cache)
|
| 54 |
assert summary["num_samples"] == int((events.fold == 9).sum())
|
| 55 |
assert all(math.isfinite(value) for value in summary["metrics"].values())
|
| 56 |
+
assert set(summary["seconds"]) == {"setup", "inference"}
|
| 57 |
|
| 58 |
assert main(["predict", path, "split=validation"]) == 0
|
| 59 |
predictions = np.load(run / "validation_predictions.npz")
|
|
|
|
| 168 |
|
| 169 |
|
| 170 |
def test_training_runs_on_two_ddp_ranks(gamgam, tmp_path):
|
| 171 |
+
config = _config_file(gamgam, tmp_path)
|
| 172 |
+
# Gloo ranks on the CPU: MPS has no collectives.
|
| 173 |
+
argv = ["train", config, "train.max_epochs=2", "train.device=cpu"]
|
| 174 |
context = mp.get_context("spawn")
|
| 175 |
queue = context.Queue()
|
| 176 |
port = _port()
|
|
@@ -1,5 +1,5 @@
|
|
| 1 |
version = 1
|
| 2 |
-
revision =
|
| 3 |
requires-python = "==3.14.*"
|
| 4 |
resolution-markers = [
|
| 5 |
"platform_machine == 'x86_64' and sys_platform == 'linux'",
|
|
@@ -228,7 +228,7 @@ name = "cuda-bindings"
|
|
| 228 |
version = "13.4.4"
|
| 229 |
source = { registry = "https://pypi.org/simple" }
|
| 230 |
dependencies = [
|
| 231 |
-
{ name = "cuda-pathfinder" },
|
| 232 |
]
|
| 233 |
wheels = [
|
| 234 |
{ url = "https://files.pythonhosted.org/packages/e6/f0/d71d1dd45caa358a39c6898dcc9f1bd22a9f22b92376b967835c047b50bc/cuda_bindings-13.4.4-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4a429132f06d0c3a853aa419aabe4c0a7bfc3f3886fb4c336d81729f63754d31", size = 6407946, upload-time = "2026-10-14T12:35:39.826Z" },
|
|
@@ -255,43 +255,43 @@ wheels = [
|
|
| 255 |
|
| 256 |
[package.optional-dependencies]
|
| 257 |
cublas = [
|
| 258 |
-
{ name = "nvidia-cublas", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 259 |
-
{ name = "nvidia-cuda-nvrtc", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 260 |
]
|
| 261 |
cudart = [
|
| 262 |
-
{ name = "nvidia-cuda-runtime", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 263 |
]
|
| 264 |
cufft = [
|
| 265 |
-
{ name = "nvidia-cufft", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 266 |
-
{ name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 267 |
]
|
| 268 |
cufile = [
|
| 269 |
{ name = "nvidia-cufile", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 270 |
]
|
| 271 |
cupti = [
|
| 272 |
-
{ name = "nvidia-cuda-cupti", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 273 |
]
|
| 274 |
curand = [
|
| 275 |
-
{ name = "nvidia-curand", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 276 |
]
|
| 277 |
cusolver = [
|
| 278 |
-
{ name = "nvidia-cublas", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 279 |
-
{ name = "nvidia-cusolver", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 280 |
-
{ name = "nvidia-cusparse", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 281 |
-
{ name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 282 |
]
|
| 283 |
cusparse = [
|
| 284 |
-
{ name = "nvidia-cusparse", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 285 |
-
{ name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 286 |
]
|
| 287 |
nvjitlink = [
|
| 288 |
-
{ name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 289 |
]
|
| 290 |
nvrtc = [
|
| 291 |
-
{ name = "nvidia-cuda-nvrtc", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 292 |
]
|
| 293 |
nvtx = [
|
| 294 |
-
{ name = "nvidia-nvtx", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')
|
| 295 |
]
|
| 296 |
|
| 297 |
[[package]]
|
|
@@ -312,6 +312,34 @@ wheels = [
|
|
| 312 |
{ url = "https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl", hash = "sha256:54058201ac7087911181bfec4af6091bb59380360f069276601256a76af08193", size = 65638, upload-time = "2026-01-26T04:13:18.546Z" },
|
| 313 |
]
|
| 314 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 315 |
[[package]]
|
| 316 |
name = "filelock"
|
| 317 |
version = "3.32.3"
|
|
@@ -381,6 +409,12 @@ onnx = [
|
|
| 381 |
{ name = "onnx" },
|
| 382 |
{ name = "onnxruntime" },
|
| 383 |
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 384 |
tabpfn = [
|
| 385 |
{ name = "tabpfn" },
|
| 386 |
]
|
|
@@ -401,12 +435,14 @@ requires-dist = [
|
|
| 401 |
{ name = "onnx", marker = "extra == 'onnx'", specifier = ">=1.16,<2" },
|
| 402 |
{ name = "onnxruntime", marker = "extra == 'onnx'", specifier = ">=1.18,<2" },
|
| 403 |
{ name = "pyyaml", specifier = ">=6" },
|
|
|
|
|
|
|
| 404 |
{ name = "tabpfn", marker = "extra == 'tabpfn'", specifier = ">=9.2,<10" },
|
| 405 |
{ name = "torch", marker = "platform_machine != 'x86_64' or sys_platform != 'linux'", specifier = "==2.14.1" },
|
| 406 |
{ name = "torch", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'", specifier = "==2.14.1", index = "https://download.pytorch.org/whl/cu130" },
|
| 407 |
{ name = "uproot", specifier = ">=5.0" },
|
| 408 |
]
|
| 409 |
-
provides-extras = ["onnx", "tabpfn"]
|
| 410 |
|
| 411 |
[package.metadata.requires-dev]
|
| 412 |
dev = [
|
|
@@ -731,7 +767,7 @@ name = "nvidia-cublas"
|
|
| 731 |
version = "13.1.1.3"
|
| 732 |
source = { registry = "https://pypi.org/simple" }
|
| 733 |
dependencies = [
|
| 734 |
-
{ name = "nvidia-cuda-nvrtc" },
|
| 735 |
]
|
| 736 |
wheels = [
|
| 737 |
{ url = "https://files.pythonhosted.org/packages/a7/a1/0bd24ee8c8d03adac032fd2909426a00c88f8c57961b1277ded97f91119f/nvidia_cublas-13.1.1.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:b7a210458267ac818974c53038fbec2e969d5c99f305ab15c72522fa9f001dd5", size = 542848918, upload-time = "2026-04-08T18:46:22.985Z" },
|
|
@@ -770,7 +806,7 @@ name = "nvidia-cudnn-cu13"
|
|
| 770 |
version = "9.24.0.43"
|
| 771 |
source = { registry = "https://pypi.org/simple" }
|
| 772 |
dependencies = [
|
| 773 |
-
{ name = "nvidia-cublas" },
|
| 774 |
]
|
| 775 |
wheels = [
|
| 776 |
{ url = "https://files.pythonhosted.org/packages/ca/30/7c257e3d5cb4fecb147b93895c66e29c93f8e76d74b45bb418ff0587c4ec/nvidia_cudnn_cu13-9.24.0.43-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:a6812a554a1ff0413e9c52b84c26c050380649ab9615f9c16bded368ce9f421f", size = 650976863, upload-time = "2026-07-02T16:23:39.248Z" },
|
|
@@ -782,7 +818,7 @@ name = "nvidia-cufft"
|
|
| 782 |
version = "12.0.0.61"
|
| 783 |
source = { registry = "https://pypi.org/simple" }
|
| 784 |
dependencies = [
|
| 785 |
-
{ name = "nvidia-nvjitlink" },
|
| 786 |
]
|
| 787 |
wheels = [
|
| 788 |
{ url = "https://files.pythonhosted.org/packages/8b/ae/f417a75c0259e85c1d2f83ca4e960289a5f814ed0cea74d18c353d3e989d/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2708c852ef8cd89d1d2068bdbece0aa188813a0c934db3779b9b1faa8442e5f5", size = 214053554, upload-time = "2025-09-04T08:31:38.196Z" },
|
|
@@ -812,9 +848,9 @@ name = "nvidia-cusolver"
|
|
| 812 |
version = "12.0.4.66"
|
| 813 |
source = { registry = "https://pypi.org/simple" }
|
| 814 |
dependencies = [
|
| 815 |
-
{ name = "nvidia-cublas" },
|
| 816 |
-
{ name = "nvidia-cusparse" },
|
| 817 |
-
{ name = "nvidia-nvjitlink" },
|
| 818 |
]
|
| 819 |
wheels = [
|
| 820 |
{ url = "https://files.pythonhosted.org/packages/c8/c3/b30c9e935fc01e3da443ec0116ed1b2a009bb867f5324d3f2d7e533e776b/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:02c2457eaa9e39de20f880f4bd8820e6a1cfb9f9a34f820eb12a155aa5bc92d2", size = 223467760, upload-time = "2025-09-04T08:33:04.222Z" },
|
|
@@ -826,7 +862,7 @@ name = "nvidia-cusparse"
|
|
| 826 |
version = "12.6.3.3"
|
| 827 |
source = { registry = "https://pypi.org/simple" }
|
| 828 |
dependencies = [
|
| 829 |
-
{ name = "nvidia-nvjitlink" },
|
| 830 |
]
|
| 831 |
wheels = [
|
| 832 |
{ url = "https://files.pythonhosted.org/packages/f8/94/5c26f33738ae35276672f12615a64bd008ed5be6d1ebcb23579285d960a9/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:80bcc4662f23f1054ee334a15c72b8940402975e0eab63178fc7e670aa59472c", size = 162155568, upload-time = "2025-09-04T08:33:42.864Z" },
|
|
@@ -878,6 +914,18 @@ wheels = [
|
|
| 878 |
{ url = "https://files.pythonhosted.org/packages/a8/64/3708a90d1ebe202ffdeb7185f878a3c84d15c2b2c31858da2ce0583e2def/nvidia_nvtx-13.0.85-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb7780edb6b14107373c835bf8b72e7a178bac7367e23da7acb108f973f157a6", size = 148878, upload-time = "2025-09-04T08:28:53.627Z" },
|
| 879 |
]
|
| 880 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 881 |
[[package]]
|
| 882 |
name = "onnx"
|
| 883 |
version = "1.22.0"
|
|
@@ -1015,6 +1063,28 @@ wheels = [
|
|
| 1015 |
{ url = "https://files.pythonhosted.org/packages/19/c7/5f7c636ec43e0c545e28d1f1db71990108306f7bdcb89f069ba97e428e7f/protobuf-7.35.1-py3-none-any.whl", hash = "sha256:4bc97768d8fe4ad6743c8a19403e314511ed9f6d13205b687e52421c023ac1b9", size = 171659, upload-time = "2026-06-11T21:55:39.155Z" },
|
| 1016 |
]
|
| 1017 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1018 |
[[package]]
|
| 1019 |
name = "pydantic"
|
| 1020 |
version = "2.14.1"
|
|
@@ -1353,6 +1423,47 @@ wheels = [
|
|
| 1353 |
{ url = "https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", size = 6299353, upload-time = "2025-04-27T18:04:59.103Z" },
|
| 1354 |
]
|
| 1355 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1356 |
[[package]]
|
| 1357 |
name = "tabpfn"
|
| 1358 |
version = "9.2.0"
|
|
|
|
| 1 |
version = 1
|
| 2 |
+
revision = 2
|
| 3 |
requires-python = "==3.14.*"
|
| 4 |
resolution-markers = [
|
| 5 |
"platform_machine == 'x86_64' and sys_platform == 'linux'",
|
|
|
|
| 228 |
version = "13.4.4"
|
| 229 |
source = { registry = "https://pypi.org/simple" }
|
| 230 |
dependencies = [
|
| 231 |
+
{ name = "cuda-pathfinder", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
|
| 232 |
]
|
| 233 |
wheels = [
|
| 234 |
{ url = "https://files.pythonhosted.org/packages/e6/f0/d71d1dd45caa358a39c6898dcc9f1bd22a9f22b92376b967835c047b50bc/cuda_bindings-13.4.4-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4a429132f06d0c3a853aa419aabe4c0a7bfc3f3886fb4c336d81729f63754d31", size = 6407946, upload-time = "2026-10-14T12:35:39.826Z" },
|
|
|
|
| 255 |
|
| 256 |
[package.optional-dependencies]
|
| 257 |
cublas = [
|
| 258 |
+
{ name = "nvidia-cublas", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 259 |
+
{ name = "nvidia-cuda-nvrtc", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 260 |
]
|
| 261 |
cudart = [
|
| 262 |
+
{ name = "nvidia-cuda-runtime", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 263 |
]
|
| 264 |
cufft = [
|
| 265 |
+
{ name = "nvidia-cufft", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 266 |
+
{ name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 267 |
]
|
| 268 |
cufile = [
|
| 269 |
{ name = "nvidia-cufile", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 270 |
]
|
| 271 |
cupti = [
|
| 272 |
+
{ name = "nvidia-cuda-cupti", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 273 |
]
|
| 274 |
curand = [
|
| 275 |
+
{ name = "nvidia-curand", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 276 |
]
|
| 277 |
cusolver = [
|
| 278 |
+
{ name = "nvidia-cublas", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 279 |
+
{ name = "nvidia-cusolver", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 280 |
+
{ name = "nvidia-cusparse", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 281 |
+
{ name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 282 |
]
|
| 283 |
cusparse = [
|
| 284 |
+
{ name = "nvidia-cusparse", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 285 |
+
{ name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 286 |
]
|
| 287 |
nvjitlink = [
|
| 288 |
+
{ name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 289 |
]
|
| 290 |
nvrtc = [
|
| 291 |
+
{ name = "nvidia-cuda-nvrtc", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 292 |
]
|
| 293 |
nvtx = [
|
| 294 |
+
{ name = "nvidia-nvtx", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
|
| 295 |
]
|
| 296 |
|
| 297 |
[[package]]
|
|
|
|
| 312 |
{ url = "https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl", hash = "sha256:54058201ac7087911181bfec4af6091bb59380360f069276601256a76af08193", size = 65638, upload-time = "2026-01-26T04:13:18.546Z" },
|
| 313 |
]
|
| 314 |
|
| 315 |
+
[[package]]
|
| 316 |
+
name = "faiss-cpu"
|
| 317 |
+
version = "1.12.0"
|
| 318 |
+
source = { registry = "https://pypi.org/simple" }
|
| 319 |
+
dependencies = [
|
| 320 |
+
{ name = "numpy" },
|
| 321 |
+
{ name = "packaging" },
|
| 322 |
+
]
|
| 323 |
+
sdist = { url = "https://files.pythonhosted.org/packages/7d/80/bb75a7ed6e824dea452a24d3434a72ed799324a688b10b047d441d270185/faiss_cpu-1.12.0.tar.gz", hash = "sha256:2f87cbcd603f3ed464ebceb857971fdebc318de938566c9ae2b82beda8e953c0", size = 69292, upload-time = "2025-08-13T06:07:26.553Z" }
|
| 324 |
+
wheels = [
|
| 325 |
+
{ url = "https://files.pythonhosted.org/packages/12/35/01a4a7c179d67bee0d8a027b95c3eae19cb354ae69ef2bc50ac3b93bc853/faiss_cpu-1.12.0-cp314-cp314-macosx_13_0_x86_64.whl", hash = "sha256:ff7db774968210d08cd0331287f3f66a6ffef955a7aa9a7fcd3eb4432a4ce5f5", size = 8036142, upload-time = "2025-08-13T06:06:38.894Z" },
|
| 326 |
+
{ url = "https://files.pythonhosted.org/packages/08/23/bac2859490096608c9d527f3041b44c2e43f8df0d4aadd53a4cc5ce678ac/faiss_cpu-1.12.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:220b5bb5439c64e417b35f9ade4c7dc3bf7df683d6123901ba84d6d764ecd486", size = 3363747, upload-time = "2025-08-13T06:06:40.73Z" },
|
| 327 |
+
{ url = "https://files.pythonhosted.org/packages/7b/1d/e18023e1f43a18ec593adcd69d356f1fa94bde20344e38334d5985e5c5cc/faiss_cpu-1.12.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:693d0bf16f79e8d16a1baaeda459f3375f37da0354e97dc032806b48a2a54151", size = 3835232, upload-time = "2025-08-13T06:06:42.172Z" },
|
| 328 |
+
{ url = "https://files.pythonhosted.org/packages/cd/2b/1c1fea423d3f550f44c5ec3f14d8400919b49c285c3bd146687c63e40186/faiss_cpu-1.12.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bcc6587dee21e17430fb49ddc5200625d6f5e1de2bdf436f14827bad4ca78d19", size = 31432677, upload-time = "2025-08-13T06:06:44.348Z" },
|
| 329 |
+
{ url = "https://files.pythonhosted.org/packages/de/d2/3483e92a02f30e2d8491a256f470f54b7f5483266dfe09126d28741d31ec/faiss_cpu-1.12.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:b80e5965f001822cc99ec65c715169af1b70bdae72eccd573520a2dec485b3ee", size = 9765504, upload-time = "2025-08-13T06:06:46.567Z" },
|
| 330 |
+
{ url = "https://files.pythonhosted.org/packages/ce/2f/d97792211a9bd84b8d6b1dcaa1dcd69ac11e026c6ef19c641b6a87e31025/faiss_cpu-1.12.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:98279f1b4876ef9902695a329b81a99002782ab6e26def472022009df6f1ac68", size = 24169930, upload-time = "2025-08-13T06:06:48.916Z" },
|
| 331 |
+
{ url = "https://files.pythonhosted.org/packages/ee/b8/b707ca4d88af472509a053c39d3cced53efd19d096b8dff2fadc18c4b82d/faiss_cpu-1.12.0-cp314-cp314-win_amd64.whl", hash = "sha256:11670337f9f5ee9ff3490e30683eea80add060c300cf6f6cb0e8faf3155fd20e", size = 18475400, upload-time = "2025-08-13T06:06:51.233Z" },
|
| 332 |
+
{ url = "https://files.pythonhosted.org/packages/77/11/42e41ddebde4dfe77e36e92d0110b4f733c8640883abffde54f802482deb/faiss_cpu-1.12.0-cp314-cp314-win_arm64.whl", hash = "sha256:7ac1c8b53609b5c722ab60f1749260a7cb3c72fdfb720a0e3033067e73591da5", size = 8281229, upload-time = "2025-08-13T06:06:53.735Z" },
|
| 333 |
+
{ url = "https://files.pythonhosted.org/packages/1c/9a/8ae5bbeabe70eb673c37fc7c77e2e476746331afb6654b2df97d8b6d380d/faiss_cpu-1.12.0-cp314-cp314t-macosx_13_0_x86_64.whl", hash = "sha256:110b21b7bb4c93c4f1a5eb2ffb8ef99dcdb4725f8ab2e5cd161324e4d981f204", size = 8087247, upload-time = "2025-08-13T06:06:55.407Z" },
|
| 334 |
+
{ url = "https://files.pythonhosted.org/packages/f4/df/b3d79098860b67b126da351788c04ac243c29718dadc4a678a6f5e7209c0/faiss_cpu-1.12.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:82eb5515ce72be9a43f4cf74447a0d090e014231981df91aff7251204b506fbf", size = 3411043, upload-time = "2025-08-13T06:06:56.983Z" },
|
| 335 |
+
{ url = "https://files.pythonhosted.org/packages/bc/2f/b1a2a03dd3cce22ff9fc434aa3c7390125087260c1d1349311da36eaa432/faiss_cpu-1.12.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:754eef89cdf2b35643df6b0923a5a098bdfecf63b5f4bd86c385042ee511b287", size = 3801789, upload-time = "2025-08-13T06:06:58.688Z" },
|
| 336 |
+
{ url = "https://files.pythonhosted.org/packages/a3/a8/16ad0c6a966e93d04bfd5248d2be1d8b5849842b0e2611c5ecd26fcaf036/faiss_cpu-1.12.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7285c71c8f5e9c58b55175f5f74c78c518c52c421a88a430263f34e3e31f719c", size = 31231388, upload-time = "2025-08-13T06:07:00.55Z" },
|
| 337 |
+
{ url = "https://files.pythonhosted.org/packages/62/a1/9c16eca0b8f8b13c32c47a5e4ff7a4bc0ca3e7d263140312088811230871/faiss_cpu-1.12.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:84a50d7a2f711f79cc8b65aa28956dba6435e47b71a38b2daea44c94c9b8e458", size = 9737605, upload-time = "2025-08-13T06:07:03.018Z" },
|
| 338 |
+
{ url = "https://files.pythonhosted.org/packages/a8/4a/2c2d615078c9d816a836fb893aaef551ad152f2eb00bc258698273c240c0/faiss_cpu-1.12.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:7f3e0a14e4edec6a3959a9f51afccb89e863138f184ff2cc24c13f9ad788740b", size = 23922880, upload-time = "2025-08-13T06:07:05.099Z" },
|
| 339 |
+
{ url = "https://files.pythonhosted.org/packages/30/aa/99b8402a4dac678794f13f8f4f29d666c2ef0a91594418147f47034ebc81/faiss_cpu-1.12.0-cp314-cp314t-win_amd64.whl", hash = "sha256:8b3239cc371df6826ac43c62ac04eec7cc497bedb43f681fcd8ea494f520ddbb", size = 18750661, upload-time = "2025-08-13T06:07:07.551Z" },
|
| 340 |
+
{ url = "https://files.pythonhosted.org/packages/a3/a2/b546e9a20ba157eb2fbe141289f1752f157ee6d932899f4853df4ded6d4b/faiss_cpu-1.12.0-cp314-cp314t-win_arm64.whl", hash = "sha256:58b23456db725ee1bd605a6135d2ef55b2ac3e0b6fe873fd99a909e8ef4bd0ff", size = 8302032, upload-time = "2025-08-13T06:07:09.602Z" },
|
| 341 |
+
]
|
| 342 |
+
|
| 343 |
[[package]]
|
| 344 |
name = "filelock"
|
| 345 |
version = "3.32.3"
|
|
|
|
| 409 |
{ name = "onnx" },
|
| 410 |
{ name = "onnxruntime" },
|
| 411 |
]
|
| 412 |
+
tabdpt = [
|
| 413 |
+
{ name = "tabdpt" },
|
| 414 |
+
]
|
| 415 |
+
tabicl = [
|
| 416 |
+
{ name = "tabicl" },
|
| 417 |
+
]
|
| 418 |
tabpfn = [
|
| 419 |
{ name = "tabpfn" },
|
| 420 |
]
|
|
|
|
| 435 |
{ name = "onnx", marker = "extra == 'onnx'", specifier = ">=1.16,<2" },
|
| 436 |
{ name = "onnxruntime", marker = "extra == 'onnx'", specifier = ">=1.18,<2" },
|
| 437 |
{ name = "pyyaml", specifier = ">=6" },
|
| 438 |
+
{ name = "tabdpt", marker = "extra == 'tabdpt'", specifier = ">=1.3.1,<2" },
|
| 439 |
+
{ name = "tabicl", marker = "extra == 'tabicl'", specifier = ">=2.2,<3" },
|
| 440 |
{ name = "tabpfn", marker = "extra == 'tabpfn'", specifier = ">=9.2,<10" },
|
| 441 |
{ name = "torch", marker = "platform_machine != 'x86_64' or sys_platform != 'linux'", specifier = "==2.14.1" },
|
| 442 |
{ name = "torch", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'", specifier = "==2.14.1", index = "https://download.pytorch.org/whl/cu130" },
|
| 443 |
{ name = "uproot", specifier = ">=5.0" },
|
| 444 |
]
|
| 445 |
+
provides-extras = ["onnx", "tabpfn", "tabicl", "tabdpt"]
|
| 446 |
|
| 447 |
[package.metadata.requires-dev]
|
| 448 |
dev = [
|
|
|
|
| 767 |
version = "13.1.1.3"
|
| 768 |
source = { registry = "https://pypi.org/simple" }
|
| 769 |
dependencies = [
|
| 770 |
+
{ name = "nvidia-cuda-nvrtc", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
|
| 771 |
]
|
| 772 |
wheels = [
|
| 773 |
{ url = "https://files.pythonhosted.org/packages/a7/a1/0bd24ee8c8d03adac032fd2909426a00c88f8c57961b1277ded97f91119f/nvidia_cublas-13.1.1.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:b7a210458267ac818974c53038fbec2e969d5c99f305ab15c72522fa9f001dd5", size = 542848918, upload-time = "2026-04-08T18:46:22.985Z" },
|
|
|
|
| 806 |
version = "9.24.0.43"
|
| 807 |
source = { registry = "https://pypi.org/simple" }
|
| 808 |
dependencies = [
|
| 809 |
+
{ name = "nvidia-cublas", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
|
| 810 |
]
|
| 811 |
wheels = [
|
| 812 |
{ url = "https://files.pythonhosted.org/packages/ca/30/7c257e3d5cb4fecb147b93895c66e29c93f8e76d74b45bb418ff0587c4ec/nvidia_cudnn_cu13-9.24.0.43-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:a6812a554a1ff0413e9c52b84c26c050380649ab9615f9c16bded368ce9f421f", size = 650976863, upload-time = "2026-07-02T16:23:39.248Z" },
|
|
|
|
| 818 |
version = "12.0.0.61"
|
| 819 |
source = { registry = "https://pypi.org/simple" }
|
| 820 |
dependencies = [
|
| 821 |
+
{ name = "nvidia-nvjitlink", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
|
| 822 |
]
|
| 823 |
wheels = [
|
| 824 |
{ url = "https://files.pythonhosted.org/packages/8b/ae/f417a75c0259e85c1d2f83ca4e960289a5f814ed0cea74d18c353d3e989d/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2708c852ef8cd89d1d2068bdbece0aa188813a0c934db3779b9b1faa8442e5f5", size = 214053554, upload-time = "2025-09-04T08:31:38.196Z" },
|
|
|
|
| 848 |
version = "12.0.4.66"
|
| 849 |
source = { registry = "https://pypi.org/simple" }
|
| 850 |
dependencies = [
|
| 851 |
+
{ name = "nvidia-cublas", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
|
| 852 |
+
{ name = "nvidia-cusparse", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
|
| 853 |
+
{ name = "nvidia-nvjitlink", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
|
| 854 |
]
|
| 855 |
wheels = [
|
| 856 |
{ url = "https://files.pythonhosted.org/packages/c8/c3/b30c9e935fc01e3da443ec0116ed1b2a009bb867f5324d3f2d7e533e776b/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:02c2457eaa9e39de20f880f4bd8820e6a1cfb9f9a34f820eb12a155aa5bc92d2", size = 223467760, upload-time = "2025-09-04T08:33:04.222Z" },
|
|
|
|
| 862 |
version = "12.6.3.3"
|
| 863 |
source = { registry = "https://pypi.org/simple" }
|
| 864 |
dependencies = [
|
| 865 |
+
{ name = "nvidia-nvjitlink", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
|
| 866 |
]
|
| 867 |
wheels = [
|
| 868 |
{ url = "https://files.pythonhosted.org/packages/f8/94/5c26f33738ae35276672f12615a64bd008ed5be6d1ebcb23579285d960a9/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:80bcc4662f23f1054ee334a15c72b8940402975e0eab63178fc7e670aa59472c", size = 162155568, upload-time = "2025-09-04T08:33:42.864Z" },
|
|
|
|
| 914 |
{ url = "https://files.pythonhosted.org/packages/a8/64/3708a90d1ebe202ffdeb7185f878a3c84d15c2b2c31858da2ce0583e2def/nvidia_nvtx-13.0.85-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb7780edb6b14107373c835bf8b72e7a178bac7367e23da7acb108f973f157a6", size = 148878, upload-time = "2025-09-04T08:28:53.627Z" },
|
| 915 |
]
|
| 916 |
|
| 917 |
+
[[package]]
|
| 918 |
+
name = "omegaconf"
|
| 919 |
+
version = "2.4.0"
|
| 920 |
+
source = { registry = "https://pypi.org/simple" }
|
| 921 |
+
dependencies = [
|
| 922 |
+
{ name = "pyyaml" },
|
| 923 |
+
]
|
| 924 |
+
sdist = { url = "https://files.pythonhosted.org/packages/cc/10/1cabf3188a4d9684a06568c7fd76f9536eede625733c8399cf1f5337b83e/omegaconf-2.4.0.tar.gz", hash = "sha256:7e219a70bea371da250285b9c01c7db027bc421722799c2cc4ff14bc4a2a753b", size = 3544508, upload-time = "2026-10-09T12:44:02.326Z" }
|
| 925 |
+
wheels = [
|
| 926 |
+
{ url = "https://files.pythonhosted.org/packages/4f/bd/0d9f8247b0819385213d09564eda5c52d3673313c2414d51f72883ec24cd/omegaconf-2.4.0-py3-none-any.whl", hash = "sha256:f141a664c777582405e3cc6e0545ecc79429e79f844995b1a6e71c35946a025c", size = 254152, upload-time = "2026-10-09T12:43:59.764Z" },
|
| 927 |
+
]
|
| 928 |
+
|
| 929 |
[[package]]
|
| 930 |
name = "onnx"
|
| 931 |
version = "1.22.0"
|
|
|
|
| 1063 |
{ url = "https://files.pythonhosted.org/packages/19/c7/5f7c636ec43e0c545e28d1f1db71990108306f7bdcb89f069ba97e428e7f/protobuf-7.35.1-py3-none-any.whl", hash = "sha256:4bc97768d8fe4ad6743c8a19403e314511ed9f6d13205b687e52421c023ac1b9", size = 171659, upload-time = "2026-06-11T21:55:39.155Z" },
|
| 1064 |
]
|
| 1065 |
|
| 1066 |
+
[[package]]
|
| 1067 |
+
name = "psutil"
|
| 1068 |
+
version = "7.2.2"
|
| 1069 |
+
source = { registry = "https://pypi.org/simple" }
|
| 1070 |
+
sdist = { url = "https://files.pythonhosted.org/packages/aa/c6/d1ddf4abb55e93cebc4f2ed8b5d6dbad109ecb8d63748dd2b20ab5e57ebe/psutil-7.2.2.tar.gz", hash = "sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372", size = 493740, upload-time = "2026-01-28T18:14:54.428Z" }
|
| 1071 |
+
wheels = [
|
| 1072 |
+
{ url = "https://files.pythonhosted.org/packages/81/69/ef179ab5ca24f32acc1dac0c247fd6a13b501fd5534dbae0e05a1c48b66d/psutil-7.2.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:eed63d3b4d62449571547b60578c5b2c4bcccc5387148db46e0c2313dad0ee00", size = 130664, upload-time = "2026-01-28T18:15:09.469Z" },
|
| 1073 |
+
{ url = "https://files.pythonhosted.org/packages/7b/64/665248b557a236d3fa9efc378d60d95ef56dd0a490c2cd37dafc7660d4a9/psutil-7.2.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7b6d09433a10592ce39b13d7be5a54fbac1d1228ed29abc880fb23df7cb694c9", size = 131087, upload-time = "2026-01-28T18:15:11.724Z" },
|
| 1074 |
+
{ url = "https://files.pythonhosted.org/packages/d5/2e/e6782744700d6759ebce3043dcfa661fb61e2fb752b91cdeae9af12c2178/psutil-7.2.2-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fa4ecf83bcdf6e6c8f4449aff98eefb5d0604bf88cb883d7da3d8d2d909546a", size = 182383, upload-time = "2026-01-28T18:15:13.445Z" },
|
| 1075 |
+
{ url = "https://files.pythonhosted.org/packages/57/49/0a41cefd10cb7505cdc04dab3eacf24c0c2cb158a998b8c7b1d27ee2c1f5/psutil-7.2.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e452c464a02e7dc7822a05d25db4cde564444a67e58539a00f929c51eddda0cf", size = 185210, upload-time = "2026-01-28T18:15:16.002Z" },
|
| 1076 |
+
{ url = "https://files.pythonhosted.org/packages/dd/2c/ff9bfb544f283ba5f83ba725a3c5fec6d6b10b8f27ac1dc641c473dc390d/psutil-7.2.2-cp314-cp314t-win_amd64.whl", hash = "sha256:c7663d4e37f13e884d13994247449e9f8f574bc4655d509c3b95e9ec9e2b9dc1", size = 141228, upload-time = "2026-01-28T18:15:18.385Z" },
|
| 1077 |
+
{ url = "https://files.pythonhosted.org/packages/f2/fc/f8d9c31db14fcec13748d373e668bc3bed94d9077dbc17fb0eebc073233c/psutil-7.2.2-cp314-cp314t-win_arm64.whl", hash = "sha256:11fe5a4f613759764e79c65cf11ebdf26e33d6dd34336f8a337aa2996d71c841", size = 136284, upload-time = "2026-01-28T18:15:19.912Z" },
|
| 1078 |
+
{ url = "https://files.pythonhosted.org/packages/e7/36/5ee6e05c9bd427237b11b3937ad82bb8ad2752d72c6969314590dd0c2f6e/psutil-7.2.2-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486", size = 129090, upload-time = "2026-01-28T18:15:22.168Z" },
|
| 1079 |
+
{ url = "https://files.pythonhosted.org/packages/80/c4/f5af4c1ca8c1eeb2e92ccca14ce8effdeec651d5ab6053c589b074eda6e1/psutil-7.2.2-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979", size = 129859, upload-time = "2026-01-28T18:15:23.795Z" },
|
| 1080 |
+
{ url = "https://files.pythonhosted.org/packages/b5/70/5d8df3b09e25bce090399cf48e452d25c935ab72dad19406c77f4e828045/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9", size = 155560, upload-time = "2026-01-28T18:15:25.976Z" },
|
| 1081 |
+
{ url = "https://files.pythonhosted.org/packages/63/65/37648c0c158dc222aba51c089eb3bdfa238e621674dc42d48706e639204f/psutil-7.2.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e", size = 156997, upload-time = "2026-01-28T18:15:27.794Z" },
|
| 1082 |
+
{ url = "https://files.pythonhosted.org/packages/8e/13/125093eadae863ce03c6ffdbae9929430d116a246ef69866dad94da3bfbc/psutil-7.2.2-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8", size = 148972, upload-time = "2026-01-28T18:15:29.342Z" },
|
| 1083 |
+
{ url = "https://files.pythonhosted.org/packages/04/78/0acd37ca84ce3ddffaa92ef0f571e073faa6d8ff1f0559ab1272188ea2be/psutil-7.2.2-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc", size = 148266, upload-time = "2026-01-28T18:15:31.597Z" },
|
| 1084 |
+
{ url = "https://files.pythonhosted.org/packages/b4/90/e2159492b5426be0c1fef7acba807a03511f97c5f86b3caeda6ad92351a7/psutil-7.2.2-cp37-abi3-win_amd64.whl", hash = "sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988", size = 137737, upload-time = "2026-01-28T18:15:33.849Z" },
|
| 1085 |
+
{ url = "https://files.pythonhosted.org/packages/8c/c7/7bb2e321574b10df20cbde462a94e2b71d05f9bbda251ef27d104668306a/psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee", size = 134617, upload-time = "2026-01-28T18:15:36.514Z" },
|
| 1086 |
+
]
|
| 1087 |
+
|
| 1088 |
[[package]]
|
| 1089 |
name = "pydantic"
|
| 1090 |
version = "2.14.1"
|
|
|
|
| 1423 |
{ url = "https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", size = 6299353, upload-time = "2025-04-27T18:04:59.103Z" },
|
| 1424 |
]
|
| 1425 |
|
| 1426 |
+
[[package]]
|
| 1427 |
+
name = "tabdpt"
|
| 1428 |
+
version = "1.3.1"
|
| 1429 |
+
source = { registry = "https://pypi.org/simple" }
|
| 1430 |
+
dependencies = [
|
| 1431 |
+
{ name = "faiss-cpu" },
|
| 1432 |
+
{ name = "huggingface-hub" },
|
| 1433 |
+
{ name = "numpy" },
|
| 1434 |
+
{ name = "omegaconf" },
|
| 1435 |
+
{ name = "safetensors" },
|
| 1436 |
+
{ name = "scikit-learn" },
|
| 1437 |
+
{ name = "scipy" },
|
| 1438 |
+
{ name = "torch", version = "2.14.1", source = { registry = "https://pypi.org/simple" }, marker = "platform_machine != 'x86_64' or sys_platform != 'linux'" },
|
| 1439 |
+
{ name = "torch", version = "2.14.1+cu130", source = { registry = "https://download.pytorch.org/whl/cu130" }, marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
|
| 1440 |
+
{ name = "tqdm" },
|
| 1441 |
+
]
|
| 1442 |
+
sdist = { url = "https://files.pythonhosted.org/packages/68/f0/cda7df9f41705c162d075255fec8e9e53d63aa9eee0bb419a01a56fe8f53/tabdpt-1.3.1.tar.gz", hash = "sha256:42e3e5c13ededb3ff7e6ab128d6491de5537642f9bba7a66da54701bdb7b99ff", size = 33832, upload-time = "2026-09-21T18:22:02.147Z" }
|
| 1443 |
+
wheels = [
|
| 1444 |
+
{ url = "https://files.pythonhosted.org/packages/f6/1c/fc711dd8ec3bcf15c3c3c9320a35cc2f798c4f3de3bf7e59aef6e7b7263d/tabdpt-1.3.1-py3-none-any.whl", hash = "sha256:b3ad31443cf20d6ee0d2d9398c61f09d3d2e7b796fb2f9346a716ddf576e933f", size = 27204, upload-time = "2026-09-21T18:22:01.192Z" },
|
| 1445 |
+
]
|
| 1446 |
+
|
| 1447 |
+
[[package]]
|
| 1448 |
+
name = "tabicl"
|
| 1449 |
+
version = "2.2.0"
|
| 1450 |
+
source = { registry = "https://pypi.org/simple" }
|
| 1451 |
+
dependencies = [
|
| 1452 |
+
{ name = "einops" },
|
| 1453 |
+
{ name = "huggingface-hub" },
|
| 1454 |
+
{ name = "numpy" },
|
| 1455 |
+
{ name = "psutil" },
|
| 1456 |
+
{ name = "scikit-learn" },
|
| 1457 |
+
{ name = "scipy" },
|
| 1458 |
+
{ name = "torch", version = "2.14.1", source = { registry = "https://pypi.org/simple" }, marker = "platform_machine != 'x86_64' or sys_platform != 'linux'" },
|
| 1459 |
+
{ name = "torch", version = "2.14.1+cu130", source = { registry = "https://download.pytorch.org/whl/cu130" }, marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
|
| 1460 |
+
{ name = "tqdm" },
|
| 1461 |
+
]
|
| 1462 |
+
sdist = { url = "https://files.pythonhosted.org/packages/d4/81/ee11ac72b33c2b87f7337c99238fd2700128791edceb558b63bb30556b73/tabicl-2.2.0.tar.gz", hash = "sha256:dfc41f2acd9603e4402cedae9e42f9922cfb8cfc9fcf7901d773e26609e47e21", size = 268084, upload-time = "2026-09-02T15:13:38.59Z" }
|
| 1463 |
+
wheels = [
|
| 1464 |
+
{ url = "https://files.pythonhosted.org/packages/77/22/bbac2df399b8ed5a699bde38496ebb129a045c5d160fe52786727de932f3/tabicl-2.2.0-py3-none-any.whl", hash = "sha256:c80b8ec2719be26bc0bf1dd758a2361f58019d8f7877038b785529a425789ce4", size = 310448, upload-time = "2026-09-02T15:13:36.629Z" },
|
| 1465 |
+
]
|
| 1466 |
+
|
| 1467 |
[[package]]
|
| 1468 |
name = "tabpfn"
|
| 1469 |
version = "9.2.0"
|