ho22joshua Claude Opus 5.5 commited on
Commit
8dff74c
·
1 Parent(s): 9402518

feat: tabular in-context models, LoRA meta-training, pretraining objectives and probes

Browse files

In-context models (tabular.py):
- TabPFN-3.5 meta-trained on the PFN's Delphes tasks (model.name: tabpfn,
task.name: pfn), LoRA on its 24 in-context blocks by default
(model.lora_rank; 0 trains all weights); checkpoints hold only trained
weights. configs/delphes_tabpfn.yaml.
- TabICLv2 (BSD-3) as released and meta-trained the same way
(configs/gamgam_tabicl.yaml, configs/delphes_tabicl.yaml); LoRA is a
weight parametrization, as TabICL reads fused projections directly.
- TabDPT (Apache-2.0) as released, with optional per-event retrieval of the
context (configs/gamgam_tabdpt.yaml).
- Row options: azimuths from the leading object (model.relative_phi), pair
masses and Delta R (model.pair_objects); encoder embeddings instead of or
beside rows (model.tabular_inputs).
- model.tabpfn_checkpoint; evaluate records setup/inference seconds and
CUDA peak memory.

Pretraining (all label-free, encoders load as JEPA's):
- contrastive.py: SimCLR with physics augmentations
(configs/delphes_contrastive.yaml).
- JEPA with kinematic targets, masked object modelling
(model.jepa_target: features; configs/delphes_masked.yaml).
- probe command (probe.py): k-NN and linear probes of frozen encoders.

Development on Apple GPUs: train.device auto picks MPS; float64 metrics move
to the CPU on MPS (PyTorch 2.14 zeroes a combined device/dtype cast);
pytest --device mps.

scripts/label_efficiency.py runs label-efficiency sweeps.

Tested on fixtures on CPU and MPS (114 passed; CUDA-only tests not run);
first TabPFN-3.5 GamGam numbers in docs/design.md. Existing models'
features, splits, losses and metrics are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

configs/delphes_contrastive.yaml ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Contrastive (SimCLR) pretraining of the object transformer on the 12
2
+ # Delphes samples of configs/delphes_jepa.yaml, without their labels (same
3
+ # event cache): two random views of every event (rotation, eta flip, energy
4
+ # scale, object dropout) are pulled together (src/gnn4colliders/contrastive.py).
5
+ # Fine-tune with configs/gamgam_jepa_finetune.yaml
6
+ # model.pretrained=outputs/delphes_contrastive/last.pt, or compare encoders
7
+ # without fine-tuning: gnn4colliders probe configs/gamgam_jepa_finetune.yaml
8
+ # data.train_events=1000 split=validation model.pretrained=...
9
+ #
10
+ # gnn4colliders train configs/delphes_contrastive.yaml
11
+ name: delphes_contrastive
12
+
13
+ data:
14
+ hf_repo: HWresearch/Delphes
15
+ hf_revision: 76a6c362bfba8e766ba7255a5c08a55257f78d0e
16
+ files:
17
+ - {path: samples/higgs/top-associated/tth/ttH_NLO_inc.root, label: 0}
18
+ - {path: samples/higgs/top-associated/thjb/tHjb_NLO_inc.root, label: 1}
19
+ - {path: samples/higgs/ggf/ggF_NLO_inc.root, label: 2}
20
+ - {path: samples/higgs/vbf/VBF_NLO_inc.root, label: 3}
21
+ - {path: samples/higgs/vh/wh/WH_NLO_inc.root, label: 4}
22
+ - {path: samples/higgs/vh/zh/ZH_NLO_inc.root, label: 5}
23
+ - {path: samples/photon/ttyy/ttyy.root, label: 6}
24
+ - {path: samples/top/multitop/tttt.root, label: 7}
25
+ - {path: samples/top/single-top/SingleT_schan.root, label: 8}
26
+ - {path: samples/top/ttbar/ttbar.root, label: 9}
27
+ - {path: samples/top/ttv/ttW.root, label: 10}
28
+ - {path: samples/top/multitop/ttt.root, label: 11}
29
+ tree_name: output
30
+ feature_branches:
31
+ - [jet_pt, ele_pt, mu_pt, ph_pt, MET_met]
32
+ - [jet_eta, ele_eta, mu_eta, ph_eta, 0]
33
+ - [jet_phi, ele_phi, mu_phi, ph_phi, MET_phi]
34
+ - CALC_E
35
+ - [jet_btag, 0, 0, 0, 0]
36
+ - [0, ele_charge, mu_charge, 0, 0]
37
+ - NODE_TYPE
38
+ object_types: [vector, vector, vector, vector, single]
39
+ scales: [0.1, 1, 1, 0.1, 1, 1, 1]
40
+ # Number is the event index; Number % 10 is the fold. Pretraining trained
41
+ # on folds 2-9 and tested on 0-1; here fold 0 validates and fold 1 tests.
42
+ fold_var: Number
43
+ fold_modulus: 10
44
+ train_folds: [2, 3, 4, 5, 6, 7, 8, 9]
45
+ validation_folds: [0]
46
+ test_folds: [1]
47
+ # JEPA ignores labels and weights; the cache is that of
48
+ # configs/delphes_pretraining.yaml.
49
+ weight_var: weight
50
+ cache: outputs/delphes_pretraining/events.pt
51
+ batch_size: 1024
52
+ prepare_workers: 8
53
+ loader_workers: 4
54
+ pin_memory: true
55
+
56
+ model:
57
+ name: contrastive
58
+ backbone: transformer
59
+ # The encoder; configs/gamgam_jepa_finetune.yaml must match it.
60
+ hid_size: 128
61
+ n_layers: 4
62
+ n_heads: 8
63
+ ffn_size: 512
64
+ dropout: 0.0
65
+ pair_bias: true
66
+ temperature: 0.1
67
+ scale_jitter: 0.1
68
+ object_dropout: 0.1
69
+
70
+ task:
71
+ name: contrastive
72
+
73
+ train:
74
+ max_epochs: 30
75
+ optimizer: adamw
76
+ learning_rate: 5.0e-4
77
+ weight_decay: 0.05
78
+ lr_decay: 0.95
79
+ early_stopping: null
80
+ # Fine-tune from last.pt.
81
+ best_metric: loss
configs/delphes_masked.yaml ADDED
@@ -0,0 +1,78 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Masked object modelling of the object transformer on the 12 Delphes
2
+ # samples of configs/delphes_jepa.yaml, without their labels (same event
3
+ # cache): JEPA's masking and predictor, but the predictor predicts the hidden
4
+ # objects' kinematics instead of their target encodings (model.jepa_target:
5
+ # features; src/gnn4colliders/jepa.py). Fine-tune or probe with
6
+ # configs/gamgam_jepa_finetune.yaml model.pretrained=outputs/delphes_masked/last.pt.
7
+ #
8
+ # gnn4colliders train configs/delphes_masked.yaml
9
+ name: delphes_masked
10
+
11
+ data:
12
+ hf_repo: HWresearch/Delphes
13
+ hf_revision: 76a6c362bfba8e766ba7255a5c08a55257f78d0e
14
+ files:
15
+ - {path: samples/higgs/top-associated/tth/ttH_NLO_inc.root, label: 0}
16
+ - {path: samples/higgs/top-associated/thjb/tHjb_NLO_inc.root, label: 1}
17
+ - {path: samples/higgs/ggf/ggF_NLO_inc.root, label: 2}
18
+ - {path: samples/higgs/vbf/VBF_NLO_inc.root, label: 3}
19
+ - {path: samples/higgs/vh/wh/WH_NLO_inc.root, label: 4}
20
+ - {path: samples/higgs/vh/zh/ZH_NLO_inc.root, label: 5}
21
+ - {path: samples/photon/ttyy/ttyy.root, label: 6}
22
+ - {path: samples/top/multitop/tttt.root, label: 7}
23
+ - {path: samples/top/single-top/SingleT_schan.root, label: 8}
24
+ - {path: samples/top/ttbar/ttbar.root, label: 9}
25
+ - {path: samples/top/ttv/ttW.root, label: 10}
26
+ - {path: samples/top/multitop/ttt.root, label: 11}
27
+ tree_name: output
28
+ feature_branches:
29
+ - [jet_pt, ele_pt, mu_pt, ph_pt, MET_met]
30
+ - [jet_eta, ele_eta, mu_eta, ph_eta, 0]
31
+ - [jet_phi, ele_phi, mu_phi, ph_phi, MET_phi]
32
+ - CALC_E
33
+ - [jet_btag, 0, 0, 0, 0]
34
+ - [0, ele_charge, mu_charge, 0, 0]
35
+ - NODE_TYPE
36
+ object_types: [vector, vector, vector, vector, single]
37
+ scales: [0.1, 1, 1, 0.1, 1, 1, 1]
38
+ # Number is the event index; Number % 10 is the fold. Pretraining trained
39
+ # on folds 2-9 and tested on 0-1; here fold 0 validates and fold 1 tests.
40
+ fold_var: Number
41
+ fold_modulus: 10
42
+ train_folds: [2, 3, 4, 5, 6, 7, 8, 9]
43
+ validation_folds: [0]
44
+ test_folds: [1]
45
+ # JEPA ignores labels and weights; the cache is that of
46
+ # configs/delphes_pretraining.yaml.
47
+ weight_var: weight
48
+ cache: outputs/delphes_pretraining/events.pt
49
+ batch_size: 1024
50
+ prepare_workers: 8
51
+ loader_workers: 4
52
+ pin_memory: true
53
+
54
+ model:
55
+ name: jepa
56
+ jepa_target: features
57
+ # The encoder; configs/gamgam_jepa_finetune.yaml must match it.
58
+ hid_size: 128
59
+ n_layers: 4
60
+ n_heads: 8
61
+ ffn_size: 512
62
+ dropout: 0.0
63
+ pair_bias: true
64
+ predictor_layers: 2
65
+ mask_fraction: 0.4
66
+
67
+ task:
68
+ name: jepa
69
+
70
+ train:
71
+ max_epochs: 30
72
+ optimizer: adamw
73
+ learning_rate: 5.0e-4
74
+ weight_decay: 0.05
75
+ lr_decay: 0.95
76
+ early_stopping: null
77
+ # Fine-tune from last.pt.
78
+ best_metric: loss
configs/delphes_tabicl.yaml ADDED
@@ -0,0 +1,89 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Meta-training of the TabICL network (src/gnn4colliders/tabular.py; BSD-3,
2
+ # so meta-trained weights can be shared freely) on the classification tasks
3
+ # of configs/delphes_pfn.yaml, on the fixed-width event rows of
4
+ # configs/gamgam_tabicl.yaml. By default TabICL is frozen and its in-context
5
+ # blocks learn LoRA updates (model.lora_rank; 0 trains all 28M weights).
6
+ # task.prior_classes=[1,6,7,8,9,10,11] leaves out the five processes of the
7
+ # GamGam task. Evaluate in context with
8
+ # gnn4colliders evaluate configs/gamgam_tabicl.yaml split=validation \
9
+ # checkpoint=outputs/delphes_tabicl/best.pt
10
+ # Needs `uv sync --extra tabicl`.
11
+ #
12
+ # gnn4colliders train configs/delphes_tabicl.yaml
13
+ name: delphes_tabicl
14
+
15
+ data:
16
+ hf_repo: HWresearch/Delphes
17
+ hf_revision: 76a6c362bfba8e766ba7255a5c08a55257f78d0e
18
+ files:
19
+ - {path: samples/higgs/top-associated/tth/ttH_NLO_inc.root, label: 0}
20
+ - {path: samples/higgs/top-associated/thjb/tHjb_NLO_inc.root, label: 1}
21
+ - {path: samples/higgs/ggf/ggF_NLO_inc.root, label: 2}
22
+ - {path: samples/higgs/vbf/VBF_NLO_inc.root, label: 3}
23
+ - {path: samples/higgs/vh/wh/WH_NLO_inc.root, label: 4}
24
+ - {path: samples/higgs/vh/zh/ZH_NLO_inc.root, label: 5}
25
+ - {path: samples/photon/ttyy/ttyy.root, label: 6}
26
+ - {path: samples/top/multitop/tttt.root, label: 7}
27
+ - {path: samples/top/single-top/SingleT_schan.root, label: 8}
28
+ - {path: samples/top/ttbar/ttbar.root, label: 9}
29
+ - {path: samples/top/ttv/ttW.root, label: 10}
30
+ - {path: samples/top/multitop/ttt.root, label: 11}
31
+ tree_name: output
32
+ feature_branches:
33
+ - [jet_pt, ele_pt, mu_pt, ph_pt, MET_met]
34
+ - [jet_eta, ele_eta, mu_eta, ph_eta, 0]
35
+ - [jet_phi, ele_phi, mu_phi, ph_phi, MET_phi]
36
+ - CALC_E
37
+ - [jet_btag, 0, 0, 0, 0]
38
+ - [0, ele_charge, mu_charge, 0, 0]
39
+ - NODE_TYPE
40
+ object_types: [vector, vector, vector, vector, single]
41
+ scales: [0.1, 1, 1, 0.1, 1, 1, 1]
42
+ # Number is the event index; Number % 10 is the fold. Pretraining trained
43
+ # on folds 2-9 and tested on 0-1; here fold 0 validates and fold 1 tests.
44
+ fold_var: Number
45
+ fold_modulus: 10
46
+ train_folds: [2, 3, 4, 5, 6, 7, 8, 9]
47
+ validation_folds: [0]
48
+ test_folds: [1]
49
+ # Tasks use the labels, not the weights; the cache is that of
50
+ # configs/delphes_pretraining.yaml.
51
+ weight_var: weight
52
+ cache: outputs/delphes_pretraining/events.pt
53
+ # Events per step (tasks of context and query events). TabICL (28M
54
+ # weights) recomputes activations in the backward pass.
55
+ batch_size: 4096
56
+ prepare_workers: 8
57
+ loader_workers: 4
58
+ pin_memory: true
59
+
60
+ model:
61
+ name: tabicl
62
+ tabicl_checkpoint: tabicl-classifier-v2-20260212.ckpt
63
+ # Leading jets, electrons, muons, photons and MET in a table row.
64
+ slots: [6, 2, 2, 3, 1]
65
+ # As in configs/gamgam_tabicl.yaml, which evaluates this model.
66
+ relative_phi: false
67
+ pair_objects: 0
68
+ max_classes: 10
69
+ lora_rank: 8
70
+
71
+ task:
72
+ name: pfn
73
+ # The prior: classes per task, context events (log-uniform) and queries.
74
+ prior_classes: null
75
+ task_classes: [2, 10]
76
+ context_events: [16, 2048]
77
+ query_events: 256
78
+ energy_jitter: 0.2
79
+ steps_per_epoch: 500
80
+ validation_steps: 50
81
+
82
+ train:
83
+ max_epochs: 20
84
+ optimizer: adamw
85
+ learning_rate: 1.0e-4
86
+ weight_decay: 0.01
87
+ lr_decay: 0.95
88
+ early_stopping: null
89
+ best_metric: loss
configs/delphes_tabpfn.yaml ADDED
@@ -0,0 +1,89 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Meta-training of the TabPFN-3.5 network (src/gnn4colliders/tabular.py) on
2
+ # the classification tasks of configs/delphes_pfn.yaml, on the fixed-width
3
+ # event rows of configs/gamgam_tabpfn.yaml. By default TabPFN is frozen and
4
+ # its in-context blocks learn LoRA updates (model.lora_rank; 0 trains all
5
+ # weights). task.prior_classes=[1,6,7,8,9,10,11] leaves out the five
6
+ # processes of the GamGam task. Evaluate in context with
7
+ # gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation \
8
+ # checkpoint=outputs/delphes_tabpfn/best.pt
9
+ # Needs `uv sync --extra tabpfn` and the TabPFN-3.5 weights (docs/usage.md).
10
+ #
11
+ # gnn4colliders train configs/delphes_tabpfn.yaml
12
+ name: delphes_tabpfn
13
+
14
+ data:
15
+ hf_repo: HWresearch/Delphes
16
+ hf_revision: 76a6c362bfba8e766ba7255a5c08a55257f78d0e
17
+ files:
18
+ - {path: samples/higgs/top-associated/tth/ttH_NLO_inc.root, label: 0}
19
+ - {path: samples/higgs/top-associated/thjb/tHjb_NLO_inc.root, label: 1}
20
+ - {path: samples/higgs/ggf/ggF_NLO_inc.root, label: 2}
21
+ - {path: samples/higgs/vbf/VBF_NLO_inc.root, label: 3}
22
+ - {path: samples/higgs/vh/wh/WH_NLO_inc.root, label: 4}
23
+ - {path: samples/higgs/vh/zh/ZH_NLO_inc.root, label: 5}
24
+ - {path: samples/photon/ttyy/ttyy.root, label: 6}
25
+ - {path: samples/top/multitop/tttt.root, label: 7}
26
+ - {path: samples/top/single-top/SingleT_schan.root, label: 8}
27
+ - {path: samples/top/ttbar/ttbar.root, label: 9}
28
+ - {path: samples/top/ttv/ttW.root, label: 10}
29
+ - {path: samples/top/multitop/ttt.root, label: 11}
30
+ tree_name: output
31
+ feature_branches:
32
+ - [jet_pt, ele_pt, mu_pt, ph_pt, MET_met]
33
+ - [jet_eta, ele_eta, mu_eta, ph_eta, 0]
34
+ - [jet_phi, ele_phi, mu_phi, ph_phi, MET_phi]
35
+ - CALC_E
36
+ - [jet_btag, 0, 0, 0, 0]
37
+ - [0, ele_charge, mu_charge, 0, 0]
38
+ - NODE_TYPE
39
+ object_types: [vector, vector, vector, vector, single]
40
+ scales: [0.1, 1, 1, 0.1, 1, 1, 1]
41
+ # Number is the event index; Number % 10 is the fold. Pretraining trained
42
+ # on folds 2-9 and tested on 0-1; here fold 0 validates and fold 1 tests.
43
+ fold_var: Number
44
+ fold_modulus: 10
45
+ train_folds: [2, 3, 4, 5, 6, 7, 8, 9]
46
+ validation_folds: [0]
47
+ test_folds: [1]
48
+ # Tasks use the labels, not the weights; the cache is that of
49
+ # configs/delphes_pretraining.yaml.
50
+ weight_var: weight
51
+ cache: outputs/delphes_pretraining/events.pt
52
+ # Events per step (tasks of context and query events): one task of the
53
+ # largest context. TabPFN recomputes activations in the backward pass.
54
+ batch_size: 2304
55
+ prepare_workers: 8
56
+ loader_workers: 4
57
+ pin_memory: true
58
+
59
+ model:
60
+ name: tabpfn
61
+ tabpfn_version: v3.5
62
+ tabpfn_checkpoint: null
63
+ # Leading jets, electrons, muons, photons and MET in a table row.
64
+ slots: [6, 2, 2, 3, 1]
65
+ # As in configs/gamgam_tabpfn.yaml, which evaluates this model.
66
+ relative_phi: false
67
+ pair_objects: 0
68
+ max_classes: 10
69
+ lora_rank: 8
70
+
71
+ task:
72
+ name: pfn
73
+ # The prior: classes per task, context events (log-uniform) and queries.
74
+ prior_classes: null
75
+ task_classes: [2, 10]
76
+ context_events: [16, 2048]
77
+ query_events: 256
78
+ energy_jitter: 0.2
79
+ steps_per_epoch: 500
80
+ validation_steps: 50
81
+
82
+ train:
83
+ max_epochs: 20
84
+ optimizer: adamw
85
+ learning_rate: 1.0e-4
86
+ weight_decay: 0.01
87
+ lr_decay: 0.95
88
+ early_stopping: null
89
+ best_metric: loss
configs/gamgam_tabdpt.yaml ADDED
@@ -0,0 +1,91 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # The GamGam 5-class task classified in context by TabDPT (Layer 6;
2
+ # Apache-2.0 weights from Hugging Face), with no training: data.train_events
3
+ # labelled training events, an equal number per class, are the context of
4
+ # every prediction (src/gnn4colliders/tabular.py). TabDPT is trained on real
5
+ # tables; with model.tabdpt_context_size each event's context is its nearest
6
+ # context events (retrieval), which allows contexts beyond what fits at once.
7
+ # Needs `uv sync --extra tabdpt`. Uses the cache of configs/gamgam.yaml.
8
+ #
9
+ # gnn4colliders evaluate configs/gamgam_tabdpt.yaml split=validation
10
+ name: gamgam_tabdpt
11
+
12
+ data:
13
+ hf_repo: ho22joshua/atlas_opendata
14
+ hf_revision: d1bcc9ccaa304ea09e11f7d724342568f72a0d6e
15
+ files:
16
+ - path: GamGam/2020/MC/mc_341081.ttH125_gamgam.GamGam.root
17
+ label: 0
18
+ size_bytes: 217819076
19
+ sha256: c498974926f6c55143d17fc4d3d6fa69bd3718ff92cf7d5cad5c3d67a836c7b7
20
+ - path: GamGam/2020/MC/mc_343981.ggH125_gamgam.GamGam.root
21
+ label: 1
22
+ size_bytes: 218418347
23
+ sha256: 05b3e6709e8836a7b862a7fde09086b3a9d4d494d62a6984d5df39d8daad13eb
24
+ - path: GamGam/2020/MC/mc_345041.VBFH125_gamgam.GamGam.root
25
+ label: 2
26
+ size_bytes: 108616148
27
+ sha256: 92334fce3bffc9eb6e0745dd6ef354d5ce33da764858511b8f9ca49b862b44e2
28
+ - path: GamGam/2020/MC/mc_345318.WpH125J_Wincl_gamgam.GamGam.root
29
+ label: 3
30
+ size_bytes: 29757932
31
+ sha256: f255a9c3c6935f976f788b6f32d9a8f6bf576eb12ecd03a6f3fd0a5dea9d1248
32
+ - path: GamGam/2020/MC/mc_345319.ZH125J_Zincl_gamgam.GamGam.root
33
+ label: 4
34
+ size_bytes: 58701804
35
+ sha256: c85623b04ac0176c64acdb2ecf8e992a63824cfc430a421df742319518ff6748
36
+ class_names: [ttH, ggF, VBF, WH, ZH]
37
+ tree_name: mini
38
+ # Electrons and muons share the lep_* collection; b-tagging uses the
39
+ # MV2c10 70% efficiency working point of the Open Data documentation.
40
+ derived_branches:
41
+ ele_pt: {source: lep_pt, where: {lep_type: 11}}
42
+ ele_eta: {source: lep_eta, where: {lep_type: 11}}
43
+ ele_phi: {source: lep_phi, where: {lep_type: 11}}
44
+ ele_charge: {source: lep_charge, where: {lep_type: 11}}
45
+ mu_pt: {source: lep_pt, where: {lep_type: 13}}
46
+ mu_eta: {source: lep_eta, where: {lep_type: 13}}
47
+ mu_phi: {source: lep_phi, where: {lep_type: 13}}
48
+ mu_charge: {source: lep_charge, where: {lep_type: 13}}
49
+ jet_btag: {source: jet_MV2c10, above: 0.8244273}
50
+ # Object types (node types) as in pretraining: jet, electron, muon, photon, MET.
51
+ feature_branches:
52
+ - [jet_pt, ele_pt, mu_pt, photon_pt, met_et]
53
+ - [jet_eta, ele_eta, mu_eta, photon_eta, 0]
54
+ - [jet_phi, ele_phi, mu_phi, photon_phi, met_phi]
55
+ - CALC_E
56
+ - [jet_btag, 0, 0, 0, 0]
57
+ - [0, ele_charge, mu_charge, 0, 0]
58
+ - NODE_TYPE
59
+ object_types: [vector, vector, vector, vector, single]
60
+ # Momenta are in MeV; 1e-4 gives the pretraining unit of 10 GeV.
61
+ scales: [1.0e-4, 1, 1, 1.0e-4, 1, 1, 1]
62
+ # eventNumber % 10 gives the paper's 80:10:10 train:validation:test split.
63
+ fold_var: eventNumber
64
+ fold_modulus: 10
65
+ train_folds: [0, 1, 2, 3, 4, 5, 6, 7]
66
+ validation_folds: [8]
67
+ test_folds: [9]
68
+ # Labelled events (equal per class) that form the context.
69
+ train_events: 1000
70
+ # Unweighted: each event counts once and classes are balanced by the
71
+ # per-class mean loss (23% of ttH events have mcWeight = -1).
72
+ weight_var: null
73
+ cache: outputs/gamgam/events.pt
74
+ # Events classified per TabDPT call.
75
+ batch_size: 8192
76
+ prepare_workers: 8
77
+ loader_workers: 4
78
+ pin_memory: true
79
+
80
+ model:
81
+ name: tabdpt
82
+ out_size: 5
83
+ tabdpt_estimators: 8
84
+ tabdpt_context_size: null
85
+ # Leading jets, electrons, muons, photons and MET in a table row.
86
+ slots: [6, 2, 2, 3, 1]
87
+ relative_phi: false
88
+ pair_objects: 0
89
+
90
+ task:
91
+ name: multiclass
configs/gamgam_tabicl.yaml ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # The GamGam 5-class task classified in context by TabICLv2 (Inria; BSD-3
2
+ # weights from Hugging Face), with no training: data.train_events labelled
3
+ # training events, an equal number per class, are the context of every
4
+ # prediction (src/gnn4colliders/tabular.py). Needs `uv sync --extra tabicl`.
5
+ # Uses the cache of configs/gamgam.yaml. The rows are those of
6
+ # configs/gamgam_tabpfn.yaml; model.relative_phi=true and
7
+ # model.pair_objects=4 add the physics features.
8
+ #
9
+ # gnn4colliders evaluate configs/gamgam_tabicl.yaml split=validation
10
+ #
11
+ # With checkpoint=outputs/delphes_tabicl/best.pt, the TabICL network
12
+ # meta-trained by configs/delphes_tabicl.yaml (model.lora_rank and
13
+ # model.max_classes must match it) classifies with its own forward pass.
14
+ name: gamgam_tabicl
15
+
16
+ data:
17
+ hf_repo: ho22joshua/atlas_opendata
18
+ hf_revision: d1bcc9ccaa304ea09e11f7d724342568f72a0d6e
19
+ files:
20
+ - path: GamGam/2020/MC/mc_341081.ttH125_gamgam.GamGam.root
21
+ label: 0
22
+ size_bytes: 217819076
23
+ sha256: c498974926f6c55143d17fc4d3d6fa69bd3718ff92cf7d5cad5c3d67a836c7b7
24
+ - path: GamGam/2020/MC/mc_343981.ggH125_gamgam.GamGam.root
25
+ label: 1
26
+ size_bytes: 218418347
27
+ sha256: 05b3e6709e8836a7b862a7fde09086b3a9d4d494d62a6984d5df39d8daad13eb
28
+ - path: GamGam/2020/MC/mc_345041.VBFH125_gamgam.GamGam.root
29
+ label: 2
30
+ size_bytes: 108616148
31
+ sha256: 92334fce3bffc9eb6e0745dd6ef354d5ce33da764858511b8f9ca49b862b44e2
32
+ - path: GamGam/2020/MC/mc_345318.WpH125J_Wincl_gamgam.GamGam.root
33
+ label: 3
34
+ size_bytes: 29757932
35
+ sha256: f255a9c3c6935f976f788b6f32d9a8f6bf576eb12ecd03a6f3fd0a5dea9d1248
36
+ - path: GamGam/2020/MC/mc_345319.ZH125J_Zincl_gamgam.GamGam.root
37
+ label: 4
38
+ size_bytes: 58701804
39
+ sha256: c85623b04ac0176c64acdb2ecf8e992a63824cfc430a421df742319518ff6748
40
+ class_names: [ttH, ggF, VBF, WH, ZH]
41
+ tree_name: mini
42
+ # Electrons and muons share the lep_* collection; b-tagging uses the
43
+ # MV2c10 70% efficiency working point of the Open Data documentation.
44
+ derived_branches:
45
+ ele_pt: {source: lep_pt, where: {lep_type: 11}}
46
+ ele_eta: {source: lep_eta, where: {lep_type: 11}}
47
+ ele_phi: {source: lep_phi, where: {lep_type: 11}}
48
+ ele_charge: {source: lep_charge, where: {lep_type: 11}}
49
+ mu_pt: {source: lep_pt, where: {lep_type: 13}}
50
+ mu_eta: {source: lep_eta, where: {lep_type: 13}}
51
+ mu_phi: {source: lep_phi, where: {lep_type: 13}}
52
+ mu_charge: {source: lep_charge, where: {lep_type: 13}}
53
+ jet_btag: {source: jet_MV2c10, above: 0.8244273}
54
+ # Object types (node types) as in pretraining: jet, electron, muon, photon, MET.
55
+ feature_branches:
56
+ - [jet_pt, ele_pt, mu_pt, photon_pt, met_et]
57
+ - [jet_eta, ele_eta, mu_eta, photon_eta, 0]
58
+ - [jet_phi, ele_phi, mu_phi, photon_phi, met_phi]
59
+ - CALC_E
60
+ - [jet_btag, 0, 0, 0, 0]
61
+ - [0, ele_charge, mu_charge, 0, 0]
62
+ - NODE_TYPE
63
+ object_types: [vector, vector, vector, vector, single]
64
+ # Momenta are in MeV; 1e-4 gives the pretraining unit of 10 GeV.
65
+ scales: [1.0e-4, 1, 1, 1.0e-4, 1, 1, 1]
66
+ # eventNumber % 10 gives the paper's 80:10:10 train:validation:test split.
67
+ fold_var: eventNumber
68
+ fold_modulus: 10
69
+ train_folds: [0, 1, 2, 3, 4, 5, 6, 7]
70
+ validation_folds: [8]
71
+ test_folds: [9]
72
+ # Labelled events (equal per class) that form the context.
73
+ train_events: 1000
74
+ # Unweighted: each event counts once and classes are balanced by the
75
+ # per-class mean loss (23% of ttH events have mcWeight = -1).
76
+ weight_var: null
77
+ cache: outputs/gamgam/events.pt
78
+ # Events classified per TabICL call.
79
+ batch_size: 8192
80
+ prepare_workers: 8
81
+ loader_workers: 4
82
+ pin_memory: true
83
+
84
+ model:
85
+ name: tabicl
86
+ out_size: 5
87
+ tabicl_checkpoint: tabicl-classifier-v2-20260212.ckpt
88
+ tabicl_estimators: 8
89
+ # Leading jets, electrons, muons, photons and MET in a table row.
90
+ slots: [6, 2, 2, 3, 1]
91
+ relative_phi: false
92
+ pair_objects: 0
93
+
94
+ task:
95
+ name: multiclass
configs/gamgam_tabpfn.yaml CHANGED
@@ -6,6 +6,10 @@
6
  # configs/gamgam.yaml.
7
  #
8
  # gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation
 
 
 
 
9
  name: gamgam_tabpfn
10
 
11
  data:
@@ -83,6 +87,10 @@ model:
83
  tabpfn_estimators: 4
84
  # Leading jets, electrons, muons, photons and MET in a table row.
85
  slots: [6, 2, 2, 3, 1]
 
 
 
 
86
 
87
  task:
88
  name: multiclass
 
6
  # configs/gamgam.yaml.
7
  #
8
  # gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation
9
+ #
10
+ # With checkpoint=outputs/delphes_tabpfn/best.pt, the TabPFN network
11
+ # meta-trained by configs/delphes_tabpfn.yaml (model.lora_rank and
12
+ # model.max_classes must match it) classifies with its own forward pass.
13
  name: gamgam_tabpfn
14
 
15
  data:
 
87
  tabpfn_estimators: 4
88
  # Leading jets, electrons, muons, photons and MET in a table row.
89
  slots: [6, 2, 2, 3, 1]
90
+ # Physics features: azimuths from the leading object's, and the mass and
91
+ # Delta R of every pair of the leading pair_objects objects.
92
+ relative_phi: false
93
+ pair_objects: 0
94
 
95
  task:
96
  name: multiclass
docs/design.md CHANGED
@@ -8,16 +8,20 @@
8
  | `data.py` | ROOT reading, Hugging Face download, derived branches, object features, `PackedEvents` and its cache |
9
  | `loader.py` | `EventLoader`: deterministic shuffling, DDP sharding, collation workers |
10
  | `graphs.py` | fully connected `TensorGraph` batches, edge features, CUDA-graph padding buffers |
11
- | `sequences.py` | padded token batches for the transformer and Deep Sets |
12
  | `models.py` | `EdgeNetwork`, `FineTunedEdgeNetwork`, `ObjectTransformer`, `TransformerClassifier`, `DeepSets`, checkpoint loading |
13
- | `jepa.py` | JEPA pretraining of the transformer or edge network encoder |
14
- | `tasks.py` | multiclass and binary losses and metrics, the JEPA loss |
 
 
 
 
15
  | `train.py` | `Trainer`, early stopping, checkpoints |
16
  | `cuda_graphs.py` | captured training steps |
17
  | `distributed.py` | `torchrun` process group and collectives |
18
  | `inference.py` | model outputs over a loader, prediction files |
19
  | `export.py` | ONNX export |
20
- | `cli.py` | the five commands |
21
 
22
  ## Data flow
23
 
@@ -108,7 +112,31 @@ The edge network's JEPA encodings collapse without the variance term
108
  (`target_spread` 0.09 against about 1 with it), and none of the edge
109
  network variants yet beats training from scratch.
110
 
111
- ### In-context models: TabPFN and a collider PFN
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
112
 
113
  Both classify a dataset's events given `data.train_events` labelled
114
  training events (an equal number per class) as context, without training on
@@ -131,17 +159,66 @@ that dataset; `cli.in_context_forward` builds them for `evaluate` and
131
  scale within `task.energy_jitter`. The GamGam processes are among the
132
  Delphes samples; `task.prior_classes=[1,6,7,8,9,10,11]` leaves them out.
133
 
134
- Status: implemented and tested on CPU fixtures (TabPFN-v2 included); no GPU
135
- run yet. Planned study, in interactive allocations:
136
-
137
- 1. TabPFN-3.5 on GamGam validation with `train_events` 100, 1000, 10000 and
138
- 50000 (AUC and time).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
139
  2. Meta-train `configs/delphes_pfn.yaml` for about an hour, on all 12
140
  processes and with the GamGam processes left out, then GamGam in context
141
- with the same `train_events`.
 
 
142
  3. Supervised references with the same `train_events`: the edge network
143
  from scratch and from `Pretrained_GNN`, the transformer from scratch and
144
- from JEPA.
145
  4. One table of validation AUC against labelled events here.
146
 
147
  The loss is a weighted mean per class present in the batch, averaged over
 
8
  | `data.py` | ROOT reading, Hugging Face download, derived branches, object features, `PackedEvents` and its cache |
9
  | `loader.py` | `EventLoader`: deterministic shuffling, DDP sharding, collation workers |
10
  | `graphs.py` | fully connected `TensorGraph` batches, edge features, CUDA-graph padding buffers |
11
+ | `sequences.py` | padded token batches for the transformer and Deep Sets, event augmentations |
12
  | `models.py` | `EdgeNetwork`, `FineTunedEdgeNetwork`, `ObjectTransformer`, `TransformerClassifier`, `DeepSets`, checkpoint loading |
13
+ | `jepa.py` | JEPA pretraining of the transformer or edge network encoder (encoding or kinematic targets), event embeddings |
14
+ | `contrastive.py` | contrastive (SimCLR) pretraining of the same encoders |
15
+ | `probe.py` | k-NN and linear probes of frozen encoders |
16
+ | `pfn.py` | our prior-fitted network over events and the task prior of meta-training |
17
+ | `tabular.py` | event rows and encoder embeddings for TabPFN, TabICL and TabDPT; LoRA meta-training of TabPFN and TabICL |
18
+ | `tasks.py` | multiclass and binary losses and metrics, the pretraining and PFN losses |
19
  | `train.py` | `Trainer`, early stopping, checkpoints |
20
  | `cuda_graphs.py` | captured training steps |
21
  | `distributed.py` | `torchrun` process group and collectives |
22
  | `inference.py` | model outputs over a loader, prediction files |
23
  | `export.py` | ONNX export |
24
+ | `cli.py` | the six commands |
25
 
26
  ## Data flow
27
 
 
112
  (`target_spread` 0.09 against about 1 with it), and none of the edge
113
  network variants yet beats training from scratch.
114
 
115
+ ### Contrastive and masked-object pretraining, and probes
116
+
117
+ Two more label-free objectives pretrain the same encoders on the same
118
+ samples, and their encoders load the same way (`context_encoder`):
119
+
120
+ * `contrastive.py` (`configs/delphes_contrastive.yaml`): SimCLR, as JetCLR
121
+ and RS3L for jets. Two views of each event (random phi rotation, eta
122
+ flip, momentum scale within `model.scale_jitter`, each object but the
123
+ first dropped with `model.object_dropout`) are embedded and projected;
124
+ the NT-Xent loss at `model.temperature` contrasts them with the other
125
+ events of the batch. `embedding_spread` watches for collapse.
126
+ * `model.jepa_target: features` (`configs/delphes_masked.yaml`): masked
127
+ object modelling (Golling et al. 2024). JEPA's masking and predictor
128
+ predict the hidden objects' kinematics instead of their target
129
+ encodings, with no target encoder: the comparison with JEPA isolates
130
+ predicting in embedding space.
131
+
132
+ `gnn4colliders probe` compares encoders without fine-tuning (as jBOT):
133
+ k-NN (cosine, `model.probe_neighbours`) and linear (L-BFGS logistic
134
+ regression, `model.probe_l2`) classifiers of frozen event embeddings,
135
+ trained on `data.train_events` events. A randomly initialized encoder
136
+ (`model.pretrained=null`) is the reference; supervised checkpoints probe
137
+ too. Not yet run beyond fixtures.
138
+
139
+ ### In-context models: TabPFN, TabICL, TabDPT and a collider PFN
140
 
141
  Both classify a dataset's events given `data.train_events` labelled
142
  training events (an equal number per class) as context, without training on
 
159
  scale within `task.energy_jitter`. The GamGam processes are among the
160
  Delphes samples; `task.prior_classes=[1,6,7,8,9,10,11]` leaves them out.
161
 
162
+ * `tabular.py` also runs TabICLv2 (Qu et al., ICML 2026; BSD-3, open
163
+ pretraining code) on the same rows, as a second tabular in-context model.
164
+ Rows can measure azimuths from the leading object's (`model.relative_phi`)
165
+ and end with the invariant mass and Delta R of every pair of the
166
+ `model.pair_objects` leading objects by pt (energies are `pt cosh(eta)`,
167
+ so objects are massless; a pair with MET gives a transverse-like mass).
168
+ Both are off by default, so the default rows are unchanged.
169
+ * `tabular.py` also runs TabDPT (Ma et al. 2024; Apache-2.0), trained on
170
+ real tables rather than a synthetic prior; with
171
+ `model.tabdpt_context_size` each event's context is its nearest context
172
+ events (faiss retrieval), beyond what fits in one pass. It computes in
173
+ float64 in places (on a Mac it runs on the CPU), and on macOS faiss and
174
+ torch crash together unless `OMP_NUM_THREADS=1`.
175
+ * Tabular models can see, instead of or beside the rows, the event
176
+ embeddings of a pretrained object encoder (`model.tabular_inputs:
177
+ embeddings` or `both`, the encoder of `model.backbone` and
178
+ `model.pretrained`): physics-aware embeddings under a generic tabular
179
+ prior. In meta-training the encoder trains with the LoRA updates unless
180
+ `model.freeze_backbone`.
181
+ * `tabular.py` also meta-trains the TabPFN-3.5 network itself on the
182
+ PFN's tasks (`model.name: tabpfn`, `task.name: pfn`), on the rows above
183
+ (built after the augmentation), starting from Prior Labs' weights. Calling
184
+ the network directly skips TabPFN's preprocessing and ensembling, which
185
+ on the GamGam fixtures cost nothing (validation AUC 0.852 against 0.843
186
+ with `predict_proba`, 61 events). By default the network is frozen and
187
+ the four attention projections and two MLP layers of each of its 24
188
+ in-context blocks get rank-8 LoRA updates (2.8M parameters, zero at the
189
+ start, so training starts from TabPFN as released); `model.lora_rank=0`
190
+ trains all weights. In context, the context's keys and values are cached
191
+ (the same logits as one pass over context and queries, to 2e-5). It
192
+ separates what Prior Labs' prior and what collider tasks contribute:
193
+ TabPFN as released, TabPFN meta-trained here, and our PFN. TabICL
194
+ (`configs/delphes_tabicl.yaml`; BSD-3, so meta-trained weights can be
195
+ shared) meta-trains the same way. Its attention reads fused projection
196
+ weights directly, so LoRA is a parametrization of the weights (12 blocks,
197
+ 0.59M of 28M parameters at rank 8), for TabPFN too. TabICL's network takes
198
+ no NaN: rows are standardized by the context's statistics, absent values
199
+ at the mean, clipped at 8 sigma. On the fixtures this gives AUC 0.827
200
+ against 0.820 with its classifier, and the cached in-context forward
201
+ equals the training forward only in float32 (with mixed precision on the
202
+ CPU, AUC 0.59).
203
+
204
+ Status: implemented and tested on fixtures on the CPU and on MPS. First
205
+ result, TabPFN-3.5 as released (default rows, 4 estimators) on the 246,922
206
+ GamGam validation events on an M2 Max: AUC 80.26 with 100 labelled events
207
+ (3.1 min) and 86.93 with 1000 (3.9 min). Full-scale runs are for
208
+ Perlmutter. `scripts/label_efficiency.py` runs the sweeps below. Planned
209
+ study, in interactive allocations:
210
+
211
+ 1. TabPFN-3.5, TabICLv2 and TabDPT on GamGam validation with
212
+ `train_events` 100, 1000, 10000 and 50000 (AUC and time), with and
213
+ without the physics row features, and with encoder embeddings.
214
  2. Meta-train `configs/delphes_pfn.yaml` for about an hour, on all 12
215
  processes and with the GamGam processes left out, then GamGam in context
216
+ with the same `train_events`. The same for TabPFN-3.5 and TabICL
217
+ (`configs/delphes_tabpfn.yaml`, `configs/delphes_tabicl.yaml`; LoRA, and
218
+ `model.lora_rank=0`).
219
  3. Supervised references with the same `train_events`: the edge network
220
  from scratch and from `Pretrained_GNN`, the transformer from scratch and
221
+ from JEPA, contrastive and masked-object pretraining (also probed).
222
  4. One table of validation AUC against labelled events here.
223
 
224
  The loss is a weighted mean per class present in the batch, averaged over
docs/usage.md CHANGED
@@ -13,7 +13,10 @@ Python 3.14 and PyTorch 2.14 (CUDA 13), managed by [uv](https://docs.astral.sh/u
13
  ```bash
14
  uv sync --dev # includes pytest, ruff and the ONNX packages
15
  uv sync --dev --extra tabpfn # also TabPFN (configs/gamgam_tabpfn.yaml)
 
 
16
  uv run pytest # about 2 minutes; GPU tests run when CUDA is available
 
17
  ```
18
 
19
  ## Commands
@@ -27,6 +30,8 @@ uv run gnn4colliders train configs/gamgam.yaml # -> run_dir/{best,last}.pt
27
  uv run gnn4colliders evaluate configs/gamgam.yaml # -> run_dir/test_metrics.json
28
  uv run gnn4colliders predict configs/gamgam.yaml # -> run_dir/test_predictions.npz
29
  uv run gnn4colliders export configs/gamgam.yaml # -> run_dir/model.onnx
 
 
30
  ```
31
 
32
  `train` writes, in `run_dir` (default `outputs/<name>`):
@@ -39,6 +44,11 @@ uv run gnn4colliders export configs/gamgam.yaml # -> run_dir/model.onnx
39
  `train.resume=<run_dir>/last.pt`), and `epoch_NNNN.pt` every
40
  `train.save_every` epochs.
41
 
 
 
 
 
 
42
  `evaluate`, `predict` and `export` use `checkpoint` (default `best.pt`, else
43
  `last.pt`) on `split` (default `test`); `output` overrides the output file.
44
  Predictions hold the logits, scores, predicted class, label, weight, fold,
@@ -57,10 +67,16 @@ adds scores to a copy of a ROOT tree.
57
  | `configs/delphes_pretraining.yaml` | 12-class pretraining on the Delphes samples |
58
  | `configs/delphes_jepa.yaml` | JEPA pretraining of the transformer encoder on the same samples, without labels (same event cache) |
59
  | `configs/delphes_jepa_edge.yaml` | the same for the edge network (fine-tune with `configs/gamgam_finetune.yaml model.pretrained=outputs/delphes_jepa_edge/last.pt`) |
60
- | `configs/gamgam_jepa_finetune.yaml` | GamGam with the JEPA-pretrained encoder (`model.pretrained=null`: from scratch) |
 
 
61
  | `configs/gamgam_tabpfn.yaml` | GamGam classified in context by TabPFN-3.5, no training (`evaluate` only) |
 
 
62
  | `configs/delphes_pfn.yaml` | meta-training of our own PFN on tasks drawn from the Delphes samples (same event cache) |
63
  | `configs/gamgam_pfn.yaml` | GamGam classified in context by that PFN, no training (`evaluate` only) |
 
 
64
 
65
  Each file documents its keys; `src/gnn4colliders/config.py` lists every key
66
  with its default. Common overrides:
@@ -110,11 +126,70 @@ uv run gnn4colliders evaluate configs/gamgam_pfn.yaml split=validation data.trai
110
  uv run gnn4colliders train configs/gamgam.yaml data.train_events=1000 # supervised reference
111
  ```
112
 
113
- TabPFN-2.5 and later weights are non-commercial and download after a
114
- one-time login at https://ux.priorlabs.ai: the token is kept in
115
- `~/.cache/tabpfn/auth_token`, or set `TABPFN_TOKEN`. Put the weights on
116
- scratch storage with `TABPFN_MODEL_CACHE_DIR` (on Perlmutter
117
- `$SCRATCH/.cache/tabpfn`). `model.tabpfn_version=v2` needs no login.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
118
 
119
  ### Fine-tuning
120
 
@@ -129,6 +204,10 @@ initializes the edge network or the transformer encoder.
129
 
130
  ## GPUs, Perlmutter and DDP
131
 
 
 
 
 
132
  On a single GPU, `train.fast=true` is about 3x faster than float32 for the
133
  GamGam model (see [design.md](design.md)). Set `OMP_NUM_THREADS` to the CPU
134
  allocation and, on Perlmutter compute nodes, `UV_CACHE_DIR=$SCRATCH/.cache/uv`
 
13
  ```bash
14
  uv sync --dev # includes pytest, ruff and the ONNX packages
15
  uv sync --dev --extra tabpfn # also TabPFN (configs/gamgam_tabpfn.yaml)
16
+ uv sync --dev --extra tabicl # also TabICL (configs/gamgam_tabicl.yaml)
17
+ uv sync --dev --extra tabdpt # also TabDPT (configs/gamgam_tabdpt.yaml)
18
  uv run pytest # about 2 minutes; GPU tests run when CUDA is available
19
+ uv run pytest --device mps # the fixture runs on an Apple GPU
20
  ```
21
 
22
  ## Commands
 
30
  uv run gnn4colliders evaluate configs/gamgam.yaml # -> run_dir/test_metrics.json
31
  uv run gnn4colliders predict configs/gamgam.yaml # -> run_dir/test_predictions.npz
32
  uv run gnn4colliders export configs/gamgam.yaml # -> run_dir/model.onnx
33
+ uv run gnn4colliders probe configs/gamgam_jepa_finetune.yaml \
34
+ model.pretrained=outputs/delphes_jepa/last.pt data.train_events=1000 # -> run_dir/test_probe.json
35
  ```
36
 
37
  `train` writes, in `run_dir` (default `outputs/<name>`):
 
44
  `train.resume=<run_dir>/last.pt`), and `epoch_NNNN.pt` every
45
  `train.save_every` epochs.
46
 
47
+ `probe` fits k-NN and linear classifiers to the frozen event embeddings of
48
+ the encoder of `model.pretrained` (JEPA, contrastive or supervised; random
49
+ without) on `data.train_events` training events and reports their metrics
50
+ on `split`, to compare pretraining without fine-tuning.
51
+
52
  `evaluate`, `predict` and `export` use `checkpoint` (default `best.pt`, else
53
  `last.pt`) on `split` (default `test`); `output` overrides the output file.
54
  Predictions hold the logits, scores, predicted class, label, weight, fold,
 
67
  | `configs/delphes_pretraining.yaml` | 12-class pretraining on the Delphes samples |
68
  | `configs/delphes_jepa.yaml` | JEPA pretraining of the transformer encoder on the same samples, without labels (same event cache) |
69
  | `configs/delphes_jepa_edge.yaml` | the same for the edge network (fine-tune with `configs/gamgam_finetune.yaml model.pretrained=outputs/delphes_jepa_edge/last.pt`) |
70
+ | `configs/delphes_contrastive.yaml` | contrastive (SimCLR) pretraining of the transformer encoder on the same samples (`model.backbone=edge_network` for the edge network) |
71
+ | `configs/delphes_masked.yaml` | masked object modelling: JEPA predicting hidden objects' kinematics |
72
+ | `configs/gamgam_jepa_finetune.yaml` | GamGam with the JEPA-pretrained encoder (`model.pretrained=null`: from scratch); also takes contrastive and masked-object encoders, and `probe` |
73
  | `configs/gamgam_tabpfn.yaml` | GamGam classified in context by TabPFN-3.5, no training (`evaluate` only) |
74
+ | `configs/gamgam_tabicl.yaml` | the same by TabICLv2 (BSD-3 weights, downloaded from Hugging Face on first use) |
75
+ | `configs/gamgam_tabdpt.yaml` | the same by TabDPT (Apache-2.0 weights; `model.tabdpt_context_size` retrieves each event's context) |
76
  | `configs/delphes_pfn.yaml` | meta-training of our own PFN on tasks drawn from the Delphes samples (same event cache) |
77
  | `configs/gamgam_pfn.yaml` | GamGam classified in context by that PFN, no training (`evaluate` only) |
78
+ | `configs/delphes_tabpfn.yaml` | meta-training of TabPFN-3.5 (LoRA by default) on the same tasks; evaluate with `configs/gamgam_tabpfn.yaml checkpoint=outputs/delphes_tabpfn/best.pt` |
79
+ | `configs/delphes_tabicl.yaml` | the same for TabICL; evaluate with `configs/gamgam_tabicl.yaml checkpoint=outputs/delphes_tabicl/best.pt` |
80
 
81
  Each file documents its keys; `src/gnn4colliders/config.py` lists every key
82
  with its default. Common overrides:
 
126
  uv run gnn4colliders train configs/gamgam.yaml data.train_events=1000 # supervised reference
127
  ```
128
 
129
+ `scripts/label_efficiency.py` runs such a sweep over several `N` and
130
+ configs (restartable) and prints validation AUC and seconds per run; the
131
+ metrics JSON records `seconds` (setup, inference) and, on CUDA,
132
+ `peak_memory_gib`:
133
+
134
+ ```bash
135
+ uv run python scripts/label_efficiency.py configs/gamgam_tabpfn.yaml configs/gamgam.yaml \
136
+ --events 100 1000 10000 50000
137
+ ```
138
+
139
+ TabPFN-2.5 and later weights are non-commercial (research and internal
140
+ benchmarking are allowed). tabpfn downloads them after a one-time login at
141
+ https://ux.priorlabs.ai (token in `~/.cache/tabpfn/auth_token` or
142
+ `TABPFN_TOKEN`), but a checkpoint already in its cache directory loads
143
+ without the login, so on compute nodes download it from Hugging Face:
144
+
145
+ ```bash
146
+ export TABPFN_MODEL_CACHE_DIR=$SCRATCH/.cache/tabpfn # default ~/Library/Caches/tabpfn on macOS
147
+ uvx --from huggingface_hub hf download Prior-Labs/tabpfn_3_5 config.json \
148
+ tabpfn-v3.5-20260909.safetensors tabpfn-v3.5-20260909_multiclass.safetensors \
149
+ --local-dir $TABPFN_MODEL_CACHE_DIR
150
+ ```
151
+
152
+ `model.tabpfn_checkpoint=tabpfn-v3.5-20260909_multiclass.safetensors`
153
+ selects the multiclass checkpoint. `model.tabpfn_version=v2` needs no login.
154
+
155
+ TabDPT retrieval (`model.tabdpt_context_size`) on macOS needs
156
+ `OMP_NUM_THREADS=1`: faiss and torch load separate OpenMP runtimes.
157
+
158
+ Tabular models can use a pretrained encoder's event embeddings instead of
159
+ or beside the rows: `model.tabular_inputs=embeddings` (or `both`) with
160
+ `model.pretrained=outputs/delphes_jepa/last.pt` and the encoder's sizes
161
+ (those of `configs/delphes_jepa.yaml`: `model.hid_size=128 model.n_layers=4
162
+ model.n_heads=8 model.ffn_size=512 model.pair_bias=true`).
163
+
164
+ TabPFN and TabICL classify the same table rows (`model.slots` objects of
165
+ each type). Two options add physics they cannot learn from a few events:
166
+ `model.relative_phi=true` measures every azimuth from the leading object's,
167
+ and `model.pair_objects=4` appends the invariant mass and Delta R of every
168
+ pair of the four leading objects (12 columns). Compare with and without,
169
+ one run name per variant:
170
+
171
+ ```bash
172
+ uv run python scripts/label_efficiency.py configs/gamgam_tabicl.yaml --events 100 1000 \
173
+ --override name=gamgam_tabicl_physics model.relative_phi=true model.pair_objects=4
174
+ ```
175
+
176
+ TabPFN-3.5 can also be meta-trained on the Delphes tasks, like our PFN.
177
+ By default only LoRA updates of its in-context blocks train
178
+ (`model.lora_rank`, 8; `model.lora_rank=0` trains all 219M weights), and
179
+ checkpoints hold only those (33 MB with the optimizer state, against
180
+ 2.6 GB for all weights). With a `checkpoint`, TabPFN is
181
+ evaluated with its own forward pass, not the classifier:
182
+
183
+ ```bash
184
+ uv run gnn4colliders train configs/delphes_tabpfn.yaml
185
+ uv run gnn4colliders evaluate configs/gamgam_tabpfn.yaml split=validation \
186
+ data.train_events=1000 checkpoint=outputs/delphes_tabpfn/best.pt
187
+ ```
188
+
189
+ Activations are recomputed in the backward pass. On a 32 GB Apple M2 Max
190
+ one task of 2048 context events and 256 queries takes about 4.5 s and
191
+ 16 GiB per step with all weights trained, so the Mac is for development;
192
+ meta-train on a GPU node.
193
 
194
  ### Fine-tuning
195
 
 
204
 
205
  ## GPUs, Perlmutter and DDP
206
 
207
+ `train.device: auto` uses CUDA, else an Apple GPU (MPS), else the CPU. MPS
208
+ is for development on fixtures and small runs (no float64: metrics are
209
+ computed on the CPU; no DDP, CUDA graphs or TF32).
210
+
211
  On a single GPU, `train.fast=true` is about 3x faster than float32 for the
212
  GamGam model (see [design.md](design.md)). Set `OMP_NUM_THREADS` to the CPU
213
  allocation and, on Perlmutter compute nodes, `UV_CACHE_DIR=$SCRATCH/.cache/uv`
pyproject.toml CHANGED
@@ -25,6 +25,12 @@ onnx = [
25
  tabpfn = [
26
  "tabpfn>=9.2,<10",
27
  ]
 
 
 
 
 
 
28
 
29
  [project.scripts]
30
  gnn4colliders = "gnn4colliders.cli:main"
 
25
  tabpfn = [
26
  "tabpfn>=9.2,<10",
27
  ]
28
+ tabicl = [
29
+ "tabicl>=2.2,<3",
30
+ ]
31
+ tabdpt = [
32
+ "tabdpt>=1.3.1,<2",
33
+ ]
34
 
35
  [project.scripts]
36
  gnn4colliders = "gnn4colliders.cli:main"
scripts/label_efficiency.py ADDED
@@ -0,0 +1,83 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Validation AUC against the number of labelled training events.
2
+
3
+ For every config and every ``--events`` count N, runs the model with
4
+ ``data.train_events=N`` in ``<out>/<name>/n<N>``: supervised models are
5
+ trained, then evaluated; in-context models (TabPFN, TabICL, PFN) are only evaluated,
6
+ with the N events as context. Runs whose metrics exist are skipped, so an
7
+ interrupted sweep restarts where it stopped. Prints a table of AUC (%) and
8
+ evaluation seconds. The event cache must exist (``gnn4colliders prepare``).
9
+
10
+ uv run python scripts/label_efficiency.py configs/gamgam_tabpfn.yaml \\
11
+ configs/gamgam.yaml --events 100 1000 10000 50000
12
+
13
+ Meta-trained in-context models (the PFN; TabPFN or TabICL after
14
+ meta-training) need ``--override checkpoint=<path>``. Overrides apply to
15
+ every config; ``name=...`` keeps variants apart (run one config at a time
16
+ then, as names must differ):
17
+
18
+ uv run python scripts/label_efficiency.py configs/gamgam_tabpfn.yaml \\
19
+ --override name=gamgam_tabpfn_multiclass \\
20
+ model.tabpfn_checkpoint=tabpfn-v3.5-20260909_multiclass.safetensors
21
+ """
22
+
23
+ from __future__ import annotations
24
+
25
+ import argparse
26
+ import json
27
+ from pathlib import Path
28
+
29
+ from gnn4colliders.cli import main as gnn4colliders
30
+ from gnn4colliders.config import load_config
31
+ from gnn4colliders.models import IN_CONTEXT_MODELS
32
+
33
+
34
+ def _run(config: Path, events: int, out: Path, overrides: list[str]) -> dict:
35
+ name = load_config(config, overrides).name
36
+ run_dir = out / name / f"n{events}"
37
+ metrics = run_dir / "validation_metrics.json"
38
+ if not metrics.exists():
39
+ args = [*overrides, f"data.train_events={events}", f"run_dir={run_dir}"]
40
+ resolved = load_config(config, args)
41
+ if resolved.model.name == "pfn" and resolved.checkpoint is None:
42
+ # The run directory is new: the meta-trained weights are elsewhere.
43
+ raise SystemExit("the PFN needs --override checkpoint=<meta-trained>.pt")
44
+ if resolved.model.name not in IN_CONTEXT_MODELS:
45
+ if gnn4colliders(["train", str(config), *args]) != 0:
46
+ raise SystemExit(f"training {name} with {events} events failed")
47
+ if gnn4colliders(["evaluate", str(config), *args, "split=validation"]) != 0:
48
+ raise SystemExit(f"evaluating {name} with {events} events failed")
49
+ return {"name": name, "events": events, **json.loads(metrics.read_text())}
50
+
51
+
52
+ def main() -> None:
53
+ parser = argparse.ArgumentParser(
54
+ description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
55
+ )
56
+ parser.add_argument("configs", nargs="+", type=Path)
57
+ parser.add_argument("--events", nargs="+", type=int, required=True)
58
+ parser.add_argument("--override", nargs="*", default=[], help="key=value")
59
+ parser.add_argument("--out", type=Path, default=Path("outputs/label_efficiency"))
60
+ args = parser.parse_args()
61
+ names = [load_config(config, args.override).name for config in args.configs]
62
+ if len(set(names)) < len(names):
63
+ raise SystemExit(f"configs share a run name ({names}); override name per run")
64
+ runs = [
65
+ _run(config, events, args.out, args.override)
66
+ for config in args.configs
67
+ for events in args.events
68
+ ]
69
+ print("| model | " + " | ".join(f"N = {n}" for n in args.events) + " |")
70
+ print("|---" * (len(args.events) + 1) + "|")
71
+ for name in dict.fromkeys(run["name"] for run in runs):
72
+ cells = []
73
+ for events in args.events:
74
+ run = next(r for r in runs if r["name"] == name and r["events"] == events)
75
+ auc = 100 * run["metrics"]["auc"]
76
+ seconds = run.get("seconds")
77
+ total = f", {sum(seconds.values()):.0f} s" if seconds else ""
78
+ cells.append(f"{auc:.2f}{total}")
79
+ print(f"| {name} | " + " | ".join(cells) + " |")
80
+
81
+
82
+ if __name__ == "__main__":
83
+ main()
src/gnn4colliders/cli.py CHANGED
@@ -8,10 +8,16 @@ Commands:
8
  * ``evaluate``: metrics of ``checkpoint`` on ``split``, written as JSON.
9
  * ``predict``: per-event scores of ``checkpoint`` on ``split``, as ``.npz``.
10
  * ``export``: ONNX model of ``checkpoint``.
 
 
 
11
 
12
- The in-context models (``model.name: pfn`` or ``tabpfn``) are evaluated
 
13
  without training on the task: their context is ``data.train_events``
14
- labelled training events. The PFN is meta-trained with ``task.name: pfn``.
 
 
15
  """
16
 
17
  from __future__ import annotations
@@ -29,7 +35,14 @@ import torch
29
 
30
  from .config import Config, load_config, save_config
31
  from .data import PackedEvents, load_events, prepare
32
- from .distributed import DistributedContext, barrier, finalize, initialize
 
 
 
 
 
 
 
33
  from .graphs import collate_graph_batch
34
  from .inference import collect_outputs, write_predictions
35
  from .loader import EventLoader
@@ -260,6 +273,20 @@ def _checkpoint(config: Config) -> Path:
260
  return best if best.exists() else config.path / "last.pt"
261
 
262
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
263
  def in_context_forward(
264
  config: Config, events: PackedEvents, device: torch.device
265
  ) -> tuple[Any, str]:
@@ -272,25 +299,32 @@ def in_context_forward(
272
  raise ValueError("in-context models classify with task.name: multiclass")
273
  train = events.split_indices(data.train_folds)
274
  context = events.balanced_subset(train, data.train_events, data.seed)
275
- if model_config.name == "tabpfn":
276
- from .tabular import tabpfn_forward
277
 
278
- forward = tabpfn_forward(
279
  model_config,
280
  events,
281
  context,
282
  num_classes=model_config.out_size,
283
  num_node_types=len(data.object_types),
 
284
  device=device,
285
  seed=config.train.seed,
286
  )
287
- return forward, f"TabPFN {model_config.tabpfn_version}"
288
  from .pfn import in_context_forward as pfn_forward
 
289
 
290
  checkpoint = _checkpoint(config)
291
  model = make_model(config, events, load_pretrained=False)
292
  load_weights(model, load_checkpoint(checkpoint))
293
- forward = pfn_forward(
 
 
 
 
 
294
  model.to(device),
295
  events,
296
  context,
@@ -308,6 +342,9 @@ def run_inference(config: Config, *, predict: bool) -> Path:
308
  torch.backends.cuda.matmul.allow_tf32 = config.train.allow_tf32
309
  events = load_events(config.data.cache)
310
  loader = make_loaders(config, events, (config.split,), context)[config.split]
 
 
 
311
  if config.model.name in IN_CONTEXT_MODELS:
312
  forward, checkpoint = in_context_forward(config, events, context.device)
313
  else:
@@ -315,9 +352,20 @@ def run_inference(config: Config, *, predict: bool) -> Path:
315
  forward = make_model(config, events, load_pretrained=False)
316
  load_weights(forward, load_checkpoint(checkpoint))
317
  forward = forward.to(context.device).eval()
 
 
318
  outputs = collect_outputs(
319
  forward, loader, device=context.device, context=context
320
  )
 
 
 
 
 
 
 
 
 
321
  task = make_task(config)
322
  suffix = "predictions.npz" if predict else "metrics.json"
323
  target = Path(config.output or config.path / f"{config.split}_{suffix}")
@@ -331,6 +379,8 @@ def run_inference(config: Config, *, predict: bool) -> Path:
331
  "split": config.split,
332
  "num_samples": int(outputs.labels.shape[0]),
333
  "metrics": metrics,
 
 
334
  }
335
  target.parent.mkdir(parents=True, exist_ok=True)
336
  target.write_text(json.dumps(summary, indent=2) + "\n")
@@ -359,12 +409,61 @@ def run_export(config: Config) -> Path:
359
  )
360
 
361
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
362
  COMMANDS = {
363
  "prepare": run_prepare,
364
  "train": run_train,
365
  "evaluate": lambda config: run_inference(config, predict=False),
366
  "predict": lambda config: run_inference(config, predict=True),
367
  "export": run_export,
 
368
  }
369
 
370
 
 
8
  * ``evaluate``: metrics of ``checkpoint`` on ``split``, written as JSON.
9
  * ``predict``: per-event scores of ``checkpoint`` on ``split``, as ``.npz``.
10
  * ``export``: ONNX model of ``checkpoint``.
11
+ * ``probe``: k-NN and linear classifiers of the frozen encodings of the
12
+ encoder of ``model.pretrained`` (random without), trained on
13
+ ``data.train_events`` events; their metrics on ``split``, as JSON.
14
 
15
+ The in-context models (``model.name: pfn``, ``tabpfn``, ``tabicl`` or
16
+ ``tabdpt``) are evaluated
17
  without training on the task: their context is ``data.train_events``
18
+ labelled training events. The PFN, TabPFN and TabICL are meta-trained with
19
+ ``task.name: pfn``; TabPFN and TabICL without a ``checkpoint`` are the
20
+ classifiers as released, and TabDPT is only used as released.
21
  """
22
 
23
  from __future__ import annotations
 
35
 
36
  from .config import Config, load_config, save_config
37
  from .data import PackedEvents, load_events, prepare
38
+ from .distributed import (
39
+ DistributedContext,
40
+ barrier,
41
+ finalize,
42
+ initialize,
43
+ resolve_device,
44
+ synchronize,
45
+ )
46
  from .graphs import collate_graph_batch
47
  from .inference import collect_outputs, write_predictions
48
  from .loader import EventLoader
 
273
  return best if best.exists() else config.path / "last.pt"
274
 
275
 
276
+ #: The tabular in-context models used as released (without a meta-trained
277
+ #: ``checkpoint``), and how the metrics name them.
278
+ RELEASED_MODELS = {
279
+ "tabpfn": lambda model: (
280
+ f"TabPFN {model.tabpfn_version} "
281
+ f"({model.tabpfn_checkpoint or 'default checkpoint'})"
282
+ ),
283
+ "tabicl": lambda model: f"TabICL ({model.tabicl_checkpoint})",
284
+ "tabdpt": lambda model: (
285
+ f"TabDPT (context {model.tabdpt_context_size or 'all'} per query)"
286
+ ),
287
+ }
288
+
289
+
290
  def in_context_forward(
291
  config: Config, events: PackedEvents, device: torch.device
292
  ) -> tuple[Any, str]:
 
299
  raise ValueError("in-context models classify with task.name: multiclass")
300
  train = events.split_indices(data.train_folds)
301
  context = events.balanced_subset(train, data.train_events, data.seed)
302
+ if model_config.name in RELEASED_MODELS and config.checkpoint is None:
303
+ from . import tabular
304
 
305
+ forward = getattr(tabular, f"{model_config.name}_forward")(
306
  model_config,
307
  events,
308
  context,
309
  num_classes=model_config.out_size,
310
  num_node_types=len(data.object_types),
311
+ batch_size=data.batch_size,
312
  device=device,
313
  seed=config.train.seed,
314
  )
315
+ return forward, RELEASED_MODELS[model_config.name](model_config)
316
  from .pfn import in_context_forward as pfn_forward
317
+ from .tabular import meta_tabular_forward
318
 
319
  checkpoint = _checkpoint(config)
320
  model = make_model(config, events, load_pretrained=False)
321
  load_weights(model, load_checkpoint(checkpoint))
322
+ build = (
323
+ meta_tabular_forward
324
+ if model_config.name in ("tabpfn", "tabicl")
325
+ else pfn_forward
326
+ )
327
+ forward = build(
328
  model.to(device),
329
  events,
330
  context,
 
342
  torch.backends.cuda.matmul.allow_tf32 = config.train.allow_tf32
343
  events = load_events(config.data.cache)
344
  loader = make_loaders(config, events, (config.split,), context)[config.split]
345
+ if context.device.type == "cuda":
346
+ torch.cuda.reset_peak_memory_stats(context.device)
347
+ start = time.perf_counter()
348
  if config.model.name in IN_CONTEXT_MODELS:
349
  forward, checkpoint = in_context_forward(config, events, context.device)
350
  else:
 
352
  forward = make_model(config, events, load_pretrained=False)
353
  load_weights(forward, load_checkpoint(checkpoint))
354
  forward = forward.to(context.device).eval()
355
+ synchronize(context.device)
356
+ loaded = time.perf_counter()
357
  outputs = collect_outputs(
358
  forward, loader, device=context.device, context=context
359
  )
360
+ synchronize(context.device)
361
+ # Setup: loading the checkpoint, or the in-context model's pass over
362
+ # the context; inference: this rank's events.
363
+ seconds = {"setup": loaded - start, "inference": time.perf_counter() - loaded}
364
+ peak_memory = (
365
+ torch.cuda.max_memory_allocated(context.device) / 2**30
366
+ if context.device.type == "cuda"
367
+ else None
368
+ )
369
  task = make_task(config)
370
  suffix = "predictions.npz" if predict else "metrics.json"
371
  target = Path(config.output or config.path / f"{config.split}_{suffix}")
 
379
  "split": config.split,
380
  "num_samples": int(outputs.labels.shape[0]),
381
  "metrics": metrics,
382
+ "seconds": seconds,
383
+ "peak_memory_gib": peak_memory,
384
  }
385
  target.parent.mkdir(parents=True, exist_ok=True)
386
  target.write_text(json.dumps(summary, indent=2) + "\n")
 
409
  )
410
 
411
 
412
+ def run_probe(config: Config) -> Path:
413
+ from .probe import probe
414
+
415
+ if int(os.environ.get("WORLD_SIZE", "1")) > 1:
416
+ raise ValueError("probe runs in one process")
417
+ data = config.data
418
+ if data.train_events is None:
419
+ raise ValueError("probe needs data.train_events (the probes' training events)")
420
+ events = load_events(data.cache)
421
+ folds = {
422
+ "train": data.train_folds,
423
+ "validation": data.validation_folds,
424
+ "test": data.test_folds,
425
+ }
426
+ train = events.balanced_subset(
427
+ events.split_indices(data.train_folds), data.train_events, data.seed
428
+ )
429
+ queries = events.split_indices(folds[config.split])
430
+ # The probes are not scored on the events they are fitted to.
431
+ queries = queries[~torch.isin(queries, train)]
432
+ device = resolve_device(config.train.device)
433
+ start = time.perf_counter()
434
+ # All the dataset's classes, whatever the config's task (a pretraining
435
+ # config probes too).
436
+ num_classes = int(events.labels.max()) + 1
437
+ logits = probe(
438
+ config, events, train, queries, num_classes=num_classes, device=device
439
+ )
440
+ task = build_task("multiclass")
441
+ labels, weight = events.labels[queries], events.weight[queries]
442
+ summary = {
443
+ "encoder": config.model.pretrained,
444
+ "split": config.split,
445
+ "train_events": int(len(train)),
446
+ "num_samples": int(len(queries)),
447
+ **{
448
+ name: task.metrics(values.cpu(), labels, weight)
449
+ for name, values in logits.items()
450
+ },
451
+ "seconds": time.perf_counter() - start,
452
+ }
453
+ target = Path(config.output or config.path / f"{config.split}_probe.json")
454
+ target.parent.mkdir(parents=True, exist_ok=True)
455
+ target.write_text(json.dumps(summary, indent=2) + "\n")
456
+ print(json.dumps({name: summary[name] for name in logits}, indent=2))
457
+ return target
458
+
459
+
460
  COMMANDS = {
461
  "prepare": run_prepare,
462
  "train": run_train,
463
  "evaluate": lambda config: run_inference(config, predict=False),
464
  "predict": lambda config: run_inference(config, predict=True),
465
  "export": run_export,
466
+ "probe": run_probe,
467
  }
468
 
469
 
src/gnn4colliders/config.py CHANGED
@@ -65,8 +65,9 @@ class DataConfig:
65
 
66
  @dataclass
67
  class ModelConfig:
68
- #: ``edge_network``, ``transformer``, ``deep_sets``, ``jepa``
69
- #: (pretraining), or the in-context models ``pfn`` and ``tabpfn``.
 
70
  name: str = "edge_network"
71
  hid_size: int = 64
72
  out_size: int = 1
@@ -83,10 +84,14 @@ class ModelConfig:
83
  ffn_size: int | None = None
84
  #: Attention bias from object-pair features (Particle Transformer).
85
  pair_bias: bool = False
86
- # JEPA only.
87
- #: The encoder pretrained: ``transformer`` or ``edge_network`` (the
88
- #: sizes above apply to it; the predictor is a transformer).
89
  backbone: str = "transformer"
 
 
 
 
90
  predictor_layers: int = 2
91
  #: Fraction of each event's objects to predict (at least one, and at
92
  #: least one left visible).
@@ -96,6 +101,14 @@ class ModelConfig:
96
  #: Weight of a VICReg variance term on the context encodings (and the
97
  #: edge network's global state) that counters collapse.
98
  variance_weight: float = 0.0
 
 
 
 
 
 
 
 
99
  # PFN only (the encoder is the transformer above).
100
  #: Classes the PFN can tell apart in one task.
101
  max_classes: int = 10
@@ -103,16 +116,41 @@ class ModelConfig:
103
  pfn_layers: int = 6
104
  # TabPFN only.
105
  tabpfn_version: str = "v3.5"
 
 
 
 
106
  tabpfn_estimators: int = 4
 
 
 
 
 
 
 
 
 
 
 
 
107
  #: Objects of each type (in ``data.object_types`` order) in a table row.
108
  slots: list[int] = field(default_factory=lambda: [6, 2, 2, 3, 1])
 
 
 
 
 
 
 
 
109
 
110
 
111
  @dataclass
112
  class TaskConfig:
113
  #: ``multiclass`` (``model.out_size`` classes), ``binary`` (one logit)
114
- #: ``jepa`` (pretraining with ``model.name: jepa``) or ``pfn``
115
- #: (meta-training with ``model.name: pfn``).
 
116
  name: str = "multiclass"
117
  #: Use ``|weight|`` in the loss (signed NLO weights).
118
  absolute_weights: bool = False
@@ -134,7 +172,7 @@ class TaskConfig:
134
 
135
  @dataclass
136
  class TrainConfig:
137
- #: ``auto`` (CUDA when available), ``cpu``, ``cuda`` or ``mps``.
138
  device: str = "auto"
139
  seed: int = 42
140
  max_epochs: int = 100
@@ -186,8 +224,19 @@ class Config:
186
  self.run_dir = self.run_dir or f"outputs/{self.name}"
187
  if self.task.name == "binary" and self.model.out_size != 1:
188
  raise ValueError("binary classification needs model.out_size: 1")
189
- if self.task.name == "pfn" and self.model.name != "pfn":
190
- raise ValueError("task.name: pfn meta-trains model.name: pfn")
 
 
 
 
 
 
 
 
 
 
 
191
  if self.split not in ("train", "validation", "test"):
192
  raise ValueError("split must be train, validation or test")
193
 
 
65
 
66
  @dataclass
67
  class ModelConfig:
68
+ #: ``edge_network``, ``transformer``, ``deep_sets``, ``jepa`` or
69
+ #: ``contrastive`` (pretraining), or the in-context models ``pfn``,
70
+ #: ``tabpfn``, ``tabicl`` and ``tabdpt``.
71
  name: str = "edge_network"
72
  hid_size: int = 64
73
  out_size: int = 1
 
84
  ffn_size: int | None = None
85
  #: Attention bias from object-pair features (Particle Transformer).
86
  pair_bias: bool = False
87
+ # JEPA and contrastive pretraining, probes and tabular embeddings.
88
+ #: The encoder: ``transformer`` or ``edge_network`` (the sizes above
89
+ #: apply to it; JEPA's predictor is a transformer).
90
  backbone: str = "transformer"
91
+ # JEPA only.
92
+ #: What the predictor predicts for hidden objects: their ``encoding`` by
93
+ #: the target encoder, or their kinematic ``features`` (masked objects).
94
+ jepa_target: str = "encoding"
95
  predictor_layers: int = 2
96
  #: Fraction of each event's objects to predict (at least one, and at
97
  #: least one left visible).
 
101
  #: Weight of a VICReg variance term on the context encodings (and the
102
  #: edge network's global state) that counters collapse.
103
  variance_weight: float = 0.0
104
+ # Contrastive only (contrastive.py): the NT-Xent temperature and the
105
+ # views' momentum-scale jitter and object dropout probability.
106
+ temperature: float = 0.1
107
+ scale_jitter: float = 0.1
108
+ object_dropout: float = 0.1
109
+ # Probes of frozen encoders (probe.py).
110
+ probe_neighbours: int = 20
111
+ probe_l2: float = 1e-4
112
  # PFN only (the encoder is the transformer above).
113
  #: Classes the PFN can tell apart in one task.
114
  max_classes: int = 10
 
116
  pfn_layers: int = 6
117
  # TabPFN only.
118
  tabpfn_version: str = "v3.5"
119
+ #: A checkpoint of that version (a path, or a file name in the TabPFN
120
+ #: cache) instead of its default, e.g.
121
+ #: ``tabpfn-v3.5-20260909_multiclass.safetensors``.
122
+ tabpfn_checkpoint: str | None = None
123
  tabpfn_estimators: int = 4
124
+ #: Meta-training of the TabPFN or TabICL network (``task.name: pfn``):
125
+ #: rank of the LoRA updates of its in-context blocks, or 0 to train all
126
+ #: its weights.
127
+ lora_rank: int = 8
128
+ # TabICL only.
129
+ tabicl_checkpoint: str = "tabicl-classifier-v2-20260212.ckpt"
130
+ tabicl_estimators: int = 8
131
+ # TabDPT only.
132
+ tabdpt_estimators: int = 8
133
+ #: Context events per query, its nearest rows (retrieval); null: all.
134
+ tabdpt_context_size: int | None = None
135
+ # The table rows of TabPFN, TabICL and TabDPT (tabular.py).
136
  #: Objects of each type (in ``data.object_types`` order) in a table row.
137
  slots: list[int] = field(default_factory=lambda: [6, 2, 2, 3, 1])
138
+ #: Azimuths measured from that of the event's leading object.
139
+ relative_phi: bool = False
140
+ #: Invariant mass and Delta R of every pair of this many leading objects.
141
+ pair_objects: int = 0
142
+ #: What a tabular model sees of an event: its ``rows``, the
143
+ #: ``embeddings`` of the encoder of ``model.backbone`` and
144
+ #: ``model.pretrained`` (sizes as above), or ``both``.
145
+ tabular_inputs: str = "rows"
146
 
147
 
148
  @dataclass
149
  class TaskConfig:
150
  #: ``multiclass`` (``model.out_size`` classes), ``binary`` (one logit)
151
+ #: ``jepa`` or ``contrastive`` (pretraining, with the same
152
+ #: ``model.name``) or ``pfn``
153
+ #: (meta-training with ``model.name: pfn``, ``tabpfn`` or ``tabicl``).
154
  name: str = "multiclass"
155
  #: Use ``|weight|`` in the loss (signed NLO weights).
156
  absolute_weights: bool = False
 
172
 
173
  @dataclass
174
  class TrainConfig:
175
+ #: ``auto`` (CUDA, else MPS, else CPU), ``cpu``, ``cuda`` or ``mps``.
176
  device: str = "auto"
177
  seed: int = 42
178
  max_epochs: int = 100
 
224
  self.run_dir = self.run_dir or f"outputs/{self.name}"
225
  if self.task.name == "binary" and self.model.out_size != 1:
226
  raise ValueError("binary classification needs model.out_size: 1")
227
+ if self.task.name == "pfn" and self.model.name not in (
228
+ "pfn",
229
+ "tabpfn",
230
+ "tabicl",
231
+ ):
232
+ raise ValueError(
233
+ "task.name: pfn meta-trains model.name: pfn, tabpfn or tabicl"
234
+ )
235
+ for pretraining in ("jepa", "contrastive"):
236
+ if (self.model.name == pretraining) != (self.task.name == pretraining):
237
+ raise ValueError(
238
+ f"model.name: {pretraining} trains task.name: {pretraining}"
239
+ )
240
  if self.split not in ("train", "validation", "test"):
241
  raise ValueError("split must be train, validation or test")
242
 
src/gnn4colliders/contrastive.py ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Contrastive pretraining of an object encoder (SimCLR, Chen et al. 2020;
2
+ for jets JetCLR, Dillon et al. 2022, and RS3L, Harris et al. 2024).
3
+
4
+ Every event is seen in two random views. Each view rotates the event in
5
+ phi, flips it in eta (probability 1/2), scales its momenta and energies by
6
+ a factor uniform in ``1 +- model.scale_jitter`` and drops each object
7
+ other than the first with probability ``model.object_dropout``. The encoder
8
+ (``model.backbone``, as for JEPA) embeds each view (the mean of its object
9
+ encodings, or the edge network's decoded global state), and a projection MLP
10
+ maps the embedding to a unit vector. The NT-Xent loss at temperature
11
+ ``model.temperature`` pulls the two views of an event together and pushes
12
+ them from the other events of the batch (on each rank) apart; labels are not
13
+ used. The pretrained encoder (``context_encoder``, as JEPA names it)
14
+ initializes a classifier through ``model.pretrained``.
15
+ """
16
+
17
+ from __future__ import annotations
18
+
19
+ import math
20
+ from typing import Any
21
+
22
+ import torch
23
+ import torch.nn.functional as F
24
+ from torch import nn
25
+
26
+ from .jepa import event_embedding
27
+ from .sequences import augment_tokens
28
+
29
+
30
+ # Random augmentations from a generator do not trace; they are cheap.
31
+ @torch.compiler.disable
32
+ def random_view(
33
+ tokens: torch.Tensor,
34
+ mask: torch.Tensor,
35
+ *,
36
+ scale_jitter: float,
37
+ object_dropout: float,
38
+ generator: torch.Generator | None = None,
39
+ ) -> tuple[torch.Tensor, torch.Tensor]:
40
+ """An augmented view ``(tokens, mask)`` of every event (see the module
41
+ documentation)."""
42
+ events, device = tokens.shape[0], tokens.device
43
+
44
+ def uniform(*shape: int) -> torch.Tensor:
45
+ return torch.rand(shape, generator=generator, device=device)
46
+
47
+ scale = 1 + scale_jitter * (2 * uniform(events) - 1)
48
+ rotation = math.pi * (2 * uniform(events) - 1)
49
+ flip = torch.where(uniform(events) < 0.5, -1.0, 1.0)
50
+ view = augment_tokens(tokens, mask, scale=scale, rotation=rotation, flip=flip)
51
+ dropped = uniform(*mask.shape) < object_dropout
52
+ dropped[:, 0] = False # every event keeps an object
53
+ return view, mask & ~dropped
54
+
55
+
56
+ class Contrastive(nn.Module):
57
+ """The encoder and the projection MLP.
58
+
59
+ ``forward`` returns ``[events, 3]``: each event's loss, 1 (every event
60
+ counts), and the spread of the embeddings over the batch (their standard
61
+ deviation, averaged over dimensions; it falls towards zero if the
62
+ representation collapses).
63
+ """
64
+
65
+ def __init__(
66
+ self,
67
+ encoder: nn.Module,
68
+ *,
69
+ temperature: float = 0.1,
70
+ scale_jitter: float = 0.1,
71
+ object_dropout: float = 0.1,
72
+ ) -> None:
73
+ super().__init__()
74
+ hid_size = encoder.hid_size
75
+ self.temperature = temperature
76
+ self.scale_jitter, self.object_dropout = scale_jitter, object_dropout
77
+ self.context_encoder = encoder
78
+ self.projector = nn.Sequential(
79
+ nn.Linear(hid_size, hid_size), nn.ReLU(), nn.Linear(hid_size, hid_size)
80
+ )
81
+
82
+ def forward(self, batch: Any) -> torch.Tensor:
83
+ tokens, mask = batch.tokens, batch.token_mask
84
+ # Validation sees the same views every epoch.
85
+ generator = (
86
+ None if self.training else torch.Generator(tokens.device).manual_seed(0)
87
+ )
88
+ embedded = [
89
+ event_embedding(
90
+ self.context_encoder,
91
+ *random_view(
92
+ tokens,
93
+ mask,
94
+ scale_jitter=self.scale_jitter,
95
+ object_dropout=self.object_dropout,
96
+ generator=generator,
97
+ ),
98
+ )
99
+ for _ in range(2)
100
+ ]
101
+ spread = embedded[0].float().std(dim=0).mean()
102
+ z = F.normalize(self.projector(torch.cat(embedded)).float(), dim=1)
103
+ events = tokens.shape[0]
104
+ similarity = (z @ z.T) / self.temperature
105
+ similarity = similarity.masked_fill(
106
+ torch.eye(2 * events, dtype=torch.bool, device=z.device), float("-inf")
107
+ )
108
+ # The other view of each event is its positive.
109
+ positive = torch.arange(2 * events, device=z.device).roll(events)
110
+ loss = F.cross_entropy(similarity, positive, reduction="none")
111
+ loss = loss.view(2, events).mean(dim=0)
112
+ return torch.stack((loss, torch.ones_like(loss), spread.expand_as(loss)), dim=1)
src/gnn4colliders/distributed.py CHANGED
@@ -30,10 +30,20 @@ class DistributedContext:
30
 
31
  def resolve_device(device: str) -> torch.device:
32
  if device == "auto":
33
- return torch.device("cuda" if torch.cuda.is_available() else "cpu")
 
 
34
  return torch.device(device)
35
 
36
 
 
 
 
 
 
 
 
 
37
  def initialize(device: str = "auto") -> DistributedContext:
38
  world_size = int(os.environ.get("WORLD_SIZE", "1"))
39
  requested = resolve_device(device)
 
30
 
31
  def resolve_device(device: str) -> torch.device:
32
  if device == "auto":
33
+ if torch.cuda.is_available():
34
+ return torch.device("cuda")
35
+ return torch.device("mps" if torch.backends.mps.is_available() else "cpu")
36
  return torch.device(device)
37
 
38
 
39
+ def synchronize(device: torch.device) -> None:
40
+ """Wait for the work queued on ``device``, for timing."""
41
+ if device.type == "cuda":
42
+ torch.cuda.synchronize(device)
43
+ elif device.type == "mps":
44
+ torch.mps.synchronize()
45
+
46
+
47
  def initialize(device: str = "auto") -> DistributedContext:
48
  world_size = int(os.environ.get("WORLD_SIZE", "1"))
49
  requested = resolve_device(device)
src/gnn4colliders/jepa.py CHANGED
@@ -10,6 +10,12 @@ encoder is an exponential moving average of the context encoder and gets no
10
  gradient. The loss is the mean squared error between predicted and
11
  layer-normalized target encodings; labels are not used.
12
 
 
 
 
 
 
 
13
  The pretrained context encoder initializes a classifier of the same
14
  backbone (``model.name: transformer`` or ``edge_network`` with
15
  ``model.pretrained``).
@@ -34,6 +40,7 @@ from torch import nn
34
 
35
  from .graphs import build_graph
36
  from .models import NODE_TYPE, EdgeNetwork, EncoderBlock
 
37
 
38
  #: Objects of one type beyond this index share the last index embedding.
39
  MAX_TYPE_INDEX = 16
@@ -93,6 +100,42 @@ def encode(
93
  return states, encoder.global_decoder(global_h)
94
 
95
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
96
  def _variance_penalty(encodings: torch.Tensor) -> torch.Tensor:
97
  """VICReg variance term (Bardes et al. 2022): the mean shortfall of each
98
  dimension's standard deviation over ``encodings [n, dims]`` below 1."""
@@ -106,7 +149,8 @@ class JEPA(nn.Module):
106
  ``forward`` returns ``[events, 3]``: each event's loss, whether it has
107
  hidden objects, and the spread of the target encodings over the batch
108
  (their standard deviation over objects, averaged over dimensions; it
109
- falls towards zero if the representation collapses).
 
110
  """
111
 
112
  def __init__(
@@ -121,13 +165,22 @@ class JEPA(nn.Module):
121
  mask_fraction: float = 0.4,
122
  ema_momentum: float = 0.996,
123
  variance_weight: float = 0.0,
 
124
  ) -> None:
125
  super().__init__()
 
 
 
 
126
  hid_size = encoder.hid_size
127
  self.mask_fraction, self.ema_momentum = mask_fraction, ema_momentum
128
  self.variance_weight = variance_weight
129
  self.context_encoder = encoder
130
- self.target_encoder = copy.deepcopy(encoder).requires_grad_(False)
 
 
 
 
131
  self.query_type = nn.Embedding(num_node_types, hid_size)
132
  self.query_index = nn.Embedding(MAX_TYPE_INDEX, hid_size)
133
  self.predictor = nn.ModuleList(
@@ -135,7 +188,9 @@ class JEPA(nn.Module):
135
  for _ in range(predictor_layers)
136
  )
137
  self.predictor_norm = nn.LayerNorm(hid_size)
138
- self.predictor_out = nn.Linear(hid_size, hid_size)
 
 
139
  #: Optimizer steps taken, and the steps over which the momentum
140
  #: reaches 1 (set before training; see :meth:`update_target`).
141
  self.register_buffer("step", torch.zeros((), dtype=torch.long))
@@ -149,10 +204,15 @@ class JEPA(nn.Module):
149
  )
150
  hidden = sample_targets(mask, self.mask_fraction, generator)
151
  with torch.no_grad():
152
- target = encode(self.target_encoder, tokens, mask)[0].float()
153
- spread = target[mask].std(dim=0).mean()
154
- target = nn.functional.layer_norm(target, target.shape[-1:])
 
 
 
155
  predicted, context, event = self.predict(tokens, mask, hidden)
 
 
156
  error = (predicted - target).square().mean(dim=-1) * hidden
157
  count = hidden.sum(dim=1)
158
  loss = error.sum(dim=1) / count.clamp_min(1)
@@ -193,6 +253,8 @@ class JEPA(nn.Module):
193
  """Move the target encoder towards the context encoder, with a
194
  momentum rising linearly from ``ema_momentum`` to 1 over
195
  ``total_steps``. Called by the trainer after each optimizer step."""
 
 
196
  progress = (self.step.float() / self.total_steps).clamp(max=1)
197
  momentum = self.ema_momentum + (1 - self.ema_momentum) * progress
198
  for target, context in zip(
 
10
  gradient. The loss is the mean squared error between predicted and
11
  layer-normalized target encodings; labels are not used.
12
 
13
+ With ``model.jepa_target: features`` the predictor instead predicts the
14
+ hidden objects' kinematics, ``log(1 + pt)``, eta, ``cos phi``, ``sin phi``,
15
+ ``log(1 + energy)``, b-tag and charge, as in masked particle modelling
16
+ (Golling et al. 2024); there is no target encoder. Both differ only in
17
+ whether the target is an encoding or the input.
18
+
19
  The pretrained context encoder initializes a classifier of the same
20
  backbone (``model.name: transformer`` or ``edge_network`` with
21
  ``model.pretrained``).
 
40
 
41
  from .graphs import build_graph
42
  from .models import NODE_TYPE, EdgeNetwork, EncoderBlock
43
+ from .sequences import BTAG, CHARGE, ENERGY, ETA, PHI, PT
44
 
45
  #: Objects of one type beyond this index share the last index embedding.
46
  MAX_TYPE_INDEX = 16
 
100
  return states, encoder.global_decoder(global_h)
101
 
102
 
103
+ def event_embedding(
104
+ encoder: nn.Module, tokens: torch.Tensor, mask: torch.Tensor
105
+ ) -> torch.Tensor:
106
+ """Embeddings ``[events, hid_size]`` of events by an object encoder: the
107
+ mean of the encodings of the objects in ``mask``, or the edge network's
108
+ decoded global state (what its classifier reads)."""
109
+ states, event = encode(encoder, tokens, mask)
110
+ if event is not None:
111
+ return event
112
+ counts = mask.sum(dim=1, keepdim=True).clamp_min(1)
113
+ return (states * mask.unsqueeze(-1)).sum(dim=1) / counts
114
+
115
+
116
+ #: Width of the kinematic targets of :func:`object_features`.
117
+ FEATURE_TARGETS = 7
118
+
119
+
120
+ def object_features(tokens: torch.Tensor) -> torch.Tensor:
121
+ """Kinematic targets ``[events, objects, FEATURE_TARGETS]`` (see the
122
+ module documentation)."""
123
+ pt, energy, phi = tokens[..., PT], tokens[..., ENERGY], tokens[..., PHI]
124
+ btag, charge = tokens[..., BTAG], tokens[..., CHARGE]
125
+ return torch.stack(
126
+ (
127
+ pt.clamp_min(0).log1p(),
128
+ tokens[..., ETA],
129
+ phi.cos(),
130
+ phi.sin(),
131
+ energy.clamp_min(0).log1p(),
132
+ btag,
133
+ charge,
134
+ ),
135
+ dim=-1,
136
+ )
137
+
138
+
139
  def _variance_penalty(encodings: torch.Tensor) -> torch.Tensor:
140
  """VICReg variance term (Bardes et al. 2022): the mean shortfall of each
141
  dimension's standard deviation over ``encodings [n, dims]`` below 1."""
 
149
  ``forward`` returns ``[events, 3]``: each event's loss, whether it has
150
  hidden objects, and the spread of the target encodings over the batch
151
  (their standard deviation over objects, averaged over dimensions; it
152
+ falls towards zero if the representation collapses). With kinematic
153
+ targets, the spread is that of the context encodings.
154
  """
155
 
156
  def __init__(
 
165
  mask_fraction: float = 0.4,
166
  ema_momentum: float = 0.996,
167
  variance_weight: float = 0.0,
168
+ target: str = "encoding",
169
  ) -> None:
170
  super().__init__()
171
+ if target not in ("encoding", "features"):
172
+ raise ValueError(
173
+ f"unknown jepa_target {target!r}; use encoding or features"
174
+ )
175
  hid_size = encoder.hid_size
176
  self.mask_fraction, self.ema_momentum = mask_fraction, ema_momentum
177
  self.variance_weight = variance_weight
178
  self.context_encoder = encoder
179
+ self.target_encoder = (
180
+ copy.deepcopy(encoder).requires_grad_(False)
181
+ if target == "encoding"
182
+ else None
183
+ )
184
  self.query_type = nn.Embedding(num_node_types, hid_size)
185
  self.query_index = nn.Embedding(MAX_TYPE_INDEX, hid_size)
186
  self.predictor = nn.ModuleList(
 
188
  for _ in range(predictor_layers)
189
  )
190
  self.predictor_norm = nn.LayerNorm(hid_size)
191
+ self.predictor_out = nn.Linear(
192
+ hid_size, hid_size if target == "encoding" else FEATURE_TARGETS
193
+ )
194
  #: Optimizer steps taken, and the steps over which the momentum
195
  #: reaches 1 (set before training; see :meth:`update_target`).
196
  self.register_buffer("step", torch.zeros((), dtype=torch.long))
 
204
  )
205
  hidden = sample_targets(mask, self.mask_fraction, generator)
206
  with torch.no_grad():
207
+ if self.target_encoder is None:
208
+ target = object_features(tokens).float()
209
+ else:
210
+ target = encode(self.target_encoder, tokens, mask)[0].float()
211
+ spread = target[mask].std(dim=0).mean()
212
+ target = nn.functional.layer_norm(target, target.shape[-1:])
213
  predicted, context, event = self.predict(tokens, mask, hidden)
214
+ if self.target_encoder is None:
215
+ spread = context[mask & ~hidden].detach().float().std(dim=0).mean()
216
  error = (predicted - target).square().mean(dim=-1) * hidden
217
  count = hidden.sum(dim=1)
218
  loss = error.sum(dim=1) / count.clamp_min(1)
 
253
  """Move the target encoder towards the context encoder, with a
254
  momentum rising linearly from ``ema_momentum`` to 1 over
255
  ``total_steps``. Called by the trainer after each optimizer step."""
256
+ if self.target_encoder is None:
257
+ return
258
  progress = (self.step.float() / self.total_steps).clamp(max=1)
259
  momentum = self.ema_momentum + (1 - self.ema_momentum) * progress
260
  for target, context in zip(
src/gnn4colliders/models.py CHANGED
@@ -14,11 +14,11 @@ from torch import nn
14
 
15
  from .data import NODE_FEATURE_NAMES
16
  from .graphs import EDGE_FEATURE_NAMES, TensorGraph
 
17
 
18
  if TYPE_CHECKING:
19
  from .config import ModelConfig
20
 
21
- PT, ETA, PHI = (NODE_FEATURE_NAMES.index(name) for name in ("pt", "eta", "phi"))
22
  NODE_TYPE = NODE_FEATURE_NAMES.index("node_type")
23
 
24
 
@@ -413,9 +413,18 @@ class DeepSets(nn.Module):
413
 
414
 
415
  #: Models that take :class:`~gnn4colliders.sequences.SequenceBatch` inputs.
416
- SEQUENCE_MODELS = ("transformer", "deep_sets", "jepa", "pfn", "tabpfn")
 
 
 
 
 
 
 
 
 
417
  #: Models that classify events given labelled context events (see cli.py).
418
- IN_CONTEXT_MODELS = ("pfn", "tabpfn")
419
 
420
 
421
  def state_dict_of(checkpoint: Mapping[str, Any]) -> dict[str, Any]:
@@ -440,17 +449,47 @@ def load_weights(model: nn.Module, checkpoint: Mapping[str, Any]) -> None:
440
  model.load_state_dict(state_dict_of(checkpoint), strict=True)
441
 
442
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
443
  def pretrained_state(path: str) -> dict[str, Any]:
444
  """Weights of a pretrained backbone: those of a supervised checkpoint, or
445
- the context encoder of a JEPA checkpoint."""
446
- state = state_dict_of(torch.load(path, map_location="cpu", weights_only=False))
447
- prefix = "context_encoder."
448
- encoder = {
449
- key.removeprefix(prefix): value
450
- for key, value in state.items()
451
- if key.startswith(prefix)
 
 
 
 
 
 
 
 
 
 
 
452
  }
453
- return encoder or state
454
 
455
 
456
  def _load_encoder(encoder: ObjectTransformer, path: str) -> None:
@@ -460,23 +499,10 @@ def _load_encoder(encoder: ObjectTransformer, path: str) -> None:
460
  encoder.load_state_dict(state, strict=True)
461
 
462
 
463
- def build_model(
464
- config: ModelConfig,
465
- *,
466
- num_features: int,
467
- num_node_types: int,
468
- num_global_features: int = 0,
469
- load_pretrained: bool = True,
470
- ) -> nn.Module:
471
- """The model of ``config`` for inputs of the given widths.
472
-
473
- With ``config.pretrained`` the edge network is a
474
- :class:`FineTunedEdgeNetwork` whose backbone is loaded from a supervised
475
- or JEPA checkpoint, and the transformer's encoder is loaded from a JEPA
476
- checkpoint, as is the PFN's (all skipped with ``load_pretrained=False``,
477
- e.g. when a trained checkpoint is loaded next).
478
- """
479
- transformer = {
480
  "input_size": num_features,
481
  "num_node_types": num_node_types,
482
  "hid_size": config.hid_size,
@@ -486,7 +512,10 @@ def build_model(
486
  "dropout": config.dropout,
487
  "pair_bias": config.pair_bias,
488
  }
489
- edge_network = {
 
 
 
490
  "node_size": num_features,
491
  "edge_size": len(EDGE_FEATURE_NAMES),
492
  "hid_size": config.hid_size,
@@ -494,19 +523,87 @@ def build_model(
494
  "n_proc_steps": config.n_proc_steps,
495
  "dropout": config.dropout,
496
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
497
  if config.name == "jepa":
498
  from .jepa import JEPA
499
 
500
- if config.backbone == "transformer":
501
- encoder = ObjectTransformer(**transformer)
502
- elif config.backbone == "edge_network":
503
- # Global features are not used in pretraining.
504
- encoder = EdgeNetwork(global_size=0, out_size=1, **edge_network)
505
- encoder.classifier = nn.Identity()
506
- else:
507
- raise ValueError(
508
- f"unknown backbone {config.backbone!r}; use transformer or edge_network"
509
- )
510
  return JEPA(
511
  encoder,
512
  num_node_types=num_node_types,
@@ -517,6 +614,22 @@ def build_model(
517
  mask_fraction=config.mask_fraction,
518
  ema_momentum=config.ema_momentum,
519
  variance_weight=config.variance_weight,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
520
  )
521
  if config.name == "transformer":
522
  model = TransformerClassifier(
@@ -542,8 +655,30 @@ def build_model(
542
  ffn_size=config.ffn_size or 4 * config.hid_size,
543
  dropout=config.dropout,
544
  )
545
- if config.name == "tabpfn":
546
- raise ValueError("tabpfn is not trained; run evaluate or predict")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
547
  if config.name == "deep_sets":
548
  return DeepSets(
549
  input_size=num_features,
@@ -557,7 +692,7 @@ def build_model(
557
  if config.name != "edge_network":
558
  raise ValueError(
559
  f"unknown model {config.name!r}; use edge_network, transformer, "
560
- "deep_sets, jepa, pfn or tabpfn"
561
  )
562
  sizes = {"global_size": num_global_features, **edge_network}
563
  if not config.pretrained:
 
14
 
15
  from .data import NODE_FEATURE_NAMES
16
  from .graphs import EDGE_FEATURE_NAMES, TensorGraph
17
+ from .sequences import ETA, PHI, PT
18
 
19
  if TYPE_CHECKING:
20
  from .config import ModelConfig
21
 
 
22
  NODE_TYPE = NODE_FEATURE_NAMES.index("node_type")
23
 
24
 
 
413
 
414
 
415
  #: Models that take :class:`~gnn4colliders.sequences.SequenceBatch` inputs.
416
+ SEQUENCE_MODELS = (
417
+ "transformer",
418
+ "deep_sets",
419
+ "jepa",
420
+ "contrastive",
421
+ "pfn",
422
+ "tabpfn",
423
+ "tabicl",
424
+ "tabdpt",
425
+ )
426
  #: Models that classify events given labelled context events (see cli.py).
427
+ IN_CONTEXT_MODELS = ("pfn", "tabpfn", "tabicl", "tabdpt")
428
 
429
 
430
  def state_dict_of(checkpoint: Mapping[str, Any]) -> dict[str, Any]:
 
449
  model.load_state_dict(state_dict_of(checkpoint), strict=True)
450
 
451
 
452
+ def _strip_prefix(
453
+ state: dict[str, Any], prefixes: tuple[str, ...]
454
+ ) -> dict[str, Any] | None:
455
+ """The entries of ``state`` under the first of ``prefixes`` it has, the
456
+ prefix removed, or None."""
457
+ for prefix in prefixes:
458
+ part = {
459
+ key.removeprefix(prefix): value
460
+ for key, value in state.items()
461
+ if key.startswith(prefix)
462
+ }
463
+ if part:
464
+ return part
465
+ return None
466
+
467
+
468
+ def _load_state(path: str) -> dict[str, Any]:
469
+ return state_dict_of(torch.load(path, map_location="cpu", weights_only=False))
470
+
471
+
472
  def pretrained_state(path: str) -> dict[str, Any]:
473
  """Weights of a pretrained backbone: those of a supervised checkpoint, or
474
+ the context encoder of a JEPA or contrastive checkpoint."""
475
+ state = _load_state(path)
476
+ return _strip_prefix(state, ("context_encoder.",)) or state
477
+
478
+
479
+ #: Prefixes of an encoder's weights in the checkpoints this package writes:
480
+ #: pretraining (JEPA, contrastive), a supervised transformer, a fine-tuned
481
+ #: edge network and the encoder of a meta-trained tabular model.
482
+ ENCODER_PREFIXES = ("context_encoder.", "encoder.", "backbone.", "inputs.encoder.")
483
+
484
+
485
+ def encoder_state(path: str) -> dict[str, torch.Tensor]:
486
+ """The encoder weights of a checkpoint (see ``ENCODER_PREFIXES``; else a
487
+ supervised edge network), without a classifier."""
488
+ state = _load_state(path)
489
+ state = _strip_prefix(state, ENCODER_PREFIXES) or state
490
+ return {
491
+ key: value for key, value in state.items() if not key.startswith("classifier.")
492
  }
 
493
 
494
 
495
  def _load_encoder(encoder: ObjectTransformer, path: str) -> None:
 
499
  encoder.load_state_dict(state, strict=True)
500
 
501
 
502
+ def _transformer_sizes(
503
+ config: ModelConfig, num_features: int, num_node_types: int
504
+ ) -> dict[str, Any]:
505
+ return {
 
 
 
 
 
 
 
 
 
 
 
 
 
506
  "input_size": num_features,
507
  "num_node_types": num_node_types,
508
  "hid_size": config.hid_size,
 
512
  "dropout": config.dropout,
513
  "pair_bias": config.pair_bias,
514
  }
515
+
516
+
517
+ def _edge_network_sizes(config: ModelConfig, num_features: int) -> dict[str, Any]:
518
+ return {
519
  "node_size": num_features,
520
  "edge_size": len(EDGE_FEATURE_NAMES),
521
  "hid_size": config.hid_size,
 
523
  "n_proc_steps": config.n_proc_steps,
524
  "dropout": config.dropout,
525
  }
526
+
527
+
528
+ def build_encoder(
529
+ config: ModelConfig, backbone: str, *, num_features: int, num_node_types: int
530
+ ) -> nn.Module:
531
+ """The object encoder ``backbone`` (``transformer`` or ``edge_network``)
532
+ with the sizes of ``config``, as pretrained (JEPA, contrastive) and
533
+ probed: the edge network without global features or classifier."""
534
+ if backbone == "transformer":
535
+ return ObjectTransformer(
536
+ **_transformer_sizes(config, num_features, num_node_types)
537
+ )
538
+ if backbone == "edge_network":
539
+ encoder = EdgeNetwork(
540
+ global_size=0, out_size=1, **_edge_network_sizes(config, num_features)
541
+ )
542
+ encoder.classifier = nn.Identity()
543
+ return encoder
544
+ raise ValueError(f"unknown backbone {backbone!r}; use transformer or edge_network")
545
+
546
+
547
+ #: Seed of the initialization of encoders without pretrained weights, so
548
+ #: that a frozen random encoder is rebuilt identically at evaluation and on
549
+ #: every rank.
550
+ RANDOM_ENCODER_SEED = 0
551
+
552
+
553
+ def pretrained_encoder(
554
+ config: ModelConfig,
555
+ *,
556
+ num_features: int,
557
+ num_node_types: int,
558
+ backbone: str | None = None,
559
+ freeze: bool | None = None,
560
+ ) -> nn.Module:
561
+ """The object encoder ``backbone`` (default ``config.backbone``) with
562
+ the weights of ``config.pretrained``, or randomly initialized with
563
+ ``RANDOM_ENCODER_SEED`` without; frozen if ``freeze`` (default
564
+ ``config.freeze_backbone``)."""
565
+ with torch.random.fork_rng(devices=[]):
566
+ torch.manual_seed(RANDOM_ENCODER_SEED)
567
+ encoder = build_encoder(
568
+ config,
569
+ backbone or config.backbone,
570
+ num_features=num_features,
571
+ num_node_types=num_node_types,
572
+ )
573
+ if config.pretrained:
574
+ encoder.load_state_dict(encoder_state(config.pretrained), strict=True)
575
+ freeze = config.freeze_backbone if freeze is None else freeze
576
+ return encoder.requires_grad_(not freeze)
577
+
578
+
579
+ def build_model(
580
+ config: ModelConfig,
581
+ *,
582
+ num_features: int,
583
+ num_node_types: int,
584
+ num_global_features: int = 0,
585
+ load_pretrained: bool = True,
586
+ ) -> nn.Module:
587
+ """The model of ``config`` for inputs of the given widths.
588
+
589
+ With ``config.pretrained`` the edge network is a
590
+ :class:`FineTunedEdgeNetwork` whose backbone is loaded from a supervised
591
+ or JEPA checkpoint, and the transformer's encoder is loaded from a JEPA
592
+ checkpoint, as is the PFN's (all skipped with ``load_pretrained=False``,
593
+ e.g. when a trained checkpoint is loaded next).
594
+ """
595
+ transformer = _transformer_sizes(config, num_features, num_node_types)
596
+ edge_network = _edge_network_sizes(config, num_features)
597
  if config.name == "jepa":
598
  from .jepa import JEPA
599
 
600
+ # Global features are not used in pretraining.
601
+ encoder = build_encoder(
602
+ config,
603
+ config.backbone,
604
+ num_features=num_features,
605
+ num_node_types=num_node_types,
606
+ )
 
 
 
607
  return JEPA(
608
  encoder,
609
  num_node_types=num_node_types,
 
614
  mask_fraction=config.mask_fraction,
615
  ema_momentum=config.ema_momentum,
616
  variance_weight=config.variance_weight,
617
+ target=config.jepa_target,
618
+ )
619
+ if config.name == "contrastive":
620
+ from .contrastive import Contrastive
621
+
622
+ encoder = build_encoder(
623
+ config,
624
+ config.backbone,
625
+ num_features=num_features,
626
+ num_node_types=num_node_types,
627
+ )
628
+ return Contrastive(
629
+ encoder,
630
+ temperature=config.temperature,
631
+ scale_jitter=config.scale_jitter,
632
+ object_dropout=config.object_dropout,
633
  )
634
  if config.name == "transformer":
635
  model = TransformerClassifier(
 
655
  ffn_size=config.ffn_size or 4 * config.hid_size,
656
  dropout=config.dropout,
657
  )
658
+ if config.name in ("tabpfn", "tabicl"):
659
+ from . import tabular
660
+
661
+ # The released weights load whatever load_pretrained says.
662
+ if config.name == "tabpfn":
663
+ network = tabular.load_tabpfn_network(
664
+ config.tabpfn_version, config.tabpfn_checkpoint
665
+ )
666
+ meta = tabular.TabularPFN
667
+ else:
668
+ network = tabular.load_tabicl_network(config.tabicl_checkpoint)
669
+ meta = tabular.TabularICL
670
+ # So do those of a frozen encoder of the inputs.
671
+ inputs = tabular.EventInputs.of(
672
+ config, num_features=num_features, num_node_types=num_node_types
673
+ )
674
+ return meta(
675
+ network,
676
+ inputs=inputs,
677
+ max_classes=config.max_classes,
678
+ lora_rank=config.lora_rank,
679
+ )
680
+ if config.name == "tabdpt":
681
+ raise ValueError("tabdpt is not trained; run evaluate or predict")
682
  if config.name == "deep_sets":
683
  return DeepSets(
684
  input_size=num_features,
 
692
  if config.name != "edge_network":
693
  raise ValueError(
694
  f"unknown model {config.name!r}; use edge_network, transformer, "
695
+ "deep_sets, jepa, contrastive, pfn, tabpfn, tabicl or tabdpt"
696
  )
697
  sizes = {"global_size": num_global_features, **edge_network}
698
  if not config.pretrained:
src/gnn4colliders/pfn.py CHANGED
@@ -29,11 +29,10 @@ import torch
29
  from torch import nn
30
 
31
  from .config import TaskConfig
32
- from .data import NODE_FEATURE_NAMES, PackedEvents, map_tensors
33
- from .models import ETA, PHI, PT, EncoderBlock, ObjectTransformer
34
- from .sequences import SequenceBatch, collate_sequence_batch
35
 
36
- ENERGY = NODE_FEATURE_NAMES.index("energy")
37
  #: Logit of the classes a task does not have.
38
  ABSENT_LOGIT = -1e4
39
 
@@ -177,13 +176,10 @@ class _Step:
177
 
178
  def _augment(batch: SequenceBatch, step: _Step) -> SequenceBatch:
179
  """Scale momenta and energies, rotate in phi and flip eta, per event."""
180
- tokens, mask = batch.tokens.clone(), batch.token_mask
181
- for column in (PT, ENERGY):
182
- tokens[..., column] *= step.scale[:, None]
183
- tokens[..., ETA] *= step.flip[:, None]
184
- phi = tokens[..., PHI] + step.rotation[:, None]
185
- tokens[..., PHI] = torch.remainder(phi + math.pi, 2 * math.pi) - math.pi
186
- tokens = torch.where(mask.unsqueeze(-1), tokens, batch.tokens)
187
  return SequenceBatch(
188
  tokens=tokens,
189
  token_mask=mask,
 
29
  from torch import nn
30
 
31
  from .config import TaskConfig
32
+ from .data import PackedEvents, map_tensors
33
+ from .models import EncoderBlock, ObjectTransformer
34
+ from .sequences import SequenceBatch, augment_tokens, collate_sequence_batch
35
 
 
36
  #: Logit of the classes a task does not have.
37
  ABSENT_LOGIT = -1e4
38
 
 
176
 
177
  def _augment(batch: SequenceBatch, step: _Step) -> SequenceBatch:
178
  """Scale momenta and energies, rotate in phi and flip eta, per event."""
179
+ mask = batch.token_mask
180
+ tokens = augment_tokens(
181
+ batch.tokens, mask, scale=step.scale, rotation=step.rotation, flip=step.flip
182
+ )
 
 
 
183
  return SequenceBatch(
184
  tokens=tokens,
185
  token_mask=mask,
src/gnn4colliders/probe.py ADDED
@@ -0,0 +1,161 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """k-NN and linear probes of a frozen object encoder (as for jBOT, Tsoi and
2
+ Rankin 2026), to compare pretraining objectives without fine-tuning.
3
+
4
+ The encoder of ``model.pretrained`` embeds every event: the mean of its
5
+ object encodings, or the edge network's decoded global state (what its
6
+ classifier reads). The checkpoint can be a JEPA, contrastive or masked-object
7
+ one (its context encoder) or a supervised transformer or edge network; with
8
+ none, the encoder is randomly initialized (with a fixed seed), the
9
+ reference every pretraining should beat. The config can be that of the
10
+ pretraining or of a classifier with the encoder's sizes. Two classifiers of
11
+ the frozen embeddings, of all the dataset's classes, are trained on
12
+ ``data.train_events`` labelled training events and evaluated on ``split``
13
+ (without those events):
14
+
15
+ * k-NN: the class fractions among the ``model.probe_neighbours`` training
16
+ embeddings of largest cosine similarity;
17
+ * linear: multinomial logistic regression of the standardized embeddings
18
+ (full-batch L-BFGS, L2 penalty ``model.probe_l2``).
19
+ """
20
+
21
+ from __future__ import annotations
22
+
23
+ from typing import Any
24
+
25
+ import torch
26
+ import torch.nn.functional as F
27
+ from torch import nn
28
+
29
+ from .config import Config
30
+ from .data import PackedEvents
31
+ from .jepa import event_embedding
32
+ from .models import pretrained_encoder
33
+ from .sequences import collate_sequence_batch
34
+
35
+
36
+ def backbone_of(config: Config) -> str:
37
+ """The encoder architecture of the model of ``config``."""
38
+ name = config.model.name
39
+ if name in ("jepa", "contrastive"):
40
+ return config.model.backbone
41
+ if name in ("transformer", "edge_network"):
42
+ return name
43
+ raise ValueError(f"cannot probe model.name {name!r}")
44
+
45
+
46
+ def load_encoder(config: Config, events: PackedEvents) -> nn.Module:
47
+ """The (pretrained) frozen encoder of ``config``."""
48
+ encoder = pretrained_encoder(
49
+ config.model,
50
+ num_features=events.num_features,
51
+ num_node_types=len(config.data.object_types),
52
+ backbone=backbone_of(config),
53
+ freeze=True,
54
+ )
55
+ return encoder.eval()
56
+
57
+
58
+ @torch.inference_mode()
59
+ def embed(
60
+ encoder: nn.Module,
61
+ events: PackedEvents,
62
+ indices: torch.Tensor,
63
+ *,
64
+ batch_size: int,
65
+ device: torch.device,
66
+ ) -> torch.Tensor:
67
+ """Event embeddings ``[events, hid_size]`` of ``indices``
68
+ (:func:`~gnn4colliders.jepa.event_embedding`)."""
69
+ parts = []
70
+ for part in indices.split(batch_size):
71
+ batch = collate_sequence_batch(events, part).to(device)
72
+ parts.append(event_embedding(encoder, batch.tokens, batch.token_mask).float())
73
+ return torch.cat(parts)
74
+
75
+
76
+ def knn_logits(
77
+ train: torch.Tensor,
78
+ labels: torch.Tensor,
79
+ queries: torch.Tensor,
80
+ *,
81
+ num_classes: int,
82
+ neighbours: int,
83
+ chunk: int = 4096,
84
+ ) -> torch.Tensor:
85
+ """Log class fractions ``[queries, num_classes]`` among each query's
86
+ ``neighbours`` most cosine-similar ``train`` embeddings."""
87
+ train, queries = F.normalize(train, dim=1), F.normalize(queries, dim=1)
88
+ neighbours = min(neighbours, len(train))
89
+ one_hot = F.one_hot(labels, num_classes).float()
90
+ parts = []
91
+ for part in queries.split(chunk):
92
+ nearest = (part @ train.T).topk(neighbours, dim=1).indices
93
+ fractions = one_hot[nearest].mean(dim=1)
94
+ parts.append((fractions + 1e-3).log())
95
+ return torch.cat(parts)
96
+
97
+
98
+ def linear_logits(
99
+ train: torch.Tensor,
100
+ labels: torch.Tensor,
101
+ queries: torch.Tensor,
102
+ *,
103
+ num_classes: int,
104
+ l2: float,
105
+ steps: int = 200,
106
+ ) -> torch.Tensor:
107
+ """Logits ``[queries, num_classes]`` of a multinomial logistic regression
108
+ fitted on the standardized ``train`` embeddings."""
109
+ mean, std = train.mean(dim=0), train.std(dim=0).clamp_min(1e-6)
110
+ x = (train - mean) / std
111
+ linear = nn.Linear(x.shape[1], num_classes).to(x.device)
112
+ optimizer = torch.optim.LBFGS(
113
+ linear.parameters(), max_iter=steps, line_search_fn="strong_wolfe"
114
+ )
115
+
116
+ def closure() -> torch.Tensor:
117
+ optimizer.zero_grad()
118
+ loss = F.cross_entropy(linear(x), labels) + l2 * linear.weight.square().sum()
119
+ loss.backward()
120
+ return loss
121
+
122
+ with torch.enable_grad():
123
+ optimizer.step(closure)
124
+ with torch.no_grad():
125
+ return linear((queries - mean) / std)
126
+
127
+
128
+ def probe(
129
+ config: Config,
130
+ events: PackedEvents,
131
+ train: torch.Tensor,
132
+ queries: torch.Tensor,
133
+ *,
134
+ num_classes: int,
135
+ device: torch.device,
136
+ ) -> dict[str, Any]:
137
+ """Logits ``[queries, num_classes]`` of the k-NN and linear probes on the
138
+ ``queries`` events, given the labelled ``train`` events."""
139
+ encoder = load_encoder(config, events).to(device)
140
+ batch_size = config.data.batch_size
141
+ embedded = {
142
+ name: embed(encoder, events, indices, batch_size=batch_size, device=device)
143
+ for name, indices in (("train", train), ("queries", queries))
144
+ }
145
+ labels = events.labels[train].to(device, torch.long)
146
+ return {
147
+ "knn": knn_logits(
148
+ embedded["train"],
149
+ labels,
150
+ embedded["queries"],
151
+ num_classes=num_classes,
152
+ neighbours=config.model.probe_neighbours,
153
+ ),
154
+ "linear": linear_logits(
155
+ embedded["train"],
156
+ labels,
157
+ embedded["queries"],
158
+ num_classes=num_classes,
159
+ l2=config.model.probe_l2,
160
+ ),
161
+ }
src/gnn4colliders/sequences.py CHANGED
@@ -2,11 +2,17 @@
2
 
3
  from __future__ import annotations
4
 
 
5
  from dataclasses import dataclass
6
 
7
  import torch
8
 
9
- from .data import PackedEvents, map_tensors
 
 
 
 
 
10
 
11
 
12
  @dataclass(frozen=True)
@@ -49,3 +55,23 @@ def collate_sequence_batch(
49
  else events.global_features[indices],
50
  event_index=indices,
51
  )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
 
3
  from __future__ import annotations
4
 
5
+ import math
6
  from dataclasses import dataclass
7
 
8
  import torch
9
 
10
+ from .data import NODE_FEATURE_NAMES, PackedEvents, map_tensors
11
+
12
+ PT, ETA, PHI, ENERGY, BTAG, CHARGE = (
13
+ NODE_FEATURE_NAMES.index(name)
14
+ for name in ("pt", "eta", "phi", "energy", "btag", "charge")
15
+ )
16
 
17
 
18
  @dataclass(frozen=True)
 
55
  else events.global_features[indices],
56
  event_index=indices,
57
  )
58
+
59
+
60
+ def augment_tokens(
61
+ tokens: torch.Tensor,
62
+ mask: torch.Tensor,
63
+ *,
64
+ scale: torch.Tensor,
65
+ rotation: torch.Tensor,
66
+ flip: torch.Tensor,
67
+ ) -> torch.Tensor:
68
+ """``tokens`` with each event's momenta and energies scaled, azimuths
69
+ rotated and pseudorapidities multiplied by ``flip`` (1 or -1), all
70
+ ``[events]``; padding is left unchanged."""
71
+ augmented = tokens.clone()
72
+ for column in (PT, ENERGY):
73
+ augmented[..., column] *= scale[:, None]
74
+ augmented[..., ETA] *= flip[:, None]
75
+ phi = augmented[..., PHI] + rotation[:, None]
76
+ augmented[..., PHI] = torch.remainder(phi + math.pi, 2 * math.pi) - math.pi
77
+ return torch.where(mask.unsqueeze(-1), augmented, tokens)
src/gnn4colliders/tabular.py CHANGED
@@ -1,44 +1,125 @@
1
- """TabPFN (Hollmann et al. 2025; Prior Labs) on events as fixed-width rows.
2
-
3
- TabPFN classifies the rows of a table in context, given labelled rows, in
4
- one forward pass and without training. A row of an event holds the leading
5
- ``model.slots[t]`` objects of each object type ``t`` in stored (pt) order,
6
- with their six kinematic columns (``pt, eta, phi, energy, btag, charge``;
7
- NaN for absent objects, which TabPFN handles), then the number of objects of
8
- each type. The context is ``data.train_events`` labelled training events.
9
-
10
- The ``tabpfn`` package is an optional dependency (``uv sync --extra
11
- tabpfn``). TabPFN-2.5 and later weights are under a non-commercial license
12
- and download after a one-time login (a token in ``~/.cache/tabpfn`` or
13
- ``TABPFN_TOKEN``); ``TABPFN_MODEL_CACHE_DIR`` sets where they are stored.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14
  """
15
 
16
  from __future__ import annotations
17
 
 
18
  import math
 
 
19
  from collections.abc import Callable
 
20
 
21
  import torch
22
  import torch.nn.functional as F
 
 
23
 
24
  from .config import ModelConfig
25
  from .data import NODE_FEATURE_NAMES, PackedEvents
26
- from .sequences import SequenceBatch
 
 
 
27
 
28
  NODE_TYPE = NODE_FEATURE_NAMES.index("node_type")
29
- #: Log-probability of the classes TabPFN gives no probability.
30
  MIN_LOG_PROBABILITY = math.log(1e-12)
31
 
32
 
33
- def table_features(
34
- events: PackedEvents, indices: torch.Tensor, slots: list[int]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
35
  ) -> torch.Tensor:
36
- """Rows ``[events, sum(slots) * 6 + len(slots)]`` of ``indices``."""
 
37
  if NODE_TYPE != len(NODE_FEATURE_NAMES) - 1:
38
- raise ValueError("table_features expects node_type as the last column")
39
- rows, counts = events.gather_objects(indices)
40
- count, types = len(indices), len(slots)
41
- event = torch.repeat_interleave(torch.arange(count), counts, output_size=len(rows))
 
 
42
  node_type = rows[:, NODE_TYPE].long()
43
  one_hot = F.one_hot(node_type, types)
44
  # Index of every object among its event's objects of the same type.
@@ -46,13 +127,130 @@ def table_features(
46
  first = counts.cumsum(0) - counts
47
  before = (seen - one_hot)[first[event]]
48
  rank = (seen - before).gather(1, node_type[:, None]).squeeze(1) - 1
49
- capacity = torch.tensor(slots)
50
  offset = capacity.cumsum(0) - capacity
51
  keep = rank < capacity[node_type]
52
- table = torch.full((count, sum(slots), NODE_TYPE), math.nan)
53
  table[event[keep], (offset[node_type] + rank)[keep]] = rows[keep, :NODE_TYPE]
54
- multiplicity = torch.zeros(count, types).index_add_(0, event, one_hot.float())
55
- return torch.cat((table.flatten(1), multiplicity), dim=1)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
 
57
 
58
  def tabpfn_forward(
@@ -62,6 +260,7 @@ def tabpfn_forward(
62
  *,
63
  num_classes: int,
64
  num_node_types: int,
 
65
  device: torch.device,
66
  seed: int,
67
  ) -> Callable[[SequenceBatch], torch.Tensor]:
@@ -72,8 +271,7 @@ def tabpfn_forward(
72
  from tabpfn.constants import ModelVersion
73
  except ImportError as error:
74
  raise ImportError("tabpfn is not installed: uv sync --extra tabpfn") from error
75
- if len(config.slots) != num_node_types:
76
- raise ValueError("model.slots needs one count per object type")
77
  classifier = TabPFNClassifier.create_default_for_version(
78
  ModelVersion(config.tabpfn_version),
79
  n_estimators=config.tabpfn_estimators,
@@ -81,18 +279,427 @@ def tabpfn_forward(
81
  random_state=seed,
82
  # The context is processed once for all batches of events.
83
  fit_mode="fit_with_cache",
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
84
  )
85
- classifier.fit(
86
- table_features(events, context, config.slots).numpy(),
87
- events.labels[context].numpy(),
 
 
 
 
 
 
 
 
 
 
 
 
 
 
88
  )
89
- classes = torch.as_tensor(classifier.classes_, dtype=torch.long)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
90
 
91
  def forward(batch: SequenceBatch) -> torch.Tensor:
92
- rows = table_features(events, batch.event_index.cpu(), config.slots)
93
- probability = torch.as_tensor(classifier.predict_proba(rows.numpy()))
94
- logits = torch.full((len(rows), num_classes), MIN_LOG_PROBABILITY)
95
- logits[:, classes] = probability.float().clamp_min(1e-12).log()
96
- return logits.to(device)
97
 
98
  return forward
 
1
+ """Tabular in-context models on events as fixed-width rows: TabPFN
2
+ (Hollmann et al. 2025; Prior Labs), TabICL (Qu et al. 2025, 2026; Inria)
3
+ and TabDPT (Ma et al. 2024; Layer 6), which is trained on real tables and
4
+ can retrieve each query's nearest context rows.
5
+
6
+ Both classify the rows of a table in context, given labelled rows, in one
7
+ forward pass and without training. A row of an event (:class:`RowLayout`)
8
+ holds the leading ``model.slots[t]`` objects of each object type ``t`` in
9
+ stored (pt) order, with their six kinematic columns (``pt, eta, phi, energy,
10
+ btag, charge``; NaN for absent objects, which both models handle), then the
11
+ number of objects of each type. Optionally (``model.relative_phi``) every
12
+ azimuth is measured from that of the event's leading object, which removes
13
+ the azimuthal rotation the models cannot learn from a few events, and
14
+ (``model.pair_objects``) the row ends with the invariant mass and Delta R of
15
+ every pair of the event's leading objects. The context is
16
+ ``data.train_events`` labelled training events.
17
+
18
+ The ``tabpfn``, ``tabicl`` and ``tabdpt`` packages are optional
19
+ dependencies (``uv sync --extra tabpfn --extra tabicl --extra tabdpt``).
20
+ TabICL's (BSD-3) and TabDPT's (Apache-2.0) weights download from Hugging
21
+ Face (``jingang/TabICL``, ``Layer6/TabDPT``). TabPFN-2.5 and later weights
22
+ are under a non-commercial license and download after a one-time login (a
23
+ token in ``~/.cache/tabpfn`` or ``TABPFN_TOKEN``); ``TABPFN_MODEL_CACHE_DIR``
24
+ sets where they are stored. A checkpoint already in that directory loads
25
+ without the login, so the weights can also be downloaded from Hugging Face
26
+ (``Prior-Labs/tabpfn_3_5``).
27
+
28
+ :class:`TabularPFN` and :class:`TabularICL` meta-train the TabPFN-3.5 and
29
+ TabICL networks on the same rows, like the PFN of :mod:`gnn4colliders.pfn`
30
+ (``task.name: pfn``, the same tasks and augmentations), and classify in
31
+ context with the network's own forward pass instead of the classifier (no
32
+ preprocessing beyond TabICL's standardization, no ensembling). By default
33
+ the network is frozen and its in-context blocks learn low-rank updates
34
+ (``model.lora_rank``).
35
  """
36
 
37
  from __future__ import annotations
38
 
39
+ import dataclasses
40
  import math
41
+ import os
42
+ import sys
43
  from collections.abc import Callable
44
+ from typing import Any
45
 
46
  import torch
47
  import torch.nn.functional as F
48
+ from torch import nn
49
+ from torch.nn.utils import parametrize
50
 
51
  from .config import ModelConfig
52
  from .data import NODE_FEATURE_NAMES, PackedEvents
53
+ from .jepa import event_embedding
54
+ from .models import pretrained_encoder
55
+ from .pfn import ABSENT_LOGIT, TaskBatch
56
+ from .sequences import ENERGY, ETA, PHI, PT, SequenceBatch, collate_sequence_batch
57
 
58
  NODE_TYPE = NODE_FEATURE_NAMES.index("node_type")
59
+ #: Log-probability of the classes a classifier gives no probability.
60
  MIN_LOG_PROBABILITY = math.log(1e-12)
61
 
62
 
63
+ @dataclasses.dataclass(frozen=True)
64
+ class RowLayout:
65
+ """What a row of an event holds (see the module documentation)."""
66
+
67
+ slots: tuple[int, ...]
68
+ relative_phi: bool = False
69
+ pair_objects: int = 0
70
+
71
+ def __post_init__(self) -> None:
72
+ if not 0 <= self.pair_objects <= sum(self.slots):
73
+ raise ValueError("model.pair_objects must be between 0 and sum(slots)")
74
+
75
+ @classmethod
76
+ def of(cls, config: ModelConfig) -> RowLayout:
77
+ return cls(tuple(config.slots), config.relative_phi, config.pair_objects)
78
+
79
+ @property
80
+ def width(self) -> int:
81
+ pairs = self.pair_objects * (self.pair_objects - 1) // 2
82
+ return sum(self.slots) * NODE_TYPE + len(self.slots) + 2 * pairs
83
+
84
+
85
+ def _wrap(phi: torch.Tensor) -> torch.Tensor:
86
+ return torch.remainder(phi + math.pi, 2 * math.pi) - math.pi
87
+
88
+
89
+ def _pair_features(table: torch.Tensor, objects: int) -> torch.Tensor:
90
+ """Invariant mass and Delta R ``[events, 2 * pairs]`` of every pair of
91
+ the ``objects`` leading slots of ``table [events, slots, columns]`` by pt
92
+ (NaN for pairs with an absent object)."""
93
+ order = table[..., PT].nan_to_num(-1.0).argsort(dim=1, descending=True)
94
+ leading = table.gather(1, order[:, :objects, None].expand(-1, -1, NODE_TYPE))
95
+ first, second = torch.triu_indices(objects, objects, 1, device=table.device)
96
+ a, b = leading[:, first], leading[:, second]
97
+
98
+ def momentum(x: torch.Tensor) -> torch.Tensor:
99
+ pt, eta, phi = x[..., PT], x[..., ETA], x[..., PHI]
100
+ return torch.stack(
101
+ (pt * torch.cos(phi), pt * torch.sin(phi), pt * torch.sinh(eta)), dim=-1
102
+ )
103
+
104
+ energy = a[..., ENERGY] + b[..., ENERGY]
105
+ squared = energy.square() - (momentum(a) + momentum(b)).square().sum(dim=-1)
106
+ mass = squared.clamp_min(0).sqrt()
107
+ delta_r = torch.hypot(a[..., ETA] - b[..., ETA], _wrap(a[..., PHI] - b[..., PHI]))
108
+ return torch.cat((mass, delta_r), dim=1)
109
+
110
+
111
+ def object_table(
112
+ rows: torch.Tensor, counts: torch.Tensor, layout: RowLayout
113
  ) -> torch.Tensor:
114
+ """Rows ``[events, layout.width]`` of the events with ``counts``
115
+ consecutive object ``rows`` each."""
116
  if NODE_TYPE != len(NODE_FEATURE_NAMES) - 1:
117
+ raise ValueError("object_table expects node_type as the last column")
118
+ slots = layout.slots
119
+ device, count, types = rows.device, len(counts), len(slots)
120
+ event = torch.repeat_interleave(
121
+ torch.arange(count, device=device), counts, output_size=len(rows)
122
+ )
123
  node_type = rows[:, NODE_TYPE].long()
124
  one_hot = F.one_hot(node_type, types)
125
  # Index of every object among its event's objects of the same type.
 
127
  first = counts.cumsum(0) - counts
128
  before = (seen - one_hot)[first[event]]
129
  rank = (seen - before).gather(1, node_type[:, None]).squeeze(1) - 1
130
+ capacity = torch.tensor(slots, device=device)
131
  offset = capacity.cumsum(0) - capacity
132
  keep = rank < capacity[node_type]
133
+ table = torch.full((count, sum(slots), NODE_TYPE), math.nan, device=device)
134
  table[event[keep], (offset[node_type] + rank)[keep]] = rows[keep, :NODE_TYPE]
135
+ if layout.relative_phi:
136
+ leading = table[..., PT].nan_to_num(-1.0).argmax(dim=1)
137
+ reference = table[torch.arange(count, device=device), leading, PHI]
138
+ table[..., PHI] = _wrap(table[..., PHI] - reference[:, None])
139
+ multiplicity = torch.zeros(count, types, device=device)
140
+ multiplicity.index_add_(0, event, one_hot.float())
141
+ parts = [table.flatten(1), multiplicity]
142
+ if layout.pair_objects > 1:
143
+ parts.append(_pair_features(table, layout.pair_objects))
144
+ return torch.cat(parts, dim=1)
145
+
146
+
147
+ def table_features(
148
+ events: PackedEvents, indices: torch.Tensor, layout: RowLayout
149
+ ) -> torch.Tensor:
150
+ """Rows ``[events, layout.width]`` of ``indices``."""
151
+ return object_table(*events.gather_objects(indices), layout)
152
+
153
+
154
+ def sequence_table(batch: SequenceBatch, layout: RowLayout) -> torch.Tensor:
155
+ """The rows of :func:`table_features` of a (possibly augmented) batch."""
156
+ mask = batch.token_mask
157
+ return object_table(batch.tokens[mask], mask.sum(dim=1), layout)
158
+
159
+
160
+ class EventInputs(nn.Module):
161
+ """The inputs ``[events, columns]`` of a tabular model: each event's row
162
+ (:class:`RowLayout`), its embedding by an object ``encoder``
163
+ (:func:`~gnn4colliders.jepa.event_embedding`), or both
164
+ (``model.tabular_inputs``: ``rows``, ``embeddings`` or ``both``). The
165
+ encoder is that of ``model.backbone`` with the weights of
166
+ ``model.pretrained`` (randomly initialized without), trained with the
167
+ LoRA updates in meta-training unless ``model.freeze_backbone``."""
168
+
169
+ def __init__(
170
+ self, layout: RowLayout, encoder: nn.Module | None, *, rows: bool
171
+ ) -> None:
172
+ super().__init__()
173
+ if not rows and encoder is None:
174
+ raise ValueError("tabular inputs need rows or an encoder")
175
+ self.layout, self.encoder, self.rows = layout, encoder, rows
176
+
177
+ @classmethod
178
+ def of(
179
+ cls, config: ModelConfig, *, num_features: int, num_node_types: int
180
+ ) -> EventInputs:
181
+ if len(config.slots) != num_node_types:
182
+ raise ValueError("model.slots needs one count per object type")
183
+ inputs = config.tabular_inputs
184
+ if inputs not in ("rows", "embeddings", "both"):
185
+ raise ValueError(
186
+ f"unknown tabular_inputs {inputs!r}; use rows, embeddings or both"
187
+ )
188
+ encoder = None
189
+ if inputs != "rows":
190
+ encoder = pretrained_encoder(
191
+ config, num_features=num_features, num_node_types=num_node_types
192
+ )
193
+ return cls(RowLayout.of(config), encoder, rows=inputs != "embeddings")
194
+
195
+ def forward(self, batch: SequenceBatch) -> torch.Tensor:
196
+ parts = []
197
+ if self.rows:
198
+ parts.append(sequence_table(batch, self.layout))
199
+ if self.encoder is not None:
200
+ mask = batch.token_mask
201
+ parts.append(event_embedding(self.encoder, batch.tokens, mask).float())
202
+ return torch.cat(parts, dim=1)
203
+
204
+ def of_events(
205
+ self,
206
+ events: PackedEvents,
207
+ indices: torch.Tensor,
208
+ *,
209
+ batch_size: int,
210
+ device: torch.device,
211
+ ) -> torch.Tensor:
212
+ """The inputs of the events ``indices``, on ``device``."""
213
+ return torch.cat(
214
+ [
215
+ self(collate_sequence_batch(events, part).to(device))
216
+ for part in indices.split(batch_size)
217
+ ]
218
+ )
219
+
220
+
221
+ def _classifier_forward(
222
+ classifier: Any,
223
+ inputs: EventInputs,
224
+ events: PackedEvents,
225
+ context: torch.Tensor,
226
+ *,
227
+ num_classes: int,
228
+ batch_size: int,
229
+ device: torch.device,
230
+ predict: Callable[[Any], Any] | None = None,
231
+ ) -> Callable[[SequenceBatch], torch.Tensor]:
232
+ """Logits (log-probabilities) ``[events, num_classes]`` of a batch by a
233
+ scikit-learn ``classifier`` fitted on the ``inputs`` of the labelled
234
+ ``context``; its probabilities are ``predict(inputs)`` (default
235
+ ``predict_proba``)."""
236
+ predict = predict or classifier.predict_proba
237
+ inputs = inputs.to(device).eval()
238
+ # The context's classes, numbered 0, 1, ... for the classifier.
239
+ labels = events.labels[context].long()
240
+ classes = labels.unique()
241
+ with torch.no_grad():
242
+ rows = inputs.of_events(events, context, batch_size=batch_size, device=device)
243
+ classifier.fit(rows.cpu().numpy(), torch.searchsorted(classes, labels).numpy())
244
+
245
+ def forward(batch: SequenceBatch) -> torch.Tensor:
246
+ with torch.no_grad():
247
+ rows = inputs(batch).cpu()
248
+ probability = torch.as_tensor(predict(rows.numpy()))
249
+ logits = torch.full((len(rows), num_classes), MIN_LOG_PROBABILITY)
250
+ logits[:, classes] = probability.float().clamp_min(1e-12).log()
251
+ return logits.to(device)
252
+
253
+ return forward
254
 
255
 
256
  def tabpfn_forward(
 
260
  *,
261
  num_classes: int,
262
  num_node_types: int,
263
+ batch_size: int,
264
  device: torch.device,
265
  seed: int,
266
  ) -> Callable[[SequenceBatch], torch.Tensor]:
 
271
  from tabpfn.constants import ModelVersion
272
  except ImportError as error:
273
  raise ImportError("tabpfn is not installed: uv sync --extra tabpfn") from error
274
+ checkpoint = config.tabpfn_checkpoint
 
275
  classifier = TabPFNClassifier.create_default_for_version(
276
  ModelVersion(config.tabpfn_version),
277
  n_estimators=config.tabpfn_estimators,
 
279
  random_state=seed,
280
  # The context is processed once for all batches of events.
281
  fit_mode="fit_with_cache",
282
+ **({} if checkpoint is None else {"model_path": checkpoint}),
283
+ )
284
+ return _classifier_forward(
285
+ classifier,
286
+ EventInputs.of(
287
+ config, num_features=events.num_features, num_node_types=num_node_types
288
+ ),
289
+ events,
290
+ context,
291
+ num_classes=num_classes,
292
+ batch_size=batch_size,
293
+ device=device,
294
+ )
295
+
296
+
297
+ def tabicl_forward(
298
+ config: ModelConfig,
299
+ events: PackedEvents,
300
+ context: torch.Tensor,
301
+ *,
302
+ num_classes: int,
303
+ num_node_types: int,
304
+ batch_size: int,
305
+ device: torch.device,
306
+ seed: int,
307
+ ) -> Callable[[SequenceBatch], torch.Tensor]:
308
+ """A function from a batch of events to their logits (log-probabilities)
309
+ ``[events, num_classes]`` by TabICL given the labelled ``context``."""
310
+ try:
311
+ from tabicl import TabICLClassifier
312
+ except ImportError as error:
313
+ raise ImportError("tabicl is not installed: uv sync --extra tabicl") from error
314
+ classifier = TabICLClassifier(
315
+ n_estimators=config.tabicl_estimators,
316
+ checkpoint_version=config.tabicl_checkpoint,
317
+ device=str(device),
318
+ random_state=seed,
319
+ # The context is processed once for all batches of events.
320
+ kv_cache=True,
321
+ )
322
+ return _classifier_forward(
323
+ classifier,
324
+ EventInputs.of(
325
+ config, num_features=events.num_features, num_node_types=num_node_types
326
+ ),
327
+ events,
328
+ context,
329
+ num_classes=num_classes,
330
+ batch_size=batch_size,
331
+ device=device,
332
+ )
333
+
334
+
335
+ def tabdpt_forward(
336
+ config: ModelConfig,
337
+ events: PackedEvents,
338
+ context: torch.Tensor,
339
+ *,
340
+ num_classes: int,
341
+ num_node_types: int,
342
+ batch_size: int,
343
+ device: torch.device,
344
+ seed: int,
345
+ ) -> Callable[[SequenceBatch], torch.Tensor]:
346
+ """A function from a batch of events to their logits (log-probabilities)
347
+ ``[events, num_classes]`` by TabDPT given the labelled ``context``; with
348
+ ``model.tabdpt_context_size`` each query's context is its nearest rows.
349
+
350
+ TabDPT computes in float64 in places, so on MPS it runs on the CPU. Its
351
+ retrieval (faiss) and PyTorch load separate OpenMP runtimes on macOS,
352
+ which crash together unless ``OMP_NUM_THREADS=1``."""
353
+ try:
354
+ from tabdpt import TabDPTClassifier
355
+ except ImportError as error:
356
+ raise ImportError("tabdpt is not installed: uv sync --extra tabdpt") from error
357
+ size = config.tabdpt_context_size
358
+ if size is not None and sys.platform == "darwin":
359
+ if os.environ.get("OMP_NUM_THREADS") != "1":
360
+ raise ValueError(
361
+ "TabDPT retrieval on macOS needs OMP_NUM_THREADS=1 (faiss and "
362
+ "torch load separate OpenMP runtimes)"
363
+ )
364
+ classifier = TabDPTClassifier(
365
+ # Absent objects are NaN: imputed, with indicator columns.
366
+ missing_indicators=True,
367
+ context_reduction="subsample" if size is None else "retrieval",
368
+ device="cpu" if device.type == "mps" else str(device),
369
+ verbose=False,
370
  )
371
+
372
+ def predict(rows: Any) -> Any:
373
+ return classifier.ensemble_predict_proba(
374
+ rows, n_ensembles=config.tabdpt_estimators, context_size=size, seed=seed
375
+ )
376
+
377
+ return _classifier_forward(
378
+ classifier,
379
+ EventInputs.of(
380
+ config, num_features=events.num_features, num_node_types=num_node_types
381
+ ),
382
+ events,
383
+ context,
384
+ num_classes=num_classes,
385
+ batch_size=batch_size,
386
+ device=device,
387
+ predict=predict,
388
  )
389
+
390
+
391
+ # --- Meta-training of the tabular networks ------------------------------------
392
+
393
+
394
+ def load_tabpfn_network(version: str, checkpoint: str | None) -> nn.Module:
395
+ """The TabPFN-3.5 network of ``model.tabpfn_version`` and
396
+ ``model.tabpfn_checkpoint`` (default: that version's)."""
397
+ try:
398
+ from tabpfn.model_loading import load_model_criterion_config
399
+ except ImportError as error:
400
+ raise ImportError("tabpfn is not installed: uv sync --extra tabpfn") from error
401
+ if version not in ("v3.5", "v3.5-fast"):
402
+ raise ValueError("TabularPFN trains a TabPFN-3.5 network (v3.5, v3.5-fast)")
403
+ models, *_ = load_model_criterion_config(
404
+ checkpoint,
405
+ check_bar_distribution_criterion=False,
406
+ estimator_type="classifier",
407
+ version=version,
408
+ download_if_not_exists=True,
409
+ )
410
+ return models[0]
411
+
412
+
413
+ def load_tabicl_network(checkpoint: str) -> nn.Module:
414
+ """The TabICL network of ``model.tabicl_checkpoint`` (a path, or a file
415
+ name in the Hugging Face repository ``jingang/TabICL``), recomputing
416
+ activations in the backward pass (gradient checkpointing)."""
417
+ try:
418
+ from huggingface_hub import hf_hub_download
419
+ from tabicl._model.tabicl import TabICL
420
+ except ImportError as error:
421
+ raise ImportError("tabicl is not installed: uv sync --extra tabicl") from error
422
+ path = checkpoint if os.path.exists(checkpoint) else None
423
+ path = path or hf_hub_download(repo_id="jingang/TabICL", filename=checkpoint)
424
+ saved = torch.load(path, map_location="cpu", weights_only=True)
425
+ network = TabICL(**{**saved["config"], "recompute": True})
426
+ network.load_state_dict(saved["state_dict"])
427
+ return network
428
+
429
+
430
+ class LoRA(nn.Module):
431
+ """A parametrization that adds a trainable low-rank update ``up @ down``
432
+ (LoRA; Hu et al. 2022), zero at initialization, to a frozen weight. As a
433
+ parametrization it also reaches weights the code reads directly, such as
434
+ TabICL's fused attention projections."""
435
+
436
+ def __init__(self, weight: torch.Tensor, rank: int) -> None:
437
+ super().__init__()
438
+ out_features, in_features = weight.shape
439
+ down = torch.randn(rank, in_features) / math.sqrt(in_features)
440
+ self.down = nn.Parameter(down.to(weight))
441
+ self.up = nn.Parameter(weight.new_zeros(out_features, rank))
442
+
443
+ def forward(self, weight: torch.Tensor) -> torch.Tensor:
444
+ return weight + (self.up @ self.down).to(weight.dtype)
445
+
446
+
447
+ def add_lora(network: nn.Module, weights: list[str], rank: int) -> None:
448
+ """Freeze ``network`` and give its ``weights`` (dotted names) LoRA
449
+ updates of ``rank``."""
450
+ network.requires_grad_(False)
451
+ for name in weights:
452
+ owner_name, _, attribute = name.rpartition(".")
453
+ owner = network.get_submodule(owner_name)
454
+ weight = getattr(owner, attribute).detach()
455
+ parametrize.register_parametrization(owner, attribute, LoRA(weight, rank))
456
+
457
+
458
+ def merge_lora(network: nn.Module) -> None:
459
+ """Fold the LoRA updates into the weights, for inference (the network
460
+ can no longer be trained or saved as LoRA updates)."""
461
+ for module in list(network.modules()):
462
+ if parametrize.is_parametrized(module):
463
+ for name in list(module.parametrizations):
464
+ parametrize.remove_parametrizations(
465
+ module, name, leave_parametrized=True
466
+ )
467
+
468
+
469
+ class MetaTabular(nn.Module):
470
+ """Query logits ``[tasks * queries, max_classes]`` of a
471
+ :class:`~gnn4colliders.pfn.TaskBatch` by a tabular in-context
472
+ ``network`` on the events' :class:`EventInputs` (interface of
473
+ :class:`~gnn4colliders.pfn.EventPFN`); the logits of classes a task does
474
+ not have are ``ABSENT_LOGIT``.
475
+
476
+ With ``lora_rank > 0`` only the LoRA updates of the network's
477
+ :attr:`lora_weights` train (:func:`add_lora`), else all weights do; an
478
+ encoder of the inputs trains too unless frozen. The state dict holds only
479
+ the trainable weights (and buffers): the rest are those of the released
480
+ network and of ``model.pretrained``.
481
+ """
482
+
483
+ def __init__(
484
+ self,
485
+ network: nn.Module,
486
+ *,
487
+ inputs: EventInputs,
488
+ max_classes: int,
489
+ lora_rank: int,
490
+ ) -> None:
491
+ super().__init__()
492
+ self.network, self.inputs, self.max_classes = network, inputs, max_classes
493
+ if lora_rank > 0:
494
+ add_lora(network, self.lora_weights(), lora_rank)
495
+ self.register_state_dict_post_hook(_drop_frozen)
496
+ self.register_load_state_dict_post_hook(_allow_missing_frozen)
497
+
498
+ def lora_weights(self) -> list[str]:
499
+ """The weights of the network's in-context blocks that LoRA updates."""
500
+ raise NotImplementedError
501
+
502
+ def task_logits(self, rows: torch.Tensor, labels: torch.Tensor) -> torch.Tensor:
503
+ """Logits ``[tasks, queries, classes]`` of tasks of ``rows [tasks,
504
+ context + queries, columns]`` given the context ``labels [tasks,
505
+ context]``."""
506
+ raise NotImplementedError
507
+
508
+ def in_context(
509
+ self, rows: torch.Tensor, labels: torch.Tensor
510
+ ) -> Callable[[torch.Tensor], torch.Tensor]:
511
+ """A function from query rows to their logits ``[queries, classes]``
512
+ given the context ``rows`` and ``labels`` (one task, computed once)."""
513
+ raise NotImplementedError
514
+
515
+ def frozen(self) -> set[str]:
516
+ return {
517
+ name for name, value in self.named_parameters() if not value.requires_grad
518
+ }
519
+
520
+ def forward(self, batch: TaskBatch) -> torch.Tensor:
521
+ tasks = batch.context_labels.shape[0]
522
+ rows = self.inputs(batch.events)
523
+ rows = rows.view(tasks, -1, rows.shape[1])
524
+ logits = self.task_logits(rows, batch.context_labels)
525
+ logits = logits[..., : self.max_classes].float()
526
+ classes = torch.arange(self.max_classes, device=logits.device)
527
+ absent = classes >= batch.num_classes.reshape(-1, 1, 1)
528
+ return logits.masked_fill(absent, ABSENT_LOGIT).flatten(0, 1)
529
+
530
+
531
+ class TabularPFN(MetaTabular):
532
+ """:class:`MetaTabular` of the TabPFN-3.5 network: LoRA updates of the
533
+ attention projections and MLP layers of its 24 in-context blocks."""
534
+
535
+ def lora_weights(self) -> list[str]:
536
+ names = []
537
+ for index in range(len(self.network.icl_blocks)):
538
+ block = f"icl_blocks.{index}"
539
+ names += [
540
+ f"{block}.icl_attention.{name}_projection.weight"
541
+ for name in ("q", "k", "v", "out")
542
+ ]
543
+ names += [f"{block}.mlp.0.weight", f"{block}.mlp.2.weight"]
544
+ return names
545
+
546
+ def options(self) -> Any:
547
+ """TabPFN's performance options; with gradients, activations are
548
+ recomputed in the backward pass (gradient checkpointing)."""
549
+ options = self.network.get_default_performance_options()
550
+ recompute = torch.is_grad_enabled()
551
+ return dataclasses.replace(options, force_recompute_layer=recompute)
552
+
553
+ def task_logits(self, rows: torch.Tensor, labels: torch.Tensor) -> torch.Tensor:
554
+ x = rows.transpose(0, 1) # TabPFN: [rows, tasks, columns]
555
+ y = labels.T.to(x.dtype)
556
+ output = self.network(x, y, "multiclass", performance_options=self.options())
557
+ return output.transpose(0, 1)
558
+
559
+ def in_context(
560
+ self, rows: torch.Tensor, labels: torch.Tensor
561
+ ) -> Callable[[torch.Tensor], torch.Tensor]:
562
+ y = labels.to(rows.dtype)[:, None]
563
+ options = self.options()
564
+ _, cache = self.network(
565
+ rows[:, None],
566
+ y,
567
+ "multiclass",
568
+ performance_options=options,
569
+ return_kv_cache=True,
570
+ )
571
+
572
+ def forward(queries: torch.Tensor) -> torch.Tensor:
573
+ output = self.network(
574
+ queries[:, None],
575
+ y,
576
+ "multiclass",
577
+ performance_options=options,
578
+ kv_cache=cache,
579
+ x_is_test_only=True,
580
+ )
581
+ return output[:, 0]
582
+
583
+ return forward
584
+
585
+
586
+ #: Standardized TabICL inputs are clipped to this many standard deviations.
587
+ TABICL_CLIP = 8.0
588
+
589
+
590
+ def _standardizer(context: torch.Tensor) -> Callable[[torch.Tensor], torch.Tensor]:
591
+ """Standardization of rows ``[..., rows, columns]`` by the mean and
592
+ standard deviation of each task's ``context`` rows, absent (NaN) values
593
+ at the mean, clipped at ``TABICL_CLIP``: TabICL's network takes no NaN
594
+ (its classifier preprocesses similarly, without ensembling)."""
595
+ mean = context.nanmean(dim=-2, keepdim=True)
596
+ std = (context - mean).square().nanmean(dim=-2, keepdim=True).sqrt()
597
+ mean, std = mean.nan_to_num(0.0), std.nan_to_num(1.0).clamp_min(1e-6)
598
+
599
+ def standardize(rows: torch.Tensor) -> torch.Tensor:
600
+ z = ((rows - mean) / std).nan_to_num(0.0)
601
+ return z.clamp(-TABICL_CLIP, TABICL_CLIP)
602
+
603
+ return standardize
604
+
605
+
606
+ class TabularICL(MetaTabular):
607
+ """:class:`MetaTabular` of the TabICL network: LoRA updates of the
608
+ attention projections and feed-forward layers of its 12 in-context
609
+ blocks. The rows are standardized by :func:`_standardizer`."""
610
+
611
+ def lora_weights(self) -> list[str]:
612
+ names = []
613
+ for index in range(len(self.network.icl_predictor.tf_icl.blocks)):
614
+ block = f"icl_predictor.tf_icl.blocks.{index}"
615
+ names += [
616
+ f"{block}.attn.in_proj_weight",
617
+ f"{block}.attn.out_proj.weight",
618
+ f"{block}.linear1.weight",
619
+ f"{block}.linear2.weight",
620
+ ]
621
+ return names
622
+
623
+ def task_logits(self, rows: torch.Tensor, labels: torch.Tensor) -> torch.Tensor:
624
+ context = labels.shape[1]
625
+ x = _standardizer(rows[:, :context])(rows)
626
+ # The training forward, in validation too: its logits cover
627
+ # max_classes whatever the context's classes. In eval mode TabICL's
628
+ # modules switch to their inference path, which chooses its own
629
+ # device; its dropout is 0, so training mode changes nothing else.
630
+ training = self.network.training
631
+ self.network.train()
632
+ try:
633
+ return self.network._train_forward(x, labels.to(x.dtype))
634
+ finally:
635
+ self.network.train(training)
636
+
637
+ def in_context(
638
+ self, rows: torch.Tensor, labels: torch.Tensor
639
+ ) -> Callable[[torch.Tensor], torch.Tensor]:
640
+ from tabicl import InferenceConfig
641
+
642
+ # On the model's device (TabICL picks one itself by default) and in
643
+ # float32, as in meta-training: mixed precision on the CPU changes
644
+ # the predictions substantially.
645
+ inference = InferenceConfig()
646
+ for stage in (inference.COL_CONFIG, inference.ROW_CONFIG, inference.ICL_CONFIG):
647
+ stage.update({"device": rows.device, "use_amp": False})
648
+ standardize = _standardizer(rows)
649
+ self.network.forward_with_cache(
650
+ X_train=standardize(rows)[None],
651
+ y_train=labels.to(rows.dtype)[None],
652
+ store_cache=True,
653
+ inference_config=inference,
654
+ )
655
+
656
+ def forward(queries: torch.Tensor) -> torch.Tensor:
657
+ logits = self.network.forward_with_cache(
658
+ X_test=standardize(queries)[None],
659
+ use_cache=True,
660
+ store_cache=False,
661
+ return_logits=True,
662
+ inference_config=inference,
663
+ )
664
+ return logits[0]
665
+
666
+ return forward
667
+
668
+
669
+ def _drop_frozen(
670
+ module: MetaTabular, state: dict[str, Any], prefix: str, metadata: Any
671
+ ) -> None:
672
+ for name in module.frozen():
673
+ del state[prefix + name]
674
+
675
+
676
+ def _allow_missing_frozen(module: MetaTabular, keys: Any) -> None:
677
+ frozen = module.frozen()
678
+ keys.missing_keys[:] = [key for key in keys.missing_keys if key not in frozen]
679
+
680
+
681
+ @torch.inference_mode()
682
+ def meta_tabular_forward(
683
+ model: MetaTabular,
684
+ events: PackedEvents,
685
+ context: torch.Tensor,
686
+ *,
687
+ num_classes: int,
688
+ batch_size: int,
689
+ device: torch.device,
690
+ ) -> Callable[[SequenceBatch], torch.Tensor]:
691
+ """A function from a batch of events to their logits ``[events,
692
+ num_classes]``, given the labelled ``context`` events (one task). The
693
+ context is processed once."""
694
+ if num_classes > model.max_classes:
695
+ raise ValueError(f"the model tells apart at most {model.max_classes} classes")
696
+ model.eval()
697
+ merge_lora(model.network)
698
+ rows = model.inputs.of_events(events, context, batch_size=batch_size, device=device)
699
+ labels = events.labels[context].to(device)
700
+ logits = model.in_context(rows, labels)
701
 
702
  def forward(batch: SequenceBatch) -> torch.Tensor:
703
+ return logits(model.inputs(batch))[:, :num_classes]
 
 
 
 
704
 
705
  return forward
src/gnn4colliders/tasks.py CHANGED
@@ -31,6 +31,13 @@ def per_class_weighted_mean(
31
  return torch.where(present, means, torch.zeros_like(means)).sum() / present.sum()
32
 
33
 
 
 
 
 
 
 
 
34
  def weighted_binary_auc(
35
  labels: torch.Tensor, scores: torch.Tensor, weights: torch.Tensor
36
  ) -> float:
@@ -39,9 +46,10 @@ def weighted_binary_auc(
39
  Equals the weighted fraction of (positive, negative) pairs ordered
40
  correctly. Returns NaN without positive or negative weight.
41
  """
42
- labels = labels.detach().to(torch.bool).reshape(-1)
43
- scores = scores.detach().to(torch.float64).reshape(-1)
44
- weights = weights.detach().to(device=scores.device, dtype=torch.float64).reshape(-1)
 
45
  zero = weights.new_zeros(())
46
  positive = torch.where(labels, weights, zero)
47
  negative = torch.where(labels, zero, weights)
@@ -72,6 +80,7 @@ def macro_scores(
72
  minlength=num_classes * num_classes,
73
  )
74
  .reshape(num_classes, num_classes)
 
75
  .to(torch.float64)
76
  )
77
  hits, predicted, actual = confusion.diagonal(), confusion.sum(0), confusion.sum(1)
@@ -172,10 +181,14 @@ class BinaryClassificationTask:
172
  }
173
 
174
 
175
- class JepaTask:
176
- """The loss of JEPA pretraining. The model's outputs per event are
177
- ``[loss, has_targets, target_spread]`` (:class:`~gnn4colliders.jepa.JEPA`);
178
- labels and weights are not used."""
 
 
 
 
179
 
180
  @staticmethod
181
  def loss(
@@ -184,11 +197,10 @@ class JepaTask:
184
  has_targets = outputs[:, 1]
185
  return (outputs[:, 0] * has_targets).sum() / has_targets.sum().clamp_min(1)
186
 
187
- @staticmethod
188
  def metrics(
189
- outputs: torch.Tensor, labels: torch.Tensor, weight: torch.Tensor
190
  ) -> dict[str, float]:
191
- return {"target_spread": float(outputs[:, 2].mean())}
192
 
193
 
194
  class PfnTask(MulticlassClassificationTask):
@@ -213,7 +225,11 @@ def build_task(name: str, *, absolute_weights: bool = False, threshold: float =
213
  absolute_weights=absolute_weights, threshold=threshold
214
  )
215
  if name == "jepa":
216
- return JepaTask()
 
 
217
  if name == "pfn":
218
  return PfnTask()
219
- raise ValueError(f"unknown task {name!r}; use multiclass, binary, jepa or pfn")
 
 
 
31
  return torch.where(present, means, torch.zeros_like(means)).sum() / present.sum()
32
 
33
 
34
+ def float64_device(tensor: torch.Tensor) -> torch.device:
35
+ """Where float64 metrics of ``tensor`` run: the CPU for MPS, which has no
36
+ float64, else its own device. Move, then cast: on PyTorch 2.14 a combined
37
+ ``.to(device="cpu", dtype=torch.float64)`` of an MPS tensor gives zeros."""
38
+ return torch.device("cpu") if tensor.device.type == "mps" else tensor.device
39
+
40
+
41
  def weighted_binary_auc(
42
  labels: torch.Tensor, scores: torch.Tensor, weights: torch.Tensor
43
  ) -> float:
 
46
  Equals the weighted fraction of (positive, negative) pairs ordered
47
  correctly. Returns NaN without positive or negative weight.
48
  """
49
+ device = float64_device(scores)
50
+ labels = labels.detach().to(device).to(torch.bool).reshape(-1)
51
+ scores = scores.detach().to(device).to(torch.float64).reshape(-1)
52
+ weights = weights.detach().to(device).to(torch.float64).reshape(-1)
53
  zero = weights.new_zeros(())
54
  positive = torch.where(labels, weights, zero)
55
  negative = torch.where(labels, zero, weights)
 
80
  minlength=num_classes * num_classes,
81
  )
82
  .reshape(num_classes, num_classes)
83
+ .to(float64_device(labels))
84
  .to(torch.float64)
85
  )
86
  hits, predicted, actual = confusion.diagonal(), confusion.sum(0), confusion.sum(1)
 
181
  }
182
 
183
 
184
+ class PretrainingTask:
185
+ """The loss of JEPA or contrastive pretraining. The model's outputs per
186
+ event are ``[loss, counts, spread]`` (:class:`~gnn4colliders.jepa.JEPA`,
187
+ :class:`~gnn4colliders.contrastive.Contrastive`), the spread reported as
188
+ ``spread_name``; labels and weights are not used."""
189
+
190
+ def __init__(self, spread_name: str) -> None:
191
+ self.spread_name = spread_name
192
 
193
  @staticmethod
194
  def loss(
 
197
  has_targets = outputs[:, 1]
198
  return (outputs[:, 0] * has_targets).sum() / has_targets.sum().clamp_min(1)
199
 
 
200
  def metrics(
201
+ self, outputs: torch.Tensor, labels: torch.Tensor, weight: torch.Tensor
202
  ) -> dict[str, float]:
203
+ return {self.spread_name: float(outputs[:, 2].mean())}
204
 
205
 
206
  class PfnTask(MulticlassClassificationTask):
 
225
  absolute_weights=absolute_weights, threshold=threshold
226
  )
227
  if name == "jepa":
228
+ return PretrainingTask("target_spread")
229
+ if name == "contrastive":
230
+ return PretrainingTask("embedding_spread")
231
  if name == "pfn":
232
  return PfnTask()
233
+ raise ValueError(
234
+ f"unknown task {name!r}; use multiclass, binary, jepa, contrastive or pfn"
235
+ )
tests/conftest.py CHANGED
@@ -14,6 +14,17 @@ FIXTURES = ROOT / "tests/fixtures"
14
  GAMGAM_FIXTURES = sorted((FIXTURES / "atlas_opendata_gamgam").glob("mc_*_64.root"))
15
  TTH_FIXTURE = FIXTURES / "ttH_NLO_64.root"
16
  PRETRAINED = ROOT / "Pretrained_GNN"
 
 
 
 
 
 
 
 
 
 
 
17
 
18
 
19
  def random_events(count: int = 40, *, seed: int = 0, classes: int = 3) -> PackedEvents:
@@ -28,7 +39,7 @@ def random_events(count: int = 40, *, seed: int = 0, classes: int = 3) -> Packed
28
  def fixture_config(
29
  name: str, files: list[Path], labels: list[int], tmp_path: Path, *overrides: str
30
  ) -> Config:
31
- """``configs/<name>.yaml`` on local fixture files, on the CPU, in tmp_path."""
32
  entries = [
33
  {"path": str(path), "label": label} for path, label in zip(files, labels)
34
  ]
@@ -42,7 +53,7 @@ def fixture_config(
42
  "data.loader_workers=0",
43
  "data.pin_memory=false",
44
  "data.batch_size=64",
45
- "train.device=cpu",
46
  f"run_dir={tmp_path}/run",
47
  *overrides,
48
  ],
 
14
  GAMGAM_FIXTURES = sorted((FIXTURES / "atlas_opendata_gamgam").glob("mc_*_64.root"))
15
  TTH_FIXTURE = FIXTURES / "ttH_NLO_64.root"
16
  PRETRAINED = ROOT / "Pretrained_GNN"
17
+ #: ``train.device`` of the fixture runs: ``pytest --device mps`` (default cpu).
18
+ DEVICE = "cpu"
19
+
20
+
21
+ def pytest_addoption(parser: pytest.Parser) -> None:
22
+ parser.addoption("--device", default="cpu", help="train.device of fixture runs")
23
+
24
+
25
+ def pytest_configure(config: pytest.Config) -> None:
26
+ global DEVICE
27
+ DEVICE = config.getoption("--device")
28
 
29
 
30
  def random_events(count: int = 40, *, seed: int = 0, classes: int = 3) -> PackedEvents:
 
39
  def fixture_config(
40
  name: str, files: list[Path], labels: list[int], tmp_path: Path, *overrides: str
41
  ) -> Config:
42
+ """``configs/<name>.yaml`` on local fixture files, on ``DEVICE``, in tmp_path."""
43
  entries = [
44
  {"path": str(path), "label": label} for path, label in zip(files, labels)
45
  ]
 
53
  "data.loader_workers=0",
54
  "data.pin_memory=false",
55
  "data.batch_size=64",
56
+ f"train.device={DEVICE}",
57
  f"run_dir={tmp_path}/run",
58
  *overrides,
59
  ],
tests/test_contrastive.py ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Contrastive pretraining of both backbones: the random views, the NT-Xent
2
+ loss, and pretraining then fine-tuning and probing."""
3
+
4
+ import json
5
+ import math
6
+
7
+ import pytest
8
+ import torch
9
+ from conftest import GAMGAM_FIXTURES, TTH_FIXTURE, fixture_config, random_events
10
+
11
+ from gnn4colliders.cli import main
12
+ from gnn4colliders.config import ModelConfig, save_config
13
+ from gnn4colliders.contrastive import random_view
14
+ from gnn4colliders.models import build_model
15
+ from gnn4colliders.sequences import ETA, PHI, PT, collate_sequence_batch
16
+
17
+ BACKBONES = {
18
+ "transformer": {"hid_size": 16, "n_layers": 1, "n_heads": 2, "pair_bias": True},
19
+ "edge_network": {"hid_size": 16, "n_layers": 2, "n_proc_steps": 2, "n_heads": 2},
20
+ }
21
+ backbones = pytest.mark.parametrize("backbone", BACKBONES)
22
+
23
+
24
+ def _batch():
25
+ events = random_events()
26
+ events.object_features[:, 6] = torch.randint(
27
+ 0, 5, (events.object_features.shape[0],)
28
+ ).float()
29
+ return collate_sequence_batch(events, torch.arange(16))
30
+
31
+
32
+ def test_views_rotate_flip_scale_and_drop_objects_but_keep_the_first():
33
+ batch = _batch()
34
+ tokens, mask = batch.tokens, batch.token_mask
35
+ generator = torch.Generator().manual_seed(0)
36
+ view, kept = random_view(
37
+ tokens, mask, scale_jitter=0.2, object_dropout=0.5, generator=generator
38
+ )
39
+ assert kept[:, 0].all() and not (kept & ~mask).any() and (mask & ~kept).any()
40
+ # One factor per event on pt, one sign on eta, azimuths in [-pi, pi).
41
+ ratio = view[..., PT][mask] / tokens[..., PT][mask]
42
+ assert ((ratio > 0.8 - 1e-6) & (ratio < 1.2 + 1e-6)).all()
43
+ sign = view[..., ETA][mask] / tokens[..., ETA][mask]
44
+ torch.testing.assert_close(sign.abs(), torch.ones_like(sign))
45
+ assert (view[..., PHI].abs() <= math.pi).all()
46
+ torch.testing.assert_close(view[~mask], tokens[~mask])
47
+
48
+
49
+ @backbones
50
+ def test_loss_is_finite_and_trains_every_encoder_parameter(backbone):
51
+ torch.manual_seed(0)
52
+ config = ModelConfig(name="contrastive", backbone=backbone, **BACKBONES[backbone])
53
+ model = build_model(config, num_features=7, num_node_types=5).train()
54
+ outputs = model(_batch())
55
+ assert outputs.shape == (16, 3) and torch.isfinite(outputs).all()
56
+ # A loss above log(1) and at most about log(2 * 16 - 1) at initialization.
57
+ assert 0 < outputs[:, 0].mean() < math.log(31) + 1
58
+ outputs[:, 0].mean().backward()
59
+ for name, parameter in model.context_encoder.named_parameters():
60
+ assert parameter.grad is not None and parameter.grad.abs().sum() > 0, name
61
+
62
+
63
+ @pytest.mark.parametrize(
64
+ ("backbone", "pretrain_config", "finetune_config", "overrides"),
65
+ [
66
+ ("transformer", "delphes_contrastive", "gamgam_jepa_finetune", []),
67
+ (
68
+ "edge_network",
69
+ "delphes_contrastive",
70
+ "gamgam_finetune",
71
+ ["model.backbone=edge_network"],
72
+ ),
73
+ ("transformer", "delphes_masked", "gamgam_jepa_finetune", []),
74
+ ],
75
+ )
76
+ def test_pretrain_then_fine_tune_and_probe_on_gamgam(
77
+ backbone, pretrain_config, finetune_config, overrides, tmp_path
78
+ ):
79
+ small = [f"model.{key}={value}" for key, value in BACKBONES[backbone].items()]
80
+ pretrain = fixture_config(
81
+ pretrain_config,
82
+ [TTH_FIXTURE, TTH_FIXTURE],
83
+ [0, 1],
84
+ tmp_path / "pretrain",
85
+ "train.max_epochs=2",
86
+ *small,
87
+ *overrides,
88
+ )
89
+ path = str(save_config(pretrain, tmp_path / "pretrain.yaml"))
90
+ assert main(["prepare", path]) == 0
91
+ assert main(["train", path]) == 0
92
+ history = [
93
+ json.loads(line)
94
+ for line in (pretrain.path / "history.jsonl").read_text().splitlines()
95
+ ]
96
+ spread = (
97
+ "embedding_spread" if pretrain.model.name == "contrastive" else "target_spread"
98
+ )
99
+ assert len(history) == 2 and history[1]["validation"][spread] > 0
100
+ # The pretraining config probes its own encoder, on its labels.
101
+ probe = ["data.train_events=40", "model.probe_neighbours=5", "split=validation"]
102
+ probe.append(f"model.pretrained={pretrain.path}/last.pt")
103
+ assert main(["probe", path, *probe]) == 0
104
+ summary = json.loads((pretrain.path / "validation_probe.json").read_text())
105
+ assert math.isfinite(summary["knn"]["accuracy"])
106
+
107
+ finetune = fixture_config(
108
+ finetune_config,
109
+ GAMGAM_FIXTURES,
110
+ list(range(5)),
111
+ tmp_path / "finetune",
112
+ "train.max_epochs=1",
113
+ f"model.pretrained={pretrain.path}/last.pt",
114
+ "data.train_events=100",
115
+ "model.probe_neighbours=5",
116
+ *small,
117
+ )
118
+ path = str(save_config(finetune, tmp_path / "finetune.yaml"))
119
+ assert main(["prepare", path]) == 0
120
+ assert main(["train", path]) == 0
121
+ assert main(["probe", path]) == 0
122
+ summary = json.loads((finetune.path / "test_probe.json").read_text())
123
+ assert math.isfinite(summary["linear"]["auc"])
tests/test_jepa.py CHANGED
@@ -1,7 +1,8 @@
1
- """JEPA pretraining of both backbones: masking, the predictor, the EMA target
2
- and fine-tuning."""
3
 
4
  import json
 
5
 
6
  import pytest
7
  import torch
@@ -9,7 +10,13 @@ from conftest import GAMGAM_FIXTURES, TTH_FIXTURE, fixture_config, random_events
9
 
10
  from gnn4colliders.cli import main
11
  from gnn4colliders.config import ModelConfig, save_config
12
- from gnn4colliders.jepa import _variance_penalty, sample_targets, type_index
 
 
 
 
 
 
13
  from gnn4colliders.models import build_model
14
  from gnn4colliders.sequences import collate_sequence_batch
15
  from gnn4colliders.train import load_checkpoint
@@ -23,9 +30,11 @@ BACKBONES = {
23
  backbones = pytest.mark.parametrize("backbone", BACKBONES)
24
 
25
 
26
- def _jepa(backbone="transformer"):
27
  torch.manual_seed(0)
28
- config = ModelConfig(name="jepa", backbone=backbone, **BACKBONES[backbone])
 
 
29
  return build_model(config, num_features=7, num_node_types=5)
30
 
31
 
@@ -77,6 +86,27 @@ def test_loss_is_finite_and_trains_every_context_parameter(backbone):
77
  assert parameter.grad is not None and parameter.grad.abs().sum() > 0, name
78
 
79
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80
  def test_variance_term_penalizes_only_collapsed_dimensions():
81
  spread = torch.randn(4096, 8, generator=torch.Generator().manual_seed(0))
82
  assert _variance_penalty(spread) < 0.03
 
1
+ """JEPA pretraining of both backbones: masking, the predictor, the EMA target,
2
+ kinematic targets (masked objects) and fine-tuning."""
3
 
4
  import json
5
+ import math
6
 
7
  import pytest
8
  import torch
 
10
 
11
  from gnn4colliders.cli import main
12
  from gnn4colliders.config import ModelConfig, save_config
13
+ from gnn4colliders.jepa import (
14
+ FEATURE_TARGETS,
15
+ _variance_penalty,
16
+ object_features,
17
+ sample_targets,
18
+ type_index,
19
+ )
20
  from gnn4colliders.models import build_model
21
  from gnn4colliders.sequences import collate_sequence_batch
22
  from gnn4colliders.train import load_checkpoint
 
30
  backbones = pytest.mark.parametrize("backbone", BACKBONES)
31
 
32
 
33
+ def _jepa(backbone="transformer", target="encoding"):
34
  torch.manual_seed(0)
35
+ config = ModelConfig(
36
+ name="jepa", backbone=backbone, jepa_target=target, **BACKBONES[backbone]
37
+ )
38
  return build_model(config, num_features=7, num_node_types=5)
39
 
40
 
 
86
  assert parameter.grad is not None and parameter.grad.abs().sum() > 0, name
87
 
88
 
89
+ def test_kinematic_targets_of_objects():
90
+ tokens = torch.tensor([[[1.0, 0.5, math.pi / 2, 2.0, 1.0, -1.0, 0.0]]])
91
+ torch.testing.assert_close(
92
+ object_features(tokens),
93
+ torch.tensor([[[math.log(2), 0.5, 0.0, 1.0, math.log(3), 1.0, -1.0]]]),
94
+ )
95
+
96
+
97
+ @backbones
98
+ def test_masked_objects_predict_kinematics_without_a_target_encoder(backbone):
99
+ model, batch = _jepa(backbone, "features").train(), _batch()
100
+ assert model.target_encoder is None
101
+ model.update_target() # nothing to update
102
+ outputs = model(batch)
103
+ assert outputs.shape == (16, 3) and torch.isfinite(outputs).all()
104
+ assert model.predictor_out.out_features == FEATURE_TARGETS
105
+ outputs[:, 0].mean().backward()
106
+ for name, parameter in model.context_encoder.named_parameters():
107
+ assert parameter.grad is not None and parameter.grad.abs().sum() > 0, name
108
+
109
+
110
  def test_variance_term_penalizes_only_collapsed_dimensions():
111
  spread = torch.randn(4096, 8, generator=torch.Generator().manual_seed(0))
112
  assert _variance_penalty(spread) < 0.03
tests/test_probe.py ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """k-NN and linear probes of frozen encoders: the classifiers, the encoder
2
+ weights of each kind of checkpoint, and the probe command."""
3
+
4
+ import json
5
+ import math
6
+
7
+ import pytest
8
+ import torch
9
+ from conftest import GAMGAM_FIXTURES, TTH_FIXTURE, fixture_config
10
+
11
+ from gnn4colliders.cli import main
12
+ from gnn4colliders.config import ModelConfig, save_config
13
+ from gnn4colliders.data import load_events
14
+ from gnn4colliders.models import encoder_state, pretrained_encoder
15
+ from gnn4colliders.probe import knn_logits, linear_logits
16
+
17
+ SMALL = {"hid_size": 16, "n_layers": 1, "n_heads": 2, "pair_bias": True}
18
+ BACKBONES = {
19
+ "transformer": SMALL,
20
+ "edge_network": {"hid_size": 16, "n_layers": 2, "n_proc_steps": 2, "n_heads": 2},
21
+ }
22
+
23
+
24
+ def _clusters(count=60, seed=0):
25
+ """Two well separated classes of 8-dimensional embeddings."""
26
+ generator = torch.Generator().manual_seed(seed)
27
+ labels = torch.arange(count) % 2
28
+ centres = torch.tensor([[3.0] + [0.0] * 7, [0.0, 3.0] + [0.0] * 6])
29
+ return centres[labels] + 0.3 * torch.randn(count, 8, generator=generator), labels
30
+
31
+
32
+ def test_knn_and_linear_probes_separate_clusters():
33
+ train, labels = _clusters(seed=0)
34
+ queries, expected = _clusters(seed=1)
35
+ for logits in (
36
+ knn_logits(train, labels, queries, num_classes=2, neighbours=5),
37
+ linear_logits(train, labels, queries, num_classes=2, l2=1e-4),
38
+ ):
39
+ assert logits.shape == (60, 2)
40
+ assert torch.equal(logits.argmax(dim=1), expected)
41
+
42
+
43
+ def test_encoder_weights_of_each_kind_of_checkpoint(tmp_path):
44
+ weight = torch.ones(2)
45
+ checkpoints = {
46
+ "jepa": {"context_encoder.norm.weight": weight, "target_encoder.x": weight},
47
+ "transformer": {"encoder.norm.weight": weight, "classifier.weight": weight},
48
+ "edge_network": {"node_encoder.0.weight": weight, "classifier.weight": weight},
49
+ "fine_tuned": {"backbone.node_encoder.0.weight": weight, "head.weight": weight},
50
+ "tabular": {"inputs.encoder.norm.weight": weight, "network.x.up": weight},
51
+ }
52
+ expected = {
53
+ "jepa": ["norm.weight"],
54
+ "transformer": ["norm.weight"],
55
+ "edge_network": ["node_encoder.0.weight"],
56
+ "fine_tuned": ["node_encoder.0.weight"],
57
+ "tabular": ["norm.weight"],
58
+ }
59
+ for name, state in checkpoints.items():
60
+ path = tmp_path / f"{name}.pt"
61
+ torch.save({"model_state_dict": state}, path)
62
+ assert list(encoder_state(str(path))) == expected[name]
63
+
64
+
65
+ @pytest.mark.parametrize(
66
+ ("backbone", "pretrain_config", "probe_config"),
67
+ [
68
+ ("transformer", "delphes_jepa", "gamgam_jepa_finetune"),
69
+ ("edge_network", "delphes_jepa_edge", "gamgam_finetune"),
70
+ ],
71
+ )
72
+ def test_probe_random_and_jepa_encoders_on_gamgam(
73
+ backbone, pretrain_config, probe_config, tmp_path
74
+ ):
75
+ small = [f"model.{key}={value}" for key, value in BACKBONES[backbone].items()]
76
+ pretrain = fixture_config(
77
+ pretrain_config,
78
+ [TTH_FIXTURE, TTH_FIXTURE],
79
+ [0, 1],
80
+ tmp_path / "jepa",
81
+ "train.max_epochs=1",
82
+ *small,
83
+ )
84
+ path = str(save_config(pretrain, tmp_path / "jepa.yaml"))
85
+ assert main(["prepare", path]) == 0
86
+ assert main(["train", path]) == 0
87
+
88
+ for name, encoder in (("random", "null"), ("jepa", f"{pretrain.path}/last.pt")):
89
+ config = fixture_config(
90
+ probe_config,
91
+ GAMGAM_FIXTURES,
92
+ list(range(5)),
93
+ tmp_path / name,
94
+ f"model.pretrained={encoder}",
95
+ "data.train_events=100",
96
+ "model.probe_neighbours=5",
97
+ "split=validation",
98
+ *small,
99
+ )
100
+ path = str(save_config(config, tmp_path / f"{name}.yaml"))
101
+ assert main(["prepare", path]) == 0
102
+ assert main(["probe", path]) == 0
103
+ summary = json.loads((config.path / "validation_probe.json").read_text())
104
+ assert summary["train_events"] == 100
105
+ for probe in ("knn", "linear"):
106
+ assert math.isfinite(summary[probe]["auc"])
107
+
108
+
109
+ def test_probe_a_supervised_transformer(tmp_path):
110
+ small = [f"model.{key}={value}" for key, value in SMALL.items()]
111
+ trained = fixture_config(
112
+ "gamgam_transformer",
113
+ GAMGAM_FIXTURES,
114
+ list(range(5)),
115
+ tmp_path / "supervised",
116
+ "train.max_epochs=1",
117
+ *small,
118
+ )
119
+ path = str(save_config(trained, tmp_path / "supervised.yaml"))
120
+ assert main(["prepare", path]) == 0
121
+ assert main(["train", path]) == 0
122
+ config = fixture_config(
123
+ "gamgam_transformer",
124
+ GAMGAM_FIXTURES,
125
+ list(range(5)),
126
+ tmp_path / "probe",
127
+ f"model.pretrained={trained.path}/last.pt",
128
+ "data.train_events=50",
129
+ *small,
130
+ )
131
+ path = str(save_config(config, tmp_path / "probe.yaml"))
132
+ assert main(["prepare", path]) == 0
133
+ assert main(["probe", path]) == 0
134
+ summary = json.loads((config.path / "test_probe.json").read_text())
135
+ assert summary["encoder"] == f"{trained.path}/last.pt"
136
+ # On the training split, the probes' own events are not queries.
137
+ assert main(["probe", path, "split=train"]) == 0
138
+ summary = json.loads((config.path / "train_probe.json").read_text())
139
+ events = load_events(config.data.cache)
140
+ train_fold = torch.isin(events.fold, torch.tensor(config.data.train_folds))
141
+ assert summary["num_samples"] == int(train_fold.sum()) - 50
142
+
143
+
144
+ def test_a_random_encoder_is_rebuilt_identically_without_touching_global_state():
145
+ config = ModelConfig(name="transformer", **SMALL)
146
+ state = torch.random.get_rng_state()
147
+ first, second = (
148
+ pretrained_encoder(config, num_features=7, num_node_types=5) for _ in range(2)
149
+ )
150
+ assert torch.equal(torch.random.get_rng_state(), state)
151
+ for a, b in zip(first.parameters(), second.parameters()):
152
+ assert torch.equal(a, b)
tests/test_tabular.py CHANGED
@@ -1,17 +1,31 @@
1
- """Fixed-width event tables for TabPFN, balanced label subsets and TabPFN in
2
- context (when installed)."""
 
3
 
4
  import json
5
  import math
 
 
6
 
7
  import pytest
8
  import torch
9
- from conftest import GAMGAM_FIXTURES, fixture_config
 
 
10
 
11
  from gnn4colliders.cli import main
12
- from gnn4colliders.config import save_config
13
  from gnn4colliders.data import pack_events
14
- from gnn4colliders.tabular import table_features
 
 
 
 
 
 
 
 
 
15
 
16
 
17
  def _objects(types):
@@ -23,8 +37,9 @@ def _objects(types):
23
  def test_table_keeps_the_leading_objects_of_each_type():
24
  first, second = _objects([0, 0, 0, 3, 3, 4]), _objects([1, 4])
25
  events = pack_events([first, second], [0, 1])
26
- table = table_features(events, torch.tensor([0, 1]), [2, 1, 1, 1, 1])
27
- assert table.shape == (2, 6 * 6 + 5)
 
28
  slots = table[:, :36].view(2, 6, 6)
29
  # Slots: jet, jet, electron, muon, photon, MET.
30
  torch.testing.assert_close(slots[0, :2], first[:2, :6])
@@ -36,6 +51,67 @@ def test_table_keeps_the_leading_objects_of_each_type():
36
  assert table[:, 36:].tolist() == [[3, 0, 0, 2, 1], [0, 1, 0, 0, 1]]
37
 
38
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
  def test_balanced_subsets_are_equal_per_class_and_nested():
40
  labels = torch.tensor([0] * 50 + [1] * 30 + [2] * 5)
41
  events = pack_events([torch.zeros(1, 7)] * len(labels), labels)
@@ -67,3 +143,261 @@ def test_tabpfn_classifies_gamgam_in_context(tmp_path):
67
  assert main(["evaluate", path]) == 0
68
  metrics = json.loads((config.path / "validation_metrics.json").read_text())
69
  assert not math.isnan(metrics["metrics"]["auc"])
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Fixed-width event tables, balanced label subsets, LoRA, TabICL in
2
+ context, and TabPFN in context and meta-trained (when installed, with its
3
+ weights)."""
4
 
5
  import json
6
  import math
7
+ import os
8
+ import sys
9
 
10
  import pytest
11
  import torch
12
+ from conftest import GAMGAM_FIXTURES, TTH_FIXTURE, fixture_config
13
+ from torch import nn
14
+ from torch.nn.utils import parametrize
15
 
16
  from gnn4colliders.cli import main
17
+ from gnn4colliders.config import ModelConfig, save_config
18
  from gnn4colliders.data import pack_events
19
+ from gnn4colliders.sequences import collate_sequence_batch
20
+ from gnn4colliders.tabular import (
21
+ EventInputs,
22
+ RowLayout,
23
+ add_lora,
24
+ merge_lora,
25
+ sequence_table,
26
+ table_features,
27
+ )
28
+ from gnn4colliders.train import load_checkpoint
29
 
30
 
31
  def _objects(types):
 
37
  def test_table_keeps_the_leading_objects_of_each_type():
38
  first, second = _objects([0, 0, 0, 3, 3, 4]), _objects([1, 4])
39
  events = pack_events([first, second], [0, 1])
40
+ layout = RowLayout((2, 1, 1, 1, 1))
41
+ table = table_features(events, torch.tensor([0, 1]), layout)
42
+ assert table.shape == (2, layout.width) == (2, 6 * 6 + 5)
43
  slots = table[:, :36].view(2, 6, 6)
44
  # Slots: jet, jet, electron, muon, photon, MET.
45
  torch.testing.assert_close(slots[0, :2], first[:2, :6])
 
51
  assert table[:, 36:].tolist() == [[3, 0, 0, 2, 1], [0, 1, 0, 0, 1]]
52
 
53
 
54
+ def test_sequence_batches_give_the_same_table():
55
+ events = pack_events(
56
+ [_objects([0, 0, 0, 3, 3, 4]), _objects([1, 4]), _objects([2, 2, 2, 0])],
57
+ [0, 1, 2],
58
+ )
59
+ indices = torch.tensor([2, 0, 1])
60
+ layout = RowLayout((2, 1, 1, 1, 1), relative_phi=True, pair_objects=3)
61
+ torch.testing.assert_close(
62
+ sequence_table(collate_sequence_batch(events, indices), layout),
63
+ table_features(events, indices, layout),
64
+ equal_nan=True,
65
+ )
66
+
67
+
68
+ def _object(pt, eta, phi, node_type):
69
+ return [pt, eta, phi, pt * math.cosh(eta), 0.0, 0.0, node_type]
70
+
71
+
72
+ def test_relative_phi_and_the_masses_and_delta_r_of_leading_pairs():
73
+ # Two back-to-back photons (mass 2, Delta R pi) and a softer jet; an
74
+ # event with one photon has no pair.
75
+ pair = torch.tensor(
76
+ [_object(1.0, 0.0, 0.5, 3), _object(1.0, 0.0, 0.5 - math.pi, 3)]
77
+ + [_object(0.5, 1.0, 2.0, 0)]
78
+ )
79
+ single = torch.tensor([_object(1.0, 0.3, 1.0, 3)])
80
+ events = pack_events([pair, single], [0, 1])
81
+ layout = RowLayout((1, 0, 0, 2, 0), relative_phi=True, pair_objects=2)
82
+ table = table_features(events, torch.tensor([0, 1]), layout)
83
+ assert table.shape == (2, layout.width) == (2, 3 * 6 + 5 + 2)
84
+ slots = table[:, :18].view(2, 3, 6)
85
+ # Azimuths from the leading photon's.
86
+ torch.testing.assert_close(slots[0, :, 2], torch.tensor([1.5, 0.0, -math.pi]))
87
+ assert slots[1, 1, 2] == 0
88
+ torch.testing.assert_close(table[0, -2:], torch.tensor([2.0, math.pi]))
89
+ assert table[1, -2:].isnan().all()
90
+
91
+
92
+ def test_lora_starts_at_the_frozen_weight_and_trains_only_its_update():
93
+ network = nn.Sequential(nn.Linear(6, 4))
94
+ weight, x = network[0].weight.detach().clone(), torch.randn(5, 6)
95
+ expected = network(x)
96
+ add_lora(network, ["0.weight"], rank=2)
97
+ torch.testing.assert_close(network(x), expected)
98
+ trainable = [name for name, p in network.named_parameters() if p.requires_grad]
99
+ assert sorted(trainable) == [
100
+ "0.parametrizations.weight.0.down",
101
+ "0.parametrizations.weight.0.up",
102
+ ]
103
+ # Code that reads the weight directly sees the update too.
104
+ with torch.no_grad():
105
+ network[0].parametrizations.weight[0].up.fill_(1.0)
106
+ lora = network[0].parametrizations.weight[0]
107
+ torch.testing.assert_close(network[0].weight, weight + lora.up @ lora.down)
108
+ # Merged for inference, the network computes the same.
109
+ expected = network(x)
110
+ merge_lora(network)
111
+ assert not parametrize.is_parametrized(network[0])
112
+ torch.testing.assert_close(network(x), expected)
113
+
114
+
115
  def test_balanced_subsets_are_equal_per_class_and_nested():
116
  labels = torch.tensor([0] * 50 + [1] * 30 + [2] * 5)
117
  events = pack_events([torch.zeros(1, 7)] * len(labels), labels)
 
143
  assert main(["evaluate", path]) == 0
144
  metrics = json.loads((config.path / "validation_metrics.json").read_text())
145
  assert not math.isnan(metrics["metrics"]["auc"])
146
+ assert metrics["checkpoint"] == "TabPFN v2 (default checkpoint)"
147
+
148
+
149
+ def _tabpfn_3_5_weights() -> None:
150
+ pytest.importorskip("tabpfn")
151
+ from tabpfn.model_loading import get_cache_dir
152
+
153
+ if not (get_cache_dir() / "tabpfn-v3.5-20260909.safetensors").exists():
154
+ pytest.skip("TabPFN-3.5 weights are not in the TabPFN cache")
155
+
156
+
157
+ def test_meta_train_tabpfn_with_lora_then_classify_gamgam(tmp_path):
158
+ _tabpfn_3_5_weights()
159
+ meta = fixture_config(
160
+ "delphes_tabpfn",
161
+ [TTH_FIXTURE, TTH_FIXTURE],
162
+ [0, 1],
163
+ tmp_path / "meta",
164
+ "train.max_epochs=1",
165
+ "task.context_events=[4,16]",
166
+ "task.query_events=4",
167
+ "task.steps_per_epoch=2",
168
+ "task.validation_steps=1",
169
+ "model.relative_phi=true",
170
+ "model.pair_objects=4",
171
+ )
172
+ path = str(save_config(meta, tmp_path / "meta.yaml"))
173
+ assert main(["prepare", path]) == 0
174
+ assert main(["train", path]) == 0
175
+ # Of the weights, only the LoRA updates are saved: 24 blocks of 4
176
+ # projections and 2 MLP layers.
177
+ state = load_checkpoint(meta.path / "best.pt")["model_state_dict"]
178
+ weights = [key for key in state if key.endswith(".weight")]
179
+ updates = [key for key in state if key.endswith((".down", ".up"))]
180
+ assert not weights and len(updates) == 24 * 6 * 2
181
+
182
+ gamgam = fixture_config(
183
+ "gamgam_tabpfn",
184
+ GAMGAM_FIXTURES,
185
+ list(range(5)),
186
+ tmp_path / "gamgam",
187
+ f"checkpoint={meta.path}/best.pt",
188
+ "data.train_events=40",
189
+ "split=validation",
190
+ "model.relative_phi=true",
191
+ "model.pair_objects=4",
192
+ )
193
+ path = str(save_config(gamgam, tmp_path / "gamgam.yaml"))
194
+ assert main(["prepare", path]) == 0
195
+ assert main(["evaluate", path]) == 0
196
+ metrics = json.loads((gamgam.path / "validation_metrics.json").read_text())
197
+ assert metrics["checkpoint"] == f"{meta.path}/best.pt"
198
+ assert math.isfinite(metrics["metrics"]["auc"])
199
+
200
+
201
+ def test_tabicl_classifies_gamgam_in_context_with_physics_features(tmp_path):
202
+ pytest.importorskip("tabicl")
203
+ config = fixture_config(
204
+ "gamgam_tabicl",
205
+ GAMGAM_FIXTURES,
206
+ list(range(5)),
207
+ tmp_path,
208
+ "data.train_events=100",
209
+ "model.tabicl_estimators=1",
210
+ "model.relative_phi=true",
211
+ "model.pair_objects=4",
212
+ "split=validation",
213
+ )
214
+ path = str(save_config(config, tmp_path / "tabicl.yaml"))
215
+ assert main(["prepare", path]) == 0
216
+ assert main(["evaluate", path]) == 0
217
+ metrics = json.loads((config.path / "validation_metrics.json").read_text())
218
+ assert metrics["checkpoint"] == "TabICL (tabicl-classifier-v2-20260212.ckpt)"
219
+ assert math.isfinite(metrics["metrics"]["auc"])
220
+
221
+
222
+ #: faiss and torch crash together on macOS unless OpenMP has one thread.
223
+ RETRIEVAL = sys.platform != "darwin" or os.environ.get("OMP_NUM_THREADS") == "1"
224
+
225
+
226
+ @pytest.mark.parametrize(
227
+ "context_size",
228
+ [
229
+ None,
230
+ pytest.param(
231
+ 50,
232
+ marks=pytest.mark.skipif(
233
+ not RETRIEVAL, reason="retrieval on macOS needs OMP_NUM_THREADS=1"
234
+ ),
235
+ ),
236
+ ],
237
+ )
238
+ def test_tabdpt_classifies_gamgam_in_context(context_size, tmp_path):
239
+ pytest.importorskip("tabdpt")
240
+ config = fixture_config(
241
+ "gamgam_tabdpt",
242
+ GAMGAM_FIXTURES,
243
+ list(range(5)),
244
+ tmp_path,
245
+ "data.train_events=100",
246
+ "model.tabdpt_estimators=1",
247
+ f"model.tabdpt_context_size={context_size or 'null'}",
248
+ "split=validation",
249
+ )
250
+ path = str(save_config(config, tmp_path / "tabdpt.yaml"))
251
+ assert main(["prepare", path]) == 0
252
+ assert main(["evaluate", path]) == 0
253
+ metrics = json.loads((config.path / "validation_metrics.json").read_text())
254
+ assert (
255
+ metrics["checkpoint"] == f"TabDPT (context {context_size or 'all'} per query)"
256
+ )
257
+ assert math.isfinite(metrics["metrics"]["auc"])
258
+
259
+
260
+ def test_meta_train_tabicl_with_lora_then_classify_gamgam(tmp_path):
261
+ pytest.importorskip("tabicl")
262
+ meta = fixture_config(
263
+ "delphes_tabicl",
264
+ [TTH_FIXTURE, TTH_FIXTURE],
265
+ [0, 1],
266
+ tmp_path / "meta",
267
+ "train.max_epochs=1",
268
+ "task.context_events=[4,16]",
269
+ "task.query_events=4",
270
+ "task.steps_per_epoch=2",
271
+ "task.validation_steps=1",
272
+ "model.pair_objects=4",
273
+ )
274
+ path = str(save_config(meta, tmp_path / "meta.yaml"))
275
+ assert main(["prepare", path]) == 0
276
+ assert main(["train", path]) == 0
277
+ # Only the LoRA updates are saved: 12 blocks of 4 weights.
278
+ state = load_checkpoint(meta.path / "best.pt")["model_state_dict"]
279
+ assert len(state) == 12 * 4 * 2
280
+ assert all(key.endswith((".down", ".up")) for key in state)
281
+
282
+ gamgam = fixture_config(
283
+ "gamgam_tabicl",
284
+ GAMGAM_FIXTURES,
285
+ list(range(5)),
286
+ tmp_path / "gamgam",
287
+ f"checkpoint={meta.path}/best.pt",
288
+ "data.train_events=40",
289
+ "split=validation",
290
+ "model.pair_objects=4",
291
+ )
292
+ path = str(save_config(gamgam, tmp_path / "gamgam.yaml"))
293
+ assert main(["prepare", path]) == 0
294
+ assert main(["evaluate", path]) == 0
295
+ metrics = json.loads((gamgam.path / "validation_metrics.json").read_text())
296
+ assert metrics["checkpoint"] == f"{meta.path}/best.pt"
297
+ assert math.isfinite(metrics["metrics"]["auc"])
298
+
299
+
300
+ #: The encoder of the hybrid tests, as pretrained by JEPA on the fixtures.
301
+ ENCODER = {
302
+ "hid_size": 16,
303
+ "n_layers": 1,
304
+ "n_heads": 2,
305
+ "ffn_size": 64,
306
+ "pair_bias": True,
307
+ }
308
+
309
+
310
+ def test_event_inputs_are_rows_embeddings_or_both():
311
+ events = pack_events([_objects([0, 0, 3, 3, 4]), _objects([1, 3, 4])], [0, 1])
312
+ batch = collate_sequence_batch(events, torch.tensor([0, 1]))
313
+ width = RowLayout((6, 2, 2, 3, 1)).width
314
+ for inputs, expected in (("rows", width), ("embeddings", 16), ("both", width + 16)):
315
+ config = ModelConfig(name="tabicl", tabular_inputs=inputs, **ENCODER)
316
+ model = EventInputs.of(config, num_features=7, num_node_types=5)
317
+ assert model(batch).shape == (2, expected)
318
+ with pytest.raises(ValueError, match="tabular_inputs"):
319
+ EventInputs.of(
320
+ ModelConfig(tabular_inputs="objects"), num_features=7, num_node_types=5
321
+ )
322
+
323
+
324
+ @pytest.fixture
325
+ def jepa_encoder(tmp_path):
326
+ small = [f"model.{key}={value}" for key, value in ENCODER.items()]
327
+ config = fixture_config(
328
+ "delphes_jepa",
329
+ [TTH_FIXTURE, TTH_FIXTURE],
330
+ [0, 1],
331
+ tmp_path / "jepa",
332
+ "train.max_epochs=1",
333
+ *small,
334
+ )
335
+ path = str(save_config(config, tmp_path / "jepa.yaml"))
336
+ assert main(["prepare", path]) == 0
337
+ assert main(["train", path]) == 0
338
+ return f"{config.path}/last.pt"
339
+
340
+
341
+ def _hybrid(name, tmp_path, jepa_encoder, *overrides):
342
+ small = [f"model.{key}={value}" for key, value in ENCODER.items()]
343
+ config = fixture_config(
344
+ name,
345
+ GAMGAM_FIXTURES if name.startswith("gamgam") else [TTH_FIXTURE, TTH_FIXTURE],
346
+ list(range(5)) if name.startswith("gamgam") else [0, 1],
347
+ tmp_path / name,
348
+ "model.tabular_inputs=both",
349
+ f"model.pretrained={jepa_encoder}",
350
+ *small,
351
+ *overrides,
352
+ )
353
+ path = str(save_config(config, tmp_path / f"{name}.yaml"))
354
+ assert main(["prepare", path]) == 0
355
+ return config, path
356
+
357
+
358
+ def test_tabicl_classifies_gamgam_on_rows_and_jepa_embeddings(tmp_path, jepa_encoder):
359
+ pytest.importorskip("tabicl")
360
+ config, path = _hybrid(
361
+ "gamgam_tabicl",
362
+ tmp_path,
363
+ jepa_encoder,
364
+ "data.train_events=100",
365
+ "model.tabicl_estimators=1",
366
+ "split=validation",
367
+ )
368
+ assert main(["evaluate", path]) == 0
369
+ metrics = json.loads((config.path / "validation_metrics.json").read_text())
370
+ assert math.isfinite(metrics["metrics"]["auc"])
371
+
372
+
373
+ @pytest.mark.parametrize("freeze", [False, True])
374
+ def test_meta_train_tabicl_with_its_encoder(tmp_path, jepa_encoder, freeze):
375
+ pytest.importorskip("tabicl")
376
+ meta, path = _hybrid(
377
+ "delphes_tabicl",
378
+ tmp_path,
379
+ jepa_encoder,
380
+ "train.max_epochs=1",
381
+ "task.context_events=[4,16]",
382
+ "task.query_events=4",
383
+ "task.steps_per_epoch=2",
384
+ "task.validation_steps=1",
385
+ f"model.freeze_backbone={str(freeze).lower()}",
386
+ )
387
+ assert main(["train", path]) == 0
388
+ state = load_checkpoint(meta.path / "best.pt")["model_state_dict"]
389
+ encoder = [key for key in state if key.startswith("inputs.encoder.")]
390
+ # A trained encoder is saved; a frozen one comes from model.pretrained.
391
+ assert bool(encoder) != freeze
392
+ gamgam, path = _hybrid(
393
+ "gamgam_tabicl",
394
+ tmp_path,
395
+ jepa_encoder,
396
+ f"checkpoint={meta.path}/best.pt",
397
+ "data.train_events=40",
398
+ "split=validation",
399
+ f"model.freeze_backbone={str(freeze).lower()}",
400
+ )
401
+ assert main(["evaluate", path]) == 0
402
+ metrics = json.loads((gamgam.path / "validation_metrics.json").read_text())
403
+ assert math.isfinite(metrics["metrics"]["auc"])
tests/test_workflows.py CHANGED
@@ -53,6 +53,7 @@ def test_gamgam_train_evaluate_predict_export_and_resume(gamgam, tmp_path):
53
  events = load_events(gamgam.data.cache)
54
  assert summary["num_samples"] == int((events.fold == 9).sum())
55
  assert all(math.isfinite(value) for value in summary["metrics"].values())
 
56
 
57
  assert main(["predict", path, "split=validation"]) == 0
58
  predictions = np.load(run / "validation_predictions.npz")
@@ -167,7 +168,9 @@ def _rank(rank: int, port: int, argv: list[str], queue) -> None:
167
 
168
 
169
  def test_training_runs_on_two_ddp_ranks(gamgam, tmp_path):
170
- argv = ["train", _config_file(gamgam, tmp_path), "train.max_epochs=2"]
 
 
171
  context = mp.get_context("spawn")
172
  queue = context.Queue()
173
  port = _port()
 
53
  events = load_events(gamgam.data.cache)
54
  assert summary["num_samples"] == int((events.fold == 9).sum())
55
  assert all(math.isfinite(value) for value in summary["metrics"].values())
56
+ assert set(summary["seconds"]) == {"setup", "inference"}
57
 
58
  assert main(["predict", path, "split=validation"]) == 0
59
  predictions = np.load(run / "validation_predictions.npz")
 
168
 
169
 
170
  def test_training_runs_on_two_ddp_ranks(gamgam, tmp_path):
171
+ config = _config_file(gamgam, tmp_path)
172
+ # Gloo ranks on the CPU: MPS has no collectives.
173
+ argv = ["train", config, "train.max_epochs=2", "train.device=cpu"]
174
  context = mp.get_context("spawn")
175
  queue = context.Queue()
176
  port = _port()
uv.lock CHANGED
@@ -1,5 +1,5 @@
1
  version = 1
2
- revision = 3
3
  requires-python = "==3.14.*"
4
  resolution-markers = [
5
  "platform_machine == 'x86_64' and sys_platform == 'linux'",
@@ -228,7 +228,7 @@ name = "cuda-bindings"
228
  version = "13.4.4"
229
  source = { registry = "https://pypi.org/simple" }
230
  dependencies = [
231
- { name = "cuda-pathfinder" },
232
  ]
233
  wheels = [
234
  { url = "https://files.pythonhosted.org/packages/e6/f0/d71d1dd45caa358a39c6898dcc9f1bd22a9f22b92376b967835c047b50bc/cuda_bindings-13.4.4-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4a429132f06d0c3a853aa419aabe4c0a7bfc3f3886fb4c336d81729f63754d31", size = 6407946, upload-time = "2026-10-14T12:35:39.826Z" },
@@ -255,43 +255,43 @@ wheels = [
255
 
256
  [package.optional-dependencies]
257
  cublas = [
258
- { name = "nvidia-cublas", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
259
- { name = "nvidia-cuda-nvrtc", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
260
  ]
261
  cudart = [
262
- { name = "nvidia-cuda-runtime", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
263
  ]
264
  cufft = [
265
- { name = "nvidia-cufft", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
266
- { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
267
  ]
268
  cufile = [
269
  { name = "nvidia-cufile", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
270
  ]
271
  cupti = [
272
- { name = "nvidia-cuda-cupti", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
273
  ]
274
  curand = [
275
- { name = "nvidia-curand", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
276
  ]
277
  cusolver = [
278
- { name = "nvidia-cublas", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
279
- { name = "nvidia-cusolver", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
280
- { name = "nvidia-cusparse", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
281
- { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
282
  ]
283
  cusparse = [
284
- { name = "nvidia-cusparse", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
285
- { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
286
  ]
287
  nvjitlink = [
288
- { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
289
  ]
290
  nvrtc = [
291
- { name = "nvidia-cuda-nvrtc", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
292
  ]
293
  nvtx = [
294
- { name = "nvidia-nvtx", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" },
295
  ]
296
 
297
  [[package]]
@@ -312,6 +312,34 @@ wheels = [
312
  { url = "https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl", hash = "sha256:54058201ac7087911181bfec4af6091bb59380360f069276601256a76af08193", size = 65638, upload-time = "2026-01-26T04:13:18.546Z" },
313
  ]
314
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
315
  [[package]]
316
  name = "filelock"
317
  version = "3.32.3"
@@ -381,6 +409,12 @@ onnx = [
381
  { name = "onnx" },
382
  { name = "onnxruntime" },
383
  ]
 
 
 
 
 
 
384
  tabpfn = [
385
  { name = "tabpfn" },
386
  ]
@@ -401,12 +435,14 @@ requires-dist = [
401
  { name = "onnx", marker = "extra == 'onnx'", specifier = ">=1.16,<2" },
402
  { name = "onnxruntime", marker = "extra == 'onnx'", specifier = ">=1.18,<2" },
403
  { name = "pyyaml", specifier = ">=6" },
 
 
404
  { name = "tabpfn", marker = "extra == 'tabpfn'", specifier = ">=9.2,<10" },
405
  { name = "torch", marker = "platform_machine != 'x86_64' or sys_platform != 'linux'", specifier = "==2.14.1" },
406
  { name = "torch", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'", specifier = "==2.14.1", index = "https://download.pytorch.org/whl/cu130" },
407
  { name = "uproot", specifier = ">=5.0" },
408
  ]
409
- provides-extras = ["onnx", "tabpfn"]
410
 
411
  [package.metadata.requires-dev]
412
  dev = [
@@ -731,7 +767,7 @@ name = "nvidia-cublas"
731
  version = "13.1.1.3"
732
  source = { registry = "https://pypi.org/simple" }
733
  dependencies = [
734
- { name = "nvidia-cuda-nvrtc" },
735
  ]
736
  wheels = [
737
  { url = "https://files.pythonhosted.org/packages/a7/a1/0bd24ee8c8d03adac032fd2909426a00c88f8c57961b1277ded97f91119f/nvidia_cublas-13.1.1.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:b7a210458267ac818974c53038fbec2e969d5c99f305ab15c72522fa9f001dd5", size = 542848918, upload-time = "2026-04-08T18:46:22.985Z" },
@@ -770,7 +806,7 @@ name = "nvidia-cudnn-cu13"
770
  version = "9.24.0.43"
771
  source = { registry = "https://pypi.org/simple" }
772
  dependencies = [
773
- { name = "nvidia-cublas" },
774
  ]
775
  wheels = [
776
  { url = "https://files.pythonhosted.org/packages/ca/30/7c257e3d5cb4fecb147b93895c66e29c93f8e76d74b45bb418ff0587c4ec/nvidia_cudnn_cu13-9.24.0.43-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:a6812a554a1ff0413e9c52b84c26c050380649ab9615f9c16bded368ce9f421f", size = 650976863, upload-time = "2026-07-02T16:23:39.248Z" },
@@ -782,7 +818,7 @@ name = "nvidia-cufft"
782
  version = "12.0.0.61"
783
  source = { registry = "https://pypi.org/simple" }
784
  dependencies = [
785
- { name = "nvidia-nvjitlink" },
786
  ]
787
  wheels = [
788
  { url = "https://files.pythonhosted.org/packages/8b/ae/f417a75c0259e85c1d2f83ca4e960289a5f814ed0cea74d18c353d3e989d/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2708c852ef8cd89d1d2068bdbece0aa188813a0c934db3779b9b1faa8442e5f5", size = 214053554, upload-time = "2025-09-04T08:31:38.196Z" },
@@ -812,9 +848,9 @@ name = "nvidia-cusolver"
812
  version = "12.0.4.66"
813
  source = { registry = "https://pypi.org/simple" }
814
  dependencies = [
815
- { name = "nvidia-cublas" },
816
- { name = "nvidia-cusparse" },
817
- { name = "nvidia-nvjitlink" },
818
  ]
819
  wheels = [
820
  { url = "https://files.pythonhosted.org/packages/c8/c3/b30c9e935fc01e3da443ec0116ed1b2a009bb867f5324d3f2d7e533e776b/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:02c2457eaa9e39de20f880f4bd8820e6a1cfb9f9a34f820eb12a155aa5bc92d2", size = 223467760, upload-time = "2025-09-04T08:33:04.222Z" },
@@ -826,7 +862,7 @@ name = "nvidia-cusparse"
826
  version = "12.6.3.3"
827
  source = { registry = "https://pypi.org/simple" }
828
  dependencies = [
829
- { name = "nvidia-nvjitlink" },
830
  ]
831
  wheels = [
832
  { url = "https://files.pythonhosted.org/packages/f8/94/5c26f33738ae35276672f12615a64bd008ed5be6d1ebcb23579285d960a9/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:80bcc4662f23f1054ee334a15c72b8940402975e0eab63178fc7e670aa59472c", size = 162155568, upload-time = "2025-09-04T08:33:42.864Z" },
@@ -878,6 +914,18 @@ wheels = [
878
  { url = "https://files.pythonhosted.org/packages/a8/64/3708a90d1ebe202ffdeb7185f878a3c84d15c2b2c31858da2ce0583e2def/nvidia_nvtx-13.0.85-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb7780edb6b14107373c835bf8b72e7a178bac7367e23da7acb108f973f157a6", size = 148878, upload-time = "2025-09-04T08:28:53.627Z" },
879
  ]
880
 
 
 
 
 
 
 
 
 
 
 
 
 
881
  [[package]]
882
  name = "onnx"
883
  version = "1.22.0"
@@ -1015,6 +1063,28 @@ wheels = [
1015
  { url = "https://files.pythonhosted.org/packages/19/c7/5f7c636ec43e0c545e28d1f1db71990108306f7bdcb89f069ba97e428e7f/protobuf-7.35.1-py3-none-any.whl", hash = "sha256:4bc97768d8fe4ad6743c8a19403e314511ed9f6d13205b687e52421c023ac1b9", size = 171659, upload-time = "2026-06-11T21:55:39.155Z" },
1016
  ]
1017
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1018
  [[package]]
1019
  name = "pydantic"
1020
  version = "2.14.1"
@@ -1353,6 +1423,47 @@ wheels = [
1353
  { url = "https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", size = 6299353, upload-time = "2025-04-27T18:04:59.103Z" },
1354
  ]
1355
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1356
  [[package]]
1357
  name = "tabpfn"
1358
  version = "9.2.0"
 
1
  version = 1
2
+ revision = 2
3
  requires-python = "==3.14.*"
4
  resolution-markers = [
5
  "platform_machine == 'x86_64' and sys_platform == 'linux'",
 
228
  version = "13.4.4"
229
  source = { registry = "https://pypi.org/simple" }
230
  dependencies = [
231
+ { name = "cuda-pathfinder", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
232
  ]
233
  wheels = [
234
  { url = "https://files.pythonhosted.org/packages/e6/f0/d71d1dd45caa358a39c6898dcc9f1bd22a9f22b92376b967835c047b50bc/cuda_bindings-13.4.4-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4a429132f06d0c3a853aa419aabe4c0a7bfc3f3886fb4c336d81729f63754d31", size = 6407946, upload-time = "2026-10-14T12:35:39.826Z" },
 
255
 
256
  [package.optional-dependencies]
257
  cublas = [
258
+ { name = "nvidia-cublas", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
259
+ { name = "nvidia-cuda-nvrtc", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
260
  ]
261
  cudart = [
262
+ { name = "nvidia-cuda-runtime", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
263
  ]
264
  cufft = [
265
+ { name = "nvidia-cufft", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
266
+ { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
267
  ]
268
  cufile = [
269
  { name = "nvidia-cufile", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
270
  ]
271
  cupti = [
272
+ { name = "nvidia-cuda-cupti", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
273
  ]
274
  curand = [
275
+ { name = "nvidia-curand", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
276
  ]
277
  cusolver = [
278
+ { name = "nvidia-cublas", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
279
+ { name = "nvidia-cusolver", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
280
+ { name = "nvidia-cusparse", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
281
+ { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
282
  ]
283
  cusparse = [
284
+ { name = "nvidia-cusparse", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
285
+ { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
286
  ]
287
  nvjitlink = [
288
+ { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
289
  ]
290
  nvrtc = [
291
+ { name = "nvidia-cuda-nvrtc", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
292
  ]
293
  nvtx = [
294
+ { name = "nvidia-nvtx", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" },
295
  ]
296
 
297
  [[package]]
 
312
  { url = "https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl", hash = "sha256:54058201ac7087911181bfec4af6091bb59380360f069276601256a76af08193", size = 65638, upload-time = "2026-01-26T04:13:18.546Z" },
313
  ]
314
 
315
+ [[package]]
316
+ name = "faiss-cpu"
317
+ version = "1.12.0"
318
+ source = { registry = "https://pypi.org/simple" }
319
+ dependencies = [
320
+ { name = "numpy" },
321
+ { name = "packaging" },
322
+ ]
323
+ sdist = { url = "https://files.pythonhosted.org/packages/7d/80/bb75a7ed6e824dea452a24d3434a72ed799324a688b10b047d441d270185/faiss_cpu-1.12.0.tar.gz", hash = "sha256:2f87cbcd603f3ed464ebceb857971fdebc318de938566c9ae2b82beda8e953c0", size = 69292, upload-time = "2025-08-13T06:07:26.553Z" }
324
+ wheels = [
325
+ { url = "https://files.pythonhosted.org/packages/12/35/01a4a7c179d67bee0d8a027b95c3eae19cb354ae69ef2bc50ac3b93bc853/faiss_cpu-1.12.0-cp314-cp314-macosx_13_0_x86_64.whl", hash = "sha256:ff7db774968210d08cd0331287f3f66a6ffef955a7aa9a7fcd3eb4432a4ce5f5", size = 8036142, upload-time = "2025-08-13T06:06:38.894Z" },
326
+ { url = "https://files.pythonhosted.org/packages/08/23/bac2859490096608c9d527f3041b44c2e43f8df0d4aadd53a4cc5ce678ac/faiss_cpu-1.12.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:220b5bb5439c64e417b35f9ade4c7dc3bf7df683d6123901ba84d6d764ecd486", size = 3363747, upload-time = "2025-08-13T06:06:40.73Z" },
327
+ { url = "https://files.pythonhosted.org/packages/7b/1d/e18023e1f43a18ec593adcd69d356f1fa94bde20344e38334d5985e5c5cc/faiss_cpu-1.12.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:693d0bf16f79e8d16a1baaeda459f3375f37da0354e97dc032806b48a2a54151", size = 3835232, upload-time = "2025-08-13T06:06:42.172Z" },
328
+ { url = "https://files.pythonhosted.org/packages/cd/2b/1c1fea423d3f550f44c5ec3f14d8400919b49c285c3bd146687c63e40186/faiss_cpu-1.12.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bcc6587dee21e17430fb49ddc5200625d6f5e1de2bdf436f14827bad4ca78d19", size = 31432677, upload-time = "2025-08-13T06:06:44.348Z" },
329
+ { url = "https://files.pythonhosted.org/packages/de/d2/3483e92a02f30e2d8491a256f470f54b7f5483266dfe09126d28741d31ec/faiss_cpu-1.12.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:b80e5965f001822cc99ec65c715169af1b70bdae72eccd573520a2dec485b3ee", size = 9765504, upload-time = "2025-08-13T06:06:46.567Z" },
330
+ { url = "https://files.pythonhosted.org/packages/ce/2f/d97792211a9bd84b8d6b1dcaa1dcd69ac11e026c6ef19c641b6a87e31025/faiss_cpu-1.12.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:98279f1b4876ef9902695a329b81a99002782ab6e26def472022009df6f1ac68", size = 24169930, upload-time = "2025-08-13T06:06:48.916Z" },
331
+ { url = "https://files.pythonhosted.org/packages/ee/b8/b707ca4d88af472509a053c39d3cced53efd19d096b8dff2fadc18c4b82d/faiss_cpu-1.12.0-cp314-cp314-win_amd64.whl", hash = "sha256:11670337f9f5ee9ff3490e30683eea80add060c300cf6f6cb0e8faf3155fd20e", size = 18475400, upload-time = "2025-08-13T06:06:51.233Z" },
332
+ { url = "https://files.pythonhosted.org/packages/77/11/42e41ddebde4dfe77e36e92d0110b4f733c8640883abffde54f802482deb/faiss_cpu-1.12.0-cp314-cp314-win_arm64.whl", hash = "sha256:7ac1c8b53609b5c722ab60f1749260a7cb3c72fdfb720a0e3033067e73591da5", size = 8281229, upload-time = "2025-08-13T06:06:53.735Z" },
333
+ { url = "https://files.pythonhosted.org/packages/1c/9a/8ae5bbeabe70eb673c37fc7c77e2e476746331afb6654b2df97d8b6d380d/faiss_cpu-1.12.0-cp314-cp314t-macosx_13_0_x86_64.whl", hash = "sha256:110b21b7bb4c93c4f1a5eb2ffb8ef99dcdb4725f8ab2e5cd161324e4d981f204", size = 8087247, upload-time = "2025-08-13T06:06:55.407Z" },
334
+ { url = "https://files.pythonhosted.org/packages/f4/df/b3d79098860b67b126da351788c04ac243c29718dadc4a678a6f5e7209c0/faiss_cpu-1.12.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:82eb5515ce72be9a43f4cf74447a0d090e014231981df91aff7251204b506fbf", size = 3411043, upload-time = "2025-08-13T06:06:56.983Z" },
335
+ { url = "https://files.pythonhosted.org/packages/bc/2f/b1a2a03dd3cce22ff9fc434aa3c7390125087260c1d1349311da36eaa432/faiss_cpu-1.12.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:754eef89cdf2b35643df6b0923a5a098bdfecf63b5f4bd86c385042ee511b287", size = 3801789, upload-time = "2025-08-13T06:06:58.688Z" },
336
+ { url = "https://files.pythonhosted.org/packages/a3/a8/16ad0c6a966e93d04bfd5248d2be1d8b5849842b0e2611c5ecd26fcaf036/faiss_cpu-1.12.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7285c71c8f5e9c58b55175f5f74c78c518c52c421a88a430263f34e3e31f719c", size = 31231388, upload-time = "2025-08-13T06:07:00.55Z" },
337
+ { url = "https://files.pythonhosted.org/packages/62/a1/9c16eca0b8f8b13c32c47a5e4ff7a4bc0ca3e7d263140312088811230871/faiss_cpu-1.12.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:84a50d7a2f711f79cc8b65aa28956dba6435e47b71a38b2daea44c94c9b8e458", size = 9737605, upload-time = "2025-08-13T06:07:03.018Z" },
338
+ { url = "https://files.pythonhosted.org/packages/a8/4a/2c2d615078c9d816a836fb893aaef551ad152f2eb00bc258698273c240c0/faiss_cpu-1.12.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:7f3e0a14e4edec6a3959a9f51afccb89e863138f184ff2cc24c13f9ad788740b", size = 23922880, upload-time = "2025-08-13T06:07:05.099Z" },
339
+ { url = "https://files.pythonhosted.org/packages/30/aa/99b8402a4dac678794f13f8f4f29d666c2ef0a91594418147f47034ebc81/faiss_cpu-1.12.0-cp314-cp314t-win_amd64.whl", hash = "sha256:8b3239cc371df6826ac43c62ac04eec7cc497bedb43f681fcd8ea494f520ddbb", size = 18750661, upload-time = "2025-08-13T06:07:07.551Z" },
340
+ { url = "https://files.pythonhosted.org/packages/a3/a2/b546e9a20ba157eb2fbe141289f1752f157ee6d932899f4853df4ded6d4b/faiss_cpu-1.12.0-cp314-cp314t-win_arm64.whl", hash = "sha256:58b23456db725ee1bd605a6135d2ef55b2ac3e0b6fe873fd99a909e8ef4bd0ff", size = 8302032, upload-time = "2025-08-13T06:07:09.602Z" },
341
+ ]
342
+
343
  [[package]]
344
  name = "filelock"
345
  version = "3.32.3"
 
409
  { name = "onnx" },
410
  { name = "onnxruntime" },
411
  ]
412
+ tabdpt = [
413
+ { name = "tabdpt" },
414
+ ]
415
+ tabicl = [
416
+ { name = "tabicl" },
417
+ ]
418
  tabpfn = [
419
  { name = "tabpfn" },
420
  ]
 
435
  { name = "onnx", marker = "extra == 'onnx'", specifier = ">=1.16,<2" },
436
  { name = "onnxruntime", marker = "extra == 'onnx'", specifier = ">=1.18,<2" },
437
  { name = "pyyaml", specifier = ">=6" },
438
+ { name = "tabdpt", marker = "extra == 'tabdpt'", specifier = ">=1.3.1,<2" },
439
+ { name = "tabicl", marker = "extra == 'tabicl'", specifier = ">=2.2,<3" },
440
  { name = "tabpfn", marker = "extra == 'tabpfn'", specifier = ">=9.2,<10" },
441
  { name = "torch", marker = "platform_machine != 'x86_64' or sys_platform != 'linux'", specifier = "==2.14.1" },
442
  { name = "torch", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'", specifier = "==2.14.1", index = "https://download.pytorch.org/whl/cu130" },
443
  { name = "uproot", specifier = ">=5.0" },
444
  ]
445
+ provides-extras = ["onnx", "tabpfn", "tabicl", "tabdpt"]
446
 
447
  [package.metadata.requires-dev]
448
  dev = [
 
767
  version = "13.1.1.3"
768
  source = { registry = "https://pypi.org/simple" }
769
  dependencies = [
770
+ { name = "nvidia-cuda-nvrtc", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
771
  ]
772
  wheels = [
773
  { url = "https://files.pythonhosted.org/packages/a7/a1/0bd24ee8c8d03adac032fd2909426a00c88f8c57961b1277ded97f91119f/nvidia_cublas-13.1.1.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:b7a210458267ac818974c53038fbec2e969d5c99f305ab15c72522fa9f001dd5", size = 542848918, upload-time = "2026-04-08T18:46:22.985Z" },
 
806
  version = "9.24.0.43"
807
  source = { registry = "https://pypi.org/simple" }
808
  dependencies = [
809
+ { name = "nvidia-cublas", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
810
  ]
811
  wheels = [
812
  { url = "https://files.pythonhosted.org/packages/ca/30/7c257e3d5cb4fecb147b93895c66e29c93f8e76d74b45bb418ff0587c4ec/nvidia_cudnn_cu13-9.24.0.43-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:a6812a554a1ff0413e9c52b84c26c050380649ab9615f9c16bded368ce9f421f", size = 650976863, upload-time = "2026-07-02T16:23:39.248Z" },
 
818
  version = "12.0.0.61"
819
  source = { registry = "https://pypi.org/simple" }
820
  dependencies = [
821
+ { name = "nvidia-nvjitlink", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
822
  ]
823
  wheels = [
824
  { url = "https://files.pythonhosted.org/packages/8b/ae/f417a75c0259e85c1d2f83ca4e960289a5f814ed0cea74d18c353d3e989d/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2708c852ef8cd89d1d2068bdbece0aa188813a0c934db3779b9b1faa8442e5f5", size = 214053554, upload-time = "2025-09-04T08:31:38.196Z" },
 
848
  version = "12.0.4.66"
849
  source = { registry = "https://pypi.org/simple" }
850
  dependencies = [
851
+ { name = "nvidia-cublas", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
852
+ { name = "nvidia-cusparse", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
853
+ { name = "nvidia-nvjitlink", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
854
  ]
855
  wheels = [
856
  { url = "https://files.pythonhosted.org/packages/c8/c3/b30c9e935fc01e3da443ec0116ed1b2a009bb867f5324d3f2d7e533e776b/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:02c2457eaa9e39de20f880f4bd8820e6a1cfb9f9a34f820eb12a155aa5bc92d2", size = 223467760, upload-time = "2025-09-04T08:33:04.222Z" },
 
862
  version = "12.6.3.3"
863
  source = { registry = "https://pypi.org/simple" }
864
  dependencies = [
865
+ { name = "nvidia-nvjitlink", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
866
  ]
867
  wheels = [
868
  { url = "https://files.pythonhosted.org/packages/f8/94/5c26f33738ae35276672f12615a64bd008ed5be6d1ebcb23579285d960a9/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:80bcc4662f23f1054ee334a15c72b8940402975e0eab63178fc7e670aa59472c", size = 162155568, upload-time = "2025-09-04T08:33:42.864Z" },
 
914
  { url = "https://files.pythonhosted.org/packages/a8/64/3708a90d1ebe202ffdeb7185f878a3c84d15c2b2c31858da2ce0583e2def/nvidia_nvtx-13.0.85-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb7780edb6b14107373c835bf8b72e7a178bac7367e23da7acb108f973f157a6", size = 148878, upload-time = "2025-09-04T08:28:53.627Z" },
915
  ]
916
 
917
+ [[package]]
918
+ name = "omegaconf"
919
+ version = "2.4.0"
920
+ source = { registry = "https://pypi.org/simple" }
921
+ dependencies = [
922
+ { name = "pyyaml" },
923
+ ]
924
+ sdist = { url = "https://files.pythonhosted.org/packages/cc/10/1cabf3188a4d9684a06568c7fd76f9536eede625733c8399cf1f5337b83e/omegaconf-2.4.0.tar.gz", hash = "sha256:7e219a70bea371da250285b9c01c7db027bc421722799c2cc4ff14bc4a2a753b", size = 3544508, upload-time = "2026-10-09T12:44:02.326Z" }
925
+ wheels = [
926
+ { url = "https://files.pythonhosted.org/packages/4f/bd/0d9f8247b0819385213d09564eda5c52d3673313c2414d51f72883ec24cd/omegaconf-2.4.0-py3-none-any.whl", hash = "sha256:f141a664c777582405e3cc6e0545ecc79429e79f844995b1a6e71c35946a025c", size = 254152, upload-time = "2026-10-09T12:43:59.764Z" },
927
+ ]
928
+
929
  [[package]]
930
  name = "onnx"
931
  version = "1.22.0"
 
1063
  { url = "https://files.pythonhosted.org/packages/19/c7/5f7c636ec43e0c545e28d1f1db71990108306f7bdcb89f069ba97e428e7f/protobuf-7.35.1-py3-none-any.whl", hash = "sha256:4bc97768d8fe4ad6743c8a19403e314511ed9f6d13205b687e52421c023ac1b9", size = 171659, upload-time = "2026-06-11T21:55:39.155Z" },
1064
  ]
1065
 
1066
+ [[package]]
1067
+ name = "psutil"
1068
+ version = "7.2.2"
1069
+ source = { registry = "https://pypi.org/simple" }
1070
+ sdist = { url = "https://files.pythonhosted.org/packages/aa/c6/d1ddf4abb55e93cebc4f2ed8b5d6dbad109ecb8d63748dd2b20ab5e57ebe/psutil-7.2.2.tar.gz", hash = "sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372", size = 493740, upload-time = "2026-01-28T18:14:54.428Z" }
1071
+ wheels = [
1072
+ { url = "https://files.pythonhosted.org/packages/81/69/ef179ab5ca24f32acc1dac0c247fd6a13b501fd5534dbae0e05a1c48b66d/psutil-7.2.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:eed63d3b4d62449571547b60578c5b2c4bcccc5387148db46e0c2313dad0ee00", size = 130664, upload-time = "2026-01-28T18:15:09.469Z" },
1073
+ { url = "https://files.pythonhosted.org/packages/7b/64/665248b557a236d3fa9efc378d60d95ef56dd0a490c2cd37dafc7660d4a9/psutil-7.2.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7b6d09433a10592ce39b13d7be5a54fbac1d1228ed29abc880fb23df7cb694c9", size = 131087, upload-time = "2026-01-28T18:15:11.724Z" },
1074
+ { url = "https://files.pythonhosted.org/packages/d5/2e/e6782744700d6759ebce3043dcfa661fb61e2fb752b91cdeae9af12c2178/psutil-7.2.2-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fa4ecf83bcdf6e6c8f4449aff98eefb5d0604bf88cb883d7da3d8d2d909546a", size = 182383, upload-time = "2026-01-28T18:15:13.445Z" },
1075
+ { url = "https://files.pythonhosted.org/packages/57/49/0a41cefd10cb7505cdc04dab3eacf24c0c2cb158a998b8c7b1d27ee2c1f5/psutil-7.2.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e452c464a02e7dc7822a05d25db4cde564444a67e58539a00f929c51eddda0cf", size = 185210, upload-time = "2026-01-28T18:15:16.002Z" },
1076
+ { url = "https://files.pythonhosted.org/packages/dd/2c/ff9bfb544f283ba5f83ba725a3c5fec6d6b10b8f27ac1dc641c473dc390d/psutil-7.2.2-cp314-cp314t-win_amd64.whl", hash = "sha256:c7663d4e37f13e884d13994247449e9f8f574bc4655d509c3b95e9ec9e2b9dc1", size = 141228, upload-time = "2026-01-28T18:15:18.385Z" },
1077
+ { url = "https://files.pythonhosted.org/packages/f2/fc/f8d9c31db14fcec13748d373e668bc3bed94d9077dbc17fb0eebc073233c/psutil-7.2.2-cp314-cp314t-win_arm64.whl", hash = "sha256:11fe5a4f613759764e79c65cf11ebdf26e33d6dd34336f8a337aa2996d71c841", size = 136284, upload-time = "2026-01-28T18:15:19.912Z" },
1078
+ { url = "https://files.pythonhosted.org/packages/e7/36/5ee6e05c9bd427237b11b3937ad82bb8ad2752d72c6969314590dd0c2f6e/psutil-7.2.2-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486", size = 129090, upload-time = "2026-01-28T18:15:22.168Z" },
1079
+ { url = "https://files.pythonhosted.org/packages/80/c4/f5af4c1ca8c1eeb2e92ccca14ce8effdeec651d5ab6053c589b074eda6e1/psutil-7.2.2-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979", size = 129859, upload-time = "2026-01-28T18:15:23.795Z" },
1080
+ { url = "https://files.pythonhosted.org/packages/b5/70/5d8df3b09e25bce090399cf48e452d25c935ab72dad19406c77f4e828045/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9", size = 155560, upload-time = "2026-01-28T18:15:25.976Z" },
1081
+ { url = "https://files.pythonhosted.org/packages/63/65/37648c0c158dc222aba51c089eb3bdfa238e621674dc42d48706e639204f/psutil-7.2.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e", size = 156997, upload-time = "2026-01-28T18:15:27.794Z" },
1082
+ { url = "https://files.pythonhosted.org/packages/8e/13/125093eadae863ce03c6ffdbae9929430d116a246ef69866dad94da3bfbc/psutil-7.2.2-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8", size = 148972, upload-time = "2026-01-28T18:15:29.342Z" },
1083
+ { url = "https://files.pythonhosted.org/packages/04/78/0acd37ca84ce3ddffaa92ef0f571e073faa6d8ff1f0559ab1272188ea2be/psutil-7.2.2-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc", size = 148266, upload-time = "2026-01-28T18:15:31.597Z" },
1084
+ { url = "https://files.pythonhosted.org/packages/b4/90/e2159492b5426be0c1fef7acba807a03511f97c5f86b3caeda6ad92351a7/psutil-7.2.2-cp37-abi3-win_amd64.whl", hash = "sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988", size = 137737, upload-time = "2026-01-28T18:15:33.849Z" },
1085
+ { url = "https://files.pythonhosted.org/packages/8c/c7/7bb2e321574b10df20cbde462a94e2b71d05f9bbda251ef27d104668306a/psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee", size = 134617, upload-time = "2026-01-28T18:15:36.514Z" },
1086
+ ]
1087
+
1088
  [[package]]
1089
  name = "pydantic"
1090
  version = "2.14.1"
 
1423
  { url = "https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", size = 6299353, upload-time = "2025-04-27T18:04:59.103Z" },
1424
  ]
1425
 
1426
+ [[package]]
1427
+ name = "tabdpt"
1428
+ version = "1.3.1"
1429
+ source = { registry = "https://pypi.org/simple" }
1430
+ dependencies = [
1431
+ { name = "faiss-cpu" },
1432
+ { name = "huggingface-hub" },
1433
+ { name = "numpy" },
1434
+ { name = "omegaconf" },
1435
+ { name = "safetensors" },
1436
+ { name = "scikit-learn" },
1437
+ { name = "scipy" },
1438
+ { name = "torch", version = "2.14.1", source = { registry = "https://pypi.org/simple" }, marker = "platform_machine != 'x86_64' or sys_platform != 'linux'" },
1439
+ { name = "torch", version = "2.14.1+cu130", source = { registry = "https://download.pytorch.org/whl/cu130" }, marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
1440
+ { name = "tqdm" },
1441
+ ]
1442
+ sdist = { url = "https://files.pythonhosted.org/packages/68/f0/cda7df9f41705c162d075255fec8e9e53d63aa9eee0bb419a01a56fe8f53/tabdpt-1.3.1.tar.gz", hash = "sha256:42e3e5c13ededb3ff7e6ab128d6491de5537642f9bba7a66da54701bdb7b99ff", size = 33832, upload-time = "2026-09-21T18:22:02.147Z" }
1443
+ wheels = [
1444
+ { url = "https://files.pythonhosted.org/packages/f6/1c/fc711dd8ec3bcf15c3c3c9320a35cc2f798c4f3de3bf7e59aef6e7b7263d/tabdpt-1.3.1-py3-none-any.whl", hash = "sha256:b3ad31443cf20d6ee0d2d9398c61f09d3d2e7b796fb2f9346a716ddf576e933f", size = 27204, upload-time = "2026-09-21T18:22:01.192Z" },
1445
+ ]
1446
+
1447
+ [[package]]
1448
+ name = "tabicl"
1449
+ version = "2.2.0"
1450
+ source = { registry = "https://pypi.org/simple" }
1451
+ dependencies = [
1452
+ { name = "einops" },
1453
+ { name = "huggingface-hub" },
1454
+ { name = "numpy" },
1455
+ { name = "psutil" },
1456
+ { name = "scikit-learn" },
1457
+ { name = "scipy" },
1458
+ { name = "torch", version = "2.14.1", source = { registry = "https://pypi.org/simple" }, marker = "platform_machine != 'x86_64' or sys_platform != 'linux'" },
1459
+ { name = "torch", version = "2.14.1+cu130", source = { registry = "https://download.pytorch.org/whl/cu130" }, marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
1460
+ { name = "tqdm" },
1461
+ ]
1462
+ sdist = { url = "https://files.pythonhosted.org/packages/d4/81/ee11ac72b33c2b87f7337c99238fd2700128791edceb558b63bb30556b73/tabicl-2.2.0.tar.gz", hash = "sha256:dfc41f2acd9603e4402cedae9e42f9922cfb8cfc9fcf7901d773e26609e47e21", size = 268084, upload-time = "2026-09-02T15:13:38.59Z" }
1463
+ wheels = [
1464
+ { url = "https://files.pythonhosted.org/packages/77/22/bbac2df399b8ed5a699bde38496ebb129a045c5d160fe52786727de932f3/tabicl-2.2.0-py3-none-any.whl", hash = "sha256:c80b8ec2719be26bc0bf1dd758a2361f58019d8f7877038b785529a425789ce4", size = 310448, upload-time = "2026-09-02T15:13:36.629Z" },
1465
+ ]
1466
+
1467
  [[package]]
1468
  name = "tabpfn"
1469
  version = "9.2.0"