amorarc commited on
Commit
522b012
·
verified ·
1 Parent(s): e75cdcc

Func-JEPA pretrained weights

Browse files
README.md ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ library_name: pytorch
4
+ tags: [symbolic-mathematics, jepa, representation-learning, symbolic-regression, function-encoder]
5
+ ---
6
+
7
+ # Func-JEPA
8
+
9
+ Pretrained weights of *Func-JEPA: Learning a Sobolev Space of Functions as Latent Vectors* (Alejandro Moreno Arcas,
10
+ 2026). Code: https://github.com/amorarc/Func-JEPA.
11
+
12
+ Func-JEPA encodes the symbolic form of a univariate function into a vector of R^256 whose norm is the exact H¹ norm of
13
+ the function and whose dot products approximate H¹ inner products. Differentiation is a learned latent operator. Small
14
+ modules on the frozen encoder decode latents to formulas, compose and integrate in latent space, and map sampled values
15
+ to latents.
16
+
17
+ ## Usage
18
+
19
+ ```bash
20
+ pip install git+https://github.com/amorarc/Func-JEPA
21
+ ```
22
+
23
+ ```python
24
+ import numpy as np
25
+ from fjepa import FuncJEPA
26
+
27
+ m = FuncJEPA.from_pretrained("amorarc/func-jepa")
28
+ z = m.encode("4*sin(7*x) - 3*cos(7*x)")
29
+ m.norm(z) # 12.87, the H1 norm of f on [-pi, pi]
30
+ m.decode(m.derivative(z))[0] # 21*sin(7*x) + 28*cos(7*x)
31
+ m.regress(32 * np.cos(6 * m.nodes) + 8) # 32*cos(6*x) + 8, from 64 samples
32
+ ```
33
+
34
+ ## Files
35
+
36
+ | Folder | Parameters | Role |
37
+ |---|---|---|
38
+ | `encoder_pi` | 35.1M | encoder + derivative operator on `[-π, π]` (all results of the paper) |
39
+ | `encoder_05` | 35.1M | the same model trained on `[0, 5]` |
40
+ | `decoder` | 17.7M | latent → prefix tokens |
41
+ | `compose` | 52.9M | composition operators: sin, cos, log, sqrt, square, cube, quadratic |
42
+ | `integrator` | 7.6M | latent of `f` → latent of the zero-mean antiderivative |
43
+ | `value_encoder` | 7.4M | 64 samples → latent direction |
44
+ | `decoder_lc`, `value_encoder_lc` | 17.7M, 7.4M | variants fine-tuned for the Lample–Charton ODE benchmark |
45
+
46
+ Each folder holds `model.safetensors` and `config.json`; `SHA256.json` lists the hashes.
47
+
48
+ ## Training
49
+
50
+ Synthetic data only: 2.0M functions from 17 parameterised families and short random expression trees, generated with
51
+ SymPy. Objective: H¹ angle matching, linearity, and a JEPA term predicting the latent of `f'` from that of `f`. One
52
+ training run of 40k steps on a single 8 GB GPU (6.6 hours); the modules were trained afterwards on the frozen encoder.
53
+
54
+ ## Results (from the paper, single run)
55
+
56
+ - Latent cosines and dot products correlate 0.993 and 0.998 with their H¹ counterparts.
57
+ - The derivative operator retrieves `f'` among 3,956 candidates for 82.0% of test functions.
58
+ - Exact symbolic regression from 64 samples: 76% / 76% / 62% at noise 0 / 0.1% / 1% on 50 test functions.
59
+ - Symbolic integration on the paper's test set: 91.7% verified by SymPy (full solver, which needs the training data).
60
+
61
+ ## Limitations
62
+
63
+ - Latent operators are approximate (median relative error of the derivative: 7%). Predicted latents decode to the
64
+ right shape more often than to exact constants; refit and verify the candidates.
65
+ - Weak outside the pretraining families: 20–25% on the integration test sets of Lample and Charton and 16–25% on
66
+ their ODE test sets, where task-specific models exceed 90%.
67
+ - The symbolic-regression advantage is measured on functions of the pretraining families.
68
+ - Univariate functions on a fixed interval, 37-token vocabulary without special functions, at most 126 tokens.
69
+ - Single training run: variability across seeds was not measured.
70
+
71
+ ## Citation
72
+
73
+ ```bibtex
74
+ @article{morenoarcas2026funcjepa,
75
+ title = {Func-JEPA: Learning a Sobolev Space of Functions as Latent Vectors},
76
+ author = {Moreno Arcas, Alejandro},
77
+ year = {2026}
78
+ }
79
+ ```
SHA256.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "encoder_pi/model.safetensors": "f8b7e49f05fc5473ae9b1d6c3b60932010e9021d914003efd455d0cba687c40f",
3
+ "encoder_05/model.safetensors": "e47f5d95abc1c7def7bb8c51e46c599034de8c0e327c1627b4cb0ce90dba68e5",
4
+ "decoder/model.safetensors": "f63a5b17f49f98f492805981dcac4315598878513926f5b109cc99e2592641e2",
5
+ "compose/model.safetensors": "75ab293392c0833dd8a0c8fe13b43361271bc5e2297f0ac774ce10391819c094",
6
+ "integrator/model.safetensors": "84807277397c886fa91b76634ddc1e5cbdadbfeb882f61bc9b817dbb91a1aec1",
7
+ "value_encoder/model.safetensors": "b94e7a37cd7c8488bd30a2a5d59b164d019f483a263469407873144bd593827f",
8
+ "decoder_lc/model.safetensors": "51390496190a3d4dbdf5a13017edf81cbe6becbf8396cbb6fd3a0c77cb4535ae",
9
+ "value_encoder_lc/model.safetensors": "7656cd9bc2c363b0c225e5051b27245a169013c16b2de97debb16b170b7bd106"
10
+ }
compose/config.json ADDED
@@ -0,0 +1,99 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "ops": [
3
+ "sin",
4
+ "cos",
5
+ "log",
6
+ "sqrt",
7
+ "square",
8
+ "cube",
9
+ "quad"
10
+ ],
11
+ "d": 256,
12
+ "config": {
13
+ "run_name": "compose_v8",
14
+ "data": "data/compose_v3",
15
+ "latents": "latents_v14.npz",
16
+ "ops": [
17
+ "sin",
18
+ "cos",
19
+ "log",
20
+ "sqrt",
21
+ "square",
22
+ "cube",
23
+ "quad"
24
+ ],
25
+ "seed": 0,
26
+ "model": {
27
+ "hidden": 768,
28
+ "depth": 3,
29
+ "mult": 2,
30
+ "dropout": 0.1,
31
+ "skip_init": "identity",
32
+ "head": "factored",
33
+ "structure": "exact",
34
+ "quad_from_square": true
35
+ },
36
+ "loss": {
37
+ "w_cos": 0.0,
38
+ "w_nce": 0.1,
39
+ "tau": 0.05,
40
+ "w_gain": 1.0
41
+ },
42
+ "train": {
43
+ "steps": 40000,
44
+ "batch_size": 1024,
45
+ "lr": 5e-05,
46
+ "weight_decay": 0.05,
47
+ "warmup": 500,
48
+ "min_lr_frac": 0.05,
49
+ "schedule": "wsd",
50
+ "decay_frac": 0.0,
51
+ "grad_clip": 1.0,
52
+ "eval_every": 2000,
53
+ "eval_n": 3000,
54
+ "patience": 0,
55
+ "ema": 0.0,
56
+ "max_lift_frac": 0.25,
57
+ "clean_exclude": "data/v4",
58
+ "aug_sign": true
59
+ }
60
+ },
61
+ "best": {
62
+ "sin": {
63
+ "val_top1": 0.7093333333333334,
64
+ "step": 34000
65
+ },
66
+ "cos": {
67
+ "val_top1": 0.649,
68
+ "step": 32000
69
+ },
70
+ "log": {
71
+ "val_top1": 0.6013834846519671,
72
+ "step": 38000
73
+ },
74
+ "sqrt": {
75
+ "val_top1": 0.5649109792284867,
76
+ "step": 40000
77
+ },
78
+ "square": {
79
+ "val_top1": 0.9226666666666666,
80
+ "step": 40000
81
+ },
82
+ "cube": {
83
+ "val_top1": 0.8953333333333333,
84
+ "step": 40000
85
+ },
86
+ "quad": {
87
+ "val_top1": 0.8016666666666666,
88
+ "step": 38000
89
+ }
90
+ },
91
+ "component": "compose",
92
+ "description": "7 composition operators (sin cos log sqrt square cube quad)",
93
+ "source_checkpoint": "runs/compose_v8/opbank.pt",
94
+ "domain": [
95
+ -3.141592653589793,
96
+ 3.141592653589793
97
+ ],
98
+ "format": "func-jepa-1"
99
+ }
compose/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:75ab293392c0833dd8a0c8fe13b43361271bc5e2297f0ac774ce10391819c094
3
+ size 211531352
decoder/config.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "run_name": "dec_v3",
4
+ "data": "data/v4",
5
+ "cache": "data/v4/dec_v14",
6
+ "seed": 0,
7
+ "model": {
8
+ "d": 384,
9
+ "layers": 6,
10
+ "heads": 6,
11
+ "ff": 1536,
12
+ "K": 8,
13
+ "dropout": 0.05
14
+ },
15
+ "train": {
16
+ "steps": 105000,
17
+ "batch": 512,
18
+ "lr": 1.5e-05,
19
+ "wd": 0.01,
20
+ "warmup": 2000,
21
+ "min_lr_frac": 0.05,
22
+ "schedule": "wsd",
23
+ "decay_frac": 0.0,
24
+ "label_smoothing": 0.02,
25
+ "grad_clip": 1.0,
26
+ "p_noise": 0.4,
27
+ "max_angle_cos": 0.99,
28
+ "logn_jitter": 0.02,
29
+ "p_dz": 0.3,
30
+ "rewarm": 500,
31
+ "log_every": 100,
32
+ "eval_every": 5000,
33
+ "eval_n": 1024,
34
+ "eval_ce_n": 4096,
35
+ "len_buckets": 16,
36
+ "max_tokens": 32768
37
+ },
38
+ "encoder": "runs/v14/best.pt"
39
+ },
40
+ "step": 105000,
41
+ "d_lat": 256,
42
+ "best": 0.8779296875,
43
+ "component": "decoder",
44
+ "description": "latent -> prefix tokens, on the frozen encoder_pi",
45
+ "source_checkpoint": "runs/dec_v3/best.pt",
46
+ "domain": [
47
+ -3.141592653589793,
48
+ 3.141592653589793
49
+ ],
50
+ "format": "func-jepa-1"
51
+ }
decoder/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f63a5b17f49f98f492805981dcac4315598878513926f5b109cc99e2592641e2
3
+ size 70730584
decoder_lc/config.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "run_name": "dec_v4",
4
+ "data": "data/lc_mix",
5
+ "cache": "data/lc_mix/dec_v14",
6
+ "seed": 0,
7
+ "model": {
8
+ "d": 384,
9
+ "layers": 6,
10
+ "heads": 6,
11
+ "ff": 1536,
12
+ "K": 8,
13
+ "dropout": 0.05
14
+ },
15
+ "train": {
16
+ "steps": 130000,
17
+ "batch": 512,
18
+ "lr": 0.0002,
19
+ "wd": 0.01,
20
+ "warmup": 2000,
21
+ "min_lr_frac": 0.05,
22
+ "schedule": "wsd",
23
+ "decay_frac": 0.2,
24
+ "label_smoothing": 0.02,
25
+ "grad_clip": 1.0,
26
+ "p_noise": 0.4,
27
+ "max_angle_cos": 0.99,
28
+ "logn_jitter": 0.02,
29
+ "p_dz": 0.3,
30
+ "rewarm": 500,
31
+ "log_every": 100,
32
+ "eval_every": 2500,
33
+ "eval_n": 1024,
34
+ "eval_ce_n": 4096,
35
+ "len_buckets": 16,
36
+ "max_tokens": 32768
37
+ },
38
+ "encoder": "runs/v14/best.pt"
39
+ },
40
+ "step": 130000,
41
+ "d_lat": 256,
42
+ "best": 0.791015625,
43
+ "component": "decoder_lc",
44
+ "description": "decoder fine-tuned on Lample-Charton-type expressions (ODE benchmark only)",
45
+ "source_checkpoint": "runs/dec_v4/best.pt",
46
+ "domain": [
47
+ -3.141592653589793,
48
+ 3.141592653589793
49
+ ],
50
+ "format": "func-jepa-1"
51
+ }
decoder_lc/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:51390496190a3d4dbdf5a13017edf81cbe6becbf8396cbb6fd3a0c77cb4535ae
3
+ size 70730584
encoder_05/config.json ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "step": 38000,
3
+ "config": {
4
+ "run_name": "e05",
5
+ "data": "data/v05",
6
+ "seed": 0,
7
+ "model": {
8
+ "hidden": 512,
9
+ "layers": 10,
10
+ "heads": 8,
11
+ "intermediate": 2176,
12
+ "max_pos": 130,
13
+ "dropout": 0.0,
14
+ "latent_dim": 256,
15
+ "pooling": "cls",
16
+ "A_init": "identity",
17
+ "A_inv": false,
18
+ "deriv": "factored",
19
+ "deriv_hidden": 1024,
20
+ "gain_hidden": 256,
21
+ "proj_init": 1.0,
22
+ "init": "scaled_normal",
23
+ "init_std": 0.02,
24
+ "vocab_size": null,
25
+ "magnitude": "computed"
26
+ },
27
+ "train": {
28
+ "batch_size": 384,
29
+ "steps": 40000,
30
+ "lr": 0.0003,
31
+ "A_lr": 0.001,
32
+ "weight_decay": 0.01,
33
+ "warmup": 700,
34
+ "min_lr_frac": 0.1,
35
+ "grad_clip": 1.0,
36
+ "bf16": true,
37
+ "mask_p": 0.15,
38
+ "log_every": 50,
39
+ "eval_every": 1000,
40
+ "ckpt_every": 1000,
41
+ "batch_size_after_mlm": 384,
42
+ "grad_ckpt": true,
43
+ "len_buckets": 4,
44
+ "p_group": 0.15,
45
+ "p_sweep": 0.25,
46
+ "p_combo": 0.15,
47
+ "p_single": 0.5,
48
+ "combo_coefs": [
49
+ 1,
50
+ -1,
51
+ 2,
52
+ -2,
53
+ 3,
54
+ -3,
55
+ 5,
56
+ -5,
57
+ 10,
58
+ -10,
59
+ "1/2",
60
+ "-1/2"
61
+ ],
62
+ "combo_min_rel": 0.1
63
+ },
64
+ "loss": {
65
+ "w_mlm": 0.0,
66
+ "w_jepa": 1.0,
67
+ "w_gram": 1.0,
68
+ "gram_mode": "cosine",
69
+ "jepa_start": 1000,
70
+ "jepa_ramp": 1000,
71
+ "jepa_enc_grad": 1.0,
72
+ "norm_eps": 0.05,
73
+ "jepa_stopgrad": true,
74
+ "combine": "sum",
75
+ "jepa_max_ratio": 0.5,
76
+ "lam_K": 0.1,
77
+ "guard": false,
78
+ "w_ijepa": 0.0,
79
+ "w_inv_reg": 0.0,
80
+ "mlm_hold": 0,
81
+ "mlm_anneal": 0,
82
+ "w_lin": 1.0,
83
+ "lin_ramp": 1000,
84
+ "lin_max_ratio": 1.0,
85
+ "w_jepa_d2": 0.0,
86
+ "w_jepa_pow": 0.0,
87
+ "gram_inner": "H1",
88
+ "sobolev_lambda": 0.5,
89
+ "lambda_ramp": 0,
90
+ "magnitude_ramp": 0,
91
+ "w_vnorm": 0.05,
92
+ "vnorm_ramp": 0,
93
+ "fprime_nograd": false,
94
+ "w_cent": 1.5,
95
+ "w_var": 1.0,
96
+ "var_gamma": 0.5,
97
+ "w_lrvar": 0.0,
98
+ "lin_start": 1000,
99
+ "jepa_norm": "split",
100
+ "jepa_gain_w": 1.0
101
+ }
102
+ },
103
+ "K": null,
104
+ "r": 0.5,
105
+ "sobolev_lambda": 0.5,
106
+ "mag_t": 1.0,
107
+ "component": "encoder_05",
108
+ "description": "encoder + derivative operator D on [0, 5] (paper: E_[0,5])",
109
+ "source_checkpoint": "runs/e05/best.pt",
110
+ "domain": [
111
+ 0.0,
112
+ 5.0
113
+ ],
114
+ "format": "func-jepa-1"
115
+ }
encoder_05/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e47f5d95abc1c7def7bb8c51e46c599034de8c0e327c1627b4cb0ce90dba68e5
3
+ size 140382720
encoder_pi/config.json ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "step": 39000,
3
+ "config": {
4
+ "run_name": "v14",
5
+ "data": "data/v4",
6
+ "seed": 0,
7
+ "model": {
8
+ "hidden": 512,
9
+ "layers": 10,
10
+ "heads": 8,
11
+ "intermediate": 2176,
12
+ "max_pos": 130,
13
+ "dropout": 0.0,
14
+ "latent_dim": 256,
15
+ "pooling": "cls",
16
+ "A_init": "identity",
17
+ "A_inv": false,
18
+ "deriv": "factored",
19
+ "deriv_hidden": 1024,
20
+ "gain_hidden": 256,
21
+ "proj_init": 1.0,
22
+ "init": "scaled_normal",
23
+ "init_std": 0.02,
24
+ "vocab_size": null,
25
+ "magnitude": "computed"
26
+ },
27
+ "train": {
28
+ "batch_size": 384,
29
+ "steps": 40000,
30
+ "lr": 0.0003,
31
+ "A_lr": 0.001,
32
+ "weight_decay": 0.01,
33
+ "warmup": 700,
34
+ "min_lr_frac": 0.1,
35
+ "grad_clip": 1.0,
36
+ "bf16": true,
37
+ "mask_p": 0.15,
38
+ "log_every": 50,
39
+ "eval_every": 1000,
40
+ "ckpt_every": 1000,
41
+ "batch_size_after_mlm": 384,
42
+ "grad_ckpt": true,
43
+ "len_buckets": 4,
44
+ "p_group": 0.15,
45
+ "p_sweep": 0.25,
46
+ "p_combo": 0.15,
47
+ "p_single": 0.5,
48
+ "combo_coefs": [
49
+ 1,
50
+ -1,
51
+ 2,
52
+ -2,
53
+ 3,
54
+ -3,
55
+ 5,
56
+ -5,
57
+ 10,
58
+ -10,
59
+ "1/2",
60
+ "-1/2"
61
+ ],
62
+ "combo_min_rel": 0.1
63
+ },
64
+ "loss": {
65
+ "w_mlm": 0.0,
66
+ "w_jepa": 1.0,
67
+ "w_gram": 1.0,
68
+ "gram_mode": "cosine",
69
+ "jepa_start": 1000,
70
+ "jepa_ramp": 1000,
71
+ "jepa_enc_grad": 1.0,
72
+ "norm_eps": 0.05,
73
+ "jepa_stopgrad": true,
74
+ "combine": "sum",
75
+ "jepa_max_ratio": 0.5,
76
+ "lam_K": 0.1,
77
+ "guard": false,
78
+ "w_ijepa": 0.0,
79
+ "w_inv_reg": 0.0,
80
+ "mlm_hold": 0,
81
+ "mlm_anneal": 0,
82
+ "w_lin": 1.0,
83
+ "lin_ramp": 1000,
84
+ "lin_max_ratio": 1.0,
85
+ "w_jepa_d2": 0.0,
86
+ "w_jepa_pow": 0.0,
87
+ "gram_inner": "H1",
88
+ "sobolev_lambda": 0.5,
89
+ "lambda_ramp": 0,
90
+ "magnitude_ramp": 0,
91
+ "w_vnorm": 0.05,
92
+ "vnorm_ramp": 0,
93
+ "fprime_nograd": false,
94
+ "w_cent": 1.5,
95
+ "w_var": 1.0,
96
+ "var_gamma": 0.5,
97
+ "w_lrvar": 0.0,
98
+ "lin_start": 1000,
99
+ "jepa_norm": "split",
100
+ "jepa_gain_w": 1.0
101
+ }
102
+ },
103
+ "K": null,
104
+ "r": 0.5,
105
+ "sobolev_lambda": 0.5,
106
+ "mag_t": 1.0,
107
+ "component": "encoder_pi",
108
+ "description": "encoder + derivative operator D on [-pi, pi] (paper: E_[-pi,pi])",
109
+ "source_checkpoint": "runs/v14/best.pt",
110
+ "domain": [
111
+ -3.141592653589793,
112
+ 3.141592653589793
113
+ ],
114
+ "format": "func-jepa-1"
115
+ }
encoder_pi/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f8b7e49f05fc5473ae9b1d6c3b60932010e9021d914003efd455d0cba687c40f
3
+ size 140382720
integrator/config.json ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "run_name": "int_v2",
4
+ "data": "data/v4",
5
+ "cache": "data/v4/dec_v14",
6
+ "seed": 0,
7
+ "model": {
8
+ "hidden": 768,
9
+ "depth": 3,
10
+ "mult": 2,
11
+ "dropout": 0.1
12
+ },
13
+ "loss": {
14
+ "w_gain": 1.0,
15
+ "w_nce": 0.1,
16
+ "tau": 0.05,
17
+ "w_cyc": 0.5
18
+ },
19
+ "train": {
20
+ "steps": 60000,
21
+ "batch": 1024,
22
+ "lr": 0.0003,
23
+ "wd": 0.05,
24
+ "warmup": 500,
25
+ "decay_frac": 0.33333334,
26
+ "min_lr_frac": 0.05,
27
+ "grad_clip": 1.0,
28
+ "eval_every": 1000,
29
+ "eval_n": 4096,
30
+ "aug_sign": true
31
+ }
32
+ },
33
+ "step": 56000,
34
+ "component": "integrator",
35
+ "description": "latent of f -> latent of the zero-mean antiderivative",
36
+ "source_checkpoint": "runs/int_v2/best.pt",
37
+ "domain": [
38
+ -3.141592653589793,
39
+ 3.141592653589793
40
+ ],
41
+ "format": "func-jepa-1"
42
+ }
integrator/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:84807277397c886fa91b76634ddc1e5cbdadbfeb882f61bc9b817dbb91a1aec1
3
+ size 30208140
value_encoder/config.json ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "run_name": "venc_v2",
4
+ "data": "data/v4",
5
+ "cache": "data/v4/dec_v14",
6
+ "seed": 0,
7
+ "model": {
8
+ "hidden": 768,
9
+ "depth": 3,
10
+ "mult": 2,
11
+ "dropout": 0.1,
12
+ "max_deg": 40
13
+ },
14
+ "loss": {
15
+ "w_nce": 0.1,
16
+ "tau": 0.05,
17
+ "w_dec": 0.5,
18
+ "dec_batch": 256
19
+ },
20
+ "train": {
21
+ "steps": 20000,
22
+ "batch": 1024,
23
+ "lr": 0.0003,
24
+ "wd": 0.05,
25
+ "warmup": 500,
26
+ "decay_frac": 0.33333334,
27
+ "min_lr_frac": 0.05,
28
+ "grad_clip": 1.0,
29
+ "eval_every": 1000,
30
+ "eval_n": 4096,
31
+ "noise_max": 0.02,
32
+ "p_noise": 0.5,
33
+ "eval_decode_n": 256
34
+ }
35
+ },
36
+ "step": 20000,
37
+ "component": "value_encoder",
38
+ "description": "64 samples -> unit direction of the latent",
39
+ "source_checkpoint": "runs/venc_v2/best.pt",
40
+ "domain": [
41
+ -3.141592653589793,
42
+ 3.141592653589793
43
+ ],
44
+ "format": "func-jepa-1"
45
+ }
value_encoder/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b94e7a37cd7c8488bd30a2a5d59b164d019f483a263469407873144bd593827f
3
+ size 29552696
value_encoder_lc/config.json ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "run_name": "venc_v4",
4
+ "data": "data/lc_mix",
5
+ "cache": "data/lc_mix/dec_v14",
6
+ "seed": 0,
7
+ "model": {
8
+ "hidden": 768,
9
+ "depth": 3,
10
+ "mult": 2,
11
+ "dropout": 0.1,
12
+ "max_deg": 40
13
+ },
14
+ "loss": {
15
+ "w_nce": 0.1,
16
+ "tau": 0.05,
17
+ "w_dec": 0.5,
18
+ "dec_batch": 256
19
+ },
20
+ "train": {
21
+ "steps": 10000,
22
+ "batch": 1024,
23
+ "lr": 0.0001,
24
+ "wd": 0.05,
25
+ "warmup": 300,
26
+ "decay_frac": 0.33333334,
27
+ "min_lr_frac": 0.05,
28
+ "grad_clip": 1.0,
29
+ "eval_every": 1000,
30
+ "eval_n": 4096,
31
+ "noise_max": 0.02,
32
+ "p_noise": 0.5,
33
+ "eval_decode_n": 256
34
+ }
35
+ },
36
+ "step": 9000,
37
+ "component": "value_encoder_lc",
38
+ "description": "value encoder fine-tuned with decoder_lc (ODE benchmark only)",
39
+ "source_checkpoint": "runs/venc_v4/best.pt",
40
+ "domain": [
41
+ -3.141592653589793,
42
+ 3.141592653589793
43
+ ],
44
+ "format": "func-jepa-1"
45
+ }
value_encoder_lc/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7656cd9bc2c363b0c225e5051b27245a169013c16b2de97debb16b170b7bd106
3
+ size 29552696