Func-JEPA pretrained weights
Browse files- README.md +79 -0
- SHA256.json +10 -0
- compose/config.json +99 -0
- compose/model.safetensors +3 -0
- decoder/config.json +51 -0
- decoder/model.safetensors +3 -0
- decoder_lc/config.json +51 -0
- decoder_lc/model.safetensors +3 -0
- encoder_05/config.json +115 -0
- encoder_05/model.safetensors +3 -0
- encoder_pi/config.json +115 -0
- encoder_pi/model.safetensors +3 -0
- integrator/config.json +42 -0
- integrator/model.safetensors +3 -0
- value_encoder/config.json +45 -0
- value_encoder/model.safetensors +3 -0
- value_encoder_lc/config.json +45 -0
- value_encoder_lc/model.safetensors +3 -0
README.md
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
tags: [symbolic-mathematics, jepa, representation-learning, symbolic-regression, function-encoder]
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
# Func-JEPA
|
| 8 |
+
|
| 9 |
+
Pretrained weights of *Func-JEPA: Learning a Sobolev Space of Functions as Latent Vectors* (Alejandro Moreno Arcas,
|
| 10 |
+
2026). Code: https://github.com/amorarc/Func-JEPA.
|
| 11 |
+
|
| 12 |
+
Func-JEPA encodes the symbolic form of a univariate function into a vector of R^256 whose norm is the exact H¹ norm of
|
| 13 |
+
the function and whose dot products approximate H¹ inner products. Differentiation is a learned latent operator. Small
|
| 14 |
+
modules on the frozen encoder decode latents to formulas, compose and integrate in latent space, and map sampled values
|
| 15 |
+
to latents.
|
| 16 |
+
|
| 17 |
+
## Usage
|
| 18 |
+
|
| 19 |
+
```bash
|
| 20 |
+
pip install git+https://github.com/amorarc/Func-JEPA
|
| 21 |
+
```
|
| 22 |
+
|
| 23 |
+
```python
|
| 24 |
+
import numpy as np
|
| 25 |
+
from fjepa import FuncJEPA
|
| 26 |
+
|
| 27 |
+
m = FuncJEPA.from_pretrained("amorarc/func-jepa")
|
| 28 |
+
z = m.encode("4*sin(7*x) - 3*cos(7*x)")
|
| 29 |
+
m.norm(z) # 12.87, the H1 norm of f on [-pi, pi]
|
| 30 |
+
m.decode(m.derivative(z))[0] # 21*sin(7*x) + 28*cos(7*x)
|
| 31 |
+
m.regress(32 * np.cos(6 * m.nodes) + 8) # 32*cos(6*x) + 8, from 64 samples
|
| 32 |
+
```
|
| 33 |
+
|
| 34 |
+
## Files
|
| 35 |
+
|
| 36 |
+
| Folder | Parameters | Role |
|
| 37 |
+
|---|---|---|
|
| 38 |
+
| `encoder_pi` | 35.1M | encoder + derivative operator on `[-π, π]` (all results of the paper) |
|
| 39 |
+
| `encoder_05` | 35.1M | the same model trained on `[0, 5]` |
|
| 40 |
+
| `decoder` | 17.7M | latent → prefix tokens |
|
| 41 |
+
| `compose` | 52.9M | composition operators: sin, cos, log, sqrt, square, cube, quadratic |
|
| 42 |
+
| `integrator` | 7.6M | latent of `f` → latent of the zero-mean antiderivative |
|
| 43 |
+
| `value_encoder` | 7.4M | 64 samples → latent direction |
|
| 44 |
+
| `decoder_lc`, `value_encoder_lc` | 17.7M, 7.4M | variants fine-tuned for the Lample–Charton ODE benchmark |
|
| 45 |
+
|
| 46 |
+
Each folder holds `model.safetensors` and `config.json`; `SHA256.json` lists the hashes.
|
| 47 |
+
|
| 48 |
+
## Training
|
| 49 |
+
|
| 50 |
+
Synthetic data only: 2.0M functions from 17 parameterised families and short random expression trees, generated with
|
| 51 |
+
SymPy. Objective: H¹ angle matching, linearity, and a JEPA term predicting the latent of `f'` from that of `f`. One
|
| 52 |
+
training run of 40k steps on a single 8 GB GPU (6.6 hours); the modules were trained afterwards on the frozen encoder.
|
| 53 |
+
|
| 54 |
+
## Results (from the paper, single run)
|
| 55 |
+
|
| 56 |
+
- Latent cosines and dot products correlate 0.993 and 0.998 with their H¹ counterparts.
|
| 57 |
+
- The derivative operator retrieves `f'` among 3,956 candidates for 82.0% of test functions.
|
| 58 |
+
- Exact symbolic regression from 64 samples: 76% / 76% / 62% at noise 0 / 0.1% / 1% on 50 test functions.
|
| 59 |
+
- Symbolic integration on the paper's test set: 91.7% verified by SymPy (full solver, which needs the training data).
|
| 60 |
+
|
| 61 |
+
## Limitations
|
| 62 |
+
|
| 63 |
+
- Latent operators are approximate (median relative error of the derivative: 7%). Predicted latents decode to the
|
| 64 |
+
right shape more often than to exact constants; refit and verify the candidates.
|
| 65 |
+
- Weak outside the pretraining families: 20–25% on the integration test sets of Lample and Charton and 16–25% on
|
| 66 |
+
their ODE test sets, where task-specific models exceed 90%.
|
| 67 |
+
- The symbolic-regression advantage is measured on functions of the pretraining families.
|
| 68 |
+
- Univariate functions on a fixed interval, 37-token vocabulary without special functions, at most 126 tokens.
|
| 69 |
+
- Single training run: variability across seeds was not measured.
|
| 70 |
+
|
| 71 |
+
## Citation
|
| 72 |
+
|
| 73 |
+
```bibtex
|
| 74 |
+
@article{morenoarcas2026funcjepa,
|
| 75 |
+
title = {Func-JEPA: Learning a Sobolev Space of Functions as Latent Vectors},
|
| 76 |
+
author = {Moreno Arcas, Alejandro},
|
| 77 |
+
year = {2026}
|
| 78 |
+
}
|
| 79 |
+
```
|
SHA256.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"encoder_pi/model.safetensors": "f8b7e49f05fc5473ae9b1d6c3b60932010e9021d914003efd455d0cba687c40f",
|
| 3 |
+
"encoder_05/model.safetensors": "e47f5d95abc1c7def7bb8c51e46c599034de8c0e327c1627b4cb0ce90dba68e5",
|
| 4 |
+
"decoder/model.safetensors": "f63a5b17f49f98f492805981dcac4315598878513926f5b109cc99e2592641e2",
|
| 5 |
+
"compose/model.safetensors": "75ab293392c0833dd8a0c8fe13b43361271bc5e2297f0ac774ce10391819c094",
|
| 6 |
+
"integrator/model.safetensors": "84807277397c886fa91b76634ddc1e5cbdadbfeb882f61bc9b817dbb91a1aec1",
|
| 7 |
+
"value_encoder/model.safetensors": "b94e7a37cd7c8488bd30a2a5d59b164d019f483a263469407873144bd593827f",
|
| 8 |
+
"decoder_lc/model.safetensors": "51390496190a3d4dbdf5a13017edf81cbe6becbf8396cbb6fd3a0c77cb4535ae",
|
| 9 |
+
"value_encoder_lc/model.safetensors": "7656cd9bc2c363b0c225e5051b27245a169013c16b2de97debb16b170b7bd106"
|
| 10 |
+
}
|
compose/config.json
ADDED
|
@@ -0,0 +1,99 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"ops": [
|
| 3 |
+
"sin",
|
| 4 |
+
"cos",
|
| 5 |
+
"log",
|
| 6 |
+
"sqrt",
|
| 7 |
+
"square",
|
| 8 |
+
"cube",
|
| 9 |
+
"quad"
|
| 10 |
+
],
|
| 11 |
+
"d": 256,
|
| 12 |
+
"config": {
|
| 13 |
+
"run_name": "compose_v8",
|
| 14 |
+
"data": "data/compose_v3",
|
| 15 |
+
"latents": "latents_v14.npz",
|
| 16 |
+
"ops": [
|
| 17 |
+
"sin",
|
| 18 |
+
"cos",
|
| 19 |
+
"log",
|
| 20 |
+
"sqrt",
|
| 21 |
+
"square",
|
| 22 |
+
"cube",
|
| 23 |
+
"quad"
|
| 24 |
+
],
|
| 25 |
+
"seed": 0,
|
| 26 |
+
"model": {
|
| 27 |
+
"hidden": 768,
|
| 28 |
+
"depth": 3,
|
| 29 |
+
"mult": 2,
|
| 30 |
+
"dropout": 0.1,
|
| 31 |
+
"skip_init": "identity",
|
| 32 |
+
"head": "factored",
|
| 33 |
+
"structure": "exact",
|
| 34 |
+
"quad_from_square": true
|
| 35 |
+
},
|
| 36 |
+
"loss": {
|
| 37 |
+
"w_cos": 0.0,
|
| 38 |
+
"w_nce": 0.1,
|
| 39 |
+
"tau": 0.05,
|
| 40 |
+
"w_gain": 1.0
|
| 41 |
+
},
|
| 42 |
+
"train": {
|
| 43 |
+
"steps": 40000,
|
| 44 |
+
"batch_size": 1024,
|
| 45 |
+
"lr": 5e-05,
|
| 46 |
+
"weight_decay": 0.05,
|
| 47 |
+
"warmup": 500,
|
| 48 |
+
"min_lr_frac": 0.05,
|
| 49 |
+
"schedule": "wsd",
|
| 50 |
+
"decay_frac": 0.0,
|
| 51 |
+
"grad_clip": 1.0,
|
| 52 |
+
"eval_every": 2000,
|
| 53 |
+
"eval_n": 3000,
|
| 54 |
+
"patience": 0,
|
| 55 |
+
"ema": 0.0,
|
| 56 |
+
"max_lift_frac": 0.25,
|
| 57 |
+
"clean_exclude": "data/v4",
|
| 58 |
+
"aug_sign": true
|
| 59 |
+
}
|
| 60 |
+
},
|
| 61 |
+
"best": {
|
| 62 |
+
"sin": {
|
| 63 |
+
"val_top1": 0.7093333333333334,
|
| 64 |
+
"step": 34000
|
| 65 |
+
},
|
| 66 |
+
"cos": {
|
| 67 |
+
"val_top1": 0.649,
|
| 68 |
+
"step": 32000
|
| 69 |
+
},
|
| 70 |
+
"log": {
|
| 71 |
+
"val_top1": 0.6013834846519671,
|
| 72 |
+
"step": 38000
|
| 73 |
+
},
|
| 74 |
+
"sqrt": {
|
| 75 |
+
"val_top1": 0.5649109792284867,
|
| 76 |
+
"step": 40000
|
| 77 |
+
},
|
| 78 |
+
"square": {
|
| 79 |
+
"val_top1": 0.9226666666666666,
|
| 80 |
+
"step": 40000
|
| 81 |
+
},
|
| 82 |
+
"cube": {
|
| 83 |
+
"val_top1": 0.8953333333333333,
|
| 84 |
+
"step": 40000
|
| 85 |
+
},
|
| 86 |
+
"quad": {
|
| 87 |
+
"val_top1": 0.8016666666666666,
|
| 88 |
+
"step": 38000
|
| 89 |
+
}
|
| 90 |
+
},
|
| 91 |
+
"component": "compose",
|
| 92 |
+
"description": "7 composition operators (sin cos log sqrt square cube quad)",
|
| 93 |
+
"source_checkpoint": "runs/compose_v8/opbank.pt",
|
| 94 |
+
"domain": [
|
| 95 |
+
-3.141592653589793,
|
| 96 |
+
3.141592653589793
|
| 97 |
+
],
|
| 98 |
+
"format": "func-jepa-1"
|
| 99 |
+
}
|
compose/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:75ab293392c0833dd8a0c8fe13b43361271bc5e2297f0ac774ce10391819c094
|
| 3 |
+
size 211531352
|
decoder/config.json
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": {
|
| 3 |
+
"run_name": "dec_v3",
|
| 4 |
+
"data": "data/v4",
|
| 5 |
+
"cache": "data/v4/dec_v14",
|
| 6 |
+
"seed": 0,
|
| 7 |
+
"model": {
|
| 8 |
+
"d": 384,
|
| 9 |
+
"layers": 6,
|
| 10 |
+
"heads": 6,
|
| 11 |
+
"ff": 1536,
|
| 12 |
+
"K": 8,
|
| 13 |
+
"dropout": 0.05
|
| 14 |
+
},
|
| 15 |
+
"train": {
|
| 16 |
+
"steps": 105000,
|
| 17 |
+
"batch": 512,
|
| 18 |
+
"lr": 1.5e-05,
|
| 19 |
+
"wd": 0.01,
|
| 20 |
+
"warmup": 2000,
|
| 21 |
+
"min_lr_frac": 0.05,
|
| 22 |
+
"schedule": "wsd",
|
| 23 |
+
"decay_frac": 0.0,
|
| 24 |
+
"label_smoothing": 0.02,
|
| 25 |
+
"grad_clip": 1.0,
|
| 26 |
+
"p_noise": 0.4,
|
| 27 |
+
"max_angle_cos": 0.99,
|
| 28 |
+
"logn_jitter": 0.02,
|
| 29 |
+
"p_dz": 0.3,
|
| 30 |
+
"rewarm": 500,
|
| 31 |
+
"log_every": 100,
|
| 32 |
+
"eval_every": 5000,
|
| 33 |
+
"eval_n": 1024,
|
| 34 |
+
"eval_ce_n": 4096,
|
| 35 |
+
"len_buckets": 16,
|
| 36 |
+
"max_tokens": 32768
|
| 37 |
+
},
|
| 38 |
+
"encoder": "runs/v14/best.pt"
|
| 39 |
+
},
|
| 40 |
+
"step": 105000,
|
| 41 |
+
"d_lat": 256,
|
| 42 |
+
"best": 0.8779296875,
|
| 43 |
+
"component": "decoder",
|
| 44 |
+
"description": "latent -> prefix tokens, on the frozen encoder_pi",
|
| 45 |
+
"source_checkpoint": "runs/dec_v3/best.pt",
|
| 46 |
+
"domain": [
|
| 47 |
+
-3.141592653589793,
|
| 48 |
+
3.141592653589793
|
| 49 |
+
],
|
| 50 |
+
"format": "func-jepa-1"
|
| 51 |
+
}
|
decoder/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f63a5b17f49f98f492805981dcac4315598878513926f5b109cc99e2592641e2
|
| 3 |
+
size 70730584
|
decoder_lc/config.json
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": {
|
| 3 |
+
"run_name": "dec_v4",
|
| 4 |
+
"data": "data/lc_mix",
|
| 5 |
+
"cache": "data/lc_mix/dec_v14",
|
| 6 |
+
"seed": 0,
|
| 7 |
+
"model": {
|
| 8 |
+
"d": 384,
|
| 9 |
+
"layers": 6,
|
| 10 |
+
"heads": 6,
|
| 11 |
+
"ff": 1536,
|
| 12 |
+
"K": 8,
|
| 13 |
+
"dropout": 0.05
|
| 14 |
+
},
|
| 15 |
+
"train": {
|
| 16 |
+
"steps": 130000,
|
| 17 |
+
"batch": 512,
|
| 18 |
+
"lr": 0.0002,
|
| 19 |
+
"wd": 0.01,
|
| 20 |
+
"warmup": 2000,
|
| 21 |
+
"min_lr_frac": 0.05,
|
| 22 |
+
"schedule": "wsd",
|
| 23 |
+
"decay_frac": 0.2,
|
| 24 |
+
"label_smoothing": 0.02,
|
| 25 |
+
"grad_clip": 1.0,
|
| 26 |
+
"p_noise": 0.4,
|
| 27 |
+
"max_angle_cos": 0.99,
|
| 28 |
+
"logn_jitter": 0.02,
|
| 29 |
+
"p_dz": 0.3,
|
| 30 |
+
"rewarm": 500,
|
| 31 |
+
"log_every": 100,
|
| 32 |
+
"eval_every": 2500,
|
| 33 |
+
"eval_n": 1024,
|
| 34 |
+
"eval_ce_n": 4096,
|
| 35 |
+
"len_buckets": 16,
|
| 36 |
+
"max_tokens": 32768
|
| 37 |
+
},
|
| 38 |
+
"encoder": "runs/v14/best.pt"
|
| 39 |
+
},
|
| 40 |
+
"step": 130000,
|
| 41 |
+
"d_lat": 256,
|
| 42 |
+
"best": 0.791015625,
|
| 43 |
+
"component": "decoder_lc",
|
| 44 |
+
"description": "decoder fine-tuned on Lample-Charton-type expressions (ODE benchmark only)",
|
| 45 |
+
"source_checkpoint": "runs/dec_v4/best.pt",
|
| 46 |
+
"domain": [
|
| 47 |
+
-3.141592653589793,
|
| 48 |
+
3.141592653589793
|
| 49 |
+
],
|
| 50 |
+
"format": "func-jepa-1"
|
| 51 |
+
}
|
decoder_lc/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:51390496190a3d4dbdf5a13017edf81cbe6becbf8396cbb6fd3a0c77cb4535ae
|
| 3 |
+
size 70730584
|
encoder_05/config.json
ADDED
|
@@ -0,0 +1,115 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"step": 38000,
|
| 3 |
+
"config": {
|
| 4 |
+
"run_name": "e05",
|
| 5 |
+
"data": "data/v05",
|
| 6 |
+
"seed": 0,
|
| 7 |
+
"model": {
|
| 8 |
+
"hidden": 512,
|
| 9 |
+
"layers": 10,
|
| 10 |
+
"heads": 8,
|
| 11 |
+
"intermediate": 2176,
|
| 12 |
+
"max_pos": 130,
|
| 13 |
+
"dropout": 0.0,
|
| 14 |
+
"latent_dim": 256,
|
| 15 |
+
"pooling": "cls",
|
| 16 |
+
"A_init": "identity",
|
| 17 |
+
"A_inv": false,
|
| 18 |
+
"deriv": "factored",
|
| 19 |
+
"deriv_hidden": 1024,
|
| 20 |
+
"gain_hidden": 256,
|
| 21 |
+
"proj_init": 1.0,
|
| 22 |
+
"init": "scaled_normal",
|
| 23 |
+
"init_std": 0.02,
|
| 24 |
+
"vocab_size": null,
|
| 25 |
+
"magnitude": "computed"
|
| 26 |
+
},
|
| 27 |
+
"train": {
|
| 28 |
+
"batch_size": 384,
|
| 29 |
+
"steps": 40000,
|
| 30 |
+
"lr": 0.0003,
|
| 31 |
+
"A_lr": 0.001,
|
| 32 |
+
"weight_decay": 0.01,
|
| 33 |
+
"warmup": 700,
|
| 34 |
+
"min_lr_frac": 0.1,
|
| 35 |
+
"grad_clip": 1.0,
|
| 36 |
+
"bf16": true,
|
| 37 |
+
"mask_p": 0.15,
|
| 38 |
+
"log_every": 50,
|
| 39 |
+
"eval_every": 1000,
|
| 40 |
+
"ckpt_every": 1000,
|
| 41 |
+
"batch_size_after_mlm": 384,
|
| 42 |
+
"grad_ckpt": true,
|
| 43 |
+
"len_buckets": 4,
|
| 44 |
+
"p_group": 0.15,
|
| 45 |
+
"p_sweep": 0.25,
|
| 46 |
+
"p_combo": 0.15,
|
| 47 |
+
"p_single": 0.5,
|
| 48 |
+
"combo_coefs": [
|
| 49 |
+
1,
|
| 50 |
+
-1,
|
| 51 |
+
2,
|
| 52 |
+
-2,
|
| 53 |
+
3,
|
| 54 |
+
-3,
|
| 55 |
+
5,
|
| 56 |
+
-5,
|
| 57 |
+
10,
|
| 58 |
+
-10,
|
| 59 |
+
"1/2",
|
| 60 |
+
"-1/2"
|
| 61 |
+
],
|
| 62 |
+
"combo_min_rel": 0.1
|
| 63 |
+
},
|
| 64 |
+
"loss": {
|
| 65 |
+
"w_mlm": 0.0,
|
| 66 |
+
"w_jepa": 1.0,
|
| 67 |
+
"w_gram": 1.0,
|
| 68 |
+
"gram_mode": "cosine",
|
| 69 |
+
"jepa_start": 1000,
|
| 70 |
+
"jepa_ramp": 1000,
|
| 71 |
+
"jepa_enc_grad": 1.0,
|
| 72 |
+
"norm_eps": 0.05,
|
| 73 |
+
"jepa_stopgrad": true,
|
| 74 |
+
"combine": "sum",
|
| 75 |
+
"jepa_max_ratio": 0.5,
|
| 76 |
+
"lam_K": 0.1,
|
| 77 |
+
"guard": false,
|
| 78 |
+
"w_ijepa": 0.0,
|
| 79 |
+
"w_inv_reg": 0.0,
|
| 80 |
+
"mlm_hold": 0,
|
| 81 |
+
"mlm_anneal": 0,
|
| 82 |
+
"w_lin": 1.0,
|
| 83 |
+
"lin_ramp": 1000,
|
| 84 |
+
"lin_max_ratio": 1.0,
|
| 85 |
+
"w_jepa_d2": 0.0,
|
| 86 |
+
"w_jepa_pow": 0.0,
|
| 87 |
+
"gram_inner": "H1",
|
| 88 |
+
"sobolev_lambda": 0.5,
|
| 89 |
+
"lambda_ramp": 0,
|
| 90 |
+
"magnitude_ramp": 0,
|
| 91 |
+
"w_vnorm": 0.05,
|
| 92 |
+
"vnorm_ramp": 0,
|
| 93 |
+
"fprime_nograd": false,
|
| 94 |
+
"w_cent": 1.5,
|
| 95 |
+
"w_var": 1.0,
|
| 96 |
+
"var_gamma": 0.5,
|
| 97 |
+
"w_lrvar": 0.0,
|
| 98 |
+
"lin_start": 1000,
|
| 99 |
+
"jepa_norm": "split",
|
| 100 |
+
"jepa_gain_w": 1.0
|
| 101 |
+
}
|
| 102 |
+
},
|
| 103 |
+
"K": null,
|
| 104 |
+
"r": 0.5,
|
| 105 |
+
"sobolev_lambda": 0.5,
|
| 106 |
+
"mag_t": 1.0,
|
| 107 |
+
"component": "encoder_05",
|
| 108 |
+
"description": "encoder + derivative operator D on [0, 5] (paper: E_[0,5])",
|
| 109 |
+
"source_checkpoint": "runs/e05/best.pt",
|
| 110 |
+
"domain": [
|
| 111 |
+
0.0,
|
| 112 |
+
5.0
|
| 113 |
+
],
|
| 114 |
+
"format": "func-jepa-1"
|
| 115 |
+
}
|
encoder_05/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e47f5d95abc1c7def7bb8c51e46c599034de8c0e327c1627b4cb0ce90dba68e5
|
| 3 |
+
size 140382720
|
encoder_pi/config.json
ADDED
|
@@ -0,0 +1,115 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"step": 39000,
|
| 3 |
+
"config": {
|
| 4 |
+
"run_name": "v14",
|
| 5 |
+
"data": "data/v4",
|
| 6 |
+
"seed": 0,
|
| 7 |
+
"model": {
|
| 8 |
+
"hidden": 512,
|
| 9 |
+
"layers": 10,
|
| 10 |
+
"heads": 8,
|
| 11 |
+
"intermediate": 2176,
|
| 12 |
+
"max_pos": 130,
|
| 13 |
+
"dropout": 0.0,
|
| 14 |
+
"latent_dim": 256,
|
| 15 |
+
"pooling": "cls",
|
| 16 |
+
"A_init": "identity",
|
| 17 |
+
"A_inv": false,
|
| 18 |
+
"deriv": "factored",
|
| 19 |
+
"deriv_hidden": 1024,
|
| 20 |
+
"gain_hidden": 256,
|
| 21 |
+
"proj_init": 1.0,
|
| 22 |
+
"init": "scaled_normal",
|
| 23 |
+
"init_std": 0.02,
|
| 24 |
+
"vocab_size": null,
|
| 25 |
+
"magnitude": "computed"
|
| 26 |
+
},
|
| 27 |
+
"train": {
|
| 28 |
+
"batch_size": 384,
|
| 29 |
+
"steps": 40000,
|
| 30 |
+
"lr": 0.0003,
|
| 31 |
+
"A_lr": 0.001,
|
| 32 |
+
"weight_decay": 0.01,
|
| 33 |
+
"warmup": 700,
|
| 34 |
+
"min_lr_frac": 0.1,
|
| 35 |
+
"grad_clip": 1.0,
|
| 36 |
+
"bf16": true,
|
| 37 |
+
"mask_p": 0.15,
|
| 38 |
+
"log_every": 50,
|
| 39 |
+
"eval_every": 1000,
|
| 40 |
+
"ckpt_every": 1000,
|
| 41 |
+
"batch_size_after_mlm": 384,
|
| 42 |
+
"grad_ckpt": true,
|
| 43 |
+
"len_buckets": 4,
|
| 44 |
+
"p_group": 0.15,
|
| 45 |
+
"p_sweep": 0.25,
|
| 46 |
+
"p_combo": 0.15,
|
| 47 |
+
"p_single": 0.5,
|
| 48 |
+
"combo_coefs": [
|
| 49 |
+
1,
|
| 50 |
+
-1,
|
| 51 |
+
2,
|
| 52 |
+
-2,
|
| 53 |
+
3,
|
| 54 |
+
-3,
|
| 55 |
+
5,
|
| 56 |
+
-5,
|
| 57 |
+
10,
|
| 58 |
+
-10,
|
| 59 |
+
"1/2",
|
| 60 |
+
"-1/2"
|
| 61 |
+
],
|
| 62 |
+
"combo_min_rel": 0.1
|
| 63 |
+
},
|
| 64 |
+
"loss": {
|
| 65 |
+
"w_mlm": 0.0,
|
| 66 |
+
"w_jepa": 1.0,
|
| 67 |
+
"w_gram": 1.0,
|
| 68 |
+
"gram_mode": "cosine",
|
| 69 |
+
"jepa_start": 1000,
|
| 70 |
+
"jepa_ramp": 1000,
|
| 71 |
+
"jepa_enc_grad": 1.0,
|
| 72 |
+
"norm_eps": 0.05,
|
| 73 |
+
"jepa_stopgrad": true,
|
| 74 |
+
"combine": "sum",
|
| 75 |
+
"jepa_max_ratio": 0.5,
|
| 76 |
+
"lam_K": 0.1,
|
| 77 |
+
"guard": false,
|
| 78 |
+
"w_ijepa": 0.0,
|
| 79 |
+
"w_inv_reg": 0.0,
|
| 80 |
+
"mlm_hold": 0,
|
| 81 |
+
"mlm_anneal": 0,
|
| 82 |
+
"w_lin": 1.0,
|
| 83 |
+
"lin_ramp": 1000,
|
| 84 |
+
"lin_max_ratio": 1.0,
|
| 85 |
+
"w_jepa_d2": 0.0,
|
| 86 |
+
"w_jepa_pow": 0.0,
|
| 87 |
+
"gram_inner": "H1",
|
| 88 |
+
"sobolev_lambda": 0.5,
|
| 89 |
+
"lambda_ramp": 0,
|
| 90 |
+
"magnitude_ramp": 0,
|
| 91 |
+
"w_vnorm": 0.05,
|
| 92 |
+
"vnorm_ramp": 0,
|
| 93 |
+
"fprime_nograd": false,
|
| 94 |
+
"w_cent": 1.5,
|
| 95 |
+
"w_var": 1.0,
|
| 96 |
+
"var_gamma": 0.5,
|
| 97 |
+
"w_lrvar": 0.0,
|
| 98 |
+
"lin_start": 1000,
|
| 99 |
+
"jepa_norm": "split",
|
| 100 |
+
"jepa_gain_w": 1.0
|
| 101 |
+
}
|
| 102 |
+
},
|
| 103 |
+
"K": null,
|
| 104 |
+
"r": 0.5,
|
| 105 |
+
"sobolev_lambda": 0.5,
|
| 106 |
+
"mag_t": 1.0,
|
| 107 |
+
"component": "encoder_pi",
|
| 108 |
+
"description": "encoder + derivative operator D on [-pi, pi] (paper: E_[-pi,pi])",
|
| 109 |
+
"source_checkpoint": "runs/v14/best.pt",
|
| 110 |
+
"domain": [
|
| 111 |
+
-3.141592653589793,
|
| 112 |
+
3.141592653589793
|
| 113 |
+
],
|
| 114 |
+
"format": "func-jepa-1"
|
| 115 |
+
}
|
encoder_pi/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f8b7e49f05fc5473ae9b1d6c3b60932010e9021d914003efd455d0cba687c40f
|
| 3 |
+
size 140382720
|
integrator/config.json
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": {
|
| 3 |
+
"run_name": "int_v2",
|
| 4 |
+
"data": "data/v4",
|
| 5 |
+
"cache": "data/v4/dec_v14",
|
| 6 |
+
"seed": 0,
|
| 7 |
+
"model": {
|
| 8 |
+
"hidden": 768,
|
| 9 |
+
"depth": 3,
|
| 10 |
+
"mult": 2,
|
| 11 |
+
"dropout": 0.1
|
| 12 |
+
},
|
| 13 |
+
"loss": {
|
| 14 |
+
"w_gain": 1.0,
|
| 15 |
+
"w_nce": 0.1,
|
| 16 |
+
"tau": 0.05,
|
| 17 |
+
"w_cyc": 0.5
|
| 18 |
+
},
|
| 19 |
+
"train": {
|
| 20 |
+
"steps": 60000,
|
| 21 |
+
"batch": 1024,
|
| 22 |
+
"lr": 0.0003,
|
| 23 |
+
"wd": 0.05,
|
| 24 |
+
"warmup": 500,
|
| 25 |
+
"decay_frac": 0.33333334,
|
| 26 |
+
"min_lr_frac": 0.05,
|
| 27 |
+
"grad_clip": 1.0,
|
| 28 |
+
"eval_every": 1000,
|
| 29 |
+
"eval_n": 4096,
|
| 30 |
+
"aug_sign": true
|
| 31 |
+
}
|
| 32 |
+
},
|
| 33 |
+
"step": 56000,
|
| 34 |
+
"component": "integrator",
|
| 35 |
+
"description": "latent of f -> latent of the zero-mean antiderivative",
|
| 36 |
+
"source_checkpoint": "runs/int_v2/best.pt",
|
| 37 |
+
"domain": [
|
| 38 |
+
-3.141592653589793,
|
| 39 |
+
3.141592653589793
|
| 40 |
+
],
|
| 41 |
+
"format": "func-jepa-1"
|
| 42 |
+
}
|
integrator/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:84807277397c886fa91b76634ddc1e5cbdadbfeb882f61bc9b817dbb91a1aec1
|
| 3 |
+
size 30208140
|
value_encoder/config.json
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": {
|
| 3 |
+
"run_name": "venc_v2",
|
| 4 |
+
"data": "data/v4",
|
| 5 |
+
"cache": "data/v4/dec_v14",
|
| 6 |
+
"seed": 0,
|
| 7 |
+
"model": {
|
| 8 |
+
"hidden": 768,
|
| 9 |
+
"depth": 3,
|
| 10 |
+
"mult": 2,
|
| 11 |
+
"dropout": 0.1,
|
| 12 |
+
"max_deg": 40
|
| 13 |
+
},
|
| 14 |
+
"loss": {
|
| 15 |
+
"w_nce": 0.1,
|
| 16 |
+
"tau": 0.05,
|
| 17 |
+
"w_dec": 0.5,
|
| 18 |
+
"dec_batch": 256
|
| 19 |
+
},
|
| 20 |
+
"train": {
|
| 21 |
+
"steps": 20000,
|
| 22 |
+
"batch": 1024,
|
| 23 |
+
"lr": 0.0003,
|
| 24 |
+
"wd": 0.05,
|
| 25 |
+
"warmup": 500,
|
| 26 |
+
"decay_frac": 0.33333334,
|
| 27 |
+
"min_lr_frac": 0.05,
|
| 28 |
+
"grad_clip": 1.0,
|
| 29 |
+
"eval_every": 1000,
|
| 30 |
+
"eval_n": 4096,
|
| 31 |
+
"noise_max": 0.02,
|
| 32 |
+
"p_noise": 0.5,
|
| 33 |
+
"eval_decode_n": 256
|
| 34 |
+
}
|
| 35 |
+
},
|
| 36 |
+
"step": 20000,
|
| 37 |
+
"component": "value_encoder",
|
| 38 |
+
"description": "64 samples -> unit direction of the latent",
|
| 39 |
+
"source_checkpoint": "runs/venc_v2/best.pt",
|
| 40 |
+
"domain": [
|
| 41 |
+
-3.141592653589793,
|
| 42 |
+
3.141592653589793
|
| 43 |
+
],
|
| 44 |
+
"format": "func-jepa-1"
|
| 45 |
+
}
|
value_encoder/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b94e7a37cd7c8488bd30a2a5d59b164d019f483a263469407873144bd593827f
|
| 3 |
+
size 29552696
|
value_encoder_lc/config.json
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": {
|
| 3 |
+
"run_name": "venc_v4",
|
| 4 |
+
"data": "data/lc_mix",
|
| 5 |
+
"cache": "data/lc_mix/dec_v14",
|
| 6 |
+
"seed": 0,
|
| 7 |
+
"model": {
|
| 8 |
+
"hidden": 768,
|
| 9 |
+
"depth": 3,
|
| 10 |
+
"mult": 2,
|
| 11 |
+
"dropout": 0.1,
|
| 12 |
+
"max_deg": 40
|
| 13 |
+
},
|
| 14 |
+
"loss": {
|
| 15 |
+
"w_nce": 0.1,
|
| 16 |
+
"tau": 0.05,
|
| 17 |
+
"w_dec": 0.5,
|
| 18 |
+
"dec_batch": 256
|
| 19 |
+
},
|
| 20 |
+
"train": {
|
| 21 |
+
"steps": 10000,
|
| 22 |
+
"batch": 1024,
|
| 23 |
+
"lr": 0.0001,
|
| 24 |
+
"wd": 0.05,
|
| 25 |
+
"warmup": 300,
|
| 26 |
+
"decay_frac": 0.33333334,
|
| 27 |
+
"min_lr_frac": 0.05,
|
| 28 |
+
"grad_clip": 1.0,
|
| 29 |
+
"eval_every": 1000,
|
| 30 |
+
"eval_n": 4096,
|
| 31 |
+
"noise_max": 0.02,
|
| 32 |
+
"p_noise": 0.5,
|
| 33 |
+
"eval_decode_n": 256
|
| 34 |
+
}
|
| 35 |
+
},
|
| 36 |
+
"step": 9000,
|
| 37 |
+
"component": "value_encoder_lc",
|
| 38 |
+
"description": "value encoder fine-tuned with decoder_lc (ODE benchmark only)",
|
| 39 |
+
"source_checkpoint": "runs/venc_v4/best.pt",
|
| 40 |
+
"domain": [
|
| 41 |
+
-3.141592653589793,
|
| 42 |
+
3.141592653589793
|
| 43 |
+
],
|
| 44 |
+
"format": "func-jepa-1"
|
| 45 |
+
}
|
value_encoder_lc/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7656cd9bc2c363b0c225e5051b27245a169013c16b2de97debb16b170b7bd106
|
| 3 |
+
size 29552696
|