IdentiDeform-prior
A video prior for deformable-object material identification. From a short 3-view video of an object being pressed, dropped, bent, sheared, stretched or tapped, it predicts
- the constitutive class:
jelly(elastic),metal(elasto-plastic),plasticine; - the material parameters in the units of the simulator it was trained on: specific stiffness E/ρ,
Poisson ratio ν, density ρ and specific yield stress σ_y/ρ (σ_y is undefined for
jelly); - a predictive uncertainty for each parameter.
It is a 9-member ensemble of small heads (2.85 M parameters each) on top of a frozen V-JEPA 2 ViT-L video encoder. The heads are trained on simulated data only.
(a, b) Identifiable-parameter error vs. number of simulator calls during simulation-in-the-loop refinement, on validation and test. Starting from this prior with 0 simulations gives the error that an initialisation from the privileged ground-truth-class mean only reaches after about 10–12 simulations. (c) A downstream gripper task from the accompanying paper.
Only E/ρ and σ_y/ρ are identifiable from video. In MPM simulation the motion stays the same when E, σ_y and ρ are scaled together, and ν has only a weak effect on it. ρ and ν are still predicted, but they stay close to the training mean (their errors are about as large as the constant baseline's) and should be treated as weak priors.
Intended use
- Initialising simulation-in-the-loop refinement (system identification) or digital twins in MPM-style simulators: take the predicted class and parameters as the starting point, and the predicted σ as the initial search scale or prior width.
- Research on physical-property inference from video.
Not intended for safety-critical material certification, or for real-world SI values without calibration (see Limitations).
How to use
pip install -r requirements.txt
# sample pack: npz with rgb [V=3, T>=64, 3, H, W] uint8 (as in the IdentiDeform dataset)
python inference.py --pack sample_pack.npz --timestamps times.json # or --fps 30
from huggingface_hub import snapshot_download
import sys; root = snapshot_download("HandsomeHusky/IdentiDeform-prior"); sys.path.insert(0, root)
from inference import IdentiDeformPrior
prior = IdentiDeformPrior(root, device="cuda") # also downloads facebook/vjepa2-vitl-fpc64-256
out = prior.predict(rgb, times) # rgb: [3, T, 3, H, W] uint8, times: T seconds
out["class"], out["class_votes"] # e.g. 'metal', {'jelly': 0, 'metal': 8, 'plasticine': 1}
out["parameters"]["E_over_rho"] # {'value': ..., 'interval_1sigma': [lo, hi]}
out["z_mean"], out["z_std_total"] # standardized mean and total std (E/ρ, ν, ρ, σ_y/ρ)
What the code does, step by step (the same pipeline as in training):
| Step | Detail |
|---|---|
| Input | 3 synchronised views, 64 frames uniformly sub-sampled per view, 224×224 px, ImageNet normalisation |
| Encoder | frozen V-JEPA 2 ViT-L (facebook/vjepa2-vitl-fpc64-256, tubelet 2 → 32 × 14×14 × 1024 tokens per view), bf16 on GPU |
| Heads | 9 × fieldbridge.CachedFieldModel (variants B/C/D × seeds 17/29/43): view–time transformer, visual-field bridge, Gaussian parameter head, class head |
| Ensemble | mean of standardized predictions; class = majority vote (ties → lowest id) |
| Output | de-standardized as expm1(z·std + mean) (ν: linear) with the training statistics in stats.json |
| Uncertainty | σ_total = sqrt(mean over heads of exp(logvar) + variance across heads), in z space |
E/ρ and σ_y/ρ are reported as E/(ρ/1000) and σ_y/(ρ/1000), i.e. relative to the density of water.
--calibrate multiplies σ by the per-target temperatures fitted on validation (0.88–1.00, see ensemble.json).
Files
| File | Content |
|---|---|
heads/v2main_{B,C,D}_seed{17,29,43}.safetensors |
head weights (fp32 state dicts, 11.4 MB each; optimizer state removed) |
configs/*.json |
full training configs (paths removed; the backbone points to the Hub id) |
stats.json |
train-split standardization statistics (mean/std of the transformed targets; manifest SHA-256) |
classes.json, ensemble.json |
class list; ensemble rule, members, calibration temperatures and coverage |
inference.py, fieldbridge/ |
inference script and the minimal model code from the project repository |
V-JEPA 2 weights are not included in this repository. They are downloaded from facebook/vjepa2-vitl-fpc64-256 at load time.
Training data
IdentiDeform V2 (simulated, MPM): 87 / 10 / 10 objects (train / val / test, no object shared between splits); 1,955 / 283 / 300 simulations; 6 actions (press, drop, bend, shear, stretch, tap-and-hold); 3 constitutive classes; 3 rendered views, 128 frames per view. Targets are standardized on the train split only. The dataset release is coming at huggingface.co/HandsomeHusky.
Training: frozen V-JEPA 2 features, AdamW (lr 3e-4, wd 0.05), bf16, batch 8, smooth-L1 + Gaussian NLL (weight 0.05) + class cross-entropy (weight 0.1), early stopping on validation standardized MAE. Variants: B = no field supervision; C = field supervision with fields fed back into the head; D = field supervision as an auxiliary task only.
Evaluation
Standardized MAE = mean absolute error in train-standardized z units, pooled over the 4 targets (σ_y only where it is defined). Within-class r = Pearson correlation inside each class, which removes the gain from simply recognising the class.
| Method | val MAE ↓ | test MAE ↓ | val class acc. | test class acc. | test within-class r, E/ρ | test within-class r, σ_y/ρ |
|---|---|---|---|---|---|---|
| This ensemble (9 heads) | 0.616 | 0.646 | 0.700 | 0.743 | 0.629 | 0.793 |
| Single head (variant C, 3 seeds) | 0.644 | 0.676 ± 0.012 | 0.661 | 0.718 | 0.608 | 0.754 |
| GT-class train mean (privileged) | 0.786 | 0.792 | – | – | – | – |
| Constant (train mean) | 0.858 | 0.853 | 0.329 | 0.333 | – | – |
Per-target MAE of the ensemble (val / test): E/ρ 0.357 / 0.387, σ_y/ρ 0.341 / 0.431, ν 0.861 / 0.841, ρ 0.806 / 0.848. The two identifiable targets are learned. ν and ρ are not.
Uncertainty calibration (σ_total, raw): 1σ / 2σ coverage on test is 0.76 / 0.96 for E/ρ and 0.69 / 0.93 for σ_y/ρ (the nominal values are 0.68 / 0.95). The across-head spread on its own is strongly over-confident, so use σ_total.
Simulation-in-the-loop refinement (compass search over log E/ρ and log σ_y/ρ against the observed 3-view video; identifiable z-MAE after k simulations; val: 60 samples / 10 objects, test: 30 samples / 9 objects):
| Initialisation | val k=0 | k=4 | k=12 | k=24 | test k=0 | k=4 | k=12 | k=24 |
|---|---|---|---|---|---|---|---|---|
| This prior, top-1 class | 0.344 | 0.327 | 0.283 | 0.236 | 0.496 | 0.573 | 0.430 | 0.298 |
| GT-class mean (privileged) | 0.674 | 0.577 | 0.339 | 0.257 | 0.725 | 0.693 | 0.430 | 0.277 |
| Best of 3 class means (non-privileged) | 0.831 (k=3) | 0.771 | 0.536 | 0.482 | 0.830 (k=3) | 0.859 | 0.634 | 0.515 |
- The prior with 0 simulations (0.344 val / 0.496 test) is already better than non-privileged uninformed fitting after 24 simulations (0.482 / 0.515).
- The privileged GT-class-mean initialisation needs about 9.7 simulations (test) to reach the prior's 0-simulation error. It is matched again around k = 12. At k = 24 the two are statistically indistinguishable (test difference +0.013, 95% CI [−0.027, +0.080]).
Limitations
- Simulation-only training. All training videos are rendered from one MPM engine with fixed appearance, lighting and camera setup.
- Zero-shot transfer to other datasets or engines is unreliable. Parameter semantics and scales depend on the engine (units, hardening model, damping). For a new domain, use the prior to initialise refinement in the target simulator, or calibrate it with a few simulated samples from that domain.
- Refinement floor. Refinement stops improving at an identifiable error of about 0.2–0.3, because of model mismatch (fixed hardening, ν taken from the prediction). Initialisations already near that floor gain little.
- ρ and ν are poorly identifiable (see above). σ_y/ρ is meaningless for the elastic class.
- The class accuracy (0.70–0.74) is the main source of remaining error. Votes split close to 5/4 indicate ambiguity.
- Small test set (10 objects), so the confidence intervals are wide.
- Numerical precision: the training features were computed with the backbone in bf16 on GPU. Running V-JEPA 2 in fp32, or on CPU, changes the ensemble mean by up to about 0.05 z, and can flip near-tied class votes.
License
The heads, configs and code in this repository are released under the MIT license.
The V-JEPA 2 backbone (facebook/vjepa2-vitl-fpc64-256, not redistributed here) is released by Meta under its
own license. Its Hub page currently lists MIT. Check that page before use.
Code
Training and evaluation code: github.com/3365538768/IdentiDeform
Citation
@inproceedings{identideform2026,
title = {IdentiDeform},
author = {Anonymous},
booktitle = {Under review},
year = {2026}
}
Model tree for HandsomeHusky/IdentiDeform-prior
Base model
facebook/vjepa2-vitl-fpc64-256