IdentiDeform-prior

A video prior for deformable-object material identification. From a short 3-view video of an object being pressed, dropped, bent, sheared, stretched or tapped, it predicts

  • the constitutive class: jelly (elastic), metal (elasto-plastic), plasticine;
  • the material parameters in the units of the simulator it was trained on: specific stiffness E/ρ, Poisson ratio ν, density ρ and specific yield stress σ_y/ρ (σ_y is undefined for jelly);
  • a predictive uncertainty for each parameter.

It is a 9-member ensemble of small heads (2.85 M parameters each) on top of a frozen V-JEPA 2 ViT-L video encoder. The heads are trained on simulated data only.

Refinement from the video prior vs. uninformed initialisations

(a, b) Identifiable-parameter error vs. number of simulator calls during simulation-in-the-loop refinement, on validation and test. Starting from this prior with 0 simulations gives the error that an initialisation from the privileged ground-truth-class mean only reaches after about 10–12 simulations. (c) A downstream gripper task from the accompanying paper.

Only E/ρ and σ_y/ρ are identifiable from video. In MPM simulation the motion stays the same when E, σ_y and ρ are scaled together, and ν has only a weak effect on it. ρ and ν are still predicted, but they stay close to the training mean (their errors are about as large as the constant baseline's) and should be treated as weak priors.

Intended use

  • Initialising simulation-in-the-loop refinement (system identification) or digital twins in MPM-style simulators: take the predicted class and parameters as the starting point, and the predicted σ as the initial search scale or prior width.
  • Research on physical-property inference from video.

Not intended for safety-critical material certification, or for real-world SI values without calibration (see Limitations).

How to use

pip install -r requirements.txt
# sample pack: npz with rgb [V=3, T>=64, 3, H, W] uint8 (as in the IdentiDeform dataset)
python inference.py --pack sample_pack.npz --timestamps times.json   # or --fps 30
from huggingface_hub import snapshot_download
import sys; root = snapshot_download("HandsomeHusky/IdentiDeform-prior"); sys.path.insert(0, root)
from inference import IdentiDeformPrior

prior = IdentiDeformPrior(root, device="cuda")      # also downloads facebook/vjepa2-vitl-fpc64-256
out = prior.predict(rgb, times)                      # rgb: [3, T, 3, H, W] uint8, times: T seconds
out["class"], out["class_votes"]                     # e.g. 'metal', {'jelly': 0, 'metal': 8, 'plasticine': 1}
out["parameters"]["E_over_rho"]                      # {'value': ..., 'interval_1sigma': [lo, hi]}
out["z_mean"], out["z_std_total"]                    # standardized mean and total std (E/ρ, ν, ρ, σ_y/ρ)

What the code does, step by step (the same pipeline as in training):

Step Detail
Input 3 synchronised views, 64 frames uniformly sub-sampled per view, 224×224 px, ImageNet normalisation
Encoder frozen V-JEPA 2 ViT-L (facebook/vjepa2-vitl-fpc64-256, tubelet 2 → 32 × 14×14 × 1024 tokens per view), bf16 on GPU
Heads 9 × fieldbridge.CachedFieldModel (variants B/C/D × seeds 17/29/43): view–time transformer, visual-field bridge, Gaussian parameter head, class head
Ensemble mean of standardized predictions; class = majority vote (ties → lowest id)
Output de-standardized as expm1(z·std + mean) (ν: linear) with the training statistics in stats.json
Uncertainty σ_total = sqrt(mean over heads of exp(logvar) + variance across heads), in z space

E/ρ and σ_y/ρ are reported as E/(ρ/1000) and σ_y/(ρ/1000), i.e. relative to the density of water. --calibrate multiplies σ by the per-target temperatures fitted on validation (0.88–1.00, see ensemble.json).

Files

File Content
heads/v2main_{B,C,D}_seed{17,29,43}.safetensors head weights (fp32 state dicts, 11.4 MB each; optimizer state removed)
configs/*.json full training configs (paths removed; the backbone points to the Hub id)
stats.json train-split standardization statistics (mean/std of the transformed targets; manifest SHA-256)
classes.json, ensemble.json class list; ensemble rule, members, calibration temperatures and coverage
inference.py, fieldbridge/ inference script and the minimal model code from the project repository

V-JEPA 2 weights are not included in this repository. They are downloaded from facebook/vjepa2-vitl-fpc64-256 at load time.

Training data

IdentiDeform V2 (simulated, MPM): 87 / 10 / 10 objects (train / val / test, no object shared between splits); 1,955 / 283 / 300 simulations; 6 actions (press, drop, bend, shear, stretch, tap-and-hold); 3 constitutive classes; 3 rendered views, 128 frames per view. Targets are standardized on the train split only. The dataset release is coming at huggingface.co/HandsomeHusky.

Training: frozen V-JEPA 2 features, AdamW (lr 3e-4, wd 0.05), bf16, batch 8, smooth-L1 + Gaussian NLL (weight 0.05) + class cross-entropy (weight 0.1), early stopping on validation standardized MAE. Variants: B = no field supervision; C = field supervision with fields fed back into the head; D = field supervision as an auxiliary task only.

Evaluation

Standardized MAE = mean absolute error in train-standardized z units, pooled over the 4 targets (σ_y only where it is defined). Within-class r = Pearson correlation inside each class, which removes the gain from simply recognising the class.

Method val MAE ↓ test MAE ↓ val class acc. test class acc. test within-class r, E/ρ test within-class r, σ_y/ρ
This ensemble (9 heads) 0.616 0.646 0.700 0.743 0.629 0.793
Single head (variant C, 3 seeds) 0.644 0.676 ± 0.012 0.661 0.718 0.608 0.754
GT-class train mean (privileged) 0.786 0.792 – – – –
Constant (train mean) 0.858 0.853 0.329 0.333 – –

Per-target MAE of the ensemble (val / test): E/ρ 0.357 / 0.387, σ_y/ρ 0.341 / 0.431, ν 0.861 / 0.841, ρ 0.806 / 0.848. The two identifiable targets are learned. ν and ρ are not.

Uncertainty calibration (σ_total, raw): 1σ / 2σ coverage on test is 0.76 / 0.96 for E/ρ and 0.69 / 0.93 for σ_y/ρ (the nominal values are 0.68 / 0.95). The across-head spread on its own is strongly over-confident, so use σ_total.

Simulation-in-the-loop refinement (compass search over log E/ρ and log σ_y/ρ against the observed 3-view video; identifiable z-MAE after k simulations; val: 60 samples / 10 objects, test: 30 samples / 9 objects):

Initialisation val k=0 k=4 k=12 k=24 test k=0 k=4 k=12 k=24
This prior, top-1 class 0.344 0.327 0.283 0.236 0.496 0.573 0.430 0.298
GT-class mean (privileged) 0.674 0.577 0.339 0.257 0.725 0.693 0.430 0.277
Best of 3 class means (non-privileged) 0.831 (k=3) 0.771 0.536 0.482 0.830 (k=3) 0.859 0.634 0.515
  • The prior with 0 simulations (0.344 val / 0.496 test) is already better than non-privileged uninformed fitting after 24 simulations (0.482 / 0.515).
  • The privileged GT-class-mean initialisation needs about 9.7 simulations (test) to reach the prior's 0-simulation error. It is matched again around k = 12. At k = 24 the two are statistically indistinguishable (test difference +0.013, 95% CI [−0.027, +0.080]).

Limitations

  • Simulation-only training. All training videos are rendered from one MPM engine with fixed appearance, lighting and camera setup.
  • Zero-shot transfer to other datasets or engines is unreliable. Parameter semantics and scales depend on the engine (units, hardening model, damping). For a new domain, use the prior to initialise refinement in the target simulator, or calibrate it with a few simulated samples from that domain.
  • Refinement floor. Refinement stops improving at an identifiable error of about 0.2–0.3, because of model mismatch (fixed hardening, ν taken from the prediction). Initialisations already near that floor gain little.
  • ρ and ν are poorly identifiable (see above). σ_y/ρ is meaningless for the elastic class.
  • The class accuracy (0.70–0.74) is the main source of remaining error. Votes split close to 5/4 indicate ambiguity.
  • Small test set (10 objects), so the confidence intervals are wide.
  • Numerical precision: the training features were computed with the backbone in bf16 on GPU. Running V-JEPA 2 in fp32, or on CPU, changes the ensemble mean by up to about 0.05 z, and can flip near-tied class votes.

License

The heads, configs and code in this repository are released under the MIT license. The V-JEPA 2 backbone (facebook/vjepa2-vitl-fpc64-256, not redistributed here) is released by Meta under its own license. Its Hub page currently lists MIT. Check that page before use.

Code

Training and evaluation code: github.com/3365538768/IdentiDeform

Citation

@inproceedings{identideform2026,
  title     = {IdentiDeform},
  author    = {Anonymous},
  booktitle = {Under review},
  year      = {2026}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for HandsomeHusky/IdentiDeform-prior

Finetuned
(14)
this model