File size: 8,190 Bytes
0f71828 2a9189f 0f71828 2a9189f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 | ---
license: apache-2.0
language:
- en
- zh
tags:
- eeg
- video-generation
- world-model
- videorae3d
- safetensors
---
# IWM: VideoRAE3D + EEG-conditioned action models
Five selected **inference-only checkpoints** from Intention World Model.
The visual representation is VideoRAE3D. Our learned IDM/FDM teacher and EEG
controllers are provided here; third-party visual encoder/decoder weights are
not redistributed.
| Model ID | Selected step | Purpose |
| --- | ---: | --- |
| `teacher-best87600` | 87,600 | Shared frozen IDM + FDM, selected from training completed at 160,000 steps |
| `cg2-ng-seed26102751` | 800 | Reference CG2-NG controller, seed 26102751 |
| `cg2-ng-seed26102752` | 1,000 | Reference CG2-NG controller, seed 26102752 |
| `ng-d-seed26102751` | 2,400 | Exploratory independent motion-reader refinement, seed 26102751 |
| `ng-d-seed26102752` | 2,400 | Exploratory independent motion-reader refinement, seed 26102752 |
Controller steps refer to their own response-training stage, not teacher steps.
NG-D training completed at total 3,000; these are the selected **best 2,400**
checkpoints. Both seeds are included, rather than only the better-scoring seed.
## Model and input contract
- Visual latent: `[B,1024,64]`, spatial-temporal grid `4 x 16 x 16`.
- IDM: width 512, depth 6, 8 heads, continuous `[B,1,256]` action.
- FDM: width 512, depth 6, 8 heads, direct next-latent prediction. No output
residual, transport, LoRA, pixel enhancement or post-hoc action gain.
- EEG: `[B,32,500]`, 200 Hz, 2.5 seconds; four 512-wide layers, 20 ordered
time tokens read by a visual-conditioned query. Predicted classes are
`Drink`, `Eat`, `Pen`, `Digital`, `Manipulation`, in that order.
- CG2-NG combines a frozen visual prior, EEG residual, predicted-class response
and learned synthetic-noise fallback gate. NG-D freezes the original CG2-NG
and adds an independent target-EEG motion reader and bounded correction.
**This release uses target-period EEG conditioning, not past-only forecasting.**
Observed video is `[-2,0)`, EEG is `[-0.5,2)`, and the output is H1 `[0,2)`:
16 native frames at 8 fps, 256 x 256. No future video or true class is supplied
to the EEG controller. Oracle IDM inference does use the true future latent.
The channel order is recorded in every controller's `config.json`. Inputs must
match the original EEG preprocessing, physical scale and ordering. These are
preprocessed EEG values **before** the model's TRAIN-fit normalization; the
included mean/std and clipping are applied inside the model. Do not normalize
twice or pass arbitrary device units. The inherited cache uses EDF physical
microvolts, a causal 0.1–75 Hz fourth-order bandpass and 50 Hz notch at 256 Hz,
past-sample selection to 200 Hz, then float32 values divided by 100; see the
[preprocessing arithmetic](https://github.com/fly0520/intention-world-model/blob/516a71c252ff8f895dcd036bb343c996254c7a76/scripts/audit_conditional_probe_eeg.py).
All-zero EEG means raw zero before
normalization and intentionally returns exactly the visual prior. This fallback
is an explicit interface design, not evidence of learned physiological validity.
## Loading
Use the matching source archive at commit
[`516a71c252ff8f895dcd036bb343c996254c7a76`](https://github.com/fly0520/intention-world-model/tree/516a71c252ff8f895dcd036bb343c996254c7a76).
The complete repository is needed, not only an individual experiment folder.
The loader requires PyTorch, NumPy and safetensors. Export verification used
PyTorch 2.3.0 and safetensors 0.4.3 on CPU. Downloading through the Hub additionally
requires `huggingface_hub`. This is not a Transformers `AutoModel` checkpoint.
```bash
git clone --branch debug-3dvrae https://github.com/fly0520/intention-world-model.git
cd intention-world-model
git checkout 516a71c252ff8f895dcd036bb343c996254c7a76
```
```python
from pathlib import Path
import sys
import torch
from huggingface_hub import snapshot_download
release = Path(snapshot_download("SnoopyFan/IWM"))
source = Path("/your/checkout/intention-world-model")
sys.path.insert(0, str(release))
from load_models import load_checkpoint
device = "cuda" if torch.cuda.is_available() else "cpu"
teacher = load_checkpoint(release, "teacher-best87600", source, device)
controller = load_checkpoint(release, "cg2-ng-seed26102751", source, device)
# Or: "ng-d-seed26102751"; both second-seed versions are also available.
# z_current: native VideoRAE3D latent [B,1024,64]
# eeg_raw: correctly ordered/preprocessed target-period EEG [B,32,500]
with torch.inference_mode():
logits, action = controller(z_current.to(device), eeg_raw.to(device))
z_h1 = teacher.fdm(z_current.to(device), action)
z_visual = teacher.fdm(z_current.to(device), controller.visual(z_current.to(device)))
```
For RGB encoding/decoding obtain the compatible released
`AutoEncoderVJepa21_3D_C64Repa` VideoRAE weights and V-JEPA2.1 encoder separately.
See the pinned [visual integration](https://github.com/fly0520/intention-world-model/blob/516a71c252ff8f895dcd036bb343c996254c7a76/experiments/videorae3d/teacher/visual.py)
and [runtime contract](https://github.com/fly0520/intention-world-model/blob/516a71c252ff8f895dcd036bb343c996254c7a76/experiments/videorae3d/README.md).
Follow their licenses and preprocessing; visual weights are not part of this
repository. The example above starts at the latent/EEG interface and does not
download or contain any EEG/video data.
## Results and intended use
CG2-NG remains the reference demonstration model. Its two-seed subject-balanced
Oracle motion-consistency J is **0.256243**, versus **0.232376** for the shared
visual/zero baseline on the repeated development set (148 cases, 24 subjects).
NG-D best2400 reaches **0.258135** on the same aggregate, essentially unchanged
from its shorter-training parent; it is an exploratory alternative, not a
confirmed general upgrade. Selected examples show local forearm/wrist responses,
not uniformly correct full-action reconstruction. J is not a visible-success rate.
Full comparisons, Gaussian/PSD controls and seed-specific results:
- [CG2-NG protocol and results](https://github.com/fly0520/intention-world-model/blob/516a71c252ff8f895dcd036bb343c996254c7a76/docs/experiments/videorae3d-cg2-ng-20261029.md)
- [Recent variants and continuation results](https://github.com/fly0520/intention-world-model/blob/516a71c252ff8f895dcd036bb343c996254c7a76/docs/experiments/videorae3d-cg2-dualcontext-20261110.md)
The teacher was selected on repeated full-150 development evaluation; these EEG
controllers were selected on TRAIN-inner. The 148-case EEG development set is
used repeatedly for reporting and demonstration selection, not an untouched
confirmation set. Subjects/phases are shared. Do not interpret these models as
cross-person generalization, causal intention reading, medical diagnostics or
proof that synthetic-noise discrimination alone establishes correct motion.
## Packaging, provenance and privacy
`manifest.json` gives the original checkpoint SHA, exported tensor/file SHA,
selected step, exact removed-buffer list and export checks for each model.
The safetensors retain original FP32 values without quantization. The loader
also restores archived dense buffer layouts to preserve the original matrix
multiplication path. Required
aggregate normalization, learned visual prior and class-expert parameters are
included. Raw EEG/video, individual features/actions, participant/sample IDs,
relation memories, optimizer/RNG/sampler state, logs and private paths are not.
The loader deletes unused training-only buffers before strict loading.
This is an inference release, **not an optimizer-resume training checkpoint**.
CPU synthetic checks verify exact tensor round-trips, identical original/exported
class/action outputs, teacher IDM/FDM outputs and exact raw-zero visual fallback.
They validate packaging, not new scientific performance or cross-device bitwise
equivalence. All original private checkpoints are retained unchanged.
The existing repository Apache-2.0 license metadata is preserved. It does not
relicense third-party VideoRAE/V-JEPA code, weights or source datasets.
|