|
Download README.md from SnoopyFan/IWM: direct link, hf CLI and curl.
- Browser
- Download file 8.19 kB
-
https://huggingface.co/SnoopyFan/IWM/resolve/main/README.md
- Command line
-
hf download hf://SnoopyFan/IWM/README.md
-
curl -L -o README.md https://huggingface.co/SnoopyFan/IWM/resolve/main/README.md
8.19 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| - zh | |
| tags: | |
| - eeg | |
| - video-generation | |
| - world-model | |
| - videorae3d | |
| - safetensors | |
| # IWM: VideoRAE3D + EEG-conditioned action models | |
| Five selected **inference-only checkpoints** from Intention World Model. | |
| The visual representation is VideoRAE3D. Our learned IDM/FDM teacher and EEG | |
| controllers are provided here; third-party visual encoder/decoder weights are | |
| not redistributed. | |
| | Model ID | Selected step | Purpose | | |
| | --- | ---: | --- | | |
| | `teacher-best87600` | 87,600 | Shared frozen IDM + FDM, selected from training completed at 160,000 steps | | |
| | `cg2-ng-seed26102751` | 800 | Reference CG2-NG controller, seed 26102751 | | |
| | `cg2-ng-seed26102752` | 1,000 | Reference CG2-NG controller, seed 26102752 | | |
| | `ng-d-seed26102751` | 2,400 | Exploratory independent motion-reader refinement, seed 26102751 | | |
| | `ng-d-seed26102752` | 2,400 | Exploratory independent motion-reader refinement, seed 26102752 | | |
| Controller steps refer to their own response-training stage, not teacher steps. | |
| NG-D training completed at total 3,000; these are the selected **best 2,400** | |
| checkpoints. Both seeds are included, rather than only the better-scoring seed. | |
| ## Model and input contract | |
| - Visual latent: `[B,1024,64]`, spatial-temporal grid `4 x 16 x 16`. | |
| - IDM: width 512, depth 6, 8 heads, continuous `[B,1,256]` action. | |
| - FDM: width 512, depth 6, 8 heads, direct next-latent prediction. No output | |
| residual, transport, LoRA, pixel enhancement or post-hoc action gain. | |
| - EEG: `[B,32,500]`, 200 Hz, 2.5 seconds; four 512-wide layers, 20 ordered | |
| time tokens read by a visual-conditioned query. Predicted classes are | |
| `Drink`, `Eat`, `Pen`, `Digital`, `Manipulation`, in that order. | |
| - CG2-NG combines a frozen visual prior, EEG residual, predicted-class response | |
| and learned synthetic-noise fallback gate. NG-D freezes the original CG2-NG | |
| and adds an independent target-EEG motion reader and bounded correction. | |
| **This release uses target-period EEG conditioning, not past-only forecasting.** | |
| Observed video is `[-2,0)`, EEG is `[-0.5,2)`, and the output is H1 `[0,2)`: | |
| 16 native frames at 8 fps, 256 x 256. No future video or true class is supplied | |
| to the EEG controller. Oracle IDM inference does use the true future latent. | |
| The channel order is recorded in every controller's `config.json`. Inputs must | |
| match the original EEG preprocessing, physical scale and ordering. These are | |
| preprocessed EEG values **before** the model's TRAIN-fit normalization; the | |
| included mean/std and clipping are applied inside the model. Do not normalize | |
| twice or pass arbitrary device units. The inherited cache uses EDF physical | |
| microvolts, a causal 0.1–75 Hz fourth-order bandpass and 50 Hz notch at 256 Hz, | |
| past-sample selection to 200 Hz, then float32 values divided by 100; see the | |
| [preprocessing arithmetic](https://github.com/fly0520/intention-world-model/blob/516a71c252ff8f895dcd036bb343c996254c7a76/scripts/audit_conditional_probe_eeg.py). | |
| All-zero EEG means raw zero before | |
| normalization and intentionally returns exactly the visual prior. This fallback | |
| is an explicit interface design, not evidence of learned physiological validity. | |
| ## Loading | |
| Use the matching source archive at commit | |
| [`516a71c252ff8f895dcd036bb343c996254c7a76`](https://github.com/fly0520/intention-world-model/tree/516a71c252ff8f895dcd036bb343c996254c7a76). | |
| The complete repository is needed, not only an individual experiment folder. | |
| The loader requires PyTorch, NumPy and safetensors. Export verification used | |
| PyTorch 2.3.0 and safetensors 0.4.3 on CPU. Downloading through the Hub additionally | |
| requires `huggingface_hub`. This is not a Transformers `AutoModel` checkpoint. | |
| ```bash | |
| git clone --branch debug-3dvrae https://github.com/fly0520/intention-world-model.git | |
| cd intention-world-model | |
| git checkout 516a71c252ff8f895dcd036bb343c996254c7a76 | |
| ``` | |
| ```python | |
| from pathlib import Path | |
| import sys | |
| import torch | |
| from huggingface_hub import snapshot_download | |
| release = Path(snapshot_download("SnoopyFan/IWM")) | |
| source = Path("/your/checkout/intention-world-model") | |
| sys.path.insert(0, str(release)) | |
| from load_models import load_checkpoint | |
| device = "cuda" if torch.cuda.is_available() else "cpu" | |
| teacher = load_checkpoint(release, "teacher-best87600", source, device) | |
| controller = load_checkpoint(release, "cg2-ng-seed26102751", source, device) | |
| # Or: "ng-d-seed26102751"; both second-seed versions are also available. | |
| # z_current: native VideoRAE3D latent [B,1024,64] | |
| # eeg_raw: correctly ordered/preprocessed target-period EEG [B,32,500] | |
| with torch.inference_mode(): | |
| logits, action = controller(z_current.to(device), eeg_raw.to(device)) | |
| z_h1 = teacher.fdm(z_current.to(device), action) | |
| z_visual = teacher.fdm(z_current.to(device), controller.visual(z_current.to(device))) | |
| ``` | |
| For RGB encoding/decoding obtain the compatible released | |
| `AutoEncoderVJepa21_3D_C64Repa` VideoRAE weights and V-JEPA2.1 encoder separately. | |
| See the pinned [visual integration](https://github.com/fly0520/intention-world-model/blob/516a71c252ff8f895dcd036bb343c996254c7a76/experiments/videorae3d/teacher/visual.py) | |
| and [runtime contract](https://github.com/fly0520/intention-world-model/blob/516a71c252ff8f895dcd036bb343c996254c7a76/experiments/videorae3d/README.md). | |
| Follow their licenses and preprocessing; visual weights are not part of this | |
| repository. The example above starts at the latent/EEG interface and does not | |
| download or contain any EEG/video data. | |
| ## Results and intended use | |
| CG2-NG remains the reference demonstration model. Its two-seed subject-balanced | |
| Oracle motion-consistency J is **0.256243**, versus **0.232376** for the shared | |
| visual/zero baseline on the repeated development set (148 cases, 24 subjects). | |
| NG-D best2400 reaches **0.258135** on the same aggregate, essentially unchanged | |
| from its shorter-training parent; it is an exploratory alternative, not a | |
| confirmed general upgrade. Selected examples show local forearm/wrist responses, | |
| not uniformly correct full-action reconstruction. J is not a visible-success rate. | |
| Full comparisons, Gaussian/PSD controls and seed-specific results: | |
| - [CG2-NG protocol and results](https://github.com/fly0520/intention-world-model/blob/516a71c252ff8f895dcd036bb343c996254c7a76/docs/experiments/videorae3d-cg2-ng-20261029.md) | |
| - [Recent variants and continuation results](https://github.com/fly0520/intention-world-model/blob/516a71c252ff8f895dcd036bb343c996254c7a76/docs/experiments/videorae3d-cg2-dualcontext-20261110.md) | |
| The teacher was selected on repeated full-150 development evaluation; these EEG | |
| controllers were selected on TRAIN-inner. The 148-case EEG development set is | |
| used repeatedly for reporting and demonstration selection, not an untouched | |
| confirmation set. Subjects/phases are shared. Do not interpret these models as | |
| cross-person generalization, causal intention reading, medical diagnostics or | |
| proof that synthetic-noise discrimination alone establishes correct motion. | |
| ## Packaging, provenance and privacy | |
| `manifest.json` gives the original checkpoint SHA, exported tensor/file SHA, | |
| selected step, exact removed-buffer list and export checks for each model. | |
| The safetensors retain original FP32 values without quantization. The loader | |
| also restores archived dense buffer layouts to preserve the original matrix | |
| multiplication path. Required | |
| aggregate normalization, learned visual prior and class-expert parameters are | |
| included. Raw EEG/video, individual features/actions, participant/sample IDs, | |
| relation memories, optimizer/RNG/sampler state, logs and private paths are not. | |
| The loader deletes unused training-only buffers before strict loading. | |
| This is an inference release, **not an optimizer-resume training checkpoint**. | |
| CPU synthetic checks verify exact tensor round-trips, identical original/exported | |
| class/action outputs, teacher IDM/FDM outputs and exact raw-zero visual fallback. | |
| They validate packaging, not new scientific performance or cross-device bitwise | |
| equivalence. All original private checkpoints are retained unchanged. | |
| The existing repository Apache-2.0 license metadata is preserved. It does not | |
| relicense third-party VideoRAE/V-JEPA code, weights or source datasets. | |