| --- |
| license: mit |
| library_name: pytorch |
| tags: |
| - world-models |
| - spectral-learning |
| - operator-learning |
| - multimodal |
| - wavelets |
| - pytorch |
| - research |
| --- |
| |
| # Spectral World Models — small runnable prototype |
|
|
| This is a collaborator-oriented Hugging Face prototype of **Spectral World Models: Multimodal Operator Learning in Hilbert Space for Imagination-Based Planning** by Andrew J. Kiruluta. |
|
|
| The implementation is grounded in the accompanying manuscript and source repository. It uses a fixed 2-level Haar image encoder, DCT-style spectral text encoder, low-rank multimodal fusion, an action-conditioned low-rank spectral transition operator, modality decoders, and recursive imagination rollouts. |
|
|
| ## First benchmark |
|
|
| The included **Mini Moving Shapes** dataset is a deliberately small multimodal smoke benchmark: 32×32 grayscale moving shapes, symbolic text descriptions, and five discrete actions. The repository compares SWM with Dreamer-style, transformer, Koopman, and neural-operator baselines and includes architecture ablations. |
|
|
| The checked-in results are actual quick CPU measurements from the source repository, not the hypothetical tables in the manuscript. They are intended to validate the pipeline, not establish superiority. |
|
|
| | Method | Params | Test PSNR ↑ | Test SSIM ↑ | Test NLL ↓ | Rollout MSE@5 ↓ | |
| |---|---:|---:|---:|---:|---:| |
| | Dreamer-style | 497,682 | 14.571 | 0.072 | 0.0107 | 0.03490 | |
| | Transformer | 503,322 | 14.368 | 0.017 | 0.0687 | 0.03657 | |
| | Koopman | 465,306 | 14.619 | 0.090 | -0.0027 | 0.03487 | |
| | Neural operator | 403,794 | 14.534 | 0.058 | 0.0210 | 0.03511 | |
| | SWM | 405,422 | 14.591 | 0.080 | 0.0050 | 0.03577 | |
|
|
| These smoke results are mixed: SWM is competitive on some one-step metrics, while Koopman has lower rollout MSE@5 in this short run. That is precisely why larger multi-seed experiments are needed. |
|
|
| ## Run |
|
|
| `python -m venv .venv` |
|
|
| `source .venv/bin/activate` |
|
|
| `pip install -e .` |
|
|
| `PYTHONPATH=. python scripts/run_quick_benchmark.py` |
|
|
| Tests: `PYTHONPATH=. python -m pytest -q` |
|
|
| See `COLLABORATORS.md` for the GPU scaling and experimental roadmap. |
|
|
