File size: 2,111 Bytes
4fd79a1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | ---
license: mit
library_name: pytorch
tags:
- world-models
- spectral-learning
- operator-learning
- multimodal
- wavelets
- pytorch
- research
---
# Spectral World Models — small runnable prototype
This is a collaborator-oriented Hugging Face prototype of **Spectral World Models: Multimodal Operator Learning in Hilbert Space for Imagination-Based Planning** by Andrew J. Kiruluta.
The implementation is grounded in the accompanying manuscript and source repository. It uses a fixed 2-level Haar image encoder, DCT-style spectral text encoder, low-rank multimodal fusion, an action-conditioned low-rank spectral transition operator, modality decoders, and recursive imagination rollouts.
## First benchmark
The included **Mini Moving Shapes** dataset is a deliberately small multimodal smoke benchmark: 32×32 grayscale moving shapes, symbolic text descriptions, and five discrete actions. The repository compares SWM with Dreamer-style, transformer, Koopman, and neural-operator baselines and includes architecture ablations.
The checked-in results are actual quick CPU measurements from the source repository, not the hypothetical tables in the manuscript. They are intended to validate the pipeline, not establish superiority.
| Method | Params | Test PSNR ↑ | Test SSIM ↑ | Test NLL ↓ | Rollout MSE@5 ↓ |
|---|---:|---:|---:|---:|---:|
| Dreamer-style | 497,682 | 14.571 | 0.072 | 0.0107 | 0.03490 |
| Transformer | 503,322 | 14.368 | 0.017 | 0.0687 | 0.03657 |
| Koopman | 465,306 | 14.619 | 0.090 | -0.0027 | 0.03487 |
| Neural operator | 403,794 | 14.534 | 0.058 | 0.0210 | 0.03511 |
| SWM | 405,422 | 14.591 | 0.080 | 0.0050 | 0.03577 |
These smoke results are mixed: SWM is competitive on some one-step metrics, while Koopman has lower rollout MSE@5 in this short run. That is precisely why larger multi-seed experiments are needed.
## Run
`python -m venv .venv`
`source .venv/bin/activate`
`pip install -e .`
`PYTHONPATH=. python scripts/run_quick_benchmark.py`
Tests: `PYTHONPATH=. python -m pytest -q`
See `COLLABORATORS.md` for the GPU scaling and experimental roadmap.
|