---
license: mit
library_name: pytorch
tags:
- world-models
- spectral-learning
- operator-learning
- multimodal
- wavelets
- pytorch
- research
---

# Spectral World Models — small runnable prototype

This is a collaborator-oriented Hugging Face prototype of **Spectral World Models: Multimodal Operator Learning in Hilbert Space for Imagination-Based Planning** by Andrew J. Kiruluta.

The implementation is grounded in the accompanying manuscript and source repository. It uses a fixed 2-level Haar image encoder, DCT-style spectral text encoder, low-rank multimodal fusion, an action-conditioned low-rank spectral transition operator, modality decoders, and recursive imagination rollouts.

## First benchmark

The included **Mini Moving Shapes** dataset is a deliberately small multimodal smoke benchmark: 32×32 grayscale moving shapes, symbolic text descriptions, and five discrete actions. The repository compares SWM with Dreamer-style, transformer, Koopman, and neural-operator baselines and includes architecture ablations.

The checked-in results are actual quick CPU measurements from the source repository, not the hypothetical tables in the manuscript. They are intended to validate the pipeline, not establish superiority.

| Method | Params | Test PSNR ↑ | Test SSIM ↑ | Test NLL ↓ | Rollout MSE@5 ↓ |
|---|---:|---:|---:|---:|---:|
| Dreamer-style | 497,682 | 14.571 | 0.072 | 0.0107 | 0.03490 |
| Transformer | 503,322 | 14.368 | 0.017 | 0.0687 | 0.03657 |
| Koopman | 465,306 | 14.619 | 0.090 | -0.0027 | 0.03487 |
| Neural operator | 403,794 | 14.534 | 0.058 | 0.0210 | 0.03511 |
| SWM | 405,422 | 14.591 | 0.080 | 0.0050 | 0.03577 |

These smoke results are mixed: SWM is competitive on some one-step metrics, while Koopman has lower rollout MSE@5 in this short run. That is precisely why larger multi-seed experiments are needed.

## Run

`python -m venv .venv`

`source .venv/bin/activate`

`pip install -e .`

`PYTHONPATH=. python scripts/run_quick_benchmark.py`

Tests: `PYTHONPATH=. python -m pytest -q`

See `COLLABORATORS.md` for the GPU scaling and experimental roadmap.