diff --git a/COLLABORATORS.md b/COLLABORATORS.md new file mode 100644 index 0000000000000000000000000000000000000000..9c345e2f1573a5db50a733eb1f261a784bd54a45 --- /dev/null +++ b/COLLABORATORS.md @@ -0,0 +1,24 @@ +# Collaborator Roadmap + +This repository is the small, runnable reference implementation for the Spectral World Models (SWM) research program. + +## What is implemented now + +The prototype follows the manuscript's concrete architecture: a 2-level Haar image representation, a DCT-style spectral text representation, multimodal fusion, an action-conditioned low-rank spectral transition, image/text decoders, and recursive latent rollouts. The bundled Mini Moving Shapes benchmark exercises image + symbolic-text + action next-state prediction. + +## What we want collaborators to test + +1. **Replication first.** Re-run the included benchmark across multiple seeds and report confidence intervals rather than optimizing against the test set. +2. **Harder visual dynamics.** Replace Mini Moving Shapes with larger physical/video/control benchmarks while retaining matched baselines. +3. **Multimodal grounding.** Test whether language actually changes future-state predictions in instruction-conditioned environments. +4. **Long-horizon imagination.** Measure error-vs-horizon, latent norm growth, planning return, and rollout throughput. +5. **Scale and systems.** Optimize sparse/low-rank operator kernels on A100/H100/H200/B200-class GPUs and multi-GPU systems. +6. **Ablations and controls.** Preserve Dreamer/PlaNet-style, transformer, Koopman, and neural-operator controls, plus spectral-state/transition/text/stability ablations. + +## Evidence standard + +The manuscript explicitly states that its results tables are illustrative placeholders until real experiments are run. Do not copy those placeholder values into claims. The `results/` directory in this repo contains measured *quick smoke-benchmark* values only. Treat them as engineering validation, not paper-grade evidence. + +## Contributions + +Open an issue/discussion with hardware, software versions, dataset, exact command/config, seeds, wall time, peak memory, and result files. Pull requests for reproducibility, GPU kernels, distributed training, stronger baselines, datasets, and HF integration are welcome. diff --git a/HF_README.md b/HF_README.md new file mode 100644 index 0000000000000000000000000000000000000000..8ef7882db463d490e0fc2b7d02975e732793c13f --- /dev/null +++ b/HF_README.md @@ -0,0 +1,48 @@ +--- +license: mit +library_name: pytorch +tags: +- world-models +- spectral-learning +- operator-learning +- multimodal +- wavelets +- pytorch +- research +--- + +# Spectral World Models — small runnable prototype + +This is a collaborator-oriented Hugging Face prototype of **Spectral World Models: Multimodal Operator Learning in Hilbert Space for Imagination-Based Planning** by Andrew J. Kiruluta. + +The implementation is grounded in the accompanying manuscript and source repository. It uses a fixed 2-level Haar image encoder, DCT-style spectral text encoder, low-rank multimodal fusion, an action-conditioned low-rank spectral transition operator, modality decoders, and recursive imagination rollouts. + +## First benchmark + +The included **Mini Moving Shapes** dataset is a deliberately small multimodal smoke benchmark: 32×32 grayscale moving shapes, symbolic text descriptions, and five discrete actions. The repository compares SWM with Dreamer-style, transformer, Koopman, and neural-operator baselines and includes architecture ablations. + +The checked-in results are actual quick CPU measurements from the source repository, not the hypothetical tables in the manuscript. They are intended to validate the pipeline, not establish superiority. + +| Method | Params | Test PSNR ↑ | Test SSIM ↑ | Test NLL ↓ | Rollout MSE@5 ↓ | +|---|---:|---:|---:|---:|---:| +| Dreamer-style | 497,682 | 14.571 | 0.072 | 0.0107 | 0.03490 | +| Transformer | 503,322 | 14.368 | 0.017 | 0.0687 | 0.03657 | +| Koopman | 465,306 | 14.619 | 0.090 | -0.0027 | 0.03487 | +| Neural operator | 403,794 | 14.534 | 0.058 | 0.0210 | 0.03511 | +| SWM | 405,422 | 14.591 | 0.080 | 0.0050 | 0.03577 | + +These smoke results are mixed: SWM is competitive on some one-step metrics, while Koopman has lower rollout MSE@5 in this short run. That is precisely why larger multi-seed experiments are needed. + +## Run + +`python -m venv .venv` + +`source .venv/bin/activate` + +`pip install -e .` + +`PYTHONPATH=. python scripts/run_quick_benchmark.py` + +Tests: `PYTHONPATH=. python -m pytest -q` + +See `COLLABORATORS.md` for the GPU scaling and experimental roadmap. diff --git a/HUGGINGFACE_UPLOAD.md b/HUGGINGFACE_UPLOAD.md new file mode 100644 index 0000000000000000000000000000000000000000..a33756ef9c922771986c419c708f3582e1790069 --- /dev/null +++ b/HUGGINGFACE_UPLOAD.md @@ -0,0 +1,71 @@ +# Upload V14 to Hugging Face + +Suggested model/research repository: `kiruluta/Spectral-World-Models-External-Control` + +## 1. Unpack and enter the repository + +`unzip Spectral-World-Models-HF-v14.zip` + +`cd Spectral-World-Models-HF-v14` + +## 2. Create an environment + +`conda create -n swm-v14 python=3.11 -y` + +`conda activate swm-v14` + +`pip install -e '.[external]'` + +## 3. Validate before publishing + +`PYTHONPATH=. pytest -q` + +Optional external smoke run: + +`PYTHONPATH=. python scripts/run_v14_external_benchmark.py --envs CartPole-v1 --models swm_structured_cf_v9 no_spectral_transition --seeds 0 --horizons 5 10 --epochs 1 --train-episodes 16 --val-episodes 4 --test-episodes 4 --device cpu --out results/v14_smoke` + +## 4. Run the benchmark on DGX Spark + +`PYTHONPATH=. python scripts/run_v14_external_benchmark.py --seeds 0 1 2 3 4 5 6 7 8 9 --epochs 5 --device cuda` + +Review `results/v14_external_control/metrics_summary.csv` before publishing benchmark claims. + +## 5. Install and authenticate the Hugging Face CLI + +`pip install -U huggingface_hub` + +`hf auth login` + +Use a Hugging Face access token with write permission. Do not put the token in the repository. + +## 6. Create the Hub repository + +`hf repos create kiruluta/Spectral-World-Models-External-Control` + +The Hub creates model repositories by default. This project is a research/model repository rather than a Space. + +## 7. Upload the complete repository + +From the directory containing `Spectral-World-Models-HF-v14/`: + +`hf upload kiruluta/Spectral-World-Models-External-Control ./Spectral-World-Models-HF-v14 . --exclude '.pytest_cache/**' --exclude '**/__pycache__/**'` + +The current `hf upload` path handles large files through the Hub upload stack. Re-run the same command if an upload is interrupted. + +## 8. Verify the Hub page + +Confirm that the root contains `README.md`, `V14_EXTERNAL_CONTROL_BENCHMARK.md`, `pyproject.toml`, `spectral_world_models/`, `scripts/`, and `tests/`. Check that no access token, local cache, or private data was uploaded. + +## 9. Add trained V14 checkpoints after the full run + +After inspecting the ten-seed results, upload the generated checkpoint and metric directories: + +`hf upload kiruluta/Spectral-World-Models-External-Control ./results/v14_external_control results/v14_external_control` + +If you want checkpoints in the same repository: + +`hf upload kiruluta/Spectral-World-Models-External-Control ./results/v14_external_control/checkpoints checkpoints` + +## 10. Tag the milestone + +For a reproducible release, clone the Hub repo with Git, create a `v14.0` tag, and push the tag, or record the Hub commit hash in the manuscript. Do not update paper numbers from an unversioned moving `main` branch. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..e41b0ae0a99abb8f7d4278f3f1352d01c06bb5ba --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Andrew J. Kiruluta + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/LONG_HORIZON_BENCHMARK.md b/LONG_HORIZON_BENCHMARK.md new file mode 100644 index 0000000000000000000000000000000000000000..7dc9c421c888c6efb8a172ce177cdf8034d6c7b3 --- /dev/null +++ b/LONG_HORIZON_BENCHMARK.md @@ -0,0 +1,35 @@ +# Long-Horizon Multi-Seed Benchmark + +This repository is the integrated SWM prototype with multi-seed and long-horizon evaluation support. + +## Recommended DGX Spark run + +Install the project first: + + pip install -e . + +Run the five-seed benchmark on CUDA: + + PYTHONPATH=. python scripts/run_quick_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 --device cuda + +The runner keeps the original short-sequence training/validation/test benchmark unchanged. For rollout horizons longer than the original sequence length, it creates a separate held-out synthetic rollout dataset with enough time steps for the largest requested horizon. + +Primary aggregate output: + + results/multiseed_horizons/metrics_summary.csv + +Per-seed output: + + results/multiseed_horizons/metrics_by_seed.csv + +Run metadata: + + results/multiseed_horizons/run_config.json + +## Fairness + +For a given seed, every model receives the same seed. The long-horizon rollout dataset is held out from training. This is still a compact synthetic benchmark and should be treated as prototype evidence, not a paper-grade claim of superiority. + +## Quick smoke test + + PYTHONPATH=. python scripts/run_quick_benchmark.py --seeds 0 --horizons 5 10 --models swm koopman --epochs 1 --device auto diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..0a8023ada7de2823d38ed517246348f4936b7d44 --- /dev/null +++ b/README.md @@ -0,0 +1,98 @@ +# V13: Axis Symmetry & Intervention Equivariance Audit + +## V15.1 bug-fix release + +V15.1 replaces V15 after correcting the external state-token diagnostic. The original runner called the flat `text_decoder` layer directly; V15.1 uses the canonical `decode_text()` API and validates `[B,L,V]` logits against `[B,L]` discretized Gym-state token targets. No model architecture, checkpoint, dataset, training recipe, or scientific protocol changed. See `V15_1_BUGFIX.md`. + + +V13 freezes the V9 learning recipe and diagnoses the V12 vertical-action collapse with exact rotations/reflections/x-y swaps plus deliberately constructed collision episodes. See `V13_AXIS_SYMMETRY_EQUIVARIANCE.md`. + +Main run: `PYTHONPATH=. python scripts/run_v13_axis_audit.py --seeds 0 1 2 3 4 5 6 7 8 9 --pairs 64 --horizon 30 --device cuda` + +--- + +# V12: Causal Response Decomposition + +V12 freezes the V9 learning recipe and decomposes causal-response behavior by action program, horizon, effect size, initial speed, boundary contact, occlusion-zone exposure, and physics regime. It adds paired episode-clustered bootstrap confidence intervals against neural-operator and no-spectral-transition controls. See `V12_CAUSAL_RESPONSE_DECOMPOSITION.md`. + +Main run: `PYTHONPATH=. python scripts/run_v12_decomposition.py --seeds 0 1 2 3 4 5 6 7 8 9 --cf-horizon 30 --cf-pairs 64 --bootstrap 2000 --device cuda` + +--- + +--- + +## V9: causal generalization across physics regimes + +V9 freezes the structured spectral transport mechanism and targets the failure exposed by V8: excellent in-distribution counterfactual alignment did not extrapolate to changed physics. V9 trains on a family of dynamics regimes and adds trajectory-level counterfactual supervision, then evaluates interpolation, parameter extrapolation, and structural OOD shifts separately. See `V9_CAUSAL_GENERALIZATION.md`. + +Run: `PYTHONPATH=. python scripts/run_v9_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --cf-horizon 30 --planning-horizon 20 --planning-episodes 64 --device cuda` + +license: mit +library_name: pytorch +tags: +- world-models +- spectral-learning +- operator-learning +- multimodal +- wavelets +- pytorch +- research +--- + +# Spectral World Models — V7 + +V7 is the counterfactual-action training revision of the runnable Spectral World Models research prototype. + +## Why V7 + +V6 showed that structured transport can preserve long-horizon latent state, but the model substantially under-reacted to interventions. V7 keeps the V6 structured transition and adds a paired counterfactual objective so action response is learned explicitly rather than expected to emerge from reconstruction loss alone. + +The new `swm_structured_cf` model uses the same architecture as `swm_structured`; only the training objective changes. For the same starting state it rolls out opposing left/right interventions and optimizes: action-effect direction alignment, action-effect magnitude calibration, and endpoint reconstruction. Defaults are intentionally modest so the auxiliary objective does not overwhelm next-state learning. + +## Main experiment + +Install with `pip install -e .` and run: + +`PYTHONPATH=. python scripts/run_v7_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --cf-horizon 30 --device cuda` + +Results are written to `results/v7_counterfactual_training/metrics_summary.csv` and `metrics_by_seed.csv`. + +The primary comparison is `swm_structured_cf` versus unchanged `swm_structured`, `swm_selective`, `no_spectral_transition`, and `neural_operator`. Inspect counterfactual direction cosine and magnitude ratio together with H100 image MSE, text accuracy, and latent cosine. Do not select a model from a single scalar metric. + +## Tunable V7 objective + +The runner exposes `--cf-train-horizon`, `--cf-pairs`, `--cf-every`, `--lambda-cf-dir`, `--lambda-cf-mag`, and `--lambda-cf-branch`. Defaults are 8, 64, 4, 0.02, 0.005, and 0.10 respectively. Hyperparameters should be selected using training/validation evidence, not the held-out test set. + +## Validation + +Run `PYTHONPATH=. python -m pytest -q`. The V7 suite includes the earlier architecture/stability tests plus a backward-pass test for the new counterfactual objective. + +See `V7_COUNTERFACTUAL_TRAINING.md` for the experiment rationale. Earlier stages are documented in `V4_MULTIMODAL_STABILITY.md`, `V5_HARD_DYNAMICS.md`, and `V6_STRUCTURED_TRANSPORT.md`. + +## V8: frozen-architecture generalization and planning + +V8 freezes the V7 architectures and evaluates OOD dynamics, unseen intervention conditions, and imagination-based planning. See `V8_GENERALIZATION_AND_PLANNING.md`. + +Run: `PYTHONPATH=. python scripts/run_v8_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --cf-horizon 30 --planning-horizon 20 --planning-episodes 96 --device cuda` + +## V10 — counterfactual benchmark integrity + +V10 freezes the learned architectures and audits the causal benchmark itself. See `V10_COUNTERFACTUAL_INTEGRITY.md`. + +Run: `PYTHONPATH=. python scripts/run_v10_audit.py --seeds 0 1 2 3 4 --cf-horizon 30 --cf-pairs 48 --device cuda` + +## V11: causal gain calibration +V11 freezes the V9 structured transport architecture and replaces endpoint-oriented causal magnitude fitting with scale-aware trajectory-effect calibration. It reports direction, gain error, magnitude correlation, and calibration slope/intercept across interpolation, extrapolation, and structural OOD. See `V11_CAUSAL_GAIN_CALIBRATION.md`. + +## V14: external control benchmark + +V14 freezes the V9 architecture family and moves evaluation outside the project's synthetic simulator. The external suite uses Gymnasium Classic Control (`CartPole-v1`, `MountainCar-v0`, `Acrobot-v1`) with official RGB rendering converted to the model's 32×32 grayscale input. Models are trained from scratch per environment; architecture and comparison logic are frozen. See `V14_EXTERNAL_CONTROL_BENCHMARK.md`. + +## V15 — External Dynamics Diagnostic +V15 is intentionally the **final diagnostic milestone**, not another model revision. It reloads the frozen V14 checkpoints and tests whether V14's negative full-frame MSE result reflects genuine dynamics failure or static-background metric confounding. See `V15_EXTERNAL_DYNAMICS_DIAGNOSTIC.md`. + +--- + +## Final reproducibility release (V6–V15.1) + +This tree is the frozen final reproducibility artifact for the SWM falsification program. See `docs/REPRODUCIBILITY_RELEASE.md` for the experimental history and interpretation, and `results/` for the measured V12–V15.1 audit CSVs included with this release. The project intentionally stops architecture iteration at V15.1. diff --git a/README_SOURCE.md b/README_SOURCE.md new file mode 100644 index 0000000000000000000000000000000000000000..3f28ffdf1add8ee903e9810d40a1badf9c38f7f9 --- /dev/null +++ b/README_SOURCE.md @@ -0,0 +1,167 @@ +# Spectral World Models + +### Andrew Kiruluta, UC Berkeley, June 2026 + +This repository is a compact PyTorch implementation of the manuscript architecture **Spectral World Models: Multimodal Operator Learning in Hilbert Space for Imagination-Based Planning**. + +It implements a concrete Spectral World Model (SWM) rather than a broad design space: + +- fixed 2-level Haar wavelet image encoder and inverse spectral decoder, +- DCT-style spectral text encoder over token-position embeddings, +- low-rank multimodal fusion into a shared latent Hilbert-style state, +- action-conditioned low-rank spectral transition operator, +- predictive, reconstruction, latent consistency, text, and stability losses, +- imagination rollout evaluation, +- benchmark baselines and ablations. + +The bundled benchmark is **Mini Moving Shapes**, a generated multimodal next-state prediction dataset. Each sequence contains a moving square or circle, a symbolic text description, and discrete action controls. The task is to predict the next image/text state and maintain stable multi-step rollouts. + +Core idea: +Fuse image and text modalities into a shared spectral latent and use low-rank spectral (or baseline) transition operators to predict next-frame images and text, enabling short rollouts for imagination-based evaluation. + +Key components +Models: +- models.py — wavelet/DCT/FFT/conv encoders, decoders, and transition modules (low-rank spectral, Koopman, MLP, transformer) plus build_model. +- Data: data.py — Mini Moving Shapes synthetic generator and TransitionDataset / SequenceDataset. +- Training: train.py — training loop, validation/test evaluation, rollout evaluation, checkpointing, and metric export. +- Metrics: metrics.py — MSE, PSNR, SSIM, Gaussian NLL, token accuracy, parameter counting. +- Utilities: haar.py — dependency-free 2D Haar wavelet transform. + +## Repository layout + +```text +spectral-world-models/ +├── data/ +│ ├── generate_benchmark.py +│ └── mini_moving_shapes.npz +├── spectral_world_models/ +│ ├── data.py +│ ├── haar.py +│ ├── metrics.py +│ ├── models.py +│ └── train.py +├── scripts/ +│ └── run_quick_benchmark.py +├── results/ +│ ├── README.md +│ ├── metrics.csv +│ ├── metrics.json +│ └── *_metrics.json / *_history.csv / *.pt +├── tests/ +│ └── test_smoke.py +├── pyproject.toml +└── requirements.txt +``` + +## Installation + +```bash +python -m venv .venv +source .venv/bin/activate +pip install -e . +``` + +Or, without editable installation: + +```bash +pip install -r requirements.txt +export PYTHONPATH=$PWD +``` + +## Generate the benchmark dataset + +The dataset is already included at `data/mini_moving_shapes.npz`. To regenerate it: + +```bash +PYTHONPATH=. python data/generate_benchmark.py +``` + +Default split sizes: + +- train: 256 sequences +- validation: 64 sequences +- test: 64 sequences +- sequence length: 8 +- image size: 32x32 grayscale +- actions: stay, up, down, left, right + +## Run the full quick benchmark + +```bash +PYTHONPATH=. python scripts/run_quick_benchmark.py +``` + +This trains and evaluates: + +1. Dreamer-style latent baseline +2. Transformer latent predictor baseline +3. Koopman baseline +4. Neural-operator / Fourier-feature baseline +5. Spectral World Model (ours) +6. SWM ablations: + - no spectral state + - no spectral transition + - no text branch + - no stability penalty + +## Train selected models + +```bash +PYTHONPATH=. python -m spectral_world_models.train \ + --data data/mini_moving_shapes.npz \ + --out results \ + --epochs 3 \ + --batch-size 64 \ + --models swm dreamer transformer koopman neural_operator +``` + +## Metrics + +The repo reports: + +- training loss per epoch, +- validation loss, PSNR, SSIM, NLL, text accuracy, +- test loss, PSNR, SSIM, NLL, text accuracy, +- rollout MSE over horizon 5, +- average rollout step time. + +The included `results/` folder contains actual quick CPU benchmark metrics generated in the sandbox. These are meant as reproducibility smoke-test metrics, not final paper-grade multi-seed values. + +### Main quick benchmark results + +| Method | Params | Test PSNR ↑ | Test SSIM ↑ | Test NLL ↓ | Rollout MSE@5 ↓ | Time/step ms ↓ | +|---|---:|---:|---:|---:|---:|---:| +| Dreamer-style latent model | 497,682 | 14.571 | 0.072 | 0.0107 | 0.03490 | 0.0528 | +| Transformer latent predictor | 503,322 | 14.368 | 0.017 | 0.0687 | 0.03657 | 0.0290 | +| Koopman baseline | 465,306 | 14.619 | 0.090 | -0.0027 | 0.03487 | 0.0170 | +| Neural-operator baseline | 403,794 | 14.534 | 0.058 | 0.0210 | 0.03511 | 0.0194 | +| Spectral World Model (ours) | 405,422 | 14.591 | 0.080 | 0.0050 | 0.03577 | 0.0343 | + +### Ablation quick benchmark results + +| Variant | Params | Test PSNR ↑ | Test NLL ↓ | Rollout MSE@5 ↓ | Time/step ms ↓ | +|---|---:|---:|---:|---:|---:| +| Spectral World Model (ours) | 405,422 | 14.591 | 0.0050 | 0.03577 | 0.0343 | +| No spectral state | 497,682 | 14.567 | 0.0117 | 0.03492 | 0.0184 | +| No spectral transition | 446,802 | 14.562 | 0.0131 | 0.03505 | 0.0318 | +| No text branch | 402,350 | 14.729 | -0.0325 | 0.03508 | 0.0302 | +| No stability penalty | 405,422 | 14.433 | 0.0498 | 0.03609 | 0.0294 | + +## Important interpretation note + +The quick benchmark is deliberately small so the repository can be run on CPU. It verifies that the architecture, baselines, ablations, dataset, training loop, validation loop, test loop, and rollout metrics are implemented end-to-end. It is not intended as a final NeurIPS-grade empirical study. For paper-quality claims, rerun with larger sequence sets, longer training, multiple seeds, and a harder image/control benchmark. + +## Test + +```bash +PYTHONPATH=. python -m pytest -q +``` + +Expected result: + +```text +1 passed +``` +This repo is based on the following citation. Please cite this reference when using this repo in your work: + + diff --git a/SHA256SUMS.txt b/SHA256SUMS.txt new file mode 100644 index 0000000000000000000000000000000000000000..e5f0de31ddff3007f869462c1b9776c43ddd25ee --- /dev/null +++ b/SHA256SUMS.txt @@ -0,0 +1,174 @@ +3ed731b65d06150c138e2dadb0be0697550888a6b47eb8c45ecc9adba8b8e9bd ./.pytest_cache/.gitignore +37dc88ef9a0abeddbe81053a6dd8fdfb13afb613045ea1eb4a5c815a74a3bde4 ./.pytest_cache/CACHEDIR.TAG +73fd6fccdd802c419a6b2d983d6c3173b7da97558ac4b589edec2dfe443db9ad ./.pytest_cache/README.md +44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a ./.pytest_cache/v/cache/lastfailed +b01b056f0602ebcafba839b5c7e8823af7636e5f0d586da920ca01d6b6d31c33 ./.pytest_cache/v/cache/nodeids +bf571929e8e8c20597c605c7d4a982e6d612a60cbf0cd2b70fc0cde0002ce279 ./COLLABORATORS.md +a3caa209da0594667ce389291a99fe89ab202ba7e57b46819862b55730be0b7e ./HF_README.md +63510bf8419fa733ec1d56692c3d1edc9191320b446e73a893d9549574171e35 ./HUGGINGFACE_UPLOAD.md +a08f3530b8ae26ac3b4bd0dd3aba206bab940e978c3dcaf7b3a5681cf352c8bd ./LICENSE +29512728279597fdc3661f9cd033574ccc96756dc1bbf9f20cd29f0ab348c1fd ./LONG_HORIZON_BENCHMARK.md +9796e532c7cc33964600a6c4ff0fc59470c1e0f382f873e87737e126e0319f86 ./README.md +83e1be70172dbb9df5868f199813e8855022d46815a2d7a565dbf70139be2147 ./README_SOURCE.md +33b0723122c453f978502192ac0b81bb11f9beee7b694678b1756776c0aad647 ./SHA256SUMS.txt +9801b3c771837f52c7bc6ec7b305b025cff397fc2e408e35db25da54e8eedb69 ./STABILITY_REFACTOR.md +2ce36a27a5d90882897199044e15f5b44387a639d061483d08601a77d46b03f6 ./V10_COUNTERFACTUAL_INTEGRITY.md +0191c5e5589884aaabdeaf2a50d353b170257b1994bfccd0cc8e632a12d1bea7 ./V11_CAUSAL_GAIN_CALIBRATION.md +ea626169ecba36a1a83f686ccca6f08d53c4bc95bcc0e36749456577fdb1b811 ./V12_CAUSAL_RESPONSE_DECOMPOSITION.md +ad6b59f95d4d8f2c0db18b4e4400390befac13d7ca1c670cdd233a42ba8c1890 ./V13_AXIS_SYMMETRY_EQUIVARIANCE.md +bedc9f71a8ec8f190e05b6b9e0cd6179921365ac9d43a0ed6bdf4ed23eee68cc ./V13_SMOKE_VALIDATION.md +fb6d5a3a3f01ca1337b4dcfa567dd835cf05cc5005feba751089317fb6f6119e ./V14_EXTERNAL_CONTROL_BENCHMARK.md +16c5a849801d2514bf3447d2d1446cd8caebe54009282137550286b2ad177b09 ./V15_1_BUGFIX.md +f84953431287830f30bf695634143bd83be939e5ce9a1dfdfb5a0471958c3474 ./V15_EXTERNAL_DYNAMICS_DIAGNOSTIC.md +8b9b326921aa8ba82488ed8723467f08836e36b70a1ddedbe28ce92d4bc92a84 ./V4_MULTIMODAL_STABILITY.md +ac3e7e9b93f0576297a89b7d864db4a62362f2d74cf182ee0f572b642fb9d047 ./V5_HARD_DYNAMICS.md +724d8aabaf1f7e2273d13ee6642e1c7151449ce51ce5b3a4065cde03b1c1e7f2 ./V6_STRUCTURED_TRANSPORT.md +2883f7995d57b6f2f3ee353c7b36253c5bdabfd68feb17ac84c6909208e291d9 ./V7_COUNTERFACTUAL_TRAINING.md +e20a30f8238e336e6e3c18e54631104c1d3a1aa5196d6c932171a5c7425f1ae0 ./V8_GENERALIZATION_AND_PLANNING.md +4517cd17b2fecb9830ca58cb290ebb2a77e1ac910601180106d7467118f48739 ./V8_SMOKE_VALIDATION.md +cf111fe065bc41b2b56bfdd289432a4ae1d2ead8e54a7fd821c7612967db8805 ./V9_CAUSAL_GENERALIZATION.md +24ec48221c3469fabbf2cb0c130af1cd755644a0e161b98d5595bfecc3cde9ff ./configs/prototype.json +02e24aa69ebd0c13cb675faf51cb0c8a437d5bd04150e431704e3c66012b2534 ./data/generate_benchmark.py +3688d9b074059a297d2f9163a93b8a07a5aaeaf5a94f5e7b7792421350acbd71 ./data/hard_controlled_dynamics_v6.npz +15b91073dfd882b528416bb1fb53d08c53eeaae201fbd468ac9f1bce041b73b4 ./data/hard_controlled_dynamics_v6_rollout_h6.npz +e3dea41e2fd4303760e7abb526f4404434cfca74093d45df235a27f7828cc190 ./data/mini_moving_shapes.npz +4593ad92dab9ff7ffc6250d368a7b6eb68ab1d09c3bae749209f6dbc4cbcf491 ./data/mini_moving_shapes_rollout_h11.npz +4593ad92dab9ff7ffc6250d368a7b6eb68ab1d09c3bae749209f6dbc4cbcf491 ./data/mini_moving_shapes_rollout_h31.npz +7dcc2e791d591c7810dc2295779e816e3b4cf38e8554397c33a4949538860143 ./data/v9_extrapolation_h6.npz +0aab7d41b0596bd72a24100710f6f7aafe8a94289ed33a6bd921fcbcee0c97be ./data/v9_interpolation_h6.npz +288cd80b247bb1396897579bd2b86363c1936d60e904431a22eee9a0642c42c6 ./data/v9_structural_ood_h6.npz +d26d2a2c08b9c84f7613e31099b6274e29a326f58f7dadb50b3ce4dc5d9d1181 ./data/v9_train_family.npz +2cb323a3a4adb7970631b5bd300d28d3dcdc9a813b2c75202f436beae63295b6 ./docs/HF_RELEASE_STEPS.md +9132263a8b224d800e22df68e6670a80f0aa3e18b3f4b2fcb01ae312904f546c ./docs/REPRODUCIBILITY_RELEASE.md +5538a4705bbe9dbcbe5662d5055769f26589735d216e28431a9505afd71bb737 ./hf/configuration_swm.py +7c8e0e9a56b2cae193e53955c1b3c696ae79bc27a6fd197961f8a8db69700ecd ./pyproject.toml +c24a1f9491090ed37fd2382de19d5b82dd2afd5dc94fbd0cd33687422d93eac5 ./requirements.txt +8affbf4f9c7e1a74747e71075f7ae1ec96cc639c761c70b9591110e732030a34 ./results/README.md +7754579118d0a89cfb2781b7e97f5b2c4f02dc950cc6269339ec7daefcd04ecc ./results/dreamer.pt +8fcc90ed9a6230ce17bdaccaacc7beda5c3e6cadd46f6cf6138e9a4d532009e9 ./results/dreamer_history.csv +aa1e9cbf5da301d04ec319b94cd6268646683deb979ada0c1f94d9dfe6b711c1 ./results/dreamer_metrics.json +a24d327994545a03f46d9f9db6b7fe6ea29b1bf407bcaac76bcb88f897377a23 ./results/koopman.pt +cdcf7c3980af6372de63ae908f96cc88f8e3dd5de2f07e5703048053bcfcf087 ./results/koopman_history.csv +7ce64e628468149203e217b49d5e9d5b59fb032d8631a3400afe67142e497a4e ./results/koopman_metrics.json +c296bf1f9a61fc48ce6c1ead8e2e511eff8a0e956e1d94c553d80949016ddf76 ./results/metrics.csv +1b2ef28d4d169ec60993a01865600dfc139a76bbb50e334b836851ccf5471aef ./results/metrics.json +d2b7aea7c1936d2e3bcda86bf27e6148bd96889448259f009ba20daae42abeb7 ./results/neural_operator.pt +52852d61012de0ce6dc2c950b85893f1630a369542593043c75d9006838f1748 ./results/neural_operator_history.csv +23859f5efea3a990cc4a99a014b77616d63025e5161a82a87e9066709b02874b ./results/neural_operator_metrics.json +610d0e774236353811f1958c2ae12008ee102d32c2ccd1e74531f6a5acb2dede ./results/no_spectral_state.pt +e968e0acd2f85163cf96fc80b9cc032999656701bb29ece1a5458efd2f4a868b ./results/no_spectral_state_history.csv +ca896a44c825acf931cc7ed42f0789f61ff071f8050d3ab2173f8235750a7520 ./results/no_spectral_state_metrics.json +9cf788c11ec1cc7d72a39fb8237ce49be12da3aa7fc195ccb301afca5beca0e8 ./results/no_spectral_transition.pt +29aab44eee560594f44571f93822b7cb80ab81a003bdd9a3d10005d882e99740 ./results/no_spectral_transition_history.csv +d06ee5e855eef67f64459fd164ba2d1d14e4cedfdaa91573067d5f187e24b79c ./results/no_spectral_transition_metrics.json +c35225bb09b697aedad2dd26b20f57f2ed42619cad1ae2f32fdf4fa0379cb2af ./results/no_stability_penalty.pt +11755c07b5a689b567affeaaefc2d980fe7b4a8b31cb563bc8d3a509d1ddb166 ./results/no_stability_penalty_history.csv +c72922852bcd46f7f2f446436d26355807d425dd01af54660eb13cfa9a0b314a ./results/no_stability_penalty_metrics.json +b2c9b293ea4f61d9202182b1ffd13dceeb9f76d1c48630da2f03caef4b7ef835 ./results/no_text_branch.pt +249b3761157a99e2ef73b22d3727cea2ca4e79b81156bb2cd3646f9025fc1746 ./results/no_text_branch_history.csv +988259dc26dd92810aa8d075b07b82d8bd733a948f578260649e71e1605384b0 ./results/no_text_branch_metrics.json +39c834750c8f021f7880a0b18dbd3c3e557df8198a0dc5dcd002214aaeed0482 ./results/swm.pt +732cdec082c187acbf67124912233a33ee6a9e5b442a94577ae68eb08eedab2c ./results/swm_history.csv +f30f06654accaef32d05bf15f93d3851110cd09fd5f640949aa37b1261951c36 ./results/swm_metrics.json +46656b6ee4f1f06f4768b06e68df7a2079f2adacc28de3bffb5c8e4b45b481fa ./results/transformer.pt +2944519b035562ddd7fede340e09102c935c0579d49fb2732a259857b04a8c12 ./results/transformer_history.csv +ca29fbf514dd76f64a453c6f479601fd4d1e96170177ee5cd85f0c4e35fa183c ./results/transformer_metrics.json +cc0df50a489358558bd68f25e73d7689afd1b98112bcd5af6703b6403a3e9115 ./results/v10_cf_integrity/metrics_by_seed.csv +f7c77ec1d724385b596b2de3e7860c7b3fe3330de4a44138be88cb8d0db724be ./results/v10_cf_integrity/metrics_summary.csv +c5923fe86118a65a91093d7d4e21f4204171ce8ccbbb54d9b8600594e72ae31e ./results/v10_cf_integrity/oracle_physics_audit.csv +d0c46469f469e1df40db0405ef7d52f8e0136f5c80f36136aca231001a8c807a ./results/v10_cf_integrity/run_config.json +3eaba744687ad48f1dfa96a62c2a93f5c01747793f5eb1233345020f8d4bc633 ./results/v10_cf_integrity/seed_0/no_spectral_transition.pt +d4906b6fa6bd2afe8366cefa1f94c5101784b1c97a4a6f5753e015473e9dca02 ./results/v10_cf_integrity/seed_0/no_spectral_transition_history.csv +17dedaa3d4ab3ec79f95c72ff73c18d06de287b8c1f44b47e5326370d0693388 ./results/v10_cf_integrity/seed_0/no_spectral_transition_metrics.json +34d1b6a9b1593f44db69357bad97d29541ca32e311b0083027739a78d032b9bc ./results/v10_cf_integrity/seed_0/swm_structured_cf_v9.pt +fbe41c47207dd739510756b1c02f18cc78c5eef60fb164cbe540f0deb6bbe29a ./results/v10_cf_integrity/seed_0/swm_structured_cf_v9_history.csv +dfd9695fb723fd36a9bf7af658a8d43836796cd3cfd44dd05e73d5273893b8aa ./results/v10_cf_integrity/seed_0/swm_structured_cf_v9_metrics.json +6b26cd6b9c8e3c75711fd02207709a223994f705aaa2a6806d47b7fbfb9e6289 ./results/v12_causal_decomposition/paired_bootstrap_v9_vs_controls.csv +8b4c72089ed638907dc1f65985a43c1093f4a261cffa89f4b49bde81225d5b29 ./results/v12_causal_decomposition/stratified_metrics_summary.csv +183712aafe59c29cfe576f725b73c3aa34f4b121f8a9c52fb6620dba96b03744 ./results/v13_axis_equivariance/axis_equivariance_summary.csv +4cfb2ad9348ff93ef88402d768b18272fe7d92c468fffd8ce891711b24aab464 ./results/v13_axis_equivariance/forced_collision_summary.csv +8a01d838e931bab4a3078e840dce0a4c77dda5cf77c0f6b5f518370337583459 ./results/v14_external_control/metrics_summary.csv +c888e7c9faca105d5d7503ce36411a6912ea9cad4ab72ef0b24258cb49ed7d56 ./results/v15_external_diagnostics/diagnostics_by_seed.csv +a5324309539550513108b9a65620775f37bc1a5e37ab57ad7e296c0af1bc5324 ./results/v15_external_diagnostics/diagnostics_summary.csv +c942607a39c008c952979a5823b3852f003fd2b4dd44b2e6a5c2a297540e5a9d ./results/v6_structured_transport/metrics_by_seed.csv +ed217f086d40894b4d13494ddfc8773fab621195d56ca4452cc9036dced49e97 ./results/v6_structured_transport/metrics_summary.csv +59308c3a4abce83e107c4b6368d95f3d5dee9c9ef25f90c1674ce0cb3230c560 ./results/v6_structured_transport/run_config.json +b55e6b924e0ef2595042db0e607d01eddb069207e17d5e8ec1e18ef752765bb2 ./results/v6_structured_transport/seed_0/no_spectral_transition.pt +a7fbbb511b9ec055e96a3e3fa365d72e8bfb382a11e84a177f4bdc8ec6baea8c ./results/v6_structured_transport/seed_0/no_spectral_transition_history.csv +e3eb7947a654ff3d877cf832228e184543cc12eaa9f3f19f25ca33caa0d4c5f2 ./results/v6_structured_transport/seed_0/no_spectral_transition_metrics.json +9e232e5f3e0d28283c4e7ae16c19f423734a6e9d80d3970ef6dc94675f283a6c ./results/v6_structured_transport/seed_0/swm_structured.pt +4bcbdca4b9275bf925015fa7318ffa73c3116e8fd46e856991a32f739dbd0fc1 ./results/v6_structured_transport/seed_0/swm_structured_history.csv +9ea995efe9c7eb306a6ced9cf57925ed5a0ca313c9fb09c7f5f82ed2bdf38cc3 ./results/v6_structured_transport/seed_0/swm_structured_metrics.json +6e59f78ae13ca5e3992ffe439992bb1daf54c17c4739028b1f567ac9b48214d0 ./scripts/__pycache__/run_v15_external_diagnostics.cpython-313.pyc +291532f8cd8d3a224bd8989925bc822972f21b6e75cfdc40a2305ba0d3064524 ./scripts/run_quick_benchmark.py +467ac49c4f9e5c816432afcc15fd4ba5d1319398a8a31db55a20a26ab3c04ba7 ./scripts/run_stability_benchmark.py +7db49776922dce21b8ed7f64a3f1d19059f1c07f51f62363943b4f0065128ec0 ./scripts/run_v10_audit.py +3e11ff518ff6fd455e35de6a077726fbd43e2fdbca007ba3bcbf4a3885112b38 ./scripts/run_v11_benchmark.py +f9459e5e383f4b714e542658c9f24d58871ee3ed37ba5bff1ab96b748f0b10ad ./scripts/run_v12_decomposition.py +5f83890b73d6cb6a9159a317957c19fb6fd76f7d25c308d32d75c734ddddf886 ./scripts/run_v13_axis_audit.py +e4b797fadb87241cfd8cb5952eb18a6a2d872d1dcc0d5d31297e0b4aca23c841 ./scripts/run_v14_external_benchmark.py +3304c32537129587acf9dd9dac0ed64e00faee5be90f61672e4c396922bb8b9d ./scripts/run_v15_external_diagnostics.py +7f30119ceb40b3a6e75d2f09ccfab5cc6daf56e8c171107f6f50787dd01c5ca0 ./scripts/run_v5_benchmark.py +5d66b632bc04c185e2eb9799d4f8a2be4ea15083a60d0f7be706bd8d23058e0d ./scripts/run_v6_benchmark.py +4c1213fe463c0c2c561ff20d4e2623b0ba1dd459c8dfa457c8f4563e66e06a85 ./scripts/run_v7_benchmark.py +97e185c74a973c80a28e5dee4859254577f17cdafea51196eaa3de096d33dce7 ./scripts/run_v8_benchmark.py +66e4ffa268f8692cf58309a4bf076ae143d8c91e409efcaa666a7f1479aaecae ./scripts/run_v9_benchmark.py +39777f448c9519e9dca29bf61bb4eb7140a2fc4e85b8bd9b616ff9e9820e0076 ./scripts/system_check.py +f97b4a3577609d33d76b7015c74648e6568ae038b15a8cbb9742f13f37452664 ./spectral_world_models/__init__.py +726010ce8be26595d175feb4406d1d1d061562c98cc243df32c8587909c91b30 ./spectral_world_models/__pycache__/__init__.cpython-313.pyc +5d554017109781ebb2c2e76185a14d3e5afe258d5bf6c018e207d803a0423a02 ./spectral_world_models/__pycache__/counterfactual.cpython-313.pyc +88981ab807349fefdc000d6fc4ec68ab9b66447ad25667b37d30af1f816fb236 ./spectral_world_models/__pycache__/data.cpython-313.pyc +2824638abc6745f82b8e91219606057581fb28c8b6e3fc62b1ec515bfbccf7d3 ./spectral_world_models/__pycache__/haar.cpython-313.pyc +7f0f406b3948fb585cfa3a72a96dabf6b196000c254aa4af64a7e694c1097508 ./spectral_world_models/__pycache__/hard_dynamics.cpython-313.pyc +d1566985362ee3d303c7d5d66a86b403c7bf66b8799e7542628971d1f4f7c2b1 ./spectral_world_models/__pycache__/models.cpython-313.pyc +c3384e41ae2574a9c879d34122e29c9f2bfb8ddfbadfdaab50dc059b75e5ddd0 ./spectral_world_models/__pycache__/v10_cf_audit.cpython-313.pyc +a3eac2249f23e1c4ecd5a2e9ce8c0d7aaded46fc3a51aa868ab0c61da858d842 ./spectral_world_models/__pycache__/v11_causal_calibration.cpython-313.pyc +7e539c5c1e43a243588894297f13dedd7a09c95a92a0c6ed9ac5521df3deb5e0 ./spectral_world_models/__pycache__/v12_causal_decomposition.cpython-313.pyc +d62494f6c1951afa737a0d19dcddd407dad8e4c7dd87b3988fad75612a2ad7e1 ./spectral_world_models/__pycache__/v13_axis_equivariance.cpython-313.pyc +70dc53cb53c9dd986d5b7cfb3f02fed92fdc6b8896f2ad6d105ead7b53c52989 ./spectral_world_models/__pycache__/v14_external_control.cpython-313.pyc +192df8d927d44eadff2b182affbfe9d26df0ca23921d9db121412bf4a3eb594a ./spectral_world_models/__pycache__/v8_generalization.cpython-313.pyc +c65a6faa10d07b54e7b7bb9e40b74e17cd42639cc165d38287ddd06ac0992571 ./spectral_world_models/__pycache__/v9_causal_generalization.cpython-313.pyc +e3ae7e1167dcd7d512718eac9c9d995d2a22386c095d4e0beef9527abd02e9e5 ./spectral_world_models/counterfactual.py +3fc2ece9186263b4678cfdb3a548b708c82379b7b7715fd9bfe2dbf744d8af18 ./spectral_world_models/data.py +74d0fd184709031ed50b9fd2cac6b5e5481293d74c0b69b19c997107ab95b5ba ./spectral_world_models/haar.py +656e54ed27e711cac2a3cdf93987d78f01497ed74346b116d6d1ae1694b00ebd ./spectral_world_models/hard_dynamics.py +e3e5996dfd31a57a241178ccf0e86220bd9fb8ec1c73cfd4516292045c094c07 ./spectral_world_models/metrics.py +baa1c87e76de49b6f40e53871fd7de139634f694d053b3194451d63c94394db5 ./spectral_world_models/models.py +f0671bb715f7606753000091eb13e969969d907f4ae94bd43d6564930e82990b ./spectral_world_models/train.py +681f70d09ade8b429509ad875ff8831df877ebf41d80f1bf7ce767caa5a9ad56 ./spectral_world_models/v10_cf_audit.py +030576b5437f99c204fd9955582cb297ce5aeed205317c7ca2764a1142f7be65 ./spectral_world_models/v11_causal_calibration.py +84e0055e6cd70f9fa6358656fb7f81e76270278b2ec7880681d3619dc6a57c1f ./spectral_world_models/v12_causal_decomposition.py +ef4b9dca0532f2eda7ece8ae9fbb7ce08d9d25e98138254ebf3366c5c6bc7a03 ./spectral_world_models/v13_axis_equivariance.py +43122e7d4065d998948f434633c823e26ecf9750b81cbe804ebe679a018cb3c5 ./spectral_world_models/v14_external_control.py +f141a401541fb9369d20ed97bef8468a4c4645726b21ad784832c27878ddb0ad ./spectral_world_models/v8_generalization.py +1e71a1f372e6dbcb78a91ac97d5a1bca618783ba5541861539294e6e117e264d ./spectral_world_models/v9_causal_generalization.py +95d1fea81d96cea36f8ccded6ea1ec75e714da84e62f353d8f30b70b60d4669b ./tests/__pycache__/test_smoke.cpython-313-pytest-9.0.2.pyc +583d4a9f5fdfd22e36e58ecaaea7f809a3dff1050aa91941d34f95dad1653282 ./tests/__pycache__/test_stability_constraint.cpython-313-pytest-9.0.2.pyc +4141ec2598cf166493059d7b6b16ac4e79b7a95d1718c83632152116ceab0eb8 ./tests/__pycache__/test_v10_cf_audit.cpython-313-pytest-9.0.2.pyc +86d146aaf18c0caa5f19ee709b4acc55ba03a7d726b3a23f1e77a199d0dc1cfd ./tests/__pycache__/test_v11_calibration.cpython-313-pytest-9.0.2.pyc +7e42fd4ef45424dd5876a1cc23e4625fca2c07ff7a710909d8fd0bcefcad654b ./tests/__pycache__/test_v12_decomposition.cpython-313-pytest-9.0.2.pyc +ab959b3e9d7be194f9ceedb8049a995972580b5600696b7352367579de267dbd ./tests/__pycache__/test_v13_axis_equivariance.cpython-313-pytest-9.0.2.pyc +509776790ff7b89ebfd4749a2ee93735759861c71387719aca87b4834afcdb6b ./tests/__pycache__/test_v14_external_control.cpython-313-pytest-9.0.2.pyc +e79dfb8bf4d366f5a1a02a5846093e1c9e8aeb99b7f9ac6705eb43ac0c4e7acb ./tests/__pycache__/test_v15_1_state_token_decode.cpython-313-pytest-9.0.2.pyc +26f83a960a0750a4f0f0c1897626de7c0e53927d6a71b317ec666371946a8a96 ./tests/__pycache__/test_v15_diagnostics.cpython-313-pytest-9.0.2.pyc +5f807a38e2113fa94ad13c4dd81e5b7ebfca575dc7e989fb891216af4507ecf1 ./tests/__pycache__/test_v4_selective.cpython-313-pytest-9.0.2.pyc +1114b5fa6a3b784a256e73315160a4480206f27f8ca9c098fd60f4f334afdd44 ./tests/__pycache__/test_v5_hard_dynamics.cpython-313-pytest-9.0.2.pyc +a600a9eb3d3b86eaef730347c7a527bfdcdf302f057c784ef378a842b345ccec ./tests/__pycache__/test_v6_structured.cpython-313-pytest-9.0.2.pyc +fd8d9de4a26f075221638e0143b3affd4c659091926955901db389a369a4f8e3 ./tests/__pycache__/test_v7_counterfactual_training.cpython-313-pytest-9.0.2.pyc +2f8475c83eba08a537164ff7bee4102228e2c948f20422e80a9dce20b15f1608 ./tests/__pycache__/test_v8_generalization.cpython-313-pytest-9.0.2.pyc +9499c06a9c9d3998cfb787b5c9b12b9910d0f82243a7d3a8a78bd346c471c586 ./tests/__pycache__/test_v9_causal_generalization.cpython-313-pytest-9.0.2.pyc +d45c477853469ba42a643f2476b4a5c7b88075a662d91c793cdcd2bb68f6385b ./tests/test_smoke.py +4c3ca6cf55763475b01b2833d264f57101d30e6fb9170df057444503b00a95e5 ./tests/test_stability_constraint.py +198d3795b20d3edc3bcf0261d2e89d1e3090192eb65c22e80851970c7d86e5ef ./tests/test_v10_cf_audit.py +f119a86597445b83ff941da301a97e7de6201c651a356b2fa4c6489b5d81f54f ./tests/test_v11_calibration.py +5a657d824c3b1931e65197c8950475645c79a014e7d82c3ce0d514706a1d2828 ./tests/test_v12_decomposition.py +31c6511e30869159e23b0baeed999817a783412ee4c7a4f9d13d3cc8945ba583 ./tests/test_v13_axis_equivariance.py +f9d30b3d98964a206f49895182111a32bfe6cfa75becbbc18dce5eead42432ad ./tests/test_v14_external_control.py +68a7c2f6ed0169c0d309f60fdf51d169bd823900910142843940e6b4aec339cb ./tests/test_v15_1_state_token_decode.py +1ae4c75254a63188fb40fb4f4611739408d8d5d7ae4294bfaa002a8ca1327ad3 ./tests/test_v15_diagnostics.py +481ded46bb953cd21b63115241f7d6b6c9dab4b3f87d77096af23c816a4bfd79 ./tests/test_v4_selective.py +fe8a68538759fc011c0a7131d909506410f69b300349a47888a8bebb4c6ab942 ./tests/test_v5_hard_dynamics.py +daba44887294567466959c4743e29a5bb6dc97852d2a20b7dc91166ba3678914 ./tests/test_v6_structured.py +904c55073258023939e029823cee314800c81d89896a20f4772dae70046ee598 ./tests/test_v7_counterfactual_training.py +b29fa14d5c1cb39992283f418ab3cfa874f0e05c5d42f386af0b195bf4027cdc ./tests/test_v8_generalization.py +b8785289ac99b79da780bb946561f4f85799cd3676dafad0a2a8c6a048750706 ./tests/test_v9_causal_generalization.py diff --git a/STABILITY_REFACTOR.md b/STABILITY_REFACTOR.md new file mode 100644 index 0000000000000000000000000000000000000000..ccfd3e26689e6e2724284e62ab4c0f289f368633 --- /dev/null +++ b/STABILITY_REFACTOR.md @@ -0,0 +1,27 @@ +# SWM stability-attribution refactor + +This revision responds directly to the five-seed H=5/10/20/30 finding that full SWM remained much flatter than the Koopman baseline, while the existing Frobenius-energy stability penalty had effectively no measurable effect and removing the spectral transition improved rollout MSE. + +## New controlled variant + +`swm_spectral_norm` retains the SWM wavelet image state, DCT text state, fusion, and low-rank action-conditioned operator dictionary. For each discrete action it explicitly constructs the effective linear operator + +`A(a) = W0 + sum_r beta_r(a) U_r V_r` + +and caps its largest singular value at `rho=0.95` in the forward pass. The unconstrained nonlinear residual is omitted in this diagnostic variant, making the contraction mechanism directly testable rather than conflating a linear operator bound with an unbounded learned correction. `tanh` is retained and is non-expansive. + +The auxiliary stability loss now measures violations of the same spectral-norm threshold before forward rescaling. The benchmark records raw and effective operator norms so a claimed constraint can be checked from the results rather than inferred from a training hyperparameter. + +## Attribution experiment + +The targeted benchmark compares `koopman`, `neural_operator`, `swm`, `swm_spectral_norm`, `no_spectral_transition`, and `no_stability_penalty` using identical seeds. Default rollout horizons are 5, 10, 20, 30, 50, and 100. A separate held-out long-sequence dataset is generated automatically, so the original short training data are not changed to make the long-horizon test easier. + +Run on CUDA: + +`PYTHONPATH=. python scripts/run_stability_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --device cuda` + +Primary output: `results/stability_attribution/metrics_summary.csv`. + +## Interpretation rule + +The hard-constrained variant supports the stability-mechanism hypothesis only if its long-horizon behavior improves reproducibly relative to ordinary SWM/current-penalty SWM without an unacceptable collapse in one-step predictive or multimodal metrics. If it does not, the correct conclusion is that this benchmark does not attribute SWM's observed rollout flatness to the proposed operator-norm mechanism. diff --git a/V10_COUNTERFACTUAL_INTEGRITY.md b/V10_COUNTERFACTUAL_INTEGRITY.md new file mode 100644 index 0000000000000000000000000000000000000000..c86cf92dfd75478d6a38656e082787c1d4bd0e6a --- /dev/null +++ b/V10_COUNTERFACTUAL_INTEGRITY.md @@ -0,0 +1,29 @@ +# V10 — Counterfactual Benchmark Integrity & Physics Audit + +V10 is an evaluation-integrity release. It does **not** introduce a new world-model architecture. + +V9 produced nearly identical endpoint counterfactual cosine values across interpolation, extrapolation, and structural-OOD regimes. That repetition is not sufficient evidence of an evaluator bug: cosine can remain unchanged when physics changes only the magnitude of a collinear intervention effect. V10 therefore makes the oracle causal signal observable and auditable. + +## What changes + +- freezes the V9 comparison architectures; +- uses identical initial physical states across regimes so physics is the controlled variable; +- records acceleration, speed cap, damping, restitution, occlusion and a physics fingerprint; +- hashes every oracle counterfactual trajectory and asserts hashes differ across regimes; +- replaces the single constant left/right probe with left/right, up/down, turning and impulse/reversal interventions; +- records oracle endpoint magnitude, path magnitude, RMS effect and temporal variation; +- reports both endpoint and trajectory-level counterfactual direction/magnitude metrics; +- fails fast if the oracle trajectories or causal-effect diagnostics are insensitive to the regime. + +## Full run + +`PYTHONPATH=. python scripts/run_v10_audit.py --seeds 0 1 2 3 4 --cf-horizon 30 --cf-pairs 48 --device cuda` + +Primary outputs: + +- `results/v10_cf_integrity/oracle_physics_audit.csv` +- `results/v10_cf_integrity/metrics_by_seed.csv` +- `results/v10_cf_integrity/metrics_summary.csv` +- `results/v10_cf_integrity/run_config.json` + +Interpret V10 as an audit first. Only after `oracle_physics_audit.csv` demonstrates distinct regime-specific oracle trajectories should learned-model OOD counterfactual metrics be used for manuscript claims. diff --git a/V11_CAUSAL_GAIN_CALIBRATION.md b/V11_CAUSAL_GAIN_CALIBRATION.md new file mode 100644 index 0000000000000000000000000000000000000000..7e868e3fa39f81e50f1ae8c9c8c926ffe1a8adfb --- /dev/null +++ b/V11_CAUSAL_GAIN_CALIBRATION.md @@ -0,0 +1,14 @@ +# V11 — Causal Gain Calibration + +V11 follows the V10 integrity audit. The architecture remains the V9 structured spectral transport model; the intervention objective is changed to target causal gain calibration. + +## Change +At every counterfactual rollout step V11 compares the predicted action-effect vector with the oracle action-effect vector after normalization by the oracle effect scale. A symmetric log-gain penalty remains as a calibration term, and branch reconstruction prevents degenerate effect matching. + +## Evaluation +The benchmark reports trajectory cosine, mean magnitude ratio, magnitude-ratio MAE from 1, absolute log-gain calibration error, predicted-vs-oracle magnitude correlation, and a linear calibration fit (slope/intercept) for interpolation, parameter extrapolation, and structural OOD. + +Ten seeds are the default. V9, neural-operator, and no-transition controls are trained and evaluated under the same protocol. No composite score or superiority claim is built into the benchmark. + +## Main run +`PYTHONPATH=. python scripts/run_v11_benchmark.py --seeds 0 1 2 3 4 5 6 7 8 9 --cf-horizon 30 --cf-pairs 64 --device cuda` diff --git a/V12_CAUSAL_RESPONSE_DECOMPOSITION.md b/V12_CAUSAL_RESPONSE_DECOMPOSITION.md new file mode 100644 index 0000000000000000000000000000000000000000..f875f74ef6b16349ad614b0e8fbdf4575fb601bc --- /dev/null +++ b/V12_CAUSAL_RESPONSE_DECOMPOSITION.md @@ -0,0 +1,33 @@ +# V12 — Causal Response Decomposition + +V12 is a diagnostic release. It introduces **no new model architecture and no new training loss**. The V9 structured counterfactual model is treated as the primary frozen experimental recipe, with neural-operator and no-spectral-transition controls. + +## Question + +V11 showed that a scalar gain-calibration objective can lower log-gain error while suppressing intervention magnitude. V12 therefore asks *where* V9's causal response succeeds and fails before another optimization change is attempted. + +## Decomposition axes + +The evaluator reports causal trajectory cosine, magnitude ratio, magnitude-ratio absolute error, and log-gain error by physics regime; intervention program (`left_right`, `up_down`, `turn_cw`, `impulse_reverse`); rollout horizon (5, 10, 20, 30); oracle-effect tertile; initial-speed tertile; boundary-contact status; and central occlusion-zone exposure. + +The effect and speed bins are computed from matched oracle episodes within each regime. Boundary contact is derived from oracle target trajectories. Occlusion-zone exposure records whether the target path enters the central occluder strip; it is not a claim that every such frame is stochastically occluded. + +## Statistical comparison + +`paired_bootstrap_v9_vs_controls.csv` reports episode-clustered paired bootstrap 95% intervals for V9 minus each control on trajectory cosine, magnitude-ratio MAE, and log-gain error. The bootstrap preserves the matched episode/program/horizon structure. No composite score is produced. + +## Main run + +`PYTHONPATH=. python scripts/run_v12_decomposition.py --seeds 0 1 2 3 4 5 6 7 8 9 --cf-horizon 30 --cf-pairs 64 --bootstrap 2000 --device cuda` + +## Outputs + +- `decomposition_by_episode.csv`: finest-grained matched observations. +- `stratified_metrics_by_seed.csv`: seed-level stratum summaries. +- `stratified_metrics_summary.csv`: across-seed descriptive summaries. +- `paired_bootstrap_v9_vs_controls.csv`: paired uncertainty analysis. +- `run_config.json`: complete invocation metadata. + +## Interpretation rule + +A positive V9-minus-control trajectory-cosine interval supports stronger directional alignment on this synthetic benchmark only when the interval excludes zero. For magnitude error, negative V9-minus-control deltas are favorable. Subgroup findings are diagnostic and should not be presented as confirmatory without multiplicity-aware follow-up. diff --git a/V13_AXIS_SYMMETRY_EQUIVARIANCE.md b/V13_AXIS_SYMMETRY_EQUIVARIANCE.md new file mode 100644 index 0000000000000000000000000000000000000000..125dc7fdffd5767a6b7e63a19c29f7ad3a06b454 --- /dev/null +++ b/V13_AXIS_SYMMETRY_EQUIVARIANCE.md @@ -0,0 +1,19 @@ +# V13 — Axis Symmetry & Intervention Equivariance Audit + +V13 is a diagnostic release. It freezes the V9 architecture, training distribution, and counterfactual objective. It does **not** tune away the V12 vertical-action failure. + +## Question +V12 found that V9's causal direction was strong for left/right, turning, and impulse/reverse interventions but nearly collapsed for up/down. V13 asks whether this is genuine learned axis anisotropy. + +## Exact symmetry tests +For each matched episode and action program, V13 constructs identity, 90-degree rotation, horizontal reflection, vertical reflection, and x/y-swap versions. The image, controlled state, velocity, text position tokens, and action sequence are transformed consistently. The isotropic ground-truth dynamics provide an oracle commutation check. + +The central diagnostic is whether the model's transformed causal-effect trajectory, mapped back into the original coordinates, matches its original prediction. Outputs include transformed causal cosine, causal-gain ratio, equivariance-effect cosine, relative equivariance error, and oracle equivariance error. + +A 90-degree rotation of a successful horizontal intervention is especially diagnostic: if the rotated equivalent collapses, the failure is attributable to model/data anisotropy rather than a different set of sampled episodes. + +## Collision audit +V12's incidental boundary-contact stratum did not activate. V13 therefore creates near-wall states with velocities directed toward each of the four walls and verifies contact from oracle trajectories. Collision metrics are reported by wall, action program, and physics regime. + +## Interpretation +This release is intentionally falsification-oriented. A strong result is not required. If V9 violates these exact symmetries, the anisotropy should be reported as a limitation rather than hidden by another loss change. After this audit, the recommended next step is an external dynamics/control benchmark. diff --git a/V13_SMOKE_VALIDATION.md b/V13_SMOKE_VALIDATION.md new file mode 100644 index 0000000000000000000000000000000000000000..4024b039efc80fadbc7e23ed1d352208b859387c --- /dev/null +++ b/V13_SMOKE_VALIDATION.md @@ -0,0 +1,7 @@ +# V13 smoke validation + +Validated before packaging: + +- Full test suite: 25 passed (one pre-existing tensor-to-scalar warning in the stability test). +- End-to-end CPU smoke run: seed 0, `swm_structured_cf_v9` and `no_spectral_transition`, one epoch, two paired episodes, horizon 5, collision horizon 5. +- The smoke run produced both axis-equivariance and forced-collision outputs successfully; generated smoke results were removed before packaging. diff --git a/V14_EXTERNAL_CONTROL_BENCHMARK.md b/V14_EXTERNAL_CONTROL_BENCHMARK.md new file mode 100644 index 0000000000000000000000000000000000000000..3b5dc26a4f137258ec0dd6827f9ca1816f5c73bc --- /dev/null +++ b/V14_EXTERNAL_CONTROL_BENCHMARK.md @@ -0,0 +1,45 @@ +# V14 — External Control Benchmark + +V14 is the first SWM milestone outside the project's synthetic Moving Shapes / hard-controlled-dynamics simulator. It freezes the V9 architecture family and evaluates fresh training runs on established Gymnasium Classic Control environments. + +## Environments + +- `CartPole-v1` +- `MountainCar-v0` +- `Acrobot-v1` + +Each environment is rendered with Gymnasium's official `rgb_array` renderer, converted to 32×32 grayscale, and paired with a compact quantized text/state description. Models are trained **from scratch per environment**. This is architecture transfer, not checkpoint transfer. + +## Frozen comparison + +The primary comparison is: + +- `swm_structured_cf_v9` — frozen structured SWM architecture alias +- `neural_operator` — Fourier image-state control +- `no_spectral_transition` — spectral state without the structured spectral transition + +No V12/V13 diagnostic result is used to tune the V14 architecture. + +## Primary endpoint + +Long-horizon pixel prediction MSE at H=5,10,20,30, reported per seed and as mean±SD. The benchmark is deliberately simple: it asks whether the V9 architecture's long-horizon advantage or failure modes survive outside the project's own simulator. + +## Run + +Install the external benchmark dependency: + +`pip install -e '.[external]'` + +Five-seed full suite: + +`PYTHONPATH=. python scripts/run_v14_external_benchmark.py --seeds 0 1 2 3 4 --epochs 5 --device cuda` + +Ten-seed publication run: + +`PYTHONPATH=. python scripts/run_v14_external_benchmark.py --seeds 0 1 2 3 4 5 6 7 8 9 --epochs 5 --device cuda` + +Outputs are written under `results/v14_external_control/`. + +## Interpretation rules + +V14 is an external benchmark milestone, not a claim that Classic Control fully represents real-world world-modeling. A favorable result must be replicated across environments and seeds. A negative result is informative because V9 is frozen rather than retuned to the external suite. diff --git a/V15_1_BUGFIX.md b/V15_1_BUGFIX.md new file mode 100644 index 0000000000000000000000000000000000000000..15f8812aeec62515a65f2eead419f65090f008b8 --- /dev/null +++ b/V15_1_BUGFIX.md @@ -0,0 +1,34 @@ +# V15.1 — External Dynamics Diagnostic Bug-Fix Release + +V15.1 is a **bug-fixed replacement for V15**, not a new research iteration and not a new model version. +It uses the same frozen V14 checkpoints and the same diagnostic protocol. + +## Bug fixed + +V15 called `model.text_decoder(z)` directly. `text_decoder` is a flat linear layer whose raw output has shape +`[B, max_text_len * vocab_size]`. V15 then took `argmax(-1)` and compared that result with the environment-state +token tensor `[B, max_text_len]`. This caused the observed dimension mismatch and, more importantly, would not +have been a valid state-token diagnostic. + +V15.1 calls `model.decode_text(z)`, the model's canonical decoder API. It reshapes logits to +`[B, max_text_len, vocab_size]`; `argmax(-1)` therefore yields `[B, max_text_len]`, exactly matching the discretized +Gym observation tokens used as the target. Padding tokens are excluded from accuracy. + +An explicit shape assertion now fails early with an informative error if a future model violates this contract. + +## Interpretation + +`token_acc_h*` is **discretized Gym-state token accuracy**, not continuous-state regression accuracy. The tokens were +created by V14 from the Gym observation vector using the fixed binning procedure in `obs_to_tokens`. It is a +state-sensitive auxiliary diagnostic because the model is trained to predict these tokens, but it should not be +reported as exact continuous environment-state error. + +No V9 architecture, training objective, V14 checkpoint, dataset, horizon, or primary diagnostic has been changed. + +## Run + +`PYTHONPATH=. python scripts/run_v15_external_diagnostics.py --v14-results ../Spectral-World-Models-HF-v14/results/v14_external_control --seeds 0 1 2 3 4 5 6 7 8 9 --device cuda` + +Outputs remain: +- `results/v15_external_diagnostics/diagnostics_by_seed.csv` +- `results/v15_external_diagnostics/diagnostics_summary.csv` diff --git a/V15_EXTERNAL_DYNAMICS_DIAGNOSTIC.md b/V15_EXTERNAL_DYNAMICS_DIAGNOSTIC.md new file mode 100644 index 0000000000000000000000000000000000000000..85fbe2f9d6f740f0e60fbbf0c65cc49e8dd56097 --- /dev/null +++ b/V15_EXTERNAL_DYNAMICS_DIAGNOSTIC.md @@ -0,0 +1,14 @@ +# V15 — External Dynamics Diagnostic + +V15 is a **stop/go diagnostic milestone**, not another architecture iteration. It freezes and reloads the exact V14 checkpoints. There is no retraining and no V15 model. + +## Question +Did V14 expose a genuine dynamics failure, or is full-frame MSE dominated by sparse moving foregrounds and static backgrounds? + +## Diagnostics +For H5/H10/H20/H30 V15 reports full-frame MSE, static-frame baseline MSE, foreground-weighted MSE (pixels whose oracle frame differs from the initial frame), one-step-at-horizon motion-increment MSE, PSNR, dependency-free SSIM, latent rollout cosine, and state-token accuracy. The state-token metric uses the Gym observation bins already supplied to every V14 model; it is a coarse state diagnostic, **not continuous state RMSE**. + +For models exposing stability diagnostics, V15 also records spectral/operator diagnostics per seed so catastrophic rollout seeds can be compared with transition statistics. + +## Decision rule +This is the final diagnostic before a decision. If V9 remains substantially worse than controls on foreground motion, latent consistency, and state-token prediction, treat V14 as a genuine external-transfer failure and stop iterating on this architecture. If the gap largely disappears on dynamics-sensitive metrics while static prediction is competitive on full-frame MSE, revise the interpretation of V14 and move directly to a more appropriate external benchmark/task metric. Do not create V16 merely to optimize these three environments. diff --git a/V4_MULTIMODAL_STABILITY.md b/V4_MULTIMODAL_STABILITY.md new file mode 100644 index 0000000000000000000000000000000000000000..3d065546ccbc4c29527562b6e93e8270fbd8569e --- /dev/null +++ b/V4_MULTIMODAL_STABILITY.md @@ -0,0 +1,19 @@ +# SWM v4 — Selective Spectral Regulation + Multimodal Rollout Evaluation + +v3 showed that a hard `rho=0.95` contraction cap was numerically enforced but damaged predictive rollouts, while the original stability penalty was empirically inert. v4 therefore tests a different hypothesis: preserve near-unit transport modes and constrain only genuinely expansive modes. + +## New model: `swm_selective` + +The action-conditioned low-rank operator is decomposed with an SVD. Singular values at or below `max_gain=1.05` are left unchanged; only values above the threshold are clipped. The nonlinear residual is retained. This is intentionally not a claim that 1.05 is theoretically optimal; it is a diagnostic architecture designed to separate selective control of expansive modes from uniform contraction. + +## New long-horizon metrics + +For every requested horizon, the benchmark reports image rollout MSE, text accuracy decoded from the recurrent latent, cosine alignment between recurrent and target multimodal latent states, and a bounded joint diagnostic score. The components remain the primary metrics; the joint score is only a compact diagnostic and should not be used as a sole publication claim. + +## Recommended DGX Spark experiment + +`PYTHONPATH=. python scripts/run_stability_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --device cuda` + +Output: `results/stability_attribution/metrics_summary.csv` + +The key comparisons are `swm_selective` vs `swm`, `swm_spectral_norm`, `no_spectral_transition`, `neural_operator`, and `koopman`. diff --git a/V5_HARD_DYNAMICS.md b/V5_HARD_DYNAMICS.md new file mode 100644 index 0000000000000000000000000000000000000000..67eefb4f5422fd53f221fb25068da92567faf84e --- /dev/null +++ b/V5_HARD_DYNAMICS.md @@ -0,0 +1,13 @@ +# V5 — Hard Controlled Dynamics + +V4 showed that selective spectral regulation repaired the failure of global contraction, but the no-spectral-transition control remained highly competitive. V5 therefore changes the **benchmark difficulty rather than the stability mechanism**. + +The new benchmark preserves the same image/text/action API while adding inertial velocity, action-conditioned acceleration, wall bounces, a moving distractor, a central occluder, stochastic target occlusion, and longer training sequences. This is designed to make simple latent persistence insufficient. + +## Primary experiment + +`PYTHONPATH=. python scripts/run_v5_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --device cuda` + +Primary output: `results/v5_hard_dynamics/metrics_summary.csv`. + +Interpret the benchmark as an attribution experiment. Compare `swm_selective` with `swm`, `no_spectral_transition`, `koopman`, and `neural_operator`. Report image MSE, recurrent text accuracy, latent cosine separately; the joint score is diagnostic only. diff --git a/V6_STRUCTURED_TRANSPORT.md b/V6_STRUCTURED_TRANSPORT.md new file mode 100644 index 0000000000000000000000000000000000000000..e621b6d137f4690de376fb440109580073c7e640 --- /dev/null +++ b/V6_STRUCTURED_TRANSPORT.md @@ -0,0 +1,11 @@ +# V6 — Structured spectral transport and counterfactual control + +V6 follows the V5 finding that selective clipping prevents catastrophic rollout growth, while the no-spectral-transition control remains highly competitive. It therefore changes the transition hypothesis rather than increasing benchmark difficulty again. + +`swm_structured` decomposes latent dynamics into: (1) an approximately energy-preserving Cayley/orthogonal transport block, (2) learned diagonal dissipation, and (3) low-rank action-conditioned transport. Only expansive singular modes of the composed operator are clipped at 1.05. + +The new counterfactual benchmark fixes the same initial observation and compares opposing persistent action sequences. It reports final counterfactual image MSE, cosine agreement of the predicted versus true action-effect displacement vector, and action-effect magnitude ratio. This is intended to distinguish causal control sensitivity from simple persistence. + +Run: `PYTHONPATH=. python scripts/run_v6_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --cf-horizon 30 --device cuda` + +Primary output: `results/v6_structured_transport/metrics_summary.csv`. diff --git a/V7_COUNTERFACTUAL_TRAINING.md b/V7_COUNTERFACTUAL_TRAINING.md new file mode 100644 index 0000000000000000000000000000000000000000..9c7ab7aec1aa3566b987670972331632f2988cde --- /dev/null +++ b/V7_COUNTERFACTUAL_TRAINING.md @@ -0,0 +1,13 @@ +# V7 — Counterfactual Action-Contrastive Training + +V7 keeps the V6 structured transport architecture and changes the learning objective. V6 was very stable at H=100 but under-reacted to interventions: the structured model's counterfactual effect-vector cosine was about 0.44 and its effect-magnitude ratio about 0.021. V7 therefore trains an explicit paired-intervention objective. + +For the same initial image/text state, the auxiliary training batch rolls out persistent left and right interventions. The loss has three terms: (1) direction loss, `1 - cosine(delta_pred, delta_true)`; (2) magnitude loss, the absolute log ratio of predicted to true action-effect magnitude; and (3) branch reconstruction MSE for both counterfactual endpoints. The default weights are deliberately modest so the counterfactual objective augments rather than replaces next-state prediction. + +The primary comparison is `swm_structured_cf` against the unchanged `swm_structured`, `swm_selective`, `no_spectral_transition`, and `neural_operator` controls. The target is not a single winner metric: V7 should improve causal action sensitivity while preserving the long-horizon stability achieved by V6. + +Run: + +`PYTHONPATH=. python scripts/run_v7_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --cf-horizon 30 --device cuda` + +Output: `results/v7_counterfactual_training/metrics_summary.csv`. diff --git a/V8_GENERALIZATION_AND_PLANNING.md b/V8_GENERALIZATION_AND_PLANNING.md new file mode 100644 index 0000000000000000000000000000000000000000..ad429c3c94f295e37093d6e3b53fc5935175a614 --- /dev/null +++ b/V8_GENERALIZATION_AND_PLANNING.md @@ -0,0 +1,25 @@ +# V8 — Frozen Architecture: OOD Generalization, Causal Control, and Planning + +V8 freezes the V7 model architectures and changes the scientific question. V7 showed that explicit counterfactual training can produce accurate action-effect direction and magnitude while preserving stable long-horizon rollouts. V8 tests whether that behavior generalizes rather than adding another transition mechanism. + +## Distribution shift + +Training remains on the V5/V7 controlled-dynamics distribution (acceleration 0.70, max speed 3.0, occlusion probability 0.12). OOD evaluation increases acceleration to 1.10, max speed to 4.0, and occlusion probability to 0.28. No OOD examples are used for optimization or model selection. + +## Three evaluation axes + +1. **ID/OOD prediction and rollout:** one-step MSE/text accuracy and H=5,10,20,30,50,100 image MSE, recurrent text accuracy, and latent cosine. +2. **ID/OOD interventions:** paired left/right action interventions report endpoint image MSE, action-effect vector cosine, and action-effect magnitude ratio. +3. **Imagination-based planning:** from the same observed initial state, each model imagines a fixed library of candidate action sequences toward a randomly selected corner target. The chosen sequence is executed in the ground-truth simulator. Primary planning metrics are normalized regret (lower is better) and oracle-action agreement (higher is better). + +The planning candidate set is identical for every model and seed. V8 does not train on the planning targets. + +## Main command + +`PYTHONPATH=. python scripts/run_v8_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --cf-horizon 30 --planning-horizon 20 --planning-episodes 96 --device cuda` + +Primary output: `results/v8_generalization/metrics_summary.csv`. + +## Interpretation discipline + +V8 is designed as a generalization/falsification experiment. Strong ID counterfactual performance alone is insufficient. The structured counterfactual SWM is supported only if its intervention behavior survives OOD shift and, separately, if its imagined rollouts produce useful action selection. Planning metrics should be reported independently from reconstruction metrics; no arbitrary composite winner score is used. diff --git a/V8_SMOKE_VALIDATION.md b/V8_SMOKE_VALIDATION.md new file mode 100644 index 0000000000000000000000000000000000000000..6747923d3bd7ba08b922708af029ee64c825bc0a --- /dev/null +++ b/V8_SMOKE_VALIDATION.md @@ -0,0 +1,9 @@ +# V8 validation + +Repository validation completed before packaging: + +- `PYTHONPATH=. python -m pytest -q` — 12 tests passed. +- `PYTHONPATH=. python scripts/run_v8_benchmark.py --help` — CLI parsed successfully. +- End-to-end CPU smoke benchmark completed for `swm_structured_cf` and `no_spectral_transition` with a one-seed/two-step configuration. + +The smoke outputs were removed before packaging so the five-seed DGX run writes a clean `results/v8_generalization/` directory. diff --git a/V9_CAUSAL_GENERALIZATION.md b/V9_CAUSAL_GENERALIZATION.md new file mode 100644 index 0000000000000000000000000000000000000000..a59f9f16dfde1f2a69878fa0eef51aef52350139 --- /dev/null +++ b/V9_CAUSAL_GENERALIZATION.md @@ -0,0 +1,21 @@ +# V9 — Causal Generalization + +V8 showed that near-perfect in-distribution counterfactual alignment did not survive a physics shift. V9 freezes the structured transport design and changes the training/evaluation protocol rather than adding another stability operator. + +## What changes + +- Training data is a mixture of three controlled-dynamics regimes spanning acceleration, speed, occlusion, damping, and restitution. +- `swm_structured_cf_v9` uses the same structured transition as V7 but replaces endpoint-only counterfactual supervision with a discounted trajectory-level intervention loss. +- Evaluation is separated into **interpolation**, **parameter extrapolation**, and **structural OOD** (changed damping/restitution). +- Every regime reports one-step prediction, H=5/10/20/30/50/100 rollout metrics, counterfactual direction/magnitude/endpoint error, and planning regret/oracle agreement. +- Controls remain neural operator, selective SWM, endpoint-CF structured SWM, and no-spectral-transition. + +## Main hypothesis + +Trajectory-level counterfactual training across a family of physics regimes should reduce the V8 gap between ID causal response and OOD causal response without sacrificing the stable long-horizon rollout behavior of structured SWM. + +## DGX Spark command + +`PYTHONPATH=. python scripts/run_v9_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --cf-horizon 30 --planning-horizon 20 --planning-episodes 64 --device cuda` + +Primary output: `results/v9_causal_generalization/metrics_summary.csv`. diff --git a/configs/prototype.json b/configs/prototype.json new file mode 100644 index 0000000000000000000000000000000000000000..6e34f9e3292b84b5657a7e483f7e6bbccf271e03 --- /dev/null +++ b/configs/prototype.json @@ -0,0 +1 @@ +{"model":"swm","image_size":32,"max_text_len":5,"latent_dim":96,"image_latent_dim":48,"text_latent_dim":32,"benchmark":"mini_moving_shapes","epochs":3,"batch_size":64,"seed":13} diff --git a/data/generate_benchmark.py b/data/generate_benchmark.py new file mode 100644 index 0000000000000000000000000000000000000000..7476b5800cdce6a88be6722655c4b24f1ae614f9 --- /dev/null +++ b/data/generate_benchmark.py @@ -0,0 +1,7 @@ +from pathlib import Path +from spectral_world_models.data import DatasetConfig, generate_benchmark_npz + +if __name__ == "__main__": + out = Path(__file__).resolve().parent / "mini_moving_shapes.npz" + generate_benchmark_npz(out, DatasetConfig()) + print(f"wrote {out}") diff --git a/data/hard_controlled_dynamics_v6.npz b/data/hard_controlled_dynamics_v6.npz new file mode 100644 index 0000000000000000000000000000000000000000..2121cd0f0e7942fb7b5fa6edf0c3df7609bd4d89 --- /dev/null +++ b/data/hard_controlled_dynamics_v6.npz @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3688d9b074059a297d2f9163a93b8a07a5aaeaf5a94f5e7b7792421350acbd71 +size 25959244 diff --git a/data/hard_controlled_dynamics_v6_rollout_h6.npz b/data/hard_controlled_dynamics_v6_rollout_h6.npz new file mode 100644 index 0000000000000000000000000000000000000000..bb83960a423d8cd7526ff29a7a48d85b00e73917 --- /dev/null +++ b/data/hard_controlled_dynamics_v6_rollout_h6.npz @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:15b91073dfd882b528416bb1fb53d08c53eeaae201fbd468ac9f1bce041b73b4 +size 1361864 diff --git a/data/mini_moving_shapes.npz b/data/mini_moving_shapes.npz new file mode 100644 index 0000000000000000000000000000000000000000..b6f97c1ca9ba54619b338283c43dbf044ea6f80c --- /dev/null +++ b/data/mini_moving_shapes.npz @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e3dea41e2fd4303760e7abb526f4404434cfca74093d45df235a27f7828cc190 +size 6822485 diff --git a/data/mini_moving_shapes_rollout_h11.npz b/data/mini_moving_shapes_rollout_h11.npz new file mode 100644 index 0000000000000000000000000000000000000000..af3e36c57e3574a8ebf0c2276601f046c4476f92 --- /dev/null +++ b/data/mini_moving_shapes_rollout_h11.npz @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4593ad92dab9ff7ffc6250d368a7b6eb68ab1d09c3bae749209f6dbc4cbcf491 +size 1617145 diff --git a/data/mini_moving_shapes_rollout_h31.npz b/data/mini_moving_shapes_rollout_h31.npz new file mode 100644 index 0000000000000000000000000000000000000000..af3e36c57e3574a8ebf0c2276601f046c4476f92 --- /dev/null +++ b/data/mini_moving_shapes_rollout_h31.npz @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4593ad92dab9ff7ffc6250d368a7b6eb68ab1d09c3bae749209f6dbc4cbcf491 +size 1617145 diff --git a/data/v9_extrapolation_h6.npz b/data/v9_extrapolation_h6.npz new file mode 100644 index 0000000000000000000000000000000000000000..48d63582229566a98e46bfd93dce28728451cbf1 --- /dev/null +++ b/data/v9_extrapolation_h6.npz @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7dcc2e791d591c7810dc2295779e816e3b4cf38e8554397c33a4949538860143 +size 9733974 diff --git a/data/v9_interpolation_h6.npz b/data/v9_interpolation_h6.npz new file mode 100644 index 0000000000000000000000000000000000000000..3e6d5b054f7ca69fada0c255487fc4332f8b8693 --- /dev/null +++ b/data/v9_interpolation_h6.npz @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0aab7d41b0596bd72a24100710f6f7aafe8a94289ed33a6bd921fcbcee0c97be +size 9730656 diff --git a/data/v9_structural_ood_h6.npz b/data/v9_structural_ood_h6.npz new file mode 100644 index 0000000000000000000000000000000000000000..b177a4415db132dcbf06b60a7425625e8ab03f76 --- /dev/null +++ b/data/v9_structural_ood_h6.npz @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:288cd80b247bb1396897579bd2b86363c1936d60e904431a22eee9a0642c42c6 +size 9741893 diff --git a/data/v9_train_family.npz b/data/v9_train_family.npz new file mode 100644 index 0000000000000000000000000000000000000000..26bf98bc3fe226ef56411914fa858fcfde5b07a7 --- /dev/null +++ b/data/v9_train_family.npz @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d26d2a2c08b9c84f7613e31099b6274e29a326f58f7dadb50b3ce4dc5d9d1181 +size 28287014 diff --git a/docs/HF_RELEASE_STEPS.md b/docs/HF_RELEASE_STEPS.md new file mode 100644 index 0000000000000000000000000000000000000000..10eb8052d73497a70524726713b84842a9221cb4 --- /dev/null +++ b/docs/HF_RELEASE_STEPS.md @@ -0,0 +1,14 @@ +# Hugging Face release steps + +Suggested repository ID: `kiruluta/Spectral-World-Models-Reproducibility` + +1. Install/update CLI: `pip install -U huggingface_hub` +2. Authenticate: `hf auth login` +3. Create repository: `hf repos create kiruluta/Spectral-World-Models-Reproducibility --exist-ok` +4. From the directory containing this folder, upload it to repository root: + `hf upload kiruluta/Spectral-World-Models-Reproducibility ./Spectral-World-Models-HF-Reproducibility-Final . --exclude '.pytest_cache/**' --exclude '**/__pycache__/**' --exclude '*.pyc'` +5. Verify README, source, scripts, tests, versioned V6–V15.1 documents, and `results/` on the Hub. +6. Create a frozen tag after verification: + `hf repos tag create kiruluta/Spectral-World-Models-Reproducibility v15.1-final -m "Final SWM reproducibility release through V15.1"` + +Do not upload access tokens, virtual environments, caches, or unrelated local checkpoints. diff --git a/docs/REPRODUCIBILITY_RELEASE.md b/docs/REPRODUCIBILITY_RELEASE.md new file mode 100644 index 0000000000000000000000000000000000000000..669d77758512e1249fb7b167bb87a0da53d49f04 --- /dev/null +++ b/docs/REPRODUCIBILITY_RELEASE.md @@ -0,0 +1,36 @@ +# Spectral World Models — Final Reproducibility Release + +This repository freezes the Spectral World Models experimental program at V15.1. It is a reproducibility artifact, not a claim that the final structured SWM outperforms all baselines. + +## Experimental history + +- **V6 — Structured Transport:** separated transport, dissipation, and action-conditioned response; exposed the tension between stability and causal responsiveness. +- **V7 — Counterfactual Training:** restored strong in-distribution intervention direction/magnitude on the synthetic benchmark. +- **V8 — Generalization and Planning:** showed that strong in-distribution counterfactual response did not automatically generalize under shifted physics. +- **V9 — Causal Generalization:** introduced multi-physics trajectory supervision and became the frozen principal SWM variant used in later audits. +- **V10 — Counterfactual Integrity Audit:** verified that interpolation, extrapolation, and structural-OOD regimes generated genuinely different oracle trajectories; trajectory-level metrics replaced insensitive endpoint-only interpretation. +- **V11 — Causal Gain Calibration:** tested scale-aware calibration; reduced some log-gain errors but tended to suppress intervention magnitude, so V9 remained the principal variant. +- **V12 — Causal Response Decomposition:** paired/stratified analysis found an aggregate V9 trajectory-direction advantage over the neural-operator and no-transition controls while exposing a severe up/down action failure. +- **V13 — Axis Symmetry & Equivariance:** controlled rotations/reflections/x-y swaps showed that the vertical failure was learned axis anisotropy rather than asymmetric simulator physics; forced-collision evaluation repaired the V12 contact-data gap. +- **V14 — External Control Benchmark:** froze V9 and moved to CartPole-v1, MountainCar-v0, and Acrobot-v1. V9 did not transfer competitively on long-horizon full-frame rollout MSE. +- **V15/V15.1 — External Dynamics Diagnostic:** tested whether V14 was mainly a static-background metric artifact. Foreground, motion, latent, and discretized-state diagnostics supported a genuine external dynamics-transfer limitation. V15.1 fixes the state-token diagnostic shape/API bug without changing the model or protocol. + +## Included measured result files + +This package contains the measured CSVs available from the final audit sequence: + +- `results/v12_causal_decomposition/stratified_metrics_summary.csv` +- `results/v12_causal_decomposition/paired_bootstrap_v9_vs_controls.csv` +- `results/v13_axis_equivariance/axis_equivariance_summary.csv` +- `results/v13_axis_equivariance/forced_collision_summary.csv` +- `results/v14_external_control/metrics_summary.csv` +- `results/v15_external_diagnostics/diagnostics_summary.csv` +- `results/v15_external_diagnostics/diagnostics_by_seed.csv` + +Earlier V6–V11 protocols and rationale are preserved in their versioned Markdown documents in the repository. Raw per-run artifacts for those earlier versions are not reconstructed here unless already present in the source tree; this release does not fabricate missing historical files. + +## Final scientific interpretation + +The experiments support a narrower conclusion than the original architectural hypothesis. Structured spectral/operator dynamics can yield stable and useful counterfactual behavior in controlled synthetic settings, but the frozen V9 model developed strong coordinate anisotropy and did not transfer competitively to the three external Classic Control visual-dynamics tasks. V15.1 indicates that this external result is not explained solely by static-background bias in full-frame pixel MSE. + +This release intentionally stops architecture iteration after V15.1. diff --git a/hf/configuration_swm.py b/hf/configuration_swm.py new file mode 100644 index 0000000000000000000000000000000000000000..ba00419636031e1e89bc3d604aa814a9d1246a33 --- /dev/null +++ b/hf/configuration_swm.py @@ -0,0 +1,10 @@ +from transformers import PretrainedConfig + +class SpectralWorldModelConfig(PretrainedConfig): + model_type = "spectral_world_model" + def __init__(self, image_size=32, max_text_len=5, vocab_size=15, action_dim=5, + latent_dim=96, image_latent_dim=48, text_latent_dim=32, **kwargs): + super().__init__(**kwargs) + self.image_size=image_size; self.max_text_len=max_text_len; self.vocab_size=vocab_size + self.action_dim=action_dim; self.latent_dim=latent_dim + self.image_latent_dim=image_latent_dim; self.text_latent_dim=text_latent_dim diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000000000000000000000000000000000000..b2dc5b9705f8ebac39bce430ab8104a07e6eae46 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,24 @@ +[build-system] +requires = ["setuptools>=68", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "spectral-world-models" +version = "0.1.0" +description = "Spectral World Models: multimodal operator learning in Hilbert space for imagination-based planning" +readme = "README.md" +requires-python = ">=3.10" +authors = [{name="Andrew J. Kiruluta"}] +dependencies = [ + "torch>=2.0", + "numpy>=1.23", + "tqdm>=4.0", + "matplotlib>=3.6" +] + +[tool.setuptools.packages.find] +where = ["."] +include = ["spectral_world_models*"] + +[project.optional-dependencies] +external = ["gymnasium[classic-control]>=1.0,<2"] diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000000000000000000000000000000000000..65bd3ff9689920f4b7c4efcaf7f688e7de3470fd --- /dev/null +++ b/requirements.txt @@ -0,0 +1,4 @@ +torch>=2.0 +numpy>=1.23 +tqdm>=4.0 +matplotlib>=3.6 diff --git a/results/README.md b/results/README.md new file mode 100644 index 0000000000000000000000000000000000000000..f302a47a8b3483deaf84ab7070b827f5e668680a --- /dev/null +++ b/results/README.md @@ -0,0 +1,19 @@ +# Quick Benchmark Results +These metrics were generated by running `scripts/run_quick_benchmark.py` on CPU with the bundled Mini Moving Shapes dataset. They are intended as reproducibility smoke-test metrics, not final paper-grade multi-seed numbers. +## Main baselines +| Method | Params | Test PSNR ↑ | Test SSIM ↑ | Test NLL ↓ | Rollout MSE@5 ↓ | Time/step ms ↓ | +|---|---:|---:|---:|---:|---:|---:| +| Dreamer-style latent model | 497,682 | 14.571 | 0.072 | 0.0107 | 0.03490 | 0.0528 | +| Transformer latent predictor | 503,322 | 14.368 | 0.017 | 0.0687 | 0.03657 | 0.0290 | +| Koopman baseline | 465,306 | 14.619 | 0.090 | -0.0027 | 0.03487 | 0.0170 | +| Neural-operator baseline | 403,794 | 14.534 | 0.058 | 0.0210 | 0.03511 | 0.0194 | +| Spectral World Model (ours) | 405,422 | 14.591 | 0.080 | 0.0050 | 0.03577 | 0.0343 | + +## Ablations +| Variant | Params | Test PSNR ↑ | Test NLL ↓ | Rollout MSE@5 ↓ | Time/step ms ↓ | +|---|---:|---:|---:|---:|---:| +| Spectral World Model (ours) | 405,422 | 14.591 | 0.0050 | 0.03577 | 0.0343 | +| No spectral state | 497,682 | 14.567 | 0.0117 | 0.03492 | 0.0184 | +| No spectral transition | 446,802 | 14.562 | 0.0131 | 0.03505 | 0.0318 | +| No text branch | 402,350 | 14.729 | -0.0325 | 0.03508 | 0.0302 | +| No stability penalty | 405,422 | 14.433 | 0.0498 | 0.03609 | 0.0294 | diff --git a/results/dreamer.pt b/results/dreamer.pt new file mode 100644 index 0000000000000000000000000000000000000000..fd0814e4f98e16cfd701f09538c1d923c417b688 --- /dev/null +++ b/results/dreamer.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7754579118d0a89cfb2781b7e97f5b2c4f02dc950cc6269339ec7daefcd04ecc +size 2002301 diff --git a/results/dreamer_history.csv b/results/dreamer_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..a8eb3693ec02dd423ae9242b1957367f3c146f42 --- /dev/null +++ b/results/dreamer_history.csv @@ -0,0 +1,4 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +dreamer,1,0.23922246268817357,0.18563266098499298,0.03763256222009659,14.244362116517717,0.00794300902634859,0.1053612008690834,0.45267857142857143 +dreamer,2,0.2113134688032525,0.11774692365101405,0.03554206341505051,14.49257362581445,0.054459843784570694,0.03277435526251793,0.4200892857142857 +dreamer,3,0.11036801284977368,0.10580005390303475,0.035179004073143005,14.537164596955208,0.07900667190551758,0.02016819454729557,0.5526785714285715 diff --git a/results/dreamer_metrics.json b/results/dreamer_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..1229ea387e0a407408625d83421f7fbc7c9a88e9 --- /dev/null +++ b/results/dreamer_metrics.json @@ -0,0 +1,61 @@ +{ + "model": "dreamer", + "params": 497682, + "epochs": 3, + "best_val_loss": 0.10580005390303475, + "val": { + "loss": 0.10580005390303475, + "mse": 0.035179004073143005, + "psnr": 14.537164596955208, + "ssim": 0.07900667190551758, + "nll": 0.02016819454729557, + "text_acc": 0.5526785714285715 + }, + "test": { + "loss": 0.10651075627122607, + "mse": 0.03490705043077469, + "psnr": 14.570868465670038, + "ssim": 0.07190005481243134, + "nll": 0.010725376196205616, + "text_acc": 0.5008928571428571 + }, + "rollout": { + "rollout_mse": 0.034899838268756866, + "time_per_rollout_step_ms": 0.052804646875070915 + }, + "history": [ + { + "model": "dreamer", + "epoch": 1, + "train_loss": 0.23922246268817357, + "val_loss": 0.18563266098499298, + "val_mse": 0.03763256222009659, + "val_psnr": 14.244362116517717, + "val_ssim": 0.00794300902634859, + "val_nll": 0.1053612008690834, + "val_text_acc": 0.45267857142857143 + }, + { + "model": "dreamer", + "epoch": 2, + "train_loss": 0.2113134688032525, + "val_loss": 0.11774692365101405, + "val_mse": 0.03554206341505051, + "val_psnr": 14.49257362581445, + "val_ssim": 0.054459843784570694, + "val_nll": 0.03277435526251793, + "val_text_acc": 0.4200892857142857 + }, + { + "model": "dreamer", + "epoch": 3, + "train_loss": 0.11036801284977368, + "val_loss": 0.10580005390303475, + "val_mse": 0.035179004073143005, + "val_psnr": 14.537164596955208, + "val_ssim": 0.07900667190551758, + "val_nll": 0.02016819454729557, + "val_text_acc": 0.5526785714285715 + } + ] +} \ No newline at end of file diff --git a/results/koopman.pt b/results/koopman.pt new file mode 100644 index 0000000000000000000000000000000000000000..198837765ad1e7ca39f09ae4621ffe524d1ceb2b --- /dev/null +++ b/results/koopman.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a24d327994545a03f46d9f9db6b7fe6ea29b1bf407bcaac76bcb88f897377a23 +size 1870988 diff --git a/results/koopman_history.csv b/results/koopman_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..86c39f6b22b8bc32bdaf767a74b8565d805b484f --- /dev/null +++ b/results/koopman_history.csv @@ -0,0 +1,4 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +koopman,1,0.1982219240495137,0.1363554298877716,0.036813490092754364,14.339930074331706,0.022857418283820152,0.07692114263772964,0.5790178571428571 +koopman,2,0.12113601448280471,0.11492279065506798,0.035316966474056244,14.520166069109159,0.06720736622810364,0.024958457797765732,0.7040178571428571 +koopman,3,0.10989103439663138,0.10961772288594927,0.034873176366090775,14.575084938216127,0.09624947607517242,0.009549240581691265,0.8486607142857143 diff --git a/results/koopman_metrics.json b/results/koopman_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..1a850d32f4790753096f19b914d1b7649f634951 --- /dev/null +++ b/results/koopman_metrics.json @@ -0,0 +1,61 @@ +{ + "model": "koopman", + "params": 465306, + "epochs": 3, + "best_val_loss": 0.10961772288594927, + "val": { + "loss": 0.10961772288594927, + "mse": 0.034873176366090775, + "psnr": 14.575084938216127, + "ssim": 0.09624947607517242, + "nll": 0.009549240581691265, + "text_acc": 0.8486607142857143 + }, + "test": { + "loss": 0.1099218875169754, + "mse": 0.034520506858825684, + "psnr": 14.61922836236809, + "ssim": 0.0897933691740036, + "nll": -0.0026963024865835905, + "text_acc": 0.8419642857142857 + }, + "rollout": { + "rollout_mse": 0.034873973578214645, + "time_per_rollout_step_ms": 0.01702466562498728 + }, + "history": [ + { + "model": "koopman", + "epoch": 1, + "train_loss": 0.1982219240495137, + "val_loss": 0.1363554298877716, + "val_mse": 0.036813490092754364, + "val_psnr": 14.339930074331706, + "val_ssim": 0.022857418283820152, + "val_nll": 0.07692114263772964, + "val_text_acc": 0.5790178571428571 + }, + { + "model": "koopman", + "epoch": 2, + "train_loss": 0.12113601448280471, + "val_loss": 0.11492279065506798, + "val_mse": 0.035316966474056244, + "val_psnr": 14.520166069109159, + "val_ssim": 0.06720736622810364, + "val_nll": 0.024958457797765732, + "val_text_acc": 0.7040178571428571 + }, + { + "model": "koopman", + "epoch": 3, + "train_loss": 0.10989103439663138, + "val_loss": 0.10961772288594927, + "val_mse": 0.034873176366090775, + "val_psnr": 14.575084938216127, + "val_ssim": 0.09624947607517242, + "val_nll": 0.009549240581691265, + "val_text_acc": 0.8486607142857143 + } + ] +} \ No newline at end of file diff --git a/results/metrics.csv b/results/metrics.csv new file mode 100644 index 0000000000000000000000000000000000000000..b1520d37812b4e271e6e2d26cdb93ddce3cb00dd --- /dev/null +++ b/results/metrics.csv @@ -0,0 +1,10 @@ +model,params,epochs,val_loss,val_psnr,val_ssim,val_nll,val_text_acc,test_loss,test_psnr,test_ssim,test_nll,test_text_acc,rollout_mse_h5,time_per_rollout_step_ms +dreamer,497682,3,0.10580005390303475,14.537164596955208,0.07900667190551758,0.02016819454729557,0.5526785714285715,0.10651075627122607,14.570868465670038,0.07190005481243134,0.010725376196205616,0.5008928571428571,0.034899838268756866,0.052804646875070915 +transformer,503322,3,0.13327171334198543,14.30404067586808,0.01684718206524849,0.08752818405628204,0.47098214285714285,0.13411018465246474,14.368045538061631,0.016545191407203674,0.06867272406816483,0.4120535714285714,0.036570727825164795,0.029027615625043524 +koopman,465306,3,0.10961772288594927,14.575084938216127,0.09624947607517242,0.009549240581691265,0.8486607142857143,0.1099218875169754,14.61922836236809,0.0897933691740036,-0.0026963024865835905,0.8419642857142857,0.034873973578214645,0.01702466562498728 +neural_operator,403794,3,0.12681868778807776,14.497417828561044,0.06291486322879791,0.0313987098634243,0.6058035714285714,0.12872901665312902,14.534133092218475,0.05829298496246338,0.02102111093699932,0.584375,0.035108547657728195,0.01941507500005102 +swm,405422,3,0.11073624342679977,14.562615702489506,0.0878821536898613,0.013030804693698883,0.8651785714285715,0.11030659292425428,14.591434429751091,0.08014871180057526,0.004999135155230761,0.8535714285714285,0.03577370569109917,0.03425489374997426 +no_spectral_state,497682,3,0.11848256630556923,14.530032390072872,0.06655468046665192,0.022175762802362442,0.5571428571428572,0.11922914321933474,14.56741968803547,0.0609237365424633,0.011688289232552052,0.4861607142857143,0.0349217914044857,0.018417671874981778 +no_spectral_transition,446802,3,0.11065810599497386,14.530231662570795,0.07634083926677704,0.02211970090866089,0.7464285714285714,0.11198038820709501,14.562336300836341,0.06826915591955185,0.013108844868838787,0.765625,0.035049136728048325,0.03176636875004135 +no_text_branch,402350,3,0.1184177217738969,14.66014843918864,0.12382669746875763,-0.013937046751379967,0.725,0.11902656512601036,14.728564469992364,0.12929166853427887,-0.032495759427547455,0.7026785714285714,0.03507991507649422,0.03016817812504513 +no_stability_penalty,405422,2,0.12874157088143484,14.36634023825047,0.044199828058481216,0.06917144358158112,0.6120535714285714,0.13050702214241028,14.43291172530207,0.04435477778315544,0.04984518140554428,0.584375,0.03608756139874458,0.029396649999924307 diff --git a/results/metrics.json b/results/metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..9d59d76c126d237632816e58b6279b4434395889 --- /dev/null +++ b/results/metrics.json @@ -0,0 +1,543 @@ +{ + "note": "Actual quick CPU benchmark metrics generated in sandbox. These are smoke-test metrics, not paper-grade multi-seed results.", + "models": [ + { + "model": "dreamer", + "params": 497682, + "epochs": 3, + "best_val_loss": 0.10580005390303475, + "val": { + "loss": 0.10580005390303475, + "mse": 0.035179004073143005, + "psnr": 14.537164596955208, + "ssim": 0.07900667190551758, + "nll": 0.02016819454729557, + "text_acc": 0.5526785714285715 + }, + "test": { + "loss": 0.10651075627122607, + "mse": 0.03490705043077469, + "psnr": 14.570868465670038, + "ssim": 0.07190005481243134, + "nll": 0.010725376196205616, + "text_acc": 0.5008928571428571 + }, + "rollout": { + "rollout_mse": 0.034899838268756866, + "time_per_rollout_step_ms": 0.052804646875070915 + }, + "history": [ + { + "model": "dreamer", + "epoch": 1, + "train_loss": 0.23922246268817357, + "val_loss": 0.18563266098499298, + "val_mse": 0.03763256222009659, + "val_psnr": 14.244362116517717, + "val_ssim": 0.00794300902634859, + "val_nll": 0.1053612008690834, + "val_text_acc": 0.45267857142857143 + }, + { + "model": "dreamer", + "epoch": 2, + "train_loss": 0.2113134688032525, + "val_loss": 0.11774692365101405, + "val_mse": 0.03554206341505051, + "val_psnr": 14.49257362581445, + "val_ssim": 0.054459843784570694, + "val_nll": 0.03277435526251793, + "val_text_acc": 0.4200892857142857 + }, + { + "model": "dreamer", + "epoch": 3, + "train_loss": 0.11036801284977368, + "val_loss": 0.10580005390303475, + "val_mse": 0.035179004073143005, + "val_psnr": 14.537164596955208, + "val_ssim": 0.07900667190551758, + "val_nll": 0.02016819454729557, + "val_text_acc": 0.5526785714285715 + } + ] + }, + { + "model": "transformer", + "params": 503322, + "epochs": 3, + "best_val_loss": 0.13327171334198543, + "val": { + "loss": 0.13327171334198543, + "mse": 0.03711897134780884, + "psnr": 14.30404067586808, + "ssim": 0.01684718206524849, + "nll": 0.08752818405628204, + "text_acc": 0.47098214285714285 + }, + "test": { + "loss": 0.13411018465246474, + "mse": 0.036575935781002045, + "psnr": 14.368045538061631, + "ssim": 0.016545191407203674, + "nll": 0.06867272406816483, + "text_acc": 0.4120535714285714 + }, + "rollout": { + "rollout_mse": 0.036570727825164795, + "time_per_rollout_step_ms": 0.029027615625043524 + }, + "history": [ + { + "model": "transformer", + "epoch": 1, + "train_loss": 0.23709710474525178, + "val_loss": 0.13327171334198543, + "val_mse": 0.03711897134780884, + "val_psnr": 14.30404067586808, + "val_ssim": 0.01684718206524849, + "val_nll": 0.08752818405628204, + "val_text_acc": 0.47098214285714285 + }, + { + "model": "transformer", + "epoch": 2, + "train_loss": 0.16270889874015534, + "val_loss": 0.1869750597647258, + "val_mse": 0.035343486815690994, + "val_psnr": 14.516906073444048, + "val_ssim": 0.06779398024082184, + "val_nll": 0.025879332795739174, + "val_text_acc": 0.5580357142857143 + }, + { + "model": "transformer", + "epoch": 3, + "train_loss": 0.1466867652322565, + "val_loss": 0.13392244918005808, + "val_mse": 0.035102479159832, + "val_psnr": 14.54662209818714, + "val_ssim": 0.07712315022945404, + "val_nll": 0.01751108281314373, + "val_text_acc": 0.7321428571428571 + } + ] + }, + { + "model": "koopman", + "params": 465306, + "epochs": 3, + "best_val_loss": 0.10961772288594927, + "val": { + "loss": 0.10961772288594927, + "mse": 0.034873176366090775, + "psnr": 14.575084938216127, + "ssim": 0.09624947607517242, + "nll": 0.009549240581691265, + "text_acc": 0.8486607142857143 + }, + "test": { + "loss": 0.1099218875169754, + "mse": 0.034520506858825684, + "psnr": 14.61922836236809, + "ssim": 0.0897933691740036, + "nll": -0.0026963024865835905, + "text_acc": 0.8419642857142857 + }, + "rollout": { + "rollout_mse": 0.034873973578214645, + "time_per_rollout_step_ms": 0.01702466562498728 + }, + "history": [ + { + "model": "koopman", + "epoch": 1, + "train_loss": 0.1982219240495137, + "val_loss": 0.1363554298877716, + "val_mse": 0.036813490092754364, + "val_psnr": 14.339930074331706, + "val_ssim": 0.022857418283820152, + "val_nll": 0.07692114263772964, + "val_text_acc": 0.5790178571428571 + }, + { + "model": "koopman", + "epoch": 2, + "train_loss": 0.12113601448280471, + "val_loss": 0.11492279065506798, + "val_mse": 0.035316966474056244, + "val_psnr": 14.520166069109159, + "val_ssim": 0.06720736622810364, + "val_nll": 0.024958457797765732, + "val_text_acc": 0.7040178571428571 + }, + { + "model": "koopman", + "epoch": 3, + "train_loss": 0.10989103439663138, + "val_loss": 0.10961772288594927, + "val_mse": 0.034873176366090775, + "val_psnr": 14.575084938216127, + "val_ssim": 0.09624947607517242, + "val_nll": 0.009549240581691265, + "val_text_acc": 0.8486607142857143 + } + ] + }, + { + "model": "neural_operator", + "params": 403794, + "epochs": 3, + "best_val_loss": 0.12681868778807776, + "val": { + "loss": 0.12681868778807776, + "mse": 0.035502441227436066, + "psnr": 14.497417828561044, + "ssim": 0.06291486322879791, + "nll": 0.0313987098634243, + "text_acc": 0.6058035714285714 + }, + "test": { + "loss": 0.12872901665312902, + "mse": 0.03520356863737106, + "psnr": 14.534133092218475, + "ssim": 0.05829298496246338, + "nll": 0.02102111093699932, + "text_acc": 0.584375 + }, + "rollout": { + "rollout_mse": 0.035108547657728195, + "time_per_rollout_step_ms": 0.01941507500005102 + }, + "history": [ + { + "model": "neural_operator", + "epoch": 1, + "train_loss": 0.22099271257008826, + "val_loss": 0.14159053351197923, + "val_mse": 0.03796600177884102, + "val_psnr": 14.206051362501022, + "val_ssim": 0.0035837760660797358, + "val_nll": 0.11693888902664185, + "val_text_acc": 0.4611607142857143 + }, + { + "model": "neural_operator", + "epoch": 2, + "train_loss": 0.16356867498585156, + "val_loss": 0.19191501821790422, + "val_mse": 0.03761623427271843, + "val_psnr": 14.246246834384031, + "val_ssim": 0.014090199954807758, + "val_nll": 0.10479438304901123, + "val_text_acc": 0.5651785714285714 + }, + { + "model": "neural_operator", + "epoch": 3, + "train_loss": 0.16067370080522128, + "val_loss": 0.12681868778807776, + "val_mse": 0.035502441227436066, + "val_psnr": 14.497417828561044, + "val_ssim": 0.06291486322879791, + "val_nll": 0.0313987098634243, + "val_text_acc": 0.6058035714285714 + } + ] + }, + { + "model": "swm", + "params": 405422, + "epochs": 3, + "best_val_loss": 0.11073624342679977, + "val": { + "loss": 0.11073624342679977, + "mse": 0.034973446279764175, + "psnr": 14.562615702489506, + "ssim": 0.0878821536898613, + "nll": 0.013030804693698883, + "text_acc": 0.8651785714285715 + }, + "test": { + "loss": 0.11030659292425428, + "mse": 0.03474213927984238, + "psnr": 14.591434429751091, + "ssim": 0.08014871180057526, + "nll": 0.004999135155230761, + "text_acc": 0.8535714285714285 + }, + "rollout": { + "rollout_mse": 0.03577370569109917, + "time_per_rollout_step_ms": 0.03425489374997426 + }, + "history": [ + { + "model": "swm", + "epoch": 1, + "train_loss": 0.21506715645747526, + "val_loss": 0.12526373671633856, + "val_mse": 0.03644835948944092, + "val_psnr": 14.383220141420779, + "val_ssim": 0.047720521688461304, + "val_nll": 0.06424301862716675, + "val_text_acc": 0.5825892857142857 + }, + { + "model": "swm", + "epoch": 2, + "train_loss": 0.1268984486481973, + "val_loss": 0.1202295475772449, + "val_mse": 0.03607780858874321, + "val_psnr": 14.427598499194552, + "val_ssim": 0.05312095209956169, + "val_nll": 0.05137684568762779, + "val_text_acc": 0.7339285714285714 + }, + { + "model": "swm", + "epoch": 3, + "train_loss": 0.12062653500054564, + "val_loss": 0.11073624342679977, + "val_mse": 0.034973446279764175, + "val_psnr": 14.562615702489506, + "val_ssim": 0.0878821536898613, + "val_nll": 0.013030804693698883, + "val_text_acc": 0.8651785714285715 + } + ] + }, + { + "model": "no_spectral_state", + "params": 497682, + "epochs": 3, + "best_val_loss": 0.11848256630556923, + "val": { + "loss": 0.11848256630556923, + "mse": 0.03523682430386543, + "psnr": 14.530032390072872, + "ssim": 0.06655468046665192, + "nll": 0.022175762802362442, + "text_acc": 0.5571428571428572 + }, + "test": { + "loss": 0.11922914321933474, + "mse": 0.03493478149175644, + "psnr": 14.56741968803547, + "ssim": 0.0609237365424633, + "nll": 0.011688289232552052, + "text_acc": 0.4861607142857143 + }, + "rollout": { + "rollout_mse": 0.0349217914044857, + "time_per_rollout_step_ms": 0.018417671874981778 + }, + "history": [ + { + "model": "no_spectral_state", + "epoch": 1, + "train_loss": 0.231919277459383, + "val_loss": 0.16919530289513723, + "val_mse": 0.037993550300598145, + "val_psnr": 14.202901219778813, + "val_ssim": 0.0032276848796755075, + "val_nll": 0.11789543181657791, + "val_text_acc": 0.40267857142857144 + }, + { + "model": "no_spectral_state", + "epoch": 2, + "train_loss": 0.22328585758805275, + "val_loss": 0.16194516633238112, + "val_mse": 0.03718814626336098, + "val_psnr": 14.295954696017212, + "val_ssim": 0.031110256910324097, + "val_nll": 0.08993015438318253, + "val_text_acc": 0.45669642857142856 + }, + { + "model": "no_spectral_state", + "epoch": 3, + "train_loss": 0.13657328113913536, + "val_loss": 0.11848256630556923, + "val_mse": 0.03523682430386543, + "val_psnr": 14.530032390072872, + "val_ssim": 0.06655468046665192, + "val_nll": 0.022175762802362442, + "val_text_acc": 0.5571428571428572 + } + ] + }, + { + "model": "no_spectral_transition", + "params": 446802, + "epochs": 3, + "best_val_loss": 0.11065810599497386, + "val": { + "loss": 0.11065810599497386, + "mse": 0.0352352075278759, + "psnr": 14.530231662570795, + "ssim": 0.07634083926677704, + "nll": 0.02211970090866089, + "text_acc": 0.7464285714285714 + }, + "test": { + "loss": 0.11198038820709501, + "mse": 0.034975696355104446, + "psnr": 14.562336300836341, + "ssim": 0.06826915591955185, + "nll": 0.013108844868838787, + "text_acc": 0.765625 + }, + "rollout": { + "rollout_mse": 0.035049136728048325, + "time_per_rollout_step_ms": 0.03176636875004135 + }, + "history": [ + { + "model": "no_spectral_transition", + "epoch": 1, + "train_loss": 0.2562547139823437, + "val_loss": 0.1607078973736082, + "val_mse": 0.036143843084573746, + "val_psnr": 14.419656718722425, + "val_ssim": 0.04245064780116081, + "val_nll": 0.05366950482130051, + "val_text_acc": 0.5049107142857143 + }, + { + "model": "no_spectral_transition", + "epoch": 2, + "train_loss": 0.14521276125950472, + "val_loss": 0.12309963681868144, + "val_mse": 0.03614606335759163, + "val_psnr": 14.41938994501424, + "val_ssim": 0.053956516087055206, + "val_nll": 0.053746674209833145, + "val_text_acc": 0.5526785714285715 + }, + { + "model": "no_spectral_transition", + "epoch": 3, + "train_loss": 0.11529357039502688, + "val_loss": 0.11065810599497386, + "val_mse": 0.0352352075278759, + "val_psnr": 14.530231662570795, + "val_ssim": 0.07634083926677704, + "val_nll": 0.02211970090866089, + "val_text_acc": 0.7464285714285714 + } + ] + }, + { + "model": "no_text_branch", + "params": 402350, + "epochs": 3, + "best_val_loss": 0.1184177217738969, + "val": { + "loss": 0.1184177217738969, + "mse": 0.034196775406599045, + "psnr": 14.66014843918864, + "ssim": 0.12382669746875763, + "nll": -0.013937046751379967, + "text_acc": 0.725 + }, + "test": { + "loss": 0.11902656512601036, + "mse": 0.033662281930446625, + "psnr": 14.728564469992364, + "ssim": 0.12929166853427887, + "nll": -0.032495759427547455, + "text_acc": 0.7026785714285714 + }, + "rollout": { + "rollout_mse": 0.03507991507649422, + "time_per_rollout_step_ms": 0.03016817812504513 + }, + "history": [ + { + "model": "no_text_branch", + "epoch": 1, + "train_loss": 0.2242140466613429, + "val_loss": 0.12535723085914338, + "val_mse": 0.03669121488928795, + "val_psnr": 14.354379080047845, + "val_ssim": 0.045943256467580795, + "val_nll": 0.07267540693283081, + "val_text_acc": 0.5254464285714285 + }, + { + "model": "no_text_branch", + "epoch": 2, + "train_loss": 0.13053855379777296, + "val_loss": 0.12476341639246259, + "val_mse": 0.035839807242155075, + "val_psnr": 14.456343347770307, + "val_ssim": 0.05472106486558914, + "val_nll": 0.04311274737119675, + "val_text_acc": 0.6169642857142857 + }, + { + "model": "no_text_branch", + "epoch": 3, + "train_loss": 0.1171437967568636, + "val_loss": 0.1184177217738969, + "val_mse": 0.034196775406599045, + "val_psnr": 14.66014843918864, + "val_ssim": 0.12382669746875763, + "val_nll": -0.013937046751379967, + "val_text_acc": 0.725 + } + ] + }, + { + "model": "no_stability_penalty", + "params": 405422, + "epochs": 2, + "best_val_loss": 0.12874157088143484, + "val": { + "loss": 0.12874157088143484, + "mse": 0.036590300500392914, + "psnr": 14.36634023825047, + "ssim": 0.044199828058481216, + "nll": 0.06917144358158112, + "text_acc": 0.6120535714285714 + }, + "test": { + "loss": 0.13050702214241028, + "mse": 0.03603369742631912, + "psnr": 14.43291172530207, + "ssim": 0.04435477778315544, + "nll": 0.04984518140554428, + "text_acc": 0.584375 + }, + "rollout": { + "rollout_mse": 0.03608756139874458, + "time_per_rollout_step_ms": 0.029396649999924307 + }, + "history": [ + { + "model": "no_stability_penalty", + "epoch": 1, + "train_loss": 0.22426620152379786, + "val_loss": 0.12874157088143484, + "val_mse": 0.036590300500392914, + "val_psnr": 14.36634023825047, + "val_ssim": 0.044199828058481216, + "val_nll": 0.06917144358158112, + "val_text_acc": 0.6120535714285714 + }, + { + "model": "no_stability_penalty", + "epoch": 2, + "train_loss": 0.13447609916329384, + "val_loss": 0.13604165826525008, + "val_mse": 0.035899341106414795, + "val_psnr": 14.449135223541106, + "val_ssim": 0.06068476289510727, + "val_nll": 0.04517996311187744, + "val_text_acc": 0.846875 + } + ] + } + ] +} \ No newline at end of file diff --git a/results/neural_operator.pt b/results/neural_operator.pt new file mode 100644 index 0000000000000000000000000000000000000000..5cc78d2b3af7b4b118c572b38fb8a56d15c64406 --- /dev/null +++ b/results/neural_operator.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d2b7aea7c1936d2e3bcda86bf27e6148bd96889448259f009ba20daae42abeb7 +size 1626542 diff --git a/results/neural_operator_history.csv b/results/neural_operator_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..1123b0a5324e96cdf8f80a4ebcace1a4d7c3ee7f --- /dev/null +++ b/results/neural_operator_history.csv @@ -0,0 +1,4 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +neural_operator,1,0.22099271257008826,0.14159053351197923,0.03796600177884102,14.206051362501022,0.0035837760660797358,0.11693888902664185,0.4611607142857143 +neural_operator,2,0.16356867498585156,0.19191501821790422,0.03761623427271843,14.246246834384031,0.014090199954807758,0.10479438304901123,0.5651785714285714 +neural_operator,3,0.16067370080522128,0.12681868778807776,0.035502441227436066,14.497417828561044,0.06291486322879791,0.0313987098634243,0.6058035714285714 diff --git a/results/neural_operator_metrics.json b/results/neural_operator_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..351dc5a2b0eed376017bc717966c114ad7578126 --- /dev/null +++ b/results/neural_operator_metrics.json @@ -0,0 +1,61 @@ +{ + "model": "neural_operator", + "params": 403794, + "epochs": 3, + "best_val_loss": 0.12681868778807776, + "val": { + "loss": 0.12681868778807776, + "mse": 0.035502441227436066, + "psnr": 14.497417828561044, + "ssim": 0.06291486322879791, + "nll": 0.0313987098634243, + "text_acc": 0.6058035714285714 + }, + "test": { + "loss": 0.12872901665312902, + "mse": 0.03520356863737106, + "psnr": 14.534133092218475, + "ssim": 0.05829298496246338, + "nll": 0.02102111093699932, + "text_acc": 0.584375 + }, + "rollout": { + "rollout_mse": 0.035108547657728195, + "time_per_rollout_step_ms": 0.01941507500005102 + }, + "history": [ + { + "model": "neural_operator", + "epoch": 1, + "train_loss": 0.22099271257008826, + "val_loss": 0.14159053351197923, + "val_mse": 0.03796600177884102, + "val_psnr": 14.206051362501022, + "val_ssim": 0.0035837760660797358, + "val_nll": 0.11693888902664185, + "val_text_acc": 0.4611607142857143 + }, + { + "model": "neural_operator", + "epoch": 2, + "train_loss": 0.16356867498585156, + "val_loss": 0.19191501821790422, + "val_mse": 0.03761623427271843, + "val_psnr": 14.246246834384031, + "val_ssim": 0.014090199954807758, + "val_nll": 0.10479438304901123, + "val_text_acc": 0.5651785714285714 + }, + { + "model": "neural_operator", + "epoch": 3, + "train_loss": 0.16067370080522128, + "val_loss": 0.12681868778807776, + "val_mse": 0.035502441227436066, + "val_psnr": 14.497417828561044, + "val_ssim": 0.06291486322879791, + "val_nll": 0.0313987098634243, + "val_text_acc": 0.6058035714285714 + } + ] +} \ No newline at end of file diff --git a/results/no_spectral_state.pt b/results/no_spectral_state.pt new file mode 100644 index 0000000000000000000000000000000000000000..30d4a4a700228e366d582d0c5aa697cf7f2529b3 --- /dev/null +++ b/results/no_spectral_state.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:610d0e774236353811f1958c2ae12008ee102d32c2ccd1e74531f6a5acb2dede +size 2003277 diff --git a/results/no_spectral_state_history.csv b/results/no_spectral_state_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..78717c11445c9714b6250a4e25be17796e4d607d --- /dev/null +++ b/results/no_spectral_state_history.csv @@ -0,0 +1,4 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +no_spectral_state,1,0.231919277459383,0.16919530289513723,0.037993550300598145,14.202901219778813,0.0032276848796755075,0.11789543181657791,0.40267857142857144 +no_spectral_state,2,0.22328585758805275,0.16194516633238112,0.03718814626336098,14.295954696017212,0.031110256910324097,0.08993015438318253,0.45669642857142856 +no_spectral_state,3,0.13657328113913536,0.11848256630556923,0.03523682430386543,14.530032390072872,0.06655468046665192,0.022175762802362442,0.5571428571428572 diff --git a/results/no_spectral_state_metrics.json b/results/no_spectral_state_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..83e5849aca0ce21dfd89c51fe884fc3ea1ea5dbb --- /dev/null +++ b/results/no_spectral_state_metrics.json @@ -0,0 +1,61 @@ +{ + "model": "no_spectral_state", + "params": 497682, + "epochs": 3, + "best_val_loss": 0.11848256630556923, + "val": { + "loss": 0.11848256630556923, + "mse": 0.03523682430386543, + "psnr": 14.530032390072872, + "ssim": 0.06655468046665192, + "nll": 0.022175762802362442, + "text_acc": 0.5571428571428572 + }, + "test": { + "loss": 0.11922914321933474, + "mse": 0.03493478149175644, + "psnr": 14.56741968803547, + "ssim": 0.0609237365424633, + "nll": 0.011688289232552052, + "text_acc": 0.4861607142857143 + }, + "rollout": { + "rollout_mse": 0.0349217914044857, + "time_per_rollout_step_ms": 0.018417671874981778 + }, + "history": [ + { + "model": "no_spectral_state", + "epoch": 1, + "train_loss": 0.231919277459383, + "val_loss": 0.16919530289513723, + "val_mse": 0.037993550300598145, + "val_psnr": 14.202901219778813, + "val_ssim": 0.0032276848796755075, + "val_nll": 0.11789543181657791, + "val_text_acc": 0.40267857142857144 + }, + { + "model": "no_spectral_state", + "epoch": 2, + "train_loss": 0.22328585758805275, + "val_loss": 0.16194516633238112, + "val_mse": 0.03718814626336098, + "val_psnr": 14.295954696017212, + "val_ssim": 0.031110256910324097, + "val_nll": 0.08993015438318253, + "val_text_acc": 0.45669642857142856 + }, + { + "model": "no_spectral_state", + "epoch": 3, + "train_loss": 0.13657328113913536, + "val_loss": 0.11848256630556923, + "val_mse": 0.03523682430386543, + "val_psnr": 14.530032390072872, + "val_ssim": 0.06655468046665192, + "val_nll": 0.022175762802362442, + "val_text_acc": 0.5571428571428572 + } + ] +} \ No newline at end of file diff --git a/results/no_spectral_transition.pt b/results/no_spectral_transition.pt new file mode 100644 index 0000000000000000000000000000000000000000..8925a096c2128eca72f405eef81e4f6bc039dbf2 --- /dev/null +++ b/results/no_spectral_transition.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9cf788c11ec1cc7d72a39fb8237ce49be12da3aa7fc195ccb301afca5beca0e8 +size 1798961 diff --git a/results/no_spectral_transition_history.csv b/results/no_spectral_transition_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..5389d1f3a7490530e2d52253112b9b29e6d71009 --- /dev/null +++ b/results/no_spectral_transition_history.csv @@ -0,0 +1,4 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +no_spectral_transition,1,0.2562547139823437,0.1607078973736082,0.036143843084573746,14.419656718722425,0.04245064780116081,0.05366950482130051,0.5049107142857143 +no_spectral_transition,2,0.14521276125950472,0.12309963681868144,0.03614606335759163,14.41938994501424,0.053956516087055206,0.053746674209833145,0.5526785714285715 +no_spectral_transition,3,0.11529357039502688,0.11065810599497386,0.0352352075278759,14.530231662570795,0.07634083926677704,0.02211970090866089,0.7464285714285714 diff --git a/results/no_spectral_transition_metrics.json b/results/no_spectral_transition_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..72a80ca5af13395b678ee58894b3530770c18b19 --- /dev/null +++ b/results/no_spectral_transition_metrics.json @@ -0,0 +1,61 @@ +{ + "model": "no_spectral_transition", + "params": 446802, + "epochs": 3, + "best_val_loss": 0.11065810599497386, + "val": { + "loss": 0.11065810599497386, + "mse": 0.0352352075278759, + "psnr": 14.530231662570795, + "ssim": 0.07634083926677704, + "nll": 0.02211970090866089, + "text_acc": 0.7464285714285714 + }, + "test": { + "loss": 0.11198038820709501, + "mse": 0.034975696355104446, + "psnr": 14.562336300836341, + "ssim": 0.06826915591955185, + "nll": 0.013108844868838787, + "text_acc": 0.765625 + }, + "rollout": { + "rollout_mse": 0.035049136728048325, + "time_per_rollout_step_ms": 0.03176636875004135 + }, + "history": [ + { + "model": "no_spectral_transition", + "epoch": 1, + "train_loss": 0.2562547139823437, + "val_loss": 0.1607078973736082, + "val_mse": 0.036143843084573746, + "val_psnr": 14.419656718722425, + "val_ssim": 0.04245064780116081, + "val_nll": 0.05366950482130051, + "val_text_acc": 0.5049107142857143 + }, + { + "model": "no_spectral_transition", + "epoch": 2, + "train_loss": 0.14521276125950472, + "val_loss": 0.12309963681868144, + "val_mse": 0.03614606335759163, + "val_psnr": 14.41938994501424, + "val_ssim": 0.053956516087055206, + "val_nll": 0.053746674209833145, + "val_text_acc": 0.5526785714285715 + }, + { + "model": "no_spectral_transition", + "epoch": 3, + "train_loss": 0.11529357039502688, + "val_loss": 0.11065810599497386, + "val_mse": 0.0352352075278759, + "val_psnr": 14.530231662570795, + "val_ssim": 0.07634083926677704, + "val_nll": 0.02211970090866089, + "val_text_acc": 0.7464285714285714 + } + ] +} \ No newline at end of file diff --git a/results/no_stability_penalty.pt b/results/no_stability_penalty.pt new file mode 100644 index 0000000000000000000000000000000000000000..9eeaf77757cba3c7366856e18e63f0665e96b6ce --- /dev/null +++ b/results/no_stability_penalty.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c35225bb09b697aedad2dd26b20f57f2ed42619cad1ae2f32fdf4fa0379cb2af +size 1633915 diff --git a/results/no_stability_penalty_history.csv b/results/no_stability_penalty_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..64524ef6b24df79606769ff056de553a560d2dff --- /dev/null +++ b/results/no_stability_penalty_history.csv @@ -0,0 +1,3 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +no_stability_penalty,1,0.22426620152379786,0.12874157088143484,0.036590300500392914,14.36634023825047,0.044199828058481216,0.06917144358158112,0.6120535714285714 +no_stability_penalty,2,0.13447609916329384,0.13604165826525008,0.035899341106414795,14.449135223541106,0.06068476289510727,0.04517996311187744,0.846875 diff --git a/results/no_stability_penalty_metrics.json b/results/no_stability_penalty_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..c0e44a22cf780542cd84e69ff44f9843b02fe3fa --- /dev/null +++ b/results/no_stability_penalty_metrics.json @@ -0,0 +1,50 @@ +{ + "model": "no_stability_penalty", + "params": 405422, + "epochs": 2, + "best_val_loss": 0.12874157088143484, + "val": { + "loss": 0.12874157088143484, + "mse": 0.036590300500392914, + "psnr": 14.36634023825047, + "ssim": 0.044199828058481216, + "nll": 0.06917144358158112, + "text_acc": 0.6120535714285714 + }, + "test": { + "loss": 0.13050702214241028, + "mse": 0.03603369742631912, + "psnr": 14.43291172530207, + "ssim": 0.04435477778315544, + "nll": 0.04984518140554428, + "text_acc": 0.584375 + }, + "rollout": { + "rollout_mse": 0.03608756139874458, + "time_per_rollout_step_ms": 0.029396649999924307 + }, + "history": [ + { + "model": "no_stability_penalty", + "epoch": 1, + "train_loss": 0.22426620152379786, + "val_loss": 0.12874157088143484, + "val_mse": 0.036590300500392914, + "val_psnr": 14.36634023825047, + "val_ssim": 0.044199828058481216, + "val_nll": 0.06917144358158112, + "val_text_acc": 0.6120535714285714 + }, + { + "model": "no_stability_penalty", + "epoch": 2, + "train_loss": 0.13447609916329384, + "val_loss": 0.13604165826525008, + "val_mse": 0.035899341106414795, + "val_psnr": 14.449135223541106, + "val_ssim": 0.06068476289510727, + "val_nll": 0.04517996311187744, + "val_text_acc": 0.846875 + } + ] +} \ No newline at end of file diff --git a/results/no_text_branch.pt b/results/no_text_branch.pt new file mode 100644 index 0000000000000000000000000000000000000000..00b92f0e78044b7938749b5f3a1c731e8363562d --- /dev/null +++ b/results/no_text_branch.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b2c9b293ea4f61d9202182b1ffd13dceeb9f76d1c48630da2f03caef4b7ef835 +size 1621393 diff --git a/results/no_text_branch_history.csv b/results/no_text_branch_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..25118f491f6aadd6920353227d16002f0d457ba5 --- /dev/null +++ b/results/no_text_branch_history.csv @@ -0,0 +1,4 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +no_text_branch,1,0.2242140466613429,0.12535723085914338,0.03669121488928795,14.354379080047845,0.045943256467580795,0.07267540693283081,0.5254464285714285 +no_text_branch,2,0.13053855379777296,0.12476341639246259,0.035839807242155075,14.456343347770307,0.05472106486558914,0.04311274737119675,0.6169642857142857 +no_text_branch,3,0.1171437967568636,0.1184177217738969,0.034196775406599045,14.66014843918864,0.12382669746875763,-0.013937046751379967,0.725 diff --git a/results/no_text_branch_metrics.json b/results/no_text_branch_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..cfd196ad5097708a93163d6f540e778287907ca8 --- /dev/null +++ b/results/no_text_branch_metrics.json @@ -0,0 +1,61 @@ +{ + "model": "no_text_branch", + "params": 402350, + "epochs": 3, + "best_val_loss": 0.1184177217738969, + "val": { + "loss": 0.1184177217738969, + "mse": 0.034196775406599045, + "psnr": 14.66014843918864, + "ssim": 0.12382669746875763, + "nll": -0.013937046751379967, + "text_acc": 0.725 + }, + "test": { + "loss": 0.11902656512601036, + "mse": 0.033662281930446625, + "psnr": 14.728564469992364, + "ssim": 0.12929166853427887, + "nll": -0.032495759427547455, + "text_acc": 0.7026785714285714 + }, + "rollout": { + "rollout_mse": 0.03507991507649422, + "time_per_rollout_step_ms": 0.03016817812504513 + }, + "history": [ + { + "model": "no_text_branch", + "epoch": 1, + "train_loss": 0.2242140466613429, + "val_loss": 0.12535723085914338, + "val_mse": 0.03669121488928795, + "val_psnr": 14.354379080047845, + "val_ssim": 0.045943256467580795, + "val_nll": 0.07267540693283081, + "val_text_acc": 0.5254464285714285 + }, + { + "model": "no_text_branch", + "epoch": 2, + "train_loss": 0.13053855379777296, + "val_loss": 0.12476341639246259, + "val_mse": 0.035839807242155075, + "val_psnr": 14.456343347770307, + "val_ssim": 0.05472106486558914, + "val_nll": 0.04311274737119675, + "val_text_acc": 0.6169642857142857 + }, + { + "model": "no_text_branch", + "epoch": 3, + "train_loss": 0.1171437967568636, + "val_loss": 0.1184177217738969, + "val_mse": 0.034196775406599045, + "val_psnr": 14.66014843918864, + "val_ssim": 0.12382669746875763, + "val_nll": -0.013937046751379967, + "val_text_acc": 0.725 + } + ] +} \ No newline at end of file diff --git a/results/swm.pt b/results/swm.pt new file mode 100644 index 0000000000000000000000000000000000000000..f6b96d96bef8d14808ace9fee0a42dd48a14661a --- /dev/null +++ b/results/swm.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:39c834750c8f021f7880a0b18dbd3c3e557df8198a0dc5dcd002214aaeed0482 +size 1631332 diff --git a/results/swm_history.csv b/results/swm_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..02289db1e2d9109e958e414a1057899605cdb6a3 --- /dev/null +++ b/results/swm_history.csv @@ -0,0 +1,4 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +swm,1,0.21506715645747526,0.12526373671633856,0.03644835948944092,14.383220141420779,0.047720521688461304,0.06424301862716675,0.5825892857142857 +swm,2,0.1268984486481973,0.1202295475772449,0.03607780858874321,14.427598499194552,0.05312095209956169,0.05137684568762779,0.7339285714285714 +swm,3,0.12062653500054564,0.11073624342679977,0.034973446279764175,14.562615702489506,0.0878821536898613,0.013030804693698883,0.8651785714285715 diff --git a/results/swm_metrics.json b/results/swm_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..cea8c9852a73210180ade6a54e9ac9f33a7c2316 --- /dev/null +++ b/results/swm_metrics.json @@ -0,0 +1,61 @@ +{ + "model": "swm", + "params": 405422, + "epochs": 3, + "best_val_loss": 0.11073624342679977, + "val": { + "loss": 0.11073624342679977, + "mse": 0.034973446279764175, + "psnr": 14.562615702489506, + "ssim": 0.0878821536898613, + "nll": 0.013030804693698883, + "text_acc": 0.8651785714285715 + }, + "test": { + "loss": 0.11030659292425428, + "mse": 0.03474213927984238, + "psnr": 14.591434429751091, + "ssim": 0.08014871180057526, + "nll": 0.004999135155230761, + "text_acc": 0.8535714285714285 + }, + "rollout": { + "rollout_mse": 0.03577370569109917, + "time_per_rollout_step_ms": 0.03425489374997426 + }, + "history": [ + { + "model": "swm", + "epoch": 1, + "train_loss": 0.21506715645747526, + "val_loss": 0.12526373671633856, + "val_mse": 0.03644835948944092, + "val_psnr": 14.383220141420779, + "val_ssim": 0.047720521688461304, + "val_nll": 0.06424301862716675, + "val_text_acc": 0.5825892857142857 + }, + { + "model": "swm", + "epoch": 2, + "train_loss": 0.1268984486481973, + "val_loss": 0.1202295475772449, + "val_mse": 0.03607780858874321, + "val_psnr": 14.427598499194552, + "val_ssim": 0.05312095209956169, + "val_nll": 0.05137684568762779, + "val_text_acc": 0.7339285714285714 + }, + { + "model": "swm", + "epoch": 3, + "train_loss": 0.12062653500054564, + "val_loss": 0.11073624342679977, + "val_mse": 0.034973446279764175, + "val_psnr": 14.562615702489506, + "val_ssim": 0.0878821536898613, + "val_nll": 0.013030804693698883, + "val_text_acc": 0.8651785714285715 + } + ] +} \ No newline at end of file diff --git a/results/transformer.pt b/results/transformer.pt new file mode 100644 index 0000000000000000000000000000000000000000..f8328761435eec19635b9edd93f7e8b313e715fa --- /dev/null +++ b/results/transformer.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:46656b6ee4f1f06f4768b06e68df7a2079f2adacc28de3bffb5c8e4b45b481fa +size 2028773 diff --git a/results/transformer_history.csv b/results/transformer_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..cc219c2a08bed6785609ac65ca131e1814a69d62 --- /dev/null +++ b/results/transformer_history.csv @@ -0,0 +1,4 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +transformer,1,0.23709710474525178,0.13327171334198543,0.03711897134780884,14.30404067586808,0.01684718206524849,0.08752818405628204,0.47098214285714285 +transformer,2,0.16270889874015534,0.1869750597647258,0.035343486815690994,14.516906073444048,0.06779398024082184,0.025879332795739174,0.5580357142857143 +transformer,3,0.1466867652322565,0.13392244918005808,0.035102479159832,14.54662209818714,0.07712315022945404,0.01751108281314373,0.7321428571428571 diff --git a/results/transformer_metrics.json b/results/transformer_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..6dd77d838ab5f51c13d61e1a71cc2eb02a9065f7 --- /dev/null +++ b/results/transformer_metrics.json @@ -0,0 +1,61 @@ +{ + "model": "transformer", + "params": 503322, + "epochs": 3, + "best_val_loss": 0.13327171334198543, + "val": { + "loss": 0.13327171334198543, + "mse": 0.03711897134780884, + "psnr": 14.30404067586808, + "ssim": 0.01684718206524849, + "nll": 0.08752818405628204, + "text_acc": 0.47098214285714285 + }, + "test": { + "loss": 0.13411018465246474, + "mse": 0.036575935781002045, + "psnr": 14.368045538061631, + "ssim": 0.016545191407203674, + "nll": 0.06867272406816483, + "text_acc": 0.4120535714285714 + }, + "rollout": { + "rollout_mse": 0.036570727825164795, + "time_per_rollout_step_ms": 0.029027615625043524 + }, + "history": [ + { + "model": "transformer", + "epoch": 1, + "train_loss": 0.23709710474525178, + "val_loss": 0.13327171334198543, + "val_mse": 0.03711897134780884, + "val_psnr": 14.30404067586808, + "val_ssim": 0.01684718206524849, + "val_nll": 0.08752818405628204, + "val_text_acc": 0.47098214285714285 + }, + { + "model": "transformer", + "epoch": 2, + "train_loss": 0.16270889874015534, + "val_loss": 0.1869750597647258, + "val_mse": 0.035343486815690994, + "val_psnr": 14.516906073444048, + "val_ssim": 0.06779398024082184, + "val_nll": 0.025879332795739174, + "val_text_acc": 0.5580357142857143 + }, + { + "model": "transformer", + "epoch": 3, + "train_loss": 0.1466867652322565, + "val_loss": 0.13392244918005808, + "val_mse": 0.035102479159832, + "val_psnr": 14.54662209818714, + "val_ssim": 0.07712315022945404, + "val_nll": 0.01751108281314373, + "val_text_acc": 0.7321428571428571 + } + ] +} \ No newline at end of file diff --git a/results/v10_cf_integrity/metrics_by_seed.csv b/results/v10_cf_integrity/metrics_by_seed.csv new file mode 100644 index 0000000000000000000000000000000000000000..6fa327e07757c2d76bb4cae74351a5ec7be55138 --- /dev/null +++ b/results/v10_cf_integrity/metrics_by_seed.csv @@ -0,0 +1,3 @@ +seed,model,params,interpolation_cf_audit_final_image_mse,interpolation_cf_audit_endpoint_cosine,interpolation_cf_audit_endpoint_magnitude_ratio,interpolation_cf_audit_trajectory_cosine,interpolation_cf_audit_trajectory_magnitude_ratio,extrapolation_cf_audit_final_image_mse,extrapolation_cf_audit_endpoint_cosine,extrapolation_cf_audit_endpoint_magnitude_ratio,extrapolation_cf_audit_trajectory_cosine,extrapolation_cf_audit_trajectory_magnitude_ratio,structural_ood_cf_audit_final_image_mse,structural_ood_cf_audit_endpoint_cosine,structural_ood_cf_audit_endpoint_magnitude_ratio,structural_ood_cf_audit_trajectory_cosine,structural_ood_cf_audit_trajectory_magnitude_ratio +0,swm_structured_cf_v9,407807,0.06161266332492232,0.632532387971878,1.5380405234941037,0.31433912832289934,2.406510012628969,0.05608502682298422,0.632532387971878,0.884385335574069,0.3143391143530607,5.626601266812639,0.05101879790890962,0.6325324028730392,1.4282600050013334,0.4868762902915478,3.0733455126040687 +0,no_spectral_transition,446802,0.0479674325324595,0.0,0.0,0.0,0.0,0.04172235995065421,0.0,0.0,0.0,0.0,0.03563136118464172,0.0,0.0,0.0,0.0 diff --git a/results/v10_cf_integrity/metrics_summary.csv b/results/v10_cf_integrity/metrics_summary.csv new file mode 100644 index 0000000000000000000000000000000000000000..48133eaf5602d2fc88a2761b9e0f031ab1ab766d --- /dev/null +++ b/results/v10_cf_integrity/metrics_summary.csv @@ -0,0 +1,3 @@ +model,params,n_seeds,interpolation_cf_audit_final_image_mse_mean,interpolation_cf_audit_final_image_mse_std,interpolation_cf_audit_endpoint_cosine_mean,interpolation_cf_audit_endpoint_cosine_std,interpolation_cf_audit_endpoint_magnitude_ratio_mean,interpolation_cf_audit_endpoint_magnitude_ratio_std,interpolation_cf_audit_trajectory_cosine_mean,interpolation_cf_audit_trajectory_cosine_std,interpolation_cf_audit_trajectory_magnitude_ratio_mean,interpolation_cf_audit_trajectory_magnitude_ratio_std,extrapolation_cf_audit_final_image_mse_mean,extrapolation_cf_audit_final_image_mse_std,extrapolation_cf_audit_endpoint_cosine_mean,extrapolation_cf_audit_endpoint_cosine_std,extrapolation_cf_audit_endpoint_magnitude_ratio_mean,extrapolation_cf_audit_endpoint_magnitude_ratio_std,extrapolation_cf_audit_trajectory_cosine_mean,extrapolation_cf_audit_trajectory_cosine_std,extrapolation_cf_audit_trajectory_magnitude_ratio_mean,extrapolation_cf_audit_trajectory_magnitude_ratio_std,structural_ood_cf_audit_final_image_mse_mean,structural_ood_cf_audit_final_image_mse_std,structural_ood_cf_audit_endpoint_cosine_mean,structural_ood_cf_audit_endpoint_cosine_std,structural_ood_cf_audit_endpoint_magnitude_ratio_mean,structural_ood_cf_audit_endpoint_magnitude_ratio_std,structural_ood_cf_audit_trajectory_cosine_mean,structural_ood_cf_audit_trajectory_cosine_std,structural_ood_cf_audit_trajectory_magnitude_ratio_mean,structural_ood_cf_audit_trajectory_magnitude_ratio_std +swm_structured_cf_v9,407807,1,0.06161266332492232,0.0,0.632532387971878,0.0,1.5380405234941037,0.0,0.31433912832289934,0.0,2.406510012628969,0.0,0.05608502682298422,0.0,0.632532387971878,0.0,0.884385335574069,0.0,0.3143391143530607,0.0,5.626601266812639,0.0,0.05101879790890962,0.0,0.6325324028730392,0.0,1.4282600050013334,0.0,0.4868762902915478,0.0,3.0733455126040687,0.0 +no_spectral_transition,446802,1,0.0479674325324595,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.04172235995065421,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.03563136118464172,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 diff --git a/results/v10_cf_integrity/oracle_physics_audit.csv b/results/v10_cf_integrity/oracle_physics_audit.csv new file mode 100644 index 0000000000000000000000000000000000000000..443c8d0f2e54f9323248473bf2975bb7897bc339 --- /dev/null +++ b/results/v10_cf_integrity/oracle_physics_audit.csv @@ -0,0 +1,13 @@ +regime,program,physics_fingerprint,trajectory_set_hash,oracle_effect_rms,oracle_effect_final_mag,oracle_effect_path_mag,oracle_effect_temporal_variation,physics_acceleration,physics_max_speed,physics_occlusion_prob,physics_damping,physics_restitution,physics_image_size +interpolation,left_right,7faf93d85bc638df,2e2e1c8b3db689d21214409066afc8c5197b3f907bba7775d43a68e3d8b44b90,6.60317325592041,12.682748794555664,8.339932441711426,1.575080394744873,0.75,3.2,0.15,0.985,0.93,32 +interpolation,up_down,7faf93d85bc638df,f92481f3453aea714c2d3bbc62e8438c56744628c5ac65d5b0beabc5892b6854,3.219569444656372,6.8499979972839355,4.129770755767822,1.131497859954834,0.75,3.2,0.15,0.985,0.93,32 +interpolation,turn_cw,7faf93d85bc638df,a4d63cdddf44affcc73a442c7cc6816c4b463291df30d4cf30b8ccec90f96bd4,1.36246919631958,2.132298469543457,1.6553605794906616,1.2519466876983643,0.75,3.2,0.15,0.985,0.93,32 +interpolation,impulse_reverse,7faf93d85bc638df,8e53cf33e82aba2ca887b5fe8d65aa98b5ed1032538896e48fb940773b9b84d5,2.2695906162261963,6.287040710449219,2.703200578689575,1.5870790481567383,0.75,3.2,0.15,0.985,0.93,32 +extrapolation,left_right,8fd165fd1bdad9d5,8c45bbe7ee15108de6e16da209c1ce488c61b058b8731914cd49c904ec71c399,8.407105445861816,15.352855682373047,10.797575950622559,1.9451171159744263,1.2,4.3,0.3,0.94,0.82,32 +extrapolation,up_down,8fd165fd1bdad9d5,23259743a322a36fb60031bd741aebe23e312efdec4fadcb607c1972e158c5d1,5.728433609008789,13.862934112548828,7.215188980102539,1.8584476709365845,1.2,4.3,0.3,0.94,0.82,32 +extrapolation,turn_cw,8fd165fd1bdad9d5,e187ad79708f917d1d3bed010bb72e0064efdf422e69837ded8511996ca19902,2.043153762817383,4.421419143676758,2.3115246295928955,1.9071159362792969,1.2,4.3,0.3,0.94,0.82,32 +extrapolation,impulse_reverse,8fd165fd1bdad9d5,1fda1096d280bc322caa1735b5d46e4e4d07e60b52b4ddbde6cd6acf6f8dda7a,3.5782363414764404,10.176029205322266,4.007529258728027,2.2560203075408936,1.2,4.3,0.3,0.94,0.82,32 +structural_ood,left_right,e163ca91018500ad,2d6887e7c1409fcf2823fdfd3f0eb679cc4f7ca3bba624949d88e5c2c052fdce,5.653831481933594,13.098047256469727,6.868289947509766,1.201424479484558,0.7,3.0,0.12,0.88,0.62,32 +structural_ood,up_down,e163ca91018500ad,39a69f1bd64ecc94aa5146cab5d795e407b1da22ed54c7c4045cfdde99bf6af6,4.423787593841553,9.479574203491211,5.610476016998291,0.8877241015434265,0.7,3.0,0.12,0.88,0.62,32 +structural_ood,turn_cw,e163ca91018500ad,c11f1c60a6fb593c825940d132ecba2141025846ac1de1447c0cc7759a10cbb4,1.1051758527755737,2.826913833618164,1.2778631448745728,0.9744865894317627,0.7,3.0,0.12,0.88,0.62,32 +structural_ood,impulse_reverse,e163ca91018500ad,d3a10a5a4a5986acca26d536abccce102b46258bfaae384ec3c40f23ae3cec0d,1.8729972839355469,5.238364219665527,2.15092396736145,1.020102620124817,0.7,3.0,0.12,0.88,0.62,32 diff --git a/results/v10_cf_integrity/run_config.json b/results/v10_cf_integrity/run_config.json new file mode 100644 index 0000000000000000000000000000000000000000..7669830d42778807fbf91e562d8036728728de23 --- /dev/null +++ b/results/v10_cf_integrity/run_config.json @@ -0,0 +1,16 @@ +{ + "epochs": 1, + "batch_size": 64, + "seeds": [ + 0 + ], + "cf_horizon": 6, + "cf_pairs": 2, + "device": "cpu", + "models": [ + "swm_structured_cf_v9", + "no_spectral_transition" + ], + "device_resolved": "cpu", + "audit_seed": 10101 +} \ No newline at end of file diff --git a/results/v10_cf_integrity/seed_0/no_spectral_transition.pt b/results/v10_cf_integrity/seed_0/no_spectral_transition.pt new file mode 100644 index 0000000000000000000000000000000000000000..6b2f8bd1f6aa8e398df8ee6464fcf9400463228a --- /dev/null +++ b/results/v10_cf_integrity/seed_0/no_spectral_transition.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3eaba744687ad48f1dfa96a62c2a93f5c01747793f5eb1233345020f8d4bc633 +size 1798961 diff --git a/results/v10_cf_integrity/seed_0/no_spectral_transition_history.csv b/results/v10_cf_integrity/seed_0/no_spectral_transition_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..91736ce2327bee952a142186fed629cce293ed22 --- /dev/null +++ b/results/v10_cf_integrity/seed_0/no_spectral_transition_history.csv @@ -0,0 +1,2 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +no_spectral_transition,1,0.15143630791593482,0.1023182833324308,0.029909372329711914,15.241927008637951,0.09910410642623901,-0.1628052145242691,0.6334722222222222 diff --git a/results/v10_cf_integrity/seed_0/no_spectral_transition_metrics.json b/results/v10_cf_integrity/seed_0/no_spectral_transition_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..47c080221f0fa3cae60c982357661fdac48d1f74 --- /dev/null +++ b/results/v10_cf_integrity/seed_0/no_spectral_transition_metrics.json @@ -0,0 +1,61 @@ +{ + "model": "no_spectral_transition", + "params": 446802, + "epochs": 1, + "best_val_loss": 0.1023182833324308, + "val": { + "loss": 0.1023182833324308, + "mse": 0.029909372329711914, + "psnr": 15.241927008637951, + "ssim": 0.09910410642623901, + "nll": -0.1628052145242691, + "text_acc": 0.6334722222222222 + }, + "test": { + "loss": 0.10359841487977815, + "mse": 0.03062461130321026, + "psnr": 15.139294147887892, + "ssim": 0.10102337598800659, + "nll": -0.13797040283679962, + "text_acc": 0.6338888888888888 + }, + "rollout": { + "rollout_mse": 0.03078058920800686, + "rollout_text_acc": 0.6429166666666667, + "rollout_latent_cosine": 0.8857887983322144, + "rollout_joint_score": 0.641407818167288, + "time_per_rollout_step_ms": 0.043861587500027 + }, + "rollout_by_horizon": { + "5": { + "rollout_mse": 0.03078058920800686, + "rollout_text_acc": 0.6429166666666667, + "rollout_latent_cosine": 0.8857887983322144, + "rollout_joint_score": 0.641407818167288, + "time_per_rollout_step_ms": 0.043861587500027 + } + }, + "stability_diagnostics": {}, + "history": [ + { + "model": "no_spectral_transition", + "epoch": 1, + "train_loss": 0.15143630791593482, + "val_loss": 0.1023182833324308, + "val_mse": 0.029909372329711914, + "val_psnr": 15.241927008637951, + "val_ssim": 0.09910410642623901, + "val_nll": -0.1628052145242691, + "val_text_acc": 0.6334722222222222 + } + ], + "counterfactual_training": { + "enabled": false, + "horizon": 10, + "pairs": 48, + "every": 4, + "lambda_dir": 0.02, + "lambda_mag": 0.005, + "lambda_branch": 0.08 + } +} \ No newline at end of file diff --git a/results/v10_cf_integrity/seed_0/swm_structured_cf_v9.pt b/results/v10_cf_integrity/seed_0/swm_structured_cf_v9.pt new file mode 100644 index 0000000000000000000000000000000000000000..52b5a8d86e0d0d798d4ef277cee3ba04ca9e8339 --- /dev/null +++ b/results/v10_cf_integrity/seed_0/swm_structured_cf_v9.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:34d1b6a9b1593f44db69357bad97d29541ca32e311b0083027739a78d032b9bc +size 1643781 diff --git a/results/v10_cf_integrity/seed_0/swm_structured_cf_v9_history.csv b/results/v10_cf_integrity/seed_0/swm_structured_cf_v9_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..2695653b7fd64151e017635aad2752d397c5a390 --- /dev/null +++ b/results/v10_cf_integrity/seed_0/swm_structured_cf_v9_history.csv @@ -0,0 +1,2 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +swm_structured_cf_v9,1,0.1523468052899396,0.1251612338035003,0.03063354827463627,15.138026960812887,0.10426127165555954,-0.1376601904630661,0.7469444444444444 diff --git a/results/v10_cf_integrity/seed_0/swm_structured_cf_v9_metrics.json b/results/v10_cf_integrity/seed_0/swm_structured_cf_v9_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..83cdd783e42dfeee40bac6561ec2234c85f7993f --- /dev/null +++ b/results/v10_cf_integrity/seed_0/swm_structured_cf_v9_metrics.json @@ -0,0 +1,70 @@ +{ + "model": "swm_structured_cf_v9", + "params": 407807, + "epochs": 1, + "best_val_loss": 0.1251612338035003, + "val": { + "loss": 0.1251612338035003, + "mse": 0.03063354827463627, + "psnr": 15.138026960812887, + "ssim": 0.10426127165555954, + "nll": -0.1376601904630661, + "text_acc": 0.7469444444444444 + }, + "test": { + "loss": 0.1260391781511514, + "mse": 0.03139650449156761, + "psnr": 15.031187011155653, + "ssim": 0.10401683300733566, + "nll": -0.11116853356361389, + "text_acc": 0.7511111111111111 + }, + "rollout": { + "rollout_mse": 0.03316739946603775, + "rollout_text_acc": 0.70125, + "rollout_latent_cosine": 0.6849676966667175, + "rollout_joint_score": 0.6294955743238182, + "time_per_rollout_step_ms": 0.26648820468753165 + }, + "rollout_by_horizon": { + "5": { + "rollout_mse": 0.03316739946603775, + "rollout_text_acc": 0.70125, + "rollout_latent_cosine": 0.6849676966667175, + "rollout_joint_score": 0.6294955743238182, + "time_per_rollout_step_ms": 0.26648820468753165 + } + }, + "stability_diagnostics": { + "raw_operator_norm_max": 10.872591972351074, + "raw_operator_norm_mean": 7.083788871765137, + "effective_operator_norm_max": 1.0499999523162842, + "effective_operator_norm_mean": 1.0499999523162842, + "fraction_modes_clipped": 0.010416666977107525, + "rho": 1.05, + "transport_orthogonality_error": 2.52907142339609e-07, + "mean_dissipative_keep": 0.9822795391082764 + }, + "history": [ + { + "model": "swm_structured_cf_v9", + "epoch": 1, + "train_loss": 0.1523468052899396, + "val_loss": 0.1251612338035003, + "val_mse": 0.03063354827463627, + "val_psnr": 15.138026960812887, + "val_ssim": 0.10426127165555954, + "val_nll": -0.1376601904630661, + "val_text_acc": 0.7469444444444444 + } + ], + "counterfactual_training": { + "enabled": true, + "horizon": 10, + "pairs": 48, + "every": 4, + "lambda_dir": 0.02, + "lambda_mag": 0.005, + "lambda_branch": 0.08 + } +} \ No newline at end of file diff --git a/results/v12_causal_decomposition/paired_bootstrap_v9_vs_controls.csv b/results/v12_causal_decomposition/paired_bootstrap_v9_vs_controls.csv new file mode 100644 index 0000000000000000000000000000000000000000..824f8b106eb8a67da67b0ad9386910872bf116c3 --- /dev/null +++ b/results/v12_causal_decomposition/paired_bootstrap_v9_vs_controls.csv @@ -0,0 +1,19 @@ +metric,delta_v9_minus_control,ci95_low,ci95_high,n_matched,n_episodes,n_boot,regime,control +trajectory_cosine,0.14073388678909235,0.11754158117696331,0.16535144953696912,9600,640,2000,interpolation,neural_operator +magnitude_ratio_mae,-0.46488621293799953,-0.5423367733023673,-0.3933361005466432,9600,640,2000,interpolation,neural_operator +log_gain_error,1.20529130355649,0.7059878212272015,1.6538962324349753,9600,640,2000,interpolation,neural_operator +trajectory_cosine,0.1483581454321589,0.12569826875520804,0.17302015620269795,9600,640,2000,extrapolation,neural_operator +magnitude_ratio_mae,-0.16986458557502676,-0.20310999915587793,-0.13892420616191037,9600,640,2000,extrapolation,neural_operator +log_gain_error,1.2839331418966564,0.7889218767794663,1.735128872910456,9600,640,2000,extrapolation,neural_operator +trajectory_cosine,0.1234450834008881,0.1028966183759613,0.14589873393946615,9600,640,2000,structural_ood,neural_operator +magnitude_ratio_mae,-0.43423194664452847,-0.48484637176497686,-0.38457994564032805,9600,640,2000,structural_ood,neural_operator +log_gain_error,1.2176924126548692,0.7257556734202274,1.6639861287462214,9600,640,2000,structural_ood,neural_operator +trajectory_cosine,0.08013774271811135,0.05252004917827128,0.1087119062991446,9600,640,2000,interpolation,no_spectral_transition +magnitude_ratio_mae,-0.5771663398256835,-0.6814304209701949,-0.4873535173084044,9600,640,2000,interpolation,no_spectral_transition +log_gain_error,2.034260807829366,1.3680159992930518,2.6456570851688155,9600,640,2000,interpolation,no_spectral_transition +trajectory_cosine,0.08112493860301118,0.05260381869230683,0.11045850485206223,9600,640,2000,extrapolation,no_spectral_transition +magnitude_ratio_mae,-0.211490801468802,-0.254663007974353,-0.1734632861709688,9600,640,2000,extrapolation,no_spectral_transition +log_gain_error,2.1157730724615975,1.449922962422541,2.7310231102411633,9600,640,2000,extrapolation,no_spectral_transition +trajectory_cosine,0.05752474528992253,0.03108489484277623,0.0849763414849681,9600,640,2000,structural_ood,no_spectral_transition +magnitude_ratio_mae,-0.6006749469344504,-0.6618095658326444,-0.5412064559252274,9600,640,2000,structural_ood,no_spectral_transition +log_gain_error,2.036446463406707,1.3733236939392517,2.648762940795238,9600,640,2000,structural_ood,no_spectral_transition diff --git a/results/v12_causal_decomposition/stratified_metrics_summary.csv b/results/v12_causal_decomposition/stratified_metrics_summary.csv new file mode 100644 index 0000000000000000000000000000000000000000..23bb4b7f02abb73cc93ce5a6b7d47ee589d0f609 --- /dev/null +++ b/results/v12_causal_decomposition/stratified_metrics_summary.csv @@ -0,0 +1,289 @@ +model,regime,stratum_type,program,horizon,effect_bin,speed_bin,boundary_contact,occlusion_zone_exposure,n_seeds,trajectory_cosine_mean,trajectory_cosine_seed_std,magnitude_ratio_mean,magnitude_ratio_seed_std,magnitude_ratio_mae_mean,magnitude_ratio_mae_seed_std,log_gain_error_mean,log_gain_error_seed_std +neural_operator,extrapolation,boundary_contact,,,,,False,,10,0.08902158900125566,0.1415354460477829,0.7844612683501813,0.394400417804718,1.0672827974086005,0.13968108594417572,8.584474206947878,3.2846198081970215 +neural_operator,extrapolation,effect_bin,,,high,,,,10,0.0949718182847348,0.13574624061584473,0.5275473940480706,0.3040911555290222,0.8869143161213113,0.18237876892089844,9.13960779659184,3.4374947547912598 +neural_operator,extrapolation,effect_bin,,,low,,,,10,0.05142218020042492,0.19985903799533844,0.9614200112353501,0.5851147770881653,1.243484328291379,0.21320751309394836,8.599267474827357,3.4706108570098877 +neural_operator,extrapolation,effect_bin,,,mid,,,,10,0.12202334484670581,0.1391925811767578,0.882146347436547,0.3771926462650299,1.0811621389547852,0.13936807215213776,7.955333641678782,3.241976022720337 +neural_operator,extrapolation,horizon,,10,,,,,10,0.08708906302240393,0.14171727001667023,0.7695593399472273,0.4569760859012604,1.1258882671478205,0.16071419417858124,9.224821975640953,3.0857930183410645 +neural_operator,extrapolation,horizon,,20,,,,,10,0.08686810103980633,0.16509658098220825,0.7816067335457774,0.45562323927879333,0.9997501028352417,0.15172921121120453,8.14983998061507,3.802924156188965 +neural_operator,extrapolation,horizon,,30,,,,,10,0.08860771935785579,0.1709415763616562,0.7710788428563319,0.46716591715812683,0.9116489011910744,0.17551015317440033,7.523534997008392,4.232664108276367 +neural_operator,extrapolation,horizon,,5,,,,,10,0.09502143377952356,0.09816329926252365,0.8259797866184575,0.3309052586555481,1.286697625477488,0.15857268869876862,9.724775097053499,2.056216239929199 +neural_operator,extrapolation,occlusion_zone,,,,,,False,10,0.2738448689226061,0.2461058348417282,0.6459848262893502,0.42808565497398376,0.9497245167265647,0.21146486699581146,8.819298295665067,3.876788854598999 +neural_operator,extrapolation,occlusion_zone,,,,,,True,10,0.06058723824412482,0.15292030572891235,0.80576533635954,0.40929678082466125,1.0853686867442984,0.13651177287101746,8.548347424068309,3.2741026878356934 +neural_operator,extrapolation,overall,,,,,,,10,0.08902158900125566,0.1415354460477829,0.7844612683501813,0.394400417804718,1.0672827974086005,0.13968108594417572,8.584474206947878,3.2846198081970215 +neural_operator,extrapolation,program,impulse_reverse,,,,,,10,0.034195273507680216,0.1601971834897995,0.8892996805095663,0.4535578489303589,1.2060579843935557,0.1271229237318039,8.814084360346897,2.950760841369629 +neural_operator,extrapolation,program,left_right,,,,,,10,0.02143183753670428,0.1930435746908188,0.7613370241326265,0.2989388108253479,0.9202649696497247,0.2115631252527237,7.639454138107249,3.522106647491455 +neural_operator,extrapolation,program,turn_cw,,,,,,10,0.03313905155482643,0.23964984714984894,0.8793201956742754,0.6942440271377563,1.230551946372725,0.2779020667076111,9.107079247959579,4.014198303222656 +neural_operator,extrapolation,program,up_down,,,,,,10,0.2533495590442044,0.23148803412914276,0.6316029049152803,0.4199223518371582,0.9530735764594283,0.18898774683475494,8.907930341630708,3.775481700897217 +neural_operator,extrapolation,program_horizon,impulse_reverse,10,,,,,10,0.04241887993903219,0.1723296344280243,0.7414225713255291,0.3079591691493988,1.0535765580832959,0.1529054343700409,8.76050139744766,2.8696675300598145 +neural_operator,extrapolation,program_horizon,impulse_reverse,20,,,,,10,0.033331422111677965,0.17446191608905792,1.0513033833365626,0.7550410628318787,1.356996041443199,0.3041669726371765,8.870928971655667,3.328007459640503 +neural_operator,extrapolation,program_horizon,impulse_reverse,30,,,,,10,-0.002380128214281285,0.1893911510705948,0.8983102453546963,0.6241941452026367,1.1112789120525122,0.18357187509536743,8.108342358469963,3.9802892208099365 +neural_operator,extrapolation,program_horizon,impulse_reverse,5,,,,,10,0.06341092019429198,0.1142832562327385,0.8661625220214774,0.3393293619155884,1.3023804259952159,0.1734357327222824,9.516564713814295,2.1173152923583984 +neural_operator,extrapolation,program_horizon,left_right,10,,,,,10,0.01855664819559024,0.16988971829414368,0.7431261589073983,0.2974938452243805,0.979084360669367,0.1947675198316574,8.161690147360787,3.367837905883789 +neural_operator,extrapolation,program_horizon,left_right,20,,,,,10,-0.007900694354020743,0.23646967113018036,0.7056424543359754,0.32345834374427795,0.7500451575266197,0.275451123714447,6.849359515588731,4.320095539093018 +neural_operator,extrapolation,program_horizon,left_right,30,,,,,10,0.011660476110955642,0.2960834801197052,0.7304169612656551,0.3481510579586029,0.6495499344076962,0.3037759065628052,6.030202175665181,4.591176509857178 +neural_operator,extrapolation,program_horizon,left_right,5,,,,,10,0.06341092019429198,0.1142832562327385,0.8661625220214774,0.3393293619155884,1.3023804259952159,0.1734357327222824,9.516564713814295,2.1173152923583984 +neural_operator,extrapolation,program_horizon,turn_cw,10,,,,,10,0.07822680293465964,0.3070574998855591,0.9576998568605631,1.0414085388183594,1.4718550375197084,0.5070057511329651,10.715784074319526,3.9515233039855957 +neural_operator,extrapolation,program_horizon,turn_cw,20,,,,,10,0.01985556202562293,0.25927379727363586,0.7848230162868276,0.5901052355766296,1.073905053245835,0.22839373350143433,8.53265833845362,4.549863338470459 +neural_operator,extrapolation,program_horizon,turn_cw,30,,,,,10,0.0013347897041967372,0.21423345804214478,0.8954377138754352,0.6863091588020325,1.145895748352632,0.2580140233039856,8.07279533110559,4.486879348754883 +neural_operator,extrapolation,program_horizon,up_down,10,,,,,10,0.20915392102033364,0.20589151978492737,0.6359887726954184,0.4449504315853119,0.9990371123189107,0.15959708392620087,9.261312283435837,3.573634386062622 +neural_operator,extrapolation,program_horizon,up_down,20,,,,,10,0.30218611437594517,0.28927433490753174,0.584658080223744,0.43284788727760315,0.8180541591253132,0.24020735919475555,8.346413096762262,4.5347466468811035 +neural_operator,extrapolation,program_horizon,up_down,30,,,,,10,0.343815739830552,0.3322316110134125,0.560150450929541,0.4283926784992218,0.7398710099514574,0.29075121879577637,7.882800122792832,5.071950912475586 +neural_operator,extrapolation,program_horizon,up_down,5,,,,,10,0.1582424609499867,0.12374670803546906,0.7456143158124178,0.46566417813301086,1.255332024442032,0.2118261456489563,10.141195863531902,2.418494939804077 +neural_operator,extrapolation,speed_bin,,,,high,,,10,0.08424030281804065,0.1396811455488205,0.7740737898029162,0.39347222447395325,1.0729750960487308,0.142507866024971,8.699613199699105,3.3191449642181396 +neural_operator,extrapolation,speed_bin,,,,low,,,10,0.11824056012090296,0.16248436272144318,0.8479403039168014,0.4277835786342621,1.0324965279411387,0.16751696169376373,7.8808470290237,3.147381544113159 +neural_operator,interpolation,boundary_contact,,,,,False,,10,0.08827813897112113,0.13821415603160858,1.2138625618427856,0.6013453006744385,1.4752943537120395,0.25244519114494324,8.649376951536784,3.2510478496551514 +neural_operator,interpolation,effect_bin,,,high,,,,10,0.10959599802185038,0.13743305206298828,0.7125634235506733,0.37105879187583923,1.0036895379889756,0.19036976993083954,8.797277168137953,3.491823434829712 +neural_operator,interpolation,effect_bin,,,low,,,,10,0.05642676099451831,0.19655682146549225,1.5670879417822603,0.9140930771827698,1.8190440695732832,0.492054283618927,8.594978844671498,3.3896822929382324 +neural_operator,interpolation,effect_bin,,,mid,,,,10,0.09828838756486535,0.1279163956642151,1.3644690562643969,0.6337078213691711,1.605814819203483,0.2978510558605194,8.556357607268442,3.153367757797241 +neural_operator,interpolation,horizon,,10,,,,,10,0.08481106902636384,0.13703858852386475,1.2564551308019873,0.7698513269424438,1.595316626125714,0.3675946891307831,9.304831234528683,3.025973081588745 +neural_operator,interpolation,horizon,,20,,,,,10,0.08551073291225748,0.16328297555446625,1.141124091544043,0.6396961212158203,1.3393559528281913,0.26942527294158936,8.20522771385149,3.7798848152160645 +neural_operator,interpolation,horizon,,30,,,,,10,0.0911357200843895,0.17003250122070312,1.0325766470394229,0.5837492346763611,1.1516872964450158,0.23442095518112183,7.567345598054817,4.220839977264404 +neural_operator,interpolation,horizon,,5,,,,,10,0.09278066549159121,0.08813249319791794,1.495771650033324,0.6029372215270996,1.9279919346949708,0.3747951090335846,9.810345362437268,2.018282651901245 +neural_operator,interpolation,occlusion_zone,,,,,,False,10,0.35294322161469605,0.2932398021221161,1.398205002527684,1.0995291471481323,1.5234567855671048,0.6765588521957397,7.840869271270931,4.600857734680176 +neural_operator,interpolation,occlusion_zone,,,,,,True,10,0.07677096146487875,0.14109209179878235,1.2058476731173553,0.6003941297531128,1.4732003349357323,0.2519551217556,8.68452945937443,3.2168352603912354 +neural_operator,interpolation,overall,,,,,,,10,0.08827813897112113,0.13821415603160858,1.2138625618427856,0.6013453006744385,1.4752943537120395,0.25244519114494324,8.649376951536784,3.2510478496551514 +neural_operator,interpolation,program,impulse_reverse,,,,,,10,0.03243827876262913,0.15225768089294434,1.2817526847923546,0.576431155204773,1.577886345394654,0.23713572323322296,8.873222843342228,2.9288718700408936 +neural_operator,interpolation,program,left_right,,,,,,10,0.02161239778754407,0.18919679522514343,1.1678749744970447,0.46467843651771545,1.2967462288506795,0.17038823664188385,7.71267411619192,3.4884865283966064 +neural_operator,interpolation,program,turn_cw,,,,,,10,0.036066209160041274,0.23000463843345642,1.4257303308462723,1.1756560802459717,1.7674200862335663,0.7034627199172974,9.167752086557448,3.9607949256896973 +neural_operator,interpolation,program,up_down,,,,,,10,0.24994268772150008,0.22849448025226593,1.0330591994863425,0.7261916995048523,1.3321561874996406,0.3608110845088959,8.973452543810708,3.7153637409210205 +neural_operator,interpolation,program_horizon,impulse_reverse,10,,,,,10,0.04268863399393012,0.1678348332643509,1.2108015993901062,0.5033595561981201,1.5000071931630372,0.1891048699617386,8.841761285997928,2.825232744216919 +neural_operator,interpolation,program_horizon,impulse_reverse,20,,,,,10,0.028779098160339345,0.16306333243846893,1.30755362864802,0.8186987638473511,1.6013724321499467,0.39055851101875305,8.909800252318382,3.317917823791504 +neural_operator,interpolation,program_horizon,impulse_reverse,30,,,,,10,-0.005719232191222545,0.18346835672855377,1.0944140593343037,0.6726284623146057,1.2894123955629766,0.26555243134498596,8.13647478222847,3.983572006225586 +neural_operator,interpolation,program_horizon,impulse_reverse,5,,,,,10,0.06400461508746957,0.10253346711397171,1.5142414517969882,0.5905136466026306,1.9207533607026561,0.37065932154655457,9.604855052824131,2.0876147747039795 +neural_operator,interpolation,program_horizon,left_right,10,,,,,10,0.018723259084254096,0.16532979905605316,1.198197197978152,0.4928571581840515,1.4081943466793745,0.17261773347854614,8.239465498644858,3.3283581733703613 +neural_operator,interpolation,program_horizon,left_right,20,,,,,10,-0.007926933353974163,0.23511329293251038,1.0011203403148101,0.45744243264198303,1.012163707637228,0.19727177917957306,6.916976193571463,4.287064552307129 +neural_operator,interpolation,program_horizon,left_right,30,,,,,10,0.011648650332426768,0.29537805914878845,0.9579409078982281,0.44746243953704834,0.845873500383459,0.24266716837882996,6.089399719727226,4.564111709594727 +neural_operator,interpolation,program_horizon,left_right,5,,,,,10,0.06400461508746957,0.10253346711397171,1.5142414517969882,0.5905136466026306,1.9207533607026561,0.37065932154655457,9.604855052824131,2.0876147747039795 +neural_operator,interpolation,program_horizon,turn_cw,10,,,,,10,0.07212495002313517,0.2917441427707672,1.5474680876359344,1.8227299451828003,2.058576207794249,1.220977783203125,10.801147693023086,3.8107686042785645 +neural_operator,interpolation,program_horizon,turn_cw,20,,,,,10,0.02042742315079522,0.25273123383522034,1.4060020522680134,1.0777015686035156,1.6820464799180628,0.616844117641449,8.586197684146464,4.520978927612305 +neural_operator,interpolation,program_horizon,turn_cw,30,,,,,10,0.01564625430619344,0.20702804625034332,1.3237208526348696,0.9551836252212524,1.561637570988387,0.5685814619064331,8.115910882502794,4.4955153465271 +neural_operator,interpolation,program_horizon,up_down,10,,,,,10,0.20570743300413596,0.2026204615831375,1.0693536382037565,0.7964993119239807,1.4144887568661944,0.3943718373775482,9.336950460448861,3.502190113067627 +neural_operator,interpolation,program_horizon,up_down,20,,,,,10,0.3007633436918695,0.2881874740123749,0.8498203449453285,0.6283493041992188,1.0618411916075274,0.22427597641944885,8.407936725369655,4.482443332672119 +neural_operator,interpolation,program_horizon,up_down,30,,,,,10,0.34296720789016033,0.33178818225860596,0.7542307682902901,0.5601972341537476,0.9098257188452408,0.24141739308834076,7.927597007760778,5.03757905960083 +neural_operator,interpolation,program_horizon,up_down,5,,,,,10,0.15033276629983447,0.11534890532493591,1.4588320465059952,1.0445222854614258,1.9424690826795996,0.7739107012748718,10.22132598166354,2.332700252532959 +neural_operator,interpolation,speed_bin,,,,high,,,10,0.08343194253824537,0.1357109695672989,1.215070633346215,0.6069506406784058,1.4931888052785036,0.25531843304634094,8.76553570827932,3.285079002380371 +neural_operator,interpolation,speed_bin,,,,low,,,10,0.11789378383869513,0.16248643398284912,1.2064799026551623,0.6140981912612915,1.365939371916983,0.3051246702671051,7.93951788255462,3.1159489154815674 +neural_operator,structural_ood,boundary_contact,,,,,False,,10,0.0885614043883273,0.13054999709129333,1.1918491303513292,0.5700254440307617,1.4541879992761337,0.23744000494480133,8.659862738158555,3.2429866790771484 +neural_operator,structural_ood,effect_bin,,,high,,,,10,0.12270583051955326,0.13491986691951752,0.7901707722209539,0.40638986229896545,1.071118017011322,0.2036260962486267,8.708119576182217,3.436582565307617 +neural_operator,structural_ood,effect_bin,,,low,,,,10,0.031134339980837773,0.16511377692222595,1.5787597449692934,0.9735803008079529,1.891051064760654,0.5293140411376953,9.038316522408035,3.551518201828003 +neural_operator,structural_ood,effect_bin,,,mid,,,,10,0.10892682449529856,0.13440686464309692,1.2200678210121338,0.46164363622665405,1.4181294756481446,0.1956656277179718,8.262254443832012,2.9979684352874756 +neural_operator,structural_ood,horizon,,10,,,,,10,0.08777896720312128,0.14253468811511993,1.1681183042448537,0.7121753096580505,1.501833619188983,0.3348388075828552,9.309347952762618,3.0243053436279297 +neural_operator,structural_ood,horizon,,20,,,,,10,0.08679142738728274,0.15242129564285278,1.0065621156309135,0.5531929135322571,1.2151769660704304,0.16814225912094116,8.197369155177148,3.7707338333129883 +neural_operator,structural_ood,horizon,,30,,,,,10,0.08626879677146918,0.14842262864112854,1.268093473823137,0.6977611184120178,1.397155498265056,0.337123304605484,7.612560792145087,4.1863555908203125 +neural_operator,structural_ood,horizon,,5,,,,,10,0.09502143345913888,0.09816329926252365,1.3688804601581068,0.5381486415863037,1.7853852183480439,0.34461647272109985,9.806943157346298,2.0264687538146973 +neural_operator,structural_ood,occlusion_zone,,,,,,False,10,0.2837216050605639,0.253699392080307,0.896231387942098,0.5759764909744263,1.1712388583272695,0.2300838828086853,8.754340821551159,3.8919825553894043 +neural_operator,structural_ood,occlusion_zone,,,,,,True,10,0.04952936425387998,0.1429673135280609,1.2509726788331754,0.6099997162818909,1.510777827465907,0.26384857296943665,8.640967121480033,3.2269723415374756 +neural_operator,structural_ood,overall,,,,,,,10,0.0885614043883273,0.13054999709129333,1.1918491303513292,0.5700254440307617,1.4541879992761337,0.23744000494480133,8.659862738158555,3.2429866790771484 +neural_operator,structural_ood,program,impulse_reverse,,,,,,10,0.03381646420507991,0.12346756458282471,1.417555771248135,0.687743604183197,1.702664165094029,0.34401214122772217,8.896419988194248,2.9130380153656006 +neural_operator,structural_ood,program,left_right,,,,,,10,0.021431836739509436,0.1930435746908188,1.0646040454561443,0.40108492970466614,1.2019454355060588,0.2338321954011917,7.692763393447967,3.501683473587036 +neural_operator,structural_ood,program,turn_cw,,,,,,10,0.03134320768661686,0.2335602045059204,1.4885318557654197,1.071209192276001,1.8294272441261756,0.5670233368873596,9.240202333560834,3.9004364013671875 +neural_operator,structural_ood,program,up_down,,,,,,10,0.2533495597466754,0.23148803412914276,0.8708755302891404,0.5525100827217102,1.1765249635907822,0.2148117870092392,8.955150136281736,3.7335376739501953 +neural_operator,structural_ood,program_horizon,impulse_reverse,10,,,,,10,0.04241887939933804,0.1723296344280243,1.087096001644386,0.4273742437362671,1.3679785022512079,0.20390065014362335,8.8253656541463,2.8439598083496094 +neural_operator,structural_ood,program_horizon,impulse_reverse,20,,,,,10,0.03217018874095175,0.1026415079832077,1.5129448735009645,0.9444199800491333,1.7997312292456626,0.5271705389022827,8.933025238197297,3.3121564388275146 +neural_operator,structural_ood,program_horizon,impulse_reverse,30,,,,,10,-0.0027341310238625737,0.12752965092658997,1.6212091971097835,1.0947188138961792,1.804207717999816,0.5767513513565063,8.225206154119224,3.9005539417266846 +neural_operator,structural_ood,program_horizon,impulse_reverse,5,,,,,10,0.06341091970389243,0.1142832487821579,1.4489730127374059,0.5729736089706421,1.8387392108794303,0.3868088126182556,9.602082906314171,2.0894598960876465 +neural_operator,structural_ood,program_horizon,left_right,10,,,,,10,0.01855664680369955,0.16988971829414368,1.1067791410081553,0.4323230981826782,1.3083451717626304,0.21617989242076874,8.235261501092463,3.338348865509033 +neural_operator,structural_ood,program_horizon,left_right,20,,,,,10,-0.007900694744466818,0.23646968603134155,0.8733906517998549,0.37154269218444824,0.9100867273984476,0.2545561194419861,6.883835300523788,4.303523540496826 +neural_operator,structural_ood,program_horizon,left_right,30,,,,,10,0.01166047519491258,0.2960834801197052,0.8292733762791613,0.367446631193161,0.7506106319837272,0.2848482131958008,6.0498738658614455,4.580366611480713 +neural_operator,structural_ood,program_horizon,left_right,5,,,,,10,0.06341091970389243,0.1142832487821579,1.4489730127374059,0.5729736089706421,1.8387392108794303,0.3868088126182556,9.602082906314171,2.0894598960876465 +neural_operator,structural_ood,program_horizon,turn_cw,10,,,,,10,0.08098642015247606,0.31012770533561707,1.555249204672873,1.752695918083191,2.068048223108053,1.1741992235183716,10.849253056943416,3.782686948776245 +neural_operator,structural_ood,program_horizon,turn_cw,20,,,,,10,0.020710100876749495,0.26044806838035583,0.9250549440970645,0.7586477994918823,1.2071384756360204,0.2747816741466522,8.597125082742423,4.470954895019531 +neural_operator,structural_ood,program_horizon,turn_cw,30,,,,,10,-0.007666897969374986,0.16235201060771942,1.9852914185263217,1.3297067880630493,2.213095033634454,0.9255161881446838,8.274228860996663,4.363203048706055 +neural_operator,structural_ood,program_horizon,up_down,10,,,,,10,0.20915392245697148,0.20589151978492737,0.9233488696540008,0.6234409213066101,1.2629625796340407,0.2151578664779663,9.327511598868295,3.508342981338501 +neural_operator,structural_ood,program_horizon,up_down,20,,,,,10,0.3021861146758965,0.28927433490753174,0.71485799312577,0.5044617652893066,0.9437514320015907,0.20644858479499817,8.375490999245084,4.507490158081055 +neural_operator,structural_ood,program_horizon,up_down,30,,,,,10,0.34381574088420164,0.33223164081573486,0.6365999033772823,0.46084320545196533,0.8207086094422266,0.268205463886261,7.900934287603013,5.055881023406982 +neural_operator,structural_ood,program_horizon,up_down,5,,,,,10,0.1582424609696318,0.12374670803546906,1.2086953549995088,0.7128564715385437,1.6786772332852706,0.4234691858291626,10.216663659410552,2.3528566360473633 +neural_operator,structural_ood,speed_bin,,,,high,,,10,0.08404935859970283,0.12882359325885773,1.1633742325246674,0.5683196783065796,1.4426036684079604,0.2426776885986328,8.772453414954922,3.2789483070373535 +neural_operator,structural_ood,speed_bin,,,,low,,,10,0.11613501754103236,0.15066303312778473,1.3658623948475968,0.6610814332962036,1.5249811323594165,0.3305712342262268,7.9718086021807455,3.095627784729004 +no_spectral_transition,extrapolation,boundary_contact,,,,,False,,10,0.15625479583040336,0.17917177081108093,0.9280649735128639,0.40313440561294556,1.108909013302376,0.14840608835220337,7.752634276382937,3.0486655235290527 +no_spectral_transition,extrapolation,effect_bin,,,high,,,,10,0.1340285564389411,0.1853398233652115,0.6669316064991818,0.36498725414276123,0.9321785820958516,0.14055444300174713,8.309872194375133,3.147982358932495 +no_spectral_transition,extrapolation,effect_bin,,,low,,,,10,0.15620067853253203,0.18922097980976105,1.127179589504085,0.5157432556152344,1.2979034310090356,0.21434761583805084,7.729231420867146,3.245591878890991 +no_spectral_transition,extrapolation,effect_bin,,,mid,,,,10,0.18087760494504196,0.20220515131950378,1.0070917307477534,0.3657086193561554,1.105301155470998,0.1640428751707077,7.1613743201976545,2.9709599018096924 +no_spectral_transition,extrapolation,horizon,,10,,,,,10,0.13411592834029307,0.15217497944831848,0.8932460849382533,0.4083082675933838,1.1534633661125553,0.17988084256649017,8.313200467958813,3.011124610900879 +no_spectral_transition,extrapolation,horizon,,20,,,,,10,0.16094045332115456,0.2088679075241089,0.8492891360506283,0.3887864947319031,0.994132668027305,0.12582992017269135,7.516665683954488,3.318857192993164 +no_spectral_transition,extrapolation,horizon,,30,,,,,10,0.19152676304113356,0.2572992146015167,0.9668638418391609,0.474636435508728,0.9783022258110577,0.18148940801620483,6.602131888622535,3.4505183696746826 +no_spectral_transition,extrapolation,horizon,,5,,,,,10,0.13249645288190853,0.0955435037612915,1.0277927837935963,0.4075678288936615,1.3766807199106552,0.19323915243148804,8.853840661200229,2.4502973556518555 +no_spectral_transition,extrapolation,occlusion_zone,,,,,,False,10,0.12053954404473188,0.2774733006954193,0.616832741892722,0.33923736214637756,0.915883656044025,0.17487570643424988,8.273805852083024,3.9664511680603027 +no_spectral_transition,extrapolation,occlusion_zone,,,,,,True,10,0.16174944995127588,0.19365105032920837,0.975946855300578,0.42281103134155273,1.138605222111353,0.16303272545337677,7.672454033967537,2.94905686378479 +no_spectral_transition,extrapolation,overall,,,,,,,10,0.15625479583040336,0.17917177081108093,0.9280649735128639,0.40313440561294556,1.108909013302376,0.14840608835220337,7.752634276382937,3.0486655235290527 +no_spectral_transition,extrapolation,program,impulse_reverse,,,,,,10,0.12777685917346843,0.2026381492614746,1.0288711477138588,0.4196052849292755,1.2514469474350336,0.15553289651870728,8.228578069159994,2.5744621753692627 +no_spectral_transition,extrapolation,program,left_right,,,,,,10,0.20884451275057928,0.3030702471733093,1.0195723308719573,0.38739123940467834,0.9913508599682246,0.17891784012317657,6.473105907457648,2.8974475860595703 +no_spectral_transition,extrapolation,program,turn_cw,,,,,,10,0.14821809037621278,0.18072058260440826,1.0039900686960512,0.6262004971504211,1.2843292070242265,0.2752659022808075,8.437030981403465,3.775705099105835 +no_spectral_transition,extrapolation,program,up_down,,,,,,10,0.1381705446578053,0.2764863073825836,0.6788076205653851,0.34764793515205383,0.9523640872124816,0.16915178298950195,8.042921323765768,3.9074389934539795 +no_spectral_transition,extrapolation,program_horizon,impulse_reverse,10,,,,,10,0.12429677050395185,0.23222629725933075,0.9141578457470974,0.3672524094581604,1.1210119195166044,0.1714816838502884,8.14418505083304,2.7412614822387695 +no_spectral_transition,extrapolation,program_horizon,impulse_reverse,20,,,,,10,0.09248832715857133,0.21200044453144073,1.0818661690263947,0.5135517120361328,1.3457824387587607,0.20373393595218658,8.650895680394024,2.526895046234131 +no_spectral_transition,extrapolation,program_horizon,impulse_reverse,30,,,,,10,0.16048156674396524,0.26493820548057556,1.0190226432086373,0.49397364258766174,1.1050821063574403,0.14970463514328003,7.324714569188655,3.195073127746582 +no_spectral_transition,extrapolation,program_horizon,impulse_reverse,5,,,,,10,0.13384077228738533,0.1305028349161148,1.1004379328733058,0.420566588640213,1.4339113251073285,0.21894177794456482,8.794516976224259,2.2750487327575684 +no_spectral_transition,extrapolation,program_horizon,left_right,10,,,,,10,0.172175396383318,0.27356472611427307,0.984085011910065,0.3854798674583435,1.0614140480291099,0.17329205572605133,7.261814667633734,2.7693233489990234 +no_spectral_transition,extrapolation,program_horizon,left_right,20,,,,,10,0.2411788950850678,0.3930352032184601,0.9893197040524683,0.3994944095611572,0.7923182710073888,0.1916675567626953,5.391125130816363,3.335624933242798 +no_spectral_transition,extrapolation,program_horizon,left_right,30,,,,,10,0.28818298724654595,0.44125935435295105,1.0044466746519902,0.4028828740119934,0.6777597957290709,0.2103351354598999,4.444966855156236,3.572227716445923 +no_spectral_transition,extrapolation,program_horizon,left_right,5,,,,,10,0.13384077228738533,0.1305028349161148,1.1004379328733058,0.420566588640213,1.4339113251073285,0.21894177794456482,8.794516976224259,2.2750487327575684 +no_spectral_transition,extrapolation,program_horizon,turn_cw,10,,,,,10,0.13373253379759262,0.19142308831214905,1.0074527863413096,0.7517151236534119,1.436461486760527,0.46716541051864624,9.265932742971927,3.884121894836426 +no_spectral_transition,extrapolation,program_horizon,turn_cw,20,,,,,10,0.1619149173828191,0.20026275515556335,0.74368684416404,0.5177875757217407,1.0248710501939058,0.1892944574356079,8.378943945188075,4.14966344833374 +no_spectral_transition,extrapolation,program_horizon,turn_cw,30,,,,,10,0.14900681994822662,0.22378018498420715,1.2608305755828042,0.8672085404396057,1.3916550841182471,0.46895360946655273,7.666216256050393,3.686199188232422 +no_spectral_transition,extrapolation,program_horizon,up_down,10,,,,,10,0.1062590126763098,0.21219167113304138,0.6672886957545415,0.34197917580604553,0.9949660101439803,0.1403367519378662,8.580869410396554,3.5545032024383545 +no_spectral_transition,extrapolation,program_horizon,up_down,20,,,,,10,0.14817967365816004,0.33619099855422974,0.5822838269596105,0.33882832527160645,0.8135589121491649,0.2112729549407959,7.645697979419492,4.461745738983154 +no_spectral_transition,extrapolation,program_horizon,up_down,30,,,,,10,0.16843567822579644,0.40932193398475647,0.5831554739132117,0.3667818605899811,0.7387119170394726,0.2526492774486542,6.972629874094855,4.956290245056152 +no_spectral_transition,extrapolation,program_horizon,up_down,5,,,,,10,0.12980781407095493,0.176365926861763,0.882502485634177,0.42435556650161743,1.2622195095173083,0.1707955300807953,8.97248803115217,2.952303886413574 +no_spectral_transition,extrapolation,speed_bin,,,,high,,,10,0.1483945773067376,0.17673172056674957,0.920192559454032,0.4106564223766327,1.1163352167073524,0.15075579285621643,7.896505970107786,3.037606954574585 +no_spectral_transition,extrapolation,speed_bin,,,,low,,,10,0.20428946458613845,0.19952300190925598,0.976174170539059,0.38631877303123474,1.0635266591608523,0.15497641265392303,6.873418370286624,3.217860698699951 +no_spectral_transition,interpolation,boundary_contact,,,,,False,,10,0.14887428304210212,0.17443352937698364,1.4367482931045634,0.6614460349082947,1.5875744805997238,0.3467252552509308,7.820407447263909,3.0198893547058105 +no_spectral_transition,interpolation,effect_bin,,,high,,,,10,0.14839742540136286,0.18436415493488312,0.8745184615053585,0.4476762115955353,1.0727067809132858,0.19403426349163055,8.013937580140773,3.1716887950897217 +no_spectral_transition,interpolation,effect_bin,,,low,,,,10,0.14515599341867078,0.18577641248703003,1.7725036951094542,0.8433489203453064,1.9014659968546674,0.5181126594543457,7.682375273561176,3.1830873489379883 +no_spectral_transition,interpolation,effect_bin,,,mid,,,,10,0.15297173355457397,0.18653088808059692,1.6645718705064154,0.7436606884002686,1.7899446804858286,0.41895467042922974,7.763890547169671,2.8968636989593506 +no_spectral_transition,interpolation,horizon,,10,,,,,10,0.129077203689053,0.14906562864780426,1.4654715260014104,0.6784600615501404,1.697859454594436,0.3631373345851898,8.394906660917332,2.9717347621917725 +no_spectral_transition,interpolation,horizon,,20,,,,,10,0.1535188326894513,0.20256514847278595,1.2363328860692442,0.6351677179336548,1.3522249651286984,0.29541972279548645,7.574731422535843,3.2958383560180664 +no_spectral_transition,interpolation,horizon,,30,,,,,10,0.18302593021489882,0.2526461184024811,1.2432210551730014,0.6477404236793518,1.2251656158245168,0.3025588393211365,6.644247016654117,3.434323310852051 +no_spectral_transition,interpolation,horizon,,5,,,,,10,0.12354212641930644,0.08879618346691132,1.923707509197942,0.8125997185707092,2.2375390222684173,0.5674736499786377,8.95019043617649,2.409449815750122 +no_spectral_transition,interpolation,occlusion_zone,,,,,,False,10,0.12845691192196682,0.310713529586792,0.8993355583003723,0.6158859729766846,1.072532300706953,0.3320233225822449,7.731656903680414,4.563676834106445 +no_spectral_transition,interpolation,occlusion_zone,,,,,,True,10,0.1497619948299341,0.1789924055337906,1.4601140641830064,0.6741341352462769,1.609967618855931,0.36080119013786316,7.8242661665501485,2.9708337783813477 +no_spectral_transition,interpolation,overall,,,,,,,10,0.14887428304210212,0.17443352937698364,1.4367482931045634,0.6614460349082947,1.5875744805997238,0.3467252552509308,7.820407447263909,3.0198893547058105 +no_spectral_transition,interpolation,program,impulse_reverse,,,,,,10,0.11786442195689233,0.19499358534812927,1.5522240325266705,0.6585912108421326,1.7535194662283176,0.3922218382358551,8.296266334370012,2.5473480224609375 +no_spectral_transition,interpolation,program,left_right,,,,,,10,0.20426238453057977,0.30164042115211487,1.573843876640948,0.618441104888916,1.5185436834319261,0.3526862561702728,6.563755028307787,2.8690104484558105 +no_spectral_transition,interpolation,program,turn_cw,,,,,,10,0.13456003166320443,0.17056353390216827,1.467296845105011,0.9736872315406799,1.7239070012699813,0.5560628771781921,8.493287121403652,3.7184112071990967 +no_spectral_transition,interpolation,program,up_down,,,,,,10,0.13523173117300757,0.2747984230518341,1.161265556145736,0.6325189471244812,1.3884109016362345,0.2626762390136719,8.09654122350912,3.8893842697143555 +no_spectral_transition,interpolation,program_horizon,impulse_reverse,10,,,,,10,0.11934497869779079,0.22967149317264557,1.5258722514888703,0.6302701830863953,1.712565673654899,0.37636083364486694,8.239419880835339,2.7032642364501953 +no_spectral_transition,interpolation,program_horizon,impulse_reverse,20,,,,,10,0.07726265155351938,0.1982739120721817,1.4058055800305547,0.7015210390090942,1.656009496562183,0.39076516032218933,8.696517044119537,2.504685878753662 +no_spectral_transition,interpolation,program_horizon,impulse_reverse,30,,,,,10,0.1511285934476291,0.2571934163570404,1.2749100609153652,0.6379413604736328,1.3415716478135438,0.29694664478302,7.359098244877532,3.1871442794799805 +no_spectral_transition,interpolation,program_horizon,impulse_reverse,5,,,,,10,0.12372146412863003,0.12225574254989624,2.0023082376718917,0.7941293120384216,2.3039310468826444,0.5831684470176697,8.890030167647637,2.237670660018921 +no_spectral_transition,interpolation,program_horizon,left_right,10,,,,,10,0.1674788088625064,0.27201056480407715,1.5893591764004669,0.648736298084259,1.6440530011430383,0.3694264888763428,7.355691240588203,2.7327399253845215 +no_spectral_transition,interpolation,program_horizon,left_right,20,,,,,10,0.23899736773846597,0.3929867148399353,1.3913272353456705,0.5701014399528503,1.1676065393490718,0.2781565189361572,5.4848595571471375,3.310863733291626 +no_spectral_transition,interpolation,program_horizon,left_right,30,,,,,10,0.28685189739271666,0.4414638578891754,1.312380857145763,0.5277970433235168,0.9585841463529505,0.25881755352020264,4.52443914784817,3.552302837371826 +no_spectral_transition,interpolation,program_horizon,left_right,5,,,,,10,0.12372146412863003,0.12225574254989624,2.0023082376718917,0.7941293120384216,2.3039310468826444,0.5831684470176697,8.890030167647637,2.237670660018921 +no_spectral_transition,interpolation,program_horizon,turn_cw,10,,,,,10,0.12617040112163522,0.18158872425556183,1.5609959112829528,1.1288889646530151,1.959981853608042,0.7619680762290955,9.335830060509034,3.814868927001953 +no_spectral_transition,interpolation,program_horizon,turn_cw,20,,,,,10,0.15093863314323244,0.19183310866355896,1.2616212566383183,0.980694591999054,1.5208203142974526,0.5688409209251404,8.44034281889908,4.09510612487793 +no_spectral_transition,interpolation,program_horizon,turn_cw,30,,,,,10,0.1265710607247456,0.21739009022712708,1.5792733673937618,1.1638168096542358,1.6909188359044491,0.7457287907600403,7.703688484802842,3.6411285400390625 +no_spectral_transition,interpolation,program_horizon,up_down,10,,,,,10,0.10331462607427966,0.21037039160728455,1.1856587648333516,0.6453795433044434,1.4748372899717652,0.27998003363609314,8.64868546173675,3.5273807048797607 +no_spectral_transition,interpolation,program_horizon,up_down,20,,,,,10,0.14687667832258738,0.33590155839920044,0.8865774722624338,0.5135164260864258,1.0644635103060864,0.17467132210731506,7.6772062699776145,4.460873126983643 +no_spectral_transition,interpolation,program_horizon,up_down,30,,,,,10,0.16755216929450398,0.40933987498283386,0.8063199352371158,0.4933227300643921,0.9095878332271241,0.21040761470794678,6.989762189087924,4.959923267364502 +no_spectral_transition,interpolation,program_horizon,up_down,5,,,,,10,0.12318345100065926,0.17013685405254364,1.7665060522500426,0.9695561528205872,2.104754973039962,0.6415423154830933,9.070510973234196,2.9051506519317627 +no_spectral_transition,interpolation,speed_bin,,,,high,,,10,0.14075237589809958,0.17158624529838562,1.4570058637330265,0.6832861304283142,1.624613204691446,0.3669857978820801,7.968006695326531,3.0077199935913086 +no_spectral_transition,interpolation,speed_bin,,,,low,,,10,0.19850816003322877,0.19708482921123505,1.3129520281528433,0.5783146023750305,1.36122672226142,0.26351502537727356,6.918412042436776,3.1955487728118896 +no_spectral_transition,structural_ood,boundary_contact,,,,,False,,10,0.15448174249929286,0.16137510538101196,1.4670374773750425,0.6423238515853882,1.6206309995660557,0.34490761160850525,7.841108687406717,3.0175695419311523 +no_spectral_transition,structural_ood,effect_bin,,,high,,,,10,0.1530136027685239,0.19400043785572052,0.9948520618848852,0.47254836559295654,1.1713559873169288,0.22932513058185577,7.832788147935643,3.0908610820770264 +no_spectral_transition,structural_ood,effect_bin,,,low,,,,10,0.13975922504622051,0.13956500589847565,1.931627959917228,0.9036282896995544,2.168047495408879,0.5702701807022095,8.538508899350251,3.1364798545837402 +no_spectral_transition,structural_ood,effect_bin,,,mid,,,,10,0.169555056021559,0.1802833378314972,1.4911503217541304,0.6086196303367615,1.5458229369158487,0.3266218304634094,7.202142745696278,3.020543098449707 +no_spectral_transition,structural_ood,horizon,,10,,,,,10,0.1345571163682962,0.15167738497257233,1.327790520872024,0.614462673664093,1.5476078978797887,0.3049229383468628,8.38935512942262,2.980057954788208 +no_spectral_transition,structural_ood,horizon,,20,,,,,10,0.16100903601988192,0.1879214346408844,1.2039674785984709,0.5536695718765259,1.3354466955293902,0.2818399667739868,7.580271262436872,3.2993295192718506 +no_spectral_transition,structural_ood,horizon,,30,,,,,10,0.18436804220370862,0.21195249259471893,1.7189487867413669,0.8314008116722107,1.72327231503441,0.5084824562072754,6.7221993040642705,3.4089791774749756 +no_spectral_transition,structural_ood,horizon,,5,,,,,10,0.13249645304061536,0.0955435037612915,1.6675783385927303,0.7012677192687988,1.961385786572161,0.4376623332500458,8.949775842468565,2.4124600887298584 +no_spectral_transition,structural_ood,occlusion_zone,,,,,,False,10,0.12267543733207276,0.2822658121585846,0.9234217579115647,0.44716525077819824,1.185227508218959,0.16141466796398163,8.24555161140859,3.8965463638305664 +no_spectral_transition,structural_ood,occlusion_zone,,,,,,True,10,0.1608430035327369,0.17691662907600403,1.575760621267738,0.6997020244598389,1.7077116978354752,0.3975943326950073,7.7602201026063415,2.8952980041503906 +no_spectral_transition,structural_ood,overall,,,,,,,10,0.15448174249929286,0.16137510538101196,1.4670374773750425,0.6423238515853882,1.6206309995660557,0.34490761160850525,7.841108687406717,3.0175695419311523 +no_spectral_transition,structural_ood,program,impulse_reverse,,,,,,10,0.12601290231925985,0.15966980159282684,1.8286066040492894,0.8894087076187134,2.0263989174040034,0.6400425434112549,8.357071772776544,2.5311121940612793 +no_spectral_transition,structural_ood,program,left_right,,,,,,10,0.20884451253174346,0.3030702471733093,1.3724917864698765,0.5581783652305603,1.3225333320035133,0.2810096740722656,6.53734309739666,2.8691232204437256 +no_spectral_transition,structural_ood,program,turn_cw,,,,,,10,0.1417047667866717,0.15449246764183044,1.7777669261132059,0.9574980735778809,2.0253487755234043,0.5663118362426758,8.558531695969092,3.7374658584594727 +no_spectral_transition,structural_ood,program,up_down,,,,,,10,0.1381705444313411,0.2764863073825836,0.9669669550523394,0.4612427055835724,1.2094224173226393,0.16665419936180115,8.090843935625163,3.886049747467041 +no_spectral_transition,structural_ood,program_horizon,impulse_reverse,10,,,,,10,0.12429677032178006,0.23222629725933075,1.3175212480533154,0.568085789680481,1.4979526828043164,0.30870679020881653,8.226842862134799,2.7091879844665527 +no_spectral_transition,structural_ood,program_horizon,impulse_reverse,20,,,,,10,0.09184859822695216,0.12447996437549591,1.9728319468965165,1.0786244869232178,2.2277020043693483,0.8422126770019531,8.780242888815701,2.492896318435669 +no_spectral_transition,structural_ood,program_horizon,impulse_reverse,30,,,,,10,0.15406546769354462,0.18566414713859558,2.2494948104299715,1.3531177043914795,2.3207359150983393,1.0213143825531006,7.523553910199553,3.103131055831909 +no_spectral_transition,structural_ood,program_horizon,impulse_reverse,5,,,,,10,0.1338407730347626,0.1305028349161148,1.774578410817354,0.7516716718673706,2.05920506734401,0.5115846991539001,8.897647429956123,2.2379748821258545 +no_spectral_transition,structural_ood,program_horizon,left_right,10,,,,,10,0.1721753956800967,0.27356472611427307,1.4136078417592217,0.5946577191352844,1.4627029097406195,0.3089969754219055,7.358423096593469,2.7302188873291016 +no_spectral_transition,structural_ood,program_horizon,left_right,20,,,,,10,0.241178894859695,0.3930352032184601,1.1844061451178276,0.49029597640037537,0.978723703394644,0.20974449813365936,5.430201780586503,3.315459728240967 +no_spectral_transition,structural_ood,program_horizon,left_right,30,,,,,10,0.2881829865524196,0.44125935435295105,1.1173747481851024,0.4522946774959564,0.7895016475347802,0.21037617325782776,4.463100082450547,3.5611071586608887 +no_spectral_transition,structural_ood,program_horizon,left_right,5,,,,,10,0.1338407730347626,0.1305028349161148,1.774578410817354,0.7516716718673706,2.05920506734401,0.5115846991539001,8.897647429956123,2.2379748821258545 +no_spectral_transition,structural_ood,program_horizon,turn_cw,10,,,,,10,0.1354972867331526,0.1892247498035431,1.5765790482517332,1.137485384941101,1.9498667993582786,0.7894196510314941,9.338605856406502,3.836221694946289 +no_spectral_transition,structural_ood,program_horizon,turn_cw,20,,,,,10,0.1628289776170277,0.20051071047782898,0.9219713046448306,0.5485519766807556,1.1774406467564404,0.20719867944717407,8.432358932867647,4.135646820068359 +no_spectral_transition,structural_ood,program_horizon,turn_cw,30,,,,,10,0.12678803600983488,0.12461764365434647,2.8347504254430533,1.5650557279586792,2.948738880455494,1.1421160697937012,7.904630298633128,3.617793083190918 +no_spectral_transition,structural_ood,program_horizon,up_down,10,,,,,10,0.10625901273815544,0.21219167113304138,1.0034539454238256,0.4818471372127533,1.2799091996159404,0.15735942125320435,8.633548702555709,3.5395090579986572 +no_spectral_transition,structural_ood,program_horizon,up_down,20,,,,,10,0.14817967337585286,0.33619099855422974,0.7366605177347083,0.39623624086380005,0.9579204275971278,0.18065914511680603,7.678281447477639,4.443775177001953 +no_spectral_transition,structural_ood,program_horizon,up_down,30,,,,,10,0.1684356785590353,0.40932193398475647,0.6741751629073406,0.3933298587799072,0.8341128170490265,0.22685907781124115,6.997512924973853,4.939727306365967 +no_spectral_transition,structural_ood,program_horizon,up_down,5,,,,,10,0.12980781305232086,0.176365926861763,1.4535781941434833,0.647801399230957,1.7657472250284627,0.3137415647506714,9.054032667493448,2.9182653427124023 +no_spectral_transition,structural_ood,speed_bin,,,,high,,,10,0.14698827742535905,0.15908969938755035,1.4471165839051154,0.6476490497589111,1.6188787377512817,0.34853485226631165,7.985370783159228,3.0057008266448975 +no_spectral_transition,structural_ood,speed_bin,,,,low,,,10,0.20027514017333284,0.18079686164855957,1.5887762708023743,0.6586551666259766,1.6313392662118982,0.3543471395969391,6.959506991141372,3.189197063446045 +swm_structured_cf_v9,extrapolation,boundary_contact,,,,,False,,10,0.23737973443341454,0.32115113735198975,0.26367045667020456,0.3656962811946869,0.8974182118335738,0.14862127602100372,9.868407348844533,5.406498908996582 +swm_structured_cf_v9,extrapolation,effect_bin,,,high,,,,10,0.2023521791528938,0.2880186438560486,0.1594769572339069,0.20697253942489624,0.8622301731302979,0.1792752891778946,10.053406350332356,5.238326072692871 +swm_structured_cf_v9,extrapolation,effect_bin,,,low,,,,10,0.27892320321948016,0.3868284821510315,0.40417949780372603,0.6352850794792175,0.9604038212192245,0.21860866248607635,9.709618375347926,5.670840263366699 +swm_structured_cf_v9,extrapolation,effect_bin,,,mid,,,,10,0.23236443365286838,0.31370487809181213,0.23092743906451102,0.2998024523258209,0.8700096131522992,0.16872970759868622,9.831081056143892,5.401416778564453 +swm_structured_cf_v9,extrapolation,horizon,,10,,,,,10,0.24923476197594333,0.3449823558330536,0.3382497177392029,0.515852153301239,0.9337035374395782,0.20550060272216797,9.85536868837662,5.492218494415283 +swm_structured_cf_v9,extrapolation,horizon,,20,,,,,10,0.22255788340322397,0.3027989864349365,0.21793445077198417,0.31075596809387207,0.8952432520251022,0.1444469690322876,10.036439343483654,5.277358055114746 +swm_structured_cf_v9,extrapolation,horizon,,30,,,,,10,0.2335101581496019,0.31077098846435547,0.23564295122884787,0.3216460347175598,0.8862101072329096,0.1512431502342224,9.77774376599118,5.461292743682861 +swm_structured_cf_v9,extrapolation,horizon,,5,,,,,10,0.2464949341287138,0.33188092708587646,0.26258279036430965,0.33924663066864014,0.8668818635710827,0.17520776391029358,9.782634347087393,5.437679290771484 +swm_structured_cf_v9,extrapolation,occlusion_zone,,,,,,False,10,0.016855101777133938,0.24830156564712524,0.02215521294329329,0.04731354862451553,0.977844790974632,0.04731355234980583,11.586417675111443,4.699066638946533 +swm_structured_cf_v9,extrapolation,occlusion_zone,,,,,,True,10,0.27130660099591924,0.36047181487083435,0.3008266480128063,0.41605231165885925,0.8850448919657188,0.16897572576999664,9.604098067880393,5.652997016906738 +swm_structured_cf_v9,extrapolation,overall,,,,,,,10,0.23737973443341454,0.32115113735198975,0.26367045667020456,0.3656962811946869,0.8974182118335738,0.14862127602100372,9.868407348844533,5.406498908996582 +swm_structured_cf_v9,extrapolation,program,impulse_reverse,,,,,,10,0.30522849209955893,0.39903926849365234,0.30086465502390636,0.3940518796443939,0.8478286520869005,0.19729271531105042,9.322548516769894,5.91972017288208 +swm_structured_cf_v9,extrapolation,program,left_right,,,,,,10,0.3775555724860169,0.48828446865081787,0.3233388846507296,0.41770532727241516,0.7699260341614718,0.29737260937690735,8.629166153789265,6.70414400100708 +swm_structured_cf_v9,extrapolation,program,turn_cw,,,,,,10,0.26062785547041534,0.3903440535068512,0.45520078425788746,0.8075238466262817,1.0276315853271323,0.356184184551239,9.962592256030378,5.617711067199707 +swm_structured_cf_v9,extrapolation,program,up_down,,,,,,10,0.011919047936917207,0.2480626404285431,0.02316008464521554,0.049279604107141495,0.9768399191321805,0.04927961528301239,11.582868695585057,4.706208229064941 +swm_structured_cf_v9,extrapolation,program_horizon,impulse_reverse,10,,,,,10,0.3310527079273015,0.43714430928230286,0.27654747692868115,0.35737329721450806,0.8207489777356386,0.23366501927375793,9.266635053977371,5.989770412445068 +swm_structured_cf_v9,extrapolation,program_horizon,impulse_reverse,20,,,,,10,0.241777192405425,0.31851282715797424,0.2711558248847723,0.3856070339679718,0.9105012667831034,0.13251695036888123,9.834183119516819,5.4232258796691895 +swm_structured_cf_v9,extrapolation,program_horizon,impulse_reverse,30,,,,,10,0.29399841744452715,0.381820946931839,0.26806448069401084,0.3616752326488495,0.8535582209471613,0.19397065043449402,9.378787759272381,5.839923858642578 +swm_structured_cf_v9,extrapolation,program_horizon,impulse_reverse,5,,,,,10,0.3540856506209821,0.461223304271698,0.38769083758816125,0.5006448030471802,0.8065061428816989,0.25356265902519226,8.810588134313003,6.503067970275879 +swm_structured_cf_v9,extrapolation,program_horizon,left_right,10,,,,,10,0.37671650145202873,0.48729830980300903,0.3350695937871933,0.43297579884529114,0.7624611127888784,0.3068569302558899,8.623535174899734,6.712202548980713 +swm_structured_cf_v9,extrapolation,program_horizon,left_right,20,,,,,10,0.3879031987860799,0.5009994506835938,0.29400485241785645,0.38001489639282227,0.7547935897135176,0.3167803883552551,8.550494181504472,6.799134254455566 +swm_structured_cf_v9,extrapolation,program_horizon,left_right,30,,,,,10,0.3915169390849769,0.5055328607559204,0.2765902548097074,0.3576270341873169,0.7559432912617922,0.31553786993026733,8.53204712443985,6.821743488311768 +swm_structured_cf_v9,extrapolation,program_horizon,left_right,5,,,,,10,0.3540856506209821,0.461223304271698,0.38769083758816125,0.5006448030471802,0.8065061428816989,0.25356265902519226,8.810588134313003,6.503067970275879 +swm_structured_cf_v9,extrapolation,program_horizon,turn_cw,10,,,,,10,0.27357121092112113,0.4513365626335144,0.7247059713242425,1.4348138570785522,1.1682798871654085,0.8143061399459839,9.89739140921738,6.312010765075684 +swm_structured_cf_v9,extrapolation,program_horizon,turn_cw,20,,,,,10,0.25607965541603334,0.38776421546936035,0.27762149047339335,0.491597980260849,0.9446337823756039,0.17919662594795227,10.249404169712216,5.300866603851318 +swm_structured_cf_v9,extrapolation,program_horizon,turn_cw,30,,,,,10,0.25223270007409154,0.3464969992637634,0.36327489097602667,0.551181972026825,0.9699810864403844,0.14245714247226715,9.74098118916154,5.489223480224609 +swm_structured_cf_v9,extrapolation,program_horizon,up_down,10,,,,,10,0.015598627603321846,0.2566443383693695,0.016675828916694717,0.03515789285302162,0.9833241720683873,0.03515790030360222,11.633913115411996,4.598679542541504 +swm_structured_cf_v9,extrapolation,program_horizon,up_down,20,,,,,10,0.004471487005357631,0.26666608452796936,0.028955635311914484,0.06338569521903992,0.9710443692281843,0.06338570266962051,11.511675903201104,4.8571391105651855 +swm_structured_cf_v9,extrapolation,program_horizon,up_down,30,,,,,10,-0.0037074240051879316,0.26201191544532776,0.03464217843564653,0.07583919167518616,0.9653578302823007,0.07583919912576675,11.459158991090954,4.96822452545166 +swm_structured_cf_v9,extrapolation,program_horizon,up_down,5,,,,,10,0.03131350114417728,0.2139727771282196,0.012366695916606437,0.02807939052581787,0.9876333049498498,0.02807939425110817,11.726726772636175,4.403765678405762 +swm_structured_cf_v9,extrapolation,speed_bin,,,,high,,,10,0.23758675588820544,0.32154616713523865,0.2688219992735331,0.37305697798728943,0.90018250708869,0.14568041265010834,9.863614596005643,5.413500785827637 +swm_structured_cf_v9,extrapolation,speed_bin,,,,low,,,10,0.23611460332080356,0.31882619857788086,0.23218880742764103,0.32081544399261475,0.8805252963856415,0.16724972426891327,9.89769639397109,5.364058494567871 +swm_structured_cf_v9,interpolation,boundary_contact,,,,,False,,10,0.22901202576021346,0.3108251690864563,0.424123017656744,0.5804257988929749,1.0104081407740402,0.10387910902500153,9.854668255093275,5.43179988861084 +swm_structured_cf_v9,interpolation,effect_bin,,,high,,,,10,0.21215303072428893,0.29814642667770386,0.21256049680545325,0.2757437825202942,0.8524235664098523,0.19249817728996277,9.946087456166278,5.343542575836182 +swm_structured_cf_v9,interpolation,effect_bin,,,low,,,,10,0.27018023004874736,0.3749844431877136,0.6671477604535169,1.02889084815979,1.187393382618416,0.4558229446411133,9.734148658848593,5.633162975311279 +swm_structured_cf_v9,interpolation,effect_bin,,,mid,,,,10,0.20551117963774165,0.2813495397567749,0.3960495236710426,0.5125768184661865,0.993827052421316,0.0616590790450573,9.881921489877097,5.395051956176758 +swm_structured_cf_v9,interpolation,horizon,,10,,,,,10,0.2361467578110023,0.32793962955474854,0.6092008577366146,0.916283130645752,1.1552517252188408,0.40492644906044006,9.86667956639867,5.4812726974487305 +swm_structured_cf_v9,interpolation,horizon,,20,,,,,10,0.21936190325477584,0.2997666895389557,0.31376372291874643,0.43533310294151306,0.9498049759597051,0.08850464969873428,10.009697529266123,5.319296836853027 +swm_structured_cf_v9,interpolation,horizon,,30,,,,,10,0.23828917377967934,0.3174198567867279,0.296624325236123,0.399448424577713,0.9066954661451746,0.1252129226922989,9.743302524983301,5.507813453674316 +swm_structured_cf_v9,interpolation,horizon,,5,,,,,10,0.2199963490071241,0.2982174754142761,0.4944965470950745,0.6395289301872253,1.0363711474385733,0.09343884885311127,9.780435114602248,5.4594645500183105 +swm_structured_cf_v9,interpolation,occlusion_zone,,,,,,False,10,0.013622232968298098,0.23620103299617767,0.02739240532806697,0.05776609107851982,0.9739162519574165,0.055174946784973145,11.546953334212303,4.7739152908325195 +swm_structured_cf_v9,interpolation,occlusion_zone,,,,,,True,10,0.23837679935986195,0.32119834423065186,0.4413721747145125,0.6036704778671265,1.0119947446356325,0.10986228287220001,9.781090642957665,5.496395587921143 +swm_structured_cf_v9,interpolation,overall,,,,,,,10,0.22901202576021346,0.3108251690864563,0.424123017656744,0.5804257988929749,1.0104081407740402,0.10387910902500153,9.854668255093275,5.43179988861084 +swm_structured_cf_v9,interpolation,program,impulse_reverse,,,,,,10,0.293727270924137,0.3855211138725281,0.4810564383980818,0.6215835809707642,0.970587190889637,0.06770696491003036,9.321988141280599,5.926360130310059 +swm_structured_cf_v9,interpolation,program,left_right,,,,,,10,0.3597551761573413,0.46532097458839417,0.5319234083872288,0.6877908706665039,0.8940569706115639,0.14353099465370178,8.627078869336401,6.7077507972717285 +swm_structured_cf_v9,interpolation,program,turn_cw,,,,,,10,0.25944615552980393,0.383355975151062,0.7254241076640139,1.3078148365020752,1.273415025374076,0.7703384160995483,9.981960241952523,5.568979740142822 +swm_structured_cf_v9,interpolation,program,up_down,,,,,,10,0.010728032871969238,0.24246466159820557,0.033413388679468733,0.07061707228422165,0.9693250973708928,0.0653672069311142,11.519468764518388,4.832630634307861 +swm_structured_cf_v9,interpolation,program_horizon,impulse_reverse,10,,,,,10,0.3127763994503766,0.41407066583633423,0.5101397998165339,0.6593614220619202,0.9826703576371074,0.07754860818386078,9.28070463920012,5.9789323806762695 +swm_structured_cf_v9,interpolation,program_horizon,impulse_reverse,20,,,,,10,0.24787915277993305,0.32917875051498413,0.3520963871385902,0.46417951583862305,0.9594694727100432,0.06268422305583954,9.810254617221654,5.463530540466309 +swm_structured_cf_v9,interpolation,program_horizon,impulse_reverse,30,,,,,10,0.2986476701218635,0.38882866501808167,0.3336310434620827,0.43483811616897583,0.8751587770413607,0.16170892119407654,9.34317404748872,5.888181209564209 +swm_structured_cf_v9,interpolation,program_horizon,impulse_reverse,5,,,,,10,0.31560586134437474,0.41193458437919617,0.7283585231751204,0.941669225692749,1.0650501561700367,0.1376732885837555,8.853819261211902,6.45192289352417 +swm_structured_cf_v9,interpolation,program_horizon,left_right,10,,,,,10,0.35862183509161694,0.4639550447463989,0.576546230353415,0.7457488179206848,0.9199543021153659,0.11351612955331802,8.634377540671267,6.699273586273193 +swm_structured_cf_v9,interpolation,program_horizon,left_right,20,,,,,10,0.37909701177850363,0.48962754011154175,0.4397988342680037,0.5687624216079712,0.8087828998221085,0.24735413491725922,8.521546846581623,6.8367133140563965 +swm_structured_cf_v9,interpolation,program_horizon,left_right,30,,,,,10,0.38569599641487,0.49801570177078247,0.38299004575237633,0.49528664350509644,0.7824405243387446,0.28142622113227844,8.498571828880813,6.865037441253662 +swm_structured_cf_v9,interpolation,program_horizon,left_right,5,,,,,10,0.31560586134437474,0.41193458437919617,0.7283585231751204,0.941669225692749,1.0650501561700367,0.1376732885837555,8.853819261211902,6.45192289352417 +swm_structured_cf_v9,interpolation,program_horizon,turn_cw,10,,,,,10,0.2587960135715548,0.4281156361103058,1.3232261103265377,2.5542640686035156,1.74244143293472,1.8228758573532104,9.986702962749405,6.160910606384277 +swm_structured_cf_v9,interpolation,program_horizon,turn_cw,20,,,,,10,0.24660555955379096,0.37068256735801697,0.4253438533691224,0.7512900233268738,1.0673800913151354,0.32192596793174744,10.24636126914993,5.293492317199707 +swm_structured_cf_v9,interpolation,program_horizon,turn_cw,30,,,,,10,0.2729368934640661,0.3687572777271271,0.42770235929638145,0.6590504050254822,1.0104235518723725,0.20356309413909912,9.712816493958234,5.51397705078125 +swm_structured_cf_v9,interpolation,program_horizon,up_down,10,,,,,10,0.014392783130460884,0.25121840834617615,0.02689129044997165,0.05678066611289978,0.9759408081881702,0.05074391886591911,11.564933122973889,4.736627101898193 +swm_structured_cf_v9,interpolation,program_horizon,up_down,20,,,,,10,0.0038658889068756254,0.26410847902297974,0.03781581689926937,0.08197867125272751,0.9635874399915337,0.07969606667757034,11.460627384111286,4.961090564727783 +swm_structured_cf_v9,interpolation,program_horizon,up_down,30,,,,,10,-0.004123864882082369,0.26033997535705566,0.04217385243365128,0.09178073704242706,0.9587590113282204,0.09027913212776184,11.418647729605436,5.051192283630371 +swm_structured_cf_v9,interpolation,program_horizon,up_down,5,,,,,10,0.02877732433262281,0.20160070061683655,0.026772594934982658,0.05948834866285324,0.9790131299756467,0.04517364501953125,11.63366682138294,4.584439754486084 +swm_structured_cf_v9,interpolation,speed_bin,,,,high,,,10,0.2266381180093351,0.3078136444091797,0.44436635018675885,0.6072984933853149,1.0276253903454,0.11235739290714264,9.85300560605481,5.43483829498291 +swm_structured_cf_v9,interpolation,speed_bin,,,,low,,,10,0.24351923979335913,0.3293532133102417,0.300413763306653,0.41656357049942017,0.9051916156157299,0.14798250794410706,9.864828888106125,5.413529396057129 +swm_structured_cf_v9,structural_ood,boundary_contact,,,,,False,,10,0.21200648778921538,0.289302259683609,0.4435744174124885,0.6095874905586243,1.0199560526316054,0.1440492570400238,9.877555150813423,5.401383399963379 +swm_structured_cf_v9,structural_ood,effect_bin,,,high,,,,10,0.20538571157665864,0.29227855801582336,0.2174177963011492,0.28190886974334717,0.8703202681383118,0.1704520285129547,9.999089278238825,5.299349308013916 +swm_structured_cf_v9,structural_ood,effect_bin,,,low,,,,10,0.2289850340032793,0.3282428979873657,0.7799510412517355,1.1815659999847412,1.2718993021078504,0.5908184051513672,9.784328812133376,5.57462215423584 +swm_structured_cf_v9,structural_ood,effect_bin,,,mid,,,,10,0.20263677740995784,0.27494028210639954,0.3494964614279721,0.4516652822494507,0.9304528303399502,0.10705871134996414,9.84690090845597,5.421670436859131 +swm_structured_cf_v9,structural_ood,horizon,,10,,,,,10,0.2500597730221489,0.3459983468055725,0.5478264547975822,0.8459933400154114,1.0789934732398252,0.3708212077617645,9.861504953540862,5.486119747161865 +swm_structured_cf_v9,structural_ood,horizon,,20,,,,,10,0.18991561334592005,0.2614598870277405,0.33804293230110477,0.4704790711402893,0.9763005096756388,0.09696465730667114,10.052441068540793,5.254611492156982 +swm_structured_cf_v9,structural_ood,horizon,,30,,,,,10,0.1698554475549766,0.2309385985136032,0.4547651968717947,0.6162070035934448,1.063054895051755,0.17007406055927277,9.810457270598272,5.424252510070801 +swm_structured_cf_v9,structural_ood,horizon,,5,,,,,10,0.24692466038201624,0.33225899934768677,0.4303593084351339,0.5557265281677246,0.9419817592017352,0.10463979095220566,9.755238030493881,5.495772361755371 +swm_structured_cf_v9,structural_ood,occlusion_zone,,,,,,False,10,0.01616499082213462,0.2455165684223175,0.025586437165687308,0.05419544130563736,0.9744135647639632,0.05419544503092766,11.531408545896412,4.8109354972839355 +swm_structured_cf_v9,structural_ood,occlusion_zone,,,,,,True,10,0.25117478718263153,0.3339252471923828,0.5271720134618488,0.7231206297874451,1.0290645502051339,0.17809051275253296,9.546784471796826,5.706089496612549 +swm_structured_cf_v9,structural_ood,overall,,,,,,,10,0.21200648778921538,0.289302259683609,0.4435744174124885,0.6095874905586243,1.0199560526316054,0.1440492570400238,9.877555150813423,5.401383399963379 +swm_structured_cf_v9,structural_ood,program,impulse_reverse,,,,,,10,0.24709900071525226,0.3235020637512207,0.5450360836461187,0.7069123387336731,0.9937500274274498,0.07327725738286972,9.34543848905596,5.895035266876221 +swm_structured_cf_v9,structural_ood,program,left_right,,,,,,10,0.3778644384816289,0.48864269256591797,0.45666952803730965,0.5896824598312378,0.8251033985172398,0.22932657599449158,8.632389050518395,6.7012505531311035 +swm_structured_cf_v9,structural_ood,program,turn_cw,,,,,,10,0.2108557896754064,0.33762702345848083,0.846524901271992,1.4453831911087036,1.3770487288129516,0.9002699255943298,10.050427469494753,5.482922554016113 +swm_structured_cf_v9,structural_ood,program,up_down,,,,,,10,0.011919047756121775,0.2480626404285431,0.026804777659409907,0.05662982165813446,0.9731952248141169,0.05662981793284416,11.525183673854917,4.824370861053467 +swm_structured_cf_v9,structural_ood,program_horizon,impulse_reverse,10,,,,,10,0.3313750030007213,0.43742451071739197,0.4305429899133742,0.5560615658760071,0.8728710397146642,0.17571112513542175,9.258394541637973,6.0049638748168945 +swm_structured_cf_v9,structural_ood,program_horizon,impulse_reverse,20,,,,,10,0.11042132952370594,0.15012657642364502,0.49821668984368445,0.6601319909095764,1.0726052148267626,0.13482443988323212,9.876372266840189,5.371872425079346 +swm_structured_cf_v9,structural_ood,program_horizon,impulse_reverse,30,,,,,10,0.1918694297084585,0.24963948130607605,0.6153577257879078,0.8039492964744568,1.1070391832850874,0.17624427378177643,9.42358639985323,5.786844730377197 +swm_structured_cf_v9,structural_ood,program_horizon,impulse_reverse,5,,,,,10,0.3547302406281233,0.4618634283542633,0.6360269290395081,0.8211422562599182,0.9224846718832851,0.13824711740016937,8.823400747892446,6.492198944091797 +swm_structured_cf_v9,structural_ood,program_horizon,left_right,10,,,,,10,0.3770387968048453,0.487669974565506,0.49695922788232566,0.6418489217758179,0.8130664685973897,0.2446528673171997,8.612762522068806,6.727542877197266 +swm_structured_cf_v9,structural_ood,program_horizon,left_right,20,,,,,10,0.38806434590369465,0.5011969208717346,0.3702574527822435,0.4783340394496918,0.784875194565393,0.27834275364875793,8.553057671757415,6.796154975891113 +swm_structured_cf_v9,structural_ood,program_horizon,left_right,30,,,,,10,0.39162437058985233,0.5056670308113098,0.32343450244516136,0.41796377301216125,0.7799872590228916,0.28460997343063354,8.540335260354913,6.811179161071777 +swm_structured_cf_v9,structural_ood,program_horizon,left_right,5,,,,,10,0.3547302406281233,0.4618634283542633,0.6360269290395081,0.8211422562599182,0.9224846718832851,0.13824711740016937,8.823400747892446,6.492198944091797 +swm_structured_cf_v9,structural_ood,program_horizon,turn_cw,10,,,,,10,0.27622666434617715,0.4549317955970764,1.2413461154183607,2.4400463104248047,1.6524938705726526,1.734708547592163,10.015092550613918,6.123669147491455 +swm_structured_cf_v9,structural_ood,program_horizon,turn_cw,20,,,,,10,0.2567052906706522,0.3890487849712372,0.4529023559531197,0.8036503195762634,1.078516857419163,0.3739635944366455,10.300380789674819,5.198876857757568 +swm_structured_cf_v9,structural_ood,program_horizon,turn_cw,30,,,,,10,0.09963541400938994,0.17707692086696625,0.845326232444495,1.2568364143371582,1.4001354584470391,0.7281213998794556,9.835809068195521,5.368049144744873 +swm_structured_cf_v9,structural_ood,program_horizon,up_down,10,,,,,10,0.015598627936851694,0.2566443383693695,0.02245748597626791,0.04741502180695534,0.9775425140745938,0.047415029257535934,11.55977019984275,4.751384735107422 +swm_structured_cf_v9,structural_ood,program_horizon,up_down,20,,,,,10,0.004471487285627518,0.26666608452796936,0.030795230625371383,0.06666189432144165,0.9692047718912363,0.06666188687086105,11.479953545890748,4.922208786010742 +swm_structured_cf_v9,structural_ood,program_horizon,up_down,30,,,,,10,-0.003707424087794304,0.26201191544532776,0.03494232680961455,0.07597372680902481,0.965057679452002,0.07597372680902481,11.442098353989422,5.002985954284668 +swm_structured_cf_v9,structural_ood,program_horizon,up_down,5,,,,,10,0.031313499889802185,0.2139727771282196,0.019024067226385773,0.04331737011671066,0.9809759338386357,0.04331737756729126,11.618912595696747,4.623795032501221 +swm_structured_cf_v9,structural_ood,speed_bin,,,,high,,,10,0.21229598242862696,0.2898106575012207,0.4509085702671545,0.6202887296676636,1.0254586704364328,0.1486663669347763,9.87370865497264,5.406507968902588 +swm_structured_cf_v9,structural_ood,speed_bin,,,,low,,,10,0.2102373538817003,0.28629070520401,0.39875459441175265,0.5443655252456665,0.9863289438243266,0.1242399513721466,9.901061514284876,5.37043571472168 diff --git a/results/v13_axis_equivariance/axis_equivariance_summary.csv b/results/v13_axis_equivariance/axis_equivariance_summary.csv new file mode 100644 index 0000000000000000000000000000000000000000..ba0141276b6bb62f1224d1b02bdca3ded370f594 --- /dev/null +++ b/results/v13_axis_equivariance/axis_equivariance_summary.csv @@ -0,0 +1,181 @@ +model,regime,program,transform,n,base_cosine_mean,base_cosine_std,transformed_cosine_mean,transformed_cosine_std,cosine_drop_mean,cosine_drop_std,base_magnitude_ratio_mean,base_magnitude_ratio_std,transformed_magnitude_ratio_mean,transformed_magnitude_ratio_std,equivariance_effect_cosine_mean,equivariance_effect_cosine_std,equivariance_relative_error_mean,equivariance_relative_error_std,oracle_equivariance_error_mean,oracle_equivariance_error_std +neural_operator,interpolation,left_right,identity,640,0.04453898220745032,0.3722105624436701,0.04453898220745032,0.3722105624436701,0.0,0.0,1.0559674933138012,0.9222918020528581,1.0559674933138012,0.9222918020528581,1.0000000002211675,1.4982387420565145e-08,0.0,0.0,0.0,0.0 +neural_operator,interpolation,left_right,rot90,640,0.04453898220745032,0.3722105624436701,0.3328156749885238,0.352393668143541,0.2882766927810735,0.5290569778349768,1.0559674933138012,0.9222918020528581,0.8142013989592669,0.8568765541866317,0.20187534647966732,0.4116398974432171,72244.08712172024,586044.5233715329,5.389253625587287e-07,1.295566377888212e-07 +neural_operator,interpolation,left_right,flip_x,640,0.04453898220745032,0.3722105624436701,0.05901093970142028,0.3937621765362483,0.014471957493969966,0.2137978204041235,1.0559674933138012,0.9222918020528581,1.0940873136220035,0.944092130828479,-0.18844795207756448,0.5241552376275529,362468.7284336247,2480972.9107383653,5.389253625587287e-07,1.295566377888212e-07 +neural_operator,interpolation,left_right,flip_y,640,0.04453898220745032,0.3722105624436701,0.03375956388381383,0.37086157385529356,-0.010779418323636492,0.2660530931889045,1.0559674933138012,0.9222918020528581,1.0579931622465666,0.8863019667440184,-0.23071381230652332,0.48728645053289976,121913.84456775812,901881.9498323523,0.0,0.0 +neural_operator,interpolation,left_right,swap_xy,640,0.04453898220745032,0.3722105624436701,0.33108296904915735,0.3285861162049501,0.286543986841707,0.5196964083346338,1.0559674933138012,0.9222918020528581,0.7586706614342135,0.7615989570766593,-0.05159512630760868,0.71906870021209,113230.45139226197,933357.8114858868,0.0,0.0 +neural_operator,interpolation,up_down,identity,640,0.33919345730555506,0.3408285770040521,0.33919345730555506,0.3408285770040521,0.0,0.0,0.8200802166718859,0.9025611129445046,0.8200802166718859,0.9025611129445046,1.000000001571725,3.862983240339656e-08,0.0,0.0,0.0,0.0 +neural_operator,interpolation,up_down,rot90,640,0.33919345730555506,0.3408285770040521,0.0376348848833004,0.3801047873561536,-0.30155857242225464,0.5467541044171091,0.8200802166718859,0.9025611129445046,1.1471331343782367,1.0675769031070959,0.13403906965931273,0.38796098757870867,1282861.4175230234,4667693.291732626,0.0,0.0 +neural_operator,interpolation,up_down,flip_x,640,0.33919345730555506,0.3408285770040521,0.32791347476886584,0.33146278434194293,-0.011279982536689204,0.22758526234759818,0.8200802166718859,0.9025611129445046,0.8066402197137904,0.8654173857728389,0.4216333583527905,0.4271151302512053,686932.1353763219,2793455.4479636713,0.0,0.0 +neural_operator,interpolation,up_down,flip_y,640,0.33919345730555506,0.3408285770040521,0.3448690090382115,0.3502451149473457,0.005675551732656459,0.15836749181641763,0.8200802166718859,0.9025611129445046,0.8173533535245951,0.8373851129075929,0.42927894328398775,0.40307835732016484,166288.70162753342,1060153.4960696897,5.292396072498207e-07,1.3328583716974483e-07 +neural_operator,interpolation,up_down,swap_xy,640,0.33919345730555506,0.3408285770040521,0.045072042968513415,0.3808145528237418,-0.29412141433704164,0.53608280801967,0.8200802166718859,0.9025611129445046,1.1063932646909962,1.0313803373923838,-0.022716299224530022,0.7394840394864186,1354103.6923929069,4814686.360175615,0.0,0.0 +neural_operator,interpolation,turn_cw,identity,640,0.02734147448463773,0.22413816395493008,0.02734147448463773,0.22413816395493008,0.0,0.0,1.330525564798154,1.4574772357350045,1.330525564798154,1.4574772357350045,1.0000000029075435,2.326886003499094e-08,0.0,0.0,0.0,0.0 +neural_operator,interpolation,turn_cw,rot90,640,0.02734147448463773,0.22413816395493008,0.22497659893087985,0.2525552629431056,0.19763512444624212,0.3384193974585646,1.330525564798154,1.4574772357350045,1.483104572078446,1.5231053689765508,0.13191070314735964,0.23773962996990758,244470.88580844217,1270071.3553030961,4.398326294285937e-07,1.0174013637758847e-07 +neural_operator,interpolation,turn_cw,flip_x,640,0.02734147448463773,0.22413816395493008,0.03474438312696293,0.23447589377057934,0.007402908642325201,0.13590760451871023,1.330525564798154,1.4574772357350045,1.3709272721549497,1.4582812483040717,-0.20180582475722214,0.38101626714856723,530687.2624821609,2902775.0419282163,4.398326294285937e-07,1.0174013637758847e-07 +neural_operator,interpolation,turn_cw,flip_y,640,0.02734147448463773,0.22413816395493008,0.037552127657909294,0.2288544227279316,0.010210653173271567,0.15975433416480095,1.330525564798154,1.4574772357350045,1.267141032096697,1.3114955490184033,-0.2462862180405172,0.4741077156440764,534341.354031985,2449044.142445803,0.0,0.0 +neural_operator,interpolation,turn_cw,swap_xy,640,0.02734147448463773,0.22413816395493008,0.22185233675554628,0.25339878474208893,0.19451086227090855,0.3425290728062941,1.330525564798154,1.4574772357350045,1.4138168059878808,1.539204346117866,0.0027474485203776867,0.7035958392969466,137907.37314502738,954254.8443725598,0.0,0.0 +neural_operator,interpolation,impulse_reverse,identity,640,-0.00020355322038767554,0.24807025897982576,-0.00020355322038767554,0.24807025897982576,0.0,0.0,1.2004424627877597,1.1883926739885087,1.2004424627877597,1.1883926739885087,1.0000000013888461,2.163506799989145e-08,0.0,0.0,0.0,0.0 +neural_operator,interpolation,impulse_reverse,rot90,640,-0.00020355322038767554,0.24807025897982576,0.24249518828219152,0.23568102512187705,0.2426987415025792,0.35491372853108355,1.2004424627877597,1.1883926739885087,1.0287423240519273,1.0339908400385818,0.11234107264550403,0.36549767050234516,512021.4997234582,2606723.1250479063,5.808969634202299e-07,1.0736969565369344e-07 +neural_operator,interpolation,impulse_reverse,flip_x,640,-0.00020355322038767554,0.24807025897982576,0.02283431310825108,0.2705081341771392,0.023037866328638756,0.2496521446872585,1.2004424627877597,1.1883926739885087,1.236654649756383,1.2062177889709533,-0.23753622885381323,0.46729517361163675,322225.79284307535,1952424.5411672394,5.808969634202299e-07,1.0736969565369344e-07 +neural_operator,interpolation,impulse_reverse,flip_y,640,-0.00020355322038767554,0.24807025897982576,0.0009154712385679887,0.2527158793809212,0.0011190244589556642,0.2245915219644398,1.2004424627877597,1.1883926739885087,1.2187179322332669,1.171726332768948,-0.22944514579407727,0.4898860756990456,363978.33685977,2103732.1038952162,0.0,0.0 +neural_operator,interpolation,impulse_reverse,swap_xy,640,-0.00020355322038767554,0.24807025897982576,0.2279637909515956,0.2216799598069295,0.22816734417198328,0.35424599223784126,1.2004424627877597,1.1883926739885087,0.9722243359542517,0.9868214155900707,-0.08206290372503684,0.6512704169703829,216474.67610385743,1223974.2693506142,0.0,0.0 +neural_operator,extrapolation,left_right,identity,640,0.04466057784467239,0.3725857176795177,0.04466057784467239,0.3725857176795177,0.0,0.0,0.7831846151639184,0.549099631949068,0.7831846151639184,0.549099631949068,1.0000000002211675,1.4982387420565145e-08,0.0,0.0,0.0,0.0 +neural_operator,extrapolation,left_right,rot90,640,0.04466057784467239,0.3725857176795177,0.3328586943898699,0.3532855679137663,0.28819811654519756,0.5296618387625437,0.7831846151639184,0.549099631949068,0.5877036199239682,0.5276302191644489,0.20187534647966732,0.4116398974432171,72244.08712172024,586044.5233715329,5.163252607687241e-07,1.9510397047398952e-07 +neural_operator,extrapolation,left_right,flip_x,640,0.04466057784467239,0.3725857176795177,0.06010128275329407,0.39582326049878713,0.015440704908621683,0.21621179047107042,0.7831846151639184,0.549099631949068,0.7988477550607058,0.5610894236045367,-0.18844795207756448,0.5241552376275529,362468.7284336247,2480972.9107383653,5.163252607687241e-07,1.9510397047398952e-07 +neural_operator,extrapolation,left_right,flip_y,640,0.04466057784467239,0.3725857176795177,0.034222721876903964,0.37013737740838387,-0.010437855967768427,0.26736814957659255,0.7831846151639184,0.549099631949068,0.7884188019845169,0.538032086958773,-0.23071381230652332,0.48728645053289976,121913.84456775812,901881.9498323523,0.0,0.0 +neural_operator,extrapolation,left_right,swap_xy,640,0.04466057784467239,0.3725857176795177,0.33094194362429336,0.32803924672380114,0.28628136577962093,0.5187173630579063,0.7831846151639184,0.549099631949068,0.5615700232322671,0.48412998286867637,-0.05159512630760868,0.71906870021209,113230.45139226197,933357.8114858868,0.0,0.0 +neural_operator,extrapolation,up_down,identity,640,0.3394570818711145,0.34142256832200557,0.3394570818711145,0.34142256832200557,0.0,0.0,0.5686176030891147,0.4983598681560766,0.5686176030891147,0.4983598681560766,1.000000001571725,3.862983240339656e-08,0.0,0.0,0.0,0.0 +neural_operator,extrapolation,up_down,rot90,640,0.3394570818711145,0.34142256832200557,0.039412448109032994,0.3825147982505755,-0.3000446337620815,0.5483368085004082,0.5686176030891147,0.4983598681560766,0.8267145210396848,0.6160610789925256,0.13403906965931273,0.38796098757870867,1282861.4175230234,4667693.291732626,0.0,0.0 +neural_operator,extrapolation,up_down,flip_x,640,0.3394570818711145,0.34142256832200557,0.3290178577893812,0.3333771490541835,-0.010439224081733301,0.22877352430720982,0.5686176030891147,0.4983598681560766,0.5742166884917879,0.5027144690697047,0.4216333583527905,0.4271151302512053,686932.1353763219,2793455.4479636713,0.0,0.0 +neural_operator,extrapolation,up_down,flip_y,640,0.3394570818711145,0.34142256832200557,0.34539571478626385,0.35037971979747334,0.005938632915149356,0.16053579699485754,0.5686176030891147,0.4983598681560766,0.5839674728875707,0.5044918338663256,0.42927894328398775,0.40307835732016484,166288.70162753342,1060153.4960696897,4.770855359659265e-07,1.6448307046652078e-07 +neural_operator,extrapolation,up_down,swap_xy,640,0.3394570818711145,0.34142256832200557,0.04633923833052904,0.3827372536835467,-0.29311784354058545,0.5373517390946165,0.5686176030891147,0.4983598681560766,0.7878146454087982,0.579412459467293,-0.022716299224530022,0.7394840394864186,1354103.6923929069,4814686.360175615,0.0,0.0 +neural_operator,extrapolation,turn_cw,identity,640,0.014032185931500862,0.23009892629130796,0.014032185931500862,0.23009892629130796,0.0,0.0,1.0553738854359835,1.2458256134804795,1.0553738854359835,1.2458256134804795,1.0000000029075435,2.326886003499094e-08,0.0,0.0,0.0,0.0 +neural_operator,extrapolation,turn_cw,rot90,640,0.014032185931500862,0.23009892629130796,0.22653370710468151,0.25216939292282314,0.21250152117318066,0.36620720760176423,1.0553738854359835,1.2458256134804795,1.13784835744882,1.2836607248348118,0.13191070314735964,0.23773962996990758,244470.88580844217,1270071.3553030961,4.6640636996997387e-07,1.0701151166679742e-07 +neural_operator,extrapolation,turn_cw,flip_x,640,0.014032185931500862,0.23009892629130796,0.02284451136147254,0.2434064034007316,0.008812325429971679,0.14552614810609285,1.0553738854359835,1.2458256134804795,1.1079851458081975,1.3061292899895371,-0.20180582475722214,0.38101626714856723,530687.2624821609,2902775.0419282163,4.6640636996997387e-07,1.0701151166679742e-07 +neural_operator,extrapolation,turn_cw,flip_y,640,0.014032185931500862,0.23009892629130796,0.04447536080770078,0.22682799165753215,0.030443174876199918,0.15480307947521638,1.0553738854359835,1.2458256134804795,1.0620178306824528,1.1018339031841173,-0.2462862180405172,0.4741077156440764,534341.354031985,2449044.142445803,0.0,0.0 +neural_operator,extrapolation,turn_cw,swap_xy,640,0.014032185931500862,0.23009892629130796,0.22080162356287475,0.2546457553440653,0.20676943763137387,0.3689040534205614,1.0553738854359835,1.2458256134804795,1.0756537869955083,1.2881920289011393,0.0027474485203776867,0.7035958392969466,137907.37314502738,954254.8443725598,0.0,0.0 +neural_operator,extrapolation,impulse_reverse,identity,640,0.007243584340210418,0.25920275796467857,0.007243584340210418,0.25920275796467857,0.0,0.0,0.9754698474724137,0.9031405083139469,0.9754698474724137,0.9031405083139469,1.0000000013888461,2.163506799989145e-08,0.0,0.0,0.0,0.0 +neural_operator,extrapolation,impulse_reverse,rot90,640,0.007243584340210418,0.25920275796467857,0.2579608716396706,0.2514212694943209,0.2507172872994602,0.3770995899766355,0.9754698474724137,0.9031405083139469,0.8186656691065309,0.7848435533816402,0.11234107264550403,0.36549767050234516,512021.4997234582,2606723.1250479063,5.714595614492168e-07,1.486217255693445e-07 +neural_operator,extrapolation,impulse_reverse,flip_x,640,0.007243584340210418,0.25920275796467857,0.03114329327654559,0.2825896793617847,0.023899708936335175,0.25676297157537303,0.9754698474724137,0.9031405083139469,0.9820407571125542,0.8949293580680688,-0.23753622885381323,0.46729517361163675,322225.79284307535,1952424.5411672394,5.714595614492168e-07,1.486217255693445e-07 +neural_operator,extrapolation,impulse_reverse,flip_y,640,0.007243584340210418,0.25920275796467857,0.006358825412917213,0.2631899794037179,-0.0008847589272932055,0.23126159481086797,0.9754698474724137,0.9031405083139469,0.9932748981773329,0.914411656214868,-0.22944514579407727,0.4898860756990456,363978.33685977,2103732.1038952162,0.0,0.0 +neural_operator,extrapolation,impulse_reverse,swap_xy,640,0.007243584340210418,0.25920275796467857,0.24651617586250724,0.23226798785852631,0.2392725915222968,0.3733228994594088,0.9754698474724137,0.9031405083139469,0.753655497890486,0.7051162865108076,-0.08206290372503684,0.6512704169703829,216474.67610385743,1223974.2693506142,0.0,0.0 +neural_operator,structural_ood,left_right,identity,640,0.044660578547893695,0.372585716891049,0.044660578547893695,0.372585716891049,0.0,0.0,0.8618479126089369,0.6232552571255611,0.8618479126089369,0.6232552571255611,1.0000000002211675,1.4982387420565145e-08,0.0,0.0,0.0,0.0 +neural_operator,structural_ood,left_right,rot90,640,0.044660578547893695,0.372585716891049,0.332858695746836,0.3532855692875787,0.28819811719894234,0.5296618399679393,0.8618479126089369,0.6232552571255611,0.678621302484953,0.6078427529653726,0.20187534647966732,0.4116398974432171,72244.08712172024,586044.5233715329,5.051493925378736e-07,1.2967102135558228e-07 +neural_operator,structural_ood,left_right,flip_x,640,0.044660578547893695,0.372585716891049,0.06010128242933206,0.39582325925580314,0.015440703881438367,0.21621179005797644,0.8618479126089369,0.6232552571255611,0.8906612972961738,0.6380357330972711,-0.18844795207756448,0.5241552376275529,362468.7284336247,2480972.9107383653,5.051493925378736e-07,1.2967102135558228e-07 +neural_operator,structural_ood,left_right,flip_y,640,0.044660578547893695,0.372585716891049,0.03422272140905989,0.3701373759681075,-0.010437857138833806,0.26736814814672083,0.8618479126089369,0.6232552571255611,0.8780381536191271,0.6528701405098793,-0.23071381230652332,0.48728645053289976,121913.84456775812,901881.9498323523,0.0,0.0 +neural_operator,structural_ood,left_right,swap_xy,640,0.044660578547893695,0.372585716891049,0.33094194319423875,0.3280392466045861,0.28628136464634507,0.5187173623476838,0.8618479126089369,0.6232552571255611,0.6345965539637819,0.5513465309168988,-0.05159512630760868,0.71906870021209,113230.45139226197,933357.8114858868,0.0,0.0 +neural_operator,structural_ood,up_down,identity,640,0.33945708093979193,0.34142256732804555,0.33945708093979193,0.34142256732804555,0.0,0.0,0.6480062879028992,0.562630984478706,0.6480062879028992,0.562630984478706,1.000000001571725,3.862983240339656e-08,0.0,0.0,0.0,0.0 +neural_operator,structural_ood,up_down,rot90,640,0.33945708093979193,0.34142256732804555,0.03941244753932551,0.3825147969264965,-0.3000446334004664,0.5483368068769957,0.6480062879028992,0.562630984478706,0.9325333691071137,0.7459663802365649,0.13403906965931273,0.38796098757870867,1282861.4175230234,4667693.291732626,0.0,0.0 +neural_operator,structural_ood,up_down,flip_x,640,0.33945708093979193,0.34142256732804555,0.32901785848605414,0.33337714916372946,-0.010439222453737785,0.22877352414742863,0.6480062879028992,0.562630984478706,0.6639618773495386,0.6077269589273272,0.4216333583527905,0.4271151302512053,686932.1353763219,2793455.4479636713,0.0,0.0 +neural_operator,structural_ood,up_down,flip_y,640,0.33945708093979193,0.34142256732804555,0.3453957137753605,0.350379719501424,0.005938632835568569,0.16053579564540546,0.6480062879028992,0.562630984478706,0.6755305658970429,0.5846702932632444,0.42927894328398775,0.40307835732016484,166288.70162753342,1060153.4960696897,4.7311189232956963e-07,1.1569346711279908e-07 +neural_operator,structural_ood,up_down,swap_xy,640,0.33945708093979193,0.34142256732804555,0.04633923938990847,0.3827372555224704,-0.29311784154988346,0.5373517402186685,0.6480062879028992,0.562630984478706,0.8800999521336053,0.663156275460358,-0.022716299224530022,0.7394840394864186,1354103.6923929069,4814686.360175615,0.0,0.0 +neural_operator,structural_ood,turn_cw,identity,640,0.0010917708305896666,0.1894875215354979,0.0010917708305896666,0.1894875215354979,0.0,0.0,1.9276572126080282,1.6872900317011064,1.9276572126080282,1.6872900317011064,1.0000000029075435,2.326886003499094e-08,0.0,0.0,0.0,0.0 +neural_operator,structural_ood,turn_cw,rot90,640,0.0010917708305896666,0.1894875215354979,0.1576448261457699,0.1840870511171853,0.15655305531518024,0.3199479068274445,1.9276572126080282,1.6872900317011064,1.9843696640338748,1.92573605483401,0.13191070314735964,0.23773962996990758,244470.88580844217,1270071.3553030961,4.3883922140608433e-07,8.368284545369112e-08 +neural_operator,structural_ood,turn_cw,flip_x,640,0.0010917708305896666,0.1894875215354979,0.006850353814388655,0.20072335873540595,0.0057585829837989875,0.13079002232548587,1.9276572126080282,1.6872900317011064,2.00537158298539,1.759956739398661,-0.20180582475722214,0.38101626714856723,530687.2624821609,2902775.0419282163,4.3883922140608433e-07,8.368284545369112e-08 +neural_operator,structural_ood,turn_cw,flip_y,640,0.0010917708305896666,0.1894875215354979,0.022913976023482975,0.17007718680405046,0.021822205192893306,0.2065469330449422,1.9276572126080282,1.6872900317011064,1.971424444875447,1.7818705671105102,-0.2462862180405172,0.4741077156440764,534341.354031985,2449044.142445803,0.0,0.0 +neural_operator,structural_ood,turn_cw,swap_xy,640,0.0010917708305896666,0.1894875215354979,0.15304119077773065,0.1836118336936077,0.151949419947141,0.3215253732867353,1.9276572126080282,1.6872900317011064,1.891524494991927,1.9553433945684786,0.0027474485203776867,0.7035958392969466,137907.37314502738,954254.8443725598,0.0,0.0 +neural_operator,structural_ood,impulse_reverse,identity,640,0.010035325121509685,0.19144665971818528,0.010035325121509685,0.19144665971818528,0.0,0.0,1.609853234286129,1.4340207583321851,1.609853234286129,1.4340207583321851,1.0000000013888461,2.163506799989145e-08,0.0,0.0,0.0,0.0 +neural_operator,structural_ood,impulse_reverse,rot90,640,0.010035325121509685,0.19144665971818528,0.21436396094577503,0.23320390937135613,0.20432863582426536,0.3086863327053313,1.609853234286129,1.4340207583321851,1.5888796799622469,1.366481603105273,0.11234107264550403,0.36549767050234516,512021.4997234582,2606723.1250479063,5.876024880002717e-07,1.0091740803556674e-07 +neural_operator,structural_ood,impulse_reverse,flip_x,640,0.010035325121509685,0.19144665971818528,0.022469406075560982,0.2087471383718156,0.012434080954051296,0.2030050917128532,1.609853234286129,1.4340207583321851,1.6860232951934448,1.5061869059923167,-0.23753622885381323,0.46729517361163675,322225.79284307535,1952424.5411672394,5.876024880002717e-07,1.0091740803556674e-07 +neural_operator,structural_ood,impulse_reverse,flip_y,640,0.010035325121509685,0.19144665971818528,0.0013038099117693492,0.19205176472412788,-0.008731515209740337,0.16777529922872716,1.609853234286129,1.4340207583321851,1.6539761579471814,1.489101688251211,-0.22944514579407727,0.4898860756990456,363978.33685977,2103732.1038952162,0.0,0.0 +neural_operator,structural_ood,impulse_reverse,swap_xy,640,0.010035325121509685,0.19144665971818528,0.20312024690688305,0.20372378189737583,0.19308492178537334,0.29641441468856944,1.609853234286129,1.4340207583321851,1.5563086852652304,1.3845457048327252,-0.08206290372503684,0.6512704169703829,216474.67610385743,1223974.2693506142,0.0,0.0 +swm_structured_cf_v9,interpolation,left_right,identity,640,0.38634087359532715,0.4746152687956996,0.38634087359532715,0.4746152687956996,0.0,0.0,0.3960718150250614,0.5538688861361836,0.3960718150250614,0.5538688861361836,1.0,0.0,0.0,0.0,0.0,0.0 +swm_structured_cf_v9,interpolation,left_right,rot90,640,0.38634087359532715,0.4746152687956996,-0.001024081768240042,0.2537332774969322,-0.3873649553635672,0.5387577766957014,0.3960718150250614,0.5538688861361836,0.040377975437945,0.08544932394382901,-0.017820662586018443,0.5217611171553284,0.40478898454457524,0.49837941025669125,5.389253625587287e-07,1.295566377888212e-07 +swm_structured_cf_v9,interpolation,left_right,flip_x,640,0.38634087359532715,0.4746152687956996,0.38589449487626554,0.473982088493363,-0.0004463787190616131,0.012319871254784922,0.3960718150250614,0.5538688861361836,0.38958409074693917,0.5462718368407762,0.9774409884121269,0.01938239566943502,0.059400806913618,0.08981204981493189,5.389253625587287e-07,1.295566377888212e-07 +swm_structured_cf_v9,interpolation,left_right,flip_y,640,0.38634087359532715,0.4746152687956996,0.3855738382786512,0.4736192956791455,-0.0007670353166759014,0.011285758945569976,0.3960718150250614,0.5538688861361836,0.3927857158705592,0.5503219989445316,0.9722318751737475,0.02269323011339061,0.06034361852798611,0.0892922494101325,0.0,0.0 +swm_structured_cf_v9,interpolation,left_right,swap_xy,640,0.38634087359532715,0.4746152687956996,-0.00317277569200087,0.25339435482550343,-0.389513649287328,0.5413786641724463,0.3960718150250614,0.5538688861361836,0.04282384330371956,0.09037474850744505,0.003894882509484887,0.6254277465240288,0.39782201694324615,0.49260035727440327,0.0,0.0 +swm_structured_cf_v9,interpolation,up_down,identity,640,-0.002143344053911278,0.2548256931792361,-0.002143344053911278,0.2548256931792361,0.0,0.0,0.042455370008739945,0.09035257445819864,0.042455370008739945,0.09035257445819864,1.0,0.0,0.0,0.0,0.0,0.0 +swm_structured_cf_v9,interpolation,up_down,rot90,640,-0.002143344053911278,0.2548256931792361,0.3868208058178425,0.47503998109156886,0.38896414987175376,0.5410995946773525,0.042455370008739945,0.09035257445819864,0.4055681275203824,0.5786101277007512,-0.021323377965018153,0.5251750804993143,2309031.196647117,4636163.334499506,0.0,0.0 +swm_structured_cf_v9,interpolation,up_down,flip_x,640,-0.002143344053911278,0.2548256931792361,-0.005627881887946083,0.256683329385441,-0.003484537834034805,0.06406997966912953,0.042455370008739945,0.09035257445819864,0.042662929770228515,0.0908945942809078,-0.28385797678492963,0.0709365740978026,0.5053228711243719,0.6814519410907673,0.0,0.0 +swm_structured_cf_v9,interpolation,up_down,flip_y,640,-0.002143344053911278,0.2548256931792361,0.0013584938478500889,0.25400294441344773,0.003501837901761367,0.06026458380957612,0.042455370008739945,0.09035257445819864,0.04056227492869319,0.08604393201167239,-0.28370015684049577,0.06300070123385379,0.49501798921264706,0.6642716073574383,5.292396072498207e-07,1.3328583716974483e-07 +swm_structured_cf_v9,interpolation,up_down,swap_xy,640,-0.002143344053911278,0.2548256931792361,0.38766770008951423,0.47605319158130494,0.38981104414342554,0.5417708298794974,0.042455370008739945,0.09035257445819864,0.41137774251401427,0.5912763588891832,-0.00068906438536942,0.6203848414115588,2314186.614377661,4644201.743251666,0.0,0.0 +swm_structured_cf_v9,interpolation,turn_cw,identity,640,0.2728764438070357,0.3515392823715071,0.2728764438070357,0.3515392823715071,0.0,0.0,0.4170165015384555,0.6757752838368918,0.4170165015384555,0.6757752838368918,1.0,0.0,0.0,0.0,0.0,0.0 +swm_structured_cf_v9,interpolation,turn_cw,rot90,640,0.2728764438070357,0.3515392823715071,0.032244272662501315,0.1481679934798357,-0.24063217114453436,0.33861409280345334,0.4170165015384555,0.6757752838368918,0.13398746105085593,0.2599488360222174,0.02015296271058219,0.3941609446886858,0.4592267845757306,0.5747368029728748,4.398326294285937e-07,1.0174013637758847e-07 +swm_structured_cf_v9,interpolation,turn_cw,flip_x,640,0.2728764438070357,0.3515392823715071,0.27316479466389865,0.35152713380573125,0.00028835085686296225,0.0432017737881007,0.4170165015384555,0.6757752838368918,0.4131369418464601,0.6686326943513139,0.8139003622345626,0.15540762493152027,0.07723023986909539,0.10051554731680719,4.398326294285937e-07,1.0174013637758847e-07 +swm_structured_cf_v9,interpolation,turn_cw,flip_y,640,0.2728764438070357,0.3515392823715071,0.28395878556766546,0.3564791862926557,0.011082341760629787,0.0604833732564002,0.4170165015384555,0.6757752838368918,0.41435028053820133,0.585649808141093,0.8258815053850412,0.1261282512170118,62175.068951296314,381625.1407439204,0.0,0.0 +swm_structured_cf_v9,interpolation,turn_cw,swap_xy,640,0.2728764438070357,0.3515392823715071,0.033225313150978764,0.14550945850948613,-0.23965113065605692,0.3377565414825461,0.4170165015384555,0.6757752838368918,0.1346360441617435,0.26321955684823933,0.29072520143515324,0.38017402409131673,0.4530871725641191,0.5682623914383407,0.0,0.0 +swm_structured_cf_v9,interpolation,impulse_reverse,identity,640,0.29374697771854696,0.36848890862589867,0.29374697771854696,0.36848890862589867,0.0,0.0,0.3422499518375844,0.5201609200759484,0.3422499518375844,0.5201609200759484,1.0,0.0,0.0,0.0,0.0,0.0 +swm_structured_cf_v9,interpolation,impulse_reverse,rot90,640,0.29374697771854696,0.36848890862589867,0.002575720849539114,0.20480589612879513,-0.29117125686900786,0.41773275883627325,0.3422499518375844,0.5201609200759484,0.03499597006786059,0.08199367836331248,0.15199592721182853,0.46982515028340227,0.40080550061538817,0.49246957733975466,5.808969634202299e-07,1.0736969565369344e-07 +swm_structured_cf_v9,interpolation,impulse_reverse,flip_x,640,0.29374697771854696,0.36848890862589867,0.29403231078758835,0.36878708031418683,0.00028533306904137133,0.041812695518704285,0.3422499518375844,0.5201609200759484,0.334381494531408,0.5126155109437738,0.8901483335066587,0.08308537411235675,0.08608155975816771,0.11979092245071772,5.808969634202299e-07,1.0736969565369344e-07 +swm_structured_cf_v9,interpolation,impulse_reverse,flip_y,640,0.29374697771854696,0.36848890862589867,0.2933061006013304,0.3689183433785222,-0.0004408771172165871,0.04482555260764126,0.3422499518375844,0.5201609200759484,0.3391561781521887,0.5143073787344278,0.87480573146604,0.0821257859241766,0.09258643543580547,0.1302691475642171,0.0,0.0 +swm_structured_cf_v9,interpolation,impulse_reverse,swap_xy,640,0.29374697771854696,0.36848890862589867,0.0038635029325844996,0.20202125697754092,-0.2898834747859625,0.4156510186520004,0.3422499518375844,0.5201609200759484,0.03829644889494972,0.08836422648736424,-0.008875850355252624,0.4989368417362653,0.40754878148436546,0.5004025200490909,0.0,0.0 +swm_structured_cf_v9,extrapolation,left_right,identity,640,0.39134452836588024,0.4798379146810241,0.39134452836588024,0.4798379146810241,0.0,0.0,0.28239211537875236,0.3570830671664485,0.28239211537875236,0.3570830671664485,1.0,0.0,0.0,0.0,0.0,0.0 +swm_structured_cf_v9,extrapolation,left_right,rot90,640,0.39134452836588024,0.4798379146810241,-0.00033219423693177895,0.2560571459532737,-0.39167672260281206,0.5439634516838212,0.28239211537875236,0.3570830671664485,0.03326004889368032,0.07000864154773684,-0.017820662586018443,0.5217611171553284,0.40478898454457524,0.49837941025669125,5.163252607687241e-07,1.9510397047398952e-07 +swm_structured_cf_v9,extrapolation,left_right,flip_x,640,0.39134452836588024,0.4798379146810241,0.3910439571365714,0.4794713767899242,-0.00030057122930884363,0.013692991197499992,0.28239211537875236,0.3570830671664485,0.27816362483426926,0.3521987810299327,0.9774409884121269,0.01938239566943502,0.059400806913618,0.08981204981493189,5.163252607687241e-07,1.9510397047398952e-07 +swm_structured_cf_v9,extrapolation,left_right,flip_y,640,0.39134452836588024,0.4798379146810241,0.39057225016877056,0.4789306269618499,-0.0007722781971096993,0.012773272106033918,0.28239211537875236,0.3570830671664485,0.2795855902601033,0.3524848400412272,0.9722318751737475,0.02269323011339061,0.06034361852798611,0.0892922494101325,0.0,0.0 +swm_structured_cf_v9,extrapolation,left_right,swap_xy,640,0.39134452836588024,0.4798379146810241,-0.0026154210381840438,0.2556485434483456,-0.3939599494040643,0.5463248556381963,0.28239211537875236,0.3570830671664485,0.0350319120780156,0.07352162168853894,0.003894882509484887,0.6254277465240288,0.39782201694324615,0.49260035727440327,0.0,0.0 +swm_structured_cf_v9,extrapolation,up_down,identity,640,-0.000766775662486907,0.256365468550441,-0.000766775662486907,0.256365468550441,0.0,0.0,0.03410830247755601,0.07187116562344585,0.03410830247755601,0.07187116562344585,1.0,0.0,0.0,0.0,0.0,0.0 +swm_structured_cf_v9,extrapolation,up_down,rot90,640,-0.000766775662486907,0.256365468550441,0.39098501885309817,0.4794152693560506,0.39175179451558506,0.5445164315037221,0.03410830247755601,0.07187116562344585,0.2761332594323903,0.3474518299380007,-0.021323377965018153,0.5251750804993143,2309031.196647117,4636163.334499506,0.0,0.0 +swm_structured_cf_v9,extrapolation,up_down,flip_x,640,-0.000766775662486907,0.256365468550441,-0.005726307329629586,0.2593734036394857,-0.004959531667142679,0.06428608022569439,0.03410830247755601,0.07187116562344585,0.034309484823627626,0.07211740780463595,-0.28385797678492963,0.0709365740978026,0.5053228711243719,0.6814519410907673,0.0,0.0 +swm_structured_cf_v9,extrapolation,up_down,flip_y,640,-0.000766775662486907,0.256365468550441,0.0019686096057072674,0.2557458523806206,0.0027353852681941742,0.059583429858628384,0.03410830247755601,0.07187116562344585,0.03286387032563967,0.06902023573414873,-0.28370015684049577,0.06300070123385379,0.49501798921264706,0.6642716073574383,4.770855359659265e-07,1.6448307046652078e-07 +swm_structured_cf_v9,extrapolation,up_down,swap_xy,640,-0.000766775662486907,0.256365468550441,0.39183957120403645,0.4804195763579336,0.3926063468665234,0.5451614751898984,0.03410830247755601,0.07187116562344585,0.2782434820663184,0.3503303715302321,-0.00068906438536942,0.6203848414115588,2314186.614377661,4644201.743251666,0.0,0.0 +swm_structured_cf_v9,extrapolation,turn_cw,identity,640,0.25472730055917053,0.33338148639006643,0.25472730055917053,0.33338148639006643,0.0,0.0,0.36219456056132915,0.5626679937288915,0.36219456056132915,0.5626679937288915,1.0,0.0,0.0,0.0,0.0,0.0 +swm_structured_cf_v9,extrapolation,turn_cw,rot90,640,0.25472730055917053,0.33338148639006643,0.024034102686891858,0.14581006688152884,-0.23069319787227868,0.33403182156475886,0.36219456056132915,0.5626679937288915,0.10108222819981165,0.20473961886042114,0.02015296271058219,0.3941609446886858,0.4592267845757306,0.5747368029728748,4.6640636996997387e-07,1.0701151166679742e-07 +swm_structured_cf_v9,extrapolation,turn_cw,flip_x,640,0.25472730055917053,0.33338148639006643,0.25565130095928906,0.33432261137722674,0.0009240004001185298,0.047372509889874785,0.36219456056132915,0.5626679937288915,0.3626888522412628,0.5824542038279946,0.8139003622345626,0.15540762493152027,0.07723023986909539,0.10051554731680719,4.6640636996997387e-07,1.0701151166679742e-07 +swm_structured_cf_v9,extrapolation,turn_cw,flip_y,640,0.25472730055917053,0.33338148639006643,0.274290096515324,0.341761784819466,0.01956279595615342,0.07139554372886964,0.36219456056132915,0.5626679937288915,0.3467771051917225,0.46399206177821617,0.8258815053850412,0.1261282512170118,62175.068951296314,381625.1407439204,0.0,0.0 +swm_structured_cf_v9,extrapolation,turn_cw,swap_xy,640,0.25472730055917053,0.33338148639006643,0.026360273417412827,0.14496954917977178,-0.22836702714175772,0.33045047089280627,0.36219456056132915,0.5626679937288915,0.10575016093789599,0.21945334608241476,0.29072520143515324,0.38017402409131673,0.4530871725641191,0.5682623914383407,0.0,0.0 +swm_structured_cf_v9,extrapolation,impulse_reverse,identity,640,0.29395156828686597,0.36515262462029185,0.29395156828686597,0.36515262462029185,0.0,0.0,0.2732834600377828,0.37112749976353077,0.2732834600377828,0.37112749976353077,1.0,0.0,0.0,0.0,0.0,0.0 +swm_structured_cf_v9,extrapolation,impulse_reverse,rot90,640,0.29395156828686597,0.36515262462029185,-0.008019659631188433,0.20907489634338453,-0.30197122791805436,0.43787200219806804,0.2732834600377828,0.37112749976353077,0.025547227675264572,0.056729859898238255,0.15199592721182853,0.46982515028340227,0.40080550061538817,0.49246957733975466,5.714595614492168e-07,1.486217255693445e-07 +swm_structured_cf_v9,extrapolation,impulse_reverse,flip_x,640,0.29395156828686597,0.36515262462029185,0.29233406176790594,0.3636490879822448,-0.001617506518959999,0.04112783718735537,0.2732834600377828,0.37112749976353077,0.2679487102665007,0.36816207898116227,0.8901483335066587,0.08308537411235675,0.08608155975816771,0.11979092245071772,5.714595614492168e-07,1.486217255693445e-07 +swm_structured_cf_v9,extrapolation,impulse_reverse,flip_y,640,0.29395156828686597,0.36515262462029185,0.29305182537063956,0.36434843829778285,-0.000899742916226387,0.049839017608799586,0.2732834600377828,0.37112749976353077,0.2711783729493618,0.367169598676831,0.87480573146604,0.0821257859241766,0.09258643543580547,0.1302691475642171,0.0,0.0 +swm_structured_cf_v9,extrapolation,impulse_reverse,swap_xy,640,0.29395156828686597,0.36515262462029185,-0.010055159993135021,0.20510653486937858,-0.30400672828000097,0.43805700551934157,0.2732834600377828,0.37112749976353077,0.027879054124048536,0.061352506547346024,-0.008875850355252624,0.4989368417362653,0.40754878148436546,0.5004025200490909,0.0,0.0 +swm_structured_cf_v9,structural_ood,left_right,identity,640,0.39147344548255203,0.4799707734477204,0.39147344548255203,0.4799707734477204,0.0,0.0,0.3179252604022622,0.4049762100477042,0.3179252604022622,0.4049762100477042,1.0,0.0,0.0,0.0,0.0,0.0 +swm_structured_cf_v9,structural_ood,left_right,rot90,640,0.39147344548255203,0.4799707734477204,-0.00033219476645172106,0.25605714751993053,-0.3918056402490038,0.544107146374766,0.3179252604022622,0.4049762100477042,0.03322301112320279,0.0698522305340488,-0.017820662586018443,0.5217611171553284,0.40478898454457524,0.49837941025669125,5.051493925378736e-07,1.2967102135558228e-07 +swm_structured_cf_v9,structural_ood,left_right,flip_x,640,0.39147344548255203,0.4799707734477204,0.39097949787974356,0.479371696864992,-0.0004939476028084755,0.011876237311776233,0.3179252604022622,0.4049762100477042,0.31154539044946433,0.39784052695289923,0.9774409884121269,0.01938239566943502,0.059400806913618,0.08981204981493189,5.051493925378736e-07,1.2967102135558228e-07 +swm_structured_cf_v9,structural_ood,left_right,flip_y,640,0.39147344548255203,0.4799707734477204,0.39063671035692094,0.47898400569459987,-0.000836735125631094,0.010685855999199592,0.3179252604022622,0.4049762100477042,0.3143244365230203,0.4011924989549467,0.9722318751737475,0.02269323011339061,0.06034361852798611,0.0892922494101325,0.0,0.0 +swm_structured_cf_v9,structural_ood,left_right,swap_xy,640,0.39147344548255203,0.4799707734477204,-0.002615421051305021,0.25564854430061007,-0.39408886653385705,0.5464320842316023,0.3179252604022622,0.4049762100477042,0.03517215693921356,0.07363793619424254,0.003894882509484887,0.6254277465240288,0.39782201694324615,0.49260035727440327,0.0,0.0 +swm_structured_cf_v9,structural_ood,up_down,identity,640,-0.0007764234178466722,0.25636039893754536,-0.0007764234178466722,0.25636039893754536,0.0,0.0,0.034568566867077204,0.07287861836671075,0.034568566867077204,0.07287861836671075,1.0,0.0,0.0,0.0,0.0,0.0 +swm_structured_cf_v9,structural_ood,up_down,rot90,640,-0.0007764234178466722,0.25636039893754536,0.3910065063275397,0.4794205796770506,0.39178292974538637,0.544511397871936,0.034568566867077204,0.07287861836671075,0.3133527201600373,0.3991547070251864,-0.021323377965018153,0.5251750804993143,2309031.196647117,4636163.334499506,0.0,0.0 +swm_structured_cf_v9,structural_ood,up_down,flip_x,640,-0.0007764234178466722,0.25636039893754536,-0.005741452941365422,0.259365963998501,-0.00496502952351875,0.06428671075331789,0.034568566867077204,0.07287861836671075,0.034552495550109676,0.07280247500896352,-0.28385797678492963,0.0709365740978026,0.5053228711243719,0.6814519410907673,0.0,0.0 +swm_structured_cf_v9,structural_ood,up_down,flip_y,640,-0.0007764234178466722,0.25636039893754536,0.0019686090434212623,0.25574585278396533,0.0027450324612679346,0.05956612736974742,0.034568566867077204,0.07287861836671075,0.03304102417439221,0.06943710404860813,-0.28370015684049577,0.06300070123385379,0.49501798921264706,0.6642716073574383,4.7311189232956963e-07,1.1569346711279908e-07 +swm_structured_cf_v9,structural_ood,up_down,swap_xy,640,-0.0007764234178466722,0.25636039893754536,0.39194700196385385,0.4805321550776969,0.3927234253817005,0.5452860069362784,0.034568566867077204,0.07287861836671075,0.3171798333525658,0.40370831180201167,-0.00068906438536942,0.6203848414115588,2314186.614377661,4644201.743251666,0.0,0.0 +swm_structured_cf_v9,structural_ood,turn_cw,identity,640,0.10814583101919198,0.18675839394560245,0.10814583101919198,0.18675839394560245,0.0,0.0,0.840982637181878,1.260347960500151,0.840982637181878,1.260347960500151,1.0,0.0,0.0,0.0,0.0,0.0 +swm_structured_cf_v9,structural_ood,turn_cw,rot90,640,0.10814583101919198,0.18675839394560245,0.031649735929750025,0.10839123934038009,-0.07649609508944195,0.17911607592399528,0.840982637181878,1.260347960500151,0.09603483418250107,0.16299787609261038,0.02015296271058219,0.3941609446886858,0.4592267845757306,0.5747368029728748,4.3883922140608433e-07,8.368284545369112e-08 +swm_structured_cf_v9,structural_ood,turn_cw,flip_x,640,0.10814583101919198,0.18675839394560245,0.10950923142081592,0.18686799428428566,0.0013634004016239488,0.04740168588616424,0.840982637181878,1.260347960500151,0.8347731729038059,1.2508611358357025,0.8139003622345626,0.15540762493152027,0.07723023986909539,0.10051554731680719,4.3883922140608433e-07,8.368284545369112e-08 +swm_structured_cf_v9,structural_ood,turn_cw,flip_y,640,0.10814583101919198,0.18675839394560245,0.14942547537502832,0.19640344502077553,0.04127964435583635,0.12206698412617202,0.840982637181878,1.260347960500151,0.8999371130950748,1.1313554644934312,0.8258815053850412,0.1261282512170118,62175.068951296314,381625.1407439204,0.0,0.0 +swm_structured_cf_v9,structural_ood,turn_cw,swap_xy,640,0.10814583101919198,0.18675839394560245,0.03653727546161463,0.11616994755259691,-0.07160855555757735,0.1753759619112634,0.840982637181878,1.260347960500151,0.10314584212610498,0.20429955408969683,0.29072520143515324,0.38017402409131673,0.4530871725641191,0.5682623914383407,0.0,0.0 +swm_structured_cf_v9,structural_ood,impulse_reverse,identity,640,0.19574989657849073,0.25023229232303834,0.19574989657849073,0.25023229232303834,0.0,0.0,0.5815667312592268,0.7525868177302367,0.5815667312592268,0.7525868177302367,1.0,0.0,0.0,0.0,0.0,0.0 +swm_structured_cf_v9,structural_ood,impulse_reverse,rot90,640,0.19574989657849073,0.25023229232303834,-0.022519367876157048,0.17333517987157743,-0.21826926445464778,0.33655505005665864,0.5815667312592268,0.7525868177302367,0.039399811956471534,0.09726042595167726,0.15199592721182853,0.46982515028340227,0.40080550061538817,0.49246957733975466,5.876024880002717e-07,1.0091740803556674e-07 +swm_structured_cf_v9,structural_ood,impulse_reverse,flip_x,640,0.19574989657849073,0.25023229232303834,0.193232440832071,0.24725208771961968,-0.0025174557464197277,0.04248602510779071,0.5815667312592268,0.7525868177302367,0.5734183484688401,0.7492690104326286,0.8901483335066587,0.08308537411235675,0.08608155975816771,0.11979092245071772,5.876024880002717e-07,1.0091740803556674e-07 +swm_structured_cf_v9,structural_ood,impulse_reverse,flip_y,640,0.19574989657849073,0.25023229232303834,0.19517250584904106,0.24893179712550523,-0.0005773907294496893,0.0497984255744274,0.5815667312592268,0.7525868177302367,0.5832553044892848,0.7539753493236855,0.87480573146604,0.0821257859241766,0.09258643543580547,0.1302691475642171,0.0,0.0 +swm_structured_cf_v9,structural_ood,impulse_reverse,swap_xy,640,0.19574989657849073,0.25023229232303834,-0.02655330558409332,0.17604060385717024,-0.22230320216258406,0.3419672951861133,0.5815667312592268,0.7525868177302367,0.04211006811759265,0.10393527289259008,-0.008875850355252624,0.4989368417362653,0.40754878148436546,0.5004025200490909,0.0,0.0 +no_spectral_transition,interpolation,left_right,identity,640,0.3087480366140426,0.47483354319257487,0.3087480366140426,0.47483354319257487,0.0,0.0,1.4686445670594366,1.0561305056527048,1.4686445670594366,1.0561305056527048,1.0000000058793697,3.3031897111300934e-08,0.0,0.0,0.0,0.0 +no_spectral_transition,interpolation,left_right,rot90,640,0.3087480366140426,0.47483354319257487,0.19271670188900317,0.4737134062007767,-0.11603133472503942,0.6466052333510299,1.4686445670594366,1.0561305056527048,0.8779535359612055,0.8732059612372708,0.10671294408216583,0.43994682203415075,157256.05735811294,1295140.6235405423,5.389253625587287e-07,1.295566377888212e-07 +no_spectral_transition,interpolation,left_right,flip_x,640,0.3087480366140426,0.47483354319257487,0.31152833935625496,0.47834359012085764,0.0027803027422123707,0.18904217667287884,1.4686445670594366,1.0561305056527048,1.4470635744859464,0.9695748808277457,0.17946878091701313,0.42222273504789215,215297.6816428254,1698181.5500575327,5.389253625587287e-07,1.295566377888212e-07 +no_spectral_transition,interpolation,left_right,flip_y,640,0.3087480366140426,0.47483354319257487,0.294418509794923,0.47048124667991087,-0.014329526819119564,0.24682003586588164,1.4686445670594366,1.0561305056527048,1.4305552175042977,1.0217379798827912,0.17949872129071298,0.4364244295854938,173589.75882552852,1669030.1932780845,0.0,0.0 +no_spectral_transition,interpolation,left_right,swap_xy,640,0.3087480366140426,0.47483354319257487,0.20229950177672434,0.44748536203890543,-0.10644853483731823,0.6383834404592488,1.4686445670594366,1.0561305056527048,0.8844051609512462,0.9181060169698944,0.26146046695414865,0.6618994104017073,121972.6276643554,1157881.8661297392,0.0,0.0 +no_spectral_transition,interpolation,up_down,identity,640,0.19046226892187407,0.4513127477868361,0.19046226892187407,0.4513127477868361,0.0,0.0,0.9024124356339598,0.9375099803187943,0.9024124356339598,0.9375099803187943,0.9999999952028454,3.3080608608020424e-08,0.0,0.0,0.0,0.0 +no_spectral_transition,interpolation,up_down,rot90,640,0.19046226892187407,0.4513127477868361,0.292486037956769,0.4692764452631627,0.10202376903489493,0.628040417992089,0.9024124356339598,0.9375099803187943,1.4896869776479433,1.1381480432586975,0.0951617500886483,0.43914595606404006,2149395.651720482,6009089.973597164,0.0,0.0 +no_spectral_transition,interpolation,up_down,flip_x,640,0.19046226892187407,0.4513127477868361,0.20879933382362878,0.4532726704282677,0.01833706490175473,0.23187003750823065,0.9024124356339598,0.9375099803187943,0.8908994275254372,0.9544954093537955,0.21574318416727087,0.5405982959590553,133951.29701980372,1126582.9731745026,0.0,0.0 +no_spectral_transition,interpolation,up_down,flip_y,640,0.19046226892187407,0.4513127477868361,0.1950272852618582,0.4626058149269533,0.0045650163399841405,0.26021794780803725,0.9024124356339598,0.9375099803187943,0.8919253020179229,0.910902896666297,0.23847393789318533,0.542867489068983,107962.24500306493,1223899.7794815323,5.292396072498207e-07,1.3328583716974483e-07 +no_spectral_transition,interpolation,up_down,swap_xy,640,0.19046226892187407,0.4513127477868361,0.31307459143965843,0.4708190736378099,0.12261232251778438,0.6221092748027596,0.9024124356339598,0.9375099803187943,1.4843752319069607,1.0922749835675183,0.30760941383280616,0.6537036056051472,2232569.273571168,6022105.731704469,0.0,0.0 +no_spectral_transition,interpolation,turn_cw,identity,640,0.12496158737812949,0.2612235089678876,0.12496158737812949,0.2612235089678876,0.0,0.0,1.6676726740086452,1.558426300087178,1.6676726740086452,1.558426300087178,1.0000000047811213,2.3813951704560475e-08,0.0,0.0,0.0,0.0 +no_spectral_transition,interpolation,turn_cw,rot90,640,0.12496158737812949,0.2612235089678876,0.044454178442993,0.25564738188853064,-0.08050740893513648,0.31907636312236826,1.6676726740086452,1.558426300087178,1.8199152004980548,1.6492338263563542,0.02260044592261253,0.2646870944518964,9334.407117872592,160765.71181133436,4.398326294285937e-07,1.0174013637758847e-07 +no_spectral_transition,interpolation,turn_cw,flip_x,640,0.12496158737812949,0.2612235089678876,0.12615423669958545,0.27318718332384545,0.001192649321455974,0.14880612552241207,1.6676726740086452,1.558426300087178,1.6993579864036292,1.5824261479644282,0.15042538149417517,0.3295061513642587,51939.58189334608,443959.65442105033,4.398326294285937e-07,1.0174013637758847e-07 +no_spectral_transition,interpolation,turn_cw,flip_y,640,0.12496158737812949,0.2612235089678876,0.09049364859165507,0.2847274821571597,-0.0344679387864744,0.22257995572969058,1.6676726740086452,1.558426300087178,1.9329846260603518,1.5148556529068704,0.06417183970189876,0.34226133005878256,51907.202920518626,571513.1664584517,0.0,0.0 +no_spectral_transition,interpolation,turn_cw,swap_xy,640,0.12496158737812949,0.2612235089678876,0.04412857213319512,0.24332511832735915,-0.08083301524493436,0.3075175914313212,1.6676726740086452,1.558426300087178,1.8299427035631983,1.631045367723131,0.221384908901178,0.619371713508572,16788.09381715115,157288.96760848848,0.0,0.0 +no_spectral_transition,interpolation,impulse_reverse,identity,640,0.16973559683776784,0.2993348557294975,0.16973559683776784,0.2993348557294975,0.0,0.0,1.477073444173584,1.2669052888350671,1.477073444173584,1.2669052888350671,1.0000000045295412,2.9818436444338115e-08,0.0,0.0,0.0,0.0 +no_spectral_transition,interpolation,impulse_reverse,rot90,640,0.16973559683776784,0.2993348557294975,0.060285835022023095,0.3251424014425823,-0.10944976181574476,0.39229940708754707,1.477073444173584,1.2669052888350671,1.073009061067205,1.0839920718263625,0.12112216150219966,0.33423657383666694,188434.60920509015,1192955.3251682294,5.808969634202299e-07,1.0736969565369344e-07 +no_spectral_transition,interpolation,impulse_reverse,flip_x,640,0.16973559683776784,0.2993348557294975,0.16441711530055728,0.30914423336838287,-0.005318481537210573,0.15466391967269402,1.477073444173584,1.2669052888350671,1.4902556091103178,1.1971864289702672,0.188044030826832,0.38113069665351557,57637.51012191949,477364.47078330914,5.808969634202299e-07,1.0736969565369344e-07 +no_spectral_transition,interpolation,impulse_reverse,flip_y,640,0.16973559683776784,0.2993348557294975,0.15896545147101127,0.30243413051477885,-0.010770145366756575,0.16200785671344564,1.477073444173584,1.2669052888350671,1.4432982711397926,1.2507440691581804,0.18389180714450035,0.38320499654180756,152934.08584398142,901352.6149246461,0.0,0.0 +no_spectral_transition,interpolation,impulse_reverse,swap_xy,640,0.16973559683776784,0.2993348557294975,0.0732881071537804,0.3165073271638476,-0.09644748968398745,0.3851902993873829,1.477073444173584,1.2669052888350671,1.124014377642925,1.1390926843777194,0.2205291109049093,0.5533479143408648,247680.84409627813,1252200.028369553,0.0,0.0 +no_spectral_transition,extrapolation,left_right,identity,640,0.31159373231967036,0.47571342442023584,0.31159373231967036,0.47571342442023584,0.0,0.0,1.0737697619140818,0.5678673823318893,1.0737697619140818,0.5678673823318893,1.0000000058793697,3.3031897111300934e-08,0.0,0.0,0.0,0.0 +no_spectral_transition,extrapolation,left_right,rot90,640,0.31159373231967036,0.47571342442023584,0.19395260312412574,0.47692997068021503,-0.11764112919554463,0.6508360448386067,1.0737697619140818,0.5678673823318893,0.6298998987020241,0.5331595841483132,0.10671294408216583,0.43994682203415075,157256.05735811294,1295140.6235405423,5.163252607687241e-07,1.9510397047398952e-07 +no_spectral_transition,extrapolation,left_right,flip_x,640,0.31159373231967036,0.47571342442023584,0.31294577515072886,0.47923126689864287,0.0013520428310584975,0.19096354067446017,1.0737697619140818,0.5678673823318893,1.088289411040023,0.5885379710707135,0.17946878091701313,0.42222273504789215,215297.6816428254,1698181.5500575327,5.163252607687241e-07,1.9510397047398952e-07 +no_spectral_transition,extrapolation,left_right,flip_y,640,0.31159373231967036,0.47571342442023584,0.2968570764127435,0.47062213422605087,-0.0147366559069269,0.2491252480728891,1.0737697619140818,0.5678673823318893,1.0514509756107144,0.5726562422006394,0.17949872129071298,0.4364244295854938,173589.75882552852,1669030.1932780845,0.0,0.0 +no_spectral_transition,extrapolation,left_right,swap_xy,640,0.31159373231967036,0.47571342442023584,0.2048405879751954,0.44926001568516627,-0.106753144344475,0.641016827496966,1.0737697619140818,0.5678673823318893,0.6261073515358475,0.5271459022141963,0.26146046695414865,0.6618994104017073,121972.6276643554,1157881.8661297392,0.0,0.0 +no_spectral_transition,extrapolation,up_down,identity,640,0.19253133184911347,0.4543889434262181,0.19253133184911347,0.4543889434262181,0.0,0.0,0.6183534880165098,0.5245692818933856,0.6183534880165098,0.5245692818933856,0.9999999952028454,3.3080608608020424e-08,0.0,0.0,0.0,0.0 +no_spectral_transition,extrapolation,up_down,rot90,640,0.19253133184911347,0.4543889434262181,0.2950527817112743,0.47175588451238465,0.1025214498621608,0.6312247666780901,0.6183534880165098,0.5245692818933856,1.0666320074495161,0.5943877358551833,0.0951617500886483,0.43914595606404006,2149395.651720482,6009089.973597164,0.0,0.0 +no_spectral_transition,extrapolation,up_down,flip_x,640,0.19253133184911347,0.4543889434262181,0.21029855446407736,0.45558385795114525,0.017767222614963885,0.23575997792958198,0.6183534880165098,0.5245692818933856,0.6104706817683109,0.5196017790630656,0.21574318416727087,0.5405982959590553,133951.29701980372,1126582.9731745026,0.0,0.0 +no_spectral_transition,extrapolation,up_down,flip_y,640,0.19253133184911347,0.4543889434262181,0.19763366865554416,0.4643253208902146,0.005102336806430685,0.2644335640456788,0.6183534880165098,0.5245692818933856,0.6143579629195447,0.49243210717478336,0.23847393789318533,0.542867489068983,107962.24500306493,1223899.7794815323,4.770855359659265e-07,1.6448307046652078e-07 +no_spectral_transition,extrapolation,up_down,swap_xy,640,0.19253133184911347,0.4543889434262181,0.3159769149882209,0.47349768047839313,0.12344558313910739,0.626172329520431,0.6183534880165098,0.5245692818933856,1.0677178819123583,0.5808978300711671,0.30760941383280616,0.6537036056051472,2232569.273571168,6022105.731704469,0.0,0.0 +no_spectral_transition,extrapolation,turn_cw,identity,640,0.15152412885618105,0.26161741691850293,0.15152412885618105,0.26161741691850293,0.0,0.0,1.4684620361076668,1.3691230207728586,1.4684620361076668,1.3691230207728586,1.0000000047811213,2.3813951704560475e-08,0.0,0.0,0.0,0.0 +no_spectral_transition,extrapolation,turn_cw,rot90,640,0.15152412885618105,0.26161741691850293,0.05707815104153724,0.26152492614257944,-0.09444597781464381,0.29438118951227615,1.4684620361076668,1.3691230207728586,1.4763808131555887,1.4819912273742009,0.02260044592261253,0.2646870944518964,9334.407117872592,160765.71181133436,4.6640636996997387e-07,1.0701151166679742e-07 +no_spectral_transition,extrapolation,turn_cw,flip_x,640,0.15152412885618105,0.26161741691850293,0.15537014261044532,0.26694629501264744,0.003846013754264277,0.13477648887186403,1.4684620361076668,1.3691230207728586,1.5057828101911581,1.3952332447543814,0.15042538149417517,0.3295061513642587,51939.58189334608,443959.65442105033,4.6640636996997387e-07,1.0701151166679742e-07 +no_spectral_transition,extrapolation,turn_cw,flip_y,640,0.15152412885618105,0.26161741691850293,0.0974346314977538,0.3192267679755275,-0.05408949735842725,0.22228061384836956,1.4684620361076668,1.3691230207728586,1.5007829871028662,1.261121432206418,0.06417183970189876,0.34226133005878256,51907.202920518626,571513.1664584517,0.0,0.0 +no_spectral_transition,extrapolation,turn_cw,swap_xy,640,0.15152412885618105,0.26161741691850293,0.05683988069722545,0.2523056763274765,-0.0946842481589556,0.28219731642546375,1.4684620361076668,1.3691230207728586,1.4324714073712466,1.3972312676565628,0.221384908901178,0.619371713508572,16788.09381715115,157288.96760848848,0.0,0.0 +no_spectral_transition,extrapolation,impulse_reverse,identity,640,0.18478133232929395,0.3089531144036716,0.18478133232929395,0.3089531144036716,0.0,0.0,1.1116540825139964,0.7977183144365138,1.1116540825139964,0.7977183144365138,1.0000000045295412,2.9818436444338115e-08,0.0,0.0,0.0,0.0 +no_spectral_transition,extrapolation,impulse_reverse,rot90,640,0.18478133232929395,0.3089531144036716,0.07290944815582634,0.33526217951391274,-0.1118718841734676,0.4110050419758618,1.1116540825139964,0.7977183144365138,0.9230596377074107,0.9125986098286359,0.12112216150219966,0.33423657383666694,188434.60920509015,1192955.3251682294,5.714595614492168e-07,1.486217255693445e-07 +no_spectral_transition,extrapolation,impulse_reverse,flip_x,640,0.18478133232929395,0.3089531144036716,0.17352022096310976,0.3229334339193539,-0.011261111366184195,0.1652226063046068,1.1116540825139964,0.7977183144365138,1.1313272421652072,0.8243484636170563,0.188044030826832,0.38113069665351557,57637.51012191949,477364.47078330914,5.714595614492168e-07,1.486217255693445e-07 +no_spectral_transition,extrapolation,impulse_reverse,flip_y,640,0.18478133232929395,0.3089531144036716,0.17415565657866025,0.30545549189496896,-0.01062567575063369,0.161016393632022,1.1116540825139964,0.7977183144365138,1.1185582002861338,0.8328659600904036,0.18389180714450035,0.38320499654180756,152934.08584398142,901352.6149246461,0.0,0.0 +no_spectral_transition,extrapolation,impulse_reverse,swap_xy,640,0.18478133232929395,0.3089531144036716,0.08935862336268201,0.32133750456536847,-0.09542270896661194,0.3991064300399343,1.1116540825139964,0.7977183144365138,0.9520515174278443,0.8841980782320361,0.2205291109049093,0.5533479143408648,247680.84409627813,1252200.028369553,0.0,0.0 +no_spectral_transition,structural_ood,left_right,identity,640,0.31159373267432783,0.4757134242178992,0.31159373267432783,0.4757134242178992,0.0,0.0,1.1911146774427834,0.6912302155443588,1.1911146774427834,0.6912302155443588,1.0000000058793697,3.3031897111300934e-08,0.0,0.0,0.0,0.0 +no_spectral_transition,structural_ood,left_right,rot90,640,0.31159373267432783,0.4757134242178992,0.1939526032481808,0.4769299722134695,-0.11764112942614702,0.650836044638539,1.1911146774427834,0.6912302155443588,0.7171544223347155,0.592582250478893,0.10671294408216583,0.43994682203415075,157256.05735811294,1295140.6235405423,5.051493925378736e-07,1.2967102135558228e-07 +no_spectral_transition,structural_ood,left_right,flip_x,640,0.31159373267432783,0.4757134242178992,0.3129457731047296,0.479231267252096,0.0013520404304017575,0.19096354211023903,1.1911146774427834,0.6912302155443588,1.1979980661271838,0.6682058583114749,0.17946878091701313,0.42222273504789215,215297.6816428254,1698181.5500575327,5.051493925378736e-07,1.2967102135558228e-07 +no_spectral_transition,structural_ood,left_right,flip_y,640,0.31159373267432783,0.4757134242178992,0.29685707782282406,0.4706221342457604,-0.014736654851503772,0.2491252482050984,1.1911146774427834,0.6912302155443588,1.164557081746898,0.7021404079188445,0.17949872129071298,0.4364244295854938,173589.75882552852,1669030.1932780845,0.0,0.0 +no_spectral_transition,structural_ood,left_right,swap_xy,640,0.31159373267432783,0.4757134242178992,0.20484058807687688,0.4492600159841399,-0.10675314459745096,0.6410168266840799,1.1911146774427834,0.6912302155443588,0.7269538061804269,0.6191037288079857,0.26146046695414865,0.6618994104017073,121972.6276643554,1157881.8661297392,0.0,0.0 +no_spectral_transition,structural_ood,up_down,identity,640,0.19253133183292448,0.454388942165686,0.19253133183292448,0.454388942165686,0.0,0.0,0.7206584809398919,0.6308717245241404,0.7206584809398919,0.6308717245241404,0.9999999952028454,3.3080608608020424e-08,0.0,0.0,0.0,0.0 +no_spectral_transition,structural_ood,up_down,rot90,640,0.19253133183292448,0.454388942165686,0.2950527829761995,0.4717558856018507,0.10252145114327504,0.6312247661936732,0.7206584809398919,0.6308717245241404,1.1859195327357157,0.7103998674257201,0.0951617500886483,0.43914595606404006,2149395.651720482,6009089.973597164,0.0,0.0 +no_spectral_transition,structural_ood,up_down,flip_x,640,0.19253133183292448,0.454388942165686,0.2102985556626436,0.4555838574736564,0.017767223829719115,0.23575997591613637,0.7206584809398919,0.6308717245241404,0.7123531163979351,0.6186947047182674,0.21574318416727087,0.5405982959590553,133951.29701980372,1126582.9731745026,0.0,0.0 +no_spectral_transition,structural_ood,up_down,flip_y,640,0.19253133183292448,0.454388942165686,0.19763366957267864,0.4643253211900862,0.005102337739754148,0.26443356312277166,0.7206584809398919,0.6308717245241404,0.7127238864627161,0.5757488886713646,0.23847393789318533,0.542867489068983,107962.24500306493,1223899.7794815323,4.7311189232956963e-07,1.1569346711279908e-07 +no_spectral_transition,structural_ood,up_down,swap_xy,640,0.19253133183292448,0.454388942165686,0.31597691546738815,0.47349768133751907,0.12344558363446367,0.6261723295951083,0.7206584809398919,0.6308717245241404,1.1830659954153817,0.68708855210834,0.30760941383280616,0.6537036056051472,2232569.273571168,6022105.731704469,0.0,0.0 +no_spectral_transition,structural_ood,turn_cw,identity,640,0.14149016953724244,0.18962090976610863,0.14149016953724244,0.18962090976610863,0.0,0.0,2.9581715806620195,2.053414560536454,2.9581715806620195,2.053414560536454,1.0000000047811213,2.3813951704560475e-08,0.0,0.0,0.0,0.0 +no_spectral_transition,structural_ood,turn_cw,rot90,640,0.14149016953724244,0.18962090976610863,0.03877789861981,0.2298734511523313,-0.10271227091743244,0.25001325734214125,2.9581715806620195,2.053414560536454,1.8553042842698573,1.8477839684139337,0.02260044592261253,0.2646870944518964,9334.407117872592,160765.71181133436,4.3883922140608433e-07,8.368284545369112e-08 +no_spectral_transition,structural_ood,turn_cw,flip_x,640,0.14149016953724244,0.18962090976610863,0.14435867804972244,0.18579931164390318,0.002868508512479999,0.12572438722356302,2.9581715806620195,2.053414560536454,3.0065245536272416,2.0835190644791473,0.15042538149417517,0.3295061513642587,51939.58189334608,443959.65442105033,4.3883922140608433e-07,8.368284545369112e-08 +no_spectral_transition,structural_ood,turn_cw,flip_y,640,0.14149016953724244,0.18962090976610863,0.11015663983903916,0.32298889471601033,-0.031333529698203286,0.3167603236162294,2.9581715806620195,2.053414560536454,2.6288668725639583,1.6702311850889597,0.06417183970189876,0.34226133005878256,51907.202920518626,571513.1664584517,0.0,0.0 +no_spectral_transition,structural_ood,turn_cw,swap_xy,640,0.14149016953724244,0.18962090976610863,0.04158880578233948,0.22292375517894017,-0.09990136375490297,0.240035905199398,2.9581715806620195,2.053414560536454,1.807657049450977,1.7611240028524748,0.221384908901178,0.619371713508572,16788.09381715115,157288.96760848848,0.0,0.0 +no_spectral_transition,structural_ood,impulse_reverse,identity,640,0.18331554904470976,0.23128677580648943,0.18331554904470976,0.23128677580648943,0.0,0.0,2.3223626688399235,1.6652747487186403,2.3223626688399235,1.6652747487186403,1.0000000045295412,2.9818436444338115e-08,0.0,0.0,0.0,0.0 +no_spectral_transition,structural_ood,impulse_reverse,rot90,640,0.18331554904470976,0.23128677580648943,0.08066986489882311,0.2584721893131756,-0.10264568414588666,0.31609796034419363,2.3223626688399235,1.6652747487186403,1.5250600199044129,1.442012733295154,0.12112216150219966,0.33423657383666694,188434.60920509015,1192955.3251682294,5.876024880002717e-07,1.0091740803556674e-07 +no_spectral_transition,structural_ood,impulse_reverse,flip_x,640,0.18331554904470976,0.23128677580648943,0.17205715463738974,0.24001557903153972,-0.011258394407320035,0.15705003272024512,2.3223626688399235,1.6652747487186403,2.313537397750406,1.6970073273010609,0.188044030826832,0.38113069665351557,57637.51012191949,477364.47078330914,5.876024880002717e-07,1.0091740803556674e-07 +no_spectral_transition,structural_ood,impulse_reverse,flip_y,640,0.18331554904470976,0.23128677580648943,0.17339777770059756,0.2275887182283459,-0.009917771344112225,0.155921619375928,2.3223626688399235,1.6652747487186403,2.284669076564751,1.7607530172963535,0.18389180714450035,0.38320499654180756,152934.08584398142,901352.6149246461,0.0,0.0 +no_spectral_transition,structural_ood,impulse_reverse,swap_xy,640,0.18331554904470976,0.23128677580648943,0.09723962390071392,0.2498355259738795,-0.08607592514399584,0.3050021162309816,2.3223626688399235,1.6652747487186403,1.6273605010219399,1.4858254386158054,0.2205291109049093,0.5533479143408648,247680.84409627813,1252200.028369553,0.0,0.0 diff --git a/results/v13_axis_equivariance/forced_collision_summary.csv b/results/v13_axis_equivariance/forced_collision_summary.csv new file mode 100644 index 0000000000000000000000000000000000000000..b7c47c6d8aabcc9ba9c6bb0fee4585c49fefd2f4 --- /dev/null +++ b/results/v13_axis_equivariance/forced_collision_summary.csv @@ -0,0 +1,145 @@ +model,regime,wall,program,n,trajectory_cosine_mean,trajectory_cosine_std,magnitude_ratio_mean,magnitude_ratio_std,boundary_contact_mean,boundary_contact_std +neural_operator,interpolation,left,left_right,160,-0.04834427673267783,0.2682040141050711,1.1420616762072313,0.964421974481774,0.75,0.4343722427630694 +neural_operator,interpolation,left,up_down,160,0.4184704644954763,0.38243706492495577,1.2710277236823457,1.5963467664777422,0.625,0.4856429311786321 +neural_operator,interpolation,left,turn_cw,160,-0.05179979001695756,0.1585078629711932,0.815412285644561,1.0944458685926515,0.5625,0.4976359205623749 +neural_operator,interpolation,left,impulse_reverse,160,-0.036200870273751205,0.17258713640190199,1.7513159065332728,1.7909230650568206,1.0,0.0 +neural_operator,interpolation,right,left_right,160,0.025970479496754705,0.34976223928723954,1.1837641563499346,0.9874280512337156,0.75,0.4343722427630694 +neural_operator,interpolation,right,up_down,160,0.22803693000751082,0.2553261408720017,0.9961582745425404,1.006482576277263,0.375,0.4856429311786321 +neural_operator,interpolation,right,turn_cw,160,0.10002462105767336,0.23310970051187885,1.392411917820573,1.1663211801735718,0.75,0.4343722427630694 +neural_operator,interpolation,right,impulse_reverse,160,0.05740495113350334,0.2306678656547749,1.8900220541894668,1.89985859653666,1.0,0.0 +neural_operator,interpolation,top,left_right,160,0.031442284781951456,0.34309754810311593,0.7919856256688945,1.1959492464995003,0.6875,0.4649677070006681 +neural_operator,interpolation,top,up_down,160,0.22120487818174298,0.24578408106418884,0.7692678845021874,0.7859294860685371,0.75,0.4343722427630694 +neural_operator,interpolation,top,turn_cw,160,0.046749251316487064,0.20708305857268447,1.4118354107253253,2.3171656390155895,0.75,0.4343722427630694 +neural_operator,interpolation,top,impulse_reverse,160,0.04110552841229946,0.21570496937948733,0.9227968613500707,1.1527336905288006,0.6875,0.4649677070006681 +neural_operator,interpolation,bottom,left_right,160,0.04556694092025282,0.3988889524031961,1.5802017026813702,1.077617024799949,0.6875,0.4649677070006681 +neural_operator,interpolation,bottom,up_down,160,0.16510266839904944,0.3125029194743792,0.9999860123964026,1.1472718283169547,0.75,0.4343722427630694 +neural_operator,interpolation,bottom,turn_cw,160,0.029529410286340863,0.24071700924207345,2.2019323375076056,3.285172118034921,0.25,0.4343722427630694 +neural_operator,interpolation,bottom,impulse_reverse,160,0.05955578535795212,0.2600548533777567,1.8538103994680568,1.9121318557906808,0.4375,0.4976359205623749 +neural_operator,extrapolation,left,left_right,160,-0.04220024925889447,0.25348833742200033,1.207899759153952,1.1974145399320222,1.0,0.0 +neural_operator,extrapolation,left,up_down,160,0.42600401273230093,0.38526474950210193,0.8530714369408088,0.8072477695402419,0.75,0.4343722427630694 +neural_operator,extrapolation,left,turn_cw,160,-0.05035743402549997,0.18769793628808665,0.6263567548245191,0.7825269615533585,0.125,0.3317572803749166 +neural_operator,extrapolation,left,impulse_reverse,160,-0.030866402277024464,0.16580369603738424,1.476486624867539,1.5434402795316426,0.1875,0.39153784848013456 +neural_operator,extrapolation,right,left_right,160,0.03833742692368105,0.34188932242883563,1.4010882315109483,1.545447009717692,1.0,0.0 +neural_operator,extrapolation,right,up_down,160,0.22803693035093603,0.2553261417912386,0.7146728741296101,0.6954341120670863,0.875,0.3317572803749166 +neural_operator,extrapolation,right,turn_cw,160,0.1034635583346244,0.23383558556920597,1.3372144421562553,1.0687486647923268,0.1875,0.39153784848013456 +neural_operator,extrapolation,right,impulse_reverse,160,0.06767626710716286,0.23382530402091256,1.7280063229496592,2.006098044342511,0.1875,0.39153784848013456 +neural_operator,extrapolation,top,left_right,160,0.041074886079877616,0.35452281075188297,0.5266801838297397,0.5788798756991796,0.8125,0.39153784848013456 +neural_operator,extrapolation,top,up_down,160,0.20647110154495749,0.22956402241067914,0.7781181387021207,0.9978960877878754,1.0,0.0 +neural_operator,extrapolation,top,turn_cw,160,0.03155137516005198,0.21962118856809473,0.8018090242519975,0.9749322086424411,0.3125,0.4649677070006681 +neural_operator,extrapolation,top,impulse_reverse,160,0.048249022375966885,0.23822781891592745,0.8034330793190747,1.0677218026257702,0.625,0.4856429311786321 +neural_operator,extrapolation,bottom,left_right,160,0.04556694090279052,0.3988889506931959,1.189741168054752,0.7579061754933445,0.75,0.4343722427630694 +neural_operator,extrapolation,bottom,up_down,160,0.1612074441400182,0.29908170244813514,1.0383465652586892,1.3773930304139226,1.0,0.0 +neural_operator,extrapolation,bottom,turn_cw,160,0.02711405393201858,0.24136405877970005,1.17408533138223,1.7177116938463124,1.0,0.0 +neural_operator,extrapolation,bottom,impulse_reverse,160,0.05430369699606672,0.2443037438790275,1.34281502480153,1.3078635768710878,0.3125,0.4649677070006681 +neural_operator,structural_ood,left,left_right,160,-0.04220024982787436,0.25348833751594935,1.857251062511932,1.861645506520317,1.0,0.0 +neural_operator,structural_ood,left,up_down,160,0.42600401295349,0.3852647493395423,1.1199617528822272,1.0215688162466952,1.0,0.0 +neural_operator,structural_ood,left,turn_cw,160,-0.05035743403132074,0.18769793285406428,1.7240322522819043,2.1989826700733763,1.0,0.0 +neural_operator,structural_ood,left,impulse_reverse,160,-0.028775350846990476,0.20127694125484738,2.6930875914637,2.865487799768175,1.0,0.0 +neural_operator,structural_ood,right,left_right,160,0.038337428367231044,0.3418893287266337,1.9962050439091399,2.197463993805198,1.0,0.0 +neural_operator,structural_ood,right,up_down,160,0.22803693072055467,0.25532614195069475,0.9817704193410464,0.9713491671415805,1.0,0.0 +neural_operator,structural_ood,right,turn_cw,160,0.10346355803922051,0.23383558466767246,3.5543633036315443,2.9158679001185783,1.0,0.0 +neural_operator,structural_ood,right,impulse_reverse,160,0.08424384710160666,0.26186673297155694,3.0326333813762174,3.400487013618276,1.0,0.0 +neural_operator,structural_ood,top,left_right,160,0.04107488652225584,0.35452280914838,0.7422473336337134,1.0975273107959052,1.0,0.0 +neural_operator,structural_ood,top,up_down,160,0.20647110160462034,0.22956402191868033,1.168971211160533,1.5287286845212378,1.0,0.0 +neural_operator,structural_ood,top,turn_cw,160,0.02918788402894279,0.20670073992324473,1.4732671722769737,1.7100430728413727,0.125,0.3317572803749166 +neural_operator,structural_ood,top,impulse_reverse,160,0.05301330937218154,0.22340697470108353,1.3008052829885854,1.780120837225414,0.25,0.4343722427630694 +neural_operator,structural_ood,bottom,left_right,160,0.04556694106431678,0.39888895216506065,1.5144970188615843,1.0871862920550046,1.0,0.0 +neural_operator,structural_ood,bottom,up_down,160,0.16120744250620192,0.29908169651258826,1.6547223215457052,2.174821682776973,1.0,0.0 +neural_operator,structural_ood,bottom,turn_cw,160,0.024460171069949865,0.21980137447752418,1.593873387016356,1.798140072493914,0.1875,0.39153784848013456 +neural_operator,structural_ood,bottom,impulse_reverse,160,0.05210202768794261,0.22226758325148288,2.058374688006006,1.8215470850200526,0.3125,0.4649677070006681 +swm_structured_cf_v9,interpolation,left,left_right,160,0.29817070501158016,0.36735333102077977,0.5373852109536529,0.6650938336585194,0.75,0.4343722427630694 +swm_structured_cf_v9,interpolation,left,up_down,160,0.02076830407604575,0.25424579205157083,0.027829573019631383,0.06311207629971213,0.625,0.4856429311786321 +swm_structured_cf_v9,interpolation,left,turn_cw,160,0.07406590284954291,0.17147236634005178,0.4718370326794684,0.7174833366229415,0.5625,0.4976359205623749 +swm_structured_cf_v9,interpolation,left,impulse_reverse,160,0.14652230527208304,0.19501301348152242,0.6132886642590165,0.8960001196471298,1.0,0.0 +swm_structured_cf_v9,interpolation,right,left_right,160,0.3026426640804857,0.3680636560028293,0.5268593437969684,0.6367007003684703,0.75,0.4343722427630694 +swm_structured_cf_v9,interpolation,right,up_down,160,0.0001942874165251851,0.27608690019936155,0.03363356852483992,0.0749752297300972,0.375,0.4856429311786321 +swm_structured_cf_v9,interpolation,right,turn_cw,160,0.0977017563978734,0.1605940405017863,0.4390021194703877,0.7188724188871034,0.75,0.4343722427630694 +swm_structured_cf_v9,interpolation,right,impulse_reverse,160,0.16264352910220622,0.20242423497122736,0.5463738854974508,0.7436703476927138,1.0,0.0 +swm_structured_cf_v9,interpolation,top,left_right,160,0.37052958499407396,0.46343893955170995,0.4351802980992943,0.7007845770316705,0.6875,0.4649677070006681 +swm_structured_cf_v9,interpolation,top,up_down,160,-0.005312116406275891,0.22991657985586939,0.036236869607696474,0.08090172251359684,0.75,0.4343722427630694 +swm_structured_cf_v9,interpolation,top,turn_cw,160,0.1740175570201245,0.3063952853571605,0.5548398888669908,1.0187599388270374,0.75,0.4343722427630694 +swm_structured_cf_v9,interpolation,top,impulse_reverse,160,0.2140361693687737,0.30371326569042256,0.39369013886898757,0.5932789975504573,0.6875,0.4649677070006681 +swm_structured_cf_v9,interpolation,bottom,left_right,160,0.3868037946522236,0.4755931794932355,0.416491461917758,0.539115951589364,0.6875,0.4649677070006681 +swm_structured_cf_v9,interpolation,bottom,up_down,160,0.0032434210821520536,0.23446497295645088,0.02677347376580579,0.06129900571514142,0.75,0.4343722427630694 +swm_structured_cf_v9,interpolation,bottom,turn_cw,160,0.22676672775996848,0.31693471217556957,0.6825562868267298,1.3341945044338905,0.25,0.4343722427630694 +swm_structured_cf_v9,interpolation,bottom,impulse_reverse,160,0.2599191077984869,0.33844292242788954,0.43864970803260805,0.6831244703451808,0.4375,0.4976359205623749 +swm_structured_cf_v9,extrapolation,left,left_right,160,0.30487857818370684,0.3755136957587337,0.6928024380467832,0.8631975445263589,1.0,0.0 +swm_structured_cf_v9,extrapolation,left,up_down,160,0.021538531963403024,0.25626486409636184,0.02068770664839491,0.04494968880021547,0.75,0.4343722427630694 +swm_structured_cf_v9,extrapolation,left,turn_cw,160,0.14339043903746643,0.20658102516930707,0.36782032302580775,0.5405781924901325,0.125,0.3317572803749166 +swm_structured_cf_v9,extrapolation,left,impulse_reverse,160,0.1640495157800615,0.21289828638109765,0.6987925797700882,0.9154663580600955,0.1875,0.39153784848013456 +swm_structured_cf_v9,extrapolation,right,left_right,160,0.3117901818826795,0.37420381147067605,0.7348720911890269,0.8518774372934179,1.0,0.0 +swm_structured_cf_v9,extrapolation,right,up_down,160,0.00019428811501711606,0.2760868997449508,0.026675470709030702,0.059035009162692255,0.875,0.3317572803749166 +swm_structured_cf_v9,extrapolation,right,turn_cw,160,0.15343361790291965,0.1995563311085408,0.37674873201176523,0.6290923916554652,0.1875,0.39153784848013456 +swm_structured_cf_v9,extrapolation,right,impulse_reverse,160,0.18038618308492005,0.2184487720146854,0.7055370632559061,0.8075105821498112,0.1875,0.39153784848013456 +swm_structured_cf_v9,extrapolation,top,left_right,160,0.38405024841194974,0.47290084232908997,0.30002945112064483,0.38242214860673057,0.8125,0.39153784848013456 +swm_structured_cf_v9,extrapolation,top,up_down,160,-0.0038689955545123667,0.22777579347252241,0.03582489633634491,0.07879838896988682,1.0,0.0 +swm_structured_cf_v9,extrapolation,top,turn_cw,160,0.17353925894276473,0.24501830127116306,0.45589520055800675,0.8206801436743641,0.3125,0.4649677070006681 +swm_structured_cf_v9,extrapolation,top,impulse_reverse,160,0.23324336586520075,0.29957083627878084,0.3431184935383499,0.5445078743317534,0.625,0.4856429311786321 +swm_structured_cf_v9,extrapolation,bottom,left_right,160,0.3868037942796946,0.4756521242322091,0.3166326716542244,0.3981715068969163,0.75,0.4343722427630694 +swm_structured_cf_v9,extrapolation,bottom,up_down,160,0.005848171241814271,0.2287875567762524,0.0251324048365551,0.057934794388474216,1.0,0.0 +swm_structured_cf_v9,extrapolation,bottom,turn_cw,160,0.17912427224218846,0.25270215512086275,0.5200170585419983,0.9587690151270158,1.0,0.0 +swm_structured_cf_v9,extrapolation,bottom,impulse_reverse,160,0.22976838890463114,0.2993376290140296,0.3470815885812044,0.5270900319059337,0.3125,0.4649677070006681 +swm_structured_cf_v9,structural_ood,left,left_right,160,0.3048785792896524,0.3754629474251403,1.0340855695307254,1.2879617800371992,1.0,0.0 +swm_structured_cf_v9,structural_ood,left,up_down,160,0.021538531662275394,0.25626486459292697,0.023128031908284198,0.05050018956235513,1.0,0.0 +swm_structured_cf_v9,structural_ood,left,turn_cw,160,0.14373601648549084,0.20681614458953013,0.9746890192851424,1.427900506884098,1.0,0.0 +swm_structured_cf_v9,structural_ood,left,impulse_reverse,160,0.19665141836740077,0.2577885191232127,1.1549140013754369,1.5606703527502448,1.0,0.0 +swm_structured_cf_v9,structural_ood,right,left_right,160,0.3119620716199279,0.37440451641934636,1.0583286203444005,1.232995946551395,1.0,0.0 +swm_structured_cf_v9,structural_ood,right,up_down,160,0.0001942884875461459,0.27608690568213945,0.02831543918683107,0.06235862757113248,1.0,0.0 +swm_structured_cf_v9,structural_ood,right,turn_cw,160,0.15343361510895193,0.19955632826383068,0.998109520226717,1.635582265829277,1.0,0.0 +swm_structured_cf_v9,structural_ood,right,impulse_reverse,160,0.2099838077556342,0.25942351859937934,1.119189616292715,1.3256554686406572,1.0,0.0 +swm_structured_cf_v9,structural_ood,top,left_right,160,0.3837064693099819,0.47238384353635815,0.42482939613983034,0.6383962635559397,1.0,0.0 +swm_structured_cf_v9,structural_ood,top,up_down,160,-0.003569861559662968,0.22774539177569958,0.05193286874198692,0.11570249465289749,1.0,0.0 +swm_structured_cf_v9,structural_ood,top,turn_cw,160,0.13543484934489242,0.19175425807706276,0.6678012142889201,1.1111773167417007,0.125,0.3317572803749166 +swm_structured_cf_v9,structural_ood,top,impulse_reverse,160,0.2090024863369763,0.2688278474139836,0.5101033104583621,0.8100739167397333,0.25,0.4343722427630694 +swm_structured_cf_v9,structural_ood,bottom,left_right,160,0.38680379316210745,0.4755754990399216,0.3970735318958759,0.5031520385175049,1.0,0.0 +swm_structured_cf_v9,structural_ood,bottom,up_down,160,0.006077810790156945,0.22861989947492048,0.03370584209877633,0.07840522487355094,1.0,0.0 +swm_structured_cf_v9,structural_ood,bottom,turn_cw,160,0.1313229748433514,0.19208845776689543,0.6691975269466639,1.0726259287209294,0.1875,0.39153784848013456 +swm_structured_cf_v9,structural_ood,bottom,impulse_reverse,160,0.20312410797923802,0.25892800161636914,0.5066382648423314,0.7081059067802934,0.3125,0.4649677070006681 +no_spectral_transition,interpolation,left,left_right,160,0.12121193459024653,0.32443465350418127,1.2402479228097945,0.7482759150665911,0.75,0.4343722427630694 +no_spectral_transition,interpolation,left,up_down,160,0.13741927224909886,0.42682681803337014,0.8000750377832446,0.7123560772222534,0.625,0.4856429311786321 +no_spectral_transition,interpolation,left,turn_cw,160,0.0865874866140075,0.15255563443434605,1.3141836501657962,1.2018848697138942,0.5625,0.4976359205623749 +no_spectral_transition,interpolation,left,impulse_reverse,160,0.036396690865512935,0.20114129573069614,1.763583607506007,1.5221094043522743,1.0,0.0 +no_spectral_transition,interpolation,right,left_right,160,0.24694830469670706,0.39853453462037386,2.2424981371004833,1.2819683668677295,0.75,0.4343722427630694 +no_spectral_transition,interpolation,right,up_down,160,0.29156018683861473,0.4357993033773892,1.0412073282859637,1.0222294661607145,0.375,0.4856429311786321 +no_spectral_transition,interpolation,right,turn_cw,160,0.13136645882489878,0.23264866927099673,1.7582513109780848,1.5029987240029867,0.75,0.4343722427630694 +no_spectral_transition,interpolation,right,impulse_reverse,160,0.09364840578055009,0.28498121722604464,3.544586358263041,2.4966884019766633,1.0,0.0 +no_spectral_transition,interpolation,top,left_right,160,0.20689757182262838,0.42379365727237767,1.2175587019417435,1.203655242742071,0.6875,0.4649677070006681 +no_spectral_transition,interpolation,top,up_down,160,0.05247103262809105,0.2804898357473199,0.728233573876787,0.6892137271455905,0.75,0.4343722427630694 +no_spectral_transition,interpolation,top,turn_cw,160,0.03308990450605052,0.21901005294385562,1.5274055023677646,2.0401390340393384,0.75,0.4343722427630694 +no_spectral_transition,interpolation,top,impulse_reverse,160,0.06995080570632126,0.2509300663456708,1.225161736877635,1.3837870405619985,0.6875,0.4649677070006681 +no_spectral_transition,interpolation,bottom,left_right,160,0.28343247482553124,0.42205869361832754,1.5503893604484014,0.9251078108252797,0.6875,0.4649677070006681 +no_spectral_transition,interpolation,bottom,up_down,160,0.1441725330693381,0.4735221457310022,1.7099730377551168,1.3318724192698967,0.75,0.4343722427630694 +no_spectral_transition,interpolation,bottom,turn_cw,160,0.10147273153997957,0.2408630500122048,2.8726909461431207,3.790543348649922,0.25,0.4343722427630694 +no_spectral_transition,interpolation,bottom,impulse_reverse,160,0.15533534841033542,0.2553704896857451,1.729333192575723,1.607696423037979,0.4375,0.4976359205623749 +no_spectral_transition,extrapolation,left,left_right,160,0.1170516962360125,0.30164280478582595,1.3656773093040102,1.0610068620772761,1.0,0.0 +no_spectral_transition,extrapolation,left,up_down,160,0.1394313260127092,0.4293262747294617,0.567762093670899,0.444173178772035,0.75,0.4343722427630694 +no_spectral_transition,extrapolation,left,turn_cw,160,0.06694169644906652,0.16111146962294462,0.9172187279909849,0.7805971508003772,0.125,0.3317572803749166 +no_spectral_transition,extrapolation,left,impulse_reverse,160,0.0378908004538971,0.18470846854131837,1.5850311305490323,1.4847572880579893,0.1875,0.39153784848013456 +no_spectral_transition,extrapolation,right,left_right,160,0.2554934861895163,0.3855077438093355,2.5541121397065583,1.798034054216021,1.0,0.0 +no_spectral_transition,extrapolation,right,up_down,160,0.2915601842221804,0.4357993028136978,0.7477608124981998,0.6874713495557958,0.875,0.3317572803749166 +no_spectral_transition,extrapolation,right,turn_cw,160,0.15030639662873,0.25548793639334516,1.6637332800775766,1.2698519237917427,0.1875,0.39153784848013456 +no_spectral_transition,extrapolation,right,impulse_reverse,160,0.11241358611732721,0.2834074404923384,3.2692977405051353,2.404311479254912,0.1875,0.39153784848013456 +no_spectral_transition,extrapolation,top,left_right,160,0.21828345075482503,0.435361066426877,0.8485918610589579,0.6254881432271306,0.8125,0.39153784848013456 +no_spectral_transition,extrapolation,top,up_down,160,0.05234760856837965,0.27168839446139087,0.8601154591800878,0.9385459185227945,1.0,0.0 +no_spectral_transition,extrapolation,top,turn_cw,160,0.06277456124371383,0.23944795840440633,0.9233944829553365,1.0659344509792459,0.3125,0.4649677070006681 +no_spectral_transition,extrapolation,top,impulse_reverse,160,0.07663542368973139,0.26713222214353666,0.9611136948456988,1.0333468401635828,0.625,0.4856429311786321 +no_spectral_transition,extrapolation,bottom,left_right,160,0.28343247406883165,0.4220586904412394,1.1728063545529852,0.6765881808656102,0.75,0.4343722427630694 +no_spectral_transition,extrapolation,bottom,up_down,160,0.1570599788116912,0.45970063852473697,1.926059103710577,1.6584842984467638,1.0,0.0 +no_spectral_transition,extrapolation,bottom,turn_cw,160,0.1163464741781354,0.23135628300152247,1.6061807910911738,1.8881132921428145,1.0,0.0 +no_spectral_transition,extrapolation,bottom,impulse_reverse,160,0.15221397376153617,0.24861808223340712,1.319160165823996,1.1166749020912554,0.3125,0.4649677070006681 +no_spectral_transition,structural_ood,left,left_right,160,0.11705169586348348,0.3016428027089719,1.99059567975346,1.561656452025006,1.0,0.0 +no_spectral_transition,structural_ood,left,up_down,160,0.1394313242577482,0.42932627231569903,0.781453803688055,0.6264469244816595,1.0,0.0 +no_spectral_transition,structural_ood,left,turn_cw,160,0.06694169644906652,0.16111146764565698,2.568034949526191,2.253116124004194,1.0,0.0 +no_spectral_transition,structural_ood,left,impulse_reverse,160,0.04243167567547061,0.186411188946807,2.787784045515582,2.5865294887012147,1.0,0.0 +no_spectral_transition,structural_ood,right,left_right,160,0.25549348883505446,0.38550774352093853,4.031921227590646,2.8318604962053024,1.0,0.0 +no_spectral_transition,structural_ood,right,up_down,160,0.2915601859916933,0.43579930490595903,1.0305640755566856,0.9765190646290075,1.0,0.0 +no_spectral_transition,structural_ood,right,turn_cw,160,0.15030639450997113,0.25548793737073816,4.622385439276695,3.412292370059742,1.0,0.0 +no_spectral_transition,structural_ood,right,impulse_reverse,160,0.1390182335759164,0.3065930387123917,5.962940607254859,4.369761230027924,1.0,0.0 +no_spectral_transition,structural_ood,top,left_right,160,0.21828345095273108,0.43536106635224986,1.1490419143345207,1.2455350053791,1.0,0.0 +no_spectral_transition,structural_ood,top,up_down,160,0.052347607916453855,0.2716883958515421,1.310696345200995,1.4040025658026816,1.0,0.0 +no_spectral_transition,structural_ood,top,turn_cw,160,0.07361851291789208,0.22272878852451294,1.7293492898344993,1.7384644158629208,0.125,0.3317572803749166 +no_spectral_transition,structural_ood,top,impulse_reverse,160,0.0851548783335602,0.2516039215563464,1.5556787023786456,2.046172040289813,0.25,0.4343722427630694 +no_spectral_transition,structural_ood,bottom,left_right,160,0.2834324724157341,0.4220586905712699,1.5318354909048353,0.9957416822777394,1.0,0.0 +no_spectral_transition,structural_ood,bottom,up_down,160,0.15705998102203012,0.4597006440576765,3.1154982850886883,2.680764726086525,1.0,0.0 +no_spectral_transition,structural_ood,bottom,turn_cw,160,0.09800965990871191,0.2001180853179077,2.414409533329308,2.1050953687004506,0.1875,0.39153784848013456 +no_spectral_transition,structural_ood,bottom,impulse_reverse,160,0.15968945630593226,0.23171420317288724,2.3051460828632115,1.581391198733608,0.3125,0.4649677070006681 diff --git a/results/v14_external_control/metrics_summary.csv b/results/v14_external_control/metrics_summary.csv new file mode 100644 index 0000000000000000000000000000000000000000..4a6f03a9f5abc5629eb54e66a4c059aaae9ae6d4 --- /dev/null +++ b/results/v14_external_control/metrics_summary.csv @@ -0,0 +1,10 @@ +env_id,model,params,n_seeds,rollout_mse_h5_mean,rollout_mse_h5_std,rollout_mse_h10_mean,rollout_mse_h10_std,rollout_mse_h20_mean,rollout_mse_h20_std,rollout_mse_h30_mean,rollout_mse_h30_std +CartPole-v1,neural_operator,424064,10,0.0006748462212271989,0.0001299804495714757,0.0009282927960157395,0.00015468570686310684,0.0013759312801994382,0.0001851490907062441,0.0016309754457324744,0.00019973190116870962 +CartPole-v1,swm_structured_cf_v9,420946,10,0.010556331946281717,0.01022593373688085,0.02556397336302325,0.024932835180183126,0.03417481736978516,0.042039789635205826,0.036373767792247234,0.04786999185573027 +CartPole-v1,no_spectral_transition,467072,10,0.0008691661234479398,0.0001870540576379721,0.001057595224119723,0.00021681315527813396,0.0013877597521059215,0.00018724521060319808,0.0015665917191654444,0.00014474241557245268 +MountainCar-v0,neural_operator,411288,10,0.0006089691363740712,0.0001444924119433363,0.0007360691553913057,0.00019591595872844454,0.0008317771018482745,0.00020318070655984214,0.0011551685980521142,0.00022559938956871696 +MountainCar-v0,swm_structured_cf_v9,410547,10,0.005631116358563304,0.0035773137476026434,0.01736451021861285,0.022286048839871224,0.024847829318605363,0.032388456357427355,0.028382448735646902,0.03567293497688429 +MountainCar-v0,no_spectral_transition,454296,10,0.003406702494248748,0.0005882765404263405,0.0033385258400812747,0.0005567247714356033,0.0031223797239363195,0.0006078483712004343,0.003346434934064746,0.0006434907843673941 +Acrobot-v1,neural_operator,436888,10,0.0010058330022729932,0.000135020745183597,0.0011516758240759373,0.00015800559984369415,0.001432927418500185,0.00014941575448583583,0.001738599198870361,0.00012091154193147414 +Acrobot-v1,swm_structured_cf_v9,436147,10,0.002116703090723604,0.0005501751713738825,0.008117175451479853,0.017320904838101312,0.01730848508886993,0.03601083522152292,0.02241422866936773,0.04220507663690017 +Acrobot-v1,no_spectral_transition,479896,10,0.0018341316725127399,0.00036157888794762346,0.0019610013579949735,0.00035523011936842736,0.002256414480507374,0.0003490138223016178,0.002563271764665842,0.00033099081454081547 diff --git a/results/v15_external_diagnostics/diagnostics_by_seed.csv b/results/v15_external_diagnostics/diagnostics_by_seed.csv new file mode 100644 index 0000000000000000000000000000000000000000..d9fa5709350766642c15ed65b9fdb8b6fcd84c65 --- /dev/null +++ b/results/v15_external_diagnostics/diagnostics_by_seed.csv @@ -0,0 +1,91 @@ +env_id,fg_mse_h10,fg_mse_h20,fg_mse_h30,fg_mse_h5,latent_cos_h10,latent_cos_h20,latent_cos_h30,latent_cos_h5,model,motion_mse_h10,motion_mse_h20,motion_mse_h30,motion_mse_h5,mse_h10,mse_h20,mse_h30,mse_h5,psnr_h10,psnr_h20,psnr_h30,psnr_h5,seed,ssim_h10,ssim_h20,ssim_h30,ssim_h5,static_mse_h10,static_mse_h20,static_mse_h30,static_mse_h5,token_acc_h10,token_acc_h20,token_acc_h30,token_acc_h5 +CartPole-v1,0.0674522650858762,0.0949607477268728,0.11117655242560431,0.0542440037625056,0.8597052982077003,0.8147908663377166,0.7991477097384632,0.891299257054925,neural_operator,0.00035985808662886143,0.0005021747104549945,0.00034212702618674484,0.000135766246947,0.001177776439817535,0.0016110742874388961,0.0017135325106210075,0.0009654033842707577,30.392745673656464,29.38970598578453,29.09636229276657,31.24637943506241,0,0.8883007764816284,0.8484677076339722,0.8386073112487793,0.906626284122467,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.6406250135041773,0.5343750107567757,0.5375000124331564,0.7031250135041773 +CartPole-v1,0.09630811442821141,0.13867218875020626,0.13181017873493062,0.09740044841350937,0.8176538459956646,0.6820666177081876,0.6923031851183623,0.8891848623752594,neural_operator,0.0003628969631215284,0.0005505642485417184,0.0003775029882042702,0.00013124336552650107,0.0012572821681260393,0.0024411259562384657,0.002330022910655316,0.0010530406989346375,31.127813786268234,27.757788330316544,28.329900979995728,31.23221045732498,1,0.889893114566803,0.7850590944290161,0.7998709678649902,0.9011819362640381,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.6187500117812306,0.5031250091269612,0.46250001015141606,0.721875011222437 +CartPole-v1,0.11755379474652727,0.13117089425031736,0.13712726267203834,0.10717578688888807,0.7298400122672319,0.658828136511147,0.6828874051570892,0.8367759576067328,neural_operator,0.00037996989576916107,0.0004771514420127687,0.0003256285527726277,0.00013875414293984534,0.0018933690726044006,0.0024411171837073198,0.002464319067712495,0.001106211939259083,28.954040735960007,27.52545517683029,27.659236818552017,31.050823003053665,2,0.8245065808296204,0.777686357498169,0.7743759155273438,0.8925542235374451,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.4875000095926225,0.45937500754371285,0.4218750074505806,0.6437500133179128 +CartPole-v1,0.06319059016290446,0.11584121696751026,0.14256779814286347,0.053752005072622566,0.8273205258883536,0.7426219270564616,0.7542957352707162,0.8956868555396795,neural_operator,0.0003326918157466885,0.0004848188062023251,0.00032636957929632615,0.00015934432662856324,0.0009014550186066117,0.0019828287988730153,0.0022452215994235303,0.0006956472935826241,32.63624569773674,29.127073109149933,28.836885303258896,33.14465492963791,3,0.918258786201477,0.8235655426979065,0.8036434054374695,0.9325838088989258,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.5937500118743628,0.4781250103842467,0.5250000099185854,0.753125011222437 +CartPole-v1,0.13034365029859402,0.10530103699466054,0.15253792393559706,0.07722924860968305,0.7324047419242561,0.7044431329704821,0.6922935832990333,0.8545218724757433,neural_operator,0.0003495950429197592,0.000489423994247673,0.0003128299328549744,0.00014560941507113512,0.0015969112029097232,0.002173516856146307,0.002538360949984053,0.0008387892582959466,29.80777844786644,28.632681518793106,28.08094412088394,31.934889882802963,4,0.8515440225601196,0.8051053285598755,0.7732072472572327,0.9186224937438965,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.459375009406358,0.4531250090803951,0.465625009033829,0.6218750132247806 +CartPole-v1,0.08378339531250845,0.12472855035230168,0.11434730986729846,0.07372959092034748,0.7780297663994133,0.7111516920849681,0.7140123737044632,0.8813817715272307,neural_operator,0.00036734645184921533,0.0004934551669126819,0.0003601362185419177,0.00013325109478845576,0.0017300114388945076,0.002321506401813167,0.0021364682843341143,0.0009715400507843697,29.88439640402794,27.860379844903946,28.736766457557678,31.5354180932045,5,0.8408775329589844,0.7878109216690063,0.8048341274261475,0.9062654972076416,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.5343750093597919,0.46250000898726285,0.47812500852160156,0.690625011920929 +CartPole-v1,0.09144725428875278,0.10463747884114127,0.12956243295957393,0.06668265497853554,0.7701622070744634,0.6933216713368893,0.7243663400877267,0.8676063884049654,neural_operator,0.00036171795714068367,0.0004924131470911974,0.00031752624561853793,0.00015953126428769338,0.0013688311046280432,0.00200814918866854,0.0022727564073647955,0.0008661975217592044,30.211401849985123,28.99502059817314,28.470468282699585,32.08665853738785,6,0.8729586005210876,0.8187597990036011,0.7977542877197266,0.9170386791229248,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.5187500123865902,0.4593750089406967,0.49687501043081284,0.7093750122003257 +CartPole-v1,0.06394518260299265,0.11685976079206739,0.12226273974489743,0.049164321504235886,0.7681561498902738,0.6782210785895586,0.6679085069335997,0.8302799109369516,neural_operator,0.0003529906138801664,0.0005294432941767013,0.0003668909787108987,0.0001340074744113906,0.001060090912233136,0.0020064539850181973,0.001976199474029272,0.0007398250750156876,31.438330203294754,28.293305277824402,28.68656986951828,32.56676757335663,7,0.9017666578292847,0.8153375387191772,0.8211480379104614,0.9295548796653748,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.646875012665987,0.5562500096857548,0.5281250094994903,0.7468750132247806 +CartPole-v1,0.11225260332867038,0.14105397953244392,0.14815602873909484,0.09122088049298327,0.8748081792145967,0.8083639573305845,0.801831585355103,0.9226347031071782,neural_operator,0.0003591123440447319,0.00047796916030407033,0.00031625551110181505,0.00011488916920399106,0.0016348556223420019,0.002601935864731786,0.002725487840052665,0.0010133037667401368,29.709793835878372,27.280856430530548,27.66497179865837,31.123945385217667,8,0.8472105264663696,0.7655595541000366,0.7585245966911316,0.9026415944099426,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.5687500108033419,0.42812500800937414,0.4187500092666596,0.6781250119674951 +CartPole-v1,0.07230006392125254,0.10477301414812246,0.12282154885659224,0.05690642201678209,0.76815155451186,0.6211686108727008,0.6302497442811728,0.8510179473087192,neural_operator,0.00042226830834124485,0.00053960079056381,0.00038755001190793337,0.00016064425101802726,0.0011869635979451232,0.0017204786050228904,0.001851426905432163,0.0008339430626165267,30.95762261748314,29.13851636648178,29.099135488271713,32.53572955727577,9,0.8883943557739258,0.8368001580238342,0.8247705698013306,0.9202356338500977,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.6500000110827386,0.5375000117346644,0.546875009778887,0.7687500114552677 +CartPole-v1,0.16380409993143985,0.16656650212826207,0.17378788159703618,0.13967921364383074,0.05161292683624197,0.017174288412206806,0.006658268524915911,0.44068002194399014,swm_structured_cf_v9,0.006788959218837931,0.004583281978923992,0.006212871767921513,0.012818998967759399,0.17811744270329655,0.19472916702534349,0.19714052755080047,0.06743655596892495,9.22909302264452,8.491378508508205,8.130717925727367,17.369599923491478,0,0.35946235060691833,0.30284690856933594,0.27944415807724,0.7106376886367798,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.1343750029336661,0.11875000246800482,0.10625000204890966,0.5593750106636435 +CartPole-v1,0.14612628795566707,0.13418546097818762,0.14612354626956403,0.08949771527113626,0.07169922010507435,0.07748709231964312,0.08698526781518012,0.3960901403333992,swm_structured_cf_v9,0.014888634158367609,0.015227933683689798,0.014956890612296547,0.007887487077013589,0.07227799985412275,0.07394515252963174,0.06688061191539418,0.018329114784592093,14.167988389730453,13.445249177515507,14.622675359249115,21.552939370274544,1,0.6189663410186768,0.6168190836906433,0.6187826991081238,0.848700225353241,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.26250000577419996,0.2750000054948032,0.2437500054948032,0.5781250114087015 +CartPole-v1,0.1637569413657909,0.14950262809725245,0.16397509449565306,0.08201437061590511,0.3629252786922734,0.3934796422836371,0.40365586429834366,0.49486241617705673,swm_structured_cf_v9,0.0003295159398636338,0.0004980189104779598,0.00033080406751140323,0.0003393858047004983,0.0021237070603739028,0.0020589247219504614,0.0022016272056646358,0.0011315841514942804,29.080061346292496,28.10463759303093,29.40211993455887,31.334371864795685,2,0.7986798286437988,0.8054369688034058,0.7930155992507935,0.9080536365509033,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.4843750100117177,0.5093750129453838,0.5218750128988177,0.6687500143889338 +CartPole-v1,0.10395421241992153,0.09959096148668323,0.11568765503761824,0.09973640691623586,0.1342517037410289,0.1926725661905948,0.2140829462732654,0.2622748589492403,swm_structured_cf_v9,0.010557981401689176,0.004872109434359118,0.005314041530709801,0.017046005230781702,0.02856667535525048,0.01689019082550658,0.013495141740349936,0.057653641953947954,17.39175796508789,19.99076110124588,20.31271317601204,15.343709245324135,3,0.7831428647041321,0.7897410988807678,0.7959927320480347,0.7594170570373535,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.3500000073108822,0.4125000089406967,0.47187501029111445,0.41562501015141606 +CartPole-v1,0.135110509131664,0.12951999946358228,0.14215566740676877,0.11138788672360533,0.182018041057745,0.15305640298174694,0.22705992143892217,0.42843486194033176,swm_structured_cf_v9,0.0012919478496069559,0.0011016379976069857,0.0011441241675997915,0.0025634524112057022,0.013032562907028478,0.006543793932650033,0.009877923555450252,0.01045882953360433,22.524331286549568,24.635897368192673,23.002659648656845,24.921255454421043,4,0.7844136953353882,0.8021917343139648,0.775775671005249,0.8633349537849426,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.4531250100117177,0.4812500113621354,0.462500010849908,0.6812500122468919 +CartPole-v1,0.12021797087982122,0.12830244193901308,0.14628478025363734,0.11898202849307005,0.13271191250532866,0.10229029943002388,0.08848255075281486,0.3333836929232348,swm_structured_cf_v9,0.003061950821255266,0.002109965124296842,0.001623504228248862,0.004387977913012264,0.0306647974457519,0.018609869674037327,0.020708990488657264,0.028273338649341895,17.755780816078186,20.02522937953472,20.09884576499462,19.74635261297226,5,0.716295599937439,0.7332662343978882,0.712439239025116,0.7966720461845398,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.3093750069383532,0.3656250087078661,0.3656250087078661,0.47500001010484993 +CartPole-v1,0.1142532092781039,0.10326309670927003,0.12161426108968953,0.07687877618823791,0.07121713808737695,0.05470232244988438,0.03243695976561867,0.350445541087538,swm_structured_cf_v9,0.009502649933438079,0.008225014150639254,0.007100802147817831,0.008640471594191013,0.04292601616180036,0.03170341376971919,0.03030998330723378,0.035144182982548955,15.1546019166708,16.23946762084961,16.48997274041176,16.42148844897747,6,0.706387996673584,0.728721022605896,0.704007625579834,0.8151201009750366,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.23437500419095159,0.29375000740401447,0.25625000474974513,0.48125001112930477 +CartPole-v1,0.16876706591028778,0.14792364202367025,0.152066596074745,0.10124011549476108,0.22373947415326256,0.3277819227077998,0.39825574567657895,0.44366513875866076,swm_structured_cf_v9,0.0007037500521676066,0.0004893383002517737,0.0003208428586409795,0.0008020023862833625,0.003966131610241064,0.0021056020753178473,0.0021675907659925997,0.0037781458788685995,26.46272225677967,28.310557901859283,28.777271956205368,27.729461014270782,7,0.808586061000824,0.809073805809021,0.8000062704086304,0.9113079905509949,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.4625000103842467,0.5031250107567757,0.5000000132713467,0.7156250125262886 +CartPole-v1,0.15622979912734536,0.14863025556314824,0.17099768761613632,0.10219992366083375,0.39710539672523737,0.37459286535158753,0.41158257122151554,0.5599603988812305,swm_structured_cf_v9,0.0004235666091207957,0.00048444427965255166,0.00029816107319136087,0.0007543334421704628,0.0028067652013987754,0.0021874445814802357,0.0023339406045650435,0.0024046663348258335,28.307830333709717,29.165710985660553,29.087365299463272,30.643801867961884,8,0.8022536635398865,0.8103255033493042,0.7768194675445557,0.9017024636268616,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.4843750107102096,0.44375000963918865,0.4500000106636435,0.7218750123865902 +CartPole-v1,0.13247880180423977,0.13398030142707285,0.1609976201725658,0.12934716620338804,0.05005710727709811,-0.03438225430727471,-0.026874149567447603,0.2723362408578396,swm_structured_cf_v9,0.005755580704949637,0.005808224974726883,0.005915430434029201,0.00604633823921219,0.051593445766229706,0.055512741076881866,0.05395349381183223,0.039180561610010045,14.756405115127563,13.996580749750137,13.859837099909782,17.8846553042531,9,0.6835668087005615,0.6393935680389404,0.614193856716156,0.7757323980331421,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.30312500637955964,0.24687500554136932,0.20625000400468707,0.46562500880099833 +CartPole-v1,0.06358454347004505,0.10047404621218448,0.13700948383876366,0.06186098350406155,0.7972875947598368,0.7115575382485986,0.6835132008418441,0.830407383851707,no_spectral_transition,0.0003351520281356102,0.00045265813369432806,0.00035657261158661413,0.00015362739078028653,0.001136771152005167,0.002043532463176234,0.0022340630766848335,0.0008835619116780435,30.466314285993576,28.18617820739746,27.728447794914246,31.38781887292862,0,0.9086055159568787,0.8352168798446655,0.8168257474899292,0.9234441518783569,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.6500000148080289,0.5593750120606273,0.5062500110361725,0.6625000147614628 +CartPole-v1,0.09740811991468945,0.10220451637724182,0.13534261262975633,0.0714790683089177,0.7859288458712399,0.7691874536685646,0.7656857566908002,0.8719409592449665,no_spectral_transition,0.0003476270300737383,0.0005010827610547608,0.0003295039042612302,0.00012067988206965197,0.001371612375123732,0.0015186732821348414,0.0018837007169167919,0.0010483182688858506,29.946810245513916,29.513822585344315,28.497104436159134,31.18504324555397,1,0.8710683584213257,0.8570936918258667,0.8229820728302002,0.9011902809143066,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.5625000121071935,0.5093750115483999,0.5125000104308128,0.7093750105705112 +CartPole-v1,0.11645730878058203,0.12749331190389057,0.12928967348170772,0.11461398603114503,0.7442506835795939,0.7294525098986924,0.71119747729972,0.8708743210881948,no_spectral_transition,0.0003548712567660317,0.0004870086766182169,0.00034809612726327543,0.00015410594593490146,0.0019040461097574735,0.00224542555406515,0.0021170388838527288,0.0014742988846592198,28.47034850716591,27.570573955774307,28.0864180624485,29.6142438352108,2,0.8354935646057129,0.8062950372695923,0.8146731853485107,0.8679385185241699,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.5062500108033419,0.48125001159496605,0.45625000866129994,0.6312500105705112 +CartPole-v1,0.10264395838271412,0.12817118526618287,0.13396929822465609,0.08583383417686719,0.7946473024785519,0.7251691399142146,0.7216061232611537,0.8851743470877409,no_spectral_transition,0.00032129863829410965,0.00047015336282640874,0.0003290034967555755,0.0001201532752386214,0.001274385557280766,0.0021622655340252095,0.0021440202956455323,0.0008545943225044539,29.83014354109764,27.59274861216545,27.93011635541916,31.758469969034195,3,0.8849765062332153,0.8092033863067627,0.8105349540710449,0.9207761287689209,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.581250011222437,0.4312500092200935,0.450000009033829,0.7125000120140612 +CartPole-v1,0.0988588607876224,0.10117522143673341,0.13468139819906355,0.08876347332325452,0.8418305572122335,0.7517950246110559,0.7718713977374136,0.8823347222059965,no_spectral_transition,0.00037150547398567824,0.0005180702714575602,0.00033818125633899854,0.0001533052456608175,0.0016130790072566015,0.0016679755124187068,0.002068512974346959,0.001281279730960705,29.54308369755745,29.656146705150604,28.167254775762558,30.74443805217743,4,0.8502745628356934,0.8475055694580078,0.809014618396759,0.8795123100280762,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.6125000130850822,0.534375011920929,0.5375000115018338,0.7312500125262886 +CartPole-v1,0.0686728465007036,0.10144063354528043,0.12434915253197687,0.06651727006544661,0.839124973397702,0.7682657728437334,0.7651484203524888,0.8871903782710433,no_spectral_transition,0.00036756844668772715,0.0005328812432914276,0.00039697303330044065,0.00018129424561763585,0.0010006352667915053,0.0018812299053934112,0.001931461262756784,0.0008144905052631657,30.75781312584877,28.335616141557693,28.32579466700554,31.64784362912178,5,0.9174851179122925,0.8433554172515869,0.8404136300086975,0.9306192398071289,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.6531250118277967,0.5500000137835741,0.5375000112690032,0.7656250116415322 +CartPole-v1,0.08653712284046833,0.1214136411617801,0.12578127197326694,0.07131153739396723,0.6567112338962033,0.4736614395515062,0.49191270396113396,0.7978982347995043,no_spectral_transition,0.00044275366118284865,0.0006559134144978884,0.00043487215005910684,0.00019437207567563064,0.0015170072008459101,0.002137902400136227,0.0022112420963367185,0.0011537268812844559,29.29103511571884,27.386663615703583,27.283523201942444,30.25370341539383,6,0.8732129335403442,0.8192694187164307,0.8109448552131653,0.9007405042648315,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.5187500105239451,0.4312500099185854,0.48437500977888703,0.6343750129453838 +CartPole-v1,0.11075027585282271,0.11319746793878949,0.13361513744712283,0.10166072691509953,0.686659978760872,0.6324182823300362,0.6194141220767051,0.8341380162164569,no_spectral_transition,0.00039103044504429363,0.0005570809080808203,0.00035608827991495673,0.0001805096340063983,0.0017684575107068667,0.0017948942729617556,0.0020336100496933796,0.0014527288749377476,28.961806654930115,28.94836312532425,27.926716715097427,29.78454977273941,7,0.842322826385498,0.8395810723304749,0.8189092874526978,0.8676248788833618,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.5437500136904418,0.48125001112930477,0.49375001271255314,0.6656250148080289 +CartPole-v1,0.08753031651243148,0.10282294672651915,0.12626377926790155,0.0630875019113546,0.7952527021989226,0.7868877956643701,0.7683243663050234,0.8833291251212358,no_spectral_transition,0.000336009815672611,0.0004907631791368772,0.0003333880832823155,0.00014803341015223737,0.001263127035826983,0.001635685726569136,0.001962624007546765,0.0008488811714641997,30.31924369931221,29.587965548038483,28.54525598883629,31.92003059387207,8,0.8814529180526733,0.8486361503601074,0.8180992007255554,0.9196664690971375,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.5656250100582838,0.5500000128522515,0.5468750111758709,0.6562500135041773 +CartPole-v1,0.09410704981644358,0.11163770508028392,0.13002663676570592,0.0717320595309161,0.7465890415478498,0.7015098449774086,0.6917745999526232,0.8574717724695802,no_spectral_transition,0.0004012028395763423,0.0005861003416818278,0.00042220828795791476,0.00017876294123198022,0.001258485011248922,0.0017240897652754938,0.0018828712410368098,0.0008668899914709982,30.133511036634445,28.640166461467743,28.690049320459366,32.26860648393631,9,0.8858938217163086,0.8441126346588135,0.8286219835281372,0.9196852445602417,0.0016746692103026817,0.002196556895228241,0.0029960669583708466,0.000707036524921989,0.5718750129453838,0.5125000101979822,0.5125000106636435,0.7375000107567757 +MountainCar-v0,0.06211637792708302,0.11331565142197086,0.2262225616530742,0.05876175338838863,0.8465308072045445,0.6888810095842928,0.5524784782901406,0.8604475394822657,neural_operator,0.00015339935302189495,0.00013901496348689424,0.0001200335031337485,0.00011442797181057673,0.0006959743268382113,0.0007764882076912727,0.001934681410691752,0.000710518655978376,40.43570217490196,42.44736564159393,34.019988149404526,37.63578110933304,0,0.9037383794784546,0.8912741541862488,0.730587363243103,0.8998051285743713,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.9270833376795053,0.6927083479240537,0.6354166809469461,0.9218750046566129 +MountainCar-v0,0.11815264940560155,0.10734988908620835,0.21756922601070983,0.11446826526844234,0.6950669498182833,0.638542699161917,0.5850643582816701,0.7194270696491003,neural_operator,0.00014167777554056904,0.00013952397773364655,0.00012003163966040065,0.00013032796906142785,0.0009692637263611914,0.0007411199448341677,0.0018746537900256044,0.0011664001856246387,37.777233332395554,39.501320868730545,32.65009528398514,34.52731019258499,1,0.8629803657531738,0.8960446119308472,0.7382447123527527,0.8333752155303955,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.7760416781529784,0.6927083474583924,0.6927083474583924,0.8333333428017795 +MountainCar-v0,0.10271105499850819,0.10650629023598324,0.1785792125533323,0.05850160906811652,0.713950761128217,0.6671930872835219,0.6324893244309351,0.724488583393395,neural_operator,0.000152353716027398,0.00013399738758701485,0.0001260013594519549,0.00011340083133493106,0.0008390569139535842,0.000746929883128189,0.0016174686068666233,0.0008718056702363697,34.48294943571091,35.260413467884064,30.998391151428223,34.911183923482895,2,0.8816008567810059,0.8934528827667236,0.7711974382400513,0.881514310836792,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.7447916786186397,0.7239583465270698,0.7135416809469461,0.7656250107102096 +MountainCar-v0,0.09780584995083808,0.25470441739375943,0.28423281758148633,0.0716649080429761,0.7536890283226967,0.42011998262023553,0.3764267729711719,0.7703977772034705,neural_operator,0.00014649447849190977,0.00014233782513900506,0.00012537665342238213,0.00011724487773295822,0.000635799956057781,0.0016780709844113062,0.0024824230976605577,0.0006486579164857176,37.759904980659485,31.279716670513153,29.485017627477646,36.01523914933205,3,0.9125156402587891,0.7613598108291626,0.6550157070159912,0.9108765125274658,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.7656250116415322,0.6406250107102096,0.6562500107102096,0.7812500116415322 +MountainCar-v0,0.07706989056321994,0.18353254784823175,0.24220301500509983,0.051811090199635146,0.6830177119700238,0.32405868388013914,0.31137464570929296,0.7192710433737375,neural_operator,0.00018009833785370688,0.00016605186089968527,0.00014633498420657531,0.0001270795085304144,0.0006770859899063453,0.0014665680142229576,0.0024040006278482906,0.0006602854395794111,34.4350603222847,29.86688831448555,27.624655842781067,34.31387883424759,4,0.9038494229316711,0.7883108258247375,0.6605695486068726,0.9083352088928223,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.8177083428017795,0.6718750125728548,0.6718750116415322,0.7864583446644247 +MountainCar-v0,0.075016394337581,0.1970027742900129,0.2899321003060322,0.050166647244626716,0.8595290146768093,0.6457904042908922,0.45120905071962625,0.85935039492324,neural_operator,0.0001510338020400681,0.0001600775610048899,0.0001387635833229317,9.944012087448328e-05,0.0005910842733101163,0.0012475235662918749,0.0024258252910840383,0.0005085672543430064,33.4575178027153,30.982101947069168,27.59287029504776,35.642560839653015,5,0.9153941869735718,0.8219451904296875,0.6567333936691284,0.9285438060760498,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.901041672565043,0.6614583469927311,0.598958348389715,0.9218750046566129 +MountainCar-v0,0.1882599999098602,0.12347196221057288,0.21721727797739732,0.07157402934080892,0.6506563294678926,0.6378121434827335,0.566001995379338,0.784430175088346,neural_operator,0.0001659178604632161,0.00015658568570176,0.00013214426517249578,0.0001440974271562645,0.001532622654934812,0.0013764274806362664,0.0019028976342951864,0.0010129401078700084,29.90615352988243,30.1637604534626,28.973810464143753,33.64986053109169,6,0.7774956226348877,0.798080563545227,0.722260594367981,0.8628636598587036,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.7864583441987634,0.7812500079162419,0.7604166767559946,0.8854166716337204 +MountainCar-v0,0.12271707070993898,0.2840416813922424,0.32258416328092526,0.06111205608718251,0.7814150773920119,0.5530810841009952,0.36393053582287394,0.8522255914285779,neural_operator,0.00014613752397641538,0.00013992152894885812,0.00012311257630415184,8.606726096171968e-05,0.0009103089470379189,0.0020289943091427176,0.0028306709795629104,0.0007337839227616882,35.955162525177,30.77146276831627,28.19557699561119,35.317486226558685,7,0.8707099556922913,0.7086875438690186,0.6008630990982056,0.8961949944496155,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.8489583409391344,0.66666668234393,0.65104168234393,0.9270833376795053 +MountainCar-v0,0.06544569332536337,0.11507773893093148,0.22364530016693746,0.09803008219363107,0.7184638446196914,0.6921108562964946,0.6792615595040843,0.7164717810228467,neural_operator,0.0001960771369500236,0.0001674826027497911,0.00013426057370846678,0.00013826135180128318,0.0009553514929621088,0.0008026468566413314,0.001954409696423909,0.0010750238069476836,32.01233226060867,36.61072665452957,31.266837269067764,31.795885533094406,8,0.8674340844154358,0.8891153335571289,0.7336372137069702,0.8510178327560425,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.765625009778887,0.7031250139698386,0.7291666781529784,0.770833345130086 +MountainCar-v0,0.0600409512146598,0.16895785936795793,0.2395926778976334,0.06945605165185498,0.8487653490155935,0.672300957958214,0.530674291861942,0.8111897511407733,neural_operator,0.0001507930671590385,0.0001597734942717366,0.00013645121724925602,0.00011339347649697018,0.0005156003073523152,0.0010596265656488413,0.002103359198798671,0.0006811539881255158,36.54939341545105,32.36520391702652,28.952253967523575,34.780096024274826,9,0.927790105342865,0.8503828048706055,0.7081248760223389,0.9062401056289673,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.8854166730307043,0.7343750121071935,0.7239583446644247,0.8593750083819032 +MountainCar-v0,0.15506889195285112,0.18482307872636738,0.3096142138901996,0.1557389801919271,0.531812731642276,0.5132199339568615,0.5134187964722514,0.6374134737998247,swm_structured_cf_v9,0.0001422084547388191,0.0001383857553000807,0.00011978248891489351,9.791193233033013e-05,0.004025124486361165,0.003733245415787678,0.005125135739945108,0.004590326770994579,24.16323271393776,24.491844415664673,23.24149590730667,23.602359890937805,0,0.685505747795105,0.7085684537887573,0.6028103232383728,0.6445493698120117,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.7656250135041773,0.6770833432674408,0.6197916781529784,0.8177083441987634 +MountainCar-v0,0.1291524289681547,0.27198874147507013,0.35649022085999604,0.1214892850998126,0.3648478960385546,0.21941004329710267,0.19172285322565585,0.5012753083137795,swm_structured_cf_v9,0.0017948798151564693,0.0021490969863355996,0.002510415803680041,0.0004883162480897596,0.012186532352643553,0.016825309321575332,0.014697992504807189,0.004752934044518042,22.668560564517975,21.66485472023487,21.42802242934704,23.868407994508743,1,0.7355434894561768,0.6997644901275635,0.6011759042739868,0.7041164636611938,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.828125009778887,0.619791679084301,0.5833333446644247,0.8437500093132257 +MountainCar-v0,0.14213687251321971,0.1698736097314395,0.24535160040250048,0.09616079168884539,0.08817731082672253,0.0707171299145557,0.046962811233242974,0.4989668717607856,swm_structured_cf_v9,0.04552569489950997,0.03752389967208458,0.03868491012173081,0.02529751563673699,0.09596414990301128,0.08259499256382696,0.10591076276978129,0.051243527141195955,14.711442172527313,16.096198245882988,13.679176487028599,17.226671740412712,2,0.5588879585266113,0.603071391582489,0.5040404796600342,0.6705824732780457,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.4166666748933494,0.4635416711680591,0.34895833814516664,0.7500000074505806 +MountainCar-v0,0.14639470631664153,0.18714090145658702,0.2556682003196329,0.0861376873754125,0.2713231649249792,0.056140768399927765,0.08791645395103842,0.7292271954938769,swm_structured_cf_v9,0.014112155883511335,0.013188973248134062,0.016265833347347325,0.0010667564969910615,0.04873880388913676,0.05989104804029921,0.045336573934037006,0.0058270505505788606,16.523573704063892,14.505766108632088,16.035069212317467,23.100495547056198,3,0.6784051656723022,0.6803495287895203,0.6163690090179443,0.7139432430267334,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.5677083428017795,0.33333334140479565,0.4114583400078118,0.8541666753590107 +MountainCar-v0,0.1246938744734507,0.15929961332585663,0.2173202979611233,0.09800021275441395,-0.010871336446143687,-0.050688160699792206,-0.04063402877363842,0.5830189257394522,swm_structured_cf_v9,0.022523540292695543,0.020229222680427483,0.018466554192968943,0.010270882860154984,0.12834116794692818,0.13371088344138116,0.11782434588531032,0.022839750065031694,10.042842723429203,9.580100983381271,10.390295818448067,20.18653417378664,4,0.48353463411331177,0.5200401544570923,0.4765159487724304,0.6946039199829102,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.1562500037252903,0.15104167070239782,0.2447916711680591,0.7864583428017795 +MountainCar-v0,0.1525907361137797,0.19572434830479324,0.29532125138211995,0.09561564846444526,0.5442223963327706,0.6165758934803307,0.6885941014625132,0.6995629137381911,swm_structured_cf_v9,0.0014162162367714615,0.0008745026193652805,0.0004203585783062458,0.0004617473127819238,0.008011607842490776,0.004750177817186341,0.00496719264992862,0.004801053830306046,22.76120735704899,23.93256115913391,23.478026777505875,23.558902263641357,5,0.7238668203353882,0.7361704111099243,0.6581141948699951,0.7114229202270508,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.8177083428017795,0.7708333423361182,0.7708333428017795,0.8958333395421505 +MountainCar-v0,0.1458266818372067,0.180035661556758,0.2922640305478126,0.09268839899527848,0.7772687477990985,0.7633737595751882,0.7455445192754269,0.8004232523962855,swm_structured_cf_v9,0.0001473945203738225,0.00014231081695625925,0.00012757044377487148,9.087184126377323e-05,0.00329215591045795,0.0030726842232979834,0.004315878013585461,0.003679264340462396,24.969680935144424,25.360082268714905,24.044923841953278,24.525070756673813,6,0.7322408556938171,0.7457242012023926,0.6459065675735474,0.7077203989028931,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.8072916753590107,0.7916666744276881,0.7708333418704569,0.8802083404734731 +MountainCar-v0,0.19268707968555077,0.3322502459122916,0.33110308049072046,0.1256267642186799,0.45384780724998564,0.09503090145881288,-0.004193103784928098,0.6837319480255246,swm_structured_cf_v9,0.0027808454001938543,0.010204269870815352,0.00965601295564511,0.0010336709974581026,0.0225219984567957,0.042495700530707836,0.06066311071481323,0.008508478975272737,20.61826951801777,16.662906803190708,14.56446373462677,22.687433063983917,7,0.6653368473052979,0.6194418668746948,0.5491899251937866,0.6966501474380493,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.7604166734963655,0.536458345130086,0.4114583428017795,0.8697916739620268 +MountainCar-v0,0.1315033202408813,0.18534725668723695,0.303533273317953,0.12449101766105741,0.7962804418057203,0.7989061959087849,0.8022652138024569,0.8177534649148583,swm_structured_cf_v9,0.00014805809350970645,0.00013918089119879795,0.00012128253401377798,9.095162994654515e-05,0.0026711860664363485,0.0025195998277922627,0.0038595774694840657,0.003202152791345725,26.105768531560898,26.435592502355576,24.741415858268738,25.311781227588654,8,0.756859302520752,0.7702822089195251,0.6510385274887085,0.7144365310668945,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.8697916730307043,0.7083333441987634,0.7500000102445483,0.890625006519258 +MountainCar-v0,0.11101250108913518,0.20172826977614022,0.30990929310792126,0.08344681328162551,0.6503603449091315,0.6839883923530579,0.6716247941367328,0.6495842998847365,swm_structured_cf_v9,0.00014481802729338078,0.00014850181091674575,0.00012553234169065952,0.00013477488206081745,0.0033225270017283037,0.0032978544804791454,0.004577315910864854,0.0037363899500633124,24.95391708612442,25.074667751789093,23.822584569454193,24.43583032488823,9,0.7306619882583618,0.7247704863548279,0.6174429059028625,0.7061501741409302,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.8541666734963655,0.6718750144354999,0.6302083465270698,0.8958333395421505 +MountainCar-v0,0.1369895972457016,0.11752325785892026,0.19355236916453578,0.13799281453066214,0.6738923043012619,0.6177640822716057,0.6185940806171857,0.7014781478792429,no_spectral_transition,0.0001503774901791266,0.00014255513238375928,0.00012408319915035992,8.813269128649104e-05,0.003496447930956492,0.0033278366427111905,0.004048351303936215,0.0037378270862973295,24.651942759752274,24.8629792034626,24.09053236246109,24.40084534883499,0,0.6823287010192871,0.6965698003768921,0.6311985850334167,0.66213059425354,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.770833345130086,0.7135416781529784,0.723958345130086,0.8333333428017795 +MountainCar-v0,0.1398456265568626,0.18480624993583916,0.3061247879070379,0.14189290801346033,0.7021657081786543,0.46795191790442914,0.4930320157436654,0.7742464621551335,no_spectral_transition,0.00014478777377557872,0.00013994277124147747,0.00011970856030050613,9.39606353879685e-05,0.0027241102634434355,0.0025197761351591907,0.0038866258637426654,0.0033155957753479015,26.058047205209732,26.430545538663864,24.720432430505753,25.21652165055275,1,0.7523332238197327,0.7711811661720276,0.6497693061828613,0.7023308277130127,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.7239583469927311,0.619791679084301,0.6406250116415322,0.7343750139698386 +MountainCar-v0,0.11758566555073457,0.2252829448416378,0.3239195852671344,0.11846819429410971,0.524214451899752,0.4219958451576531,0.45909539656713605,0.6145478419493884,no_spectral_transition,0.0001505672849799744,0.0001442628153587311,0.00012009865939344166,8.662265255243717e-05,0.0033384874113835394,0.00360034556069877,0.004793051459273556,0.0037529645014728885,24.91756135225296,24.627961963415146,23.560440182685852,24.46148332953453,2,0.7300287485122681,0.7145068049430847,0.6220694780349731,0.6941713690757751,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.6510416832752526,0.5468750107102096,0.5885416772216558,0.6822916842065752 +MountainCar-v0,0.16195362531288993,0.1730160744045861,0.2843186260724906,0.17643960011082527,0.7711842004209757,0.6754885315895081,0.659906460903585,0.7968283179216087,no_spectral_transition,0.00014541176615340268,0.0001421820523160866,0.00012139959732893058,8.846084135655241e-05,0.003337986876431387,0.003013351877598325,0.004253663057170343,0.0038779590140620712,25.042705535888672,25.452360957860947,24.1128026843071,24.449439108371735,3,0.7155367136001587,0.7436766624450684,0.6407389640808105,0.6716935634613037,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.8281250083819032,0.7447916776873171,0.7447916786186397,0.890625006519258 +MountainCar-v0,0.09538217913359404,0.20657645055325702,0.33085148088866845,0.11756976987635426,0.7280590045265853,0.5872229920350946,0.5403345902450383,0.6976276081986725,no_spectral_transition,0.00014932698184322302,0.00014305399676084463,0.0001242231244249492,8.901366044655745e-05,0.002047152738668956,0.002128426914168813,0.0035473290736263152,0.0026880336736212485,27.329970121383667,27.327221155166626,25.28622567653656,26.159092515707016,4,0.8028435111045837,0.791899561882019,0.6560699939727783,0.7445783615112305,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.812500009778887,0.6562500158324838,0.6093750149011612,0.8177083437331021 +MountainCar-v0,0.09981008290196769,0.19658721261657774,0.3074305541782678,0.08861765214930983,0.709924525115639,0.47880466977949254,0.468367049616063,0.7705739112570882,no_spectral_transition,0.0001441005314505861,0.000144277769904555,0.000122706357235991,8.935503466767081e-05,0.0029284861557243858,0.0031477129778068047,0.004461532658751821,0.0033148307338706218,25.473564982414246,25.291645646095276,23.961590439081192,24.997758120298386,5,0.7541120052337646,0.7357915639877319,0.6273767948150635,0.7228343486785889,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.7500000139698386,0.5729166800156236,0.5781250125728548,0.7916666786186397 +MountainCar-v0,0.1363724457694815,0.30618857503650077,0.35825307275092655,0.12370679116932592,0.8134634178131819,0.6733536715619266,0.6903241318650544,0.8388857394456863,no_spectral_transition,0.00015453071297066856,0.0001401566738714194,0.00011963114851559764,9.209852517333417e-05,0.0037163071647228207,0.004542910672171274,0.0054151682707015425,0.003640502898633713,24.388811647892,23.553114145994186,22.944053828716278,24.538230925798416,6,0.6847181916236877,0.6178779602050781,0.5501845479011536,0.6916390061378479,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.7708333446644247,0.6354166800156236,0.6927083460614085,0.7760416786186397 +MountainCar-v0,0.1355844522637284,0.10632510547644936,0.1855417755310782,0.10296256353194622,0.5683440316352062,0.7346966757904738,0.7000906278844923,0.5809930920950137,no_spectral_transition,0.00014944729276174934,0.00013894836179766112,0.00012010186713362625,0.00016619117233507286,0.0028717919740302023,0.0025568434575689025,0.003378005672857398,0.002877666149288416,25.655432254076004,26.085457503795624,25.061180859804153,25.608891546726227,7,0.7260591983795166,0.7582089900970459,0.682382345199585,0.7236517071723938,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.6458333423361182,0.7239583460614085,0.7500000121071935,0.7343750111758709 +MountainCar-v0,0.13337900144870218,0.12728002700168872,0.2373658295677714,0.07621890462314695,0.6009900572244078,0.496763281757012,0.39505610219202936,0.7288795739877969,no_spectral_transition,0.00014342951426559586,0.00013973764661961985,0.00012131525963633705,9.656558478401966e-05,0.0025187895389535697,0.0021444772191898664,0.00326403930921515,0.0027227972641412634,26.4749396443367,27.081364899873734,25.46147882938385,26.031149685382843,8,0.749943196773529,0.7859843969345093,0.6763328313827515,0.7411518692970276,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.8333333400078118,0.7031250135041773,0.6979166809469461,0.8906250060535967 +MountainCar-v0,0.16298211432876997,0.1849458350916393,0.30708472377591534,0.15140947327017784,0.6911659620236605,0.6424725046090316,0.5504813630832359,0.7835308131761849,no_spectral_transition,0.00014721050666972012,0.0001422920797317373,0.0001291081069693112,0.00010598035868227798,0.004005548638815526,0.003744989098777296,0.005111044491059147,0.004473997454624623,24.171808421611786,24.47128999233246,23.244548618793488,23.71028795838356,9,0.6869512796401978,0.7070931792259216,0.6028414964675903,0.654226541519165,0.0010133257156434983,0.002459771895925611,0.003329529117545693,0.0003009121570227147,0.8229166748933494,0.7760416748933494,0.729166675824672,0.885416672565043 +Acrobot-v1,0.05079566594940843,0.06363430281635374,0.0741171993286116,0.03626148845796706,0.8390284786000848,0.7529458091594279,0.6984941705595702,0.8655873565003276,neural_operator,0.0008117778015162003,0.0014534338604335062,0.0019478318532151206,0.0006525165903212837,0.0012849583144998178,0.0020158144438937597,0.0024093685424304567,0.0009581439971952932,29.38016039133072,27.799302369356155,26.93428459763527,30.426055669784546,0,0.8381615281105042,0.744533360004425,0.6910024881362915,0.8812721967697144,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8147321529686451,0.7522321548312902,0.7276785857975483,0.848214294295758 +Acrobot-v1,0.05041901842923835,0.06248221980422386,0.07274626140133478,0.037132377619855106,0.8406938053667545,0.7557947691529989,0.7319095104467124,0.891718483529985,neural_operator,0.0007677261564253968,0.001434033241821453,0.001915290790122981,0.0006368548692847753,0.0012778190975950565,0.0019367939730727812,0.0023325235961237922,0.0009584552753949538,29.398036271333694,27.86530366539955,27.02557909488678,30.39351189136505,1,0.8324847221374512,0.7411535978317261,0.6852099895477295,0.8764713406562805,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8191964388824999,0.7500000107102096,0.7343750139698386,0.8861607201397419 +Acrobot-v1,0.05217022429860663,0.06671323872433277,0.07911595203040633,0.0385341978399083,0.9098081197589636,0.8501038015820086,0.8067877264693379,0.9312287922948599,neural_operator,0.0007543733268562391,0.0014334349350519915,0.001913928618932914,0.0006303087704395693,0.0013302685147209559,0.0020965897901987773,0.0025529275599183165,0.0009996349654102232,29.266371369361877,27.6434668302536,26.761422246694565,30.306270718574524,2,0.8373575210571289,0.7411214113235474,0.6827311515808105,0.8804097175598145,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8147321529686451,0.7745535830035806,0.7343750149011612,0.8861607210710645 +Acrobot-v1,0.04782653604343068,0.062055867892922834,0.07322196252061985,0.035961057837994304,0.8763013081625104,0.7939372644759715,0.7428082048427314,0.9023551996797323,neural_operator,0.0007898142475255554,0.00144179967492164,0.0019256444743973589,0.0006411939838812941,0.0012242842058185488,0.0019370550335224834,0.002354823516725446,0.0009447265301787411,29.536372989416122,27.839400470256805,26.997047424316406,30.431511998176575,3,0.8383009433746338,0.7406988143920898,0.683211624622345,0.8777387142181396,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8125000102445483,0.7589285825379193,0.7254464430734515,0.8750000069849193 +Acrobot-v1,0.05261403557960875,0.0631594673395739,0.07228918262990192,0.0411642023245804,0.9200818175449967,0.8956991378217936,0.8748230691999197,0.9262853842228651,neural_operator,0.0007608444077504828,0.0014358115979575814,0.0019116273393819938,0.0006457378207329612,0.0013770815667157876,0.002061710813904938,0.0024192059545384836,0.0011445149048086023,29.15246695280075,27.644532293081284,27.016678422689438,29.820661932229996,4,0.8220305442810059,0.7266778945922852,0.6791929602622986,0.8551219701766968,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8236607243306935,0.7566964421421289,0.7075893022119999,0.8281250093132257 +Acrobot-v1,0.05458257235295605,0.06844119192464859,0.07928194548003376,0.038661632011098845,0.8866889514029026,0.8503959132358432,0.8458410920575261,0.9041631892323494,neural_operator,0.0008197316471978411,0.0014569024751835968,0.0019253798753311457,0.0006747114412917199,0.0013880389324185671,0.0021928467795078177,0.0025889127664413536,0.0010162697044506785,29.159527838230133,27.42398238182068,26.615544766187668,30.236035525798798,5,0.831429123878479,0.7319869995117188,0.6780260801315308,0.8782332539558411,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8080357247963548,0.7299107252620161,0.7232143003493547,0.8459821515716612 +Acrobot-v1,0.05218937482277397,0.06260186915460508,0.07186592915968504,0.04645179558428936,0.8201038260012865,0.7578372959978878,0.7170072696171701,0.8532803971320391,neural_operator,0.0007953653604317878,0.0014437459401506203,0.001977631959320547,0.0006799249848654654,0.0014105123027547961,0.001994584382373432,0.0023834834346416756,0.0012533395029095118,29.05394458770752,27.786148369312286,27.03702947497368,29.47148710489273,6,0.819562554359436,0.7386393547058105,0.6856632828712463,0.8418689370155334,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.7879464407451451,0.7700892980210483,0.7142857289873064,0.8214285811409354 +Acrobot-v1,0.04213807928317692,0.054542205260077026,0.06312304403400049,0.03860083606559783,0.9053720897063613,0.8817929020151496,0.8349127611145377,0.8991377176716924,neural_operator,0.0007876850769434895,0.0014561628590854525,0.0019923409392959,0.0006762416967802665,0.0013294987993504037,0.001903656197100645,0.002256453192785557,0.0012235694684932241,29.116365671157837,27.772673964500427,27.223289787769318,29.460078477859497,7,0.8306009769439697,0.753146767616272,0.7077896595001221,0.8488073348999023,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8504464374855161,0.8013392961584032,0.7544642998836935,0.8727678647264838 +Acrobot-v1,0.05074779577262234,0.06481463688396616,0.07636262923188042,0.0379858544038143,0.7937555122189224,0.6831282137427479,0.6544622157234699,0.8433179305866361,neural_operator,0.0007945955165951091,0.0014462385099704989,0.0019081777145828482,0.0006538788243517502,0.0012961798920514411,0.002037569106050796,0.0024479040112055372,0.0009751267061801627,29.289916962385178,27.728659957647324,26.812910676002502,30.301974684000015,8,0.833305835723877,0.737095832824707,0.6810996532440186,0.8779064416885376,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8035714398138225,0.7544642966240644,0.7299107285216451,0.8504464370198548 +Acrobot-v1,0.05075925244454993,0.05911683919111965,0.07050585022807354,0.043918623974605,0.8846351662650704,0.848662312142551,0.8345451764762402,0.8843986615538597,neural_operator,0.0008044711539696436,0.001443160850840286,0.0018780062080168136,0.0006559198238846875,0.001779612575774081,0.0023159509528341005,0.002577153133643151,0.0016617936198599637,27.900464206933975,26.93283173441887,26.525270253419876,28.005058139562607,9,0.7722384929656982,0.7003794312477112,0.6606652140617371,0.7919394969940186,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.7834821538999677,0.7142857275903225,0.7209821571595967,0.7745535834692419 +Acrobot-v1,0.06540654701530002,0.08263003371394007,0.09092646589851938,0.04250730366584321,0.6677293949760497,0.627536307554692,0.6347337900660932,0.7272485075518489,swm_structured_cf_v9,0.0008082255112640269,0.001470486396101478,0.0019063829622965045,0.0006666835435282792,0.0026887811782216886,0.003624333623520215,0.00398691128793871,0.002241465970655554,25.915704369544983,24.76089408993721,24.31503826379776,26.579837203025818,0,0.7441952228546143,0.6540946960449219,0.6111472845077515,0.7792878150939941,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.7566964412108064,0.7254464430734515,0.685267873108387,0.7879464393481612 +Acrobot-v1,0.05005968321347609,0.05014562717406079,0.05459932670055423,0.048061690758913755,0.7146810926496983,0.4774771969532594,0.438220476557035,0.8232072591781616,swm_structured_cf_v9,0.0008554005453333957,0.00457309831836028,0.007398442835210517,0.0006629915084488402,0.00276925292564556,0.017331971910607535,0.013868277877918445,0.0019108528267679503,25.810580521821976,20.85102167725563,22.02569715678692,27.28758391737938,1,0.6757839918136597,0.583639919757843,0.6080228090286255,0.7595535516738892,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.6785714449360967,0.5982142998836935,0.6183035867288709,0.7544642980210483 +Acrobot-v1,0.056294224807061255,0.06274626264348626,0.06587227123964112,0.05661522669834085,0.701847655698657,0.7309520840644836,0.7270818669348955,0.7872442007064819,swm_structured_cf_v9,0.00079682277765869,0.0014397582426681765,0.0018898358567867035,0.0006687623405241538,0.003080908460106002,0.0032252751079795416,0.0034041867475025356,0.0032112161788973026,25.15839245915413,25.042949676513672,24.812180638313293,24.984192848205566,2,0.6149730682373047,0.6077772378921509,0.5773234367370605,0.6143432855606079,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.6696428707800806,0.660714297555387,0.6517857294529676,0.6785714440047741 +Acrobot-v1,0.053270447126124054,0.05592259217519313,0.06272612381144427,0.044736589858075604,0.7814123407006264,0.6958149030688219,0.649183125642594,0.8463756563141942,swm_structured_cf_v9,0.0008468052085390809,0.0017549649207921902,0.0023251316647474596,0.000683294478562857,0.002275635741170845,0.004835701318370411,0.007897479308667243,0.0018739977622317383,26.588929444551468,25.355207845568657,24.6778507232666,27.365518182516098,3,0.7196948528289795,0.6586586236953735,0.6231859922409058,0.771980881690979,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.7254464407451451,0.7031250125728548,0.6450892996508628,0.7834821557626128 +Acrobot-v1,0.051695485977688804,0.06393970958015416,0.0708692887565121,0.05402347183553502,0.6259680439834483,0.370245487705688,0.2088035355845932,0.7792080994695425,swm_structured_cf_v9,0.003922323160509222,0.014240790297662898,0.019257929884133773,0.0009797847557138084,0.014455290182013414,0.021669665597073617,0.04565594583800703,0.0028868387034890475,24.07224477827549,21.61051892489195,18.752524569630623,26.158959478139877,4,0.6098225116729736,0.5743426084518433,0.5090982913970947,0.6818963885307312,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.6919642977882177,0.6160714428406209,0.4732142968568951,0.7566964407451451 +Acrobot-v1,0.04732698014413472,0.06906603094830643,0.0741130776659702,0.03880904332982027,0.5919061198364943,0.14863845481886528,0.07353244154364802,0.7805618774145842,swm_structured_cf_v9,0.0021866520086177843,0.014201881399912963,0.015954211307871446,0.0006701929751784519,0.010104978138770093,0.07924201561945665,0.06980136770289391,0.0017065219590222114,24.55735182762146,13.92097044736147,13.806905664503574,27.870007127523422,5,0.7127335667610168,0.43570753931999207,0.4571380615234375,0.8110532760620117,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.6875000128056854,0.2857142926659435,0.27008929336443543,0.8191964384168386 +Acrobot-v1,0.07994163886178285,0.08646033056356828,0.09011128068959806,0.04453036974882707,0.08925631068996154,-0.03712756733875722,-0.025427553337067366,0.6638601389713585,swm_structured_cf_v9,0.010653251927578822,0.00960414910059626,0.008816477042728366,0.003628249408393458,0.17993019137793453,0.18532222311478108,0.17778925888706,0.008691471546626417,7.916644632816315,7.559077680110931,7.701203934848309,23.65661182999611,6,0.05813480168581009,0.10105158388614655,0.10050068795681,0.660292387008667,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.08705357438884676,0.049107144586741924,0.04910714481957257,0.7500000125728548 +Acrobot-v1,0.05655000382103026,0.06962958048097789,0.07683463371358812,0.0480698458122788,0.7540259417146444,0.7128861909732223,0.6985242748633027,0.7934017879888415,swm_structured_cf_v9,0.0007917158886243669,0.00144571676491978,0.0018966410982983462,0.0006526667519040075,0.002638305433720234,0.0032689605814084643,0.003546863084920915,0.002342125022551045,25.88924351334572,25.056313812732697,24.724947214126587,26.379974991083145,7,0.7098084688186646,0.6387431025505066,0.60696941614151,0.7562750577926636,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.7723214407451451,0.7075892970897257,0.7031250139698386,0.8593750079162419 +Acrobot-v1,0.05160604068078101,0.06547163226059638,0.07403323863400146,0.04142125425642007,0.7989400895312428,0.8022825717926025,0.8039588015526533,0.8429099321365356,swm_structured_cf_v9,0.000806750949209345,0.001450358820193287,0.0018906660023816357,0.0006468713811216276,0.001968761835087207,0.0025613336238166085,0.002861118999135215,0.001689026126769022,27.198847115039825,26.251284807920456,25.762763261795044,27.808663427829742,8,0.7544062733650208,0.6813938617706299,0.6439508199691772,0.7957784533500671,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.7142857266589999,0.6852678689174354,0.6785714421421289,0.7433035830035806 +Acrobot-v1,0.08165863386238925,0.09926016098324908,0.10901153009035625,0.05861761810956523,0.6399432858452201,0.6008646888658404,0.5889084781520069,0.694062540307641,swm_structured_cf_v9,0.0008118570948454362,0.0014545552385243354,0.0019130000505924727,0.0006906489029461227,0.0030837553149467567,0.0040400645993941,0.004500546376220882,0.002482456973666558,25.381864696741104,24.310869693756104,23.825568705797195,26.198105186223984,9,0.7317610383033752,0.645045280456543,0.600976824760437,0.782677173614502,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.6718750158324838,0.6897321571595967,0.687500013038516,0.754464297555387 +Acrobot-v1,0.05661671549023595,0.07230685789727431,0.08240727485099342,0.04032297022786224,0.7189148536417633,0.5864745650324039,0.5406193117960356,0.8222415712662041,no_spectral_transition,0.0008402084548606581,0.0014879923389798932,0.0019598056836116484,0.0006877038736661234,0.0019080510619460256,0.002760215049420367,0.0031587779903929913,0.0015165962095124996,27.49418932199478,26.099345803260803,25.52498060464859,28.340108186006546,0,0.7934786081314087,0.6997053623199463,0.6529990434646606,0.8386664390563965,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.805803582072258,0.7343750111758709,0.7299107280559838,0.8750000074505806 +Acrobot-v1,0.05371663726691622,0.06583724694792181,0.07643820584053174,0.04028617373160159,0.8549254965037107,0.7774070055456832,0.7570772962644696,0.889216317795217,no_spectral_transition,0.0008193736206294489,0.0014612421382480534,0.0019136150041845212,0.0006689262368411164,0.0025516039804642787,0.0032753274044807767,0.0036378225067892345,0.0021923639469605405,26.123090744018555,25.16215941309929,24.69893965125084,26.67375761270523,1,0.7373956441879272,0.6602421998977661,0.6175901889801025,0.7774441838264465,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.7901785834692419,0.7343750135041773,0.7343750144354999,0.8325892952270806 +Acrobot-v1,0.053738823960884474,0.06594208445676486,0.07482855986745562,0.037270294546033256,0.8281510239467025,0.7441698373295367,0.7153684655204415,0.8626179397106171,no_spectral_transition,0.0007801880557281038,0.0014467426644841908,0.0019075007154469859,0.0006478992964247254,0.001986857883821358,0.0027268690082564717,0.0030464348674286157,0.001601980737177655,27.26412805914879,26.06301188468933,25.578705608844757,28.03009870648384,2,0.777828574180603,0.692150354385376,0.6490039825439453,0.8234192132949829,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8013392961584032,0.7500000116415322,0.7254464430734515,0.8482142938300967 +Acrobot-v1,0.046702092615305446,0.05912778312631417,0.06959449448913801,0.036536486601107754,0.8878812426701188,0.8510487452149391,0.8362239608541131,0.9012982053682208,no_spectral_transition,0.0007811640334693948,0.00143857153625504,0.0019140302898961181,0.0006578193162312118,0.0017694254129310139,0.00240015321651299,0.002785379627312068,0.001515189354904578,27.68167218565941,26.585874766111374,25.957092434167862,28.247671335935593,3,0.7908641695976257,0.7109684944152832,0.6641853451728821,0.8238300681114197,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8147321534343064,0.754464297555387,0.7321428717114031,0.8593750079162419 +Acrobot-v1,0.051822318535414524,0.06528392686595907,0.07488651938911062,0.0383533183921827,0.8361060041934252,0.7376226303167641,0.7011502601671964,0.8645780337974429,no_spectral_transition,0.00079497886883928,0.0014401101686871698,0.0019505475573282638,0.0006517992893861901,0.001807145501516061,0.002556597006332595,0.002921145427535521,0.001481089022490778,27.68812781572342,26.41212511062622,25.86335900425911,28.40757128596306,4,0.7973756790161133,0.7100645303726196,0.6651673316955566,0.8372557163238525,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8348214384168386,0.7745535830035806,0.7209821580909193,0.8593750079162419 +Acrobot-v1,0.04753908189741196,0.057844143877446186,0.06832496634160634,0.035493849645718,0.8775139590725303,0.8001832836307585,0.7475267457775772,0.8997095599770546,no_spectral_transition,0.0007987539926261888,0.0014619221931297943,0.001946014149268649,0.0006603310126820361,0.00241547897712735,0.0030853805801598355,0.003446173255724716,0.0021230378060863586,26.30019423365593,25.39662367105484,24.92578035593033,26.784388452768326,5,0.7458608150482178,0.6710206866264343,0.6265740394592285,0.7793423533439636,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8325892956927419,0.7522321538999677,0.7142857294529676,0.8861607210710645 +Acrobot-v1,0.05180232190468814,0.06295859233796364,0.07342593363136984,0.038572661724174395,0.8167989924550056,0.7442900068126619,0.708546370442491,0.8717523319646716,no_spectral_transition,0.0008332647684028416,0.001484078543853684,0.0019862646764750025,0.0007098022530271919,0.0019027154976356542,0.002591092014881724,0.00297457430860959,0.0015652845258955495,27.50176230072975,26.36614438891411,25.794028967618942,28.21731388568878,6,0.7922267913818359,0.7113450765609741,0.6658337116241455,0.8313980102539062,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8125000102445483,0.7544642984867096,0.7142857294529676,0.8861607210710645 +Acrobot-v1,0.05262295852298848,0.06692253918299684,0.07519678123935591,0.039954817919351626,0.8514480488374829,0.7884748699143529,0.768201743485406,0.8747733747586608,no_spectral_transition,0.0007992322556447107,0.001452492969292507,0.001912767224240497,0.0006539456477696604,0.002507685694581596,0.003216827821233892,0.0035450554842100246,0.002151292459529941,26.150469839572906,25.226057320833206,24.80021271109581,26.730232506990433,7,0.7349998950958252,0.6580475568771362,0.6218150854110718,0.7786524891853333,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8080357243306935,0.7433035834692419,0.7142857289873064,0.8727678647264838 +Acrobot-v1,0.046944184723543,0.05747354900086066,0.06550694268662483,0.036795203246583696,0.8483659299090505,0.7412498572375625,0.7041507624089718,0.8795990357175469,no_spectral_transition,0.000836115048400643,0.0014763258768653031,0.0019796057148369073,0.0006860608450978134,0.002494266263965983,0.0031553651206195354,0.003438705532971653,0.0022439019430748885,26.14346894621849,25.236115366220474,24.91711238026619,26.54572242498398,8,0.7328164577484131,0.6578186750411987,0.6233960390090942,0.7630531787872314,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.7968750111758709,0.7276785830035806,0.7187500144354999,0.8459821515716612 +Acrobot-v1,0.0525889951386489,0.06282371943962062,0.07029894033621531,0.04074052370560821,0.8037602407857776,0.6912813404924236,0.6443007127381861,0.8458583210594952,no_spectral_transition,0.0008154582709494207,0.0014702221496918355,0.001894857553708107,0.0006686974394227718,0.002584784851933364,0.003218830601326772,0.003486478892227751,0.002265293389427825,26.030766904354095,25.18989422917366,24.84380355477333,26.51942351460457,9,0.7204926609992981,0.6480824947357178,0.6151719689369202,0.758975088596344,0.0029196965099345107,0.0029245632103993557,0.0037530396159013435,0.00275761558786769,0.8102678675204515,0.7477678693830967,0.7321428721770644,0.8459821515716612 diff --git a/results/v15_external_diagnostics/diagnostics_summary.csv b/results/v15_external_diagnostics/diagnostics_summary.csv new file mode 100644 index 0000000000000000000000000000000000000000..f7abd3e9e7afcaf7abdf38482e7feaa5df0df7be --- /dev/null +++ b/results/v15_external_diagnostics/diagnostics_summary.csv @@ -0,0 +1,10 @@ +env_id,fg_mse_h10_mean,fg_mse_h10_std,fg_mse_h20_mean,fg_mse_h20_std,fg_mse_h30_mean,fg_mse_h30_std,fg_mse_h5_mean,fg_mse_h5_std,latent_cos_h10_mean,latent_cos_h10_std,latent_cos_h20_mean,latent_cos_h20_std,latent_cos_h30_mean,latent_cos_h30_std,latent_cos_h5_mean,latent_cos_h5_std,model,motion_mse_h10_mean,motion_mse_h10_std,motion_mse_h20_mean,motion_mse_h20_std,motion_mse_h30_mean,motion_mse_h30_std,motion_mse_h5_mean,motion_mse_h5_std,mse_h10_mean,mse_h10_std,mse_h20_mean,mse_h20_std,mse_h30_mean,mse_h30_std,mse_h5_mean,mse_h5_std,n_seeds,psnr_h10_mean,psnr_h10_std,psnr_h20_mean,psnr_h20_std,psnr_h30_mean,psnr_h30_std,psnr_h5_mean,psnr_h5_std,ssim_h10_mean,ssim_h10_std,ssim_h20_mean,ssim_h20_std,ssim_h30_mean,ssim_h30_std,ssim_h5_mean,ssim_h5_std,static_mse_h10_mean,static_mse_h10_std,static_mse_h20_mean,static_mse_h20_std,static_mse_h30_mean,static_mse_h30_std,static_mse_h5_mean,static_mse_h5_std,token_acc_h10_mean,token_acc_h10_std,token_acc_h20_mean,token_acc_h20_std,token_acc_h30_mean,token_acc_h30_std,token_acc_h5_mean,token_acc_h5_std +CartPole-v1,0.08985769141762902,0.023994482330403348,0.11779988683556439,0.015736412746706497,0.13123697760784908,0.013925349140851527,0.07275053626600929,0.02028269014790477,0.7926232281373814,0.05007524361458368,0.7114977690798696,0.06174182049701381,0.715929616894573,0.055484670568596274,0.8720389526337385,0.029111009485614948,neural_operator,0.0003648447479442041,2.359537828203645e-05,0.0005037014760507941,2.6477923197470525e-05,0.0003432817045196046,2.7673308412457818e-05,0.0001413040750822603,1.4923825680594977e-05,0.0013807546578107122,0.0003201980168781204,0.0021308187127658585,0.00032380390283901297,0.002225379594960941,0.00031456674854856987,0.0009083902051258974,0.00013504859443348938,10,30.512016925215722,1.0548027973706384,28.40007826387882,0.7596834347363502,28.466124141216277,0.5288417544990378,31.845747685432436,0.7252560894940735,0.8723710954189301,0.030069036511139097,0.8064152002334595,0.02690774145898966,0.7996736466884613,0.02513659903869109,0.9127305030822754,0.012944805229850094,0.0016746692103026817,0.0,0.002196556895228241,0.0,0.0029960669583708466,0.0,0.000707036524921989,0.0,0.57187501124572,0.06923754002757304,0.48718750942498446,0.043036090827767916,0.48812500964850186,0.04641674228673935,0.7037500123260543,0.0471588622227027 +CartPole-v1,0.14046988978042813,0.022831025653631383,0.13414652898161422,0.02078972109320978,0.14936907900134141,0.019427236932305993,0.10509636032110041,0.02011297954436376,0.16773381991806674,0.1257286624038551,0.16588551478198496,0.15234936718611816,0.18423259461997077,0.1719597636086723,0.39821333118525215,0.09492643286333918,swm_structured_cf_v9,0.005330453668929669,0.0050590951445949025,0.004339996883462515,0.00466728822212004,0.004321747288796729,0.004635996350416084,0.006128645306633018,0.0055764539878229175,0.0426075544065494,0.05297250119349221,0.040428630021251875,0.05945796681964293,0.03990698309459404,0.059595817191866636,0.026379062184815893,0.023450597791770916,10,19.483057244867087,6.747606167257856,20.24054703861475,7.19004088989127,20.378417890518904,7.268879352562722,22.294763510674237,5.976052811340526,0.7061755210161209,0.1369307357926813,0.7037815928459168,0.15768946244505172,0.6870477318763732,0.15945402923058644,0.8290678560733795,0.0689631795320436,0.0016746692103026817,0.0,0.002196556895228241,0.0,0.0029960669583708466,0.0,0.000707036524921989,0.0,0.34781250746455045,0.12051908041783878,0.3650000083260238,0.1289548024442333,0.35843750829808413,0.14500546655354263,0.5762500113807618,0.11441464984715258 +CartPole-v1,0.09265504028585228,0.01680238944121086,0.11100306756488862,0.011172816707598943,0.13103284443599214,0.004496209456286605,0.079686044116103,0.017608257339613145,0.7688282913703006,0.06069843057565902,0.704990480170818,0.09241422987368017,0.6990448168478907,0.08762902532060234,0.8600759260356426,0.029971316560137115,no_spectral_transition,0.0003669019635418991,3.665478246379227e-05,0.0005251712292340116,6.0955023997773774e-05,0.00036448872307204283,3.92782471249874e-05,0.00015848440463681613,2.53460700843328e-05,0.0014107606226843928,0.0002851677905124013,0.0018811674416156165,0.0002523678743168431,0.0020469144604817304,0.00012952986906983834,0.001067877054310884,0.0002578984305163562,10,29.772010990977286,0.7112152974236122,28.54182449579239,0.8694395157629585,28.118068131804467,0.42427746600663957,31.05647478699684,0.9203536238434799,0.8750786125659943,0.026776140073232826,0.8350269258022308,0.01745226999552084,0.8191019535064697,0.009588173554487956,0.9031197726726532,0.0237897276909557,0.0016746692103026817,0.0,0.002196556895228241,0.0,0.0029960669583708466,0.0,0.000707036524921989,0.0,0.5765625121071934,0.049744355936541716,0.5040625114226713,0.047072501748198596,0.5037500106263906,0.03319633388081879,0.6906250124098733,0.046654573711359476 +MountainCar-v0,0.0969335932342654,0.03934097688009415,0.1653960812177871,0.06431922511907931,0.2441778352432628,0.04262546984988092,0.07055464924856629,0.020621688103137505,0.7551084873615764,0.07562807683678727,0.5939890908659435,0.12537904625964139,0.5048911012971076,0.12314460015524945,0.7817699706705753,0.06102146037178444,neural_operator,0.00015839830515242402,1.729590697494225e-05,0.00015047668875232817,1.2685368535451437e-05,0.00013025103556323637,8.761473818306059e-06,0.00011837407957610291,1.746261391879281e-05,0.0008322148588714384,0.00029254735608582206,0.0011924395812648926,0.0004462753732046283,0.0021530390333257543,0.0003684169552798206,0.0008069136947952416,0.0002144196888510129,10,35.2771409779787,3.0778965155556297,33.924896070361136,4.36365868864559,29.975949704647064,2.182956871935696,34.858928236365315,1.5323327229012733,0.8823508620262146,0.04309490153199499,0.8298653721809387,0.06523673677322413,0.6977233946323395,0.05205442875721306,0.8878766775131226,0.03008353490787997,0.0010133257156434983,0.0,0.002459771895925611,0.0,0.003329529117545693,0.0,0.0003009121570227147,0.0,0.8218750089406968,0.06476415579927906,0.6968750128522515,0.04121573003095778,0.6833333462011069,0.04945936369686411,0.8453125081956386,0.06652047098398667 +MountainCar-v0,0.14310670931908714,0.02213301889495468,0.20682117269525407,0.05354290543758011,0.29165754622799794,0.041546650648675944,0.10793955997314982,0.022979194866993794,0.4467269505083095,0.27185217489480645,0.376667485764483,0.3302357641914893,0.37032224110007517,0.34404169798659023,0.6600957654067315,0.11014914525463271,swm_structured_cf_v9,0.008873581162375436,0.014924597465619851,0.008473834435153423,0.012395098203320645,0.008649825280807268,0.012709126294005998,0.0039033399837814287,0.008136318381374479,0.032907525385599004,0.044711599756332876,0.03528914956623339,0.04470442749061537,0.036727788559255715,0.04430603863029529,0.011318092845976934,0.015192734780576022,10,20.751849530637266,5.297025007655365,20.380457495898007,5.754247822131502,19.54254746362567,5.307961806064355,22.85034869834781,2.4104373755775015,0.6750842809677124,0.08786698409038335,0.6808183193206787,0.07746162289785581,0.5922603785991669,0.06245911616507363,0.6964175641536713,0.02236030468041272,0.0010133257156434983,0.0,0.002459771895925611,0.0,0.003329529117545693,0.0,0.0003009121570227147,0.0,0.684375008288771,0.23412667000005796,0.572395842615515,0.2046322009768628,0.5541666756384075,0.18907198734715872,0.8484375079162418,0.04974193052370073 +MountainCar-v0,0.13198847905124325,0.0225148260227656,0.18285317328170964,0.058766207614391344,0.2834442805103826,0.05861371660232967,0.12352786715693184,0.02993412927560965,0.6783403663139325,0.09002265651863703,0.5796514172456227,0.10637086964817973,0.5575281818717486,0.10578284961369783,0.7287591508065816,0.08176609829034623,no_spectral_transition,0.00014791898550496255,3.534266757024026e-06,0.00014174092999858916,1.921680719730078e-06,0.00012223758800890505,2.9561828989105877e-06,9.963811566723821e-05,2.4067918680105392e-05,0.0030985108693130313,0.0005906303220397358,0.0030726670555850434,0.0007667556250431912,0.004215881116033415,0.0007322704482152995,0.0034402174551360076,0.0005691127365819045,10,25.416478392481803,0.990633947184614,25.518394100666047,1.2067071729540135,24.244328591227532,0.8653706814339032,24.957370018959047,0.7912661563459749,0.7284854769706726,0.038241973345802885,0.7322790086269378,0.051725547225928675,0.6338964343070984,0.038157446411331224,0.7008408188819886,0.031962330756574564,0.0010133257156434983,0.0,0.002459771895925611,0.0,0.003329529117545693,0.0,0.0003009121570227147,0.0,0.7609375109430403,0.06933367705325785,0.6692708455957472,0.07544596311375042,0.6755208455026149,0.06583730687945742,0.8036458438262344,0.07312511185494172 +Acrobot-v1,0.0504242554976372,0.0033995837359064524,0.06275618389918236,0.0038651167160405047,0.07326299560445478,0.004647717742723746,0.03946720661197105,0.0033987740718341513,0.8676469075027853,0.042087267919109585,0.807029741932638,0.06864890936583294,0.7741591196507216,0.07435681193492266,0.8901473112404347,0.029080005290101626,neural_operator,0.0007886384695211746,2.1653462317755074e-05,0.0014444723945416626,8.754700532071072e-06,0.0019295859772597624,3.4162267367902106e-05,0.0006547288805833773,1.7268588951019004e-05,0.0013698254201699456,0.00015471472027374987,0.002049257147245953,0.00012685078452871892,0.002432275570845377,0.0001106871843495782,0.0011135574674881355,0.0002239749843697272,10,29.12536272406578,0.4549265455871346,27.643630203604697,0.28074765119862,26.89490567445755,0.21373320571743495,29.885264614224432,0.761171253266611,0.8255472242832184,0.01977576338008268,0.7355433464050293,0.014218960784310408,0.683459210395813,0.011691591934459521,0.8609769403934479,0.028301190574136798,0.0029196965099345107,0.0,0.0029245632103993557,0.0,0.0037530396159013435,0.0,0.00275761558786769,0.0,0.8118303676135838,0.01873611699963312,0.7562500116880984,0.023758315032396927,0.7272321574855596,0.012749076923454573,0.8488839369732887,0.0345649944845051 +Acrobot-v1,0.05938096855097683,0.012301423107102166,0.07052719605235325,0.014858842777009686,0.07690972372001852,0.01589765079572944,0.047739241407361986,0.006693100635181972,0.6365710275626043,0.20389092621013669,0.5129570318458718,0.27536691532924634,0.4797519237559754,0.2932847437790469,0.773808000003919,0.061224821900041165,swm_structured_cf_v9,0.002247980507218017,0.0031233421604539083,0.0051635759499731645,0.0054226458321553605,0.006324871870504722,0.006508757786680497,0.0009950146046321605,0.0009304933820662966,0.022299586058761634,0.05554091527807243,0.032512154509640824,0.058598145614013546,0.03333119561102649,0.05555701306633595,0.0029035973070676846,0.0020944162807651186,10,23.848980335891248,5.6712342240799805,21.471910865604876,6.086306262558284,21.040468013286592,5.946774819342807,26.428945419192313,1.3131471449276317,0.6331313796341419,0.20811135997578525,0.558045445382595,0.1750708630297379,0.5338313624262809,0.16248452115624093,0.7413138270378112,0.06557129694394628,0.0029196965099345107,0.0,0.0029245632103993557,0.0,0.0037530396159013435,0.0,0.00275761558786769,0.0,0.6455357265891507,0.19936811448197655,0.572098225634545,0.22402642810971884,0.5462053693132475,0.22016280197307928,0.7687500117346644,0.0483367583841147 +Acrobot-v1,0.05140941300560371,0.003302036496581226,0.06365204431331221,0.004616706603556715,0.07309086186724016,0.004819409983500384,0.038432629974022345,0.0018541940152403352,0.8323865792015568,0.04739065404598482,0.7462202141527087,0.07094803307600313,0.7123165629454888,0.07890956419391568,0.8711644691415131,0.024229276525416614,no_spectral_transition,0.000809873736955069,2.2215135925000496e-05,0.001461970057948747,1.7599549455115628e-05,0.00193650085689967,3.220169224221441e-05,0.0006692985210548841,1.9643584483073236e-05,0.0021928015125922684,0.00034280409386951345,0.0028986657823224958,0.000325776631582699,0.0032440547893202165,0.000301249900212126,0.0018656029395060613,0.00035113478032448126,10,26.837787035107613,0.737433704049703,25.77373519539833,0.5825097746254083,25.290401527285574,0.49737081880975137,27.449628791213037,0.8510219600657765,0.7623339295387268,0.030564977998986238,0.6819445431232453,0.025401671435785537,0.6401736736297607,0.021188889073415235,0.8012036740779876,0.032335207369590224,0.0029196965099345107,0.0,0.0029245632103993557,0.0,0.0037530396159013435,0.0,0.00275761558786769,0.0,0.8107142962515355,0.014187721421720504,0.7473214405123144,0.013384587282969818,0.7236607289873064,0.008137037347140287,0.8611607222352177,0.018340433387084555 diff --git a/results/v6_structured_transport/metrics_by_seed.csv b/results/v6_structured_transport/metrics_by_seed.csv new file mode 100644 index 0000000000000000000000000000000000000000..4aaeb655f21abf09db51bd7e0b3a080e93dd704d --- /dev/null +++ b/results/v6_structured_transport/metrics_by_seed.csv @@ -0,0 +1,3 @@ +seed,model,params,test_mse,test_text_acc,cf_final_image_mse,cf_effect_vector_cosine,cf_effect_magnitude_ratio,rollout_mse_h5,rollout_text_acc_h5,rollout_latent_cosine_h5,raw_operator_norm_max,raw_operator_norm_mean,effective_operator_norm_max,effective_operator_norm_mean,fraction_modes_clipped,rho,transport_orthogonality_error,mean_dissipative_keep +0,swm_structured,407807,0.030684633180499077,0.7447222222222222,0.03891142123999695,0.9700712518145641,0.01697352527116891,0.029978245496749878,0.7020833333333333,0.8126939535140991,8.035642623901367,6.814639091491699,1.0499999523162842,1.0499999523162842,0.010416666977107525,1.05,2.4690794475645816e-07,0.981905460357666 +0,no_spectral_transition,446802,0.03044491820037365,0.6829166666666666,0.038256316824117675,-0.9665572692950567,0.00365413070176146,0.029460513964295387,0.6529166666666667,0.9047141075134277,,,,,,,, diff --git a/results/v6_structured_transport/metrics_summary.csv b/results/v6_structured_transport/metrics_summary.csv new file mode 100644 index 0000000000000000000000000000000000000000..7304ef166d25a88027789418d1884ca218dac7a6 --- /dev/null +++ b/results/v6_structured_transport/metrics_summary.csv @@ -0,0 +1,3 @@ +model,params,n_seeds,test_mse_mean,test_mse_std,test_text_acc_mean,test_text_acc_std,cf_final_image_mse_mean,cf_final_image_mse_std,cf_effect_vector_cosine_mean,cf_effect_vector_cosine_std,cf_effect_magnitude_ratio_mean,cf_effect_magnitude_ratio_std,rollout_mse_h5_mean,rollout_mse_h5_std,rollout_text_acc_h5_mean,rollout_text_acc_h5_std,rollout_latent_cosine_h5_mean,rollout_latent_cosine_h5_std,transport_orthogonality_error_mean,transport_orthogonality_error_std,effective_operator_norm_max_mean,effective_operator_norm_max_std,effective_operator_norm_mean_mean,effective_operator_norm_mean_std,raw_operator_norm_mean_mean,raw_operator_norm_mean_std,mean_dissipative_keep_mean,mean_dissipative_keep_std,fraction_modes_clipped_mean,fraction_modes_clipped_std,raw_operator_norm_max_mean,raw_operator_norm_max_std,rho_mean,rho_std +swm_structured,407807,1,0.030684633180499077,0.0,0.7447222222222222,0.0,0.03891142123999695,0.0,0.9700712518145641,0.0,0.01697352527116891,0.0,0.029978245496749878,0.0,0.7020833333333333,0.0,0.8126939535140991,0.0,2.4690794475645816e-07,0.0,1.0499999523162842,0.0,1.0499999523162842,0.0,6.814639091491699,0.0,0.981905460357666,0.0,0.010416666977107525,0.0,8.035642623901367,0.0,1.05,0.0 +no_spectral_transition,446802,1,0.03044491820037365,0.0,0.6829166666666666,0.0,0.038256316824117675,0.0,-0.9665572692950567,0.0,0.00365413070176146,0.0,0.029460513964295387,0.0,0.6529166666666667,0.0,0.9047141075134277,0.0,,,,,,,,,,,,,,,, diff --git a/results/v6_structured_transport/run_config.json b/results/v6_structured_transport/run_config.json new file mode 100644 index 0000000000000000000000000000000000000000..6e28c8577dd568bb13fa97bfc1eee1afbd86df94 --- /dev/null +++ b/results/v6_structured_transport/run_config.json @@ -0,0 +1,17 @@ +{ + "epochs": 1, + "batch_size": 64, + "seeds": [ + 0 + ], + "horizons": [ + 5 + ], + "cf_horizon": 5, + "device": "cpu", + "models": [ + "swm_structured", + "no_spectral_transition" + ], + "device_resolved": "cpu" +} \ No newline at end of file diff --git a/results/v6_structured_transport/seed_0/no_spectral_transition.pt b/results/v6_structured_transport/seed_0/no_spectral_transition.pt new file mode 100644 index 0000000000000000000000000000000000000000..026269a04d63a447e0cdc7d0878d56e52519d1a7 --- /dev/null +++ b/results/v6_structured_transport/seed_0/no_spectral_transition.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b55e6b924e0ef2595042db0e607d01eddb069207e17d5e8ec1e18ef752765bb2 +size 1798961 diff --git a/results/v6_structured_transport/seed_0/no_spectral_transition_history.csv b/results/v6_structured_transport/seed_0/no_spectral_transition_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..3e09f6e64450251442c6b4d7eff1282e00378a71 --- /dev/null +++ b/results/v6_structured_transport/seed_0/no_spectral_transition_history.csv @@ -0,0 +1,2 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +no_spectral_transition,1,0.15102833081036807,0.0990018825168195,0.02923606149852276,15.340811332417182,0.10233747959136963,-0.18618392944335938,0.6665277777777778 diff --git a/results/v6_structured_transport/seed_0/no_spectral_transition_metrics.json b/results/v6_structured_transport/seed_0/no_spectral_transition_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..a6ad969d5a590fed4eece09e46baae26ceef91cc --- /dev/null +++ b/results/v6_structured_transport/seed_0/no_spectral_transition_metrics.json @@ -0,0 +1,52 @@ +{ + "model": "no_spectral_transition", + "params": 446802, + "epochs": 1, + "best_val_loss": 0.0990018825168195, + "val": { + "loss": 0.0990018825168195, + "mse": 0.02923606149852276, + "psnr": 15.340811332417182, + "ssim": 0.10233747959136963, + "nll": -0.18618392944335938, + "text_acc": 0.6665277777777778 + }, + "test": { + "loss": 0.09991374287916266, + "mse": 0.03044491820037365, + "psnr": 15.164851884734452, + "ssim": 0.10065317898988724, + "nll": -0.1442098468542099, + "text_acc": 0.6829166666666666 + }, + "rollout": { + "rollout_mse": 0.029460513964295387, + "rollout_text_acc": 0.6529166666666667, + "rollout_latent_cosine": 0.9047141075134277, + "rollout_joint_score": 0.6517651959337575, + "time_per_rollout_step_ms": 0.030902148436950938 + }, + "rollout_by_horizon": { + "5": { + "rollout_mse": 0.029460513964295387, + "rollout_text_acc": 0.6529166666666667, + "rollout_latent_cosine": 0.9047141075134277, + "rollout_joint_score": 0.6517651959337575, + "time_per_rollout_step_ms": 0.030902148436950938 + } + }, + "stability_diagnostics": {}, + "history": [ + { + "model": "no_spectral_transition", + "epoch": 1, + "train_loss": 0.15102833081036807, + "val_loss": 0.0990018825168195, + "val_mse": 0.02923606149852276, + "val_psnr": 15.340811332417182, + "val_ssim": 0.10233747959136963, + "val_nll": -0.18618392944335938, + "val_text_acc": 0.6665277777777778 + } + ] +} \ No newline at end of file diff --git a/results/v6_structured_transport/seed_0/swm_structured.pt b/results/v6_structured_transport/seed_0/swm_structured.pt new file mode 100644 index 0000000000000000000000000000000000000000..9266d95195fbd265c47f719147d97687fe0af0ef --- /dev/null +++ b/results/v6_structured_transport/seed_0/swm_structured.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9e232e5f3e0d28283c4e7ae16c19f423734a6e9d80d3970ef6dc94675f283a6c +size 1643477 diff --git a/results/v6_structured_transport/seed_0/swm_structured_history.csv b/results/v6_structured_transport/seed_0/swm_structured_history.csv new file mode 100644 index 0000000000000000000000000000000000000000..8ad5df3b6f46a388c52d44e3787a8e7c1b3e41e9 --- /dev/null +++ b/results/v6_structured_transport/seed_0/swm_structured_history.csv @@ -0,0 +1,2 @@ +model,epoch,train_loss,val_loss,val_mse,val_psnr,val_ssim,val_nll,val_text_acc +swm_structured,1,0.14737580275783937,0.11526541832996451,0.02945595793426037,15.308268490509393,0.09588314592838287,-0.17854872345924377,0.7309722222222222 diff --git a/results/v6_structured_transport/seed_0/swm_structured_metrics.json b/results/v6_structured_transport/seed_0/swm_structured_metrics.json new file mode 100644 index 0000000000000000000000000000000000000000..19c8063e8cfab8a4b7d48eb9068265f3ed3b1945 --- /dev/null +++ b/results/v6_structured_transport/seed_0/swm_structured_metrics.json @@ -0,0 +1,61 @@ +{ + "model": "swm_structured", + "params": 407807, + "epochs": 1, + "best_val_loss": 0.11526541832996451, + "val": { + "loss": 0.11526541832996451, + "mse": 0.02945595793426037, + "psnr": 15.308268490509393, + "ssim": 0.09588314592838287, + "nll": -0.17854872345924377, + "text_acc": 0.7309722222222222 + }, + "test": { + "loss": 0.11625830764355867, + "mse": 0.030684633180499077, + "psnr": 15.130790641231592, + "ssim": 0.09395081549882889, + "nll": -0.13588643074035645, + "text_acc": 0.7447222222222222 + }, + "rollout": { + "rollout_mse": 0.029978245496749878, + "rollout_text_acc": 0.7020833333333333, + "rollout_latent_cosine": 0.8126939535140991, + "rollout_joint_score": 0.6551427138208794, + "time_per_rollout_step_ms": 0.1993113734357621 + }, + "rollout_by_horizon": { + "5": { + "rollout_mse": 0.029978245496749878, + "rollout_text_acc": 0.7020833333333333, + "rollout_latent_cosine": 0.8126939535140991, + "rollout_joint_score": 0.6551427138208794, + "time_per_rollout_step_ms": 0.1993113734357621 + } + }, + "stability_diagnostics": { + "raw_operator_norm_max": 8.035642623901367, + "raw_operator_norm_mean": 6.814639091491699, + "effective_operator_norm_max": 1.0499999523162842, + "effective_operator_norm_mean": 1.0499999523162842, + "fraction_modes_clipped": 0.010416666977107525, + "rho": 1.05, + "transport_orthogonality_error": 2.4690794475645816e-07, + "mean_dissipative_keep": 0.981905460357666 + }, + "history": [ + { + "model": "swm_structured", + "epoch": 1, + "train_loss": 0.14737580275783937, + "val_loss": 0.11526541832996451, + "val_mse": 0.02945595793426037, + "val_psnr": 15.308268490509393, + "val_ssim": 0.09588314592838287, + "val_nll": -0.17854872345924377, + "val_text_acc": 0.7309722222222222 + } + ] +} \ No newline at end of file diff --git a/scripts/run_quick_benchmark.py b/scripts/run_quick_benchmark.py new file mode 100644 index 0000000000000000000000000000000000000000..01b927ac7ae90f8e014612d24eb211c68cc5504f --- /dev/null +++ b/scripts/run_quick_benchmark.py @@ -0,0 +1,64 @@ +"""Fair multi-seed + long-horizon benchmark for Spectral World Models.""" +from __future__ import annotations +import argparse, csv, json +from pathlib import Path +from statistics import mean, stdev +import torch +from spectral_world_models.data import DatasetConfig, generate_benchmark_npz +from spectral_world_models.train import train_one_model + +DEFAULT_MODELS=["dreamer","transformer","koopman","neural_operator","swm","no_spectral_state","no_spectral_transition","no_text_branch","no_stability_penalty"] +BASE_METRICS=[("test_psnr",lambda r:r["test"]["psnr"]),("test_ssim",lambda r:r["test"]["ssim"]),("test_nll",lambda r:r["test"]["nll"]),("test_mse",lambda r:r["test"]["mse"]),("test_text_acc",lambda r:r["test"]["text_acc"])] +def stats(xs): return (mean(xs), stdev(xs) if len(xs)>1 else 0.0) + +def main(): + ap=argparse.ArgumentParser(description="Run fair multi-seed SWM benchmark with H=5/10/20/30 rollouts.") + ap.add_argument("--data",default=None); ap.add_argument("--rollout-data",default=None); ap.add_argument("--out",default=None) + ap.add_argument("--epochs",type=int,default=3); ap.add_argument("--batch-size",type=int,default=64) + ap.add_argument("--seeds",type=int,nargs="+",default=[0,1,2,3,4]); ap.add_argument("--horizons",type=int,nargs="+",default=[5,10,20,30]) + ap.add_argument("--device",default="auto"); ap.add_argument("--models",nargs="+",default=DEFAULT_MODELS) + args=ap.parse_args(); root=Path(__file__).resolve().parents[1] + data=Path(args.data) if args.data else root/"data"/"mini_moving_shapes.npz" + out=Path(args.out) if args.out else root/"results"/"multiseed_horizons" + rollout_data=Path(args.rollout_data) if args.rollout_data else root/"data"/"mini_moving_shapes_rollout_h31.npz" + if not data.exists(): generate_benchmark_npz(data,DatasetConfig()) + need_len=max(args.horizons)+1 + regenerate=True + if rollout_data.exists(): + try: + import numpy as np + with np.load(rollout_data,allow_pickle=True) as d: regenerate=d["test_images"].shape[1] < need_len + except Exception: regenerate=True + if regenerate: + # Separate held-out long sequences: training data remain unchanged. + generate_benchmark_npz(rollout_data,DatasetConfig(seq_len=need_len,train_sequences=1,val_sequences=1,test_sequences=64,seed=7007)) + device=("cuda" if torch.cuda.is_available() else "cpu") if args.device=="auto" else args.device + if device.startswith("cuda") and not torch.cuda.is_available(): raise RuntimeError("CUDA requested but unavailable") + out.mkdir(parents=True,exist_ok=True); grouped={m:[] for m in args.models}; raw=[] + for seed in args.seeds: + sd=out/f"seed_{seed}" + for m in args.models: + print(f"\n=== seed={seed} model={m} horizons={args.horizons} ===") + r=train_one_model(m,data,sd,epochs=args.epochs,batch_size=args.batch_size,seed=seed,device=device,rollout_data_path=rollout_data,rollout_horizons=tuple(args.horizons)) + grouped[m].append(r); row={"seed":seed,"model":m,"params":r["params"]} + for k,g in BASE_METRICS: row[k]=g(r) + for h in args.horizons: + rr=r["rollout_by_horizon"][str(h)]; row[f"rollout_mse_h{h}"]=rr["rollout_mse"]; row[f"time_per_rollout_step_ms_h{h}"]=rr["time_per_rollout_step_ms"] + raw.append(row) + with (out/"metrics_by_seed.csv").open("w",newline="") as f: + w=csv.DictWriter(f,fieldnames=list(raw[0])); w.writeheader(); w.writerows(raw) + summary=[] + for m,runs in grouped.items(): + row={"model":m,"params":runs[0]["params"],"n_seeds":len(runs)} + for k,g in BASE_METRICS: + mu,sd=stats([float(g(r)) for r in runs]); row[k+"_mean"]=mu; row[k+"_std"]=sd + for h in args.horizons: + vals=[float(r["rollout_by_horizon"][str(h)]["rollout_mse"]) for r in runs]; mu,sd=stats(vals); row[f"rollout_mse_h{h}_mean"]=mu; row[f"rollout_mse_h{h}_std"]=sd + summary.append(row) + with (out/"metrics_summary.csv").open("w",newline="") as f: + w=csv.DictWriter(f,fieldnames=list(summary[0])); w.writeheader(); w.writerows(summary) + with (out/"run_config.json").open("w") as f: json.dump(vars(args)|{"device_resolved":device,"rollout_data_resolved":str(rollout_data)},f,indent=2) + print("\n=== Long-horizon summary ===") + for r in summary: print(r["model"]+": "+" | ".join(f'H{h} {r[f"rollout_mse_h{h}_mean"]:.6f} ± {r[f"rollout_mse_h{h}_std"]:.6f}' for h in args.horizons)) + print(f"\nWrote: {out/'metrics_summary.csv'}") +if __name__=="__main__": main() diff --git a/scripts/run_stability_benchmark.py b/scripts/run_stability_benchmark.py new file mode 100644 index 0000000000000000000000000000000000000000..f26a69966f98da44396ca56c8989e25e213ab421 --- /dev/null +++ b/scripts/run_stability_benchmark.py @@ -0,0 +1,72 @@ +"""Targeted attribution benchmark for SWM transition stability.""" +from __future__ import annotations +import argparse, csv, json +from pathlib import Path +from statistics import mean, stdev +import torch +from spectral_world_models.data import DatasetConfig, generate_benchmark_npz +from spectral_world_models.train import train_one_model + +DEFAULT_MODELS=["koopman","neural_operator","swm","swm_selective","swm_spectral_norm","no_spectral_transition","no_stability_penalty"] +BASE=[("test_psnr",lambda r:r["test"]["psnr"]),("test_ssim",lambda r:r["test"]["ssim"]),("test_nll",lambda r:r["test"]["nll"]),("test_mse",lambda r:r["test"]["mse"]),("test_text_acc",lambda r:r["test"]["text_acc"])] +def stats(xs): return mean(xs), stdev(xs) if len(xs)>1 else 0.0 + +def main(): + ap=argparse.ArgumentParser(description="SWM v4 benchmark: selective spectral regulation and long-horizon multimodal information preservation.") + ap.add_argument("--data",default=None); ap.add_argument("--rollout-data",default=None); ap.add_argument("--out",default=None) + ap.add_argument("--epochs",type=int,default=3); ap.add_argument("--batch-size",type=int,default=64) + ap.add_argument("--seeds",type=int,nargs="+",default=[0,1,2,3,4]); ap.add_argument("--horizons",type=int,nargs="+",default=[5,10,20,30,50,100]) + ap.add_argument("--device",default="auto"); ap.add_argument("--models",nargs="+",default=DEFAULT_MODELS); ap.add_argument("--lambda-stability",type=float,default=0.01) + a=ap.parse_args(); root=Path(__file__).resolve().parents[1] + data=Path(a.data) if a.data else root/"data"/"mini_moving_shapes.npz" + out=Path(a.out) if a.out else root/"results"/"stability_attribution" + rollout=Path(a.rollout_data) if a.rollout_data else root/f"data/mini_moving_shapes_rollout_h{max(a.horizons)+1}.npz" + if not data.exists(): generate_benchmark_npz(data,DatasetConfig()) + need=max(a.horizons)+1; regen=True + if rollout.exists(): + try: + import numpy as np + with np.load(rollout,allow_pickle=True) as d: regen=d["test_images"].shape[1] < need + except Exception: pass + if regen: generate_benchmark_npz(rollout,DatasetConfig(seq_len=need,train_sequences=1,val_sequences=1,test_sequences=64,seed=7007)) + device=("cuda" if torch.cuda.is_available() else "cpu") if a.device=="auto" else a.device + if device.startswith("cuda") and not torch.cuda.is_available(): raise RuntimeError("CUDA requested but unavailable") + out.mkdir(parents=True,exist_ok=True); grouped={m:[] for m in a.models}; raw=[] + for seed in a.seeds: + for m in a.models: + print(f"\n=== seed={seed} model={m} horizons={a.horizons} ===") + r=train_one_model(m,data,out/f"seed_{seed}",epochs=a.epochs,batch_size=a.batch_size,seed=seed,device=device,rollout_data_path=rollout,rollout_horizons=tuple(a.horizons),lambda_stability=a.lambda_stability) + grouped[m].append(r); row={"seed":seed,"model":m,"params":r["params"]} + for k,g in BASE: row[k]=g(r) + for h in a.horizons: + rr=r["rollout_by_horizon"][str(h)] + for metric in ["rollout_mse","rollout_text_acc","rollout_latent_cosine","rollout_joint_score"]: row[f"{metric}_h{h}"]=rr[metric] + for k,v in r.get("stability_diagnostics",{}).items(): row[k]=v + raw.append(row) + fields=[] + for r in raw: + for k in r: + if k not in fields: fields.append(k) + with (out/"metrics_by_seed.csv").open("w",newline="") as f: + w=csv.DictWriter(f,fieldnames=fields); w.writeheader(); w.writerows(raw) + summary=[] + for m,runs in grouped.items(): + row={"model":m,"params":runs[0]["params"],"n_seeds":len(runs)} + for k,g in BASE: + mu,sd=stats([float(g(r)) for r in runs]); row[k+"_mean"]=mu; row[k+"_std"]=sd + for h in a.horizons: + for metric in ["rollout_mse","rollout_text_acc","rollout_latent_cosine","rollout_joint_score"]: + vals=[float(r["rollout_by_horizon"][str(h)][metric]) for r in runs]; mu,sd=stats(vals); row[f"{metric}_h{h}_mean"]=mu; row[f"{metric}_h{h}_std"]=sd + diag_keys=set().union(*(r.get("stability_diagnostics",{}).keys() for r in runs)) + for k in sorted(diag_keys): + vals=[float(r["stability_diagnostics"][k]) for r in runs if k in r.get("stability_diagnostics",{})]; mu,sd=stats(vals); row[k+"_mean"]=mu; row[k+"_std"]=sd + summary.append(row) + fields=[] + for r in summary: + for k in r: + if k not in fields: fields.append(k) + with (out/"metrics_summary.csv").open("w",newline="") as f: + w=csv.DictWriter(f,fieldnames=fields); w.writeheader(); w.writerows(summary) + with (out/"run_config.json").open("w") as f: json.dump(vars(a)|{"device_resolved":device,"rollout_data_resolved":str(rollout)},f,indent=2) + print("\nWrote",out/"metrics_summary.csv") +if __name__=="__main__": main() diff --git a/scripts/run_v10_audit.py b/scripts/run_v10_audit.py new file mode 100644 index 0000000000000000000000000000000000000000..358080fa0bc2d868c0c2a9356de988614c234c8d --- /dev/null +++ b/scripts/run_v10_audit.py @@ -0,0 +1,49 @@ +"""V10: frozen-model counterfactual benchmark integrity and physics audit.""" +from __future__ import annotations +import argparse,csv,json +from pathlib import Path +from statistics import mean,stdev +import torch +from spectral_world_models.models import build_model +from spectral_world_models.train import train_one_model +from spectral_world_models.v9_causal_generalization import generate_family_npz +from spectral_world_models.v10_cf_audit import REGIMES,build_oracle_audit,assert_audit_integrity,evaluate_counterfactual_audit +DEFAULT_MODELS=['neural_operator','swm_selective','swm_structured_cf','swm_structured_cf_v9','no_spectral_transition'] +def stats(x): return mean(x),stdev(x) if len(x)>1 else 0. +def main(): + ap=argparse.ArgumentParser(description='V10 counterfactual integrity audit with regime-specific oracle trajectory hashes.') + ap.add_argument('--epochs',type=int,default=3); ap.add_argument('--batch-size',type=int,default=64); ap.add_argument('--seeds',type=int,nargs='+',default=[0,1,2,3,4]); ap.add_argument('--cf-horizon',type=int,default=30); ap.add_argument('--cf-pairs',type=int,default=48); ap.add_argument('--device',default='auto'); ap.add_argument('--models',nargs='+',default=DEFAULT_MODELS); a=ap.parse_args() + root=Path(__file__).resolve().parents[1]; out=root/'results/v10_cf_integrity'; out.mkdir(parents=True,exist_ok=True); data=root/'data/v9_train_family.npz' + if not data.exists(): generate_family_npz(data) + audit=build_oracle_audit(n=a.cf_pairs,horizon=a.cf_horizon,seed=10101); assert_audit_integrity(audit) + with (out/'oracle_physics_audit.csv').open('w',newline='') as f: w=csv.DictWriter(f,fieldnames=list(audit[0])); w.writeheader(); w.writerows(audit) + device=('cuda' if torch.cuda.is_available() else 'cpu') if a.device=='auto' else a.device; dev=torch.device(device); raw=[] + for seed in a.seeds: + for m in a.models: + print(f'V10 seed={seed} model={m}'); sd=out/f'seed_{seed}' + r=train_one_model(m,data,sd,epochs=a.epochs,batch_size=a.batch_size,seed=seed,device=device,cf_train_horizon=10,cf_pairs=48,cf_every=4,lambda_cf_dir=.02,lambda_cf_mag=.005,lambda_cf_branch=.08) + ck=torch.load(sd/f'{m}.pt',map_location=dev); model=build_model(m).to(dev); model.load_state_dict(ck['model']); row={'seed':seed,'model':m,'params':r['params']} + for rn,cfg in REGIMES.items(): + met=evaluate_counterfactual_audit(model,dev,cfg,n=a.cf_pairs,horizon=a.cf_horizon,seed=10101) + row.update({f'{rn}_{k}':v for k,v in met.items()}) + raw.append(row) + fields=[] + for r in raw: + for k in r: + if k not in fields: fields.append(k) + with (out/'metrics_by_seed.csv').open('w',newline='') as f: w=csv.DictWriter(f,fieldnames=fields); w.writeheader(); w.writerows(raw) + summary=[] + for m in a.models: + rs=[r for r in raw if r['model']==m]; row={'model':m,'params':rs[0]['params'],'n_seeds':len(rs)} + for k in fields: + if k in ('seed','model','params') or not all(k in r and isinstance(r[k],(int,float)) for r in rs): continue + mu,sd=stats([float(r[k]) for r in rs]); row[k+'_mean']=mu; row[k+'_std']=sd + summary.append(row) + sf=[] + for r in summary: + for k in r: + if k not in sf: sf.append(k) + with (out/'metrics_summary.csv').open('w',newline='') as f: w=csv.DictWriter(f,fieldnames=sf); w.writeheader(); w.writerows(summary) + with (out/'run_config.json').open('w') as f: json.dump(vars(a)|{'device_resolved':device,'audit_seed':10101},f,indent=2) + print('Oracle audit PASS; wrote',out/'metrics_summary.csv') +if __name__=='__main__': main() diff --git a/scripts/run_v11_benchmark.py b/scripts/run_v11_benchmark.py new file mode 100644 index 0000000000000000000000000000000000000000..efbff7603d7c54b7bdb26d0b36cbd5a7000a11a3 --- /dev/null +++ b/scripts/run_v11_benchmark.py @@ -0,0 +1,38 @@ +from __future__ import annotations +import argparse,csv,json +from pathlib import Path +from statistics import mean,stdev +import torch +from spectral_world_models.models import build_model +from spectral_world_models.train import train_one_model +from spectral_world_models.v9_causal_generalization import generate_family_npz +from spectral_world_models.v11_causal_calibration import REGIMES,evaluate_causal_calibration +DEFAULT_MODELS=['neural_operator','swm_structured_cf_v9','swm_structured_cf_v11','no_spectral_transition'] +def stats(x): return mean(x),stdev(x) if len(x)>1 else 0. +def main(): + ap=argparse.ArgumentParser(description='V11 causal-gain calibration benchmark') + ap.add_argument('--epochs',type=int,default=3); ap.add_argument('--batch-size',type=int,default=64); ap.add_argument('--seeds',type=int,nargs='+',default=list(range(10))); ap.add_argument('--cf-horizon',type=int,default=30); ap.add_argument('--cf-pairs',type=int,default=64); ap.add_argument('--device',default='auto'); ap.add_argument('--models',nargs='+',default=DEFAULT_MODELS); a=ap.parse_args() + root=Path(__file__).resolve().parents[1]; out=root/'results/v11_causal_calibration'; out.mkdir(parents=True,exist_ok=True); data=root/'data/v9_train_family.npz' + if not data.exists(): generate_family_npz(data) + device=('cuda' if torch.cuda.is_available() else 'cpu') if a.device=='auto' else a.device; dev=torch.device(device); raw=[] + for seed in a.seeds: + for m in a.models: + print(f'V11 seed={seed} model={m}'); sd=out/f'seed_{seed}' + r=train_one_model(m,data,sd,epochs=a.epochs,batch_size=a.batch_size,seed=seed,device=device,cf_train_horizon=12,cf_pairs=a.cf_pairs,cf_every=4,lambda_cf_dir=.03,lambda_cf_mag=.01,lambda_cf_branch=.08) + ck=torch.load(sd/f'{m}.pt',map_location=dev); model=build_model(m).to(dev); model.load_state_dict(ck['model']); row={'seed':seed,'model':m,'params':r['params']} + for rn,cfg in REGIMES.items(): row.update({f'{rn}_{k}':v for k,v in evaluate_causal_calibration(model,dev,cfg,n=a.cf_pairs,horizon=a.cf_horizon,seed=11101).items()}) + raw.append(row) + fields=list(raw[0]); + with (out/'metrics_by_seed.csv').open('w',newline='') as f: w=csv.DictWriter(f,fieldnames=fields); w.writeheader(); w.writerows(raw) + summary=[] + for m in a.models: + rs=[r for r in raw if r['model']==m]; row={'model':m,'params':rs[0]['params'],'n_seeds':len(rs)} + for k in fields: + if k in ('seed','model','params'): continue + mu,sd=stats([float(r[k]) for r in rs]); row[k+'_mean']=mu; row[k+'_std']=sd + summary.append(row) + sf=list(summary[0]); + with (out/'metrics_summary.csv').open('w',newline='') as f: w=csv.DictWriter(f,fieldnames=sf); w.writeheader(); w.writerows(summary) + with (out/'run_config.json').open('w') as f: json.dump(vars(a)|{'device_resolved':device,'architecture_base':'v9 structured transport','objective':'scale-aware trajectory causal calibration'},f,indent=2) + print('wrote',out/'metrics_summary.csv') +if __name__=='__main__': main() diff --git a/scripts/run_v12_decomposition.py b/scripts/run_v12_decomposition.py new file mode 100644 index 0000000000000000000000000000000000000000..41e33532a7237289b91b4f80d290d50364f71bfa --- /dev/null +++ b/scripts/run_v12_decomposition.py @@ -0,0 +1,67 @@ +from __future__ import annotations +import argparse,csv,json +from pathlib import Path +import torch +from spectral_world_models.models import build_model +from spectral_world_models.train import train_one_model +from spectral_world_models.v9_causal_generalization import generate_family_npz +from spectral_world_models.v12_causal_decomposition import REGIMES,decompose_model,summarize_strata,paired_bootstrap_delta +DEFAULT_MODELS=['neural_operator','swm_structured_cf_v9','no_spectral_transition'] +def writecsv(path,rows): + if not rows:return + fields=[] + for r in rows: + for k in r: + if k not in fields: fields.append(k) + with path.open('w',newline='') as f: + w=csv.DictWriter(f,fieldnames=fields,extrasaction='ignore'); w.writeheader(); w.writerows(rows) +def main(): + ap=argparse.ArgumentParser(description='V12 causal-response decomposition; frozen V9 architecture/objective') + ap.add_argument('--epochs',type=int,default=3); ap.add_argument('--batch-size',type=int,default=64); ap.add_argument('--seeds',type=int,nargs='+',default=list(range(10))) + ap.add_argument('--cf-horizon',type=int,default=30); ap.add_argument('--cf-pairs',type=int,default=64); ap.add_argument('--bootstrap',type=int,default=2000); ap.add_argument('--device',default='auto'); ap.add_argument('--models',nargs='+',default=DEFAULT_MODELS); a=ap.parse_args() + root=Path(__file__).resolve().parents[1]; out=root/'results/v12_causal_decomposition'; out.mkdir(parents=True,exist_ok=True); data=root/'data/v9_train_family.npz' + if not data.exists(): generate_family_npz(data) + device=('cuda' if torch.cuda.is_available() else 'cpu') if a.device=='auto' else a.device; dev=torch.device(device); allrows=[] + for seed in a.seeds: + for m in a.models: + print(f'V12 seed={seed} model={m}'); sd=out/f'seed_{seed}' + # Freeze the V9 experimental recipe: V12 changes evaluation, not learning. + r=train_one_model(m,data,sd,epochs=a.epochs,batch_size=a.batch_size,seed=seed,device=device,cf_train_horizon=10,cf_pairs=48,cf_every=4,lambda_cf_dir=.02,lambda_cf_mag=.005,lambda_cf_branch=.08) + ck=torch.load(sd/f'{m}.pt',map_location=dev); model=build_model(m).to(dev); model.load_state_dict(ck['model']) + for rn,cfg in REGIMES.items(): + rr=decompose_model(model,dev,cfg,rn,n=a.cf_pairs,horizon=a.cf_horizon,seed=12101) + for x in rr:x.update({'seed':seed,'model':m,'params':r['params']}) + allrows.extend(rr) + writecsv(out/'decomposition_by_episode.csv',allrows) + summaries=[] + for seed in a.seeds: + for m in a.models: + for rn in REGIMES: + rs=[r for r in allrows if r['seed']==seed and r['model']==m and r['regime']==rn] + for s in summarize_strata(rs): s.update({'seed':seed,'model':m,'regime':rn}); summaries.append(s) + writecsv(out/'stratified_metrics_by_seed.csv',summaries) + # Aggregate across seeds as observations; preserve seed-level file for uncertainty inspection. + agg=[] + keys=sorted({(r['model'],r['regime'],r['stratum_type'],str(r.get('program','')),str(r.get('horizon','')),str(r.get('effect_bin','')),str(r.get('speed_bin','')),str(r.get('boundary_contact','')),str(r.get('occlusion_zone_exposure',''))) for r in summaries}) + for k in keys: + rs=[r for r in summaries if (r['model'],r['regime'],r['stratum_type'],str(r.get('program','')),str(r.get('horizon','')),str(r.get('effect_bin','')),str(r.get('speed_bin','')),str(r.get('boundary_contact','')),str(r.get('occlusion_zone_exposure','')))==k] + row={'model':k[0],'regime':k[1],'stratum_type':k[2],'program':k[3],'horizon':k[4],'effect_bin':k[5],'speed_bin':k[6],'boundary_contact':k[7],'occlusion_zone_exposure':k[8],'n_seeds':len(rs)} + for metric in ('trajectory_cosine','magnitude_ratio','magnitude_ratio_mae','log_gain_error'): + vals=[float(x[metric+'_mean']) for x in rs]; row[metric+'_mean']=sum(vals)/len(vals); row[metric+'_seed_std']=float(torch.tensor(vals).std(unbiased=True)) if len(vals)>1 else 0. + agg.append(row) + writecsv(out/'stratified_metrics_summary.csv',agg) + # Paired, episode-clustered CIs within each seed/regime, then pooled matched rows across seeds via seed-offset episode IDs. + boots=[]; v9='swm_structured_cf_v9' + for control in [m for m in a.models if m!=v9]: + for rn in REGIMES: + A=[];B=[] + for si,seed in enumerate(a.seeds): + ar=[dict(r,episode=r['episode']+si*100000) for r in allrows if r['model']==v9 and r['regime']==rn and r['seed']==seed] + br=[dict(r,episode=r['episode']+si*100000) for r in allrows if r['model']==control and r['regime']==rn and r['seed']==seed] + A.extend(ar);B.extend(br) + for metric in ('trajectory_cosine','magnitude_ratio_mae','log_gain_error'): + z=paired_bootstrap_delta(A,B,metric=metric,n_boot=a.bootstrap,seed=12201); z.update({'regime':rn,'control':control}); boots.append(z) + writecsv(out/'paired_bootstrap_v9_vs_controls.csv',boots) + with (out/'run_config.json').open('w') as f: json.dump(vars(a)|{'device_resolved':device,'release':'V12 Causal Response Decomposition','learning_changes':'none; V9 recipe frozen','primary_model':'swm_structured_cf_v9'},f,indent=2) + print('wrote',out) +if __name__=='__main__': main() diff --git a/scripts/run_v13_axis_audit.py b/scripts/run_v13_axis_audit.py new file mode 100644 index 0000000000000000000000000000000000000000..c2bcf6aadb99430043a6250bd18ffe1db781d03b --- /dev/null +++ b/scripts/run_v13_axis_audit.py @@ -0,0 +1,48 @@ +from __future__ import annotations +import argparse,csv,json +from pathlib import Path +import numpy as np, torch +from spectral_world_models.models import build_model +from spectral_world_models.train import train_one_model +from spectral_world_models.v9_causal_generalization import generate_family_npz +from spectral_world_models.v13_axis_equivariance import REGIMES,audit_axis_equivariance,audit_forced_collisions +DEFAULT_MODELS=['neural_operator','swm_structured_cf_v9','no_spectral_transition'] +def writecsv(path,rows): + if not rows:return + fields=[] + for r in rows: + for k in r: + if k not in fields: fields.append(k) + with path.open('w',newline='') as f: + w=csv.DictWriter(f,fieldnames=fields); w.writeheader(); w.writerows(rows) +def summarize(rows,keys,metrics): + out=[]; groups={} + for r in rows: groups.setdefault(tuple(r[k] for k in keys),[]).append(r) + for kval,rs in groups.items(): + z=dict(zip(keys,kval)); z['n']=len(rs) + for m in metrics: + x=np.asarray([float(q[m]) for q in rs if np.isfinite(float(q[m]))]); z[m+'_mean']=float(x.mean()) if len(x) else float('nan'); z[m+'_std']=float(x.std(ddof=1)) if len(x)>1 else 0. + out.append(z) + return out +def main(): + ap=argparse.ArgumentParser(description='V13 frozen-V9 axis symmetry, intervention equivariance, and forced-collision audit') + ap.add_argument('--epochs',type=int,default=3); ap.add_argument('--batch-size',type=int,default=64); ap.add_argument('--seeds',type=int,nargs='+',default=list(range(10))); ap.add_argument('--pairs',type=int,default=64); ap.add_argument('--horizon',type=int,default=30); ap.add_argument('--collision-horizon',type=int,default=15); ap.add_argument('--device',default='auto'); ap.add_argument('--models',nargs='+',default=DEFAULT_MODELS); a=ap.parse_args() + root=Path(__file__).resolve().parents[1]; out=root/'results/v13_axis_equivariance'; out.mkdir(parents=True,exist_ok=True); data=root/'data/v9_train_family.npz' + if not data.exists(): generate_family_npz(data) + device=('cuda' if torch.cuda.is_available() else 'cpu') if a.device=='auto' else a.device; dev=torch.device(device); axis=[]; coll=[] + for seed in a.seeds: + for m in a.models: + print(f'V13 seed={seed} model={m}'); sd=out/f'seed_{seed}'; r=train_one_model(m,data,sd,epochs=a.epochs,batch_size=a.batch_size,seed=seed,device=device,cf_train_horizon=10,cf_pairs=48,cf_every=4,lambda_cf_dir=.02,lambda_cf_mag=.005,lambda_cf_branch=.08) + ck=torch.load(sd/f'{m}.pt',map_location=dev); model=build_model(m).to(dev); model.load_state_dict(ck['model']) + for rn,cfg in REGIMES.items(): + rr=audit_axis_equivariance(model,dev,cfg,rn,n=a.pairs,horizon=a.horizon,seed=13101); cc=audit_forced_collisions(model,dev,cfg,rn,n=a.pairs,horizon=a.collision_horizon,seed=13201) + for x in rr:x.update(seed=seed,model=m,params=r['params']) + for x in cc:x.update(seed=seed,model=m,params=r['params']) + axis.extend(rr); coll.extend(cc) + writecsv(out/'axis_equivariance_by_episode.csv',axis); writecsv(out/'forced_collision_by_episode.csv',coll) + axsum=summarize(axis,['model','regime','program','transform'],['base_cosine','transformed_cosine','cosine_drop','base_magnitude_ratio','transformed_magnitude_ratio','equivariance_effect_cosine','equivariance_relative_error','oracle_equivariance_error']) + csum=summarize(coll,['model','regime','wall','program'],['trajectory_cosine','magnitude_ratio','boundary_contact']) + writecsv(out/'axis_equivariance_summary.csv',axsum); writecsv(out/'forced_collision_summary.csv',csum) + with (out/'run_config.json').open('w') as f: json.dump(vars(a)|{'release':'V13 Axis Symmetry & Intervention Equivariance Audit','learning_changes':'none; V9 recipe frozen','device_resolved':device},f,indent=2) + print('wrote',out) +if __name__=='__main__':main() diff --git a/scripts/run_v14_external_benchmark.py b/scripts/run_v14_external_benchmark.py new file mode 100644 index 0000000000000000000000000000000000000000..4d2e7e2b06ed6ed705ab091dd7917deee1200ef3 --- /dev/null +++ b/scripts/run_v14_external_benchmark.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Train frozen architecture families from scratch on external Gymnasium controls.""" +from __future__ import annotations +import argparse,csv,json,sys +from pathlib import Path +import numpy as np, torch +import torch.nn.functional as F +from torch.utils.data import DataLoader +sys.path.insert(0,str(Path(__file__).resolve().parents[1])) +from spectral_world_models.models import build_model +from spectral_world_models.v14_external_control import ExternalConfig,collect_external_npz,ExternalTransitionDataset,ExternalSequenceDataset,load_meta,SUPPORTED_ENVS +from spectral_world_models.metrics import count_parameters + +MODELS=("neural_operator","swm_structured_cf_v9","no_spectral_transition") + +def loss_fn(model,b,vocab_size): + o=model(b['image'],b['tokens'],b['action']) + with torch.no_grad(): tz=model.encode(b['next_image'],b['next_tokens']) + im=F.mse_loss(o['image_pred'],b['next_image']); lat=F.mse_loss(o['z_pred'],tz) + txt=F.cross_entropy(o['text_logits'].reshape(-1,vocab_size),b['next_tokens'].reshape(-1),ignore_index=0) + return im+.15*lat+.05*txt+.01*model.stability_penalty() + +def rollout_metrics(model,loader,device,horizons): + sums={h:[] for h in horizons} + model.eval() + with torch.no_grad(): + for b in loader: + imgs=b['images'].to(device); toks=b['tokens'].to(device); acts=b['actions'].to(device); z=model.encode(imgs[:,0],toks[:,0]) + step=[] + for t in range(acts.shape[1]): + z=model.transition(z,acts[:,t]); p=model.image_decoder(z); step.append(((p-imgs[:,t+1])**2).flatten(1).mean(1).cpu()) + for h in horizons: + hh=min(h,len(step)); sums[h].append(torch.stack(step[:hh],1).mean(1)) + return {f'rollout_mse_h{h}':float(torch.cat(sums[h]).mean()) for h in horizons} + +def train(env_id,model_name,seed,args,out): + data=out/'data'/f'{env_id}.npz' + if not data.exists(): + collect_external_npz(data,ExternalConfig(env_id=env_id,seq_len=max(args.horizons)+1,train_episodes=args.train_episodes,val_episodes=args.val_episodes,test_episodes=args.test_episodes,seed=args.data_seed)) + meta=load_meta(data); device=torch.device(args.device) + torch.manual_seed(seed) + model=build_model(model_name,image_size=32,vocab_size=meta['vocab_size'],max_text_len=8,action_dim=meta['action_dim']).to(device) + tr=DataLoader(ExternalTransitionDataset(data,'train'),batch_size=args.batch_size,shuffle=True); te=DataLoader(ExternalSequenceDataset(data,'test'),batch_size=args.batch_size) + opt=torch.optim.AdamW(model.parameters(),lr=args.lr) + model.train() + for _ in range(args.epochs): + for b in tr: + b={k:v.to(device) for k,v in b.items()}; opt.zero_grad(set_to_none=True); L=loss_fn(model,b,meta['vocab_size']); L.backward(); torch.nn.utils.clip_grad_norm_(model.parameters(),1.0); opt.step() + m=rollout_metrics(model,te,device,args.horizons) + m.update(env_id=env_id,model=model_name,seed=seed,params=count_parameters(model)) + ck=out/'checkpoints'/env_id; ck.mkdir(parents=True,exist_ok=True); torch.save({'model':model.state_dict(),'meta':meta,'model_name':model_name,'seed':seed},ck/f'{model_name}_seed{seed}.pt') + return m + +def main(): + p=argparse.ArgumentParser(); p.add_argument('--envs',nargs='+',default=list(SUPPORTED_ENVS)); p.add_argument('--models',nargs='+',default=list(MODELS)); p.add_argument('--seeds',nargs='+',type=int,default=[0,1,2,3,4]); p.add_argument('--horizons',nargs='+',type=int,default=[5,10,20,30]); p.add_argument('--epochs',type=int,default=5); p.add_argument('--batch-size',type=int,default=64); p.add_argument('--lr',type=float,default=2e-3); p.add_argument('--train-episodes',type=int,default=256); p.add_argument('--val-episodes',type=int,default=64); p.add_argument('--test-episodes',type=int,default=64); p.add_argument('--data-seed',type=int,default=1400); p.add_argument('--device',default='cpu'); p.add_argument('--out',default='results/v14_external_control'); a=p.parse_args() + out=Path(a.out); out.mkdir(parents=True,exist_ok=True); rows=[] + for e in a.envs: + for m in a.models: + for s in a.seeds: + print(f'[V14] {e} {m} seed={s}',flush=True); rows.append(train(e,m,s,a,out)) + fields=list(rows[0]); + with open(out/'metrics_by_seed.csv','w',newline='') as f: w=csv.DictWriter(f,fieldnames=fields); w.writeheader(); w.writerows(rows) + # aggregate mean/std by env/model + summary=[] + for e in a.envs: + for m in a.models: + rr=[r for r in rows if r['env_id']==e and r['model']==m]; row={'env_id':e,'model':m,'params':rr[0]['params'],'n_seeds':len(rr)} + for h in a.horizons: + vals=np.array([r[f'rollout_mse_h{h}'] for r in rr]); row[f'rollout_mse_h{h}_mean']=vals.mean(); row[f'rollout_mse_h{h}_std']=vals.std(ddof=1) if len(vals)>1 else 0. + summary.append(row) + sf=list(summary[0]); + with open(out/'metrics_summary.csv','w',newline='') as f: w=csv.DictWriter(f,fieldnames=sf); w.writeheader(); w.writerows(summary) + (out/'run_config.json').write_text(json.dumps(vars(a),indent=2)); print(out/'metrics_summary.csv') +if __name__=='__main__': main() diff --git a/scripts/run_v15_external_diagnostics.py b/scripts/run_v15_external_diagnostics.py new file mode 100644 index 0000000000000000000000000000000000000000..1d49b7732fdec1be8c939aaa8b313c4cd76ddf90 --- /dev/null +++ b/scripts/run_v15_external_diagnostics.py @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +"""V15.1: bug-fixed diagnostic-only evaluation of frozen V14 checkpoints. No retraining. + +Fixes V15 token/state diagnostic shape handling by using WorldModel.decode_text(), +which reshapes the flat decoder output to [batch, token_position, vocabulary]. +""" +from __future__ import annotations +import argparse,csv,json,sys +from pathlib import Path +import numpy as np, torch +import torch.nn.functional as F +sys.path.insert(0,str(Path(__file__).resolve().parents[1])) +from spectral_world_models.models import build_model +from spectral_world_models.v14_external_control import ExternalSequenceDataset,load_meta,SUPPORTED_ENVS +from torch.utils.data import DataLoader + +MODELS=("neural_operator","swm_structured_cf_v9","no_spectral_transition") + +def ssim_simple(x,y): + # global SSIM per image, sufficient as a dependency-free diagnostic + mu_x=x.flatten(1).mean(1); mu_y=y.flatten(1).mean(1) + vx=((x.flatten(1)-mu_x[:,None])**2).mean(1); vy=((y.flatten(1)-mu_y[:,None])**2).mean(1) + cov=((x.flatten(1)-mu_x[:,None])*(y.flatten(1)-mu_y[:,None])).mean(1) + return ((2*mu_x*mu_y+1e-4)*(2*cov+9e-4)/((mu_x**2+mu_y**2+1e-4)*(vx+vy+9e-4))).mean().item() + +def diag(model,loader,device,horizons): + acc={h:{k:[] for k in ['mse','static_mse','fg_mse','motion_mse','psnr','ssim','latent_cos','token_acc']} for h in horizons} + model.eval() + with torch.no_grad(): + for b in loader: + imgs=b['images'].to(device); toks=b['tokens'].to(device); acts=b['actions'].to(device) + z0=model.encode(imgs[:,0],toks[:,0]); z=z0; prev_pred=imgs[:,0] + preds=[]; zs=[] + for t in range(acts.shape[1]): + z=model.transition(z,acts[:,t]); preds.append(model.image_decoder(z)); zs.append(z) + for h in horizons: + t=min(h,len(preds))-1; pred=preds[t]; gt=imgs[:,t+1]; base=imgs[:,0] + err=(pred-gt)**2; mse=err.flatten(1).mean(1) + static=((base-gt)**2).flatten(1).mean(1) + mask=(gt-base).abs()>0.03; fg=(err*mask).flatten(1).sum(1)/mask.flatten(1).sum(1).clamp_min(1) + if t>0: + pm=pred-preds[t-1]; gm=gt-imgs[:,t] + else: pm=pred-base; gm=gt-base + motion=((pm-gm)**2).flatten(1).mean(1) + truez=model.encode(gt,toks[:,t+1]); cos=F.cosine_similarity(zs[t],truez,dim=1) + logits=model.decode_text(zs[t]) + target=toks[:,t+1] + if logits.ndim != 3 or logits.shape[:2] != target.shape: + raise RuntimeError(f'Incompatible state-token shapes: logits={tuple(logits.shape)}, target={tuple(target.shape)}. Expected [B,L,V] versus [B,L].') + predtok=logits.argmax(-1); valid=target!=0 + tok=((predtok==target)&valid).sum(1)/valid.sum(1).clamp_min(1) + acc[h]['mse'] += mse.cpu().tolist(); acc[h]['static_mse'] += static.cpu().tolist(); acc[h]['fg_mse'] += fg.cpu().tolist(); acc[h]['motion_mse'] += motion.cpu().tolist(); acc[h]['psnr'] += (-10*torch.log10(mse.clamp_min(1e-10))).cpu().tolist(); acc[h]['ssim'].append(ssim_simple(pred,gt)); acc[h]['latent_cos'] += cos.cpu().tolist(); acc[h]['token_acc'] += tok.cpu().tolist() + out={} + for h,d in acc.items(): + for k,v in d.items(): out[f'{k}_h{h}']=float(np.mean(v)) + return out + +def spectral_diag(model): + d={} + try: + sd=model.stability_diagnostics() + for k,v in sd.items(): d['spectral_'+k]=float(v.detach().cpu() if torch.is_tensor(v) else v) + except Exception: pass + return d + +def main(): + p=argparse.ArgumentParser(description='V15.1 bug-fixed external dynamics diagnostic; evaluates frozen V14 checkpoints without retraining.'); p.add_argument('--v14-results',default='results/v14_external_control'); p.add_argument('--envs',nargs='+',default=list(SUPPORTED_ENVS)); p.add_argument('--models',nargs='+',default=list(MODELS)); p.add_argument('--seeds',nargs='+',type=int,default=list(range(10))); p.add_argument('--horizons',nargs='+',type=int,default=[5,10,20,30]); p.add_argument('--batch-size',type=int,default=64); p.add_argument('--device',default='cpu'); p.add_argument('--out',default='results/v15_external_diagnostics'); a=p.parse_args() + root=Path(a.v14_results); out=Path(a.out); out.mkdir(parents=True,exist_ok=True); rows=[] + for e in a.envs: + data=root/'data'/f'{e}.npz'; meta=load_meta(data); loader=DataLoader(ExternalSequenceDataset(data,'test'),batch_size=a.batch_size) + for m in a.models: + for s in a.seeds: + ck=root/'checkpoints'/e/f'{m}_seed{s}.pt'; print('[V15]',e,m,s,flush=True) + model=build_model(m,image_size=32,vocab_size=meta['vocab_size'],max_text_len=8,action_dim=meta['action_dim']).to(a.device); model.load_state_dict(torch.load(ck,map_location=a.device)['model']) + r={'env_id':e,'model':m,'seed':s}; r.update(diag(model,loader,torch.device(a.device),a.horizons)); r.update(spectral_diag(model)); rows.append(r) + fields=sorted(set().union(*(r.keys() for r in rows))); + with open(out/'diagnostics_by_seed.csv','w',newline='') as f: w=csv.DictWriter(f,fieldnames=fields); w.writeheader(); w.writerows(rows) + summary=[] + for e in a.envs: + for m in a.models: + rr=[r for r in rows if r['env_id']==e and r['model']==m]; q={'env_id':e,'model':m,'n_seeds':len(rr)} + for k in fields: + if k in ('env_id','model','seed') or not all(k in x for x in rr): continue + vals=np.array([x[k] for x in rr],float); q[k+'_mean']=vals.mean(); q[k+'_std']=vals.std(ddof=1) if len(vals)>1 else 0 + summary.append(q) + sf=sorted(set().union(*(r.keys() for r in summary))); + with open(out/'diagnostics_summary.csv','w',newline='') as f: w=csv.DictWriter(f,fieldnames=sf); w.writeheader(); w.writerows(summary) + (out/'run_config.json').write_text(json.dumps(vars(a),indent=2)); print(out/'diagnostics_summary.csv') +if __name__=='__main__': main() diff --git a/scripts/run_v5_benchmark.py b/scripts/run_v5_benchmark.py new file mode 100644 index 0000000000000000000000000000000000000000..3edcb2231b7a4d876e685379d93d267304c9fb42 --- /dev/null +++ b/scripts/run_v5_benchmark.py @@ -0,0 +1,72 @@ +"""Targeted attribution benchmark for SWM transition stability.""" +from __future__ import annotations +import argparse, csv, json +from pathlib import Path +from statistics import mean, stdev +import torch +from spectral_world_models.hard_dynamics import HardDynamicsConfig, generate_hard_benchmark_npz +from spectral_world_models.train import train_one_model + +DEFAULT_MODELS=["koopman","neural_operator","swm","swm_selective","swm_spectral_norm","no_spectral_transition","no_stability_penalty"] +BASE=[("test_psnr",lambda r:r["test"]["psnr"]),("test_ssim",lambda r:r["test"]["ssim"]),("test_nll",lambda r:r["test"]["nll"]),("test_mse",lambda r:r["test"]["mse"]),("test_text_acc",lambda r:r["test"]["text_acc"])] +def stats(xs): return mean(xs), stdev(xs) if len(xs)>1 else 0.0 + +def main(): + ap=argparse.ArgumentParser(description="SWM v5 hard controlled-dynamics benchmark.") + ap.add_argument("--data",default=None); ap.add_argument("--rollout-data",default=None); ap.add_argument("--out",default=None) + ap.add_argument("--epochs",type=int,default=3); ap.add_argument("--batch-size",type=int,default=64) + ap.add_argument("--seeds",type=int,nargs="+",default=[0,1,2,3,4]); ap.add_argument("--horizons",type=int,nargs="+",default=[5,10,20,30,50,100]) + ap.add_argument("--device",default="auto"); ap.add_argument("--models",nargs="+",default=DEFAULT_MODELS); ap.add_argument("--lambda-stability",type=float,default=0.01) + a=ap.parse_args(); root=Path(__file__).resolve().parents[1] + data=Path(a.data) if a.data else root/"data"/"hard_controlled_dynamics_v5.npz" + out=Path(a.out) if a.out else root/"results"/"v5_hard_dynamics" + rollout=Path(a.rollout_data) if a.rollout_data else root/f"data/hard_controlled_dynamics_v5_rollout_h{max(a.horizons)+1}.npz" + if not data.exists(): generate_hard_benchmark_npz(data,HardDynamicsConfig()) + need=max(a.horizons)+1; regen=True + if rollout.exists(): + try: + import numpy as np + with np.load(rollout,allow_pickle=True) as d: regen=d["test_images"].shape[1] < need + except Exception: pass + if regen: generate_hard_benchmark_npz(rollout,HardDynamicsConfig(seq_len=need,train_sequences=1,val_sequences=1,test_sequences=96,seed=7017)) + device=("cuda" if torch.cuda.is_available() else "cpu") if a.device=="auto" else a.device + if device.startswith("cuda") and not torch.cuda.is_available(): raise RuntimeError("CUDA requested but unavailable") + out.mkdir(parents=True,exist_ok=True); grouped={m:[] for m in a.models}; raw=[] + for seed in a.seeds: + for m in a.models: + print(f"\n=== seed={seed} model={m} horizons={a.horizons} ===") + r=train_one_model(m,data,out/f"seed_{seed}",epochs=a.epochs,batch_size=a.batch_size,seed=seed,device=device,rollout_data_path=rollout,rollout_horizons=tuple(a.horizons),lambda_stability=a.lambda_stability) + grouped[m].append(r); row={"seed":seed,"model":m,"params":r["params"]} + for k,g in BASE: row[k]=g(r) + for h in a.horizons: + rr=r["rollout_by_horizon"][str(h)] + for metric in ["rollout_mse","rollout_text_acc","rollout_latent_cosine","rollout_joint_score"]: row[f"{metric}_h{h}"]=rr[metric] + for k,v in r.get("stability_diagnostics",{}).items(): row[k]=v + raw.append(row) + fields=[] + for r in raw: + for k in r: + if k not in fields: fields.append(k) + with (out/"metrics_by_seed.csv").open("w",newline="") as f: + w=csv.DictWriter(f,fieldnames=fields); w.writeheader(); w.writerows(raw) + summary=[] + for m,runs in grouped.items(): + row={"model":m,"params":runs[0]["params"],"n_seeds":len(runs)} + for k,g in BASE: + mu,sd=stats([float(g(r)) for r in runs]); row[k+"_mean"]=mu; row[k+"_std"]=sd + for h in a.horizons: + for metric in ["rollout_mse","rollout_text_acc","rollout_latent_cosine","rollout_joint_score"]: + vals=[float(r["rollout_by_horizon"][str(h)][metric]) for r in runs]; mu,sd=stats(vals); row[f"{metric}_h{h}_mean"]=mu; row[f"{metric}_h{h}_std"]=sd + diag_keys=set().union(*(r.get("stability_diagnostics",{}).keys() for r in runs)) + for k in sorted(diag_keys): + vals=[float(r["stability_diagnostics"][k]) for r in runs if k in r.get("stability_diagnostics",{})]; mu,sd=stats(vals); row[k+"_mean"]=mu; row[k+"_std"]=sd + summary.append(row) + fields=[] + for r in summary: + for k in r: + if k not in fields: fields.append(k) + with (out/"metrics_summary.csv").open("w",newline="") as f: + w=csv.DictWriter(f,fieldnames=fields); w.writeheader(); w.writerows(summary) + with (out/"run_config.json").open("w") as f: json.dump(vars(a)|{"device_resolved":device,"rollout_data_resolved":str(rollout)},f,indent=2) + print("\nWrote",out/"metrics_summary.csv") +if __name__=="__main__": main() diff --git a/scripts/run_v6_benchmark.py b/scripts/run_v6_benchmark.py new file mode 100644 index 0000000000000000000000000000000000000000..6abfe3eeb766b6ba0175dd72558c64c10b273896 --- /dev/null +++ b/scripts/run_v6_benchmark.py @@ -0,0 +1,50 @@ +"""V6 structured spectral transport + counterfactual action benchmark.""" +from __future__ import annotations +import argparse,csv,json +from pathlib import Path +from statistics import mean,stdev +import torch, numpy as np +from spectral_world_models.hard_dynamics import HardDynamicsConfig,generate_hard_benchmark_npz +from spectral_world_models.train import train_one_model +from spectral_world_models.counterfactual import evaluate_counterfactual +from spectral_world_models.models import build_model +DEFAULT_MODELS=['neural_operator','swm','swm_selective','swm_structured','no_spectral_transition','no_stability_penalty'] +def stats(x): return mean(x),stdev(x) if len(x)>1 else 0. +def main(): + ap=argparse.ArgumentParser(); ap.add_argument('--epochs',type=int,default=3); ap.add_argument('--batch-size',type=int,default=64); ap.add_argument('--seeds',type=int,nargs='+',default=[0,1,2,3,4]); ap.add_argument('--horizons',type=int,nargs='+',default=[5,10,20,30,50,100]); ap.add_argument('--cf-horizon',type=int,default=30); ap.add_argument('--device',default='auto'); ap.add_argument('--models',nargs='+',default=DEFAULT_MODELS); a=ap.parse_args() + root=Path(__file__).resolve().parents[1]; data=root/'data/hard_controlled_dynamics_v6.npz'; rollout=root/f'data/hard_controlled_dynamics_v6_rollout_h{max(a.horizons)+1}.npz'; out=root/'results/v6_structured_transport'; out.mkdir(parents=True,exist_ok=True) + if not data.exists(): generate_hard_benchmark_npz(data,HardDynamicsConfig()) + if not rollout.exists(): generate_hard_benchmark_npz(rollout,HardDynamicsConfig(seq_len=max(a.horizons)+1,train_sequences=1,val_sequences=1,test_sequences=96,seed=7017)) + device=('cuda' if torch.cuda.is_available() else 'cpu') if a.device=='auto' else a.device; grouped={m:[] for m in a.models}; raw=[] + for seed in a.seeds: + for m in a.models: + print(f'\n=== seed={seed} model={m} ==='); sd=out/f'seed_{seed}'; r=train_one_model(m,data,sd,epochs=a.epochs,batch_size=a.batch_size,seed=seed,device=device,rollout_data_path=rollout,rollout_horizons=tuple(a.horizons)) + ck=torch.load(sd/f'{m}.pt',map_location=device); model=build_model(m).to(device); model.load_state_dict(ck['model']); cf=evaluate_counterfactual(model,torch.device(device),horizon=a.cf_horizon,seed=606+seed); r['counterfactual']=cf; grouped[m].append(r) + row={'seed':seed,'model':m,'params':r['params'],'test_mse':r['test']['mse'],'test_text_acc':r['test']['text_acc'],**cf} + for h in a.horizons: + rr=r['rollout_by_horizon'][str(h)]; row[f'rollout_mse_h{h}']=rr['rollout_mse']; row[f'rollout_text_acc_h{h}']=rr['rollout_text_acc']; row[f'rollout_latent_cosine_h{h}']=rr['rollout_latent_cosine'] + row.update(r.get('stability_diagnostics',{})); raw.append(row) + fields=[] + for r in raw: + for k in r: + if k not in fields: fields.append(k) + with (out/'metrics_by_seed.csv').open('w',newline='') as f: w=csv.DictWriter(f,fieldnames=fields);w.writeheader();w.writerows(raw) + summary=[] + for m,runs in grouped.items(): + row={'model':m,'params':runs[0]['params'],'n_seeds':len(runs)} + getters={'test_mse':lambda r:r['test']['mse'],'test_text_acc':lambda r:r['test']['text_acc'],'cf_final_image_mse':lambda r:r['counterfactual']['cf_final_image_mse'],'cf_effect_vector_cosine':lambda r:r['counterfactual']['cf_effect_vector_cosine'],'cf_effect_magnitude_ratio':lambda r:r['counterfactual']['cf_effect_magnitude_ratio']} + for h in a.horizons: + for q in ['rollout_mse','rollout_text_acc','rollout_latent_cosine']: getters[f'{q}_h{h}']=lambda r,q=q,h=h:r['rollout_by_horizon'][str(h)][q] + for k,g in getters.items(): mu,sd=stats([float(g(r)) for r in runs]);row[k+'_mean']=mu;row[k+'_std']=sd + dkeys=set().union(*(r.get('stability_diagnostics',{}).keys() for r in runs)) + for k in dkeys: + vals=[float(r['stability_diagnostics'][k]) for r in runs if k in r.get('stability_diagnostics',{})];mu,sd=stats(vals);row[k+'_mean']=mu;row[k+'_std']=sd + summary.append(row) + fields=[] + for r in summary: + for k in r: + if k not in fields: fields.append(k) + with (out/'metrics_summary.csv').open('w',newline='') as f:w=csv.DictWriter(f,fieldnames=fields);w.writeheader();w.writerows(summary) + with (out/'run_config.json').open('w') as f:json.dump(vars(a)|{'device_resolved':device},f,indent=2) + print('\nWrote',out/'metrics_summary.csv') +if __name__=='__main__':main() diff --git a/scripts/run_v7_benchmark.py b/scripts/run_v7_benchmark.py new file mode 100644 index 0000000000000000000000000000000000000000..480f5b7d1bdcef49f834755c28e3072b2f55f3dd --- /dev/null +++ b/scripts/run_v7_benchmark.py @@ -0,0 +1,50 @@ +"""V7 counterfactual-trained structured spectral transport benchmark.""" +from __future__ import annotations +import argparse,csv,json +from pathlib import Path +from statistics import mean,stdev +import torch, numpy as np +from spectral_world_models.hard_dynamics import HardDynamicsConfig,generate_hard_benchmark_npz +from spectral_world_models.train import train_one_model +from spectral_world_models.counterfactual import evaluate_counterfactual +from spectral_world_models.models import build_model +DEFAULT_MODELS=['neural_operator','swm_selective','swm_structured','swm_structured_cf','no_spectral_transition'] +def stats(x): return mean(x),stdev(x) if len(x)>1 else 0. +def main(): + ap=argparse.ArgumentParser(); ap.add_argument('--epochs',type=int,default=3); ap.add_argument('--batch-size',type=int,default=64); ap.add_argument('--seeds',type=int,nargs='+',default=[0,1,2,3,4]); ap.add_argument('--horizons',type=int,nargs='+',default=[5,10,20,30,50,100]); ap.add_argument('--cf-horizon',type=int,default=30); ap.add_argument('--device',default='auto'); ap.add_argument('--cf-train-horizon',type=int,default=8); ap.add_argument('--cf-pairs',type=int,default=64); ap.add_argument('--cf-every',type=int,default=4); ap.add_argument('--lambda-cf-dir',type=float,default=0.02); ap.add_argument('--lambda-cf-mag',type=float,default=0.005); ap.add_argument('--lambda-cf-branch',type=float,default=0.10); ap.add_argument('--models',nargs='+',default=DEFAULT_MODELS); a=ap.parse_args() + root=Path(__file__).resolve().parents[1]; data=root/'data/hard_controlled_dynamics_v6.npz'; rollout=root/f'data/hard_controlled_dynamics_v6_rollout_h{max(a.horizons)+1}.npz'; out=root/'results/v7_counterfactual_training'; out.mkdir(parents=True,exist_ok=True) + if not data.exists(): generate_hard_benchmark_npz(data,HardDynamicsConfig()) + if not rollout.exists(): generate_hard_benchmark_npz(rollout,HardDynamicsConfig(seq_len=max(a.horizons)+1,train_sequences=1,val_sequences=1,test_sequences=96,seed=7017)) + device=('cuda' if torch.cuda.is_available() else 'cpu') if a.device=='auto' else a.device; grouped={m:[] for m in a.models}; raw=[] + for seed in a.seeds: + for m in a.models: + print(f'\n=== seed={seed} model={m} ==='); sd=out/f'seed_{seed}'; r=train_one_model(m,data,sd,epochs=a.epochs,batch_size=a.batch_size,seed=seed,device=device,rollout_data_path=rollout,rollout_horizons=tuple(a.horizons),cf_train_horizon=a.cf_train_horizon,cf_pairs=a.cf_pairs,cf_every=a.cf_every,lambda_cf_dir=a.lambda_cf_dir,lambda_cf_mag=a.lambda_cf_mag,lambda_cf_branch=a.lambda_cf_branch) + ck=torch.load(sd/f'{m}.pt',map_location=device); model=build_model(m).to(device); model.load_state_dict(ck['model']); cf=evaluate_counterfactual(model,torch.device(device),horizon=a.cf_horizon,seed=606+seed); r['counterfactual']=cf; grouped[m].append(r) + row={'seed':seed,'model':m,'params':r['params'],'test_mse':r['test']['mse'],'test_text_acc':r['test']['text_acc'],**cf} + for h in a.horizons: + rr=r['rollout_by_horizon'][str(h)]; row[f'rollout_mse_h{h}']=rr['rollout_mse']; row[f'rollout_text_acc_h{h}']=rr['rollout_text_acc']; row[f'rollout_latent_cosine_h{h}']=rr['rollout_latent_cosine'] + row.update(r.get('stability_diagnostics',{})); raw.append(row) + fields=[] + for r in raw: + for k in r: + if k not in fields: fields.append(k) + with (out/'metrics_by_seed.csv').open('w',newline='') as f: w=csv.DictWriter(f,fieldnames=fields);w.writeheader();w.writerows(raw) + summary=[] + for m,runs in grouped.items(): + row={'model':m,'params':runs[0]['params'],'n_seeds':len(runs)} + getters={'test_mse':lambda r:r['test']['mse'],'test_text_acc':lambda r:r['test']['text_acc'],'cf_final_image_mse':lambda r:r['counterfactual']['cf_final_image_mse'],'cf_effect_vector_cosine':lambda r:r['counterfactual']['cf_effect_vector_cosine'],'cf_effect_magnitude_ratio':lambda r:r['counterfactual']['cf_effect_magnitude_ratio']} + for h in a.horizons: + for q in ['rollout_mse','rollout_text_acc','rollout_latent_cosine']: getters[f'{q}_h{h}']=lambda r,q=q,h=h:r['rollout_by_horizon'][str(h)][q] + for k,g in getters.items(): mu,sd=stats([float(g(r)) for r in runs]);row[k+'_mean']=mu;row[k+'_std']=sd + dkeys=set().union(*(r.get('stability_diagnostics',{}).keys() for r in runs)) + for k in dkeys: + vals=[float(r['stability_diagnostics'][k]) for r in runs if k in r.get('stability_diagnostics',{})];mu,sd=stats(vals);row[k+'_mean']=mu;row[k+'_std']=sd + summary.append(row) + fields=[] + for r in summary: + for k in r: + if k not in fields: fields.append(k) + with (out/'metrics_summary.csv').open('w',newline='') as f:w=csv.DictWriter(f,fieldnames=fields);w.writeheader();w.writerows(summary) + with (out/'run_config.json').open('w') as f:json.dump(vars(a)|{'device_resolved':device},f,indent=2) + print('\nWrote',out/'metrics_summary.csv') +if __name__=='__main__':main() diff --git a/scripts/run_v8_benchmark.py b/scripts/run_v8_benchmark.py new file mode 100644 index 0000000000000000000000000000000000000000..12bd04f162f48bb539661313171b6da1ade24807 --- /dev/null +++ b/scripts/run_v8_benchmark.py @@ -0,0 +1,60 @@ +"""V8 frozen-architecture OOD generalization + causal-control + planning benchmark.""" +from __future__ import annotations +import argparse,csv,json +from pathlib import Path +from statistics import mean,stdev +import torch +from torch.utils.data import DataLoader +from spectral_world_models.hard_dynamics import HardDynamicsConfig,generate_hard_benchmark_npz +from spectral_world_models.data import TransitionDataset,SequenceDataset +from spectral_world_models.train import train_one_model,evaluate_transitions,evaluate_rollout +from spectral_world_models.counterfactual import evaluate_counterfactual +from spectral_world_models.models import build_model +from spectral_world_models.v8_generalization import OOD_CONFIG,evaluate_counterfactual_cfg,evaluate_planning +DEFAULT_MODELS=['neural_operator','swm_selective','swm_structured_cf','no_spectral_transition'] +def stats(x): return mean(x),stdev(x) if len(x)>1 else 0. +def main(): + ap=argparse.ArgumentParser(description='V8: freeze V7 architectures; test ID/OOD generalization, intervention, and planning.') + ap.add_argument('--epochs',type=int,default=3); ap.add_argument('--batch-size',type=int,default=64); ap.add_argument('--seeds',type=int,nargs='+',default=[0,1,2,3,4]); ap.add_argument('--horizons',type=int,nargs='+',default=[5,10,20,30,50,100]); ap.add_argument('--cf-horizon',type=int,default=30); ap.add_argument('--planning-horizon',type=int,default=20); ap.add_argument('--planning-episodes',type=int,default=96); ap.add_argument('--device',default='auto'); ap.add_argument('--models',nargs='+',default=DEFAULT_MODELS) + ap.add_argument('--cf-train-horizon',type=int,default=8); ap.add_argument('--cf-pairs',type=int,default=64); ap.add_argument('--cf-every',type=int,default=4); ap.add_argument('--lambda-cf-dir',type=float,default=.02); ap.add_argument('--lambda-cf-mag',type=float,default=.005); ap.add_argument('--lambda-cf-branch',type=float,default=.10); a=ap.parse_args() + root=Path(__file__).resolve().parents[1]; train_data=root/'data/hard_controlled_dynamics_v6.npz'; id_roll=root/f'data/v8_id_rollout_h{max(a.horizons)+1}.npz'; ood=root/f'data/v8_ood_rollout_h{max(a.horizons)+1}.npz'; out=root/'results/v8_generalization'; out.mkdir(parents=True,exist_ok=True) + if not train_data.exists(): generate_hard_benchmark_npz(train_data,HardDynamicsConfig()) + if not id_roll.exists(): generate_hard_benchmark_npz(id_roll,HardDynamicsConfig(seq_len=max(a.horizons)+1,train_sequences=1,val_sequences=1,test_sequences=96,seed=8016)) + oc=OOD_CONFIG; oc.seq_len=max(a.horizons)+1 + if not ood.exists(): generate_hard_benchmark_npz(ood,oc) + device=('cuda' if torch.cuda.is_available() else 'cpu') if a.device=='auto' else a.device; raw=[] + for seed in a.seeds: + for m in a.models: + print(f'\n=== V8 seed={seed} model={m} ==='); sd=out/f'seed_{seed}' + r=train_one_model(m,train_data,sd,epochs=a.epochs,batch_size=a.batch_size,seed=seed,device=device,rollout_data_path=id_roll,rollout_horizons=tuple(a.horizons),cf_train_horizon=a.cf_train_horizon,cf_pairs=a.cf_pairs,cf_every=a.cf_every,lambda_cf_dir=a.lambda_cf_dir,lambda_cf_mag=a.lambda_cf_mag,lambda_cf_branch=a.lambda_cf_branch) + ck=torch.load(sd/f'{m}.pt',map_location=device); model=build_model(m).to(device); model.load_state_dict(ck['model']) + ood_t=evaluate_transitions(model,DataLoader(TransitionDataset(ood,'test'),batch_size=a.batch_size),torch.device(device)); ood_s=DataLoader(SequenceDataset(ood,'test'),batch_size=a.batch_size) + row={'seed':seed,'model':m,'params':r['params'],'id_test_mse':r['test']['mse'],'id_test_text_acc':r['test']['text_acc'],'ood_test_mse':ood_t['mse'],'ood_test_text_acc':ood_t['text_acc']} + idcf=evaluate_counterfactual(model,torch.device(device),horizon=a.cf_horizon,seed=8606+seed); row.update({'id_'+k:v for k,v in idcf.items()}) + row.update(evaluate_counterfactual_cfg(model,torch.device(device),horizon=a.cf_horizon,seed=8809+seed,cfg=oc)) + for h in a.horizons: + rr=r['rollout_by_horizon'][str(h)]; oo=evaluate_rollout(model,ood_s,torch.device(device),horizon=h) + for q in ['rollout_mse','rollout_text_acc','rollout_latent_cosine']: row[f'id_{q}_h{h}']=rr[q]; row[f'ood_{q}_h{h}']=oo[q] + for prefix,cfg in [('id',HardDynamicsConfig()),('ood',oc)]: + p=evaluate_planning(model,torch.device(device),n=a.planning_episodes,horizon=a.planning_horizon,seed=8900+seed,cfg=cfg); row.update({prefix+'_'+k:v for k,v in p.items()}) + row.update(r.get('stability_diagnostics',{})); raw.append(row) + fields=[] + for r in raw: + for k in r: + if k not in fields: fields.append(k) + with (out/'metrics_by_seed.csv').open('w',newline='') as f:w=csv.DictWriter(f,fieldnames=fields);w.writeheader();w.writerows(raw) + summary=[] + for m in a.models: + rs=[r for r in raw if r['model']==m]; row={'model':m,'params':rs[0]['params'],'n_seeds':len(rs)} + for k in fields: + if k in ('seed','model','params') or not all(k in r and isinstance(r[k],(int,float)) for r in rs): continue + mu,sd=stats([float(r[k]) for r in rs]); row[k+'_mean']=mu; row[k+'_std']=sd + summary.append(row) + sf=[] + for r in summary: + for k in r: + if k not in sf: sf.append(k) + with (out/'metrics_summary.csv').open('w',newline='') as f:w=csv.DictWriter(f,fieldnames=sf);w.writeheader();w.writerows(summary) + with (out/'run_config.json').open('w') as f:json.dump(vars(a)|{'device_resolved':device,'architecture_frozen_from':'v7','ood_config':oc.__dict__},f,indent=2) + print('\nWrote',out/'metrics_summary.csv') +if __name__=='__main__':main() diff --git a/scripts/run_v9_benchmark.py b/scripts/run_v9_benchmark.py new file mode 100644 index 0000000000000000000000000000000000000000..ba4b872e41aae21d49d7adaee2c8e5ed0da9bd8d --- /dev/null +++ b/scripts/run_v9_benchmark.py @@ -0,0 +1,61 @@ +"""V9: multi-environment trajectory-CF causal generalization benchmark.""" +from __future__ import annotations +import argparse,csv,json +from pathlib import Path +from statistics import mean,stdev +import torch +from torch.utils.data import DataLoader +from spectral_world_models.v9_causal_generalization import generate_family_npz,INTERP,EXTRAP,STRUCTURAL +from spectral_world_models.hard_dynamics import generate_hard_benchmark_npz +from spectral_world_models.data import TransitionDataset,SequenceDataset +from spectral_world_models.train import train_one_model,evaluate_transitions,evaluate_rollout +from spectral_world_models.v8_generalization import evaluate_counterfactual_cfg,evaluate_planning +from spectral_world_models.models import build_model +DEFAULT_MODELS=['neural_operator','swm_selective','swm_structured_cf','swm_structured_cf_v9','no_spectral_transition'] +def stats(x): return mean(x),stdev(x) if len(x)>1 else 0. +def main(): + ap=argparse.ArgumentParser(description='V9 causal generalization: environment-family training + trajectory counterfactual loss.') + ap.add_argument('--epochs',type=int,default=3); ap.add_argument('--batch-size',type=int,default=64); ap.add_argument('--seeds',type=int,nargs='+',default=[0,1,2,3,4]); ap.add_argument('--horizons',type=int,nargs='+',default=[5,10,20,30,50,100]); ap.add_argument('--cf-horizon',type=int,default=30); ap.add_argument('--planning-horizon',type=int,default=20); ap.add_argument('--planning-episodes',type=int,default=64); ap.add_argument('--device',default='auto'); ap.add_argument('--models',nargs='+',default=DEFAULT_MODELS); ap.add_argument('--cf-train-horizon',type=int,default=10); ap.add_argument('--cf-pairs',type=int,default=48); ap.add_argument('--cf-every',type=int,default=4); ap.add_argument('--lambda-cf-dir',type=float,default=.02); ap.add_argument('--lambda-cf-mag',type=float,default=.005); ap.add_argument('--lambda-cf-branch',type=float,default=.08); a=ap.parse_args() + root=Path(__file__).resolve().parents[1]; data=root/'data/v9_train_family.npz'; out=root/'results/v9_causal_generalization'; out.mkdir(parents=True,exist_ok=True); maxh=max(a.horizons) + if not data.exists(): generate_family_npz(data) + regimes={'interpolation':INTERP,'extrapolation':EXTRAP,'structural_ood':STRUCTURAL}; paths={} + for name,cfg in regimes.items(): + cfg.seq_len=maxh+1; p=root/f'data/v9_{name}_h{maxh+1}.npz'; paths[name]=p + if not p.exists(): generate_hard_benchmark_npz(p,cfg) + device=('cuda' if torch.cuda.is_available() else 'cpu') if a.device=='auto' else a.device; dev=torch.device(device); raw=[] + for seed in a.seeds: + for m in a.models: + print(f'\n=== V9 seed={seed} model={m} ==='); sd=out/f'seed_{seed}' + r=train_one_model(m,data,sd,epochs=a.epochs,batch_size=a.batch_size,seed=seed,device=device,rollout_data_path=paths['interpolation'],rollout_horizons=tuple(a.horizons),cf_train_horizon=a.cf_train_horizon,cf_pairs=a.cf_pairs,cf_every=a.cf_every,lambda_cf_dir=a.lambda_cf_dir,lambda_cf_mag=a.lambda_cf_mag,lambda_cf_branch=a.lambda_cf_branch) + ck=torch.load(sd/f'{m}.pt',map_location=dev); model=build_model(m).to(dev); model.load_state_dict(ck['model']); row={'seed':seed,'model':m,'params':r['params']} + for name,cfg in regimes.items(): + p=paths[name]; tm=evaluate_transitions(model,DataLoader(TransitionDataset(p,'test'),batch_size=a.batch_size),dev); row[f'{name}_test_mse']=tm['mse']; row[f'{name}_test_text_acc']=tm['text_acc'] + sl=DataLoader(SequenceDataset(p,'test'),batch_size=a.batch_size) + for h in a.horizons: + rr=evaluate_rollout(model,sl,dev,horizon=h) + for q in ('rollout_mse','rollout_text_acc','rollout_latent_cosine'): row[f'{name}_{q}_h{h}']=rr[q] + cf=evaluate_counterfactual_cfg(model,dev,horizon=a.cf_horizon,n=96,seed=9300+seed,cfg=cfg) + for k,v in cf.items(): row[f'{name}_{k.removeprefix("ood_")}']=v + pl=evaluate_planning(model,dev,n=a.planning_episodes,horizon=a.planning_horizon,seed=9400+seed,cfg=cfg) + for k,v in pl.items(): row[f'{name}_{k}']=v + row.update(r.get('stability_diagnostics',{})); raw.append(row) + fields=[] + for r in raw: + for k in r: + if k not in fields: fields.append(k) + with (out/'metrics_by_seed.csv').open('w',newline='') as f: w=csv.DictWriter(f,fieldnames=fields); w.writeheader(); w.writerows(raw) + summary=[] + for m in a.models: + rs=[r for r in raw if r['model']==m]; row={'model':m,'params':rs[0]['params'],'n_seeds':len(rs)} + for k in fields: + if k in ('seed','model','params') or not all(k in r and isinstance(r[k],(int,float)) for r in rs): continue + mu,sd=stats([float(r[k]) for r in rs]); row[k+'_mean']=mu; row[k+'_std']=sd + summary.append(row) + sf=[] + for r in summary: + for k in r: + if k not in sf: sf.append(k) + with (out/'metrics_summary.csv').open('w',newline='') as f: w=csv.DictWriter(f,fieldnames=sf); w.writeheader(); w.writerows(summary) + with (out/'run_config.json').open('w') as f: json.dump(vars(a)|{'device_resolved':device,'regimes':{k:v.__dict__ for k,v in regimes.items()}},f,indent=2) + print('\nWrote',out/'metrics_summary.csv') +if __name__=='__main__': main() diff --git a/scripts/system_check.py b/scripts/system_check.py new file mode 100644 index 0000000000000000000000000000000000000000..da42cb0592edcd37d51e42294d4a1429945ec5e4 --- /dev/null +++ b/scripts/system_check.py @@ -0,0 +1,7 @@ +import torch +from spectral_world_models.models import build_model +from spectral_world_models.metrics import count_parameters +m=build_model('swm') +print('torch:', torch.__version__) +print('cuda:', torch.cuda.is_available()) +print('parameters:', count_parameters(m)) diff --git a/spectral_world_models.egg-info/PKG-INFO b/spectral_world_models.egg-info/PKG-INFO new file mode 100644 index 0000000000000000000000000000000000000000..6839c8e438f560750c3ad7803c440542d7f44fcb --- /dev/null +++ b/spectral_world_models.egg-info/PKG-INFO @@ -0,0 +1,114 @@ +Metadata-Version: 2.4 +Name: spectral-world-models +Version: 0.1.0 +Summary: Spectral World Models: multimodal operator learning in Hilbert space for imagination-based planning +Author: Andrew J. Kiruluta +Requires-Python: >=3.10 +Description-Content-Type: text/markdown +License-File: LICENSE +Requires-Dist: torch>=2.0 +Requires-Dist: numpy>=1.23 +Requires-Dist: tqdm>=4.0 +Requires-Dist: matplotlib>=3.6 +Provides-Extra: external +Requires-Dist: gymnasium[classic-control]<2,>=1.0; extra == "external" +Dynamic: license-file + +# V13: Axis Symmetry & Intervention Equivariance Audit + +## V15.1 bug-fix release + +V15.1 replaces V15 after correcting the external state-token diagnostic. The original runner called the flat `text_decoder` layer directly; V15.1 uses the canonical `decode_text()` API and validates `[B,L,V]` logits against `[B,L]` discretized Gym-state token targets. No model architecture, checkpoint, dataset, training recipe, or scientific protocol changed. See `V15_1_BUGFIX.md`. + + +V13 freezes the V9 learning recipe and diagnoses the V12 vertical-action collapse with exact rotations/reflections/x-y swaps plus deliberately constructed collision episodes. See `V13_AXIS_SYMMETRY_EQUIVARIANCE.md`. + +Main run: `PYTHONPATH=. python scripts/run_v13_axis_audit.py --seeds 0 1 2 3 4 5 6 7 8 9 --pairs 64 --horizon 30 --device cuda` + +--- + +# V12: Causal Response Decomposition + +V12 freezes the V9 learning recipe and decomposes causal-response behavior by action program, horizon, effect size, initial speed, boundary contact, occlusion-zone exposure, and physics regime. It adds paired episode-clustered bootstrap confidence intervals against neural-operator and no-spectral-transition controls. See `V12_CAUSAL_RESPONSE_DECOMPOSITION.md`. + +Main run: `PYTHONPATH=. python scripts/run_v12_decomposition.py --seeds 0 1 2 3 4 5 6 7 8 9 --cf-horizon 30 --cf-pairs 64 --bootstrap 2000 --device cuda` + +--- + +--- + +## V9: causal generalization across physics regimes + +V9 freezes the structured spectral transport mechanism and targets the failure exposed by V8: excellent in-distribution counterfactual alignment did not extrapolate to changed physics. V9 trains on a family of dynamics regimes and adds trajectory-level counterfactual supervision, then evaluates interpolation, parameter extrapolation, and structural OOD shifts separately. See `V9_CAUSAL_GENERALIZATION.md`. + +Run: `PYTHONPATH=. python scripts/run_v9_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --cf-horizon 30 --planning-horizon 20 --planning-episodes 64 --device cuda` + +license: mit +library_name: pytorch +tags: +- world-models +- spectral-learning +- operator-learning +- multimodal +- wavelets +- pytorch +- research +--- + +# Spectral World Models — V7 + +V7 is the counterfactual-action training revision of the runnable Spectral World Models research prototype. + +## Why V7 + +V6 showed that structured transport can preserve long-horizon latent state, but the model substantially under-reacted to interventions. V7 keeps the V6 structured transition and adds a paired counterfactual objective so action response is learned explicitly rather than expected to emerge from reconstruction loss alone. + +The new `swm_structured_cf` model uses the same architecture as `swm_structured`; only the training objective changes. For the same starting state it rolls out opposing left/right interventions and optimizes: action-effect direction alignment, action-effect magnitude calibration, and endpoint reconstruction. Defaults are intentionally modest so the auxiliary objective does not overwhelm next-state learning. + +## Main experiment + +Install with `pip install -e .` and run: + +`PYTHONPATH=. python scripts/run_v7_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --cf-horizon 30 --device cuda` + +Results are written to `results/v7_counterfactual_training/metrics_summary.csv` and `metrics_by_seed.csv`. + +The primary comparison is `swm_structured_cf` versus unchanged `swm_structured`, `swm_selective`, `no_spectral_transition`, and `neural_operator`. Inspect counterfactual direction cosine and magnitude ratio together with H100 image MSE, text accuracy, and latent cosine. Do not select a model from a single scalar metric. + +## Tunable V7 objective + +The runner exposes `--cf-train-horizon`, `--cf-pairs`, `--cf-every`, `--lambda-cf-dir`, `--lambda-cf-mag`, and `--lambda-cf-branch`. Defaults are 8, 64, 4, 0.02, 0.005, and 0.10 respectively. Hyperparameters should be selected using training/validation evidence, not the held-out test set. + +## Validation + +Run `PYTHONPATH=. python -m pytest -q`. The V7 suite includes the earlier architecture/stability tests plus a backward-pass test for the new counterfactual objective. + +See `V7_COUNTERFACTUAL_TRAINING.md` for the experiment rationale. Earlier stages are documented in `V4_MULTIMODAL_STABILITY.md`, `V5_HARD_DYNAMICS.md`, and `V6_STRUCTURED_TRANSPORT.md`. + +## V8: frozen-architecture generalization and planning + +V8 freezes the V7 architectures and evaluates OOD dynamics, unseen intervention conditions, and imagination-based planning. See `V8_GENERALIZATION_AND_PLANNING.md`. + +Run: `PYTHONPATH=. python scripts/run_v8_benchmark.py --seeds 0 1 2 3 4 --horizons 5 10 20 30 50 100 --cf-horizon 30 --planning-horizon 20 --planning-episodes 96 --device cuda` + +## V10 — counterfactual benchmark integrity + +V10 freezes the learned architectures and audits the causal benchmark itself. See `V10_COUNTERFACTUAL_INTEGRITY.md`. + +Run: `PYTHONPATH=. python scripts/run_v10_audit.py --seeds 0 1 2 3 4 --cf-horizon 30 --cf-pairs 48 --device cuda` + +## V11: causal gain calibration +V11 freezes the V9 structured transport architecture and replaces endpoint-oriented causal magnitude fitting with scale-aware trajectory-effect calibration. It reports direction, gain error, magnitude correlation, and calibration slope/intercept across interpolation, extrapolation, and structural OOD. See `V11_CAUSAL_GAIN_CALIBRATION.md`. + +## V14: external control benchmark + +V14 freezes the V9 architecture family and moves evaluation outside the project's synthetic simulator. The external suite uses Gymnasium Classic Control (`CartPole-v1`, `MountainCar-v0`, `Acrobot-v1`) with official RGB rendering converted to the model's 32×32 grayscale input. Models are trained from scratch per environment; architecture and comparison logic are frozen. See `V14_EXTERNAL_CONTROL_BENCHMARK.md`. + +## V15 — External Dynamics Diagnostic +V15 is intentionally the **final diagnostic milestone**, not another model revision. It reloads the frozen V14 checkpoints and tests whether V14's negative full-frame MSE result reflects genuine dynamics failure or static-background metric confounding. See `V15_EXTERNAL_DYNAMICS_DIAGNOSTIC.md`. + +--- + +## Final reproducibility release (V6–V15.1) + +This tree is the frozen final reproducibility artifact for the SWM falsification program. See `docs/REPRODUCIBILITY_RELEASE.md` for the experimental history and interpretation, and `results/` for the measured V12–V15.1 audit CSVs included with this release. The project intentionally stops architecture iteration at V15.1. diff --git a/spectral_world_models.egg-info/SOURCES.txt b/spectral_world_models.egg-info/SOURCES.txt new file mode 100644 index 0000000000000000000000000000000000000000..ccb0093e138d7a5215ba353588399bf6de311848 --- /dev/null +++ b/spectral_world_models.egg-info/SOURCES.txt @@ -0,0 +1,38 @@ +LICENSE +README.md +pyproject.toml +spectral_world_models/__init__.py +spectral_world_models/counterfactual.py +spectral_world_models/data.py +spectral_world_models/haar.py +spectral_world_models/hard_dynamics.py +spectral_world_models/metrics.py +spectral_world_models/models.py +spectral_world_models/train.py +spectral_world_models/v10_cf_audit.py +spectral_world_models/v11_causal_calibration.py +spectral_world_models/v12_causal_decomposition.py +spectral_world_models/v13_axis_equivariance.py +spectral_world_models/v14_external_control.py +spectral_world_models/v8_generalization.py +spectral_world_models/v9_causal_generalization.py +spectral_world_models.egg-info/PKG-INFO +spectral_world_models.egg-info/SOURCES.txt +spectral_world_models.egg-info/dependency_links.txt +spectral_world_models.egg-info/requires.txt +spectral_world_models.egg-info/top_level.txt +tests/test_smoke.py +tests/test_stability_constraint.py +tests/test_v10_cf_audit.py +tests/test_v11_calibration.py +tests/test_v12_decomposition.py +tests/test_v13_axis_equivariance.py +tests/test_v14_external_control.py +tests/test_v15_1_state_token_decode.py +tests/test_v15_diagnostics.py +tests/test_v4_selective.py +tests/test_v5_hard_dynamics.py +tests/test_v6_structured.py +tests/test_v7_counterfactual_training.py +tests/test_v8_generalization.py +tests/test_v9_causal_generalization.py \ No newline at end of file diff --git a/spectral_world_models.egg-info/dependency_links.txt b/spectral_world_models.egg-info/dependency_links.txt new file mode 100644 index 0000000000000000000000000000000000000000..8b137891791fe96927ad78e64b0aad7bded08bdc --- /dev/null +++ b/spectral_world_models.egg-info/dependency_links.txt @@ -0,0 +1 @@ + diff --git a/spectral_world_models.egg-info/requires.txt b/spectral_world_models.egg-info/requires.txt new file mode 100644 index 0000000000000000000000000000000000000000..17520de4a41e3e0eb49317c62f776e1dd4bde6a6 --- /dev/null +++ b/spectral_world_models.egg-info/requires.txt @@ -0,0 +1,7 @@ +torch>=2.0 +numpy>=1.23 +tqdm>=4.0 +matplotlib>=3.6 + +[external] +gymnasium[classic-control]<2,>=1.0 diff --git a/spectral_world_models.egg-info/top_level.txt b/spectral_world_models.egg-info/top_level.txt new file mode 100644 index 0000000000000000000000000000000000000000..2a3a11fde1d3901ea7d77a6f05fc2200545be4e8 --- /dev/null +++ b/spectral_world_models.egg-info/top_level.txt @@ -0,0 +1 @@ +spectral_world_models diff --git a/spectral_world_models/__init__.py b/spectral_world_models/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..f35f2576622380db72fb4eb6f76100699a429c4b --- /dev/null +++ b/spectral_world_models/__init__.py @@ -0,0 +1,9 @@ +"""Spectral World Models research prototype.""" + +__all__ = [ + "data", + "haar", + "metrics", + "models", + "train", +] diff --git a/spectral_world_models/counterfactual.py b/spectral_world_models/counterfactual.py new file mode 100644 index 0000000000000000000000000000000000000000..7cf0139166c398c7346053123b9599741282d44e --- /dev/null +++ b/spectral_world_models/counterfactual.py @@ -0,0 +1,82 @@ +"""Counterfactual action-pair benchmark for V6.""" +from __future__ import annotations +import numpy as np, torch +from .hard_dynamics import HardDynamicsConfig, _render, _bounce +from .data import _text_tokens + +def _com(img): + # img [B,1,H,W] + w=img[:,0].clamp_min(0); B,H,W=w.shape + xs=torch.arange(W,device=w.device,dtype=w.dtype); ys=torch.arange(H,device=w.device,dtype=w.dtype) + mass=w.sum((1,2)).clamp_min(1e-6) + return torch.stack(((w.sum(1)*xs).sum(1)/mass,(w.sum(2)*ys).sum(1)/mass),-1) + +def generate_counterfactual_pairs(n=96,horizon=30,seed=606, cfg=None): + cfg=cfg or HardDynamicsConfig(); rng=np.random.default_rng(seed); pairs=[] + for _ in range(n): + shape='square' if rng.random()<.5 else 'circle'; radius=3 if shape=='circle' else 4 + x=float(rng.integers(radius+4,cfg.image_size-radius-4)); y=float(rng.integers(radius+4,cfg.image_size-radius-4)) + vx=float(rng.choice([-1.,1.])); vy=float(rng.choice([-1.,1.])); dx,dy=6.,6. + init=np.zeros((cfg.image_size,cfg.image_size),np.float32); _render(init,x,y,shape,radius,1.0); _render(init,dx,dy,'circle' if shape=='square' else 'square',3,.45) + init[:,cfg.image_size//2-2:cfg.image_size//2+2]*=.15 + tok=_text_tokens(shape,0,1,int(round(x)),int(round(y)),cfg.image_size,cfg.max_text_len) + branches=[] + # Persistent opposing horizontal controls expose causal action sensitivity. + for act in (3,4): + xx,yy,vxx,vyy=x,y,vx,vy; frames=[] + for t in range(horizon): + ax=(-cfg.acceleration if act==3 else cfg.acceleration); vxx=float(np.clip(vxx+ax,-cfg.max_speed,cfg.max_speed)); xx+=vxx; yy+=vyy + lo=radius+1; hi=cfg.image_size-radius-2; xx,vxx=_bounce(xx,vxx,lo,hi); yy,vyy=_bounce(yy,vyy,lo,hi) + c=np.zeros_like(init); _render(c,xx,yy,shape,radius,1.0); _render(c,dx,dy,'circle' if shape=='square' else 'square',3,.45); c[:,cfg.image_size//2-2:cfg.image_size//2+2]*=.15; frames.append(c) + branches.append(np.stack(frames)) + pairs.append((init,tok,branches[0],branches[1])) + return pairs + +@torch.no_grad() +def evaluate_counterfactual(model,device,horizon=30,n=96,seed=606): + model.eval(); pairs=generate_counterfactual_pairs(n,horizon,seed); mses=[]; cos=[]; ratios=[] + for init,tok,left,right in pairs: + image=torch.tensor(init[None,None],device=device); tokens=torch.tensor(tok[None],device=device); z0=model.encode(image,tokens); preds=[] + for act in (3,4): + z=z0.clone(); a=torch.tensor([act],device=device) + for _ in range(horizon): z=model.transition(z,a) + preds.append(model.image_decoder(z)) + gtL=torch.tensor(left[-1][None,None],device=device); gtR=torch.tensor(right[-1][None,None],device=device) + mses += [torch.mean((preds[0]-gtL)**2).item(),torch.mean((preds[1]-gtR)**2).item()] + pe=_com(preds[1])-_com(preds[0]); ge=_com(gtR)-_com(gtL) + pn=pe.norm(dim=-1); gn=ge.norm(dim=-1); cos.append(float((pe*ge).sum(-1)/(pn*gn).clamp_min(1e-6))); ratios.append(float(pn/(gn.clamp_min(1e-6)))) + return {'cf_final_image_mse':float(np.mean(mses)),'cf_effect_vector_cosine':float(np.mean(cos)),'cf_effect_magnitude_ratio':float(np.mean(ratios))} + + +def make_cf_training_batch(n=64, horizon=8, seed=707, device='cpu'): + """Create a differentiable-training batch from paired interventions.""" + pairs = generate_counterfactual_pairs(n=n, horizon=horizon, seed=seed) + init = torch.tensor(np.stack([p[0] for p in pairs])[:,None], dtype=torch.float32, device=device) + tok = torch.tensor(np.stack([p[1] for p in pairs]), dtype=torch.long, device=device) + left = torch.tensor(np.stack([p[2][-1] for p in pairs])[:,None], dtype=torch.float32, device=device) + right = torch.tensor(np.stack([p[3][-1] for p in pairs])[:,None], dtype=torch.float32, device=device) + return {'image':init,'tokens':tok,'left_target':left,'right_target':right,'horizon':horizon} + + +def counterfactual_training_loss(model, batch, lambda_dir=0.02, lambda_mag=0.005, lambda_branch=0.10): + """V7 paired-intervention objective. + + Direction aligns the predicted action-effect COM vector with ground truth; + magnitude uses a log-ratio penalty; branch reconstruction prevents a model + from satisfying the contrastive terms with unrealistic images. + """ + z0=model.encode(batch['image'],batch['tokens']); B=z0.shape[0]; preds=[] + for act in (3,4): + z=z0 + a=torch.full((B,),act,device=z.device,dtype=torch.long) + for _ in range(int(batch['horizon'])): z=model.transition(z,a) + preds.append(model.image_decoder(z)) + gtL,gtR=batch['left_target'],batch['right_target'] + pe=_com(preds[1])-_com(preds[0]); ge=_com(gtR)-_com(gtL) + pn=pe.norm(dim=-1).clamp_min(1e-5); gn=ge.norm(dim=-1).clamp_min(1e-5) + cos=(pe*ge).sum(-1)/(pn*gn) + dir_loss=(1.0-cos.clamp(-1,1)).mean() + mag_loss=torch.abs(torch.log(pn/gn)).clamp(max=5.0).mean() + branch_loss=0.5*(torch.mean((preds[0]-gtL)**2)+torch.mean((preds[1]-gtR)**2)) + total=lambda_dir*dir_loss+lambda_mag*mag_loss+lambda_branch*branch_loss + return total, {'cf_train_dir_loss':float(dir_loss.detach().cpu()),'cf_train_mag_loss':float(mag_loss.detach().cpu()),'cf_train_branch_mse':float(branch_loss.detach().cpu())} diff --git a/spectral_world_models/data.py b/spectral_world_models/data.py new file mode 100644 index 0000000000000000000000000000000000000000..534b93ce2a1e997ecf068db81f894cc5cbbdb05b --- /dev/null +++ b/spectral_world_models/data.py @@ -0,0 +1,171 @@ +''' +data.py defines the synthetic benchmark data pipeline for the spectral world model project. + +Specifies vocabulary and action labels for text-conditioned dynamics +Generates moving shape sequences with image frames, text tokens, actions, and rewards +Saves train/val/test splits to a compressed .npz benchmark file +Provides TransitionDataset for frame-to-frame transition learning +Provides SequenceDataset for full-sequence rollout evaluation +''' +# Andrew Kiruluta, UC Berkeley, June 2026 + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +import json +import math +import numpy as np +import torch +from torch.utils.data import Dataset + + +VOCAB = [ + "", + "square", "circle", + "stay", "up", "down", "left", "right", + "slow", "fast", + "x0", "x1", "x2", "x3", + "y0", "y1", "y2", "y3", +] +TOKEN_TO_ID = {tok: i for i, tok in enumerate(VOCAB)} +ACTION_NAMES = ["stay", "up", "down", "left", "right"] + + +@dataclass +class DatasetConfig: + image_size: int = 32 + seq_len: int = 8 + max_text_len: int = 5 + train_sequences: int = 256 + val_sequences: int = 64 + test_sequences: int = 64 + seed: int = 7 + + +def _draw_square(canvas: np.ndarray, x: int, y: int, size: int) -> None: + h, w = canvas.shape + x0 = max(0, x - size // 2); x1 = min(w, x + size // 2 + 1) + y0 = max(0, y - size // 2); y1 = min(h, y + size // 2 + 1) + canvas[y0:y1, x0:x1] = 1.0 + + +def _draw_circle(canvas: np.ndarray, x: int, y: int, radius: int) -> None: + h, w = canvas.shape + yy, xx = np.ogrid[:h, :w] + mask = (xx - x) ** 2 + (yy - y) ** 2 <= radius**2 + canvas[mask] = 1.0 + + +def _text_tokens(shape: str, action: int, speed: int, x: int, y: int, image_size: int, max_len: int) -> np.ndarray: + xb = min(3, max(0, int(4 * x / image_size))) + yb = min(3, max(0, int(4 * y / image_size))) + words = [shape, ACTION_NAMES[action], "fast" if speed == 2 else "slow", f"x{xb}", f"y{yb}"] + ids = [TOKEN_TO_ID[w] for w in words][:max_len] + ids += [0] * (max_len - len(ids)) + return np.asarray(ids, dtype=np.int64) + + +def _apply_action(x: int, y: int, action: int, speed: int, image_size: int, radius: int) -> tuple[int, int]: + dx, dy = 0, 0 + if action == 1: dy = -speed + elif action == 2: dy = speed + elif action == 3: dx = -speed + elif action == 4: dx = speed + x = int(np.clip(x + dx, radius + 1, image_size - radius - 2)) + y = int(np.clip(y + dy, radius + 1, image_size - radius - 2)) + return x, y + + +def generate_sequences(n_sequences: int, cfg: DatasetConfig, rng: np.random.Generator) -> dict[str, np.ndarray]: + images = np.zeros((n_sequences, cfg.seq_len, 1, cfg.image_size, cfg.image_size), dtype=np.float32) + tokens = np.zeros((n_sequences, cfg.seq_len, cfg.max_text_len), dtype=np.int64) + actions = np.zeros((n_sequences, cfg.seq_len - 1), dtype=np.int64) + rewards = np.zeros((n_sequences, cfg.seq_len - 1), dtype=np.float32) + + for n in range(n_sequences): + shape = "square" if rng.random() < 0.5 else "circle" + speed = int(rng.choice([1, 2])) + radius = 3 if shape == "circle" else 4 + x = int(rng.integers(radius + 2, cfg.image_size - radius - 2)) + y = int(rng.integers(radius + 2, cfg.image_size - radius - 2)) + last_action = 0 + for t in range(cfg.seq_len): + canvas = np.zeros((cfg.image_size, cfg.image_size), dtype=np.float32) + if shape == "square": + _draw_square(canvas, x, y, 6) + else: + _draw_circle(canvas, x, y, radius) + # Add tiny structured sensor noise to make denoising useful. + canvas += rng.normal(0.0, 0.01, size=canvas.shape).astype(np.float32) + canvas = np.clip(canvas, 0.0, 1.0) + images[n, t, 0] = canvas + tokens[n, t] = _text_tokens(shape, last_action, speed, x, y, cfg.image_size, cfg.max_text_len) + if t < cfg.seq_len - 1: + action = int(rng.integers(0, len(ACTION_NAMES))) + actions[n, t] = action + x, y = _apply_action(x, y, action, speed, cfg.image_size, radius) + last_action = action + # Reward favors reaching the upper-right quadrant. + rewards[n, t] = float((x > cfg.image_size * 0.65) and (y < cfg.image_size * 0.35)) + return {"images": images, "tokens": tokens, "actions": actions, "rewards": rewards} + + +def generate_benchmark_npz(path: str | Path, cfg: DatasetConfig = DatasetConfig()) -> None: + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + rng = np.random.default_rng(cfg.seed) + train = generate_sequences(cfg.train_sequences, cfg, rng) + val = generate_sequences(cfg.val_sequences, cfg, rng) + test = generate_sequences(cfg.test_sequences, cfg, rng) + np.savez_compressed( + path, + train_images=train["images"], train_tokens=train["tokens"], train_actions=train["actions"], train_rewards=train["rewards"], + val_images=val["images"], val_tokens=val["tokens"], val_actions=val["actions"], val_rewards=val["rewards"], + test_images=test["images"], test_tokens=test["tokens"], test_actions=test["actions"], test_rewards=test["rewards"], + vocab=np.asarray(VOCAB), action_names=np.asarray(ACTION_NAMES), + config=json.dumps(cfg.__dict__), + ) + + +class TransitionDataset(Dataset): + def __init__(self, npz_path: str | Path, split: str): + data = np.load(npz_path, allow_pickle=True) + self.images = data[f"{split}_images"] + self.tokens = data[f"{split}_tokens"] + self.actions = data[f"{split}_actions"] + self.seq_len = self.images.shape[1] + self.index = [(i, t) for i in range(self.images.shape[0]) for t in range(self.seq_len - 1)] + + def __len__(self) -> int: + return len(self.index) + + def __getitem__(self, idx: int) -> dict[str, torch.Tensor]: + i, t = self.index[idx] + return { + "image": torch.from_numpy(self.images[i, t]), + "tokens": torch.from_numpy(self.tokens[i, t]).long(), + "action": torch.tensor(self.actions[i, t]).long(), + "next_image": torch.from_numpy(self.images[i, t + 1]), + "next_tokens": torch.from_numpy(self.tokens[i, t + 1]).long(), + } + + +class SequenceDataset(Dataset): + def __init__(self, npz_path: str | Path, split: str): + data = np.load(npz_path, allow_pickle=True) + self.images = data[f"{split}_images"] + self.tokens = data[f"{split}_tokens"] + self.actions = data[f"{split}_actions"] + self.rewards = data[f"{split}_rewards"] + + def __len__(self) -> int: + return self.images.shape[0] + + def __getitem__(self, idx: int) -> dict[str, torch.Tensor]: + return { + "images": torch.from_numpy(self.images[idx]), + "tokens": torch.from_numpy(self.tokens[idx]).long(), + "actions": torch.from_numpy(self.actions[idx]).long(), + "rewards": torch.from_numpy(self.rewards[idx]), + } diff --git a/spectral_world_models/haar.py b/spectral_world_models/haar.py new file mode 100644 index 0000000000000000000000000000000000000000..b863d96e9fbc4e312f962f5d60a75d8e5d6e5a10 --- /dev/null +++ b/spectral_world_models/haar.py @@ -0,0 +1,87 @@ +''' +haar.py defines a dependency-free 2D Haar wavelet transform module. + +HaarWavelet2D performs a multi-level discrete wavelet transform on 2D images +forward returns packed wavelet coefficients with the same flattened size as the input +inverse reconstructs the image from those packed coefficients +The implementation supports configurable image size and decomposition levels +''' + +# Andrew Kiruluta, UC Berkeley, June 2026 + +from __future__ import annotations + +import torch +import torch.nn as nn + + +class HaarWavelet2D(nn.Module): + """Fixed orthonormal-ish 2D Haar transform with packed coefficients. + + The implementation is intentionally dependency-free. For 32x32 images and two + levels it returns a flattened coefficient vector with the same dimensionality + as the input image. The inverse reconstructs from this packed representation. + """ + + def __init__(self, image_size: int = 32, levels: int = 2): + super().__init__() + if image_size % (2**levels) != 0: + raise ValueError("image_size must be divisible by 2**levels") + self.image_size = image_size + self.levels = levels + + @staticmethod + def _dwt_level(x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + a = x[..., 0::2, 0::2] + b = x[..., 0::2, 1::2] + c = x[..., 1::2, 0::2] + d = x[..., 1::2, 1::2] + ll = (a + b + c + d) / 2.0 + lh = (a - b + c - d) / 2.0 + hl = (a + b - c - d) / 2.0 + hh = (a - b - c + d) / 2.0 + return ll, lh, hl, hh + + @staticmethod + def _idwt_level(ll: torch.Tensor, lh: torch.Tensor, hl: torch.Tensor, hh: torch.Tensor) -> torch.Tensor: + b, c, h, w = ll.shape + out = torch.zeros(b, c, h * 2, w * 2, device=ll.device, dtype=ll.dtype) + out[..., 0::2, 0::2] = (ll + lh + hl + hh) / 2.0 + out[..., 0::2, 1::2] = (ll - lh + hl - hh) / 2.0 + out[..., 1::2, 0::2] = (ll + lh - hl - hh) / 2.0 + out[..., 1::2, 1::2] = (ll - lh - hl + hh) / 2.0 + return out + + def forward(self, x: torch.Tensor) -> torch.Tensor: + coeffs = [] + current = x + detail_stack = [] + for _ in range(self.levels): + ll, lh, hl, hh = self._dwt_level(current) + detail_stack.append((lh, hl, hh)) + current = ll + coeffs.append(current.flatten(1)) + for detail in reversed(detail_stack): + for band in detail: + coeffs.append(band.flatten(1)) + return torch.cat(coeffs, dim=1) + + def inverse(self, coeffs: torch.Tensor) -> torch.Tensor: + b = coeffs.shape[0] + c = 1 + offset = 0 + low_size = self.image_size // (2**self.levels) + n_low = c * low_size * low_size + current = coeffs[:, offset : offset + n_low].view(b, c, low_size, low_size) + offset += n_low + # In forward, details are stored from coarsest to finest. + sizes = [self.image_size // (2**level) for level in range(self.levels, 0, -1)] + for size in sizes: + h = size + w = size + n = c * h * w + lh = coeffs[:, offset : offset + n].view(b, c, h, w); offset += n + hl = coeffs[:, offset : offset + n].view(b, c, h, w); offset += n + hh = coeffs[:, offset : offset + n].view(b, c, h, w); offset += n + current = self._idwt_level(current, lh, hl, hh) + return current diff --git a/spectral_world_models/hard_dynamics.py b/spectral_world_models/hard_dynamics.py new file mode 100644 index 0000000000000000000000000000000000000000..148e3e8a51d439ec09b95379b963bdbae2db1531 --- /dev/null +++ b/spectral_world_models/hard_dynamics.py @@ -0,0 +1,70 @@ +"""V5 harder controlled-dynamics benchmark. + +Keeps the v4 tensor/API schema while adding inertial velocity, acceleration actions, +wall bounces, a moving distractor, and intermittent occlusion. Text describes the +controlled target, so successful long-horizon prediction cannot rely on static +latent persistence alone. +""" +from __future__ import annotations +from dataclasses import dataclass +from pathlib import Path +import json +import numpy as np +from .data import VOCAB, ACTION_NAMES, TOKEN_TO_ID, _draw_square, _draw_circle, _text_tokens + +@dataclass +class HardDynamicsConfig: + image_size:int=32; seq_len:int=16; max_text_len:int=5 + train_sequences:int=512; val_sequences:int=96; test_sequences:int=96; seed:int=17 + occlusion_prob:float=0.12; acceleration:float=0.70; max_speed:float=3.0; damping:float=1.0; restitution:float=1.0 + +def _render(canvas,x,y,shape,radius,intensity=1.0): + tmp=np.zeros_like(canvas) + if shape=="square": _draw_square(tmp,int(round(x)),int(round(y)),6) + else: _draw_circle(tmp,int(round(x)),int(round(y)),radius) + np.maximum(canvas,tmp*intensity,out=canvas) + +def _bounce(p,v,lo,hi): + if phi: return hi-(p-hi),-abs(v) + return p,v + +def generate_hard_sequences(n,cfg,rng): + I=np.zeros((n,cfg.seq_len,1,cfg.image_size,cfg.image_size),np.float32) + T=np.zeros((n,cfg.seq_len,cfg.max_text_len),np.int64) + A=np.zeros((n,cfg.seq_len-1),np.int64); R=np.zeros((n,cfg.seq_len-1),np.float32) + for k in range(n): + shape="square" if rng.random()<.5 else "circle"; radius=3 if shape=="circle" else 4 + x=float(rng.integers(radius+3,cfg.image_size-radius-3)); y=float(rng.integers(radius+3,cfg.image_size-radius-3)) + vx=float(rng.choice([-2,-1,1,2])); vy=float(rng.choice([-2,-1,1,2])) + dx=float(rng.integers(5,cfg.image_size-5)); dy=float(rng.integers(5,cfg.image_size-5)); dvx=float(rng.choice([-1.5,1.5])); dvy=float(rng.choice([-1.5,1.5])) + last=0 + for t in range(cfg.seq_len): + c=np.zeros((cfg.image_size,cfg.image_size),np.float32) + occluded=(t>1 and rng.random()=3 else 1 + T[k,t]=_text_tokens(shape,last,speed,int(round(x)),int(round(y)),cfg.image_size,cfg.max_text_len) + if tcfg.image_size*.65) and (y torch.Tensor: + return torch.mean((pred - target) ** 2) + + +def psnr(pred: torch.Tensor, target: torch.Tensor, data_range: float = 1.0) -> float: + val = mse(pred, target).detach().item() + if val <= 1e-12: + return 99.0 + return 20.0 * math.log10(data_range) - 10.0 * math.log10(val) + + +def ssim_global(pred: torch.Tensor, target: torch.Tensor, data_range: float = 1.0) -> float: + """Simple global SSIM approximation for small benchmark reporting.""" + pred = pred.detach() + target = target.detach() + dims = tuple(range(1, pred.ndim)) + c1 = (0.01 * data_range) ** 2 + c2 = (0.03 * data_range) ** 2 + mu_x = pred.mean(dim=dims) + mu_y = target.mean(dim=dims) + var_x = ((pred - mu_x.view(-1, *([1] * (pred.ndim - 1)))) ** 2).mean(dim=dims) + var_y = ((target - mu_y.view(-1, *([1] * (target.ndim - 1)))) ** 2).mean(dim=dims) + cov = ((pred - mu_x.view(-1, *([1] * (pred.ndim - 1)))) * (target - mu_y.view(-1, *([1] * (target.ndim - 1))))).mean(dim=dims) + score = ((2 * mu_x * mu_y + c1) * (2 * cov + c2)) / ((mu_x**2 + mu_y**2 + c1) * (var_x + var_y + c2)) + return score.mean().item() + + +def gaussian_nll(pred: torch.Tensor, target: torch.Tensor, sigma: float = 0.12) -> float: + var = sigma**2 + nll = 0.5 * (((target - pred) ** 2) / var + math.log(2 * math.pi * var)) + return nll.mean().detach().item() + + +def text_accuracy(logits: torch.Tensor, tokens: torch.Tensor, pad_id: int = 0) -> float: + pred = logits.argmax(dim=-1) + mask = tokens.ne(pad_id) + if mask.sum().item() == 0: + return 0.0 + return (pred.eq(tokens) & mask).sum().item() / mask.sum().item() + + +def count_parameters(model: torch.nn.Module) -> int: + return sum(p.numel() for p in model.parameters() if p.requires_grad) diff --git a/spectral_world_models/models.py b/spectral_world_models/models.py new file mode 100644 index 0000000000000000000000000000000000000000..b9040074effd7485e85c021dc9950f2395d33f83 --- /dev/null +++ b/spectral_world_models/models.py @@ -0,0 +1,495 @@ +''' +models.py defines a configurable spectral world model family and its building blocks. It includes: + +text encoders: DCTTextEncoder (fixed DCT basis) and MeanTextEncoder +image encoders/decoders: wavelet, convolutional, and FFT-based variants +transition modules: low-rank spectral, MLP, Koopman, and mini-transformer transitions +WorldModel, which fuses image/text latents, predicts the next latent state, and decodes next-frame images plus text logits +build_model(name, **kwargs) to instantiate named model variants and ablations by aliasing configuration names. +''' + +# Andrew Kiruluta, UC Berkeley, June 2026 + +from __future__ import annotations + +import math +import torch +import torch.nn as nn +import torch.nn.functional as F +from .haar import HaarWavelet2D +from .data import VOCAB, ACTION_NAMES + + +class DCTTextEncoder(nn.Module): + """Text-to-spectral-field encoder using a fixed DCT basis over token position.""" + + def __init__(self, vocab_size: int, max_len: int, emb_dim: int, out_dim: int): + super().__init__() + self.max_len = max_len + self.emb = nn.Embedding(vocab_size, emb_dim, padding_idx=0) + dct = torch.zeros(max_len, max_len) + for k in range(max_len): + for n in range(max_len): + alpha = math.sqrt(1.0 / max_len) if k == 0 else math.sqrt(2.0 / max_len) + dct[k, n] = alpha * math.cos(math.pi * (n + 0.5) * k / max_len) + self.register_buffer("dct", dct) + self.proj = nn.Sequential( + nn.Linear(max_len * emb_dim, out_dim), + nn.LayerNorm(out_dim), + nn.GELU(), + ) + + def forward(self, tokens: torch.Tensor) -> torch.Tensor: + e = self.emb(tokens) # [B, L, E] + modes = torch.einsum("kl,ble->bke", self.dct, e) + return self.proj(modes.flatten(1)) + + +class MeanTextEncoder(nn.Module): + def __init__(self, vocab_size: int, emb_dim: int, out_dim: int): + super().__init__() + self.emb = nn.Embedding(vocab_size, emb_dim, padding_idx=0) + self.proj = nn.Sequential(nn.Linear(emb_dim, out_dim), nn.LayerNorm(out_dim), nn.GELU()) + + def forward(self, tokens: torch.Tensor) -> torch.Tensor: + mask = tokens.ne(0).float().unsqueeze(-1) + e = self.emb(tokens) + pooled = (e * mask).sum(dim=1) / mask.sum(dim=1).clamp_min(1.0) + return self.proj(pooled) + + +class WaveletImageEncoder(nn.Module): + def __init__(self, image_size: int, levels: int, out_dim: int): + super().__init__() + self.wave = HaarWavelet2D(image_size=image_size, levels=levels) + self.coeff_dim = image_size * image_size + self.net = nn.Sequential( + nn.Linear(self.coeff_dim, out_dim), + nn.LayerNorm(out_dim), + nn.GELU(), + ) + + def forward(self, image: torch.Tensor) -> torch.Tensor: + coeffs = self.wave(image) + return self.net(coeffs) + + +class ConvImageEncoder(nn.Module): + def __init__(self, image_size: int, out_dim: int): + super().__init__() + self.net = nn.Sequential( + nn.Conv2d(1, 16, 3, stride=2, padding=1), nn.GELU(), + nn.Conv2d(16, 32, 3, stride=2, padding=1), nn.GELU(), + nn.Flatten(), + nn.Linear(32 * (image_size // 4) * (image_size // 4), out_dim), + nn.LayerNorm(out_dim), + nn.GELU(), + ) + + def forward(self, image: torch.Tensor) -> torch.Tensor: + return self.net(image) + + +class FFTImageEncoder(nn.Module): + """Small Fourier feature encoder used as a neural-operator-style baseline.""" + + def __init__(self, image_size: int, out_dim: int, keep: int = 8): + super().__init__() + self.keep = keep + # rfft width keep, height keep => real+imag features. + in_dim = keep * keep * 2 + self.net = nn.Sequential(nn.Linear(in_dim, out_dim), nn.LayerNorm(out_dim), nn.GELU()) + + def forward(self, image: torch.Tensor) -> torch.Tensor: + fft = torch.fft.rfft2(image.squeeze(1), norm="ortho") + x = fft[:, : self.keep, : self.keep] + feats = torch.cat([x.real, x.imag], dim=1).flatten(1) + return self.net(feats) + + +class WaveletImageDecoder(nn.Module): + def __init__(self, image_size: int, levels: int, latent_dim: int): + super().__init__() + self.wave = HaarWavelet2D(image_size=image_size, levels=levels) + self.net = nn.Sequential( + nn.Linear(latent_dim, 256), nn.GELU(), + nn.Linear(256, image_size * image_size), + ) + + def forward(self, z: torch.Tensor) -> torch.Tensor: + coeffs = self.net(z) + return torch.sigmoid(self.wave.inverse(coeffs)) + + +class MLPImageDecoder(nn.Module): + def __init__(self, image_size: int, latent_dim: int): + super().__init__() + self.image_size = image_size + self.net = nn.Sequential( + nn.Linear(latent_dim, 256), nn.GELU(), + nn.Linear(256, image_size * image_size), nn.Sigmoid(), + ) + + def forward(self, z: torch.Tensor) -> torch.Tensor: + return self.net(z).view(z.shape[0], 1, self.image_size, self.image_size) + + +class LowRankSpectralTransition(nn.Module): + """Action-conditioned sparse/low-rank spectral operator dictionary. + + z_{t+1} = tanh(W0 z + sum_r beta_r(a) U_r V_r z + b(a) + nonlinear(z)). + """ + + def __init__(self, latent_dim: int, action_dim: int, n_ops: int = 4, rank: int = 16): + super().__init__() + self.latent_dim = latent_dim + self.n_ops = n_ops + self.rank = rank + self.base = nn.Linear(latent_dim, latent_dim, bias=False) + with torch.no_grad(): + self.base.weight.copy_(0.72 * torch.eye(latent_dim) + 0.01 * torch.randn(latent_dim, latent_dim)) + self.U = nn.Parameter(torch.randn(n_ops, latent_dim, rank) * 0.035) + self.V = nn.Parameter(torch.randn(n_ops, rank, latent_dim) * 0.035) + self.beta = nn.Embedding(action_dim, n_ops) + self.bias = nn.Embedding(action_dim, latent_dim) + self.nonlin = nn.Sequential(nn.Linear(latent_dim, latent_dim), nn.GELU(), nn.Linear(latent_dim, latent_dim)) + + def forward(self, z: torch.Tensor, action: torch.Tensor) -> torch.Tensor: + out = self.base(z) + beta = torch.tanh(self.beta(action)) + # Low-rank operator dictionary. + for r in range(self.n_ops): + vrz = z @ self.V[r].t() + urvz = vrz @ self.U[r].t() + out = out + beta[:, r : r + 1] * urvz + out = out + self.bias(action) + 0.10 * self.nonlin(z) + return torch.tanh(out) + + def stability_penalty(self) -> torch.Tensor: + # Smooth proxy for keeping operator energy bounded. + base_energy = torch.linalg.matrix_norm(self.base.weight, ord="fro") / math.sqrt(self.latent_dim) + op_energy = torch.stack([torch.linalg.matrix_norm(self.U[r] @ self.V[r], ord="fro") / math.sqrt(self.latent_dim) for r in range(self.n_ops)]).mean() + return F.relu(base_energy + op_energy - 0.95) ** 2 + + +class SpectralNormConstrainedTransition(LowRankSpectralTransition): + """Low-rank action-conditioned transition with a hard spectral-norm cap. + + For each action, form A(a)=W0+sum_r beta_r(a) U_r V_r and rescale the + effective matrix when its largest singular value exceeds ``rho``. This + variant intentionally removes the unconstrained nonlinear residual so the + latent transition has a directly testable contraction bound before tanh. + """ + def __init__(self, latent_dim: int, action_dim: int, n_ops: int = 4, rank: int = 16, rho: float = 0.95): + super().__init__(latent_dim, action_dim, n_ops=n_ops, rank=rank) + self.rho = float(rho) + self.action_dim = action_dim + + def effective_matrix(self, action_ids: torch.Tensor) -> torch.Tensor: + # [A,D,D] for unique action ids. Embedding coefficients remain trainable. + beta = torch.tanh(self.beta(action_ids)) + uv = torch.einsum("odr,ore->ode", self.U, self.V) + return self.base.weight.unsqueeze(0) + torch.einsum("ao,ode->ade", beta, uv) + + def constrained_matrix(self, action_ids: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + A = self.effective_matrix(action_ids) + sigma = torch.linalg.matrix_norm(A, ord=2, dim=(-2, -1)) + scale = torch.clamp(self.rho / sigma.clamp_min(1e-8), max=1.0) + return A * scale[:, None, None], sigma + + def forward(self, z: torch.Tensor, action: torch.Tensor) -> torch.Tensor: + unique, inverse = torch.unique(action, sorted=True, return_inverse=True) + A, _ = self.constrained_matrix(unique) + A_batch = A[inverse] + out = torch.bmm(A_batch, z.unsqueeze(-1)).squeeze(-1) + self.bias(action) + return torch.tanh(out) + + def stability_penalty(self) -> torch.Tensor: + # Hard forward constraint already enforces the cap; this auxiliary loss + # discourages repeatedly relying on rescaling. + ids = torch.arange(self.action_dim, device=self.base.weight.device) + raw = self.effective_matrix(ids) + sigma = torch.linalg.matrix_norm(raw, ord=2, dim=(-2, -1)) + return torch.mean(F.relu(sigma - self.rho) ** 2) + + @torch.no_grad() + def stability_diagnostics(self) -> dict[str, float]: + ids = torch.arange(self.action_dim, device=self.base.weight.device) + A, raw_sigma = self.constrained_matrix(ids) + constrained_sigma = torch.linalg.matrix_norm(A, ord=2, dim=(-2, -1)) + return { + "raw_operator_norm_max": float(raw_sigma.max().cpu()), + "raw_operator_norm_mean": float(raw_sigma.mean().cpu()), + "effective_operator_norm_max": float(constrained_sigma.max().cpu()), + "effective_operator_norm_mean": float(constrained_sigma.mean().cpu()), + "rho": self.rho, + } + + +class SelectiveSpectralTransition(LowRankSpectralTransition): + """Selective mode regularization for multimodal transport. + + Unlike the v3 hard contraction variant, this keeps singular modes at or below + ``max_gain`` untouched and clips only expansive singular values. Near-unit + modes are therefore preserved for persistent/action-conditioned transport. + The nonlinear residual is retained. + """ + def __init__(self, latent_dim: int, action_dim: int, n_ops: int = 4, rank: int = 16, max_gain: float = 1.05): + super().__init__(latent_dim, action_dim, n_ops=n_ops, rank=rank) + self.max_gain = float(max_gain) + self.action_dim = action_dim + + def effective_matrix(self, action_ids: torch.Tensor) -> torch.Tensor: + beta = torch.tanh(self.beta(action_ids)) + uv = torch.einsum("odr,ore->ode", self.U, self.V) + return self.base.weight.unsqueeze(0) + torch.einsum("ao,ode->ade", beta, uv) + + def selective_matrix(self, action_ids: torch.Tensor): + A = self.effective_matrix(action_ids) + U, S, Vh = torch.linalg.svd(A, full_matrices=False) + Sc = torch.clamp(S, max=self.max_gain) + Ac = (U * Sc.unsqueeze(-2)) @ Vh + return Ac, S, Sc + + def forward(self, z: torch.Tensor, action: torch.Tensor) -> torch.Tensor: + unique, inverse = torch.unique(action, sorted=True, return_inverse=True) + A, _, _ = self.selective_matrix(unique) + out = torch.bmm(A[inverse], z.unsqueeze(-1)).squeeze(-1) + out = out + self.bias(action) + 0.10 * self.nonlin(z) + return torch.tanh(out) + + def stability_penalty(self) -> torch.Tensor: + ids = torch.arange(self.action_dim, device=self.base.weight.device) + A = self.effective_matrix(ids) + sigma = torch.linalg.svdvals(A) + # Penalize only expansive modes; do not push all modes toward contraction. + return torch.mean(F.relu(sigma - self.max_gain) ** 2) + + @torch.no_grad() + def stability_diagnostics(self) -> dict[str, float]: + ids = torch.arange(self.action_dim, device=self.base.weight.device) + _, raw, clipped = self.selective_matrix(ids) + return { + "raw_operator_norm_max": float(raw[:, 0].max().cpu()), + "raw_operator_norm_mean": float(raw[:, 0].mean().cpu()), + "effective_operator_norm_max": float(clipped[:, 0].max().cpu()), + "effective_operator_norm_mean": float(clipped[:, 0].mean().cpu()), + "rho": self.max_gain, + "fraction_modes_clipped": float((raw > self.max_gain).float().mean().cpu()), + } + + +class StructuredSpectralTransportTransition(nn.Module): + """V6 structured action-conditioned transport. + + Separates persistent/energy-preserving transport, dissipative spectral modes, + and action-driven low-rank transport instead of asking one generic operator + to learn all three roles. The persistent block is parameterized by a + skew-symmetric generator and a Cayley transform, giving an orthogonal + transport matrix. Dissipation is diagonal in the learned latent basis and + action effects are low rank. Only expansive singular modes of the composed + operator are softly clipped at ``max_gain``. + """ + def __init__(self, latent_dim:int, action_dim:int, rank:int=12, max_gain:float=1.05): + super().__init__(); self.latent_dim=latent_dim; self.action_dim=action_dim; self.rank=rank; self.max_gain=float(max_gain) + self.generator=nn.Parameter(0.01*torch.randn(latent_dim,latent_dim)) + self.logit_keep=nn.Parameter(torch.full((latent_dim,), 2.2)) + self.action_U=nn.Parameter(0.025*torch.randn(action_dim,latent_dim,rank)) + self.action_V=nn.Parameter(0.025*torch.randn(action_dim,rank,latent_dim)) + self.action_gate=nn.Embedding(action_dim,1); nn.init.zeros_(self.action_gate.weight) + self.bias=nn.Embedding(action_dim,latent_dim) + self.residual=nn.Sequential(nn.Linear(latent_dim,latent_dim),nn.GELU(),nn.Linear(latent_dim,latent_dim)) + + def transport_matrix(self): + G=0.25*torch.tanh(self.generator); K=0.5*(G-G.t()); I=torch.eye(self.latent_dim,device=K.device,dtype=K.dtype) + return torch.linalg.solve(I-0.5*K, I+0.5*K) + + def effective_matrix(self, action_ids): + Q=self.transport_matrix(); keep=0.80+0.20*torch.sigmoid(self.logit_keep) + base=Q*keep.unsqueeze(0) + uv=torch.einsum('adr,are->ade',self.action_U[action_ids],self.action_V[action_ids]) + gate=torch.tanh(self.action_gate(action_ids)).view(-1,1,1) + return base.unsqueeze(0)+gate*uv + + def selective_matrix(self, action_ids): + A=self.effective_matrix(action_ids); U,S,Vh=torch.linalg.svd(A.detach(),full_matrices=False); Sc=torch.clamp(S,max=self.max_gain) + Ac=(U*Sc.unsqueeze(-2))@Vh + # Straight-through projection: clipped matrix in the forward pass while + # avoiding unstable SVD gradients during optimization. + return A + (Ac-A.detach()),S,Sc + + def forward(self,z,action): + unique,inverse=torch.unique(action,sorted=True,return_inverse=True); A,_,_=self.selective_matrix(unique) + out=torch.bmm(A[inverse],z.unsqueeze(-1)).squeeze(-1)+self.bias(action)+0.05*self.residual(z) + return torch.tanh(out) + + def stability_penalty(self): + ids=torch.arange(self.action_dim,device=self.generator.device); A=self.effective_matrix(ids); s=torch.linalg.svdvals(A) + expansive=F.relu(s-self.max_gain).pow(2).mean(); q=self.transport_matrix(); I=torch.eye(self.latent_dim,device=q.device) + orth=(q.t()@q-I).pow(2).mean(); return expansive+0.01*orth + + @torch.no_grad() + def stability_diagnostics(self): + ids=torch.arange(self.action_dim,device=self.generator.device); _,raw,clip=self.selective_matrix(ids); q=self.transport_matrix(); I=torch.eye(self.latent_dim,device=q.device) + return {'raw_operator_norm_max':float(raw[:,0].max().cpu()),'raw_operator_norm_mean':float(raw[:,0].mean().cpu()), + 'effective_operator_norm_max':float(clip[:,0].max().cpu()),'effective_operator_norm_mean':float(clip[:,0].mean().cpu()), + 'fraction_modes_clipped':float((raw>self.max_gain).float().mean().cpu()),'rho':self.max_gain, + 'transport_orthogonality_error':float(torch.linalg.matrix_norm(q.t()@q-I,ord='fro').cpu()/self.latent_dim**0.5), + 'mean_dissipative_keep':float((0.80+0.20*torch.sigmoid(self.logit_keep)).mean().cpu())} + + +class MLPTransition(nn.Module): + def __init__(self, latent_dim: int, action_dim: int): + super().__init__() + self.emb = nn.Embedding(action_dim, 24) + self.net = nn.Sequential( + nn.Linear(latent_dim + 24, 192), nn.GELU(), + nn.Linear(192, 192), nn.GELU(), + nn.Linear(192, latent_dim), nn.Tanh(), + ) + + def forward(self, z: torch.Tensor, action: torch.Tensor) -> torch.Tensor: + return self.net(torch.cat([z, self.emb(action)], dim=-1)) + + def stability_penalty(self) -> torch.Tensor: + return torch.tensor(0.0, device=next(self.parameters()).device) + + +class KoopmanTransition(nn.Module): + def __init__(self, latent_dim: int, action_dim: int): + super().__init__() + self.A = nn.Parameter(0.75 * torch.eye(latent_dim).unsqueeze(0).repeat(action_dim, 1, 1) + 0.005 * torch.randn(action_dim, latent_dim, latent_dim)) + self.bias = nn.Embedding(action_dim, latent_dim) + + def forward(self, z: torch.Tensor, action: torch.Tensor) -> torch.Tensor: + A = self.A[action] + return torch.tanh(torch.bmm(A, z.unsqueeze(-1)).squeeze(-1) + self.bias(action)) + + def stability_penalty(self) -> torch.Tensor: + fro = torch.linalg.matrix_norm(self.A, ord="fro", dim=(1, 2)).mean() / math.sqrt(self.A.shape[-1]) + return F.relu(fro - 0.95) ** 2 + + +class MiniTransformerTransition(nn.Module): + def __init__(self, latent_dim: int, action_dim: int): + super().__init__() + self.action = nn.Embedding(action_dim, latent_dim) + layer = nn.TransformerEncoderLayer(d_model=latent_dim, nhead=4, dim_feedforward=192, batch_first=True, dropout=0.0) + self.encoder = nn.TransformerEncoder(layer, num_layers=1) + self.out = nn.Sequential(nn.Linear(latent_dim, latent_dim), nn.Tanh()) + + def forward(self, z: torch.Tensor, action: torch.Tensor) -> torch.Tensor: + seq = torch.stack([z, self.action(action)], dim=1) + h = self.encoder(seq) + return self.out(h[:, 0]) + + def stability_penalty(self) -> torch.Tensor: + return torch.tensor(0.0, device=next(self.parameters()).device) + + +class WorldModel(nn.Module): + def __init__( + self, + model_type: str = "swm", + image_size: int = 32, + max_text_len: int = 5, + vocab_size: int = len(VOCAB), + action_dim: int = len(ACTION_NAMES), + latent_dim: int = 96, + image_latent_dim: int = 48, + text_latent_dim: int = 32, + no_text: bool = False, + ): + super().__init__() + self.model_type = model_type + self.no_text = no_text + self.latent_dim = latent_dim + self.image_size = image_size + if model_type in {"swm", "swm_spectral_norm", "swm_selective", "swm_structured", "swm_no_transition", "swm_no_text", "swm_no_stability"}: + self.image_encoder = WaveletImageEncoder(image_size, levels=2, out_dim=image_latent_dim) + self.text_encoder = DCTTextEncoder(vocab_size, max_text_len, emb_dim=24, out_dim=text_latent_dim) + self.image_decoder = WaveletImageDecoder(image_size, levels=2, latent_dim=latent_dim) + elif model_type == "neural_operator": + self.image_encoder = FFTImageEncoder(image_size, out_dim=image_latent_dim) + self.text_encoder = DCTTextEncoder(vocab_size, max_text_len, emb_dim=24, out_dim=text_latent_dim) + self.image_decoder = MLPImageDecoder(image_size, latent_dim=latent_dim) + else: + self.image_encoder = ConvImageEncoder(image_size, out_dim=image_latent_dim) + self.text_encoder = MeanTextEncoder(vocab_size, emb_dim=24, out_dim=text_latent_dim) + self.image_decoder = MLPImageDecoder(image_size, latent_dim=latent_dim) + + if no_text or model_type == "swm_no_text": + text_latent_dim = 0 + self.no_text = True + self.fusion = nn.Sequential( + nn.Linear(image_latent_dim + text_latent_dim, latent_dim), + nn.LayerNorm(latent_dim), + nn.GELU(), + nn.Linear(latent_dim, latent_dim), + nn.LayerNorm(latent_dim), + ) + if model_type == "swm_spectral_norm": + self.transition = SpectralNormConstrainedTransition(latent_dim, action_dim, n_ops=4, rank=12, rho=0.95) + elif model_type == "swm_selective": + self.transition = SelectiveSpectralTransition(latent_dim, action_dim, n_ops=4, rank=12, max_gain=1.05) + elif model_type == "swm_structured": + self.transition = StructuredSpectralTransportTransition(latent_dim, action_dim, rank=12, max_gain=1.05) + elif model_type in {"swm", "swm_no_text", "swm_no_stability"}: + self.transition = LowRankSpectralTransition(latent_dim, action_dim, n_ops=4, rank=12) + elif model_type == "koopman": + self.transition = KoopmanTransition(latent_dim, action_dim) + elif model_type == "transformer": + self.transition = MiniTransformerTransition(latent_dim, action_dim) + else: + self.transition = MLPTransition(latent_dim, action_dim) + self.text_decoder = nn.Linear(latent_dim, max_text_len * vocab_size) + self.max_text_len = max_text_len + self.vocab_size = vocab_size + + def encode(self, image: torch.Tensor, tokens: torch.Tensor) -> torch.Tensor: + zi = self.image_encoder(image) + if self.no_text: + fused = zi + else: + zt = self.text_encoder(tokens) + fused = torch.cat([zi, zt], dim=-1) + return self.fusion(fused) + + def decode_text(self, z: torch.Tensor) -> torch.Tensor: + return self.text_decoder(z).view(z.shape[0], self.max_text_len, self.vocab_size) + + def forward(self, image: torch.Tensor, tokens: torch.Tensor, action: torch.Tensor) -> dict[str, torch.Tensor]: + z = self.encode(image, tokens) + z_next = self.transition(z, action) + return { + "z": z, + "z_pred": z_next, + "image_pred": self.image_decoder(z_next), + "text_logits": self.decode_text(z_next), + } + + def stability_penalty(self) -> torch.Tensor: + return self.transition.stability_penalty() + + +def build_model(name: str, **kwargs) -> WorldModel: + aliases = { + "swm": "swm", + "swm_spectral_norm": "swm_spectral_norm", + "swm_selective": "swm_selective", + "swm_structured": "swm_structured", + "swm_structured_cf": "swm_structured", + "swm_structured_cf_v9": "swm_structured", + "swm_structured_cf_v11": "swm_structured", + "dreamer": "dreamer", + "transformer": "transformer", + "koopman": "koopman", + "neural_operator": "neural_operator", + "no_spectral_state": "dreamer", + "no_spectral_transition": "swm_no_transition", + "no_text_branch": "swm_no_text", + "no_stability_penalty": "swm_no_stability", + } + if name not in aliases: + raise ValueError(f"Unknown model '{name}'. Options: {sorted(aliases)}") + return WorldModel(model_type=aliases[name], **kwargs) diff --git a/spectral_world_models/train.py b/spectral_world_models/train.py new file mode 100644 index 0000000000000000000000000000000000000000..6278e560d181aeaa7c394e5d66ff4c63184432da --- /dev/null +++ b/spectral_world_models/train.py @@ -0,0 +1,293 @@ +''' +train.py is a benchmarking script for training and evaluating spectral world model +variants. It loads transition and sequence datasets, builds a specified model, +trains it with image, latent, text, and stability losses, and evaluates performance +on validation and test splits. The script also runs short rollout prediction tests, +saves the best checkpoint, exports training history to CSV, and writes aggregated +JSON/CSV metrics to the output directory. If the benchmark dataset is missing, +it will generate a synthetic one automatically. + +''' +# Andrew Kiruluta, UC Berkeley, June 2026 + +from __future__ import annotations + +import argparse +import csv +import json +import time +from pathlib import Path +from typing import Any + +import torch +import torch.nn.functional as F +from torch.utils.data import DataLoader +from tqdm import tqdm + +from .data import TransitionDataset, SequenceDataset, generate_benchmark_npz, DatasetConfig, VOCAB, ACTION_NAMES +from .metrics import psnr, ssim_global, gaussian_nll, text_accuracy, count_parameters +from .models import build_model +from .counterfactual import make_cf_training_batch, counterfactual_training_loss +from .v9_causal_generalization import make_trajectory_cf_batch, trajectory_cf_loss +from .v11_causal_calibration import make_calibration_batch, calibrated_trajectory_cf_loss + + +def seed_all(seed: int) -> None: + torch.manual_seed(seed) + torch.set_num_threads(max(1, min(4, torch.get_num_threads()))) + + +def batch_to_device(batch: dict[str, torch.Tensor], device: torch.device) -> dict[str, torch.Tensor]: + return {k: v.to(device) for k, v in batch.items()} + + +def compute_loss(model, batch, lambda_latent: float = 0.15, lambda_text: float = 0.05, lambda_stability: float = 0.01) -> tuple[torch.Tensor, dict[str, float]]: + out = model(batch["image"], batch["tokens"], batch["action"]) + with torch.no_grad(): + target_z = model.encode(batch["next_image"], batch["next_tokens"]) + image_loss = F.mse_loss(out["image_pred"], batch["next_image"]) + latent_loss = F.mse_loss(out["z_pred"], target_z) + text_loss = F.cross_entropy(out["text_logits"].reshape(-1, len(VOCAB)), batch["next_tokens"].reshape(-1), ignore_index=0) + stability = model.stability_penalty() + loss = image_loss + lambda_latent * latent_loss + lambda_text * text_loss + lambda_stability * stability + return loss, { + "loss": float(loss.detach().cpu()), + "image_mse": float(image_loss.detach().cpu()), + "latent_mse": float(latent_loss.detach().cpu()), + "text_ce": float(text_loss.detach().cpu()), + "stability_penalty": float(stability.detach().cpu()), + } + + +@torch.no_grad() +def evaluate_transitions(model, loader, device) -> dict[str, float]: + model.eval() + preds, targets, logits, toks = [], [], [], [] + losses = [] + for batch in loader: + batch = batch_to_device(batch, device) + out = model(batch["image"], batch["tokens"], batch["action"]) + loss, parts = compute_loss(model, batch, lambda_stability=0.0) + losses.append(parts["loss"]) + preds.append(out["image_pred"].cpu()) + targets.append(batch["next_image"].cpu()) + logits.append(out["text_logits"].cpu()) + toks.append(batch["next_tokens"].cpu()) + pred = torch.cat(preds, 0) + target = torch.cat(targets, 0) + text_logits = torch.cat(logits, 0) + text_tokens = torch.cat(toks, 0) + return { + "loss": float(sum(losses) / max(1, len(losses))), + "mse": float(torch.mean((pred - target) ** 2).item()), + "psnr": psnr(pred, target), + "ssim": ssim_global(pred, target), + "nll": gaussian_nll(pred, target), + "text_acc": text_accuracy(text_logits, text_tokens), + } + + +@torch.no_grad() +def evaluate_rollout(model, seq_loader, device, horizon: int = 5) -> dict[str, float]: + """Evaluate image and multimodal information preservation over a rollout. + + In addition to image MSE, report text accuracy decoded from the recurrent + latent and cosine alignment with the target multimodal latent at each step. + """ + model.eval() + image_errors, text_correct, text_total, alignments, times = [], 0, 0, [], [] + for batch in seq_loader: + images = batch["images"].to(device) + tokens = batch["tokens"].to(device) + actions = batch["actions"].to(device) + z = model.encode(images[:, 0], tokens[:, 0]) + h = min(horizon, actions.shape[1]) + start = time.perf_counter() + for t in range(h): + z = model.transition(z, actions[:, t]) + pred_img = model.image_decoder(z) + target_img = images[:, t + 1] + target_tok = tokens[:, t + 1] + image_errors.append(torch.mean((pred_img - target_img) ** 2).detach().cpu()) + logits = model.decode_text(z) + pred_tok = logits.argmax(-1) + mask = target_tok.ne(0) + text_correct += int(((pred_tok == target_tok) & mask).sum().item()) + text_total += int(mask.sum().item()) + target_z = model.encode(target_img, target_tok) + alignments.append(F.cosine_similarity(z, target_z, dim=-1).mean().detach().cpu()) + times.append((time.perf_counter() - start) / max(1, h * images.shape[0])) + mse = torch.stack(image_errors).mean().item() if image_errors else 0.0 + txt = text_correct / max(1, text_total) + align = torch.stack(alignments).mean().item() if alignments else 0.0 + # Bounded joint score: higher is better; reports components separately too. + joint = 0.5 * (1.0 / (1.0 + 30.0 * mse)) + 0.3 * txt + 0.2 * ((align + 1.0) / 2.0) + return { + "rollout_mse": float(mse), + "rollout_text_acc": float(txt), + "rollout_latent_cosine": float(align), + "rollout_joint_score": float(joint), + "time_per_rollout_step_ms": float(1000.0 * sum(times) / max(1, len(times))), + } + + +def train_one_model( + model_name: str, + data_path: str | Path, + out_dir: str | Path, + epochs: int = 3, + batch_size: int = 64, + lr: float = 2e-3, + seed: int = 0, + device: str = "cpu", + rollout_data_path: str | Path | None = None, + rollout_horizons: tuple[int, ...] = (5,), + lambda_stability: float = 0.01, + cf_train_horizon: int = 8, + cf_pairs: int = 64, + cf_every: int = 4, + lambda_cf_dir: float = 0.02, + lambda_cf_mag: float = 0.005, + lambda_cf_branch: float = 0.10, +) -> dict[str, Any]: + seed_all(seed) + device_t = torch.device(device) + out_dir = Path(out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + train_ds = TransitionDataset(data_path, "train") + val_ds = TransitionDataset(data_path, "val") + test_ds = TransitionDataset(data_path, "test") + test_seq = SequenceDataset(rollout_data_path or data_path, "test") + train_loader = DataLoader(train_ds, batch_size=batch_size, shuffle=True, num_workers=0) + val_loader = DataLoader(val_ds, batch_size=batch_size, shuffle=False, num_workers=0) + test_loader = DataLoader(test_ds, batch_size=batch_size, shuffle=False, num_workers=0) + seq_loader = DataLoader(test_seq, batch_size=batch_size, shuffle=False, num_workers=0) + + model = build_model(model_name).to(device_t) + optimizer = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=1e-4) + lambda_stab = 0.0 if model_name == "no_stability_penalty" else lambda_stability + history = [] + best_val = float("inf") + best_path = out_dir / f"{model_name}.pt" + for epoch in range(1, epochs + 1): + model.train() + train_parts = [] + cf_batch = None + if model_name == "swm_structured_cf": + cf_batch = make_cf_training_batch(n=cf_pairs, horizon=cf_train_horizon, seed=7000 + 101*seed + epoch, device=device_t) + elif model_name == "swm_structured_cf_v9": + cf_batch = make_trajectory_cf_batch(n=cf_pairs, horizon=cf_train_horizon, seed=9000 + 101*seed + epoch, device=device_t) + elif model_name == "swm_structured_cf_v11": + cf_batch = make_calibration_batch(n=cf_pairs, horizon=cf_train_horizon, seed=11000 + 101*seed + epoch, device=device_t) + for batch_idx, batch in enumerate(train_loader): + batch = batch_to_device(batch, device_t) + optimizer.zero_grad(set_to_none=True) + loss, parts = compute_loss(model, batch, lambda_stability=lambda_stab) + loss.backward() + torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) + optimizer.step() + train_parts.append(parts) + if cf_batch is not None and batch_idx % max(1, cf_every) == 0: + optimizer.zero_grad(set_to_none=True) + cf_loss, cf_parts = (calibrated_trajectory_cf_loss(model, cf_batch, lambda_vector=lambda_cf_dir, lambda_gain=lambda_cf_mag, lambda_branch=lambda_cf_branch) if model_name == "swm_structured_cf_v11" else trajectory_cf_loss(model, cf_batch, lambda_dir=lambda_cf_dir, lambda_mag=lambda_cf_mag, lambda_branch=lambda_cf_branch) if model_name == "swm_structured_cf_v9" else counterfactual_training_loss(model, cf_batch, lambda_dir=lambda_cf_dir, lambda_mag=lambda_cf_mag, lambda_branch=lambda_cf_branch)) + cf_loss.backward() + torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) + optimizer.step() + parts.update(cf_parts) + train_loss = sum(p["loss"] for p in train_parts) / len(train_parts) + val_metrics = evaluate_transitions(model, val_loader, device_t) + row = {"model": model_name, "epoch": epoch, "train_loss": train_loss, **{f"val_{k}": v for k, v in val_metrics.items()}} + history.append(row) + if val_metrics["loss"] < best_val: + best_val = val_metrics["loss"] + torch.save({"model": model.state_dict(), "model_name": model_name}, best_path) + + ckpt = torch.load(best_path, map_location=device_t) + model.load_state_dict(ckpt["model"]) + val_metrics = evaluate_transitions(model, val_loader, device_t) + test_metrics = evaluate_transitions(model, test_loader, device_t) + rollout_by_horizon = {} + for horizon in rollout_horizons: + rollout_by_horizon[str(horizon)] = evaluate_rollout(model, seq_loader, device_t, horizon=horizon) + # Backward-compatible h5 field when requested. + rollout = rollout_by_horizon.get("5", next(iter(rollout_by_horizon.values()))) + stability_diagnostics = {} + if hasattr(model.transition, "stability_diagnostics"): + stability_diagnostics = model.transition.stability_diagnostics() + result = { + "model": model_name, + "params": count_parameters(model), + "epochs": epochs, + "best_val_loss": best_val, + "val": val_metrics, + "test": test_metrics, + "rollout": rollout, + "rollout_by_horizon": rollout_by_horizon, + "stability_diagnostics": stability_diagnostics, + "history": history, + "counterfactual_training": {"enabled": model_name in {"swm_structured_cf","swm_structured_cf_v9","swm_structured_cf_v11"}, "horizon": cf_train_horizon, "pairs": cf_pairs, "every": cf_every, "lambda_dir": lambda_cf_dir, "lambda_mag": lambda_cf_mag, "lambda_branch": lambda_cf_branch}, + } + with open(out_dir / f"{model_name}_history.csv", "w", newline="") as f: + writer = csv.DictWriter(f, fieldnames=list(history[0].keys())) + writer.writeheader(); writer.writerows(history) + with open(out_dir / f"{model_name}_metrics.json", "w") as f: + json.dump(result, f, indent=2) + return result + + +def run_benchmark( + data_path: str | Path, + out_dir: str | Path, + models: list[str], + epochs: int, + batch_size: int, + seed: int, + device: str, +) -> dict[str, Any]: + out_dir = Path(out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + if not Path(data_path).exists(): + generate_benchmark_npz(data_path, DatasetConfig()) + all_results = [] + for i, name in enumerate(models): + print(f"\n=== Training {name} ===") + result = train_one_model(name, data_path, out_dir, epochs=epochs, batch_size=batch_size, seed=seed + i, device=device) + all_results.append(result) + summary = {"data_path": str(data_path), "epochs": epochs, "models": all_results} + with open(out_dir / "metrics.json", "w") as f: + json.dump(summary, f, indent=2) + # Flat CSV summary. + flat_rows = [] + for r in all_results: + flat_rows.append({ + "model": r["model"], + "params": r["params"], + "test_psnr": r["test"]["psnr"], + "test_ssim": r["test"]["ssim"], + "test_nll": r["test"]["nll"], + "test_mse": r["test"]["mse"], + "test_text_acc": r["test"]["text_acc"], + "rollout_mse_h5": r["rollout"]["rollout_mse"], + "time_per_rollout_step_ms": r["rollout"]["time_per_rollout_step_ms"], + }) + with open(out_dir / "metrics.csv", "w", newline="") as f: + writer = csv.DictWriter(f, fieldnames=list(flat_rows[0].keys())) + writer.writeheader(); writer.writerows(flat_rows) + return summary + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--data", default="data/mini_moving_shapes.npz") + parser.add_argument("--out", default="results") + parser.add_argument("--epochs", type=int, default=3) + parser.add_argument("--batch-size", type=int, default=64) + parser.add_argument("--seed", type=int, default=0) + parser.add_argument("--device", default="cpu") + parser.add_argument("--models", nargs="+", default=["dreamer", "transformer", "koopman", "neural_operator", "swm", "no_spectral_state", "no_spectral_transition", "no_text_branch", "no_stability_penalty"]) + args = parser.parse_args() + run_benchmark(args.data, args.out, args.models, args.epochs, args.batch_size, args.seed, args.device) + + +if __name__ == "__main__": + main() diff --git a/spectral_world_models/v10_cf_audit.py b/spectral_world_models/v10_cf_audit.py new file mode 100644 index 0000000000000000000000000000000000000000..14f4d36f1b384c6ce23b1f4cb53d613be6580f02 --- /dev/null +++ b/spectral_world_models/v10_cf_audit.py @@ -0,0 +1,104 @@ +"""V10 counterfactual benchmark integrity and physics audit. + +The V9 endpoint cosine can be invariant across regimes when the same initial state and +constant left/right interventions produce collinear ground-truth displacement vectors. +V10 makes this visible instead of treating repeated cosine values as proof of a wiring bug. +It records physics metadata, hashes oracle trajectories, uses varied action programs, and +adds trajectory-level oracle/model metrics whose values can respond to physics changes. +""" +from __future__ import annotations +from dataclasses import asdict +import hashlib, json +import numpy as np, torch +from .v8_generalization import _initial_episode, _simulate, _render, _target_com +from .v9_causal_generalization import INTERP, EXTRAP, STRUCTURAL + +REGIMES={"interpolation":INTERP,"extrapolation":EXTRAP,"structural_ood":STRUCTURAL} + +def physics_dict(cfg): + keys=("acceleration","max_speed","occlusion_prob","damping","restitution","image_size") + return {k:float(getattr(cfg,k)) if k!="image_size" else int(getattr(cfg,k)) for k in keys} + +def physics_fingerprint(cfg): + s=json.dumps(physics_dict(cfg),sort_keys=True,separators=(",",":")) + return hashlib.sha256(s.encode()).hexdigest()[:16] + +def action_programs(horizon): + """Paired interventions with turns/reversals; avoids a single collinear endpoint test.""" + h=int(horizon); q=max(1,h//4); r=h-3*q + def fit(x): return (x+[0]*h)[:h] + return [ + ("left_right",[3]*h,[4]*h), + ("up_down",[1]*h,[2]*h), + ("turn_cw",fit([1]*q+[4]*q+[2]*q+[3]*r),fit([1]*q+[3]*q+[2]*q+[4]*r)), + ("impulse_reverse",fit([4]*q+[0]*q+[3]*(h-2*q)),fit([3]*q+[0]*q+[4]*(h-2*q))), + ] + +def _render_state(init,xx,yy,shape,radius): + c=np.zeros_like(init); _render(c,xx,yy,shape,radius,1.0) + _render(c,6.,6.,'circle' if shape=='square' else 'square',3,.45) + c[:,c.shape[1]//2-2:c.shape[1]//2+2]*=.15 + return c + +def oracle_pair(init,state,cfg,acts_a,acts_b): + x,y,vx,vy,shape,radius=state; branches=[] + for acts in (acts_a,acts_b): + xx,yy,vxx,vyy=x,y,vx,vy; frames=[]; xy=[] + for act in acts: + xx,yy,vxx,vyy=_simulate(xx,yy,vxx,vyy,shape,radius,[act],cfg) + frames.append(_render_state(init,xx,yy,shape,radius)); xy.append((xx,yy)) + branches.append((np.stack(frames),np.asarray(xy,np.float32))) + return branches + +def trajectory_hash(a,b): + h=hashlib.sha256(); h.update(np.asarray(a,dtype=np.float32).tobytes()); h.update(np.asarray(b,dtype=np.float32).tobytes()); return h.hexdigest() + +def build_oracle_audit(n=32,horizon=30,seed=10101): + """Use identical initial latent physical states across regimes, then vary physics only.""" + rng=np.random.default_rng(seed); base=INTERP; episodes=[] + for _ in range(n): episodes.append(_initial_episode(rng,base)) + rows=[] + for rn,cfg in REGIMES.items(): + for pi,(pname,aa,bb) in enumerate(action_programs(horizon)): + effects=[]; mags=[]; hashes=[] + for init,tok,state in episodes: + br=oracle_pair(init,state,cfg,aa,bb); xa=br[0][1]; xb=br[1][1]; eff=xb-xa + effects.append(eff); mags.append(np.linalg.norm(eff,axis=1)); hashes.append(trajectory_hash(xa,xb)) + E=np.stack(effects); M=np.stack(mags) + rows.append({"regime":rn,"program":pname,"physics_fingerprint":physics_fingerprint(cfg), + "trajectory_set_hash":hashlib.sha256(''.join(hashes).encode()).hexdigest(), + "oracle_effect_rms":float(np.sqrt(np.mean(E**2))),"oracle_effect_final_mag":float(M[:,-1].mean()), + "oracle_effect_path_mag":float(M.mean()),"oracle_effect_temporal_variation":float(np.diff(E,axis=1).std()), + **{f"physics_{k}":v for k,v in physics_dict(cfg).items()}}) + return rows + +def assert_audit_integrity(rows): + fps={r['regime']:r['physics_fingerprint'] for r in rows}; assert len(set(fps.values()))==len(REGIMES) + for p in {r['program'] for r in rows}: + rr=[r for r in rows if r['program']==p] + assert len({r['trajectory_set_hash'] for r in rr})==len(REGIMES), f"oracle trajectories identical across regimes for {p}" + vals=np.asarray([[r['oracle_effect_rms'],r['oracle_effect_path_mag'],r['oracle_effect_temporal_variation']] for r in rr]) + assert np.max(np.ptp(vals,axis=0))>1e-5, f"oracle effects insensitive to physics for {p}" + return True + +@torch.no_grad() +def evaluate_counterfactual_audit(model,device,cfg,n=48,horizon=30,seed=10101): + rng=np.random.default_rng(seed); episodes=[_initial_episode(rng,INTERP) for _ in range(n)]; model.eval() + final_mse=[]; endpoint_cos=[]; endpoint_ratio=[]; traj_cos=[]; traj_ratio=[] + for init,tok,state in episodes: + image=torch.tensor(init[None,None],device=device); tokens=torch.tensor(tok[None],device=device); z0=model.encode(image,tokens) + for _,aa,bb in action_programs(horizon): + oracle=oracle_pair(init,state,cfg,aa,bb); gt=[]; pred=[] + for acts,(frames,_) in zip((aa,bb),oracle): + z=z0.clone(); pp=[] + for t,act in enumerate(acts): + z=model.transition(z,torch.tensor([act],device=device)); pp.append(_target_com(model.image_decoder(z))[0]) + pred.append(torch.stack(pp)); gt.append(torch.tensor(oracle[0 if acts is aa else 1][1],device=device,dtype=pp[0].dtype)) + final=torch.tensor(frames[-1][None,None],device=device); final_mse.append(float(((model.image_decoder(z)-final)**2).mean())) + pe=pred[1]-pred[0]; ge=gt[1]-gt[0]; pn=pe.norm(dim=-1); gn=ge.norm(dim=-1) + c=((pe*ge).sum(-1)/(pn*gn).clamp_min(1e-6)); endpoint_cos.append(float(c[-1])); endpoint_ratio.append(float(pn[-1]/gn[-1].clamp_min(1e-6))) + valid=gn>1e-4 + if valid.any(): traj_cos.append(float(c[valid].mean())); traj_ratio.append(float((pn[valid]/gn[valid]).mean())) + return {"cf_audit_final_image_mse":float(np.mean(final_mse)),"cf_audit_endpoint_cosine":float(np.mean(endpoint_cos)), + "cf_audit_endpoint_magnitude_ratio":float(np.mean(endpoint_ratio)),"cf_audit_trajectory_cosine":float(np.mean(traj_cos)), + "cf_audit_trajectory_magnitude_ratio":float(np.mean(traj_ratio))} diff --git a/spectral_world_models/v11_causal_calibration.py b/spectral_world_models/v11_causal_calibration.py new file mode 100644 index 0000000000000000000000000000000000000000..634c0eae43abd07fae2dda990fc0a82232ec792d --- /dev/null +++ b/spectral_world_models/v11_causal_calibration.py @@ -0,0 +1,65 @@ +"""V11 causal-gain calibration: scale-aware trajectory intervention training/evaluation.""" +from __future__ import annotations +import numpy as np, torch +from .v8_generalization import _target_com, _initial_episode +from .v9_causal_generalization import make_trajectory_cf_batch, INTERP, EXTRAP, STRUCTURAL +from .v10_cf_audit import action_programs, oracle_pair + +REGIMES={"interpolation":INTERP,"extrapolation":EXTRAP,"structural_ood":STRUCTURAL} + +def calibrated_trajectory_cf_loss(model,batch,lambda_vector=.03,lambda_gain=.01,lambda_branch=.08,discount=.97): + """Match normalized causal effect vectors and their gain over the full trajectory. + + Vector error is normalized by oracle effect scale, avoiding domination by large-displacement + examples. Gain uses symmetric log-ratio. Branch reconstruction keeps decoded states realistic. + """ + z0=model.encode(batch['image'],batch['tokens']); B=z0.shape[0]; zs=[z0,z0] + total=0.; vs=gs=bs=0.; ws=0. + for t in range(int(batch['horizon'])): + preds=[] + for j,act in enumerate((3,4)): + a=torch.full((B,),act,device=z0.device,dtype=torch.long); zs[j]=model.transition(zs[j],a); preds.append(model.image_decoder(zs[j])) + gtL=batch['left'][:,t]; gtR=batch['right'][:,t] + pe=_target_com(preds[1])-_target_com(preds[0]); ge=_target_com(gtR)-_target_com(gtL) + gn=ge.norm(dim=-1).clamp_min(.25); pn=pe.norm(dim=-1).clamp_min(1e-5) + scale=gn.detach().unsqueeze(-1) + vector=((pe-ge)/scale).square().sum(-1).mean() + gain=torch.abs(torch.log(pn/gn)).clamp(max=4.).mean() + branch=.5*((preds[0]-gtL).square().mean()+(preds[1]-gtR).square().mean()) + w=discount**t; total += w*(lambda_vector*vector+lambda_gain*gain+lambda_branch*branch) + vs += w*vector; gs += w*gain; bs += w*branch; ws += w + return total/ws,{"cf_cal_vector_loss":float((vs/ws).detach()),"cf_cal_gain_loss":float((gs/ws).detach()),"cf_cal_branch_mse":float((bs/ws).detach())} + +def make_calibration_batch(n=64,horizon=12,seed=11111,device='cpu'): + return make_trajectory_cf_batch(n=n,horizon=horizon,seed=seed,device=device) + +def _lin_calibration(pred,true): + p=np.asarray(pred,float); y=np.asarray(true,float) + if len(p)<2 or np.std(p)<1e-12: return 0.,float(y.mean() if len(y) else 0.),0. + slope,intercept=np.polyfit(p,y,1); corr=float(np.corrcoef(p,y)[0,1]) if np.std(y)>1e-12 else 0. + return float(slope),float(intercept),corr + +@torch.no_grad() +def evaluate_causal_calibration(model,device,cfg,n=64,horizon=30,seed=11101): + """Evaluate trajectory direction plus magnitude calibration against oracle physics.""" + rng=np.random.default_rng(seed); episodes=[_initial_episode(rng,INTERP) for _ in range(n)]; model.eval() + cosines=[]; ratios=[]; pred_mags=[]; true_mags=[] + for init,tok,state in episodes: + image=torch.tensor(init[None,None],device=device); tokens=torch.tensor(tok[None],device=device); z0=model.encode(image,tokens) + for _,aa,bb in action_programs(horizon): + oracle=oracle_pair(init,state,cfg,aa,bb); pred=[]; gt=[] + for acts,(frames,xy) in zip((aa,bb),oracle): + z=z0.clone(); pp=[] + for act in acts: + z=model.transition(z,torch.tensor([act],device=device)); pp.append(_target_com(model.image_decoder(z))[0]) + pred.append(torch.stack(pp)); gt.append(torch.tensor(xy,device=device,dtype=pp[0].dtype)) + pe=pred[1]-pred[0]; ge=gt[1]-gt[0]; pn=pe.norm(dim=-1); gn=ge.norm(dim=-1); valid=gn>.25 + if valid.any(): + c=(pe*ge).sum(-1)/(pn*gn).clamp_min(1e-6) + cosines.extend(c[valid].cpu().tolist()); ratios.extend((pn[valid]/gn[valid]).cpu().tolist()) + pred_mags.extend(pn[valid].cpu().tolist()); true_mags.extend(gn[valid].cpu().tolist()) + slope,intercept,corr=_lin_calibration(pred_mags,true_mags) + ratio=np.asarray(ratios,float) + return {"trajectory_cosine":float(np.mean(cosines)),"magnitude_ratio":float(ratio.mean()),"magnitude_ratio_mae":float(np.mean(np.abs(ratio-1))), + "calibration_slope_true_on_pred":slope,"calibration_intercept":intercept,"magnitude_correlation":corr, + "log_gain_calibration_error":float(np.mean(np.abs(np.log(np.clip(ratio,1e-6,None)))))} diff --git a/spectral_world_models/v12_causal_decomposition.py b/spectral_world_models/v12_causal_decomposition.py new file mode 100644 index 0000000000000000000000000000000000000000..ebbf282996783402478992ed3d01328a6b48d96f --- /dev/null +++ b/spectral_world_models/v12_causal_decomposition.py @@ -0,0 +1,88 @@ +"""V12 Causal Response Decomposition. + +No new learned architecture or loss is introduced. This module decomposes causal-response +errors by regime, action program, horizon, oracle effect size, initial speed, boundary contact, +and central-occlusion-zone exposure. It also provides paired bootstrap confidence intervals +for model differences on matched episodes/programs. +""" +from __future__ import annotations +import numpy as np, torch +from .v8_generalization import _initial_episode, _target_com +from .v10_cf_audit import action_programs, oracle_pair +from .v11_causal_calibration import REGIMES + +HORIZON_CUTS=(5,10,20,30) + +def _bin3(x, cuts, labels=("low","mid","high")): + return labels[int(np.digitize([x],cuts)[0])] + +def _episode_tags(state, oracle_xy_a, oracle_xy_b, cfg, effect_path): + x,y,vx,vy,shape,radius=state + speed=float(np.hypot(vx,vy)); effect=float(np.mean(np.linalg.norm(effect_path,axis=-1))) + lo=float(radius)+0.55; hi=float(cfg.image_size-1-radius)-0.55 + both=np.concatenate([oracle_xy_a,oracle_xy_b],axis=0) + boundary=bool(np.any((both[:,0]<=lo)|(both[:,0]>=hi)|(both[:,1]<=lo)|(both[:,1]>=hi))) + center=float(cfg.image_size)//2 + occlusion_zone=bool(np.any(np.abs(both[:,0]-center)<=2.5)) + return {"initial_speed":speed,"oracle_effect_path_mag":effect,"boundary_contact":boundary,"occlusion_zone_exposure":occlusion_zone} + +@torch.no_grad() +def decompose_model(model,device,cfg,regime,n=64,horizon=30,seed=12101): + """Return one matched observation per episode/program/horizon cut.""" + rng=np.random.default_rng(seed); episodes=[_initial_episode(rng,REGIMES["interpolation"]) for _ in range(n)]; model.eval(); rows=[] + cuts=sorted(set(h for h in HORIZON_CUTS if h<=horizon) | {horizon}) + for eid,(init,tok,state) in enumerate(episodes): + image=torch.tensor(init[None,None],device=device); tokens=torch.tensor(tok[None],device=device); z0=model.encode(image,tokens) + for pname,aa,bb in action_programs(horizon): + oracle=oracle_pair(init,state,cfg,aa,bb); pred=[]; gt=[] + for acts,(_,xy) in zip((aa,bb),oracle): + z=z0.clone(); pp=[] + for act in acts: + z=model.transition(z,torch.tensor([act],device=device)); pp.append(_target_com(model.image_decoder(z))[0]) + pred.append(torch.stack(pp)); gt.append(torch.tensor(xy,device=device,dtype=pp[0].dtype)) + pe=pred[1]-pred[0]; ge=gt[1]-gt[0]; pn=pe.norm(dim=-1); gn=ge.norm(dim=-1) + tags=_episode_tags(state,oracle[0][1],oracle[1][1],cfg,(oracle[1][1]-oracle[0][1])) + for h in cuts: + valid=gn[:h]>.25 + if not valid.any(): continue + c=(pe[:h]*ge[:h]).sum(-1)/(pn[:h]*gn[:h]).clamp_min(1e-6); ratio=pn[:h]/gn[:h].clamp_min(1e-6) + rv=ratio[valid].detach().cpu().numpy(); cv=c[valid].detach().cpu().numpy() + rows.append({"regime":regime,"episode":eid,"program":pname,"horizon":h, + "trajectory_cosine":float(cv.mean()),"magnitude_ratio":float(rv.mean()), + "magnitude_ratio_mae":float(np.abs(rv-1).mean()),"log_gain_error":float(np.abs(np.log(np.clip(rv,1e-6,None))).mean()),**tags}) + # deterministic within-regime tertiles, based on matched oracle quantities + speeds=np.asarray([r['initial_speed'] for r in rows]); effects=np.asarray([r['oracle_effect_path_mag'] for r in rows]) + sc=np.quantile(speeds,[1/3,2/3]); ec=np.quantile(effects,[1/3,2/3]) + for r in rows: + r['speed_bin']=_bin3(r['initial_speed'],sc); r['effect_bin']=_bin3(r['oracle_effect_path_mag'],ec) + return rows + +def summarize_strata(rows): + """Long-format summaries, deliberately avoiding a composite score.""" + specs=[('overall',()),('program',('program',)),('horizon',('horizon',)),('program_horizon',('program','horizon')), + ('effect_bin',('effect_bin',)),('speed_bin',('speed_bin',)),('boundary_contact',('boundary_contact',)), + ('occlusion_zone',('occlusion_zone_exposure',))] + metrics=('trajectory_cosine','magnitude_ratio','magnitude_ratio_mae','log_gain_error'); out=[] + for sname,keys in specs: + groups={} + for r in rows: groups.setdefault(tuple(r[k] for k in keys),[]).append(r) + for kvals,rs in groups.items(): + base={'stratum_type':sname,'n':len(rs)}; base.update(dict(zip(keys,kvals))) + for m in metrics: + x=np.asarray([r[m] for r in rs],float); base[m+'_mean']=float(x.mean()); base[m+'_std']=float(x.std(ddof=1)) if len(x)>1 else 0. + out.append(base) + return out + +def paired_bootstrap_delta(v9_rows,control_rows,metric='trajectory_cosine',n_boot=2000,seed=12121): + """Episode-clustered paired bootstrap: delta = V9 - control on matched keys.""" + key=lambda r:(r['regime'],r['episode'],r['program'],r['horizon']) + a={key(r):r for r in v9_rows}; b={key(r):r for r in control_rows}; ks=sorted(set(a)&set(b)) + if not ks: raise ValueError('no matched decomposition rows') + episodes=sorted({k[1] for k in ks}); byep={e:[k for k in ks if k[1]==e] for e in episodes} + def d(keys): return float(np.mean([a[k][metric]-b[k][metric] for k in keys])) + point=d(ks); rng=np.random.default_rng(seed); boots=[] + for _ in range(n_boot): + sample=rng.choice(episodes,size=len(episodes),replace=True); sk=[] + for e in sample: sk.extend(byep[int(e)]) + boots.append(d(sk)) + lo,hi=np.quantile(boots,[.025,.975]); return {'metric':metric,'delta_v9_minus_control':point,'ci95_low':float(lo),'ci95_high':float(hi),'n_matched':len(ks),'n_episodes':len(episodes),'n_boot':n_boot} diff --git a/spectral_world_models/v13_axis_equivariance.py b/spectral_world_models/v13_axis_equivariance.py new file mode 100644 index 0000000000000000000000000000000000000000..5ee0a431cb2b3a0ee7371b74a9a1d0712fc7ec1c --- /dev/null +++ b/spectral_world_models/v13_axis_equivariance.py @@ -0,0 +1,136 @@ +"""V13 Axis Symmetry & Intervention Equivariance Audit. + +This release freezes V9 learning and asks whether the learned causal response commutes +with exact scene/action symmetries. It also constructs collision episodes explicitly, +rather than relying on incidental boundary contacts. +""" +from __future__ import annotations +import numpy as np, torch +from .data import _text_tokens +from .v8_generalization import _initial_episode, _target_com +from .v10_cf_audit import action_programs, oracle_pair +from .v11_causal_calibration import REGIMES + +TRANSFORMS=("identity","rot90","flip_x","flip_y","swap_xy") +ACTION_VEC={0:(0,0),1:(0,-1),2:(0,1),3:(-1,0),4:(1,0)} +VEC_ACTION={v:k for k,v in ACTION_VEC.items()} + +def transform_xy(xy,n,name): + a=np.asarray(xy,dtype=np.float32); x=a[...,0]; y=a[...,1] + if name=="identity": z=np.stack([x,y],-1) + elif name=="rot90": z=np.stack([y,(n-1)-x],-1) + elif name=="flip_x": z=np.stack([(n-1)-x,y],-1) + elif name=="flip_y": z=np.stack([x,(n-1)-y],-1) + elif name=="swap_xy": z=np.stack([y,x],-1) + else: raise ValueError(name) + return z + +def transform_vec(v,name): + a=np.asarray(v,dtype=np.float32); x=a[...,0]; y=a[...,1] + if name=="identity": z=np.stack([x,y],-1) + elif name=="rot90": z=np.stack([y,-x],-1) + elif name=="flip_x": z=np.stack([-x,y],-1) + elif name=="flip_y": z=np.stack([x,-y],-1) + elif name=="swap_xy": z=np.stack([y,x],-1) + else: raise ValueError(name) + return z + +def inverse_vec(v,name): + if name in ("identity","flip_x","flip_y","swap_xy"): return transform_vec(v,name) + a=np.asarray(v,dtype=np.float32); x=a[...,0]; y=a[...,1] + return np.stack([-y,x],-1) # inverse of CCW image-coordinate rotation + +def transform_action(a,name): + v=transform_vec(np.asarray(ACTION_VEC[int(a)],np.float32),name) + return VEC_ACTION[(int(round(float(v[0]))),int(round(float(v[1]))))] + +def transform_actions(actions,name): return [transform_action(a,name) for a in actions] + +def transform_image(img,name): + if name=="identity": return np.array(img,copy=True) + if name=="rot90": return np.rot90(img,k=1).copy() + if name=="flip_x": return np.fliplr(img).copy() + if name=="flip_y": return np.flipud(img).copy() + if name=="swap_xy": return np.asarray(img).T.copy() + raise ValueError(name) + +def transform_episode(init,state,cfg,name): + x,y,vx,vy,shape,radius=state; p=transform_xy(np.asarray([x,y]),cfg.image_size,name); v=transform_vec(np.asarray([vx,vy]),name) + image=transform_image(init,name) + speed=2 if abs(float(v[0]))+abs(float(v[1]))>=3 else 1 + tok=_text_tokens(shape,0,speed,int(round(float(p[0]))),int(round(float(p[1]))),cfg.image_size,cfg.max_text_len) + return image,tok,(float(p[0]),float(p[1]),float(v[0]),float(v[1]),shape,radius) + +def _torch_transform_vec(v,name,inverse=False): + x,y=v[...,0],v[...,1] + if name=="identity": return torch.stack((x,y),-1) + if name=="rot90": return torch.stack((-y,x),-1) if inverse else torch.stack((y,-x),-1) + if name=="flip_x": return torch.stack((-x,y),-1) + if name=="flip_y": return torch.stack((x,-y),-1) + if name=="swap_xy": return torch.stack((y,x),-1) + raise ValueError(name) + +@torch.no_grad() +def _pred_effect(model,device,init,tok,aa,bb): + im=torch.tensor(init[None,None],device=device); tt=torch.tensor(tok[None],device=device); z0=model.encode(im,tt); out=[] + for acts in (aa,bb): + z=z0.clone(); xy=[] + for a in acts: + z=model.transition(z,torch.tensor([a],device=device)); xy.append(_target_com(model.image_decoder(z))[0]) + out.append(torch.stack(xy)) + return out[1]-out[0] + +def _effect_metrics(pe,ge): + pn=pe.norm(dim=-1); gn=ge.norm(dim=-1); valid=gn>.25 + if not bool(valid.any()): return (float('nan'),float('nan')) + cos=((pe*ge).sum(-1)/(pn*gn).clamp_min(1e-6))[valid].mean(); rat=(pn/gn.clamp_min(1e-6))[valid].mean() + return float(cos),float(rat) + +@torch.no_grad() +def audit_axis_equivariance(model,device,cfg,regime,n=64,horizon=30,seed=13101): + rng=np.random.default_rng(seed); episodes=[_initial_episode(rng,REGIMES["interpolation"]) for _ in range(n)]; rows=[]; model.eval() + for eid,(init,tok,state) in enumerate(episodes): + for pname,aa,bb in action_programs(horizon): + base_or=oracle_pair(init,state,cfg,aa,bb); base_ge=torch.tensor(base_or[1][1]-base_or[0][1],device=device); base_pe=_pred_effect(model,device,init,tok,aa,bb) + bc,br=_effect_metrics(base_pe,base_ge) + for tr in TRANSFORMS: + ti,tt,ts=transform_episode(init,state,cfg,tr); ta=transform_actions(aa,tr); tb=transform_actions(bb,tr) + tor=oracle_pair(ti,ts,cfg,ta,tb); tge=torch.tensor(tor[1][1]-tor[0][1],device=device); tpe=_pred_effect(model,device,ti,tt,ta,tb) + tc,tratio=_effect_metrics(tpe,tge) + aligned=_torch_transform_vec(tpe,tr,inverse=True); an=aligned.norm(dim=-1); bn=base_pe.norm(dim=-1); valid=(an>.05)&(bn>.05) + eqcos=float(((aligned*base_pe).sum(-1)/(an*bn).clamp_min(1e-6))[valid].mean()) if bool(valid.any()) else float('nan') + eqerr=float((aligned-base_pe).norm(dim=-1).mean()/base_pe.norm(dim=-1).mean().clamp_min(1e-6)) + # Oracle commutation residual should be numerical noise for isotropic dynamics. + oge=_torch_transform_vec(tge,tr,inverse=True); oracle_eq=float((oge-base_ge).norm(dim=-1).mean()) + rows.append(dict(regime=regime,episode=eid,program=pname,transform=tr,base_cosine=bc,transformed_cosine=tc, + cosine_drop=tc-bc,base_magnitude_ratio=br,transformed_magnitude_ratio=tratio, + equivariance_effect_cosine=eqcos,equivariance_relative_error=eqerr,oracle_equivariance_error=oracle_eq)) + return rows + +def make_collision_episodes(cfg,n=64,seed=13201): + """Guaranteed near-wall, wall-directed states; alternates all four walls.""" + rng=np.random.default_rng(seed); out=[] + for i in range(n): + init,tok,state=_initial_episode(rng,REGIMES['interpolation']); x,y,vx,vy,shape,r=state; wall=i%4 + if wall==0: x=r+1.25; vx=-2. + elif wall==1: x=cfg.image_size-r-2.25; vx=2. + elif wall==2: y=r+1.25; vy=-2. + else: y=cfg.image_size-r-2.25; vy=2. + # Keep the visual nuisance structure but place the controlled object at forced state. + from .v10_cf_audit import _render_state + init=_render_state(np.zeros_like(init),x,y,shape,r) + speed=2 if abs(vx)+abs(vy)>=3 else 1; tok=_text_tokens(shape,0,speed,int(round(x)),int(round(y)),cfg.image_size,cfg.max_text_len) + out.append((init,tok,(x,y,vx,vy,shape,r),('left','right','top','bottom')[wall])) + return out + +@torch.no_grad() +def audit_forced_collisions(model,device,cfg,regime,n=64,horizon=15,seed=13201): + rows=[]; model.eval() + for eid,(init,tok,state,wall) in enumerate(make_collision_episodes(cfg,n,seed)): + for pname,aa,bb in action_programs(horizon): + oracle=oracle_pair(init,state,cfg,aa,bb); ge=torch.tensor(oracle[1][1]-oracle[0][1],device=device); pe=_pred_effect(model,device,init,tok,aa,bb); c,r=_effect_metrics(pe,ge) + # verify actual contact from oracle paths + both=np.concatenate([oracle[0][1],oracle[1][1]],0); rad=state[-1]; lo=rad+1.; hi=cfg.image_size-rad-2. + contact=bool(np.any((both[:,0]<=lo+.25)|(both[:,0]>=hi-.25)|(both[:,1]<=lo+.25)|(both[:,1]>=hi-.25))) + rows.append(dict(regime=regime,episode=eid,wall=wall,program=pname,trajectory_cosine=c,magnitude_ratio=r,boundary_contact=contact)) + return rows diff --git a/spectral_world_models/v14_external_control.py b/spectral_world_models/v14_external_control.py new file mode 100644 index 0000000000000000000000000000000000000000..6a3b1afb1e0501d196772e844b696b316743c37e --- /dev/null +++ b/spectral_world_models/v14_external_control.py @@ -0,0 +1,131 @@ +"""V14 external-control benchmark utilities. + +Freezes the V9/SWM architecture family and evaluates it on Gymnasium Classic Control. +The benchmark is intentionally outside the project's synthetic Moving Shapes simulator. +""" +from __future__ import annotations +from dataclasses import dataclass +from pathlib import Path +import json +import numpy as np +import torch +import torch.nn.functional as F +from torch.utils.data import Dataset + +SUPPORTED_ENVS = ("CartPole-v1", "MountainCar-v0", "Acrobot-v1") + +@dataclass(frozen=True) +class ExternalConfig: + env_id: str = "CartPole-v1" + image_size: int = 32 + seq_len: int = 31 + max_text_len: int = 8 + train_episodes: int = 256 + val_episodes: int = 64 + test_episodes: int = 64 + seed: int = 1400 + + +def require_gymnasium(): + try: + import gymnasium as gym + except ImportError as e: + raise RuntimeError("V14 requires Gymnasium Classic Control. Install with: pip install -e '.[external]' ") from e + return gym + + +def rgb_to_gray32(rgb: np.ndarray, image_size: int = 32) -> np.ndarray: + x = torch.as_tensor(rgb, dtype=torch.float32).permute(2,0,1).unsqueeze(0) / 255.0 + # luminance before resize, [1,1,H,W] + x = 0.2989*x[:,0:1] + 0.5870*x[:,1:2] + 0.1140*x[:,2:3] + x = F.interpolate(x, size=(image_size,image_size), mode="bilinear", align_corners=False) + return x.squeeze(0).numpy().astype(np.float32) + + +def build_vocab(env_id: str, obs_dim: int, n_bins: int = 8) -> list[str]: + vocab = ["", f"env:{env_id}"] + for j in range(obs_dim): + vocab.extend([f"o{j}:b{k}" for k in range(n_bins)]) + return vocab + + +def obs_to_tokens(obs: np.ndarray, low: np.ndarray, high: np.ndarray, env_id: str, max_len: int=8, n_bins: int=8) -> np.ndarray: + obs = np.asarray(obs, dtype=np.float32).reshape(-1) + low = np.asarray(low, dtype=np.float32).reshape(-1) + high = np.asarray(high, dtype=np.float32).reshape(-1) + toks = [1] + for j, v in enumerate(obs[:max_len-1]): + lo, hi = low[j], high[j] + if not np.isfinite(lo) or not np.isfinite(hi) or hi <= lo: + # smooth bounded surrogate for unbounded velocity dimensions + q = (np.tanh(float(v)) + 1.0) * 0.5 + else: + q = (float(v)-float(lo)) / max(1e-6, float(hi-lo)) + b = int(np.clip(np.floor(q*n_bins), 0, n_bins-1)) + toks.append(2 + j*n_bins + b) + toks = toks[:max_len] + [0]*max(0,max_len-len(toks)) + return np.asarray(toks, dtype=np.int64) + + +def _rollout_episode(env, cfg: ExternalConfig, rng: np.random.Generator, seed: int): + obs, _ = env.reset(seed=seed) + low, high = env.observation_space.low, env.observation_space.high + imgs=[]; toks=[]; acts=[]; rews=[] + for t in range(cfg.seq_len): + imgs.append(rgb_to_gray32(env.render(), cfg.image_size)) + toks.append(obs_to_tokens(obs, low, high, cfg.env_id, cfg.max_text_len)) + if t == cfg.seq_len-1: break + a = int(rng.integers(0, env.action_space.n)) + obs, reward, terminated, truncated, _ = env.step(a) + acts.append(a); rews.append(float(reward)) + if terminated or truncated: + # restart to keep fixed-length tensors; reset is part of the observed external process. + obs, _ = env.reset(seed=seed+10000+t) + return np.stack(imgs), np.stack(toks), np.asarray(acts,np.int64), np.asarray(rews,np.float32) + + +def collect_external_npz(path: str|Path, cfg: ExternalConfig) -> dict: + gym = require_gymnasium() + if cfg.env_id not in SUPPORTED_ENVS: + raise ValueError(f"Unsupported env {cfg.env_id}; choose from {SUPPORTED_ENVS}") + path=Path(path); path.parent.mkdir(parents=True, exist_ok=True) + env=gym.make(cfg.env_id, render_mode="rgb_array") + rng=np.random.default_rng(cfg.seed) + splits={} + counts={"train":cfg.train_episodes,"val":cfg.val_episodes,"test":cfg.test_episodes} + cursor=0 + try: + for split,n in counts.items(): + rows=[_rollout_episode(env,cfg,rng,cfg.seed+cursor+i) for i in range(n)] + cursor += n + splits[f"{split}_images"]=np.stack([r[0] for r in rows]) + splits[f"{split}_tokens"]=np.stack([r[1] for r in rows]) + splits[f"{split}_actions"]=np.stack([r[2] for r in rows]) + splits[f"{split}_rewards"]=np.stack([r[3] for r in rows]) + obs_dim=int(np.prod(env.observation_space.shape)) + vocab=build_vocab(cfg.env_id, obs_dim) + meta={"env_id":cfg.env_id,"obs_dim":obs_dim,"action_dim":int(env.action_space.n),"vocab_size":len(vocab),**cfg.__dict__} + np.savez_compressed(path, **splits, vocab=np.asarray(vocab), action_names=np.asarray([f"a{i}" for i in range(env.action_space.n)]), config=json.dumps(meta)) + return meta + finally: + env.close() + + +class ExternalTransitionDataset(Dataset): + def __init__(self,path,split): + d=np.load(path,allow_pickle=True); self.images=d[f"{split}_images"]; self.tokens=d[f"{split}_tokens"]; self.actions=d[f"{split}_actions"] + self.index=[(i,t) for i in range(len(self.images)) for t in range(self.images.shape[1]-1)] + def __len__(self): return len(self.index) + def __getitem__(self,k): + i,t=self.index[k] + return {"image":torch.from_numpy(self.images[i,t]),"tokens":torch.from_numpy(self.tokens[i,t]).long(),"action":torch.tensor(self.actions[i,t]).long(),"next_image":torch.from_numpy(self.images[i,t+1]),"next_tokens":torch.from_numpy(self.tokens[i,t+1]).long()} + +class ExternalSequenceDataset(Dataset): + def __init__(self,path,split): + d=np.load(path,allow_pickle=True); self.images=d[f"{split}_images"]; self.tokens=d[f"{split}_tokens"]; self.actions=d[f"{split}_actions"] + def __len__(self): return len(self.images) + def __getitem__(self,i): return {"images":torch.from_numpy(self.images[i]),"tokens":torch.from_numpy(self.tokens[i]).long(),"actions":torch.from_numpy(self.actions[i]).long()} + + +def load_meta(path): + d=np.load(path,allow_pickle=True); return json.loads(str(d["config"])) diff --git a/spectral_world_models/v8_generalization.py b/spectral_world_models/v8_generalization.py new file mode 100644 index 0000000000000000000000000000000000000000..3a756afbc50eb588bfe77a1f52a305663c5c9465 --- /dev/null +++ b/spectral_world_models/v8_generalization.py @@ -0,0 +1,98 @@ +"""V8 frozen-architecture generalization, intervention, and planning evaluation.""" +from __future__ import annotations +import numpy as np, torch +from .hard_dynamics import HardDynamicsConfig, _render, _bounce +from .data import _text_tokens + +OOD_CONFIG = HardDynamicsConfig(seq_len=101, train_sequences=1, val_sequences=1, test_sequences=96, + seed=8017, occlusion_prob=.28, acceleration=1.10, max_speed=4.0) + +def _target_com(img: torch.Tensor) -> torch.Tensor: + """COM of the bright controlled object; suppress the 0.45-intensity distractor.""" + w=(img[:,0]-.55).clamp_min(0); B,H,W=w.shape + xs=torch.arange(W,device=w.device,dtype=w.dtype); ys=torch.arange(H,device=w.device,dtype=w.dtype) + mass=w.sum((1,2)).clamp_min(1e-6) + return torch.stack(((w.sum(1)*xs).sum(1)/mass,(w.sum(2)*ys).sum(1)/mass),-1) + +def _simulate(x,y,vx,vy,shape,radius,actions,cfg): + for act in actions: + ax=ay=0. + if act==1: ay=-cfg.acceleration + elif act==2: ay=cfg.acceleration + elif act==3: ax=-cfg.acceleration + elif act==4: ax=cfg.acceleration + vx=float(np.clip((vx+ax)*getattr(cfg,"damping",1.0),-cfg.max_speed,cfg.max_speed)); vy=float(np.clip((vy+ay)*getattr(cfg,"damping",1.0),-cfg.max_speed,cfg.max_speed)) + x+=vx; y+=vy; lo=radius+1; hi=cfg.image_size-radius-2 + x,vx=_bounce(x,vx,lo,hi); y,vy=_bounce(y,vy,lo,hi); vx*=getattr(cfg,"restitution",1.0); vy*=getattr(cfg,"restitution",1.0) + return x,y,vx,vy + +def _initial_episode(rng,cfg): + shape='square' if rng.random()<.5 else 'circle'; radius=3 if shape=='circle' else 4 + x=float(rng.integers(radius+4,cfg.image_size-radius-4)); y=float(rng.integers(radius+4,cfg.image_size-radius-4)) + vx=float(rng.choice([-2.,-1.,1.,2.])); vy=float(rng.choice([-2.,-1.,1.,2.])) + c=np.zeros((cfg.image_size,cfg.image_size),np.float32); _render(c,x,y,shape,radius,1.0) + _render(c,6.,6.,'circle' if shape=='square' else 'square',3,.45); c[:,cfg.image_size//2-2:cfg.image_size//2+2]*=.15 + tok=_text_tokens(shape,0,2 if abs(vx)+abs(vy)>=3 else 1,int(round(x)),int(round(y)),cfg.image_size,cfg.max_text_len) + return c,tok,(x,y,vx,vy,shape,radius) + +def _candidate_actions(h): + # Fixed candidate library: persistent controls plus two-phase turns and neutral. + q=max(1,h//2) + return [ + [0]*h,[1]*h,[2]*h,[3]*h,[4]*h, + [1]*q+[4]*(h-q),[4]*q+[1]*(h-q),[2]*q+[3]*(h-q),[3]*q+[2]*(h-q), + ] + +@torch.no_grad() +def evaluate_planning(model,device,n=96,horizon=20,seed=808,cfg=None): + """Model-predictive action selection toward a random corner target. + + Candidate sequences are scored only from imagined final states. The selected + sequence is then evaluated in the ground-truth simulator. Reports normalized + endpoint regret and oracle-action agreement; lower regret / higher agreement is better. + """ + cfg=cfg or HardDynamicsConfig(); rng=np.random.default_rng(seed); model.eval(); regrets=[]; agrees=[]; chosen_d=[]; oracle_d=[] + candidates=_candidate_actions(horizon); scale=np.sqrt(2.)*cfg.image_size + corners=np.asarray([[6.,6.],[cfg.image_size-7.,6.],[6.,cfg.image_size-7.],[cfg.image_size-7.,cfg.image_size-7.]]) + for _ in range(n): + init,tok,state=_initial_episode(rng,cfg); target=corners[int(rng.integers(0,4))] + image=torch.tensor(init[None,None],device=device); tokens=torch.tensor(tok[None],device=device); z0=model.encode(image,tokens) + pred_d=[]; true_d=[] + x,y,vx,vy,shape,radius=state + for acts in candidates: + z=z0.clone() + for a in acts: z=model.transition(z,torch.tensor([a],device=device)) + p=_target_com(model.image_decoder(z))[0].cpu().numpy(); pred_d.append(float(np.linalg.norm(p-target))) + xx,yy,_,_=_simulate(x,y,vx,vy,shape,radius,acts,cfg); true_d.append(float(np.linalg.norm(np.asarray([xx,yy])-target))) + ci=int(np.argmin(pred_d)); oi=int(np.argmin(true_d)); cd=true_d[ci]; od=true_d[oi] + regrets.append((cd-od)/scale); agrees.append(float(ci==oi)); chosen_d.append(cd/scale); oracle_d.append(od/scale) + return {'planning_regret':float(np.mean(regrets)),'planning_oracle_agreement':float(np.mean(agrees)), + 'planning_chosen_distance':float(np.mean(chosen_d)),'planning_oracle_distance':float(np.mean(oracle_d))} + +def generate_cf_pairs_cfg(n,horizon,seed,cfg): + rng=np.random.default_rng(seed); out=[] + for _ in range(n): + init,tok,state=_initial_episode(rng,cfg); x,y,vx,vy,shape,radius=state; branches=[] + for act in (3,4): + xx,yy,vxx,vyy=x,y,vx,vy; frames=[] + for _t in range(horizon): + xx,yy,vxx,vyy=_simulate(xx,yy,vxx,vyy,shape,radius,[act],cfg) + c=np.zeros_like(init); _render(c,xx,yy,shape,radius,1.0); _render(c,6.,6.,'circle' if shape=='square' else 'square',3,.45); c[:,cfg.image_size//2-2:cfg.image_size//2+2]*=.15; frames.append(c) + branches.append(np.stack(frames)) + out.append((init,tok,branches[0],branches[1])) + return out + +@torch.no_grad() +def evaluate_counterfactual_cfg(model,device,horizon=30,n=96,seed=809,cfg=None): + cfg=cfg or OOD_CONFIG; pairs=generate_cf_pairs_cfg(n,horizon,seed,cfg); model.eval(); mses=[]; cos=[]; ratios=[] + for init,tok,left,right in pairs: + image=torch.tensor(init[None,None],device=device); tokens=torch.tensor(tok[None],device=device); z0=model.encode(image,tokens); preds=[] + for act in (3,4): + z=z0.clone(); a=torch.tensor([act],device=device) + for _ in range(horizon): z=model.transition(z,a) + preds.append(model.image_decoder(z)) + gtL=torch.tensor(left[-1][None,None],device=device); gtR=torch.tensor(right[-1][None,None],device=device) + mses.extend([torch.mean((preds[0]-gtL)**2).item(),torch.mean((preds[1]-gtR)**2).item()]) + pe=_target_com(preds[1])-_target_com(preds[0]); ge=_target_com(gtR)-_target_com(gtL); pn=pe.norm(dim=-1); gn=ge.norm(dim=-1) + cos.append(float((pe*ge).sum(-1)/(pn*gn).clamp_min(1e-6))); ratios.append(float(pn/gn.clamp_min(1e-6))) + return {'ood_cf_final_image_mse':float(np.mean(mses)),'ood_cf_effect_vector_cosine':float(np.mean(cos)),'ood_cf_effect_magnitude_ratio':float(np.mean(ratios))} diff --git a/spectral_world_models/v9_causal_generalization.py b/spectral_world_models/v9_causal_generalization.py new file mode 100644 index 0000000000000000000000000000000000000000..08e4cd160bdb8f91944ffb66c403beb514ad1bab --- /dev/null +++ b/spectral_world_models/v9_causal_generalization.py @@ -0,0 +1,49 @@ +"""V9 causal-generalization utilities: environment families and trajectory-level interventions.""" +from __future__ import annotations +from dataclasses import replace +import numpy as np, torch +from .hard_dynamics import HardDynamicsConfig, generate_hard_sequences +from .v8_generalization import _initial_episode, _simulate, _render, _target_com + +TRAIN_FAMILY = [ + HardDynamicsConfig(acceleration=.55,max_speed=2.6,occlusion_prob=.08,damping=.99,restitution=.95), + HardDynamicsConfig(acceleration=.70,max_speed=3.0,occlusion_prob=.12,damping=1.00,restitution=1.00), + HardDynamicsConfig(acceleration=.85,max_speed=3.4,occlusion_prob=.18,damping=.97,restitution=.90), +] +INTERP = HardDynamicsConfig(acceleration=.75,max_speed=3.2,occlusion_prob=.15,damping=.985,restitution=.93,seq_len=101,test_sequences=96,seed=9101) +EXTRAP = HardDynamicsConfig(acceleration=1.20,max_speed=4.3,occlusion_prob=.30,damping=.94,restitution=.82,seq_len=101,test_sequences=96,seed=9102) +STRUCTURAL = HardDynamicsConfig(acceleration=.70,max_speed=3.0,occlusion_prob=.12,damping=.88,restitution=.62,seq_len=101,test_sequences=96,seed=9103) + +def generate_family_npz(path, seq_len=16, per_env_train=192, val=96, test=96, seed=9001): + """Mix multiple physics regimes in training while keeping validation/test deterministic.""" + rng=np.random.default_rng(seed); chunks=[] + for i,c in enumerate(TRAIN_FAMILY): + cc=replace(c,seq_len=seq_len,train_sequences=per_env_train,val_sequences=1,test_sequences=1,seed=seed+i) + chunks.append(generate_hard_sequences(per_env_train,cc,rng)) + tr={k:np.concatenate([x[k] for x in chunks],axis=0) for k in ('images','tokens','actions','rewards')} + base=replace(TRAIN_FAMILY[1],seq_len=seq_len); va=generate_hard_sequences(val,base,rng); te=generate_hard_sequences(test,base,rng) + np.savez_compressed(path,train_images=tr['images'],train_tokens=tr['tokens'],train_actions=tr['actions'],train_rewards=tr['rewards'],val_images=va['images'],val_tokens=va['tokens'],val_actions=va['actions'],val_rewards=va['rewards'],test_images=te['images'],test_tokens=te['tokens'],test_actions=te['actions'],test_rewards=te['rewards']) + +def make_trajectory_cf_batch(n=48,horizon=10,seed=9191,device='cpu'): + rng=np.random.default_rng(seed); inits=[]; toks=[]; left=[]; right=[] + for i in range(n): + cfg=TRAIN_FAMILY[i%len(TRAIN_FAMILY)]; init,tok,state=_initial_episode(rng,cfg); x,y,vx,vy,shape,radius=state; branches=[] + for act in (3,4): + xx,yy,vxx,vyy=x,y,vx,vy; frames=[] + for _ in range(horizon): + xx,yy,vxx,vyy=_simulate(xx,yy,vxx,vyy,shape,radius,[act],cfg) + c=np.zeros_like(init); _render(c,xx,yy,shape,radius,1.0); frames.append(c) + branches.append(np.stack(frames)) + inits.append(init); toks.append(tok); left.append(branches[0]); right.append(branches[1]) + return {'image':torch.tensor(np.stack(inits)[:,None],dtype=torch.float32,device=device),'tokens':torch.tensor(np.stack(toks),dtype=torch.long,device=device),'left':torch.tensor(np.stack(left)[:,:,None],dtype=torch.float32,device=device),'right':torch.tensor(np.stack(right)[:,:,None],dtype=torch.float32,device=device),'horizon':horizon} + +def trajectory_cf_loss(model,batch,lambda_dir=.02,lambda_mag=.005,lambda_branch=.08,discount=.97): + z0=model.encode(batch['image'],batch['tokens']); B=z0.shape[0]; zs=[z0,z0]; total=0.; dsum=msum=bsum=0.; wsum=0. + for t in range(batch['horizon']): + preds=[] + for j,act in enumerate((3,4)): + a=torch.full((B,),act,device=z0.device,dtype=torch.long); zs[j]=model.transition(zs[j],a); preds.append(model.image_decoder(zs[j])) + gtL=batch['left'][:,t]; gtR=batch['right'][:,t]; pe=_target_com(preds[1])-_target_com(preds[0]); ge=_target_com(gtR)-_target_com(gtL) + pn=pe.norm(dim=-1).clamp_min(1e-5); gn=ge.norm(dim=-1).clamp_min(1e-5); dl=(1-(pe*ge).sum(-1)/(pn*gn)).mean(); ml=torch.abs(torch.log(pn/gn)).clamp(max=5).mean(); bl=.5*((preds[0]-gtL).square().mean()+(preds[1]-gtR).square().mean()); w=discount**t + total=total+w*(lambda_dir*dl+lambda_mag*ml+lambda_branch*bl); dsum+=w*dl; msum+=w*ml; bsum+=w*bl; wsum+=w + return total/wsum,{'traj_cf_dir_loss':float((dsum/wsum).detach()),'traj_cf_mag_loss':float((msum/wsum).detach()),'traj_cf_branch_mse':float((bsum/wsum).detach())} diff --git a/tests/test_smoke.py b/tests/test_smoke.py new file mode 100644 index 0000000000000000000000000000000000000000..fdbccae82d62bcc195acc4b55f8979c596e0fb2b --- /dev/null +++ b/tests/test_smoke.py @@ -0,0 +1,16 @@ +from pathlib import Path +import torch +from spectral_world_models.data import DatasetConfig, generate_benchmark_npz, TransitionDataset +from spectral_world_models.models import build_model + + +def test_swm_forward(tmp_path: Path): + data = tmp_path / "tiny.npz" + generate_benchmark_npz(data, DatasetConfig(train_sequences=2, val_sequences=1, test_sequences=1, seq_len=4)) + ds = TransitionDataset(data, "train") + batch = ds[0] + model = build_model("swm") + out = model(batch["image"].unsqueeze(0), batch["tokens"].unsqueeze(0), batch["action"].unsqueeze(0)) + assert out["image_pred"].shape == (1, 1, 32, 32) + assert out["text_logits"].shape[1] == 5 + assert torch.isfinite(out["image_pred"]).all() diff --git a/tests/test_stability_constraint.py b/tests/test_stability_constraint.py new file mode 100644 index 0000000000000000000000000000000000000000..59156e193bc71211b8f916c7f044959cd760be11 --- /dev/null +++ b/tests/test_stability_constraint.py @@ -0,0 +1,17 @@ +import torch +from spectral_world_models.models import build_model + +def test_spectral_norm_constraint_is_enforced(): + m=build_model('swm_spectral_norm') + tr=m.transition + ids=torch.arange(tr.action_dim) + A,_=tr.constrained_matrix(ids) + sigma=torch.linalg.matrix_norm(A,ord=2,dim=(-2,-1)) + assert float(sigma.max()) <= tr.rho + 1e-5 + +def test_spectral_norm_variant_forward_shape(): + m=build_model('swm_spectral_norm') + image=torch.rand(2,1,32,32); tokens=torch.ones(2,5,dtype=torch.long); action=torch.tensor([0,1]) + out=m(image,tokens,action) + assert out['image_pred'].shape == image.shape + assert out['z_pred'].shape == (2,96) diff --git a/tests/test_v10_cf_audit.py b/tests/test_v10_cf_audit.py new file mode 100644 index 0000000000000000000000000000000000000000..0d51a3d3d7f9fdc88bbc4e6335b3b968cbea35ad --- /dev/null +++ b/tests/test_v10_cf_audit.py @@ -0,0 +1,13 @@ +from spectral_world_models.v10_cf_audit import REGIMES,physics_fingerprint,action_programs,build_oracle_audit,assert_audit_integrity + +def test_physics_fingerprints_are_distinct(): + assert len({physics_fingerprint(c) for c in REGIMES.values()})==3 + +def test_action_programs_include_non_collinear_interventions(): + ps=action_programs(12); assert len(ps)>=4; assert any(name=='up_down' for name,_,_ in ps); assert any(name=='turn_cw' for name,_,_ in ps) + +def test_oracle_trajectories_change_with_physics(): + rows=build_oracle_audit(n=4,horizon=12,seed=4); assert assert_audit_integrity(rows) + for p in {r['program'] for r in rows}: + rr=[r for r in rows if r['program']==p] + assert len({r['trajectory_set_hash'] for r in rr})==3 diff --git a/tests/test_v11_calibration.py b/tests/test_v11_calibration.py new file mode 100644 index 0000000000000000000000000000000000000000..e522fe23bc4785171bd343c1dc1438969bcffb2f --- /dev/null +++ b/tests/test_v11_calibration.py @@ -0,0 +1,12 @@ +import numpy as np, torch +from spectral_world_models.models import build_model +from spectral_world_models.v11_causal_calibration import make_calibration_batch,calibrated_trajectory_cf_loss,_lin_calibration,evaluate_causal_calibration,INTERP + +def test_v11_loss_backward(): + m=build_model('swm_structured_cf_v11'); b=make_calibration_batch(n=2,horizon=2,seed=1); loss,parts=calibrated_trajectory_cf_loss(m,b); assert torch.isfinite(loss); loss.backward(); assert 'cf_cal_gain_loss' in parts + +def test_calibration_fit_identity(): + s,i,c=_lin_calibration([1,2,3],[1,2,3]); assert abs(s-1)<1e-6 and abs(i)<1e-6 and c>.999 + +def test_v11_eval_smoke(): + m=build_model('swm_structured_cf_v11'); r=evaluate_causal_calibration(m,torch.device('cpu'),INTERP,n=1,horizon=2,seed=2); assert all(np.isfinite(v) for v in r.values()) diff --git a/tests/test_v12_decomposition.py b/tests/test_v12_decomposition.py new file mode 100644 index 0000000000000000000000000000000000000000..35f34ce466917388b50a3b3b5bb89dfa82c4ea5a --- /dev/null +++ b/tests/test_v12_decomposition.py @@ -0,0 +1,17 @@ +import numpy as np +from spectral_world_models.v12_causal_decomposition import summarize_strata,paired_bootstrap_delta,_episode_tags +from spectral_world_models.v9_causal_generalization import INTERP + +def test_tags_detect_boundary_and_bins_inputs(): + st=(10.,10.,1.,2.,'circle',2.) + a=np.array([[2.1,10.],[3.,10.]],np.float32); b=np.array([[20.,10.],[21.,10.]],np.float32) + t=_episode_tags(st,a,b,INTERP,b-a) + assert t['boundary_contact'] and t['initial_speed']>2 + +def test_summary_and_paired_bootstrap(): + rows=[] + for e in range(8): + rows.append(dict(regime='interpolation',episode=e,program='left_right',horizon=30,trajectory_cosine=.6,magnitude_ratio=.8,magnitude_ratio_mae=.2,log_gain_error=.25,initial_speed=1.,oracle_effect_path_mag=2.,boundary_contact=False,occlusion_zone_exposure=False,speed_bin='mid',effect_bin='mid')) + s=summarize_strata(rows); assert any(x['stratum_type']=='program' for x in s) + ctl=[dict(x,trajectory_cosine=.2) for x in rows] + z=paired_bootstrap_delta(rows,ctl,n_boot=100,seed=1); assert abs(z['delta_v9_minus_control']-.4)<1e-6 and z['ci95_low']>0 diff --git a/tests/test_v13_axis_equivariance.py b/tests/test_v13_axis_equivariance.py new file mode 100644 index 0000000000000000000000000000000000000000..2f86ec1052fb537ab06d2fcf96d7935480eecdff --- /dev/null +++ b/tests/test_v13_axis_equivariance.py @@ -0,0 +1,17 @@ +import numpy as np +from spectral_world_models.v13_axis_equivariance import transform_xy,transform_vec,inverse_vec,transform_action,transform_image,make_collision_episodes +from spectral_world_models.v9_causal_generalization import INTERP + +def test_rot90_coordinate_vector_action_consistency(): + n=32; p=np.array([7.,11.]); q=transform_xy(p,n,'rot90'); assert np.allclose(q,[11.,24.]) + v=np.array([2.,-3.]); assert np.allclose(inverse_vec(transform_vec(v,'rot90'),'rot90'),v) + assert transform_action(4,'rot90')==1 and transform_action(1,'rot90')==3 + +def test_image_transforms_are_exact(): + x=np.arange(16).reshape(4,4); assert np.array_equal(transform_image(x,'swap_xy'),x.T); assert np.array_equal(transform_image(x,'flip_x'),np.fliplr(x)) + +def test_collision_generator_forces_near_wall_states(): + eps=make_collision_episodes(INTERP,n=8,seed=1); assert len(eps)==8 + for _,_,st,wall in eps: + x,y,vx,vy,shape,r=st + assert min(x-(r+1), (INTERP.image_size-r-2)-x, y-(r+1), (INTERP.image_size-r-2)-y) <= .26 diff --git a/tests/test_v14_external_control.py b/tests/test_v14_external_control.py new file mode 100644 index 0000000000000000000000000000000000000000..7a642166ee043f885477bdcdf6b6ff3c0f403c70 --- /dev/null +++ b/tests/test_v14_external_control.py @@ -0,0 +1,13 @@ +import numpy as np +from spectral_world_models.v14_external_control import rgb_to_gray32, build_vocab, obs_to_tokens, ExternalConfig, SUPPORTED_ENVS + +def test_external_suite_declared(): + assert set(SUPPORTED_ENVS)=={"CartPole-v1","MountainCar-v0","Acrobot-v1"} + +def test_render_preprocess_shape_range(): + rgb=np.zeros((40,60,3),dtype=np.uint8); rgb[10:20,20:30]=255 + x=rgb_to_gray32(rgb,32); assert x.shape==(1,32,32); assert 0<=x.min()<=x.max()<=1 + +def test_external_tokens_bounded(): + obs=np.array([0.,1.,-2.],dtype=np.float32); low=np.array([-1.,-np.inf,-3.]); high=np.array([1.,np.inf,3.]) + v=build_vocab('X',3); t=obs_to_tokens(obs,low,high,'X',8); assert t.shape==(8,); assert int(t.max())0 diff --git a/tests/test_v6_structured.py b/tests/test_v6_structured.py new file mode 100644 index 0000000000000000000000000000000000000000..17116ca9fae35466e442f4306a80be321e23d72c --- /dev/null +++ b/tests/test_v6_structured.py @@ -0,0 +1,7 @@ +import torch +from spectral_world_models.models import build_model,StructuredSpectralTransportTransition + +def test_structured_model_forward(): + m=build_model('swm_structured'); x=torch.rand(2,1,32,32); tok=torch.ones(2,5,dtype=torch.long); a=torch.tensor([1,4]); o=m(x,tok,a); assert o['image_pred'].shape==(2,1,32,32) +def test_transport_is_orthogonal(): + t=StructuredSpectralTransportTransition(24,5,rank=4); q=t.transport_matrix(); eye=torch.eye(24); assert torch.max(torch.abs(q.T@q-eye)).item()<1e-4 diff --git a/tests/test_v7_counterfactual_training.py b/tests/test_v7_counterfactual_training.py new file mode 100644 index 0000000000000000000000000000000000000000..438dd9eb5605a83aa36e605d0c07777742eda3e2 --- /dev/null +++ b/tests/test_v7_counterfactual_training.py @@ -0,0 +1,13 @@ +import torch +from spectral_world_models.models import build_model +from spectral_world_models.counterfactual import make_cf_training_batch, counterfactual_training_loss + +def test_v7_alias_and_cf_loss_backward(): + m=build_model('swm_structured_cf') + b=make_cf_training_batch(n=2,horizon=2,seed=77,device='cpu') + loss,parts=counterfactual_training_loss(m,b) + assert torch.isfinite(loss) + assert loss.item() >= 0 + loss.backward() + assert any(p.grad is not None for p in m.parameters() if p.requires_grad) + assert set(parts) == {'cf_train_dir_loss','cf_train_mag_loss','cf_train_branch_mse'} diff --git a/tests/test_v8_generalization.py b/tests/test_v8_generalization.py new file mode 100644 index 0000000000000000000000000000000000000000..531cbed9853390dfabb92348235ff603745854cc --- /dev/null +++ b/tests/test_v8_generalization.py @@ -0,0 +1,16 @@ +import torch +from spectral_world_models.models import build_model +from spectral_world_models.v8_generalization import OOD_CONFIG,_candidate_actions,evaluate_planning,evaluate_counterfactual_cfg + +def test_ood_is_shifted_from_training_distribution(): + assert OOD_CONFIG.acceleration > .70 and OOD_CONFIG.max_speed > 3.0 and OOD_CONFIG.occlusion_prob > .12 + +def test_candidate_library_has_interventions(): + c=_candidate_actions(8); assert [3]*8 in c and [4]*8 in c and len(c)>=9 + +def test_v8_metrics_smoke(): + m=build_model('swm_structured_cf') + p=evaluate_planning(m,torch.device('cpu'),n=2,horizon=2,seed=1) + assert 0 <= p['planning_oracle_agreement'] <= 1 and p['planning_regret'] >= -1e-8 + c=evaluate_counterfactual_cfg(m,torch.device('cpu'),horizon=2,n=2,seed=2) + assert set(c)=={'ood_cf_final_image_mse','ood_cf_effect_vector_cosine','ood_cf_effect_magnitude_ratio'} diff --git a/tests/test_v9_causal_generalization.py b/tests/test_v9_causal_generalization.py new file mode 100644 index 0000000000000000000000000000000000000000..7b5979db3f090c9f149cd0a0d2efa283f9f7e773 --- /dev/null +++ b/tests/test_v9_causal_generalization.py @@ -0,0 +1,17 @@ +import numpy as np, torch +from spectral_world_models.v9_causal_generalization import make_trajectory_cf_batch,trajectory_cf_loss,INTERP,EXTRAP,STRUCTURAL +from spectral_world_models.models import build_model + +def test_v9_regimes_are_distinct(): + assert INTERP.acceleration < EXTRAP.acceleration + assert STRUCTURAL.restitution < INTERP.restitution + +def test_trajectory_cf_batch_and_backward(): + b=make_trajectory_cf_batch(n=2,horizon=3,seed=1,device='cpu') + assert b['left'].shape==(2,3,1,32,32) + m=build_model('swm_structured_cf_v9') + loss,parts=trajectory_cf_loss(m,b) + assert torch.isfinite(loss) + loss.backward() + assert any(p.grad is not None for p in m.parameters()) + assert 'traj_cf_dir_loss' in parts