File size: 4,641 Bytes
3ec346a 8bc9f6e 3ec346a 8bc9f6e 106b757 8bc9f6e 8b40df9 8bc9f6e 8b40df9 8bc9f6e 8b40df9 8bc9f6e c1dd0f0 8bc9f6e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 | ---
license: cc-by-nc-4.0
datasets:
- nkp37/OpenVid-1M
tags:
- motion
- planning
---
<p align="center">
<h2 align="center">Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics</h2>
<p align="center">
<b>
Timy Phan<sup>*</sup> · Jannik Wiese<sup>*</sup> · Björn Ommer
</b>
<p align="center">
CompVis Group @ LMU Munich, Munich Center for Machine Learning (MCML)
</p>
<p align="center">
ECCV 2026
</p>
<p align="center">
<sup>*</sup>Equal contribution
</p>
</p>
</p>
<div align="center">
[](https://compvis.github.io/schroedingers_cat/)
[](https://arxiv.org/abs/2607.25984)
[](https://github.com/CompVis/schroedingers_cat)
</div align="center">
Official Code for the paper "Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics" accepted at ECCV 2026.
## 💡 TL;DR
GARFIELD learns a structured latent representation of possible scene kinematics from an image and optional sparse spatio-temporal constraints. The representation supports both joint trajectory sampling and direct point-wise density estimation.
<p align="center">
<img src="docs/teaser.svg" alt="Teaser" width="500px">
</p>
## 📝 Overview
<p align="center">
<img src="docs/method_fig.png" alt="Architecture" width="500px">
</p>
Goal Aware Representations of Future kInEmatic Latent Distributions (GARFIELD) represents future scene kinematics as localized motion distributions. Given an initial image and optional sparse constraints, a joint encoder produces spatio-temporal latents for scene elements over time. These latents encode uncertainty about where each element may move, and separate decoders expose that uncertainty as point samples, coherent trajectory rollouts, or probability heatmaps.
The full pipeline consists of four components:
- **Encoder:** The joint encoder combines image features and sparse kinematic constraints into structured latents, each tied to one scene element and timestep.
- **Pointwise Decoder:** The pointwise decoder trains the latents to represent localized distributions by sampling individual future track positions from each latent component.
- **Full Decoder:** The full decoder samples complete trajectories jointly, preserving dependencies across points, timesteps, and scene elements for coherent motion realizations.
- **Density Decoder:** The density decoder deterministically maps localized latents to probability heatmaps, enabling fast uncertainty inspection without Monte-Carlo sampling.
## 📊 Results
### Motion Planning
<p align="center">
<img src="docs/prediction_min_ade.png" alt="EPE" width="500px">
</p>
GARFIELD infers accurate motion from very sparse information and, with only four conditioning points, outperforms Motion-I2V using sixteen.
### Direct Density Decoding
<p align="center">
<img src="docs/energy_score_vs_latency_new.png" alt="Discrete Energy Scores" width="500px">
</p>
The density decoder achieves superior discrete energy scores while estimating motion densities orders of magnitude faster than Monte-Carlo sampling.
### Entropy-informed Conditioning
<p align="center">
<img src="docs/entropy_vs_epe_cond_new_data.png" alt="Entropy-based cond" width="500px">
</p>
Selecting additional constraints by entropy collapses uncertainty without ground-truth errors and reaches comparable performance with fewer conditioning points.
## 🛠️ Usage
### Weights and Inference Setup
For inference, you can clone the GitHub repository by running:
```bash
git clone https://github.com/CompVis/schroedingers_cat
cd schroedingers_cat
```
Then, download pretrained model weights from 🤗 huggingface running:
```bash
hf download CompVis/schroedingers_cat --include "*.pt" --local-dir ckpts
```
Finally, create an environment to run the repository:
Conda (recommended):
```bash
conda create -n garfield python=3.12 -y
conda activate garfield
python -m pip install --upgrade pip
pip install -r requirements.txt
```
Virtual environment:
```bash
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt
```
## Citation
```bibtex
@inproceedings{phan2026schrodingerscat,
title = {Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics},
author = {Phan, Timy and Wiese, Jannik and Ommer, Björn},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026}
}
```
|