Spaces:
Paused
Paused
| # Generative Architecture Comparison for Counterfactual Scene Generation | |
| Companion to `sensorflow/nextgen/worldmodel.py`. Question: if/when the | |
| deterministic scene transformer is outgrown, what class of generative world | |
| model should back `ExternalWorldModelAdapter`? | |
| The consumer contract is fixed and generator-agnostic: full ground truth out, | |
| determinism per (source, recipe, seed), GENERATED provenance labels, and no | |
| exemption from the validity gate (`nextgen/validity.py`). | |
| ## Options | |
| **Diffusion world model** — denoising-based generation of future scene | |
| states (or sensor frames) conditioned on scene context and edit masks. | |
| **Transformer (autoregressive) world model** — tokenized scene/agent states | |
| rolled out step by step conditioned on history and control/edit tokens | |
| (the family most proprietary driving world models belong to). | |
| **Hybrid** — transformer backbone for dynamics/agent interaction with a | |
| diffusion decoder head for continuous state/appearance refinement. | |
| ## Comparison | |
| | Criterion | Diffusion | Transformer WM | Hybrid | | |
| |---|---|---|---| | |
| | Fidelity (per-frame realism) | Highest; excels at appearance and texture | Good at dynamics, weaker at continuous detail | High on both, at integration cost | | |
| | Controllability (recipe adherence: "this actor brakes at t=1.0s at 7.5 m/s^2") | Weak-to-moderate; edit conditioning is indirect (inpainting/guidance) | Strong; edits are just conditioning tokens; per-actor intervention is natural | Strong (inherits transformer conditioning) | | |
| | Temporal/kinematic consistency | Failure mode: per-frame plausibility with cross-frame drift — exactly what `validity.check_temporal` and `check_identity` exist to catch | Strong; dynamics are the training objective | Strong | | |
| | Latency / cost per scenario | High (many denoising steps × frames) | Moderate (one pass per step; KV-cached rollouts) | Highest | | |
| | Determinism / reproducibility | Achievable (fixed noise, fixed sampler) but fragile across library versions | Straightforward (greedy/temperature-0 decode, seeded sampling) | Fragile at the diffusion head | | |
| | Ground-truth availability | Hard if pixel-space (GT must be re-derived — a labeling problem again) | Natural if state-space (states ARE the GT) | State-space GT from backbone; decoder is presentation-only | | |
| | Validation burden on our gate | Highest: every sample needs full 5-check gating; expect high rejection on temporal checks | Lowest of the learned options; sensor consistency (`check_sensor`) still mandatory | Medium | | |
| | Fit to our transformation catalogue (actor edits, environment swaps, scene composition) | Better for environment/appearance edits (clear→fog) | Better for actor/behavior edits (sudden brake, crossing, emergence) | Covers both | | |
| ## Recommendation (pragmatic) | |
| **State-space transformer world model first; diffusion only as an optional | |
| appearance decoder; i.e. adopt the hybrid shape but build/buy it in that | |
| order.** The reasons are specific to this platform: (1) our counterfactual | |
| recipes are behavior edits far more often than appearance edits — 10 of the | |
| 15 transformations in `counterfactual.TRANSFORMATIONS` are actor/scene | |
| kinematics, which transformer conditioning handles natively; (2) our | |
| evaluation consumers need ground-truth states, not pixels — | |
| `closedloop.py` and `validity.py` operate on `ActorTrack` kinematics, so a | |
| state-space generator plugs into `ExternalWorldModelAdapter` without a new | |
| labeling pipeline; (3) our gate's hardest checks (temporal continuity, | |
| identity stability) are precisely diffusion's characteristic failure modes, | |
| so a diffusion-first choice maximizes validation burden; (4) determinism — | |
| required for lineage-valid launches (`lineage.py`) — is cheap for | |
| temperature-0 transformer decoding and fragile for diffusion samplers. | |
| Until any of that is bought or built, the deterministic transformer in | |
| `counterfactual.DeterministicSceneTransformer` remains the live | |
| implementation: zero fidelity risk, perfect controllability and | |
| reproducibility, bounded expressiveness — the right trade for a platform | |
| whose first duty is evidence integrity. | |