Persistent Generative World Architecture
H3 Ref2VA + Persistent Gaussian World Memory + Agentic Composition
Status: Working design / architecture thesis
Primary goal: Build a persistent generative video world whose visual and spatial state survives across shots, scenes, camera changes, and long time gaps without requiring the video model itself to remember everything.
1. Executive Summary
The core idea is to stop treating long-form video coherence as something a single video model must solve internally.
Instead, we externalize persistence into a world memory layer and let H3 act as the high-capability multimodal generator that reads from that memory, creates new observations of the world, and writes trustworthy changes back into it.
The preferred world representation is a persistent Gaussian-splat map that behaves more like a visual save-game or minimap than a perfect photogrammetric reconstruction.
The map does not need to look cinematic by itself. It only needs to preserve enough information to tell H3:
- where things are,
- what the room roughly looks like,
- what changed,
- what did not change,
- what is currently visible,
- what remains unobserved,
- and how a requested camera should move through the space.
The fundamental persistence rule is:
A new shot only has authority over regions it actually observes with sufficient confidence. Unobserved regions remain untouched.
This allows a room to evolve incrementally.
If a chair moves in one shot, we update that chair.
If a glass disappears, we remove that glass only when the old location was clearly observed.
If the far side of the room was not visible, the old state survives exactly as it was.
The video model therefore does not need infinite memory. It receives retrieval-augmented visual memory from the persistence layer whenever it needs to revisit the world.
2. Why This Architecture
Modern generative video models are becoming powerful enough that many problems previously assumed to require monolithic end-to-end model improvements can instead be solved through:
- agentic composition,
- multimodal retrieval,
- reference selection,
- persistent external state,
- targeted inpainting/editing,
- LoRAs,
- occasional full fine-tunes,
- deterministic 3D tools,
- reconstruction tools,
- and iterative validation.
The architectural thesis is:
We may already have enough raw generative capability. The remaining gap is increasingly orchestration, persistence, control, and error correction.
Rather than waiting for a hypothetical future model to perfectly understand a feature-length world's state, we can build that memory ourselves.
H3 is particularly attractive because its reference-to-video behavior makes the world memory useful even when the memory itself is visually crude.
The persistence layer does not have to generate the final frame.
It has to generate a strong enough reference observation for H3 to understand the intended world.
3. H3 Is the Center, Not Just the Renderer
An earlier version of the architecture placed H3 too far downstream, as though it were primarily a rendering endpoint.
That is not the preferred mental model.
H3 Ref2VA should be treated as the multimodal generative core.
It can combine different reference roles such as:
- character appearance,
- object appearance,
- environment appearance,
- style,
- action,
- pose,
- camera behavior,
- motion,
- temporal structure,
- prior footage,
- continuation context,
- and audio.
This means a shot does not need a single perfect reference clip.
The agent can construct a shot-specific reference package from many different sources.
Example:
IMAGE 1 -> protagonist identity closeup
IMAGE 2 -> protagonist full-body costume reference
IMAGE 3 -> canonical kitchen appearance
IMAGE 4 -> refrigerator detail
IMAGE 5 -> secondary character identity
IMAGE 6 -> lighting/look reference
VIDEO 1 -> tail of previous shot
VIDEO 2 -> crude engine/splat camera playblast
VIDEO 3 -> hand/object interaction reference
AUDIO 1 -> protagonist voice
AUDIO 2 -> secondary character voice
AUDIO 3 -> room ambience
H3 can then synthesize these into a coherent shot.
The important shift is:
The persistence system does not need to contain every piece of information in final-quality form. It only needs to supply the correct constraints and references.
4. The "Minimap" Mental Model
The persistence layer should not be thought of as a fully accurate digital twin.
A better analogy is a generative minimap.
It contains enough persistent information to keep the world coherent.
For a room, that might include:
Kitchen
├── walls
├── windows
├── doors
├── countertop
├── refrigerator
├── table
│ ├── glass_04
│ ├── plate_07
│ └── keys_01
├── chair_01
├── chair_02
├── lighting anchors
└── appearance memory
The Gaussian representation gives this map something a pure symbolic world model does not have:
visual memory attached to spatial memory.
The system remembers not only that a countertop exists, but approximately how that countertop appeared from previously observed views.
This is valuable because H3 can consume rendered observations of that state directly.
5. Persistent Gaussian World State
The preferred persistence behavior is incremental.
Start with a canonical room splat.
Each trustworthy new shot becomes a possible update.
CANONICAL ROOM STATE
Persistent Gaussian Map
|
+------------+------------+
| |
Visible in shot Not visible
| |
v |
New observation |
| |
classify regions |
| |
+------+------+ |
| | |
same changed untouched
| | |
keep replace/update keep
| | |
+------+------+------------------+
|
v
NEW CANONICAL STATE
The key rules are:
Observed and unchanged
- retain the canonical state.
Observed and changed
- update only the changed region.
Observed but uncertain
- retain old state unless confidence is sufficient.
Unobserved
- never overwrite.
This prevents one generated shot from accidentally destroying parts of the world that were not even visible.
6. World Updates Are Transactions
Every accepted shot can be viewed as a transaction against world state.
Suppose the current kitchen contains:
table
glass_04
chair_02
east_wall
west_wall
refrigerator
countertop
A generated shot observes:
table
chair_02
east_wall
glass_04 location
The analysis might produce:
table -> observed / unchanged
chair_02 -> observed / moved
east_wall -> observed / unchanged
glass_04 -> observed / removed
west_wall -> unobserved
refrigerator -> unobserved
countertop -> unobserved
Commit:
table -> keep
chair_02 -> update transform or local geometry
east_wall -> keep
glass_04 -> remove
west_wall -> preserve old state
refrigerator -> preserve old state
countertop -> preserve old state
That is the desired persistence behavior.
7. Preserve History Instead of Destructive Overwrite
The system should ideally maintain state history.
Example:
Shot 410 -> painting hanging on wall
Shot 411 -> painting falls
Shot 430 -> painting is rehung
Instead of discarding the original visual state, the world can maintain deltas:
room_001
├── base_splat
├── delta_0001
├── delta_0002
└── delta_0003
Or object history:
painting_07
├── state_01_hanging
├── state_02_floor
└── state_03_rehung
Benefits:
- rollback,
- alternate timelines,
- flashbacks,
- continuity debugging,
- reconstruction of earlier shots,
- branchable world state,
- cheaper storage than full checkpoints,
- and reusing prior object states instead of relearning them.
This also makes the architecture useful beyond linear filmmaking.
It can support persistent games, simulations, virtual worlds, and interactive narrative systems.
8. Local Coordinate Systems
Objects and regions should avoid permanently baking all Gaussian primitives into a single global coordinate frame.
Prefer a hierarchy:
world transform
*
region transform
*
object transform
*
Gaussian local position
Example:
WORLD
├── room_kitchen
│ ├── static_shell
│ ├── table_01
│ ├── chair_01
│ ├── chair_02
│ └── glass_04
└── hallway
If chair_02 moves, the system may only need to update:
chair_02.transform
instead of modifying thousands of Gaussian primitives.
Only when an object's geometry actually changes should the system perform local Gaussian reconstruction.
This gives a useful hierarchy of corrections:
KEEP
RIGID_REALIGN
SIM3_REALIGN
RECOLOR
LOCAL_OPTIMIZE
PRUNE_AND_RESPLAT
FULL_REGION_RECONSTRUCT
An agent can decide which operation is appropriate.
9. QuerySplat's Role
QuerySplat is potentially very useful, but it should not itself be the persistent memory.
Its best role is:
Lift a newly generated H3 observation back into candidate 3D.
Conceptually:
Persistent World Splat
|
v
Render requested camera/reference
|
v
H3 Ref2VA
|
v
Beautiful new shot
|
v
Select useful frames
|
v
QuerySplat
|
v
Candidate 3D observation
|
v
Register against persistent world
|
v
Detect trustworthy changes
|
v
Local commit
QuerySplat turns generated frames into something that can be compared spatially against the current world state.
Useful outputs may include:
- camera estimates,
- depth,
- depth confidence,
- point clouds,
- Gaussian primitives,
- and a coherent local coordinate interpretation.
Those are all useful for deciding whether a region is allowed to modify canonical memory.
10. QuerySplat Must Not Directly Replace the World
A newly reconstructed splat should be treated as an observation, not as truth.
Do not do:
persistent_world = QuerySplat(new_frames)
Instead:
candidate = QuerySplat(new_frames)
candidate
|
v
register to canonical map
|
v
visibility analysis
|
v
confidence analysis
|
v
semantic correspondence
|
v
geometric difference analysis
|
v
local change mask
|
v
commit approved regions only
Two independently generated splats may represent the same physical surface using different Gaussian primitives.
Therefore the merge logic must operate on spatial/semantic evidence rather than assuming Gaussian-to-Gaussian identity.
11. Related Tool Roles
The emerging stack can be separated by responsibility.
H3 Ref2VA
Role: generative world observation and cinematic synthesis.
Responsibilities:
- combine visual references,
- preserve identities,
- follow motion references,
- synthesize camera behavior,
- create final-quality shots,
- edit bad regions,
- perform continuation,
- and create plausible missing observations.
QuerySplat
Role: observation lifting.
Responsibilities:
- take H3-generated frames,
- infer coherent local 3D,
- estimate cameras/depth,
- create a candidate Gaussian representation,
- and provide data for world-state comparison.
Persistent Gaussian Map
Role: long-term visual/spatial memory.
Responsibilities:
- retain rooms and objects,
- preserve unseen geometry,
- track visual state,
- store confidence,
- store current and historical state,
- generate rough views from arbitrary cameras.
Local Update / Continual-GS Techniques
Concepts from systems such as CL-Splats and related continual reconstruction work are relevant.
Role: safe mutation.
Responsibilities:
- detect changed spatial regions,
- freeze untouched regions,
- update only affected Gaussians,
- maintain compact deltas,
- and avoid catastrophic map replacement.
Change Detection
Gaussian-space or hybrid 2D/3D change detection can act as the gatekeeper.
Role: determine what actually changed.
Possible classifications:
UNCHANGED
RIGID_MOVEMENT
GEOMETRY_CHANGE
APPEARANCE_CHANGE
REMOVED
NEW_OBJECT
UNCERTAIN
UNOBSERVED
Game Engine
Role: optional deterministic topology/physics oracle.
A game engine remains useful, but it does not need to render the final movie.
It can provide:
- collision,
- object transforms,
- door state,
- skeletal pose,
- camera trajectories,
- rough lighting direction,
- room topology,
- physics,
- object identity,
- and deterministic blocking.
The guiding principle is:
Use the game engine to tell H3 what happened, not necessarily what the movie should look like.
An ugly playblast can still be an excellent motion/spatial reference.
12. 3D -> H3 -> 3D Loop
One of the most important architectural loops is:
Persistent 3D World
|
v
Render reference
|
v
H3
repair / extend /
animate / render
|
v
Generated video
|
v
QuerySplat
|
v
Candidate new 3D
|
v
Validate/merge
|
v
Persistent 3D World
Or more simply:
3D -> H3 -> 3D -> H3 -> 3D
This lets the world evolve over time.
H3 can imagine a new observation.
QuerySplat or similar tools can lift that observation back into spatial memory.
The persistence layer then decides whether the new information is trustworthy enough to become canonical.
13. Expanding Into Previously Unseen Space
This architecture becomes especially interesting when the camera enters a region that has never been modeled.
Suppose the world contains:
kitchen
hallway
living room
unknown room
The system can:
- render the known hallway and doorway,
- request a camera trajectory into the unknown area,
- give H3 canonical style/environment references,
- ask H3 to generate the continuation,
- reconstruct useful frames into a candidate 3D region,
- align that region to the known doorway/hallway,
- validate it,
- commit it as a new part of the world.
Conceptually:
KNOWN WORLD
|
v
reference camera trajectory
|
v
H3 generates plausible unseen space
|
v
QuerySplat
|
v
new candidate geometry
|
v
align to known anchors
|
v
commit
|
v
WORLD EXPANDS
Once committed, that room is no longer pure hallucination.
It has become persistent world memory.
Future shots can retrieve and render it again.
14. Sparse Skeleton, Not Perfect Reconstruction
The Gaussian world does not need to be a perfect photorealistic scene.
This is an important design choice.
The goal is not necessarily:
render the persistent splat directly as the final image.
The goal is:
render enough spatial truth for H3 to reconstruct the intended world.
The persistent world mainly needs:
- surfaces,
- relative layout,
- rough material/color cues,
- object placement,
- depth,
- occlusion,
- camera parallax,
- persistent clutter,
- visual anchors.
If the far side of a couch contains ugly sparse Gaussians, that may be acceptable.
H3 can clean up the appearance.
This lowers the reconstruction burden dramatically.
15. Confidence Is a First-Class Property
Every persistent region should carry confidence.
Possible fields:
position
rotation
scale
appearance
semantic_id
object_id
last_observed_shot
observation_count
geometry_confidence
appearance_confidence
identity_confidence
registration_confidence
state:
canonical
changed
uncertain
historical
A low-confidence region should not be allowed to overwrite a high-confidence canonical region.
This is especially important with generated video because visual plausibility does not imply spatial truth.
The commit system should distinguish:
"H3 generated something convincing"
from:
"we have enough evidence that the canonical world really changed."
16. Persistent State vs References vs LoRAs
Different forms of memory should have different jobs.
World Map = Episodic / Mutable Facts
Examples:
- the glass currently sits on the table,
- the chair is rotated 20 degrees,
- the door is open,
- the character left the room,
- the painting fell,
- the refrigerator contains a cake.
These facts should not require fine-tuning.
Reference Library = Canonical Visual Memory
Examples:
- exact face reference,
- exact costume,
- canonical room look,
- prop details,
- lighting style,
- known camera treatment,
- prior shots from useful angles.
The agent retrieves these when preparing a shot.
LoRAs = Persistent Learned Priors
LoRAs are useful when the base model repeatedly fails to honor references reliably.
Examples:
- character identity,
- recurring creature design,
- studio aesthetic,
- specific cinematography grammar,
- hard-to-preserve wardrobe,
- unusual recurring objects,
- production-specific motion language.
The principle is:
Do not encode transient world facts into a LoRA.
A moved chair belongs in the map.
A stubborn character identity may belong in a LoRA.
Full Fine-Tunes = Behavioral / Domain Priors
Full fine-tuning may eventually be appropriate for:
- production-specific reference following,
- particular cinematographic behavior,
- better persistence responses,
- improved handling of our rendered conditioning style,
- better interaction with synthetic engine/splat references,
- or specific animation domains.
But full fine-tuning is not the first requirement.
The external architecture should work independently.
17. Agentic Reference Composition
The world may eventually contain tens of thousands of possible reference assets.
H3 should not receive all of them.
An agent should build a shot-specific multimodal context pack.
Inputs to the agent:
script state
scene state
camera request
characters present
objects visible
current world map
previous shot
continuity requirements
style requirements
motion requirements
audio requirements
Output:
selected image references
selected video references
selected audio references
persistent-world render
engine playblast if useful
previous-shot tail
LoRA set
prompt / structured instructions
The agent is effectively performing:
multimodal RAG for video generation.
Instead of retrieving text passages, it retrieves:
- images,
- videos,
- audio,
- geometry,
- world state,
- object state,
- previous shots,
- motion clips,
- and learned adapters.
18. Generation and Validation Loop
The production loop can be:
STORY / TASK
|
v
Director Agent
|
v
Shot Requirements
|
v
Retrieve Persistent State
|
v
Build Reference Package
|
v
H3
|
v
Generated Candidate
|
+-----------+-----------+
| |
v v
Visual Evaluator Spatial Evaluator
| |
+-----------+-----------+
|
v
Trustworthy result?
/ \
yes no
| |
v v
Commit observations edit/inpaint/
to world state regenerate
|
v
Updated world memory
The world map therefore becomes both:
- an input to generation,
- and a recipient of validated output.
19. The Commit Algorithm
The commit algorithm may be the most important technical component in the whole architecture.
Its responsibility:
Given the previous canonical world and a newly generated H3 clip, determine exactly which spatial regions are allowed to modify persistent state.
A possible pipeline:
H3 clip
|
v
frame selection
|
v
QuerySplat / depth / camera inference
|
v
candidate local 3D
|
v
register candidate to canonical world
|
v
visibility determination
|
v
semantic association
|
v
geometry + appearance comparison
|
v
confidence calculation
|
v
change classification
|
v
approved mutation mask
|
v
transactional world update
Important rules:
Rule 1: Unobserved Means Immutable
If the new shot does not see a region, the region cannot be modified.
Rule 2: Low Confidence Cannot Destroy High Confidence
A weak reconstruction cannot overwrite a stable canonical surface.
Rule 3: Rigid Movement Should Prefer Transform Updates
Do not reconstruct an object if its identity and geometry are still valid.
Rule 4: Appearance and Geometry Are Different
A lighting/color shift should not automatically be interpreted as geometry change.
Rule 5: Deletion Requires Evidence
An object disappears from the world only when its former location is sufficiently observed and proven empty.
Rule 6: New Geometry Begins as Provisional
Newly hallucinated regions may require repeated confirmation before becoming high-confidence canonical state.
Rule 7: Keep History
Accepted updates should preferably be stored as deltas or versioned states.
20. Example: Character Moves a Chair and Takes a Glass
Initial state:
Kitchen:
chair_02 = at table, rotation 0 degrees
glass_04 = on table, 35% full
Shot action:
Sarah enters.
Sarah rotates chair_02.
Sarah picks up glass_04.
Camera follows Sarah out.
After generation:
QuerySplat / reconstruction
|
v
compare to canonical kitchen
Detected:
chair_02 -> same object, new rigid transform
glass_04 -> removed from table
table -> unchanged
east wall -> unchanged
refrigerator -> unobserved
countertop -> unobserved
Commit:
chair_02.transform = new transform
glass_04.location = Sarah/right_hand
or later:
glass_04.location = unknown/carried
table = unchanged
east wall = unchanged
refrigerator = untouched
countertop = untouched
No full room reconstruction occurs.
21. Example: Geometry Actually Changes
Suppose a table is broken.
This is not a rigid transform.
The agent may choose:
LOCAL_OPTIMIZE
or:
PRUNE_AND_RESPLAT
Pipeline:
detect changed table region
|
v
freeze rest of room
|
v
remove invalid table Gaussians
|
v
reconstruct broken geometry
|
v
blend boundary
|
v
commit table delta
Everything else remains stable.
22. Example: H3 Drifts on Character Identity
Suppose H3 repeatedly changes a character's face at certain angles even with good reference retrieval.
Do not change the world map.
Instead:
collect successful identity observations
|
v
train/refine character identity LoRA
|
v
evaluate against reference-only baseline
|
v
use LoRA when confidence improvement is real
This separates:
WORLD FACT
"Sarah is standing by the refrigerator."
from:
MODEL PRIOR
"This is what Sarah looks like from every angle."
23. Why Gaussian Splats Instead of Only a Game Engine
A game engine is extremely useful for exact state.
But a pure engine representation has a weakness:
it stores geometry/state well, but it does not naturally retain the generative model's actual visual memory of the environment.
Gaussian splats give us:
- view-dependent appearance,
- captured texture,
- clutter,
- lighting cues,
- imperfect but useful scene appearance,
- and a renderable observation from arbitrary cameras.
The ideal system may therefore combine them:
SYMBOLIC / ENGINE STATE
exact facts
|
v
object transforms
topology / physics
|
+
|
v
GAUSSIAN WORLD MEMORY
visual/spatial state
|
v
rough rendered reference
|
v
H3
The engine is deterministic truth.
The splat is visual memory.
H3 is the cinematic imagination layer.
24. Why Not Just Use Previous Video Frames?
Previous frames are useful but insufficient.
They fail when:
- the camera visits a new viewpoint,
- a long time has passed,
- an object was last visible from the wrong side,
- the new shot needs geometry hidden in prior footage,
- the room has undergone multiple changes,
- or the previous clip does not expose the required spatial relationships.
The persistent splat can render a new approximate reference from a camera angle that was never previously filmed.
That is a major advantage.
25. Why Not Require the Video Model to Remember Everything?
Because long-term persistence is fundamentally different from generation quality.
A model can be excellent at producing a coherent 10-second clip and still fail to remember:
- which shelf an object was on 100 shots ago,
- whether a window was open,
- how far a chair had moved,
- what was behind the camera,
- or which version of a room is currently canonical.
Externalizing that memory gives us:
- inspectability,
- editability,
- determinism,
- branching,
- rollback,
- explicit state,
- and model independence.
A future H3 successor can replace H3 without requiring the entire persistence architecture to be rebuilt.
26. Model Upgrades Become Replaceable Components
One of the strongest reasons to pursue this architecture is longevity.
The stack can remain:
persistent world
reference library
agentic retrieval
commit logic
history
validation
LoRAs
engine integration
while the generative model changes:
H3
-> H4
-> future Seedance-class model
-> something better
The orchestration and world memory remain valuable.
This is one reason the system can potentially achieve capability beyond the apparent limits of any single frozen model.
27. Prototype Architecture
A practical first prototype does not need the full system.
Phase 1: Closed Room Loop
Goal:
Demonstrate that an H3-generated room can be reconstructed, re-rendered from a new angle, fed back to H3, and remain recognizably persistent.
Pipeline:
1. Generate 5-10 second H3 room shot
2. Sample useful frames
3. Run QuerySplat
4. Render novel camera angles
5. Feed rendered views back to H3 Ref2VA
6. Generate another shot
7. Compare object/layout persistence
Success criterion:
- the next H3 shot preserves major room layout and object identity better with the splat-derived reference than without it.
Phase 2: Controlled Object Movement
Goal:
Update only one changed object.
Example:
chair moves
Pipeline:
canonical room splat
|
H3 shot moves chair
|
QuerySplat candidate
|
register
|
detect chair movement
|
preserve room
|
update chair only
Success criterion:
- unchanged room regions survive multiple update cycles.
Phase 3: Object Removal and Addition
Test:
glass on table
-> character removes glass
-> later returns glass
Validate:
- safe deletion,
- preserved table geometry,
- correct historical state,
- restoration from prior object memory.
Phase 4: New Space Expansion
Test:
known hallway
-> H3 generates unseen bedroom
-> QuerySplat reconstructs bedroom
-> register bedroom to hallway
-> commit
-> revisit bedroom later
Success criterion:
- newly imagined space becomes persistent.
Phase 5: Hybrid Engine + Splat Conditioning
Use an engine to generate:
- deterministic actor blocking,
- exact camera path,
- object transforms,
- rough room geometry.
Then combine:
engine playblast
+
persistent splat render
+
identity refs
+
style refs
+
previous-shot tail
+
H3 Ref2VA
Measure whether this improves:
- camera control,
- object permanence,
- hand-object interactions,
- action continuity.
Phase 6: Agentic Commit Decisions
Replace hand-written update selection with an agent plus deterministic tools.
Agent outputs:
KEEP
RIGID_REALIGN
SIM3_REALIGN
RECOLOR
LOCAL_OPTIMIZE
PRUNE_AND_RESPLAT
FULL_REGION_RECONSTRUCT
The agent does not perform numerical registration itself.
It selects and orchestrates the appropriate specialized tools.
28. Evaluation Metrics
We should evaluate more than image beauty.
Persistence
- object position drift,
- orientation drift,
- object count consistency,
- room topology consistency,
- layout stability,
- identity stability.
Generative Quality
- visual quality,
- motion quality,
- reference fidelity,
- cinematography,
- temporal coherence.
World Update Safety
- false deletions,
- false additions,
- incorrect geometry overwrite,
- corruption of unobserved regions,
- confidence calibration.
Efficiency
- reconstruction latency,
- H3 generation cost,
- map update cost,
- storage growth,
- amount of scene needing reoptimization,
- reference retrieval overhead.
Long-Horizon Robustness
Run the world through dozens or hundreds of updates and measure whether static regions drift over time.
This may be one of the most important tests.
29. Potential Data Model
A region/object representation might contain:
region_id:
parent_region:
object_ids: []
transform:
translation:
rotation:
scale:
geometry:
gaussian_chunk:
collision_proxy:
bounds:
appearance:
canonical_refs: []
embeddings: []
material_hints:
confidence:
geometry:
appearance:
registration:
semantic_identity:
history:
created_at_shot:
last_observed_shot:
observation_count:
versions: []
state:
canonical:
provisional:
changed:
uncertain:
Objects may additionally contain:
object_id:
semantic_class:
identity_ref:
lora:
current_region:
current_transform:
state_variables:
Example:
object_id: glass_04
semantic_class: crystal_tumbler
current_region: kitchen_table
state_variables:
fill_fraction: 0.35
contents: water
The splat does not need to represent every semantic property.
The symbolic layer and visual layer can coexist.
30. Separation of Truth
A useful principle is to separate multiple kinds of "truth."
Symbolic Truth
door_02 is open
glass_04 belongs to Sarah
chair_02 moved after shot 101
Spatial Truth
door transform
glass position
chair position
room topology
Visual Truth
what Sarah looks like
what the kitchen surface looks like
how the chair is textured
Cinematic Truth
camera movement
lighting intent
lens behavior
editing rhythm
performance
No single representation needs to own all four.
Possible ownership:
symbolic truth -> world database / agent state
spatial truth -> engine + splat
visual truth -> references + splat + LoRA
cinematic truth -> H3 + shot plan + motion references
31. The Larger Thesis
This system suggests a broader direction for generative media.
Instead of asking:
"How do we make one model generate a perfectly coherent feature film?"
ask:
"How do we give a powerful multimodal generator the same kinds of external memory, tools, retrieval, state, correction loops, and specialized subsystems that a production pipeline has?"
Then the generator no longer needs to solve:
- memory,
- world modeling,
- identity,
- exact geometry,
- editing,
- reconstruction,
- state management,
- physics,
- and continuity
all inside one forward pass.
Those capabilities can be composed.
This makes a frozen model significantly more capable at the system level.
32. Current Preferred Architecture
STORY / USER INTENT
|
v
DIRECTOR AGENT
|
v
PERSISTENT WORLD STATE
+--------------+--------------+
| |
v v
SYMBOLIC STATE SPATIAL/VISUAL MAP
objects/events persistent Gaussians
relationships transforms/confidence
| |
+--------------+--------------+
|
v
REFERENCE RETRIEVAL
+--------------+--------------+
| | |
v v v
Images Videos Audio
identities motion/camera voices
environments prior shots ambience
appearance playblasts
\ | /
\ | /
+------------+------------+
|
optional LoRAs /
production adapters
|
v
H3 Ref2VA
|
v
GENERATED SHOT
|
+-------------+-------------+
| |
v v
Quality Review 3D Observation
QuerySplat
| |
+-------------+-------------+
|
v
CHANGE ANALYSIS
visibility / confidence /
semantic / geometric diff
|
v
COMMIT DECISION
+-------------+-------------+
| |
v v
reject accept
edit/regenerate local map update
|
v
VERSIONED WORLD STATE
33. The Most Important Research Question
The hardest part is probably not H3 generation or Gaussian rendering.
It is:
How do we decide whether a generated observation has enough evidence to modify canonical world state?
If that decision is too permissive:
- the world slowly corrupts,
- objects disappear,
- geometry drifts,
- and hallucinations become permanent.
If it is too conservative:
- the world never evolves,
- intended changes fail to persist.
The commit algorithm therefore deserves first-class research effort.
A strong first implementation can combine:
- camera/view visibility,
- reconstruction confidence,
- multi-view agreement,
- semantic identity,
- canonical-map confidence,
- object-level state,
- change classification,
- and explicit agent reasoning.
34. Near-Term Experiments Worth Running
Experiment A — Splat Feedback
Generate a room with H3.
QuerySplat it.
Render an unseen view.
Give that view to H3.
Measure whether H3 preserves the world better than a no-splat baseline.
Experiment B — Reference Clip From Splat
Create a rough 3D camera move through the splat.
Use the rough render as H3's spatial/motion reference.
Test whether H3 upgrades it into a photorealistic or stylistically consistent shot while preserving layout.
Experiment C — Engine vs Splat vs Both
Compare:
H3 + image references only
H3 + engine playblast
H3 + splat render
H3 + engine playblast + splat render
Measure:
- geometry,
- appearance,
- motion,
- object permanence,
- identity.
Experiment D — Incremental Room Mutation
Cycle:
generate
-> reconstruct
-> commit one change
-> render new reference
-> generate
Repeat many times.
Measure cumulative drift.
Experiment E — Confidence-Gated Deletion
Place an object in canonical state.
Generate shots where:
- its location is occluded,
- partially visible,
- clearly visible and empty.
Ensure the system deletes the object only in the third case.
Experiment F — Reference LoRA Fallback
Find a character/object H3 does not preserve reliably with reference retrieval alone.
Train a small LoRA.
Compare:
reference only
vs
reference + LoRA
The goal is to identify when parametric memory is actually necessary.
35. Design Principles
- Persistence lives outside the video model.
- H3 should receive shot-specific multimodal context, not the entire world.
- Unobserved space must survive.
- A generated frame is evidence, not automatically truth.
- Only changed regions should be rewritten.
- Prefer transforms over reconstruction for rigid movement.
- Separate geometry changes from appearance changes.
- Keep history and deltas.
- Use LoRAs for persistent learned identity/prior problems, not transient state.
- Use engines for deterministic facts and blocking when useful.
- Use splats as visual/spatial memory, not necessarily final rendering.
- Let H3 beautify crude but geometrically useful references.
- Treat reference selection as multimodal RAG.
- Treat world updates as transactions.
- Confidence must govern commits.
- The architecture should survive model upgrades.
36. Working Name Ideas
Not required for the architecture, but useful shorthand:
- Persistent World Splat
- Generative World Memory
- WorldSplat Memory
- Visual State Map
- Generative Minimap
- Persistent Scene Memory
- World State Splat
- Episodic Scene Graph
- Gaussian World Cache
- Persistent Imaginary-World SLAM
The phrase "generative minimap" is especially useful conceptually because it reinforces that the representation is intentionally sparse and functional rather than a perfect final render.
37. Current Conclusion
The current preferred direction is:
H3 Ref2VA + agentic multimodal retrieval + persistent Gaussian world memory + QuerySplat-style observation lifting + confidence-gated local updates + optional deterministic game-engine state + LoRA fallback for stubborn visual identity.
The Gaussian map functions as a long-lived, editable visual/spatial memory.
H3 renders observations of that world.
QuerySplat or similar systems lift new observations back into candidate 3D.
An agentic commit layer decides what becomes canonical.
The result is a world that can persist beyond the context window and beyond any individual generated clip.
This is potentially enough to turn a very strong video model into something much closer to a persistent generative world simulator without requiring the base model itself to solve every aspect of long-horizon consistency internally.
The key engineering challenge is not simply generating beautiful shots.
It is making the world remember the right things.