ComfyUI-MiniMaxH3-Context-Loop/ComfyUI-MAINodes/docs/roadmaps/persistent_world_design.md

# Persistent Generative World Architecture
## H3 Ref2VA + Persistent Gaussian World Memory + Agentic Composition

**Status:** Working design / architecture thesis  
**Primary goal:** Build a persistent generative video world whose visual and spatial state survives across shots, scenes, camera changes, and long time gaps without requiring the video model itself to remember everything.

---

## 1. Executive Summary

The core idea is to stop treating long-form video coherence as something a single video model must solve internally.

Instead, we externalize persistence into a **world memory layer** and let H3 act as the high-capability multimodal generator that reads from that memory, creates new observations of the world, and writes trustworthy changes back into it.

The preferred world representation is a **persistent Gaussian-splat map** that behaves more like a visual save-game or minimap than a perfect photogrammetric reconstruction.

The map does not need to look cinematic by itself. It only needs to preserve enough information to tell H3:

- where things are,
- what the room roughly looks like,
- what changed,
- what did not change,
- what is currently visible,
- what remains unobserved,
- and how a requested camera should move through the space.

The fundamental persistence rule is:

> **A new shot only has authority over regions it actually observes with sufficient confidence. Unobserved regions remain untouched.**

This allows a room to evolve incrementally.

If a chair moves in one shot, we update that chair.

If a glass disappears, we remove that glass only when the old location was clearly observed.

If the far side of the room was not visible, the old state survives exactly as it was.

The video model therefore does not need infinite memory. It receives **retrieval-augmented visual memory** from the persistence layer whenever it needs to revisit the world.

---

# 2. Why This Architecture

Modern generative video models are becoming powerful enough that many problems previously assumed to require monolithic end-to-end model improvements can instead be solved through:

- agentic composition,
- multimodal retrieval,
- reference selection,
- persistent external state,
- targeted inpainting/editing,
- LoRAs,
- occasional full fine-tunes,
- deterministic 3D tools,
- reconstruction tools,
- and iterative validation.

The architectural thesis is:

> **We may already have enough raw generative capability. The remaining gap is increasingly orchestration, persistence, control, and error correction.**

Rather than waiting for a hypothetical future model to perfectly understand a feature-length world's state, we can build that memory ourselves.

H3 is particularly attractive because its reference-to-video behavior makes the world memory useful even when the memory itself is visually crude.

The persistence layer does not have to generate the final frame.

It has to generate a strong enough **reference observation** for H3 to understand the intended world.

---

# 3. H3 Is the Center, Not Just the Renderer

An earlier version of the architecture placed H3 too far downstream, as though it were primarily a rendering endpoint.

That is not the preferred mental model.

H3 Ref2VA should be treated as the **multimodal generative core**.

It can combine different reference roles such as:

- character appearance,
- object appearance,
- environment appearance,
- style,
- action,
- pose,
- camera behavior,
- motion,
- temporal structure,
- prior footage,
- continuation context,
- and audio.

This means a shot does not need a single perfect reference clip.

The agent can construct a **shot-specific reference package** from many different sources.

Example:

```text
IMAGE 1  -> protagonist identity closeup
IMAGE 2  -> protagonist full-body costume reference
IMAGE 3  -> canonical kitchen appearance
IMAGE 4  -> refrigerator detail
IMAGE 5  -> secondary character identity
IMAGE 6  -> lighting/look reference

VIDEO 1  -> tail of previous shot
VIDEO 2  -> crude engine/splat camera playblast
VIDEO 3  -> hand/object interaction reference

AUDIO 1  -> protagonist voice
AUDIO 2  -> secondary character voice
AUDIO 3  -> room ambience
```

H3 can then synthesize these into a coherent shot.

The important shift is:

> **The persistence system does not need to contain every piece of information in final-quality form. It only needs to supply the correct constraints and references.**

---

# 4. The "Minimap" Mental Model

The persistence layer should not be thought of as a fully accurate digital twin.

A better analogy is a **generative minimap**.

It contains enough persistent information to keep the world coherent.

For a room, that might include:

```text
Kitchen
├── walls
├── windows
├── doors
├── countertop
├── refrigerator
├── table
│   ├── glass_04
│   ├── plate_07
│   └── keys_01
├── chair_01
├── chair_02
├── lighting anchors
└── appearance memory
```

The Gaussian representation gives this map something a pure symbolic world model does not have:

> **visual memory attached to spatial memory.**

The system remembers not only that a countertop exists, but approximately how that countertop appeared from previously observed views.

This is valuable because H3 can consume rendered observations of that state directly.

---

# 5. Persistent Gaussian World State

The preferred persistence behavior is incremental.

Start with a canonical room splat.

Each trustworthy new shot becomes a possible update.

```text
                 CANONICAL ROOM STATE
                 Persistent Gaussian Map
                          |
             +------------+------------+
             |                         |
       Visible in shot             Not visible
             |                         |
             v                         |
      New observation                  |
             |                         |
      classify regions                 |
             |                         |
      +------+------+                  |
      |             |                  |
    same          changed           untouched
      |             |                  |
    keep        replace/update         keep
      |             |                  |
      +------+------+------------------+
             |
             v
       NEW CANONICAL STATE
```

The key rules are:

1. **Observed and unchanged**
   - retain the canonical state.

2. **Observed and changed**
   - update only the changed region.

3. **Observed but uncertain**
   - retain old state unless confidence is sufficient.

4. **Unobserved**
   - never overwrite.

This prevents one generated shot from accidentally destroying parts of the world that were not even visible.

---

# 6. World Updates Are Transactions

Every accepted shot can be viewed as a transaction against world state.

Suppose the current kitchen contains:

```text
table
glass_04
chair_02
east_wall
west_wall
refrigerator
countertop
```

A generated shot observes:

```text
table
chair_02
east_wall
glass_04 location
```

The analysis might produce:

```text
table        -> observed / unchanged
chair_02     -> observed / moved
east_wall    -> observed / unchanged
glass_04     -> observed / removed
west_wall    -> unobserved
refrigerator -> unobserved
countertop   -> unobserved
```

Commit:

```text
table        -> keep
chair_02     -> update transform or local geometry
east_wall    -> keep
glass_04     -> remove
west_wall    -> preserve old state
refrigerator -> preserve old state
countertop   -> preserve old state
```

That is the desired persistence behavior.

---

# 7. Preserve History Instead of Destructive Overwrite

The system should ideally maintain state history.

Example:

```text
Shot 410 -> painting hanging on wall
Shot 411 -> painting falls
Shot 430 -> painting is rehung
```

Instead of discarding the original visual state, the world can maintain deltas:

```text
room_001
├── base_splat
├── delta_0001
├── delta_0002
└── delta_0003
```

Or object history:

```text
painting_07
├── state_01_hanging
├── state_02_floor
└── state_03_rehung
```

Benefits:

- rollback,
- alternate timelines,
- flashbacks,
- continuity debugging,
- reconstruction of earlier shots,
- branchable world state,
- cheaper storage than full checkpoints,
- and reusing prior object states instead of relearning them.

This also makes the architecture useful beyond linear filmmaking.

It can support persistent games, simulations, virtual worlds, and interactive narrative systems.

---

# 8. Local Coordinate Systems

Objects and regions should avoid permanently baking all Gaussian primitives into a single global coordinate frame.

Prefer a hierarchy:

```text
world transform
    *
region transform
    *
object transform
    *
Gaussian local position
```

Example:

```text
WORLD
├── room_kitchen
│   ├── static_shell
│   ├── table_01
│   ├── chair_01
│   ├── chair_02
│   └── glass_04
└── hallway
```

If `chair_02` moves, the system may only need to update:

```text
chair_02.transform
```

instead of modifying thousands of Gaussian primitives.

Only when an object's geometry actually changes should the system perform local Gaussian reconstruction.

This gives a useful hierarchy of corrections:

```text
KEEP
RIGID_REALIGN
SIM3_REALIGN
RECOLOR
LOCAL_OPTIMIZE
PRUNE_AND_RESPLAT
FULL_REGION_RECONSTRUCT
```

An agent can decide which operation is appropriate.

---

# 9. QuerySplat's Role

QuerySplat is potentially very useful, but it should not itself be the persistent memory.

Its best role is:

> **Lift a newly generated H3 observation back into candidate 3D.**

Conceptually:

```text
Persistent World Splat
        |
        v
Render requested camera/reference
        |
        v
H3 Ref2VA
        |
        v
Beautiful new shot
        |
        v
Select useful frames
        |
        v
QuerySplat
        |
        v
Candidate 3D observation
        |
        v
Register against persistent world
        |
        v
Detect trustworthy changes
        |
        v
Local commit
```

QuerySplat turns generated frames into something that can be compared spatially against the current world state.

Useful outputs may include:

- camera estimates,
- depth,
- depth confidence,
- point clouds,
- Gaussian primitives,
- and a coherent local coordinate interpretation.

Those are all useful for deciding whether a region is allowed to modify canonical memory.

---

# 10. QuerySplat Must Not Directly Replace the World

A newly reconstructed splat should be treated as an **observation**, not as truth.

Do not do:

```text
persistent_world = QuerySplat(new_frames)
```

Instead:

```text
candidate = QuerySplat(new_frames)

candidate
    |
    v
register to canonical map
    |
    v
visibility analysis
    |
    v
confidence analysis
    |
    v
semantic correspondence
    |
    v
geometric difference analysis
    |
    v
local change mask
    |
    v
commit approved regions only
```

Two independently generated splats may represent the same physical surface using different Gaussian primitives.

Therefore the merge logic must operate on spatial/semantic evidence rather than assuming Gaussian-to-Gaussian identity.

---

# 11. Related Tool Roles

The emerging stack can be separated by responsibility.

## H3 Ref2VA

**Role:** generative world observation and cinematic synthesis.

Responsibilities:

- combine visual references,
- preserve identities,
- follow motion references,
- synthesize camera behavior,
- create final-quality shots,
- edit bad regions,
- perform continuation,
- and create plausible missing observations.

---

## QuerySplat

**Role:** observation lifting.

Responsibilities:

- take H3-generated frames,
- infer coherent local 3D,
- estimate cameras/depth,
- create a candidate Gaussian representation,
- and provide data for world-state comparison.

---

## Persistent Gaussian Map

**Role:** long-term visual/spatial memory.

Responsibilities:

- retain rooms and objects,
- preserve unseen geometry,
- track visual state,
- store confidence,
- store current and historical state,
- generate rough views from arbitrary cameras.

---

## Local Update / Continual-GS Techniques

Concepts from systems such as CL-Splats and related continual reconstruction work are relevant.

**Role:** safe mutation.

Responsibilities:

- detect changed spatial regions,
- freeze untouched regions,
- update only affected Gaussians,
- maintain compact deltas,
- and avoid catastrophic map replacement.

---

## Change Detection

Gaussian-space or hybrid 2D/3D change detection can act as the gatekeeper.

**Role:** determine what actually changed.

Possible classifications:

```text
UNCHANGED
RIGID_MOVEMENT
GEOMETRY_CHANGE
APPEARANCE_CHANGE
REMOVED
NEW_OBJECT
UNCERTAIN
UNOBSERVED
```

---

## Game Engine

**Role:** optional deterministic topology/physics oracle.

A game engine remains useful, but it does not need to render the final movie.

It can provide:

- collision,
- object transforms,
- door state,
- skeletal pose,
- camera trajectories,
- rough lighting direction,
- room topology,
- physics,
- object identity,
- and deterministic blocking.

The guiding principle is:

> **Use the game engine to tell H3 what happened, not necessarily what the movie should look like.**

An ugly playblast can still be an excellent motion/spatial reference.

---

# 12. 3D -> H3 -> 3D Loop

One of the most important architectural loops is:

```text
              Persistent 3D World
                      |
                      v
              Render reference
                      |
                      v
                     H3
              repair / extend /
              animate / render
                      |
                      v
              Generated video
                      |
                      v
                 QuerySplat
                      |
                      v
             Candidate new 3D
                      |
                      v
               Validate/merge
                      |
                      v
              Persistent 3D World
```

Or more simply:

> **3D -> H3 -> 3D -> H3 -> 3D**

This lets the world evolve over time.

H3 can imagine a new observation.

QuerySplat or similar tools can lift that observation back into spatial memory.

The persistence layer then decides whether the new information is trustworthy enough to become canonical.

---

# 13. Expanding Into Previously Unseen Space

This architecture becomes especially interesting when the camera enters a region that has never been modeled.

Suppose the world contains:

```text
kitchen
hallway
living room
unknown room
```

The system can:

1. render the known hallway and doorway,
2. request a camera trajectory into the unknown area,
3. give H3 canonical style/environment references,
4. ask H3 to generate the continuation,
5. reconstruct useful frames into a candidate 3D region,
6. align that region to the known doorway/hallway,
7. validate it,
8. commit it as a new part of the world.

Conceptually:

```text
KNOWN WORLD
    |
    v
reference camera trajectory
    |
    v
H3 generates plausible unseen space
    |
    v
QuerySplat
    |
    v
new candidate geometry
    |
    v
align to known anchors
    |
    v
commit
    |
    v
WORLD EXPANDS
```

Once committed, that room is no longer pure hallucination.

It has become persistent world memory.

Future shots can retrieve and render it again.

---

# 14. Sparse Skeleton, Not Perfect Reconstruction

The Gaussian world does **not** need to be a perfect photorealistic scene.

This is an important design choice.

The goal is not necessarily:

> render the persistent splat directly as the final image.

The goal is:

> render enough spatial truth for H3 to reconstruct the intended world.

The persistent world mainly needs:

- surfaces,
- relative layout,
- rough material/color cues,
- object placement,
- depth,
- occlusion,
- camera parallax,
- persistent clutter,
- visual anchors.

If the far side of a couch contains ugly sparse Gaussians, that may be acceptable.

H3 can clean up the appearance.

This lowers the reconstruction burden dramatically.

---

# 15. Confidence Is a First-Class Property

Every persistent region should carry confidence.

Possible fields:

```text
position
rotation
scale
appearance
semantic_id
object_id

last_observed_shot
observation_count

geometry_confidence
appearance_confidence
identity_confidence
registration_confidence

state:
  canonical
  changed
  uncertain
  historical
```

A low-confidence region should not be allowed to overwrite a high-confidence canonical region.

This is especially important with generated video because visual plausibility does not imply spatial truth.

The commit system should distinguish:

> "H3 generated something convincing"

from:

> "we have enough evidence that the canonical world really changed."

---

# 16. Persistent State vs References vs LoRAs

Different forms of memory should have different jobs.

## World Map = Episodic / Mutable Facts

Examples:

- the glass currently sits on the table,
- the chair is rotated 20 degrees,
- the door is open,
- the character left the room,
- the painting fell,
- the refrigerator contains a cake.

These facts should not require fine-tuning.

---

## Reference Library = Canonical Visual Memory

Examples:

- exact face reference,
- exact costume,
- canonical room look,
- prop details,
- lighting style,
- known camera treatment,
- prior shots from useful angles.

The agent retrieves these when preparing a shot.

---

## LoRAs = Persistent Learned Priors

LoRAs are useful when the base model repeatedly fails to honor references reliably.

Examples:

- character identity,
- recurring creature design,
- studio aesthetic,
- specific cinematography grammar,
- hard-to-preserve wardrobe,
- unusual recurring objects,
- production-specific motion language.

The principle is:

> **Do not encode transient world facts into a LoRA.**

A moved chair belongs in the map.

A stubborn character identity may belong in a LoRA.

---

## Full Fine-Tunes = Behavioral / Domain Priors

Full fine-tuning may eventually be appropriate for:

- production-specific reference following,
- particular cinematographic behavior,
- better persistence responses,
- improved handling of our rendered conditioning style,
- better interaction with synthetic engine/splat references,
- or specific animation domains.

But full fine-tuning is not the first requirement.

The external architecture should work independently.

---

# 17. Agentic Reference Composition

The world may eventually contain tens of thousands of possible reference assets.

H3 should not receive all of them.

An agent should build a **shot-specific multimodal context pack**.

Inputs to the agent:

```text
script state
scene state
camera request
characters present
objects visible
current world map
previous shot
continuity requirements
style requirements
motion requirements
audio requirements
```

Output:

```text
selected image references
selected video references
selected audio references
persistent-world render
engine playblast if useful
previous-shot tail
LoRA set
prompt / structured instructions
```

The agent is effectively performing:

> **multimodal RAG for video generation.**

Instead of retrieving text passages, it retrieves:

- images,
- videos,
- audio,
- geometry,
- world state,
- object state,
- previous shots,
- motion clips,
- and learned adapters.

---

# 18. Generation and Validation Loop

The production loop can be:

```text
                     STORY / TASK
                          |
                          v
                    Director Agent
                          |
                          v
                   Shot Requirements
                          |
                          v
              Retrieve Persistent State
                          |
                          v
               Build Reference Package
                          |
                          v
                         H3
                          |
                          v
                  Generated Candidate
                          |
              +-----------+-----------+
              |                       |
              v                       v
       Visual Evaluator        Spatial Evaluator
              |                       |
              +-----------+-----------+
                          |
                          v
                 Trustworthy result?
                     /          \
                   yes           no
                    |             |
                    v             v
            Commit observations   edit/inpaint/
              to world state      regenerate
                    |
                    v
             Updated world memory
```

The world map therefore becomes both:

- an input to generation,
- and a recipient of validated output.

---

# 19. The Commit Algorithm

The commit algorithm may be the most important technical component in the whole architecture.

Its responsibility:

> Given the previous canonical world and a newly generated H3 clip, determine exactly which spatial regions are allowed to modify persistent state.

A possible pipeline:

```text
H3 clip
  |
  v
frame selection
  |
  v
QuerySplat / depth / camera inference
  |
  v
candidate local 3D
  |
  v
register candidate to canonical world
  |
  v
visibility determination
  |
  v
semantic association
  |
  v
geometry + appearance comparison
  |
  v
confidence calculation
  |
  v
change classification
  |
  v
approved mutation mask
  |
  v
transactional world update
```

Important rules:

### Rule 1: Unobserved Means Immutable

If the new shot does not see a region, the region cannot be modified.

### Rule 2: Low Confidence Cannot Destroy High Confidence

A weak reconstruction cannot overwrite a stable canonical surface.

### Rule 3: Rigid Movement Should Prefer Transform Updates

Do not reconstruct an object if its identity and geometry are still valid.

### Rule 4: Appearance and Geometry Are Different

A lighting/color shift should not automatically be interpreted as geometry change.

### Rule 5: Deletion Requires Evidence

An object disappears from the world only when its former location is sufficiently observed and proven empty.

### Rule 6: New Geometry Begins as Provisional

Newly hallucinated regions may require repeated confirmation before becoming high-confidence canonical state.

### Rule 7: Keep History

Accepted updates should preferably be stored as deltas or versioned states.

---

# 20. Example: Character Moves a Chair and Takes a Glass

Initial state:

```text
Kitchen:
  chair_02 = at table, rotation 0 degrees
  glass_04 = on table, 35% full
```

Shot action:

```text
Sarah enters.
Sarah rotates chair_02.
Sarah picks up glass_04.
Camera follows Sarah out.
```

After generation:

```text
QuerySplat / reconstruction
        |
        v
compare to canonical kitchen
```

Detected:

```text
chair_02 -> same object, new rigid transform
glass_04 -> removed from table
table -> unchanged
east wall -> unchanged
refrigerator -> unobserved
countertop -> unobserved
```

Commit:

```text
chair_02.transform = new transform

glass_04.location = Sarah/right_hand
or later:
glass_04.location = unknown/carried

table = unchanged
east wall = unchanged

refrigerator = untouched
countertop = untouched
```

No full room reconstruction occurs.

---

# 21. Example: Geometry Actually Changes

Suppose a table is broken.

This is not a rigid transform.

The agent may choose:

```text
LOCAL_OPTIMIZE
```

or:

```text
PRUNE_AND_RESPLAT
```

Pipeline:

```text
detect changed table region
        |
        v
freeze rest of room
        |
        v
remove invalid table Gaussians
        |
        v
reconstruct broken geometry
        |
        v
blend boundary
        |
        v
commit table delta
```

Everything else remains stable.

---

# 22. Example: H3 Drifts on Character Identity

Suppose H3 repeatedly changes a character's face at certain angles even with good reference retrieval.

Do not change the world map.

Instead:

```text
collect successful identity observations
        |
        v
train/refine character identity LoRA
        |
        v
evaluate against reference-only baseline
        |
        v
use LoRA when confidence improvement is real
```

This separates:

```text
WORLD FACT
"Sarah is standing by the refrigerator."
```

from:

```text
MODEL PRIOR
"This is what Sarah looks like from every angle."
```

---

# 23. Why Gaussian Splats Instead of Only a Game Engine

A game engine is extremely useful for exact state.

But a pure engine representation has a weakness:

> it stores geometry/state well, but it does not naturally retain the generative model's actual visual memory of the environment.

Gaussian splats give us:

- view-dependent appearance,
- captured texture,
- clutter,
- lighting cues,
- imperfect but useful scene appearance,
- and a renderable observation from arbitrary cameras.

The ideal system may therefore combine them:

```text
              SYMBOLIC / ENGINE STATE
                 exact facts
                     |
                     v
              object transforms
              topology / physics
                     |
                     +
                     |
                     v
              GAUSSIAN WORLD MEMORY
               visual/spatial state
                     |
                     v
              rough rendered reference
                     |
                     v
                    H3
```

The engine is deterministic truth.

The splat is visual memory.

H3 is the cinematic imagination layer.

---

# 24. Why Not Just Use Previous Video Frames?

Previous frames are useful but insufficient.

They fail when:

- the camera visits a new viewpoint,
- a long time has passed,
- an object was last visible from the wrong side,
- the new shot needs geometry hidden in prior footage,
- the room has undergone multiple changes,
- or the previous clip does not expose the required spatial relationships.

The persistent splat can render a new approximate reference from a camera angle that was never previously filmed.

That is a major advantage.

---

# 25. Why Not Require the Video Model to Remember Everything?

Because long-term persistence is fundamentally different from generation quality.

A model can be excellent at producing a coherent 10-second clip and still fail to remember:

- which shelf an object was on 100 shots ago,
- whether a window was open,
- how far a chair had moved,
- what was behind the camera,
- or which version of a room is currently canonical.

Externalizing that memory gives us:

- inspectability,
- editability,
- determinism,
- branching,
- rollback,
- explicit state,
- and model independence.

A future H3 successor can replace H3 without requiring the entire persistence architecture to be rebuilt.

---

# 26. Model Upgrades Become Replaceable Components

One of the strongest reasons to pursue this architecture is longevity.

The stack can remain:

```text
persistent world
reference library
agentic retrieval
commit logic
history
validation
LoRAs
engine integration
```

while the generative model changes:

```text
H3
 -> H4
 -> future Seedance-class model
 -> something better
```

The orchestration and world memory remain valuable.

This is one reason the system can potentially achieve capability beyond the apparent limits of any single frozen model.

---

# 27. Prototype Architecture

A practical first prototype does not need the full system.

## Phase 1: Closed Room Loop

Goal:

> Demonstrate that an H3-generated room can be reconstructed, re-rendered from a new angle, fed back to H3, and remain recognizably persistent.

Pipeline:

```text
1. Generate 5-10 second H3 room shot
2. Sample useful frames
3. Run QuerySplat
4. Render novel camera angles
5. Feed rendered views back to H3 Ref2VA
6. Generate another shot
7. Compare object/layout persistence
```

Success criterion:

- the next H3 shot preserves major room layout and object identity better with the splat-derived reference than without it.

---

## Phase 2: Controlled Object Movement

Goal:

> Update only one changed object.

Example:

```text
chair moves
```

Pipeline:

```text
canonical room splat
    |
H3 shot moves chair
    |
QuerySplat candidate
    |
register
    |
detect chair movement
    |
preserve room
    |
update chair only
```

Success criterion:

- unchanged room regions survive multiple update cycles.

---

## Phase 3: Object Removal and Addition

Test:

```text
glass on table
 -> character removes glass
 -> later returns glass
```

Validate:

- safe deletion,
- preserved table geometry,
- correct historical state,
- restoration from prior object memory.

---

## Phase 4: New Space Expansion

Test:

```text
known hallway
 -> H3 generates unseen bedroom
 -> QuerySplat reconstructs bedroom
 -> register bedroom to hallway
 -> commit
 -> revisit bedroom later
```

Success criterion:

- newly imagined space becomes persistent.

---

## Phase 5: Hybrid Engine + Splat Conditioning

Use an engine to generate:

- deterministic actor blocking,
- exact camera path,
- object transforms,
- rough room geometry.

Then combine:

```text
engine playblast
+
persistent splat render
+
identity refs
+
style refs
+
previous-shot tail
+
H3 Ref2VA
```

Measure whether this improves:

- camera control,
- object permanence,
- hand-object interactions,
- action continuity.

---

## Phase 6: Agentic Commit Decisions

Replace hand-written update selection with an agent plus deterministic tools.

Agent outputs:

```text
KEEP
RIGID_REALIGN
SIM3_REALIGN
RECOLOR
LOCAL_OPTIMIZE
PRUNE_AND_RESPLAT
FULL_REGION_RECONSTRUCT
```

The agent does not perform numerical registration itself.

It selects and orchestrates the appropriate specialized tools.

---

# 28. Evaluation Metrics

We should evaluate more than image beauty.

## Persistence

- object position drift,
- orientation drift,
- object count consistency,
- room topology consistency,
- layout stability,
- identity stability.

## Generative Quality

- visual quality,
- motion quality,
- reference fidelity,
- cinematography,
- temporal coherence.

## World Update Safety

- false deletions,
- false additions,
- incorrect geometry overwrite,
- corruption of unobserved regions,
- confidence calibration.

## Efficiency

- reconstruction latency,
- H3 generation cost,
- map update cost,
- storage growth,
- amount of scene needing reoptimization,
- reference retrieval overhead.

## Long-Horizon Robustness

Run the world through dozens or hundreds of updates and measure whether static regions drift over time.

This may be one of the most important tests.

---

# 29. Potential Data Model

A region/object representation might contain:

```yaml
region_id:
parent_region:
object_ids: []

transform:
  translation:
  rotation:
  scale:

geometry:
  gaussian_chunk:
  collision_proxy:
  bounds:

appearance:
  canonical_refs: []
  embeddings: []
  material_hints:

confidence:
  geometry:
  appearance:
  registration:
  semantic_identity:

history:
  created_at_shot:
  last_observed_shot:
  observation_count:
  versions: []

state:
  canonical:
  provisional:
  changed:
  uncertain:
```

Objects may additionally contain:

```yaml
object_id:
semantic_class:
identity_ref:
lora:
current_region:
current_transform:
state_variables:
```

Example:

```yaml
object_id: glass_04
semantic_class: crystal_tumbler
current_region: kitchen_table
state_variables:
  fill_fraction: 0.35
  contents: water
```

The splat does not need to represent every semantic property.

The symbolic layer and visual layer can coexist.

---

# 30. Separation of Truth

A useful principle is to separate multiple kinds of "truth."

## Symbolic Truth

```text
door_02 is open
glass_04 belongs to Sarah
chair_02 moved after shot 101
```

## Spatial Truth

```text
door transform
glass position
chair position
room topology
```

## Visual Truth

```text
what Sarah looks like
what the kitchen surface looks like
how the chair is textured
```

## Cinematic Truth

```text
camera movement
lighting intent
lens behavior
editing rhythm
performance
```

No single representation needs to own all four.

Possible ownership:

```text
symbolic truth  -> world database / agent state
spatial truth   -> engine + splat
visual truth    -> references + splat + LoRA
cinematic truth -> H3 + shot plan + motion references
```

---

# 31. The Larger Thesis

This system suggests a broader direction for generative media.

Instead of asking:

> "How do we make one model generate a perfectly coherent feature film?"

ask:

> "How do we give a powerful multimodal generator the same kinds of external memory, tools, retrieval, state, correction loops, and specialized subsystems that a production pipeline has?"

Then the generator no longer needs to solve:

- memory,
- world modeling,
- identity,
- exact geometry,
- editing,
- reconstruction,
- state management,
- physics,
- and continuity

all inside one forward pass.

Those capabilities can be composed.

This makes a frozen model significantly more capable at the system level.

---

# 32. Current Preferred Architecture

```text
                          STORY / USER INTENT
                                  |
                                  v
                           DIRECTOR AGENT
                                  |
                                  v
                       PERSISTENT WORLD STATE
                   +--------------+--------------+
                   |                             |
                   v                             v
             SYMBOLIC STATE               SPATIAL/VISUAL MAP
             objects/events              persistent Gaussians
             relationships               transforms/confidence
                   |                             |
                   +--------------+--------------+
                                  |
                                  v
                         REFERENCE RETRIEVAL
                   +--------------+--------------+
                   |              |              |
                   v              v              v
                Images          Videos          Audio
              identities       motion/camera    voices
              environments     prior shots      ambience
              appearance       playblasts
                   \              |              /
                    \             |             /
                     +------------+------------+
                                  |
                         optional LoRAs /
                         production adapters
                                  |
                                  v
                              H3 Ref2VA
                                  |
                                  v
                         GENERATED SHOT
                                  |
                    +-------------+-------------+
                    |                           |
                    v                           v
              Quality Review              3D Observation
                                            QuerySplat
                    |                           |
                    +-------------+-------------+
                                  |
                                  v
                          CHANGE ANALYSIS
                     visibility / confidence /
                     semantic / geometric diff
                                  |
                                  v
                           COMMIT DECISION
                    +-------------+-------------+
                    |                           |
                    v                           v
                  reject                     accept
              edit/regenerate           local map update
                                                |
                                                v
                                      VERSIONED WORLD STATE
```

---

# 33. The Most Important Research Question

The hardest part is probably not H3 generation or Gaussian rendering.

It is:

> **How do we decide whether a generated observation has enough evidence to modify canonical world state?**

If that decision is too permissive:

- the world slowly corrupts,
- objects disappear,
- geometry drifts,
- and hallucinations become permanent.

If it is too conservative:

- the world never evolves,
- intended changes fail to persist.

The commit algorithm therefore deserves first-class research effort.

A strong first implementation can combine:

- camera/view visibility,
- reconstruction confidence,
- multi-view agreement,
- semantic identity,
- canonical-map confidence,
- object-level state,
- change classification,
- and explicit agent reasoning.

---

# 34. Near-Term Experiments Worth Running

### Experiment A — Splat Feedback

Generate a room with H3.

QuerySplat it.

Render an unseen view.

Give that view to H3.

Measure whether H3 preserves the world better than a no-splat baseline.

---

### Experiment B — Reference Clip From Splat

Create a rough 3D camera move through the splat.

Use the rough render as H3's spatial/motion reference.

Test whether H3 upgrades it into a photorealistic or stylistically consistent shot while preserving layout.

---

### Experiment C — Engine vs Splat vs Both

Compare:

```text
H3 + image references only
H3 + engine playblast
H3 + splat render
H3 + engine playblast + splat render
```

Measure:

- geometry,
- appearance,
- motion,
- object permanence,
- identity.

---

### Experiment D — Incremental Room Mutation

Cycle:

```text
generate
 -> reconstruct
 -> commit one change
 -> render new reference
 -> generate
```

Repeat many times.

Measure cumulative drift.

---

### Experiment E — Confidence-Gated Deletion

Place an object in canonical state.

Generate shots where:

- its location is occluded,
- partially visible,
- clearly visible and empty.

Ensure the system deletes the object only in the third case.

---

### Experiment F — Reference LoRA Fallback

Find a character/object H3 does not preserve reliably with reference retrieval alone.

Train a small LoRA.

Compare:

```text
reference only
vs
reference + LoRA
```

The goal is to identify when parametric memory is actually necessary.

---

# 35. Design Principles

1. **Persistence lives outside the video model.**
2. **H3 should receive shot-specific multimodal context, not the entire world.**
3. **Unobserved space must survive.**
4. **A generated frame is evidence, not automatically truth.**
5. **Only changed regions should be rewritten.**
6. **Prefer transforms over reconstruction for rigid movement.**
7. **Separate geometry changes from appearance changes.**
8. **Keep history and deltas.**
9. **Use LoRAs for persistent learned identity/prior problems, not transient state.**
10. **Use engines for deterministic facts and blocking when useful.**
11. **Use splats as visual/spatial memory, not necessarily final rendering.**
12. **Let H3 beautify crude but geometrically useful references.**
13. **Treat reference selection as multimodal RAG.**
14. **Treat world updates as transactions.**
15. **Confidence must govern commits.**
16. **The architecture should survive model upgrades.**

---

# 36. Working Name Ideas

Not required for the architecture, but useful shorthand:

- Persistent World Splat
- Generative World Memory
- WorldSplat Memory
- Visual State Map
- Generative Minimap
- Persistent Scene Memory
- World State Splat
- Episodic Scene Graph
- Gaussian World Cache
- Persistent Imaginary-World SLAM

The phrase **"generative minimap"** is especially useful conceptually because it reinforces that the representation is intentionally sparse and functional rather than a perfect final render.

---

# 37. Current Conclusion

The current preferred direction is:

> **H3 Ref2VA + agentic multimodal retrieval + persistent Gaussian world memory + QuerySplat-style observation lifting + confidence-gated local updates + optional deterministic game-engine state + LoRA fallback for stubborn visual identity.**

The Gaussian map functions as a long-lived, editable visual/spatial memory.

H3 renders observations of that world.

QuerySplat or similar systems lift new observations back into candidate 3D.

An agentic commit layer decides what becomes canonical.

The result is a world that can persist beyond the context window and beyond any individual generated clip.

This is potentially enough to turn a very strong video model into something much closer to a **persistent generative world simulator** without requiring the base model itself to solve every aspect of long-horizon consistency internally.

The key engineering challenge is not simply generating beautiful shots.

It is making the world remember the right things.