viggle-mine-workflow / docs /ARCHITECTURE.md
ZIP Space Deployer
Deploy ZIP β€” 2026-10-14 12:52 UTC
0d8ca70
|
Raw History Blame Contribute Delete
10.8 kB
# Canonical V22 execution and recovered graph
Authority: the attached canonical `HANDOFF.md`, `app.py`, backend and engine files. Experimental `workflow.json` was used only as a feature inventory. The canonical event bindings near the end of `app.py` determine the mapping below. `v22_core.py` preserves the relevant definitions verbatim.
## Coordinate vocabulary
| Space | Meaning |
|---|---|
| `upload_original` | Uploaded source and full-frame Reference A geometry; timestamps are absolute seconds in the original source |
| `normalized_source` | Selected clip, normalized V22 canvas, 24 fps; TAP points are **pixel coordinates**, not fractions in [0,1] |
| `stabilized` | Virtual-camera input coordinates after the saved transforms; Reference B must match this frame geometry |
| `stabilized_generated` | Viggle-generated Pass A in that same stabilized raster; retained for recovery/fusion |
| `normalized_source` final | Generated/composited result on V22's normalized original-side canvas, with original clip audio muxed as in V22 |
Frame index is clip-relative: `round((absolute_source_time - clip_start) * 24)`. Anchor timestamps and frame indices are canonical preparation outputs and cannot be edited independently in the picker. β€œSource-side” final does not mean the uploaded source's original pixel dimensions are restored; V22 uses its normalized canvas.
## Precise action mapping
Inputs named below are canonical arguments, in canonical order. The Workflow controls may be reordered for readability; the adapters call the original functions in their original order.
| Canonical UI action | V22 function | Inputs | Outputs | Downstream consumers | Coordinates | Allocation |
|---|---|---|---|---|---|---|
| Upload/browse source | Native `gr.Video` input | Uploaded media | Source path, visible source browser | Prepare, Standard, A/B source audio | `upload_original` | CPU/browser |
| Prepare clip / anchors | `prepare` | video, duration, start, ref_time, auto_anchors, t1, t2, t3 | 3 display anchors; same 3 originals; 3 empty point JSON strings; normalized video; prep metadata; status | Point picker; TAP; SAM; camera; A/B metadata/audio | Original source β†’ `normalized_source` | CPU/FFmpeg |
| Select subject point | `add_click` in Blocks; equivalent standalone picker coordinate mapping in Workflow | Original anchor, current points, click position | Annotated display image, point list, count | TAP and optional SAM use original anchor geometry + point list | `normalized_source` pixels | Browser/CPU |
| Clear anchor | `clear_clicks`; picker clear/undo controls | Original anchor | Original display, empty list, count | Next saved point revision | `normalized_source` | Browser/CPU |
| Track subject | `run_tracking` β†’ `tap_tracking.track_anchors` | normalized, meta_path, p1, p2, p3, progress | TAP preview, tracks NPZ, status | Camera solver; optional SAM wrapper reuses point groups, not TAP trajectories as SAM prompts | `normalized_source` | One `large` GPU job; CPU preview |
| Optional camera guard | `run_sam_guard` β†’ `sam_guard.track_guard` | normalized, meta_path, p1, p2, p3, progress | SAM preview, masks NPY, status | `solve_transforms` guard input only | `normalized_source` | One `large` GPU job; CPU preview; bypass is CPU only |
| Build virtual camera | `build_zoom` | normalized, meta_path, tracks_path, optional masks_path, Reference A, ref_time, target_pct, smoothing, peripheral_blur, guard_margin, sam_center_assist, max_zoom_change, max_center_speed, progress | Stabilized video and Reference A (each duplicated for preview/state), transforms NPY, status | Pass A; Recovery browser; Pass B; final inverse transform | Source β†’ `stabilized` | CPU/FFmpeg/NumPy |
| Standard generation | `generate_standard` | video, Reference A, start, duration, segment_seconds, seed, audio_mode, progress | Standard final video, status | Final display/download only | Source β†’ normalized generated output | CPU prep + one `xlarge` job for all windows + CPU assembly |
| Generate Pass A | `generate_pass_a` | video, meta_path, stable_video, stable_ref, transforms_path, ref_time, pass_a_backward, segment_seconds, seed, feather, audio_mode, progress | Recomposed final; **generated stabilized video**; status | Final preview; optional Recovery uses generated stabilized output plus camera/preparation checkpoint | `stabilized` generation; `normalized_source` final | CPU prep + one `xlarge` job encompassing every internal window and A branch + CPU recomposition |
| Extract Reference B candidate | `extract_b_candidate` | **stable input video**, meta_path, reacq_time | Exact stabilized frame, status | External repaint, then Reference B upload | `stabilized` | CPU/FFmpeg |
| Reacquire identity | `reacquire_bidirectional` | source video, meta_path, **stable input**, transforms_path, **Pass A stabilized generated crop**, Reference B, reacq_time, backward_compute, transition_center, transition_duration, segment_seconds, seed, feather, audio_mode, progress | Recovered final, status | Final display/download | B generation and A/B fusion in stabilized space; then one inverse transform | CPU prep + a **separate** `xlarge` job encompassing B backward/forward windows + CPU fusion/inverse/audio |
`add_click` is retained byte-for-byte in the extracted core but is not bound to a Workflow `.select()` event. The standalone browser performs the equivalent pixel scaling and saves the structured list. It never asks SAM to composite a final matte.
## Computation inside a pass
`generate_pass_a` and `reacquire_bidirectional` each call `_render_branches_one_gpu`, which builds the complete branch/window lists and calls `_render_gpu_with_ui_progress` once. That invokes the original `backend.render_all_segments`, decorated `spaces.GPU(size="xlarge", duration=batch_gpu_budget)`. Technical windows default to five seconds; the ten-second V22 option remains. They are not graph nodes and are not separate allocations.
Pass A forward-only is the default. Its optional backward and forward branches still share the single A allocation. Pass B's backward computation and short visible transition remain separate controls. B keeps V22's internal seed offset of +7001. A/B are merged using `dynamic_zoom.merge_reacquisition`; only the merged stabilized result is passed through `dynamic_zoom.recompose` once for B's final. A has its own independently useful recomposed final, which is never fed to B.
Startup imports `spaces` before model-related imports. The canonical backend creates its CPU-side global pipeline before ZeroGPU fork, retains its fixed prompt/timestep/scheduler compatibility code, including the `pinned_loop(..., i, t)` signature, and moves/uses the pipeline inside its unchanged decorated rendering function. Generic documentation examples do not override this application-specific initialization strategy.
## Recovered graph
The schema has 8 operators, 3 media references, 9 subjects and 20 edges. Every operator's immediate successor is a subject checkpoint or final, not another operator. This is purposeful: native single-node Run includes one-hop downstream nodes.
| Path | Actual connected nodes |
|---|---|
| Source/spatial | Source β†’ Prepare Source β†’ Point Picker checkpoint β†’ Track Subject β†’ Tracking preview checkpoint β†’ SAM/bypass β†’ Guard checkpoint β†’ Virtual Camera β†’ Stabilized input checkpoint |
| Generation | Reference A β†’ Virtual Camera; stabilized checkpoint β†’ Pass A β†’ final A + saved A checkpoint |
| Recovery | Saved A β†’ Prepare Recovery β†’ candidate/checkpoint β†’ Pass B ← Reference B β†’ recovered final |
| Standard | Shared Source + Reference A β†’ Standard β†’ Standard final; no spatial/generation ancestor |
The two browser interactions are explicit side transactions tied to immutable IDs: Point Picker writes points for a prepared ID; Recovery Browser writes a candidate selection for a completed A ID. Track/Prepare Recovery consume those selections on their next explicit Run. The browser pages neither synthesize a graph edge nor inject events into Gradio.
## Artifacts and ownership
| Kind | Stored/reused contents | Contract / next consumer |
|---|---|---|
| prepared | Canonical metadata, normalized clip, anchors, absolute reference time, source digest | `normalized_source`; picker/TAP |
| tracked | Preparation ID, point revision, anchor groups, tracks, preview | `normalized_source`; SAM/bypass |
| guarded | Tracking/preparation IDs, optional masks, point revision | `normalized_source`; camera |
| camera | Preparation/guard IDs, stabilized input, transformed Reference A, transforms, point revision | `stabilized`; Pass A |
| pass_a | Camera/preparation IDs, final video and separate stabilized generated video, generation settings, take | `stabilized_generated` primary checkpoint; optional Recovery |
| recovery | Saved A ID, camera/preparation IDs, candidate frame, exact absolute time and index | `stabilized`; Pass B |
| pass_b / standard | Final path, settings/status provenance | `normalized_source` final |
Checkpoint cards render native Markdown previews and an Inspect link. Only the opaque capability ID is consumed by downstream adapters. Server-owned JSON stores the typed manifest and lineage; browser-provided paths, metadata or coordinate labels are not accepted as authoritative checkpoints. Media inputs/outputs use native file descriptors, resolved only within upload/output directories. This avoids relying on Workflow preserving arbitrary fields in file dictionaries.
User inputs are the three media references and saved point/frame selections. Computed artifacts are immutable checkpoints. Configurations are scalar native ports. Optional overrides include manual anchor times, SAM enable, camera tuning, A backward mode, and B settings. B inherits the completed A's window/seed/feather/audio values by default and exposes native overrides without editing A's saved inputs. A `-1` transition center selects a UI convenience default up to two seconds before B; an explicit center is forwarded unchanged. This is a documented UI default, not a new fusion algorithm.
## Why the experimental layout was not retained
The prior 46-card graph exposed prepared data and generation intermediates as repeated standalone inputs, used raw JSON control bundles, and separated many technical outputs from their consumers. Edges could be schema-valid while Recovery still required users to manually provide its source/camera/A files. The replacement carries these as one typed lineage and reuses real edges. Native output subjects remain where needed for execution stops and useful previews. Reference A remains shared with Standard; this leaves a long fan-out connection. Rendered wire crossings, spacing and mobile navigation are still an explicit acceptance gate, not a claimed visual success.