viggle-mine-workflow / docs /ARCHITECTURE.md
ZIP Space Deployer
Deploy ZIP β€” 2026-10-14 12:52 UTC
0d8ca70
|
Raw History Blame Contribute Delete
10.8 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

Canonical V22 execution and recovered graph

Authority: the attached canonical HANDOFF.md, app.py, backend and engine files. Experimental workflow.json was used only as a feature inventory. The canonical event bindings near the end of app.py determine the mapping below. v22_core.py preserves the relevant definitions verbatim.

Coordinate vocabulary

Space Meaning
upload_original Uploaded source and full-frame Reference A geometry; timestamps are absolute seconds in the original source
normalized_source Selected clip, normalized V22 canvas, 24 fps; TAP points are pixel coordinates, not fractions in [0,1]
stabilized Virtual-camera input coordinates after the saved transforms; Reference B must match this frame geometry
stabilized_generated Viggle-generated Pass A in that same stabilized raster; retained for recovery/fusion
normalized_source final Generated/composited result on V22's normalized original-side canvas, with original clip audio muxed as in V22

Frame index is clip-relative: round((absolute_source_time - clip_start) * 24). Anchor timestamps and frame indices are canonical preparation outputs and cannot be edited independently in the picker. β€œSource-side” final does not mean the uploaded source's original pixel dimensions are restored; V22 uses its normalized canvas.

Precise action mapping

Inputs named below are canonical arguments, in canonical order. The Workflow controls may be reordered for readability; the adapters call the original functions in their original order.

Canonical UI action V22 function Inputs Outputs Downstream consumers Coordinates Allocation
Upload/browse source Native gr.Video input Uploaded media Source path, visible source browser Prepare, Standard, A/B source audio upload_original CPU/browser
Prepare clip / anchors prepare video, duration, start, ref_time, auto_anchors, t1, t2, t3 3 display anchors; same 3 originals; 3 empty point JSON strings; normalized video; prep metadata; status Point picker; TAP; SAM; camera; A/B metadata/audio Original source β†’ normalized_source CPU/FFmpeg
Select subject point add_click in Blocks; equivalent standalone picker coordinate mapping in Workflow Original anchor, current points, click position Annotated display image, point list, count TAP and optional SAM use original anchor geometry + point list normalized_source pixels Browser/CPU
Clear anchor clear_clicks; picker clear/undo controls Original anchor Original display, empty list, count Next saved point revision normalized_source Browser/CPU
Track subject run_tracking β†’ tap_tracking.track_anchors normalized, meta_path, p1, p2, p3, progress TAP preview, tracks NPZ, status Camera solver; optional SAM wrapper reuses point groups, not TAP trajectories as SAM prompts normalized_source One large GPU job; CPU preview
Optional camera guard run_sam_guard β†’ sam_guard.track_guard normalized, meta_path, p1, p2, p3, progress SAM preview, masks NPY, status solve_transforms guard input only normalized_source One large GPU job; CPU preview; bypass is CPU only
Build virtual camera build_zoom normalized, meta_path, tracks_path, optional masks_path, Reference A, ref_time, target_pct, smoothing, peripheral_blur, guard_margin, sam_center_assist, max_zoom_change, max_center_speed, progress Stabilized video and Reference A (each duplicated for preview/state), transforms NPY, status Pass A; Recovery browser; Pass B; final inverse transform Source β†’ stabilized CPU/FFmpeg/NumPy
Standard generation generate_standard video, Reference A, start, duration, segment_seconds, seed, audio_mode, progress Standard final video, status Final display/download only Source β†’ normalized generated output CPU prep + one xlarge job for all windows + CPU assembly
Generate Pass A generate_pass_a video, meta_path, stable_video, stable_ref, transforms_path, ref_time, pass_a_backward, segment_seconds, seed, feather, audio_mode, progress Recomposed final; generated stabilized video; status Final preview; optional Recovery uses generated stabilized output plus camera/preparation checkpoint stabilized generation; normalized_source final CPU prep + one xlarge job encompassing every internal window and A branch + CPU recomposition
Extract Reference B candidate extract_b_candidate stable input video, meta_path, reacq_time Exact stabilized frame, status External repaint, then Reference B upload stabilized CPU/FFmpeg
Reacquire identity reacquire_bidirectional source video, meta_path, stable input, transforms_path, Pass A stabilized generated crop, Reference B, reacq_time, backward_compute, transition_center, transition_duration, segment_seconds, seed, feather, audio_mode, progress Recovered final, status Final display/download B generation and A/B fusion in stabilized space; then one inverse transform CPU prep + a separate xlarge job encompassing B backward/forward windows + CPU fusion/inverse/audio

add_click is retained byte-for-byte in the extracted core but is not bound to a Workflow .select() event. The standalone browser performs the equivalent pixel scaling and saves the structured list. It never asks SAM to composite a final matte.

Computation inside a pass

generate_pass_a and reacquire_bidirectional each call _render_branches_one_gpu, which builds the complete branch/window lists and calls _render_gpu_with_ui_progress once. That invokes the original backend.render_all_segments, decorated spaces.GPU(size="xlarge", duration=batch_gpu_budget). Technical windows default to five seconds; the ten-second V22 option remains. They are not graph nodes and are not separate allocations.

Pass A forward-only is the default. Its optional backward and forward branches still share the single A allocation. Pass B's backward computation and short visible transition remain separate controls. B keeps V22's internal seed offset of +7001. A/B are merged using dynamic_zoom.merge_reacquisition; only the merged stabilized result is passed through dynamic_zoom.recompose once for B's final. A has its own independently useful recomposed final, which is never fed to B.

Startup imports spaces before model-related imports. The canonical backend creates its CPU-side global pipeline before ZeroGPU fork, retains its fixed prompt/timestep/scheduler compatibility code, including the pinned_loop(..., i, t) signature, and moves/uses the pipeline inside its unchanged decorated rendering function. Generic documentation examples do not override this application-specific initialization strategy.

Recovered graph

The schema has 8 operators, 3 media references, 9 subjects and 20 edges. Every operator's immediate successor is a subject checkpoint or final, not another operator. This is purposeful: native single-node Run includes one-hop downstream nodes.

Path Actual connected nodes
Source/spatial Source β†’ Prepare Source β†’ Point Picker checkpoint β†’ Track Subject β†’ Tracking preview checkpoint β†’ SAM/bypass β†’ Guard checkpoint β†’ Virtual Camera β†’ Stabilized input checkpoint
Generation Reference A β†’ Virtual Camera; stabilized checkpoint β†’ Pass A β†’ final A + saved A checkpoint
Recovery Saved A β†’ Prepare Recovery β†’ candidate/checkpoint β†’ Pass B ← Reference B β†’ recovered final
Standard Shared Source + Reference A β†’ Standard β†’ Standard final; no spatial/generation ancestor

The two browser interactions are explicit side transactions tied to immutable IDs: Point Picker writes points for a prepared ID; Recovery Browser writes a candidate selection for a completed A ID. Track/Prepare Recovery consume those selections on their next explicit Run. The browser pages neither synthesize a graph edge nor inject events into Gradio.

Artifacts and ownership

Kind Stored/reused contents Contract / next consumer
prepared Canonical metadata, normalized clip, anchors, absolute reference time, source digest normalized_source; picker/TAP
tracked Preparation ID, point revision, anchor groups, tracks, preview normalized_source; SAM/bypass
guarded Tracking/preparation IDs, optional masks, point revision normalized_source; camera
camera Preparation/guard IDs, stabilized input, transformed Reference A, transforms, point revision stabilized; Pass A
pass_a Camera/preparation IDs, final video and separate stabilized generated video, generation settings, take stabilized_generated primary checkpoint; optional Recovery
recovery Saved A ID, camera/preparation IDs, candidate frame, exact absolute time and index stabilized; Pass B
pass_b / standard Final path, settings/status provenance normalized_source final

Checkpoint cards render native Markdown previews and an Inspect link. Only the opaque capability ID is consumed by downstream adapters. Server-owned JSON stores the typed manifest and lineage; browser-provided paths, metadata or coordinate labels are not accepted as authoritative checkpoints. Media inputs/outputs use native file descriptors, resolved only within upload/output directories. This avoids relying on Workflow preserving arbitrary fields in file dictionaries.

User inputs are the three media references and saved point/frame selections. Computed artifacts are immutable checkpoints. Configurations are scalar native ports. Optional overrides include manual anchor times, SAM enable, camera tuning, A backward mode, and B settings. B inherits the completed A's window/seed/feather/audio values by default and exposes native overrides without editing A's saved inputs. A -1 transition center selects a UI convenience default up to two seconds before B; an explicit center is forwarded unchanged. This is a documented UI default, not a new fusion algorithm.

Why the experimental layout was not retained

The prior 46-card graph exposed prepared data and generation intermediates as repeated standalone inputs, used raw JSON control bundles, and separated many technical outputs from their consumers. Edges could be schema-valid while Recovery still required users to manually provide its source/camera/A files. The replacement carries these as one typed lineage and reuses real edges. Native output subjects remain where needed for execution stops and useful previews. Reference A remains shared with Standard; this leaves a long fan-out connection. Rendered wire crossings, spacing and mobile navigation are still an explicit acceptance gate, not a claimed visual success.