Spaces:
Running on Zero
Running on Zero
|
Download docs/ARCHITECTURE.md from fffiloni/viggle-mine-workflow: direct link, hf CLI and curl.
- Browser
- Download file 10.8 kB
-
https://huggingface.co/spaces/fffiloni/viggle-mine-workflow/resolve/main/docs/ARCHITECTURE.md
- Command line
-
hf download hf://spaces/fffiloni/viggle-mine-workflow/docs/ARCHITECTURE.md
-
curl -L -o ARCHITECTURE.md https://huggingface.co/spaces/fffiloni/viggle-mine-workflow/resolve/main/docs/ARCHITECTURE.md
10.8 kB
| # Canonical V22 execution and recovered graph | |
| Authority: the attached canonical `HANDOFF.md`, `app.py`, backend and engine files. Experimental `workflow.json` was used only as a feature inventory. The canonical event bindings near the end of `app.py` determine the mapping below. `v22_core.py` preserves the relevant definitions verbatim. | |
| ## Coordinate vocabulary | |
| | Space | Meaning | | |
| |---|---| | |
| | `upload_original` | Uploaded source and full-frame Reference A geometry; timestamps are absolute seconds in the original source | | |
| | `normalized_source` | Selected clip, normalized V22 canvas, 24 fps; TAP points are **pixel coordinates**, not fractions in [0,1] | | |
| | `stabilized` | Virtual-camera input coordinates after the saved transforms; Reference B must match this frame geometry | | |
| | `stabilized_generated` | Viggle-generated Pass A in that same stabilized raster; retained for recovery/fusion | | |
| | `normalized_source` final | Generated/composited result on V22's normalized original-side canvas, with original clip audio muxed as in V22 | | |
| Frame index is clip-relative: `round((absolute_source_time - clip_start) * 24)`. Anchor timestamps and frame indices are canonical preparation outputs and cannot be edited independently in the picker. βSource-sideβ final does not mean the uploaded source's original pixel dimensions are restored; V22 uses its normalized canvas. | |
| ## Precise action mapping | |
| Inputs named below are canonical arguments, in canonical order. The Workflow controls may be reordered for readability; the adapters call the original functions in their original order. | |
| | Canonical UI action | V22 function | Inputs | Outputs | Downstream consumers | Coordinates | Allocation | | |
| |---|---|---|---|---|---|---| | |
| | Upload/browse source | Native `gr.Video` input | Uploaded media | Source path, visible source browser | Prepare, Standard, A/B source audio | `upload_original` | CPU/browser | | |
| | Prepare clip / anchors | `prepare` | video, duration, start, ref_time, auto_anchors, t1, t2, t3 | 3 display anchors; same 3 originals; 3 empty point JSON strings; normalized video; prep metadata; status | Point picker; TAP; SAM; camera; A/B metadata/audio | Original source β `normalized_source` | CPU/FFmpeg | | |
| | Select subject point | `add_click` in Blocks; equivalent standalone picker coordinate mapping in Workflow | Original anchor, current points, click position | Annotated display image, point list, count | TAP and optional SAM use original anchor geometry + point list | `normalized_source` pixels | Browser/CPU | | |
| | Clear anchor | `clear_clicks`; picker clear/undo controls | Original anchor | Original display, empty list, count | Next saved point revision | `normalized_source` | Browser/CPU | | |
| | Track subject | `run_tracking` β `tap_tracking.track_anchors` | normalized, meta_path, p1, p2, p3, progress | TAP preview, tracks NPZ, status | Camera solver; optional SAM wrapper reuses point groups, not TAP trajectories as SAM prompts | `normalized_source` | One `large` GPU job; CPU preview | | |
| | Optional camera guard | `run_sam_guard` β `sam_guard.track_guard` | normalized, meta_path, p1, p2, p3, progress | SAM preview, masks NPY, status | `solve_transforms` guard input only | `normalized_source` | One `large` GPU job; CPU preview; bypass is CPU only | | |
| | Build virtual camera | `build_zoom` | normalized, meta_path, tracks_path, optional masks_path, Reference A, ref_time, target_pct, smoothing, peripheral_blur, guard_margin, sam_center_assist, max_zoom_change, max_center_speed, progress | Stabilized video and Reference A (each duplicated for preview/state), transforms NPY, status | Pass A; Recovery browser; Pass B; final inverse transform | Source β `stabilized` | CPU/FFmpeg/NumPy | | |
| | Standard generation | `generate_standard` | video, Reference A, start, duration, segment_seconds, seed, audio_mode, progress | Standard final video, status | Final display/download only | Source β normalized generated output | CPU prep + one `xlarge` job for all windows + CPU assembly | | |
| | Generate Pass A | `generate_pass_a` | video, meta_path, stable_video, stable_ref, transforms_path, ref_time, pass_a_backward, segment_seconds, seed, feather, audio_mode, progress | Recomposed final; **generated stabilized video**; status | Final preview; optional Recovery uses generated stabilized output plus camera/preparation checkpoint | `stabilized` generation; `normalized_source` final | CPU prep + one `xlarge` job encompassing every internal window and A branch + CPU recomposition | | |
| | Extract Reference B candidate | `extract_b_candidate` | **stable input video**, meta_path, reacq_time | Exact stabilized frame, status | External repaint, then Reference B upload | `stabilized` | CPU/FFmpeg | | |
| | Reacquire identity | `reacquire_bidirectional` | source video, meta_path, **stable input**, transforms_path, **Pass A stabilized generated crop**, Reference B, reacq_time, backward_compute, transition_center, transition_duration, segment_seconds, seed, feather, audio_mode, progress | Recovered final, status | Final display/download | B generation and A/B fusion in stabilized space; then one inverse transform | CPU prep + a **separate** `xlarge` job encompassing B backward/forward windows + CPU fusion/inverse/audio | | |
| `add_click` is retained byte-for-byte in the extracted core but is not bound to a Workflow `.select()` event. The standalone browser performs the equivalent pixel scaling and saves the structured list. It never asks SAM to composite a final matte. | |
| ## Computation inside a pass | |
| `generate_pass_a` and `reacquire_bidirectional` each call `_render_branches_one_gpu`, which builds the complete branch/window lists and calls `_render_gpu_with_ui_progress` once. That invokes the original `backend.render_all_segments`, decorated `spaces.GPU(size="xlarge", duration=batch_gpu_budget)`. Technical windows default to five seconds; the ten-second V22 option remains. They are not graph nodes and are not separate allocations. | |
| Pass A forward-only is the default. Its optional backward and forward branches still share the single A allocation. Pass B's backward computation and short visible transition remain separate controls. B keeps V22's internal seed offset of +7001. A/B are merged using `dynamic_zoom.merge_reacquisition`; only the merged stabilized result is passed through `dynamic_zoom.recompose` once for B's final. A has its own independently useful recomposed final, which is never fed to B. | |
| Startup imports `spaces` before model-related imports. The canonical backend creates its CPU-side global pipeline before ZeroGPU fork, retains its fixed prompt/timestep/scheduler compatibility code, including the `pinned_loop(..., i, t)` signature, and moves/uses the pipeline inside its unchanged decorated rendering function. Generic documentation examples do not override this application-specific initialization strategy. | |
| ## Recovered graph | |
| The schema has 8 operators, 3 media references, 9 subjects and 20 edges. Every operator's immediate successor is a subject checkpoint or final, not another operator. This is purposeful: native single-node Run includes one-hop downstream nodes. | |
| | Path | Actual connected nodes | | |
| |---|---| | |
| | Source/spatial | Source β Prepare Source β Point Picker checkpoint β Track Subject β Tracking preview checkpoint β SAM/bypass β Guard checkpoint β Virtual Camera β Stabilized input checkpoint | | |
| | Generation | Reference A β Virtual Camera; stabilized checkpoint β Pass A β final A + saved A checkpoint | | |
| | Recovery | Saved A β Prepare Recovery β candidate/checkpoint β Pass B β Reference B β recovered final | | |
| | Standard | Shared Source + Reference A β Standard β Standard final; no spatial/generation ancestor | | |
| The two browser interactions are explicit side transactions tied to immutable IDs: Point Picker writes points for a prepared ID; Recovery Browser writes a candidate selection for a completed A ID. Track/Prepare Recovery consume those selections on their next explicit Run. The browser pages neither synthesize a graph edge nor inject events into Gradio. | |
| ## Artifacts and ownership | |
| | Kind | Stored/reused contents | Contract / next consumer | | |
| |---|---|---| | |
| | prepared | Canonical metadata, normalized clip, anchors, absolute reference time, source digest | `normalized_source`; picker/TAP | | |
| | tracked | Preparation ID, point revision, anchor groups, tracks, preview | `normalized_source`; SAM/bypass | | |
| | guarded | Tracking/preparation IDs, optional masks, point revision | `normalized_source`; camera | | |
| | camera | Preparation/guard IDs, stabilized input, transformed Reference A, transforms, point revision | `stabilized`; Pass A | | |
| | pass_a | Camera/preparation IDs, final video and separate stabilized generated video, generation settings, take | `stabilized_generated` primary checkpoint; optional Recovery | | |
| | recovery | Saved A ID, camera/preparation IDs, candidate frame, exact absolute time and index | `stabilized`; Pass B | | |
| | pass_b / standard | Final path, settings/status provenance | `normalized_source` final | | |
| Checkpoint cards render native Markdown previews and an Inspect link. Only the opaque capability ID is consumed by downstream adapters. Server-owned JSON stores the typed manifest and lineage; browser-provided paths, metadata or coordinate labels are not accepted as authoritative checkpoints. Media inputs/outputs use native file descriptors, resolved only within upload/output directories. This avoids relying on Workflow preserving arbitrary fields in file dictionaries. | |
| User inputs are the three media references and saved point/frame selections. Computed artifacts are immutable checkpoints. Configurations are scalar native ports. Optional overrides include manual anchor times, SAM enable, camera tuning, A backward mode, and B settings. B inherits the completed A's window/seed/feather/audio values by default and exposes native overrides without editing A's saved inputs. A `-1` transition center selects a UI convenience default up to two seconds before B; an explicit center is forwarded unchanged. This is a documented UI default, not a new fusion algorithm. | |
| ## Why the experimental layout was not retained | |
| The prior 46-card graph exposed prepared data and generation intermediates as repeated standalone inputs, used raw JSON control bundles, and separated many technical outputs from their consumers. Edges could be schema-valid while Recovery still required users to manually provide its source/camera/A files. The replacement carries these as one typed lineage and reuses real edges. Native output subjects remain where needed for execution stops and useful previews. Reference A remains shared with Standard; this leaves a long fan-out connection. Rendered wire crossings, spacing and mobile navigation are still an explicit acceptance gate, not a claimed visual success. | |