scripts/inference/README.md

Inference

The headline model is the 30s long-video distilled one: 3 sampling steps, no CFG, rolls out to minute scale. All four commands below run on data bundled in examples/ — no external dataset.

# t2v -- prompt only, no camera control
MODE=t2v NUM_CHUNKS=20 bash scripts/inference/infer_post_distill.sh

# i2v -- first frame + camera trajectory
MODE=i2v NUM_CHUNKS=20 bash scripts/inference/infer_post_distill.sh

# v2v -- reference video + its camera trajectory
MODE=v2v NUM_CHUNKS=20 bash scripts/inference/infer_post_distill.sh

# segment -- i2v plus a per-chunk prompt schedule; the prompt switches mid-rollout
MODE=segment NUM_CHUNKS=6 MAX_CASES=0 bash scripts/inference/infer_post_distill.sh

Each mode already points at its own bundled jsonl, so JSONL= is only needed for your own data.

Result lands in <OUT_ROOT>/<case>/geo_pred.mp4; OUT_ROOT defaults to output_evoke/infer/post_distill_<mode> here and output_evoke/infer/stage1_<mode> for stage 1.

If the launcher stops at [preflight] FATAL: ... cannot import torch + cv2, point it at the right interpreter: EVOKE_PYTHON_BIN=<your-env>/bin bash scripts/inference/....

Each launcher opens with the checkpoint it resolved and whether that came from TRANSFORMER_PATH or from its own default — check that line first, an unexported override silently evaluates the default model. The settings behind a result are also recorded in <OUT_ROOT>/run_info.json.

Watching a run

A single-GPU run (LOCAL_GPUS=1, the default) streams to the terminal and to logs/. The rollout reports two levels: a bar over chunks, and one line per chunk splitting it into its phases.

  chunk   3/20   warp   1.17s (pool=  72 cov=0.877)  |  diffusion   2.05s (3 steps)  |  decode+dump   5.78s  |  total   9.01s
chunks:  15%|█▌        | 3/20 [00:27<02:33,  9.19s/chunk]

warp is the point-cloud render (ViGeo depth by default) plus its encode to latents, diffusion the denoising steps themselves, and decode+dump the VAE decode and the per-chunk segment videos — on the bundled v2v case that last part costs more than the generation it follows. cov is warp coverage and pool the number of source frames fused, the two numbers worth watching over a long rollout.

QUIET=1 sends the run to the log only. Multi-shard runs always redirect and draw no bar: nothing reaches a terminal, so the engine's plain per-chunk lines are the better record.

The log file gets the same lines with the bar's redraws stripped, so it stays greppable — progress bars are terminal UI and a file full of \r frames is not readable. EVOKE_INFER_DEBUG=1 restores everything the bar replaces: the per-step DiT timings, the per-chunk sampling bar, the raw [GEO-da3] / segment-dump lines (the tag is historical; the backend is whatever DEPTH_BACKEND selects), and the vendored depth logger.

Length: use NUM_CHUNKS

One chunk = 36 pixel frames = 1.5s @24fps. NUM_CHUNKS=20 gives a 30s clip.

NUM_FRAMES is the older knob and it is not an output length — it only picks the chunk count, and nothing trims the result back down:

chunks = ceil(NUM_FRAMES / 33)        # 33 = (latent_window_size-1) * vae_temporal_stride + 1
frames = 36 * chunks - 3              # i2v / v2v: the final length is quantised down to 4k+1
frames = 36 * chunks - 4              # t2v: same, minus the dropped frame 0 (see below)

So NUM_FRAMES=721 yields 22 chunks = 789 frames = 32.9s, not 30s. NUM_CHUNKS sets NUM_FRAMES = 33 * chunks, which is the only way to get an exact length. (The formula above is for the shipped VAE_DECODE_TYPE=persistent; plain default decode yields 33 * chunks.)

The -3 is not a per-chunk trim: every chunk decodes 36 pixel frames (chunk 0 decodes 33 under t2v, where the cache is not warmed), and the accumulated clip is then cut back to the nearest 4k+1. segments/segment_NNN_pred.mp4 holds each chunk as decoded, so the segments can total more frames than geo_pred.mp4 — stitching them is not the same as the final video.

t2v drops frame 0 from the final geo_pred.mp4. With no real prior frame to warm the causal VAE cache, chunk 0's first pixel is the I-frame slot and carries a colour cast that the rest of the clip does not; the frame is still present in segments/segment_000_pred.mp4.

Modes

MODE	Input	Camera control
v2v	reference video + pose track	yes
i2v	first frame + pose track	yes
t2v	prompt only	no — warp carries the trajectory, and the engine forbids warp+t2v

For v2v, generation continues after the reference window: REF_VIDEO_SEC (default 5s) is how much of the clip conditions the model, so the default run conditions on [0s, 5s] and generates from there. It is forced to 0 for i2v/t2v.

START_SECONDS (default 0) offsets where that window is taken from. It also shifts the pose track, so unlike REF_VIDEO_SEC it applies to i2v as well and is never forced to 0.

i2v camera scale

Two i2v-only knobs, both about how far the camera moves. v2v/t2v are unaffected: v2v seeds its frame bank from the reference video and solves its own depth scale from real camera motion, t2v has no warp. Calibration tables live next to the values in the code.

env	default	what it is
GEO_CHUNK0_TARGET_DISP_PX	135 px	chunk 0's depth scale comes from a single frame, and pinning its median depth does not pin the near field — so its amplitude used to swing with the reference photo, running ~6× the flow of its own later chunks. This rescales chunk-0 depth so that at its largest commanded step, 90 % of pixels stay inside an N-pixel budget. 0 restores the old behaviour.
VIGEO_DEPTH_MEDIAN_TARGET	5	how deep the world is, in pose units: after scaling, the scene's median depth is this number. At 1.0 (the bare unit definition, and the default until it was calibrated) a 5 s sekai track commands ~1.8 scene-depths per chunk, so the camera leaves the geometry, the warp collapses to holes and the model re-invents the scene — which reads as a sudden acceleration even though the commanded motion is flat or falling. v2v solves ≈9.9 for the same quantity.

Both defaults are calibrated, so i2v needs no flags. 5 vs 10 for the world scale is a look call: 10 is the value v2v measures and is safer on worst-case warp coverage, 5 keeps twice the parallax. 1.0 reproduces pre-calibration runs.

Diagnosing it: read per-chunk cov= in _logs/<case>.log — below ~0.35 the model is filling holes from the prompt. Optical flow will not show it, because a mid-clip reset leaves the tail/body flow ratio near 1.0. Resolved values are logged as chunk0_target_disparity_px = ... and chunk0 disparity rescale: k=.

Models

Launcher	Weights	Steps
infer_stage1.sh	models/evoke/stage1_camera_control	50 (CFG 5.0)
infer_post_distill.sh	models/evoke/stage3_post_distillation	3

Any other checkpoint runs on the same launcher and recipe — pass TRANSFORMER_PATH=models/evoke/stage3_long_distillation (or any other parent of a transformer/ directory) and give it its own OUT_ROOT.

TRANSFORMER_PATH is always the parent of a transformer/ directory — the weights load as from_pretrained(TRANSFORMER_PATH, subfolder="transformer").

Sampling from the teacher

infer_evoke_teacher.py runs the dual-expert teacher in models/evoke/evoke_teacher directly. The teacher is normally only a frozen scorer inside stage-3 DMD, but that scoring forward is a v-prediction, so wrapping it in a flow-match Euler loop is enough to sample from it.

infer_evoke_teacher.sh is the worked i2v example — it runs on the bundled examples/i2v/ data with no arguments, ~5s @24fps:

bash scripts/inference/infer_evoke_teacher.sh                    # i2v, 121 frames, 50 steps
CLIP_SECONDS=10 STEPS=30 bash scripts/inference/infer_evoke_teacher.sh
IMAGE= PROMPT="a drone shot over a snowy mountain village" \
  bash scripts/inference/infer_evoke_teacher.sh                  # t2v instead

CLIP_SECONDS is rounded up to the nearest valid length: the VAE is temporal stride 4, so pixel frames must be 4k+1 (5s @24fps → 121). The engine underneath takes the same knobs as flags:

python scripts/inference/infer_evoke_teacher.py \
  --prompt "a drone shot flying over a snowy mountain village at sunrise" \
  --image_path examples/i2v/image.jpg \
  --num_frames 121 --num_inference_steps 50 --offload \
  --output output/evoke_teacher/i2v.mp4

Result lands in output/evoke_teacher/i2v.mp4; the engine log is logs/infer_evoke_teacher/run.log.

It is an example, not a supported launcher, and differs from the two above in three ways worth knowing before you read anything into its output:

• nocam, non-SP, single process — EvokeTeacherScoreWrapper._forward_core only ports that one path of the teacher's forward, so there is no camera conditioning and no sequence-parallel inference;
• not validated against the teacher's own sampler — the schedule, shift, v-prediction sign and expert boundary come from the training path, where they were checked for a single scoring step. A sampling loop compounds any mismatch, so A/B one prompt+seed against the upstream sampler before trusting it;
• 2 x 14B is ~56 GB of bf16 weights — pass --offload to keep only the routed expert resident, or --single_expert high|low to check the plumbing on one expert (which is wrong for half the schedule).

Both distilled models were trained on v2v conditioning only, so MODE=i2v|t2v on them is zero-shot; the launchers say so at startup. All three modes are in distribution for stage1.

3 steps and CFG-off are properties of the distilled weights, not knobs: the step count comes from STAGE2_STEPS, and raising NUM_INFERENCE_STEPS does nothing (the launcher warns).

stage1 is the opposite case — it is not distilled, so it is sampled like an ordinary diffusion model: 50 steps with CFG 5.0. Do not copy validation_config.num_inference_steps (=8) out of stage1_camera_control.yaml; that is the cheap sanity check run during training, and 8 steps with CFG off visibly smears the result.

Hour-scale rollouts

Without streaming, the pipeline accumulates the whole clip as one fp32 GPU tensor (~2.95 MB/frame at 384x640), so a long run OOMs deep into the job. The driver therefore refuses more than 7200 frames unless you opt in:

• STREAM_LONG=1 — decode per chunk, never accumulate, stitch the final geo_pred.mp4 from segments/. Needs VAE_DECODE_TYPE=persistent (the launcher default). The generated pixels are identical, but the side outputs differ: the 4-panel video becomes a concatenation of the per-chunk panels rather than the whole-video comparison, and sample_frames/ holds segment head frames instead of 5 evenly spaced ones.
• GEO_HIST_MAX_FRAMES=<N> — slide the point cloud so it keeps only the last N pixel frames. Required beyond 10min (the cloud otherwise grows by 12 dense depth frames per chunk), but note this changes what the warp sees, i.e. it is a recipe change and not just a memory knob. Must be much larger than WARP_LAG * 36; 720 (30s) is a typical value.

Bring your own data

Copy the layout of examples/ — it has one working case per mode, and every launcher points at it by default:

examples/t2v/cases.jsonl              {"name", "prompt"}
examples/i2v/cases.jsonl              + image_path, pose_path, prompt_path
examples/v2v/cases.jsonl              + video_path, pose_path, prompt_path, video_fps
examples/segment_prompts/cases.jsonl  + schedule_*.json (prompt switches mid-rollout)

Point JSONL at your own file with the same fields. Paths inside are resolved relative to the repo root. Poses are vipe .npz (cam_c2w [T,4,4] + intrinsics); declare pose_fps and pose_source_resolution per row when they differ from the i2v defaults (24 / [480, 832]) — a 30 fps track declared as 24 renders 25 % long and 25 % slow, and nothing detects it.

Output

<OUT_ROOT>/<case>/geo_pred.mp4                  generated video (always; also the resume marker)
<OUT_ROOT>/<case>/geo_pred_hud.mp4              same video with the joystick overlay (JOYSTICK_HUD=both)
<OUT_ROOT>/<case>/segments/segment_NNN_pred.mp4 per-chunk segments (SAVE_SEGMENTS=0 to skip)
<OUT_ROOT>/<case>/gt_vs_pred_cam_viz.mp4        4-panel gt | warp | visibility | pred (v2v only)
<OUT_ROOT>/_logs/<case>.log                     engine log
<OUT_ROOT>/run_info.json                        checkpoint + recipe that produced this directory

Finished cases are skipped on re-run, so an interrupted sweep resumes. Knobs: LOCAL_GPUS · GPU_OFFSET · MAX_CASES (0 = all) · TRANSFORMER_PATH · OUT_ROOT · SEED · NUM_CHUNKS · REF_VIDEO_SEC / START_SECONDS · JOYSTICK_HUD · EVOKE_PYTHON_BIN — or run any launcher with -h. For sweeps of hundreds of cases, IN_PROCESS_BATCH=1 BG_POSTPROC=1 SHARDS_PER_GPU=1 is the tuned recipe (~3.4 s/case against ~31 s/case out of the box).

Launcher knobs mirror configs/training/** of the same model. A train/infer mismatch in the warp or attention recipe degrades quality silently instead of failing, so do not tune them ad hoc.