scripts: add README.md

971526d verified 2 months ago

4.02 kB

# Scripts

The tooling that produced this LoRA and its examples, so the results can be
reproduced and checked rather than taken on trust.

## Inference

**`ltx_a2a_generate_multi.py`** — audio-to-audio generation from a TSV manifest.
The LTX trainer ships no standalone a2a inference CLI; `validation_runner.py` is
that code path, so this drives `scripts/train.py` with `optimization.steps: 1`
and `checkpoints.no_resume: true` and collects what validation writes.
Its reason for existing: the stock helper pins one `--reference` to every
prompt, so an N-source sweep costs N model loads per checkpoint. Validation
samples each carry their own `conditions[].audio`, so one load can cover every
(source, prompt) pair. A 3-source × 19-cell sweep drops from 3 model loads to 1,
and loading dominates runtime.
```
ltx_a2a_generate_multi.py --checkpoint <lora.safetensors> \
  --base-config <inference.yaml> --manifest sweep.tsv --output-dir out/
```
Manifest is `label<TAB>reference_wav<TAB>prompt`. The trigger word is prepended
by the tool, matching how training captions were built — never put it in the
manifest.
**`build_sweep_manifest.py`** — builds that manifest from the trained caption
vocabulary. Captions are assembled from the exact strings in the training
manifest with the source type substituted, so prompts stay in distribution.
Cell sets: `sweetspot`, `compare`, `all` (12 room cells), `placesfx` (outdoor +
SFX), `full` (all 19).
**`build_heldout_refs.py`** — rebuilds the 15.08 s held-out references from the
dataset's dry sources. Every source is exactly 6.000 s and the render length is
377 frames at 25 fps, so the reference is the dry clip repeated with a gap and
trimmed. No dependencies beyond numpy.
## Verification
**`measure_timing.py`** — envelope cross-correlation between a dry reference and
its render, in 2 ms RMS windows over ±250 ms. This reproduces the timing table
in the model card, so those numbers can be checked rather than believed.

Read the correlation column carefully: it measures how much the envelope
*changed*, and reverb changes it by design — most on transient-dense material
where the tail fills the gaps between hits. Low r on percussion is the effect
working. It is a timing measure, not a quality measure.

## Training / preprocessing

**`box_b_trainer_setup.sh`** — dataset preprocess plus the **mandatory flatten
step**.
`process_dataset.py` mirrors the dataset-JSON relative paths, writing
`audio_latents/audio/targets/X.pt` and
`reference_audio_latents/audio/REFERENCES/X.pt`. But `datasets.py::_discover_samples`
takes the primary source's `rel_path` and requires that identical path in every
other source, so targets and references never match. The run then reports
**"No valid samples found"** with per-source counts of zero while all three
directories are visibly full. Flattening all three to a single level fixes it.
This is silent and will hit anyone training a multi-source a2a IC-LoRA.

Also note `keep_last_n` is a rolling window — set it to `-1` to keep every
checkpoint, or early ones are deleted without warning.

Inference needs preprocessed latents too, because `train.py` builds a
dataloader before validation fires. It does not need *all* of them: a handful of
pairs satisfies it in under a minute.

## Video

**`build_flf25.py`** — first-and-last-frame LTX-2.5 graphs in ComfyUI API
format, both frames via chained `LTXVAddGuide` at `frame_idx` 0 and −1. Two
presets, `distilled` and `dev`; see `WORKING_CONFIG.md`. Every node class and
input name is validated against a live `object_info` dump before the graph is
written.
**`WORKING_CONFIG.md`** — the verified-good settings, and the trap that the two
transformers need opposite sampling regimes.

Note the a2a LoRA does **not** belong inside a video graph: it is
reference-conditioned, so with no reference it is off-distribution and yields
near-silence. Render video, take its audio, run the a2a pass, remux. That also
gives the base-model baseline for free.

	# Scripts

	The tooling that produced this LoRA and its examples, so the results can be
	reproduced and checked rather than taken on trust.

	## Inference

	`ltx_a2a_generate_multi.py` — audio-to-audio generation from a TSV manifest.

	The LTX trainer ships no standalone a2a inference CLI; `validation_runner.py` is
	that code path, so this drives `scripts/train.py` with `optimization.steps: 1`
	and `checkpoints.no_resume: true` and collects what validation writes.

	Its reason for existing: the stock helper pins one `--reference` to every
	prompt, so an N-source sweep costs N model loads per checkpoint. Validation
	samples each carry their own `conditions[].audio`, so one load can cover every
	(source, prompt) pair. A 3-source × 19-cell sweep drops from 3 model loads to 1,
	and loading dominates runtime.

	```
	ltx_a2a_generate_multi.py --checkpoint <lora.safetensors> \
	--base-config <inference.yaml> --manifest sweep.tsv --output-dir out/
	```

	Manifest is `label<TAB>reference_wav<TAB>prompt`. The trigger word is prepended
	by the tool, matching how training captions were built — never put it in the
	manifest.

	`build_sweep_manifest.py` — builds that manifest from the trained caption
	vocabulary. Captions are assembled from the exact strings in the training
	manifest with the source type substituted, so prompts stay in distribution.
	Cell sets: `sweetspot`, `compare`, `all` (12 room cells), `placesfx` (outdoor +
	SFX), `full` (all 19).

	`build_heldout_refs.py` — rebuilds the 15.08 s held-out references from the
	dataset's dry sources. Every source is exactly 6.000 s and the render length is
	377 frames at 25 fps, so the reference is the dry clip repeated with a gap and
	trimmed. No dependencies beyond numpy.

	## Verification

	`measure_timing.py` — envelope cross-correlation between a dry reference and
	its render, in 2 ms RMS windows over ±250 ms. This reproduces the timing table
	in the model card, so those numbers can be checked rather than believed.

	Read the correlation column carefully: it measures how much the envelope
	changed, and reverb changes it by design — most on transient-dense material
	where the tail fills the gaps between hits. Low r on percussion is the effect
	working. It is a timing measure, not a quality measure.

	## Training / preprocessing

	`box_b_trainer_setup.sh` — dataset preprocess plus the **mandatory flatten
	step**.

	`process_dataset.py` mirrors the dataset-JSON relative paths, writing
	`audio_latents/audio/targets/X.pt` and
	`reference_audio_latents/audio/REFERENCES/X.pt`. But `datasets.py::_discover_samples`
	takes the primary source's `rel_path` and requires that identical path in every
	other source, so targets and references never match. The run then reports
	"No valid samples found" with per-source counts of zero while all three
	directories are visibly full. Flattening all three to a single level fixes it.
	This is silent and will hit anyone training a multi-source a2a IC-LoRA.

	Also note `keep_last_n` is a rolling window — set it to `-1` to keep every
	checkpoint, or early ones are deleted without warning.

	Inference needs preprocessed latents too, because `train.py` builds a
	dataloader before validation fires. It does not need all of them: a handful of
	pairs satisfies it in under a minute.

	## Video

	`build_flf25.py` — first-and-last-frame LTX-2.5 graphs in ComfyUI API
	format, both frames via chained `LTXVAddGuide` at `frame_idx` 0 and −1. Two
	presets, `distilled` and `dev`; see `WORKING_CONFIG.md`. Every node class and
	input name is validated against a live `object_info` dump before the graph is
	written.

	`WORKING_CONFIG.md` — the verified-good settings, and the trap that the two
	transformers need opposite sampling regimes.

	Note the a2a LoRA does not belong inside a video graph: it is
	reference-conditioned, so with no reference it is off-distribution and yields
	near-silence. Render video, take its audio, run the a2a pass, remux. That also
	gives the base-model baseline for free.