SepGen Generation checkpoint

README.md

@@ -0,0 +1,75 @@
+---
+license: other
+license_name: ltx-2.x-community-license-agreement
+license_link: https://github.com/Lightricks/LTX-2/blob/main/LICENSE-2_x
+base_model: Lightricks/LTX-2.5
+pipeline_tag: text-to-video
+tags:
+- audio-video
+- source-separation
+- audio-generation
+- lora
+- ltx-2.5
+- sepgen
+---
+
+# SepGen generation checkpoint (`gen-3k`)
+
+**SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model**
+
+**Aviad Dahan**<sup>1\*</sup>, **Rajaei Khatib**<sup>1\*</sup>, **Yonatan Bitton**<sup>2</sup>,
+**Idan Szpektor**<sup>2</sup>, **Lior Wolf**<sup>1</sup>, **Raja Giryes**<sup>1</sup>
+
+<sup>1</sup>Tel Aviv University, <sup>2</sup>Google  ·  <sup>\*</sup>Equal contribution
+
+[Paper](ARXIV_URL) · [Project page](https://sepgen.github.io/) · [Code](https://github.com/AviadDahan/SepGen) · Other checkpoint: [sep-12k (separation)](https://huggingface.co/AviadDahan/SepGen-Separation)
+
+SepGen extends LTX-2.5 to emit the video, the audio-mix, and one waveform per captioned source in a
+single sampling run. The same weights generate (scene caption and per-source captions → video,
+audio-mix and stems) and separate (observed video and audio-mix → stems), selected by the noise
+level of the audio-mix.
+
+`gen-3k` is the generation checkpoint: `sep-12k` continued for 3,000 steps in which 70% of samples present the audio-mix at a noise level no higher than the stems'. From a scene caption and one caption per source it generates the video, the audio-mix and one waveform per source; it also separates, at parity with `sep-12k`.
+
+## Files
+
+| File | Content |
+|---|---|
+| `lora_weights.safetensors` | the LoRA adapter (sha256 `a72b3c662ce8d6b6e29013bbedb797d48ee1c90e8b7d219af4a27329eb58515b`) |
+| `config.json` | adapter and training summary |
+| `train_config.yaml` | the full training config |
+
+The adapter targets the audio stream of the LTX-2.5 22B dev transformer: rank and alpha 128, 480
+modules, 327M parameters, bf16, ComfyUI key format (`diffusion_model.`). It is trained with
+attention gates (block-diagonal caption routing, the protected audio-mix span, video reading audio
+from the audio-mix only, Cross-Stem Attention Guidance) that must also be installed at inference,
+so run it with the [SepGen code](https://github.com/AviadDahan/SepGen), not as a plain LoRA.
+
+## Usage
+
+```bash
+git clone https://github.com/AviadDahan/SepGen && cd SepGen
+bash setup.sh && source activate.sh && bash download_weights.sh
+python generate.py examples/generation_prompts.json --out-dir outputs/gen      # uses gen-3k
+python separate.py --batch-manifest examples/separation_manifest.json --checkpoint gen-3k --out-dir outputs/sep
+```
+
+`--checkpoint gen-3k` downloads this repo on first use.
+
+## License
+
+These weights are a Derivative of LTX-2.x and are distributed under the
+[LTX-2.x Community License Agreement](https://github.com/Lightricks/LTX-2/blob/main/LICENSE-2_x),
+including its use-based restrictions (Attachment A) and the
+[LTX Acceptable Use Policy](https://static.lightricks.com/legal/ltx-acceptable-use-policy.pdf).
+
+## Citation
+
+```bibtex
+@article{dahan2026sepgen,
+  title   = {SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model},
+  author  = {Dahan, Aviad and Khatib, Rajaei and Bitton, Yonatan and Szpektor, Idan and Wolf, Lior and Giryes, Raja},
+  journal = {arXiv preprint ARXIV_ID},
+  year    = {2026}
+}
+```

config.json

@@ -0,0 +1,50 @@
+{
+  "model_type": "sepgen",
+  "adapter_type": "lora",
+  "base_model": "Lightricks/LTX-2.5",
+  "base_model_file": "diffusion_models/ltx-2.5-22b-dev-transformer-bf16.safetensors",
+  "base_model_quantization": "int8-quanto (separation, as in training); bf16 for generation",
+  "weights_file": "lora_weights.safetensors",
+  "weights_format": "safetensors, bf16, ComfyUI keys (diffusion_model.*)",
+  "lora_rank": 128,
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "lora_modules": 480,
+  "trainable_parameters": 327155712,
+  "num_stems": 2,
+  "audio_span_layout": [
+    "stem0",
+    "stem1",
+    "audio_mix"
+  ],
+  "code": "https://github.com/AviadDahan/SepGen",
+  "paper_title": "SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model",
+  "license": "LTX-2.x Community License Agreement",
+  "checkpoint": "gen-3k",
+  "task": [
+    "generation",
+    "separation"
+  ],
+  "lora_target_modules": [
+    "audio_attn1.to_q",
+    "audio_attn1.to_k",
+    "audio_attn1.to_v",
+    "audio_attn1.to_out.0",
+    "audio_attn2.to_q",
+    "audio_attn2.to_out.0",
+    "video_to_audio_attn.to_q",
+    "video_to_audio_attn.to_out.0",
+    "audio_ff.net.0.proj",
+    "audio_ff.net.2"
+  ],
+  "training": {
+    "steps": 3000,
+    "init": "sep-12k",
+    "audio_mix_clean_fraction": 0.3,
+    "mix_sigma_mode": "independent_leq",
+    "learning_rate": 0.0001,
+    "scheduler": "linear to 0.1x over 12000 steps, stopped at step 3000",
+    "batch_size": 1
+  },
+  "sha256": "a72b3c662ce8d6b6e29013bbedb797d48ee1c90e8b7d219af4a27329eb58515b"
+}

lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a72b3c662ce8d6b6e29013bbedb797d48ee1c90e8b7d219af4a27329eb58515b
+size 654452792

train_config.yaml

@@ -0,0 +1,128 @@
+# SepGen generation checkpoint (gen-3k): warm start from sep-12k, then 3,000 steps in which 30% of
+# samples present the audio-mix clean and 70% present it at a noise level drawn independently and
+# clamped to at most the stems' level (independent_leq). Fresh optimizer and step counter
+# (no_resume); the LR schedule spans optimization.steps (12,000) and training stops at step 3000.
+#   python train.py configs/train_gen3k.yaml --init-checkpoint sep-12k --stop-at-step 3000
+acceleration:
+  load_text_encoder_in_8bit: false
+  mixed_precision_mode: bf16
+  offload_optimizer_during_validation: true
+  quantization: int8-quanto
+checkpoints:
+  interval: 250
+  keep_last_n: -1
+  no_resume: true
+  precision: bfloat16
+data:
+  num_dataloader_workers: 2
+  preprocessed_data_root: data/precomputed_train
+flow_matching:
+  timestep_sampling_mode: shifted_logit_normal
+  timestep_sampling_params: {}
+hub:
+  hub_model_id: null
+  push_to_hub: false
+joint_stem:
+  a2v_mix_only: true
+  first_frame_conditioning_p: 0.0
+  frozen_mix_lora: true
+  include_mix: true
+  mix_lead_alpha: 1.0
+  mix_loss_weight: 0.0
+  mix_sigma_mode: independent_leq
+  mix_span_energy_weight: 0.0
+  num_stems: 2
+  partition_loss_weight: 0.0
+  per_source_energy_weight: 0.0
+  positional_conditions_suffix: ''
+  positional_prompt_p: 0.0
+  ref_time_offset: 0.0
+  register_avail_dir: register_avail
+  register_mix:
+    ? ''
+    : 1
+    _pos: 1
+    _short: 1
+  rollout_training: false
+  sep_pattern_p: 0.3
+  span_attention_sibling: true
+  span_attention_topology: mix_protected
+  stem_audio_conditions_dir_prefix: conditions_stem_audio
+  video_loss_weight: 0.0
+  video_sigma_zero: false
+  positional_spatial_mask: true
+  positional_spatial_mask_eps: 1.0e-06
+joint_stem_validation:
+  dev_guidance:
+    audio_cfg_scale: 7.0
+    modality_scale: 1.0
+    rescale_scale: 0.7
+    stg_scale: 0.0
+    video_cfg_scale: 3.0
+lora:
+  alpha: 128
+  dropout: 0.0
+  rank: 128
+  target_modules:
+  - audio_attn1.to_q
+  - audio_attn1.to_k
+  - audio_attn1.to_v
+  - audio_attn1.to_out.0
+  - audio_attn2.to_q
+  - audio_attn2.to_out.0
+  - video_to_audio_attn.to_q
+  - video_to_audio_attn.to_out.0
+  - audio_ff.net.0.proj
+  - audio_ff.net.2
+model:
+  audio_vae_path: ${SEPGEN_MODELS}/vae/ltx-2.5-audio-vae-bf16.safetensors
+  load_checkpoint: null
+  model_path: ${SEPGEN_MODELS}/diffusion_models/ltx-2.5-22b-dev-transformer-bf16.safetensors
+  text_encoder_path: ${SEPGEN_MODELS}/text_encoders/gemma4-12b-with-proj-ltx-2.5-bf16.safetensors
+  training_mode: lora
+  video_vae_path: ${SEPGEN_MODELS}/vae/ltx-2.5-video-vae-bf16.safetensors
+optimization:
+  batch_size: 1
+  enable_gradient_checkpointing: true
+  gradient_accumulation_steps: 1
+  learning_rate: 0.0001
+  max_grad_norm: 1.0
+  optimizer_type: adamw
+  scheduler_params: {}
+  scheduler_type: linear
+  steps: 12000
+output_dir: outputs/gen-3k
+seed: 42
+training_strategy:
+  audio_latents_dir: audio_latents_mix
+  first_frame_conditioning_p: 0.0
+  name: text_to_video
+  with_audio: true
+validation:
+  audio_cfg_scale: 7.0
+  audio_modality_guidance_scale: 1.0
+  audio_stg_scale: 0.0
+  frame_rate: 25.0
+  generate_audio: true
+  guidance_rescale: 0.7
+  inference_steps: 30
+  interval: null
+  negative_prompt: worst quality, inconsistent motion, blurry, jittery, distorted
+  seed: 42
+  skip_initial_validation: false
+  stg_blocks:
+  - 28
+  video_cfg_scale: 3.0
+  video_dims:
+  - 768
+  - 512
+  - 113
+  video_modality_guidance_scale: 1.0
+  video_stg_scale: 0.0
+wandb:
+  enabled: false
+  entity: null
+  project: sepgen
+  log_validation_videos: false
+  tags:
+  - gen-3k