Update README.md

0b942a1 verified 9 days ago

6.57 kB

---
license: apache-2.0
library_name: pytorch
pipeline_tag: text-to-video
tags:
- video-generation
- diffusion
- sgf
- sgf-plus
---
<h1 style="display:flex;align-items:center;gap:8px;font-size:26px;line-height:1.3;margin:16px 0"><img src="assets/logo.png" width="32" height="32" style="display:block;flex-shrink:0;margin:0" alt="SGF+ logo"> Self Gradient Forcing Plus</h1>

**Decoupling Gradient Flows for Autoregressive Video Generation**

[Paper](https://arxiv.org/abs/2610.10429) · [Project Page](https://zihan-su.github.io/self-gradient-forcing-plus) · [GitHub](https://github.com/Zihan-Su/Self_Gradient_Forcing_Plus)

SGF+ separates context-writing and denoising parameters to resolve conflicting gradient updates, improving autoregressive video generation and enabling rollouts of up to 24 hours from only 5s training windows.

![SGF+ video generation comparisons and a 24-hour rollout](assets/teaser.png)

## Model Checkpoints

| Checkpoint | Description |
| :--- | :--- |
| [`chunkwise/model.pt`](chunkwise/model.pt) | SGF+ for chunkwise video generation. |
| [`framewise/model.pt`](framewise/model.pt) | SGF+ for framewise video generation. |
| [`diagnostics/sgf.pt`](diagnostics/sgf.pt) | SGF checkpoint used for gradient-conflict reproduction, containing the raw generator and paired critic. |

## Installation

The environment follows the SGF setup.

```bash
git clone https://github.com/Zihan-Su/Self_Gradient_Forcing_Plus.git
cd Self_Gradient_Forcing_Plus
conda create -n sgf_plus python=3.10 -y
conda activate sgf_plus
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
python setup.py develop
```

Run the following commands from the repository root.

## Download Weights

```bash
bash scripts/download_weights.sh
```

The script uses the Hugging Face CLI command `hf` by default. Set `HF_CLI=huggingface-cli` if your environment still uses the older command name.

It downloads:

- Wan base models to `wan_models/Wan2.1-T2V-1.3B` and `wan_models/Wan2.1-T2V-14B`.
- [Causal-Forcing](https://github.com/thu-ml/Causal-Forcing) AR initialization checkpoints to `checkpoints/init/chunkwise/ar_diffusion.pt` and `checkpoints/init/framewise/ar_diffusion.pt`.
- Released SGF+ inference checkpoints to `hf_weights/chunkwise/model.pt` and `hf_weights/framewise/model.pt`.
- The training prompt list to `prompts/vidprom_filtered_extended.txt`.

Run `hf auth login` first if authentication is required.

## Inference

The default prompt file is `prompts/test_prompt.txt` with 8 prompts. The launcher uses 8 GPUs when at least 8 GPUs are visible; otherwise it falls back to single-GPU serial inference. By default it generates `963` latent frames, which decode to about 240 seconds of video at 16 fps.

The inference script takes the release setting name (`chunkwise` or `framewise`) and a checkpoint path, and selects the matching config automatically:

- chunkwise config: `configs/sgf_plus_chunkwise.yaml`
- framewise config: `configs/sgf_plus_framewise.yaml`

The long-video KV-cache geometry is set in `scripts/infer_self_gradient_forcing.sh`, which is called by the SGF+ launcher. Chunkwise defaults to `KV_CACHE_SINK=3`, `KV_CACHE_FIFO_FRAMES=6`, and `KV_CACHE_CURRENT_FRAMES=3`, so `--kv_cache_max_frames` is `12`. Framewise defaults to `KV_CACHE_SINK=4`, `KV_CACHE_FIFO_FRAMES=16`, and `KV_CACHE_CURRENT_FRAMES=1`, so `--kv_cache_max_frames` is `21`.

### Chunkwise

```bash
bash scripts/infer_sgf_plus.sh chunkwise hf_weights/chunkwise/model.pt
```

This uses:

```text
configs/sgf_plus_chunkwise.yaml
hf_weights/chunkwise/model.pt
```

### Framewise

```bash
bash scripts/infer_sgf_plus.sh framewise hf_weights/framewise/model.pt
```

This uses:

```text
configs/sgf_plus_framewise.yaml
hf_weights/framewise/model.pt
```

### Custom checkpoint or prompt file

```bash
bash scripts/infer_sgf_plus.sh \
  chunkwise \
  hf_weights/chunkwise/model.pt \
  prompts/test_prompt.txt
```

Useful overrides:

```bash
NUM_OUTPUT_FRAMES=963 SEED=42 OUTPUT_ROOT=outputs/demo \
  bash scripts/infer_sgf_plus.sh chunkwise hf_weights/chunkwise/model.pt
```

For trained checkpoints, pass the release setting first and the produced `logs/.../checkpoint_model_*/model.pt` path as the second argument. The script uses EMA weights by default; set `USE_EMA=0` to use the non-EMA `generator` weights.

## Training

### Chunkwise SGF+

```bash
bash scripts/train_sgf_plus_chunkwise.sh
```

Equivalent explicit form:

```bash
bash scripts/train_sgf_plus_chunkwise.sh \
  configs/sgf_plus_chunkwise.yaml \
  logs/sgf_plus_chunkwise
```

### Framewise SGF+

```bash
bash scripts/train_sgf_plus_framewise.sh
```

Equivalent explicit form:

```bash
bash scripts/train_sgf_plus_framewise.sh \
  configs/sgf_plus_framewise.yaml \
  logs/sgf_plus_framewise
```

The launchers accept `[config.yaml] [logdir] [extra train.py args...]` and support single-node and multi-node training. Without explicit or scheduler-provided topology settings, nodes auto-register through `.rendezvous/` on the shared filesystem and launch static `torchrun` with an IP master address. For multi-node jobs, run the same command on every node within the gather window.

Useful overrides:

```bash
GATHER_WINDOW=90 NUM_GPUS=8 MASTER_PORT=29501 ENABLE_WANDB=1 \
  bash scripts/train_sgf_plus_chunkwise.sh logs/sgf_plus_chunkwise
```

To specify the topology manually, set `NNODES`, `NODE_RANK`, `MASTER_ADDR` and `MASTER_PORT`. Use a distinct `NODE_RANK` on each node and matching values for the other settings.

## Gradient Conflict Experiments

To reproduce our gradient-conflict experiments, see the [experiment guide](https://github.com/Zihan-Su/Self_Gradient_Forcing_Plus/blob/main/gradient_conflict/README.md).

## Citation

```bibtex
@article{zhuang2026self,
  title={Self Gradient Forcing: Native Long Video Extrapolation},
  author={Zhuang, Junhao and Zhang, Shiyi and Bian, Yuxuan and Li, Yaowei and Luo, Yawen and Liu, Yijun and Jin, Weiyang and Zhang, Songchun and He, Xianglong and Zhang, Xuying and others},
  journal={arXiv preprint arXiv:2607.20368},
  year={2026}
}
@article{su2026sgf+,
  title={SGF+: Decoupling Gradient Flows for Autoregressive Video Generation},
  author={Su, Zihan and Zhuang, Junhao and Li, Yaowei and Lu, Siwen and Li, Haoran and Li, Lingen and Wu, Haoyu and Jin, Weiyang and Zhang, Songchun and Huang, Haoyang and others},
  journal={arXiv preprint arXiv:2610.10429},
  year={2026}
}
```

## License

This project is released under the [Apache-2.0 license](https://huggingface.co/ZihanSu/Self_Gradient_Forcing_Plus/blob/main/LICENSE).

	---
	license: apache-2.0
	library_name: pytorch
	pipeline_tag: text-to-video
	tags:
	- video-generation
	- diffusion
	- sgf
	- sgf-plus
	---

	<h1 style="display:flex;align-items:center;gap:8px;font-size:26px;line-height:1.3;margin:16px 0"><img src="assets/logo.png" width="32" height="32" style="display:block;flex-shrink:0;margin:0" alt="SGF+ logo"> Self Gradient Forcing Plus</h1>

	Decoupling Gradient Flows for Autoregressive Video Generation

	[Paper](https://arxiv.org/abs/2610.10429) · [Project Page](https://zihan-su.github.io/self-gradient-forcing-plus) · [GitHub](https://github.com/Zihan-Su/Self_Gradient_Forcing_Plus)

	SGF+ separates context-writing and denoising parameters to resolve conflicting gradient updates, improving autoregressive video generation and enabling rollouts of up to 24 hours from only 5s training windows.

	![SGF+ video generation comparisons and a 24-hour rollout](assets/teaser.png)

	## Model Checkpoints

	\| Checkpoint \| Description \|
	\| :--- \| :--- \|
	\| [`chunkwise/model.pt`](chunkwise/model.pt) \| SGF+ for chunkwise video generation. \|
	\| [`framewise/model.pt`](framewise/model.pt) \| SGF+ for framewise video generation. \|
	\| [`diagnostics/sgf.pt`](diagnostics/sgf.pt) \| SGF checkpoint used for gradient-conflict reproduction, containing the raw generator and paired critic. \|

	## Installation

	The environment follows the SGF setup.

	```bash
	git clone https://github.com/Zihan-Su/Self_Gradient_Forcing_Plus.git
	cd Self_Gradient_Forcing_Plus

	conda create -n sgf_plus python=3.10 -y
	conda activate sgf_plus
	pip install -r requirements.txt
	pip install flash-attn --no-build-isolation
	python setup.py develop
	```

	Run the following commands from the repository root.

	## Download Weights

	```bash
	bash scripts/download_weights.sh
	```

	The script uses the Hugging Face CLI command `hf` by default. Set `HF_CLI=huggingface-cli` if your environment still uses the older command name.

	It downloads:

	- Wan base models to `wan_models/Wan2.1-T2V-1.3B` and `wan_models/Wan2.1-T2V-14B`.
	- [Causal-Forcing](https://github.com/thu-ml/Causal-Forcing) AR initialization checkpoints to `checkpoints/init/chunkwise/ar_diffusion.pt` and `checkpoints/init/framewise/ar_diffusion.pt`.
	- Released SGF+ inference checkpoints to `hf_weights/chunkwise/model.pt` and `hf_weights/framewise/model.pt`.
	- The training prompt list to `prompts/vidprom_filtered_extended.txt`.

	Run `hf auth login` first if authentication is required.

	## Inference

	The default prompt file is `prompts/test_prompt.txt` with 8 prompts. The launcher uses 8 GPUs when at least 8 GPUs are visible; otherwise it falls back to single-GPU serial inference. By default it generates `963` latent frames, which decode to about 240 seconds of video at 16 fps.

	The inference script takes the release setting name (`chunkwise` or `framewise`) and a checkpoint path, and selects the matching config automatically:

	- chunkwise config: `configs/sgf_plus_chunkwise.yaml`
	- framewise config: `configs/sgf_plus_framewise.yaml`

	The long-video KV-cache geometry is set in `scripts/infer_self_gradient_forcing.sh`, which is called by the SGF+ launcher. Chunkwise defaults to `KV_CACHE_SINK=3`, `KV_CACHE_FIFO_FRAMES=6`, and `KV_CACHE_CURRENT_FRAMES=3`, so `--kv_cache_max_frames` is `12`. Framewise defaults to `KV_CACHE_SINK=4`, `KV_CACHE_FIFO_FRAMES=16`, and `KV_CACHE_CURRENT_FRAMES=1`, so `--kv_cache_max_frames` is `21`.

	### Chunkwise

	```bash
	bash scripts/infer_sgf_plus.sh chunkwise hf_weights/chunkwise/model.pt
	```

	This uses:

	```text
	configs/sgf_plus_chunkwise.yaml
	hf_weights/chunkwise/model.pt
	```

	### Framewise

	```bash
	bash scripts/infer_sgf_plus.sh framewise hf_weights/framewise/model.pt
	```

	This uses:

	```text
	configs/sgf_plus_framewise.yaml
	hf_weights/framewise/model.pt
	```

	### Custom checkpoint or prompt file

	```bash
	bash scripts/infer_sgf_plus.sh \
	chunkwise \
	hf_weights/chunkwise/model.pt \
	prompts/test_prompt.txt
	```

	Useful overrides:

	```bash
	NUM_OUTPUT_FRAMES=963 SEED=42 OUTPUT_ROOT=outputs/demo \
	bash scripts/infer_sgf_plus.sh chunkwise hf_weights/chunkwise/model.pt
	```

	For trained checkpoints, pass the release setting first and the produced `logs/.../checkpoint_model_*/model.pt` path as the second argument. The script uses EMA weights by default; set `USE_EMA=0` to use the non-EMA `generator` weights.

	## Training

	### Chunkwise SGF+

	```bash
	bash scripts/train_sgf_plus_chunkwise.sh
	```

	Equivalent explicit form:

	```bash
	bash scripts/train_sgf_plus_chunkwise.sh \
	configs/sgf_plus_chunkwise.yaml \
	logs/sgf_plus_chunkwise
	```

	### Framewise SGF+

	```bash
	bash scripts/train_sgf_plus_framewise.sh
	```

	Equivalent explicit form:

	```bash
	bash scripts/train_sgf_plus_framewise.sh \
	configs/sgf_plus_framewise.yaml \
	logs/sgf_plus_framewise
	```

	The launchers accept `[config.yaml] [logdir] [extra train.py args...]` and support single-node and multi-node training. Without explicit or scheduler-provided topology settings, nodes auto-register through `.rendezvous/` on the shared filesystem and launch static `torchrun` with an IP master address. For multi-node jobs, run the same command on every node within the gather window.

	Useful overrides:

	```bash
	GATHER_WINDOW=90 NUM_GPUS=8 MASTER_PORT=29501 ENABLE_WANDB=1 \
	bash scripts/train_sgf_plus_chunkwise.sh logs/sgf_plus_chunkwise
	```

	To specify the topology manually, set `NNODES`, `NODE_RANK`, `MASTER_ADDR` and `MASTER_PORT`. Use a distinct `NODE_RANK` on each node and matching values for the other settings.

	## Gradient Conflict Experiments

	To reproduce our gradient-conflict experiments, see the [experiment guide](https://github.com/Zihan-Su/Self_Gradient_Forcing_Plus/blob/main/gradient_conflict/README.md).

	## Citation

	```bibtex
	@article{zhuang2026self,
	title={Self Gradient Forcing: Native Long Video Extrapolation},
	author={Zhuang, Junhao and Zhang, Shiyi and Bian, Yuxuan and Li, Yaowei and Luo, Yawen and Liu, Yijun and Jin, Weiyang and Zhang, Songchun and He, Xianglong and Zhang, Xuying and others},
	journal={arXiv preprint arXiv:2607.20368},
	year={2026}
	}

	@article{su2026sgf+,
	title={SGF+: Decoupling Gradient Flows for Autoregressive Video Generation},
	author={Su, Zihan and Zhuang, Junhao and Li, Yaowei and Lu, Siwen and Li, Haoran and Li, Lingen and Wu, Haoyu and Jin, Weiyang and Zhang, Songchun and Huang, Haoyang and others},
	journal={arXiv preprint arXiv:2610.10429},
	year={2026}
	}
	```

	## License

	This project is released under the [Apache-2.0 license](https://huggingface.co/ZihanSu/Self_Gradient_Forcing_Plus/blob/main/LICENSE).