CSA / README.md

Keep selected previews and refresh packaged documentation

7de67a1 verified 8 days ago

9.05 kB

---
language: en
license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: image-to-image
tags:
  - image-inpainting
  - coherent-semantic-attention
  - iccv-2019
  - pytorch
  - research
---
<div align="center">

<p><strong>ICCV 2019 · Modern PyTorch implementation</strong></p>

<h1>Coherent Semantic Attention for Image Inpainting</h1>

<p>Recover missing image regions with a rough prediction and context-aware refinement.</p>

<p><strong>Hongyu Liu · Bin Jiang · Yi Xiao · Chao Yang</strong></p>

<p>
  <a href="https://openaccess.thecvf.com/content_ICCV_2019/html/Liu_Coherent_Semantic_Attention_for_Image_Inpainting_ICCV_2019_paper.html">Paper</a> ·
  <a href="https://github.com/KumapowerLIU/CSA-inpainting/tree/master">Code</a> ·
  <a href="#try-the-checkpoint">Quickstart</a> ·
  <a href="https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/TRAINING.md">Training</a> ·
  <a href="https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/DATASETS.md">Datasets</a>
</p>

<img src="https://huggingface.co/KumaPower/CSA/resolve/main/preview.jpg" alt="Actual CSA completions: masked input, restored image, and original image." width="960">

<p><sub>Stage 2 · step 97,397. Columns: masked input · restored · original.<br>
Selected examples.</sub></p>

</div>

The author's **CSA** checkpoint combines a separately trained rough U-Net with CSA refinement. This modern release uses random masks, **L1 + LPIPS**, and the original image and feature **PatchGANs**. It updates the original paper's training recipe; see [implementation notes](https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/MIGRATION.md).

The released refinement checkpoint is **step 97,397**, with a frozen rough parent at **step 156,181**. It was selected by visual review, rather than automatic best-metric selection.

## How it works

<p align="center">
  <img src="https://huggingface.co/KumaPower/CSA/resolve/main/pipeline.png" alt="CSA architecture with a gray rough U-Net, red and blue refinement layers, a yellow CSA module at 32 by 32 encoder resolution, and dashed skip connections." width="960">
</p>

**Train rough prediction first, then freeze it and train refinement.** CSA operates on 32 × 32 encoder features at the 256 × 256 training resolution. This diagram follows the visual layout of Figure 2 in the paper and shows the modern training recipe. [Editable SVG](pipeline.svg).

## Try the checkpoint

### 1. Install

**Verified Linux/NVIDIA environment:** Python 3.10.14 · PyTorch 2.6.0 · torchvision 0.21.0 · CUDA 12.4. The [official wheel pairing](https://pytorch.org/get-started/previous-versions/#v260) needs a compatible NVIDIA driver, with no separate CUDA toolkit installation. The wheel below installs CSA without a Git checkout.

```bash
python3.10 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install torch==2.6.0 torchvision==0.21.0 \
  --index-url https://download.pytorch.org/whl/cu124
python -m pip install \
  "https://huggingface.co/KumaPower/CSA/resolve/main/csa_inpainting-0.3.0-py3-none-any.whl"
python - <<'PY'
import torch, torchvision
assert torch.cuda.is_available(), "CUDA is unavailable"
print(torch.__version__, torchvision.__version__)
PY
```

For CPU, macOS, or a source installation, use the [installation guide](https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/USAGE.md#installation-and-devices).

### 2. Download and run

The **528.5 MB** portable checkpoint contains both generators. No separate stage-1 checkpoint, discriminator, VGG, or LPIPS weights are needed for inference.

```bash
mkdir -p checkpoints examples
curl -fL https://huggingface.co/KumaPower/CSA/resolve/main/csa-stage2-step97397.pt \
  -o checkpoints/csa-stage2-step97397.pt
for name in masked mask expected; do
  curl -fL "https://huggingface.co/KumaPower/CSA/resolve/main/examples/${name}.png" \
    -o "examples/${name}.png"
done
csa infer \
  --checkpoint checkpoints/csa-stage2-step97397.pt \
  --image examples/masked.png \
  --mask examples/mask.png \
  --output outputs/restored.png \
  --device auto
```

### 3. Supply your image and mask

<table>
  <tr>
    <th align="center">Input image</th>
    <th align="center">Mask</th>
    <th align="center">CSA output</th>
  </tr>
  <tr>
    <td><img src="https://huggingface.co/KumaPower/CSA/resolve/main/examples/masked.png" alt="Example input with missing image regions." width="256"></td>
    <td><img src="https://huggingface.co/KumaPower/CSA/resolve/main/examples/mask.png" alt="Binary mask: white is missing; black is known." width="256"></td>
    <td><img src="https://huggingface.co/KumaPower/CSA/resolve/main/examples/expected.png" alt="Actual output from the released CSA checkpoint." width="256"></td>
  </tr>
</table>
**White = fill; black = keep.** Image and mask dimensions must match. An intact image also works: the CLI masks the hole internally. The example above is a real checkpoint output.

The model was trained at **256 × 256**. Other sizes are padded and cropped, but their restoration quality has not been validated. `--device auto` selects CUDA, then MPS, then CPU. See [usage](https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/USAGE.md) for custom inputs and folder evaluation.

## Train and extend

Use the [source repository](https://github.com/KumapowerLIU/CSA-inpainting/tree/master) and [two-stage training guide](https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/TRAINING.md) to prepare image indices, train rough prediction, initialize refinement, and resume a run. Install the training dependencies with `python -m pip install -e '.[perceptual]'` from the checkout.

| Objective | Fresh-training recipe |
|---|---|
| Stage 1 | `0.1 × L1 + 1 × LPIPS` |
| Stage 2 | `0.1 × L1 + 1 × LPIPS + 0.01 × consistency + 0.002 × (image GAN + feature GAN)` |

Stage 2 uses original PatchGAN architectures with random initialization and RaLSGAN losses. Rectangles, strokes, or their union cover **10–40%** of each image; masks are generated online. The [saved configuration](training-config.yaml) records the selected experiment.

### Hardware and batch size

| Setting | Stage 1 LPIPS phase | Stage 2 refinement |
|---|---|---|
| GPU / VRAM capacity | 1 × NVIDIA H100 80GB HBM3 | 1 × NVIDIA H100 80GB HBM3 |
| Optimizer batch | 256 | 192 |
| Precision | BF16 mixed precision | BF16 mixed precision |
| Image resolution | 256 × 256 | 256 × 256 |
| LPIPS activation chunk | 8 images | 8 images |
| Observed PyTorch peak allocated / reserved | 9.15 / 12.08 GiB, training snapshot | 31.18 / 34.42 GiB, two-update probe |

**80 GB is capacity; the last row is measured usage.** LPIPS chunking is separate from optimizer batch size. The guide starts at batch 8 and provides a memory probe for your hardware.

### Data and official sources

| Dataset | Used in this release | Where to find it |
|---|---|---|
| Unsplash-derived photographs | 296,145 training records; 2,000 held-out records | [Unsplash Dataset](https://unsplash.com/data) · [access and schema](https://github.com/unsplash/datasets) |
| Places365-Standard | 901,730 images: half of 1,803,460 training images | [Download page](http://places2.csail.mit.edu/download.html) · [CSAIL repository](https://github.com/CSAILVision/places365) |
| HDTF talking-head frames | Added during stage 2: 30,128 frames represented by 299,469 sampling entries | [HDTF data and preprocessing](https://github.com/MRzzm/HDTF) |

The base mixture contains **1,197,875 entries**; the final stage-2 mixture contains **1,497,344 sampling entries**, approximately 20% faces. Repeated entries are not unique images. The exact local splits are not distributed; [dataset preparation and provenance](https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/DATASETS.md) explain the selection and how to use public sources or your own images.

## Checkpoint and provenance

The portable artifact is for inference. It contains no optimizer or discriminator state and **cannot resume training**. The full training checkpoint is preserved separately and is not included in this release; fresh training is documented in the guide.

The model retains width 64, depth 8, and a `tanh` refinement head. Export preserves all generator tensors; see [export verification](export-verification.json), [software validation](validation.json), and [checksums](SHA256SUMS). The [evaluation record](evaluation.json) and [selected preview](preview.jpg) preserve the selected checkpoint's evidence.

## Citation and license

```bibtex
@InProceedings{Liu_2019_ICCV,
  author    = {Liu, Hongyu and Jiang, Bin and Xiao, Yi and Yang, Chao},
  title     = {Coherent Semantic Attention for Image Inpainting},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision},
  month     = {October},
  year      = {2019}
}
```

The release retains the repository's [CC BY-NC 4.0 / noncommercial research-use notice](LICENSE.md). Dataset and example-image rights remain with their respective owners.

	---
	language: en
	license: cc-by-nc-4.0
	library_name: pytorch
	pipeline_tag: image-to-image
	tags:
	- image-inpainting
	- coherent-semantic-attention
	- iccv-2019
	- pytorch
	- research
	---

	<div align="center">

	<p><strong>ICCV 2019 · Modern PyTorch implementation</strong></p>

	<h1>Coherent Semantic Attention for Image Inpainting</h1>

	<p>Recover missing image regions with a rough prediction and context-aware refinement.</p>

	<p><strong>Hongyu Liu · Bin Jiang · Yi Xiao · Chao Yang</strong></p>

	<p>
	<a href="https://openaccess.thecvf.com/content_ICCV_2019/html/Liu_Coherent_Semantic_Attention_for_Image_Inpainting_ICCV_2019_paper.html">Paper</a> ·
	<a href="https://github.com/KumapowerLIU/CSA-inpainting/tree/master">Code</a> ·
	<a href="#try-the-checkpoint">Quickstart</a> ·
	<a href="https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/TRAINING.md">Training</a> ·
	<a href="https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/DATASETS.md">Datasets</a>
	</p>

	<img src="https://huggingface.co/KumaPower/CSA/resolve/main/preview.jpg" alt="Actual CSA completions: masked input, restored image, and original image." width="960">

	<p><sub>Stage 2 · step 97,397. Columns: masked input · restored · original.<br>
	Selected examples.</sub></p>

	</div>

	The author's CSA checkpoint combines a separately trained rough U-Net with CSA refinement. This modern release uses random masks, L1 + LPIPS, and the original image and feature PatchGANs. It updates the original paper's training recipe; see [implementation notes](https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/MIGRATION.md).

	The released refinement checkpoint is step 97,397, with a frozen rough parent at step 156,181. It was selected by visual review, rather than automatic best-metric selection.

	## How it works

	<p align="center">
	<img src="https://huggingface.co/KumaPower/CSA/resolve/main/pipeline.png" alt="CSA architecture with a gray rough U-Net, red and blue refinement layers, a yellow CSA module at 32 by 32 encoder resolution, and dashed skip connections." width="960">
	</p>

	Train rough prediction first, then freeze it and train refinement. CSA operates on 32 × 32 encoder features at the 256 × 256 training resolution. This diagram follows the visual layout of Figure 2 in the paper and shows the modern training recipe. [Editable SVG](pipeline.svg).

	## Try the checkpoint

	### 1. Install

	Verified Linux/NVIDIA environment: Python 3.10.14 · PyTorch 2.6.0 · torchvision 0.21.0 · CUDA 12.4. The [official wheel pairing](https://pytorch.org/get-started/previous-versions/#v260) needs a compatible NVIDIA driver, with no separate CUDA toolkit installation. The wheel below installs CSA without a Git checkout.

	```bash
	python3.10 -m venv .venv
	source .venv/bin/activate
	python -m pip install --upgrade pip
	python -m pip install torch==2.6.0 torchvision==0.21.0 \
	--index-url https://download.pytorch.org/whl/cu124
	python -m pip install \
	"https://huggingface.co/KumaPower/CSA/resolve/main/csa_inpainting-0.3.0-py3-none-any.whl"
	python - <<'PY'
	import torch, torchvision
	assert torch.cuda.is_available(), "CUDA is unavailable"
	print(torch.__version__, torchvision.__version__)
	PY
	```

	For CPU, macOS, or a source installation, use the [installation guide](https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/USAGE.md#installation-and-devices).

	### 2. Download and run

	The 528.5 MB portable checkpoint contains both generators. No separate stage-1 checkpoint, discriminator, VGG, or LPIPS weights are needed for inference.

	```bash
	mkdir -p checkpoints examples
	curl -fL https://huggingface.co/KumaPower/CSA/resolve/main/csa-stage2-step97397.pt \
	-o checkpoints/csa-stage2-step97397.pt
	for name in masked mask expected; do
	curl -fL "https://huggingface.co/KumaPower/CSA/resolve/main/examples/${name}.png" \
	-o "examples/${name}.png"
	done

	csa infer \
	--checkpoint checkpoints/csa-stage2-step97397.pt \
	--image examples/masked.png \
	--mask examples/mask.png \
	--output outputs/restored.png \
	--device auto
	```

	### 3. Supply your image and mask

	<table>
	<tr>
	<th align="center">Input image</th>
	<th align="center">Mask</th>
	<th align="center">CSA output</th>
	</tr>
	<tr>
	<td><img src="https://huggingface.co/KumaPower/CSA/resolve/main/examples/masked.png" alt="Example input with missing image regions." width="256"></td>
	<td><img src="https://huggingface.co/KumaPower/CSA/resolve/main/examples/mask.png" alt="Binary mask: white is missing; black is known." width="256"></td>
	<td><img src="https://huggingface.co/KumaPower/CSA/resolve/main/examples/expected.png" alt="Actual output from the released CSA checkpoint." width="256"></td>
	</tr>
	</table>

	White = fill; black = keep. Image and mask dimensions must match. An intact image also works: the CLI masks the hole internally. The example above is a real checkpoint output.

	The model was trained at 256 × 256. Other sizes are padded and cropped, but their restoration quality has not been validated. `--device auto` selects CUDA, then MPS, then CPU. See [usage](https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/USAGE.md) for custom inputs and folder evaluation.

	## Train and extend

	Use the [source repository](https://github.com/KumapowerLIU/CSA-inpainting/tree/master) and [two-stage training guide](https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/TRAINING.md) to prepare image indices, train rough prediction, initialize refinement, and resume a run. Install the training dependencies with `python -m pip install -e '.[perceptual]'` from the checkout.

	\| Objective \| Fresh-training recipe \|
	\|---\|---\|
	\| Stage 1 \| `0.1 × L1 + 1 × LPIPS` \|
	\| Stage 2 \| `0.1 × L1 + 1 × LPIPS + 0.01 × consistency + 0.002 × (image GAN + feature GAN)` \|

	Stage 2 uses original PatchGAN architectures with random initialization and RaLSGAN losses. Rectangles, strokes, or their union cover 10–40% of each image; masks are generated online. The [saved configuration](training-config.yaml) records the selected experiment.

	### Hardware and batch size

	\| Setting \| Stage 1 LPIPS phase \| Stage 2 refinement \|
	\|---\|---\|---\|
	\| GPU / VRAM capacity \| 1 × NVIDIA H100 80GB HBM3 \| 1 × NVIDIA H100 80GB HBM3 \|
	\| Optimizer batch \| 256 \| 192 \|
	\| Precision \| BF16 mixed precision \| BF16 mixed precision \|
	\| Image resolution \| 256 × 256 \| 256 × 256 \|
	\| LPIPS activation chunk \| 8 images \| 8 images \|
	\| Observed PyTorch peak allocated / reserved \| 9.15 / 12.08 GiB, training snapshot \| 31.18 / 34.42 GiB, two-update probe \|

	80 GB is capacity; the last row is measured usage. LPIPS chunking is separate from optimizer batch size. The guide starts at batch 8 and provides a memory probe for your hardware.

	### Data and official sources

	\| Dataset \| Used in this release \| Where to find it \|
	\|---\|---\|---\|
	\| Unsplash-derived photographs \| 296,145 training records; 2,000 held-out records \| [Unsplash Dataset](https://unsplash.com/data) · [access and schema](https://github.com/unsplash/datasets) \|
	\| Places365-Standard \| 901,730 images: half of 1,803,460 training images \| [Download page](http://places2.csail.mit.edu/download.html) · [CSAIL repository](https://github.com/CSAILVision/places365) \|
	\| HDTF talking-head frames \| Added during stage 2: 30,128 frames represented by 299,469 sampling entries \| [HDTF data and preprocessing](https://github.com/MRzzm/HDTF) \|

	The base mixture contains 1,197,875 entries; the final stage-2 mixture contains 1,497,344 sampling entries, approximately 20% faces. Repeated entries are not unique images. The exact local splits are not distributed; [dataset preparation and provenance](https://github.com/KumapowerLIU/CSA-inpainting/blob/master/docs/DATASETS.md) explain the selection and how to use public sources or your own images.

	## Checkpoint and provenance

	The portable artifact is for inference. It contains no optimizer or discriminator state and cannot resume training. The full training checkpoint is preserved separately and is not included in this release; fresh training is documented in the guide.

	The model retains width 64, depth 8, and a `tanh` refinement head. Export preserves all generator tensors; see [export verification](export-verification.json), [software validation](validation.json), and [checksums](SHA256SUMS). The [evaluation record](evaluation.json) and [selected preview](preview.jpg) preserve the selected checkpoint's evidence.

	## Citation and license

	```bibtex
	@InProceedings{Liu_2019_ICCV,
	author = {Liu, Hongyu and Jiang, Bin and Xiao, Yi and Yang, Chao},
	title = {Coherent Semantic Attention for Image Inpainting},
	booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision},
	month = {October},
	year = {2019}
	}
	```

	The release retains the repository's [CC BY-NC 4.0 / noncommercial research-use notice](LICENSE.md). Dataset and example-image rights remain with their respective owners.