README.md

---
library_name: diffusers
pipeline_tag: text-to-image
tags:
  - diffusion
  - text-to-image
  - anime
  - pixel-space
  - unet
  - classifier-free-guidance
datasets:
  - huggan/anime-faces
---

# SykoDiffusion-V1.1

A small, text-conditioned **pixel-space diffusion model** that generates **64×64 anime faces** from short attribute prompts such as `anime face, purple long hair, purple eyes, smiling`.

The model was trained from scratch (~38M parameters) on a single Google TPU v5e-1. It has **no VAE**: the UNet works directly on RGB pixels and its output is the final image.

| | |
|---|---|
| **Task** | Text-to-image (structured attribute prompts) |
| **Output resolution** | 64×64 RGB |
| **Architecture** | `diffusers.UNet2DConditionModel`, ~38.4M parameters |
| **Diffusion space** | Pixel space (no VAE / no latent compression) |
| **Text encoder** | CLIP ViT-B/32 text encoder (frozen, `openai/clip-vit-base-patch32`) |
| **Sampler** | DDIM (η = 0) with classifier-free guidance |

---

## Table of contents

- [About the "no VAE" design](#about-the-no-vae-design)
- [Quick start](#quick-start)
- [Prompting guide](#prompting-guide)
- [What the model can and cannot do](#what-the-model-can-and-cannot-do)
- [Model details](#model-details)
- [Training details](#training-details)
- [Repository contents](#repository-contents)
- [Using the original script](#using-the-original-script)
- [Limitations, bias and responsible use](#limitations-bias-and-responsible-use)
- [License](#license)
- [Acknowledgements](#acknowledgements)

---

## About the "no VAE" design

Many popular text-to-image systems (for example Stable Diffusion) are *latent* diffusion models. They compress every image into a small latent tensor with a VAE, run the diffusion process on that latent, and use the VAE decoder at the end to turn the latent back into pixels.

**SykoDiffusion does not do this.**

```
Latent diffusion:  noise → UNet → latent (compressed, not an image) → VAE decoder → image
SykoDiffusion:     noise → UNet → RGB image (64×64×3)
```

- The UNet takes a 3-channel image as input and predicts 3-channel noise (`in_channels=3`, `out_channels=3`).
- Training images are scaled to `[-1, 1]` and fed straight into the UNet.
- There is nothing to decode. The sampler's final tensor *is* the picture.

At 64×64 the pixel-space approach is computationally affordable and avoids the blur and colour artefacts a VAE can introduce.

**Consequences worth knowing:**

- A VAE is not a super-resolution tool. Adding one to this model would not increase the output resolution.
- The output resolution is fixed at 64×64. To obtain larger images you would need a separate super-resolution model, which enlarges (and can amplify) any artefacts in the source image, or a model trained at a higher resolution.

---

## Quick start

The repository stores the UNet in the standard `diffusers` format, plus the CLIP tokenizer and text encoder. It is **not** a full `DiffusionPipeline`, so `DiffusionPipeline.from_pretrained(...)` will not work. The snippet below is a self-contained sampler that reproduces the original DDIM + classifier-free-guidance loop.

```bash
pip install torch diffusers transformers pillow numpy huggingface_hub
```

If the repository is private, authenticate first (`huggingface-cli login`, or pass `token=...` to each `from_pretrained` call).

```python
import numpy as np
import torch
from PIL import Image
from diffusers import DDIMScheduler, UNet2DConditionModel
from transformers import CLIPTextModelWithProjection, CLIPTokenizer

REPO = "SykoSLM/SykoDiffusion-V1.1"
RES, T_STEPS, TOK_LEN = 64, 1000, 24
device = "cuda" if torch.cuda.is_available() else "cpu"

unet = UNet2DConditionModel.from_pretrained(REPO).to(device).eval()
tok = CLIPTokenizer.from_pretrained(REPO, subfolder="tokenizer")
txt = CLIPTextModelWithProjection.from_pretrained(REPO, subfolder="text_encoder").to(device).eval()


@torch.no_grad()
def encode(prompts):
    t = tok(prompts, padding="max_length", max_length=TOK_LEN, truncation=True, return_tensors="pt").to(device)
    return txt(**t).last_hidden_state          # (B, 24, 512)


@torch.no_grad()
def generate(prompt, n=6, steps=50, cfg=4.0, seed=0):
    cond = encode([prompt]).repeat_interleave(n, dim=0)
    null = encode([""]).expand(n, -1, -1)      # empty prompt = unconditional branch
    ctx = torch.cat([null, cond])

    sched = DDIMScheduler(
        num_train_timesteps=T_STEPS,
        beta_schedule="squaredcos_cap_v2",
        clip_sample=True,
        prediction_type="epsilon",
        timestep_spacing="trailing",
    )
    sched.set_timesteps(steps)

    g = torch.Generator().manual_seed(seed)
    x = torch.randn(n, 3, RES, RES, generator=g).to(device)

    for t in sched.timesteps:
        eps = unet(torch.cat([x, x]), t.to(device), encoder_hidden_states=ctx).sample
        e_uncond, e_cond = eps.chunk(2)
        eps = e_uncond + cfg * (e_cond - e_uncond)   # classifier-free guidance
        x = sched.step(eps, t, x).prev_sample

    return (x.clamp(-1, 1) + 1) / 2                  # (n, 3, 64, 64) in [0, 1]


imgs = generate("anime face, purple long hair, purple eyes, smiling", n=6, steps=50, cfg=4.0, seed=0)
imgs = (imgs.permute(0, 2, 3, 1).cpu().numpy() * 255 + 0.5).astype(np.uint8)
Image.fromarray(np.concatenate(list(imgs), axis=1)).save("samples.png")
```

**Sampling parameters**

| Parameter | Default | Notes |
|---|---|---|
| `steps` | 50 | DDIM steps. Going to 100 usually changes little. |
| `cfg` | 4.0 | Guidance scale. Higher values follow the prompt more strongly but can produce dark or oversaturated samples. It is worth experimenting with this value. |
| `seed` | 0 | Same seed + same settings gives the same images. |

---

## Prompting guide

The model was trained on **templated captions**, so it responds best to the same format:

```
anime face, <hair colour> <long|short> hair, <eye colour> eyes[, smiling]
```

**Vocabulary seen during training**

| Attribute | Values |
|---|---|
| Hair colour | blonde, brown, black, blue, pink, purple, green, red, silver, white, orange |
| Hair length | long, short |
| Eye colour | blue, red, green, brown, purple, yellow, orange, pink, black |
| Expression | `smiling` (appended only when a smile was detected) |

Examples:

```
anime face, blue long hair, red eyes, smiling
anime face, brown short hair, green eyes
anime face, pink long hair, purple eyes, smiling
```

Notes:

- Prompts are truncated to **24 CLIP tokens**.
- There is no "neutral" or "not smiling" tag. A neutral expression corresponds to leaving `smiling` out.
- Free-form prompts (styles, accessories, backgrounds, poses) were never seen in training and are not expected to work reliably.
- An empty prompt (`""`) is the unconditional branch used for classifier-free guidance.

---

## What the model can and cannot do

**What it can do**

- Generate 64×64 anime-style faces conditioned on hair colour, hair length, eye colour and (optionally) a smile.
- Produce varied faces for the same prompt by changing the seed.
- Run on CPU, GPU or TPU. The UNet is small enough for casual local experimentation.

**What it cannot do**

- **Produce anything larger than 64×64.** The resolution is fixed.
- **Understand free-form text.** Only the templated attribute vocabulary above is covered by training.
- **Generate anything other than anime faces.** No full-body characters, backgrounds, objects, text or other subjects.
- **Guarantee clean results.** Some samples show asymmetric or blurry eyes and mouths, or are noticeably dark and oversaturated. This happens in a fraction of samples and varies by seed and prompt.
- **Follow attributes perfectly.** The training labels were produced automatically (see below) and contain errors, so the conditioning is approximate.
- **Support negative prompts or image-to-image editing.** Neither was implemented or evaluated.

No quantitative evaluation (e.g. FID) has been run. Quality has only been assessed by visually inspecting fixed-seed sample grids during training.

---

## Model details

| | |
|---|---|
| Backbone | `UNet2DConditionModel` (from `diffusers`), trained from scratch |
| Parameters | ~38.4M |
| Input / output | 3 × 64 × 64 (RGB, range `[-1, 1]`) |
| Blocks | `DownBlock2D`, `CrossAttnDownBlock2D` ×2 → mirrored up blocks (`CrossAttnUpBlock2D` ×2, `UpBlock2D`) |
| Channels per level | 96, 192, 384 |
| Layers per block | 1 |
| Attention | Cross-attention to text at the 32×32 and 16×16 levels; no attention at 64×64 |
| Attention head dim | 8 |
| Group-norm groups | 32 |
| Text conditioning | CLIP ViT-B/32 text encoder, last hidden state, 24 tokens × 512 dims (frozen) |
| Prediction target | ε (noise) |
| Noise schedule | `squaredcos_cap_v2`, 1000 training timesteps |
| Output layer | Zero-initialised at the start of training |

---

## Training details

**Data**

- Source: [`huggan/anime-faces`](https://huggingface.co/datasets/huggan/anime-faces).
- 43,102 images after removing unreadable files, centre-cropped to a square and resized to 64×64.
- **Captions are synthetic.** They were generated automatically with CLIP ViT-B/32 zero-shot classification (hair colour, hair length, eye colour, smile) and assembled into the templated format above. Because these labels are model predictions rather than human annotations, they contain noise.

**Setup**

| | |
|---|---|
| Hardware | Google TPU v5e-1 (Colab), PyTorch/XLA |
| Precision | bf16 autocast for compute; FP32 master weights, EMA and optimiser state |
| Steps | 40,000 (~59 epochs) |
| Batch size | 64 |
| Optimiser | AdamW (β = 0.9, 0.99; weight decay 0.01), gradient clipping at 1.0 |
| Learning rate | 2e-4 peak, 1,000-step linear warm-up, cosine decay to 5% of peak |
| Loss weighting | Min-SNR (γ = 5) |
| EMA | Decay 0.9995 (the released weights are the EMA weights) |
| Augmentation | Random horizontal flip |
| Conditioning dropout | 10% of captions replaced by the empty caption (enables classifier-free guidance) |

**A note on the loss.** The training loss plateaued around 0.025 for most of training even though sample quality kept improving. This is common for diffusion models: most of the loss is irreducible noise-prediction error, and the fine-detail improvements that matter visually contribute very little to it. Fixed-seed sample grids were a more reliable progress signal than the loss curve.

---

## Repository contents

```
.
├── config.json                      # UNet configuration
├── diffusion_pytorch_model.safetensors   # EMA UNet weights
├── text_encoder/                    # CLIP ViT-B/32 text encoder (frozen)
├── tokenizer/                       # CLIP tokenizer
├── anime_t2i_tpu.py                 # Full training / sampling script
└── README.md
```

The CLIP vision encoder is not included. It was only used once, during data preparation, to produce the captions, and is not needed for generation.

---

## Using the original script

`anime_t2i_tpu.py` contains the complete pipeline: data preparation, training on TPU (PyTorch/XLA) and sampling. It reads the model from `<WORK_DIR>/ckpt/ema_unet`, so download the UNet files (`config.json` and `diffusion_pytorch_model.safetensors`) into that folder first.

```bash
# Sample from a trained model
python anime_t2i_tpu.py sample \
  --prompt "anime face, blue long hair, red eyes, smiling" \
  --n 6 --cfg 4 --steps 50 --seed 0

# Prepare data and train from scratch (requires a TPU runtime for the XLA path)
python anime_t2i_tpu.py prepare
python anime_t2i_tpu.py train --steps 40000 --batch 64 --lr 2e-4
```

If you retrain or resume, keep `block_out_channels` and `layers_per_block` in `build_unet` identical to the values listed under [Model details](#model-details), otherwise checkpoint loading will fail with a size mismatch.

---

## Limitations, bias and responsible use

- **Dataset bias.** The model reproduces the style and demographic distribution of its training dataset, including any imbalance in hair colours, eye colours, expressions and character appearance. Rare attribute combinations are generated less reliably.
- **Label noise.** Automatic CLIP labelling means the attribute conditioning is imperfect.
- **Low resolution.** 64×64 output is suited to experimentation, prototyping and research, not to production artwork.
- **Dataset licensing.** The training images come from a third-party dataset. Check the [dataset card](https://huggingface.co/datasets/huggan/anime-faces) for its terms before any commercial use of the model or its outputs.
- The model generates stylised illustrations of fictional faces. It is not intended to depict real people.

---

## License

[Specify the license for the model weights here.]

The CLIP text encoder and tokenizer included in this repository are from OpenAI's CLIP (`openai/clip-vit-base-patch32`), released under the MIT license.

---

## Acknowledgements

- [OpenAI CLIP](https://github.com/openai/CLIP): frozen text encoder and zero-shot captioning.
- [Hugging Face `diffusers`](https://github.com/huggingface/diffusers): UNet implementation and schedulers.
- [`huggan/anime-faces`](https://huggingface.co/datasets/huggan/anime-faces): training images.
- Techniques used: DDPM/DDIM sampling, classifier-free guidance, Min-SNR loss weighting, and EMA of weights.