PtahX32

A text-to-pixel-art diffusion model, trained from scratch at 32×32.

_{Samples from the EMA weights at step 30,000 (fixed prompts, guidance 4.0, 30 DDIM steps).}

---

Overview

PtahX32 is a compact, text-conditional diffusion model that generates 32×32 pixel-art sprites from short text prompts. Every weight of the denoiser was trained from scratch on a single Colab T4 GPU. Only the text encoder (CLIP) is pretrained and frozen.

Because pixel art is tiny, the model denoises directly in pixel space. There is no VAE and no latent stage, which keeps the whole pipeline small, fast, and easy to inspect.

Task	Text-to-image (pixel art)
Output	32×32 RGB (upscale with nearest-neighbor)
Architecture	Conditional UNet with cross-attention (UNet2DConditionModel)
Text encoder	Frozen openai/clip-vit-base-patch32, 24 tokens
Objective	Noise (epsilon) prediction, MSE
Noise schedule	DDPM, 1000 steps, squaredcos_cap_v2
Sampler	DDIM with classifier-free guidance
Training data	m1guelpf/nouns
Hardware	1× NVIDIA T4 (16 GB), fp16 mixed precision
License	MIT (see License for data and dependencies)

Samples

All images below come straight from the training run. They are generated from the EMA weights with a fixed seed and a fixed set of prompts taken from the training captions, so the grids are directly comparable across steps. Each grid is 8 images (4 per row), upscaled 4× with nearest-neighbor.

Step 500	Step 5,000
Step 10,000	Step 25,000

Every checkpoint's grid (one every 500 steps) is in the samples/ folder, so you can scrub through the whole training run.

Quick start

Install the dependencies:

pip install torch diffusers transformers huggingface_hub pillow numpy

Generate images:

import numpy as np, torch
from PIL import Image
from huggingface_hub import hf_hub_download
from transformers import CLIPTokenizer, CLIPTextModel
from diffusers import UNet2DConditionModel, DDPMScheduler, DDIMScheduler

REPO, MAX_TOK, RES = "viknesh100/PtahX32", 24, 32
dev = "cuda" if torch.cuda.is_available() else "cpu"

# 1. Load the EMA weights and the frozen text encoder
ck = torch.load(hf_hub_download(REPO, "checkpoints/best.pt"), map_location="cpu", weights_only=False)
unet = UNet2DConditionModel(**ck["arch"]); unet.load_state_dict(ck["ema"]); unet.to(dev).eval()

tok  = CLIPTokenizer.from_pretrained("openai/clip-vit-base-patch32")
clip = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32").to(dev).eval()

def embed(prompts):
    ids = tok(prompts, padding="max_length", max_length=MAX_TOK, truncation=True,
              return_tensors="pt").input_ids.to(dev)
    return clip(input_ids=ids).last_hidden_state.float()

# 2. Sample with classifier-free guidance
@torch.no_grad()
def generate(prompt, n=4, steps=50, guidance=5.0, seed=0):
    base = DDPMScheduler(num_train_timesteps=1000, beta_schedule="squaredcos_cap_v2")
    sched = DDIMScheduler.from_config(base.config); sched.set_timesteps(steps)
    c, u = embed([prompt] * n), embed([""] * n)
    g = torch.Generator(device=dev).manual_seed(seed)
    x = torch.randn(n, 3, RES, RES, device=dev, generator=g)
    for t in sched.timesteps:
        e = unet(torch.cat([x, x]), t, encoder_hidden_states=torch.cat([u, c])).sample
        eu, ec = e.chunk(2)
        x = sched.step(eu + guidance * (ec - eu), t, x).prev_sample
    return ((x.clamp(-1, 1) + 1) * 127.5).round().byte().permute(0, 2, 3, 1).cpu().numpy()

# 3. Save as crisp, upscaled pixel art (always use NEAREST, never bilinear)
imgs = generate("your prompt here", n=4)
for i, a in enumerate(imgs):
    Image.fromarray(a).resize((256, 256), Image.NEAREST).save(f"ptahx32_{i}.png")

Tips for better results

• Write prompts like the training captions. The model only knows the caption style of its training set. Browse a few captions in m1guelpf/nouns and imitate them. Prompts far from that style give poor results.

• Guidance scale: 3 to 7. Lower is more varied, higher follows the prompt more closely but can oversaturate.

• Steps: 30 is fast, 50 to 100 is cleaner.

• Palette snapping: reducing the output to 8 to 16 colors removes muddy in-between shades and looks more like hand-made pixel art:

  im = Image.fromarray(a).quantize(colors=12, method=Image.Quantize.MEDIANCUT,
                                   dither=Image.Dither.NONE).convert("RGB")
  

• Generate several seeds and keep the best. Diffusion quality varies from seed to seed.

Model details

Architecture

Component	Setting
Denoiser	UNet2DConditionModel, 3 resolution levels
Channels per level	128, 256, 256
Layers per block	2
Down blocks	DownBlock2D, CrossAttnDownBlock2D, CrossAttnDownBlock2D
Up blocks	CrossAttnUpBlock2D, CrossAttnUpBlock2D, UpBlock2D
Cross-attention dim	512 (CLIP text hidden size)
Attention head dim	8
Input / output	3×32×32

Training procedure

Setting	Value
Optimizer	AdamW, weight decay 0.01
Learning rate	2e-4, linear warmup over 500 steps, then constant
Batch size	128
Gradient clipping	Global norm 1.0
Precision	fp16 autocast with gradient scaling
EMA	Decay 0.9995 with warmup (min(0.9995, (1+t)/(10+t)))
Caption dropout	10% (trains the unconditional path for classifier-free guidance)
Validation	512 held-out images, fixed split, fixed noise and timesteps
Checkpointing	Committed to the Hub every 500 steps (last.pt, plus best.pt on a new best val loss)
Throughput	About 1.1 to 1.2 steps/s on a T4

The pipeline was built to be fully resumable: model, optimizer, EMA, scaler, step count, and loss history are saved to the Hub, so training can be paused and continued from any session or account.

Evaluation

The model is monitored with a deterministic validation loss (noise-prediction MSE on held-out images, evaluated with the EMA weights and a fixed noise seed).

Metric	Value
Best validation loss	0.0022 (around step 31.5k)
Training loss at that point	0.0044 (raw model, with caption dropout)

Validation loss for diffusion models flattens early and is a poor proxy for visual quality, and it is not comparable across different datasets or architectures. The sample grids in samples/ are the more reliable view of progress. No FID or CLIP-score benchmark has been run.

Repository contents

PtahX32/
├── checkpoints/
│   ├── best.pt          # EMA weights at the lowest validation loss (use this for inference)
│   └── last.pt          # full training state (model, EMA, optimizer, scaler, step), for resuming
├── data/cache.pt        # preprocessed 32×32 dataset cache
├── logs/
│   ├── history.json     # train/val loss every 500 steps
│   └── curve.png        # loss curves
└── samples/             # sample grid every 500 steps

Intended use

PtahX32 is a small research and hobby model. Reasonable uses include generating sprites and avatars in its training style, learning how text-conditional diffusion works end to end, and serving as a starting point for finetuning on other pixel-art data.

Limitations

• Narrow domain. It was trained on one dataset with one art style, so it will not produce arbitrary pixel-art scenes, landscapes, or styles it hasn't seen.
• Caption-sensitive. Prompt following depends on how close your wording is to the training captions. Short, dataset-style prompts work best.
• Fixed 32×32 resolution. There is no super-resolution stage. Upscale with nearest-neighbor only.
• Small model, small data. Fine details and rare attribute combinations can be wrong or blurry.
• No safety filtering. Outputs are not filtered, though the training data is a curated set of cartoon-style avatars.

Bias and ethics

The model reproduces the content distribution of its training data, so what it generates follows the styles, attributes, and caption vocabulary of that dataset. It was not trained on photographs or on images of real people.

License

The model weights and the code in this card are released under the MIT License.

• Training data: m1guelpf/nouns. Check the dataset card for its current license terms before commercial use.
• Text encoder: openai/clip-vit-base-patch32 is a separate pretrained model under its own license and is not redistributed in this repository.

Acknowledgements

Built with PyTorch, 🤗 diffusers, 🤗 transformers, and OpenAI's CLIP. The method follows Ho et al., Denoising Diffusion Probabilistic Models; Song et al., Denoising Diffusion Implicit Models; Nichol & Dhariwal, Improved DDPM (cosine schedule); and Ho & Salimans, Classifier-Free Diffusion Guidance.

Citation

@misc{ptahx32,
  title  = {PtahX32: a text-to-pixel-art diffusion model trained from scratch},
  author = {viknesh100},
  year   = {2026},
  url    = {https://huggingface.co/viknesh100/PtahX32}
}