pixelgen2 / README.md
DJLougen's picture
revert to 6fbb543
7190edb verified
|
Raw History Blame Contribute Delete
14.5 kB
---
license: mit
library_name: pytorch
pipeline_tag: text-to-image
tags:
- pixel-art
- weight-space-programming
- hand-written-weights
- no-training
- tiny-model
- pico-8
- toy
language:
- en
---
<div align="center">
# 🕹️ PixelGen2
**A text-to-pixel-art model with 6,747 parameters. Zero of them were trained.**
Every weight was typed in by hand. It draws 21,866 distinct sprites from 28 templates.
<img src="showcase2.png" alt="12 bodies × 8 accessories, two colours each, generated by PixelGen2" width="100%"/>
<sub>Every sprite above is one prompt: <code>"&lt;colour&gt; &lt;body&gt; with &lt;colour&gt; &lt;accessory&gt;"</code>. No gradients were harmed.</sub>
<br/>
![params](https://img.shields.io/badge/parameters-6%2C747-ff004d?style=flat-square)
![outputs](https://img.shields.io/badge/distinct%20sprites-21%2C866-29adff?style=flat-square)
![training](https://img.shields.io/badge/training%20steps-0-00e436?style=flat-square)
![palette](https://img.shields.io/badge/palette-PICO--8-ffec27?style=flat-square)
![license](https://img.shields.io/badge/license-MIT-83769c?style=flat-square)
</div>
---
## What this is
A tiny PyTorch `nn.Module` that turns a prompt like `"red potion"` or `"purple ghost with gold crown"` into a 16×16 pixel-art sprite in the PICO-8 palette, upscaled to 128×128.
It's an experiment in **weight-space programming**, inspired by [this post](https://rikiyat.github.io/blog/weight-space-programming/): instead of training a network, you write a program and *compile it into weights*. Linear layers become lookup tables, argmax becomes a decision, a 1×1 conv becomes a palette. The model is a real `nn.Module` — you can call `.forward()`, save a `state_dict`, load it back — but no optimizer ever touched it.
There are two models in the repo:
| | `PixelGen` (v1) | `PixelGen2` (v2) |
|---|---|---|
| prompt | `"<colour> <body>"` | `"[small] <colour> <body> [with <colour> <accessory>]"` |
| parameters | 4,227 | 6,747 |
| hand-drawn templates | 21 bodies | 21 bodies + 7 accessories |
| distinct sprites (seed 0) | 189 | **21,866** |
| shading | fixed row bands | structural rim light (two 3×3 convs) |
v1 is the minimal version, published separately as [DJLougen/pixelgen](https://huggingface.co/DJLougen/pixelgen); v2 is the same thing plus four compositional layers, and is what the CLI uses by default. This repo contains both. Everything below was measured, not estimated — scripts in `experiments/`.
## How it works
Nine steps for v1. Each is an ordinary tensor op; together they're a program.
```
"red potion"
│
▼
1. bag of words prompt → count vector over a 138-word vocab
2. class WTA Linear(138→21) → argmax → one-hot body [potion]
3. shape bank one-hot selects a 16×8 half-template;
'?' cells flip on when seed-noise ≥ 0.5
4. mirror half + horizontal flip → 16×16 body (always symmetric)
5. outline 3×3 dilate(body) − body → outline pixels (palette index 1)
6. shading rows 0–6 light, 7–10 mid, 11–15 dark
7. ramp WTA Linear(138→9) → argmax → colour ramp [red]
8. one-hot palette every pixel → one-hot over 16 PICO-8 colours
9. palette conv Conv2d(16→3, 1×1) → RGB
│
▼
16×16 sprite → nearest-neighbour ×8 → 128×128 PNG
```
v2 inserts four more fixed-weight layers (`pixelgen/compose.py`):
| layer | mechanism | params |
|---|---|---|
| **accessory** | `Linear(138→8)` WTA selects a 16×8 part (crown, halo, horns, helmet, antenna, wings, torch) overlaid on the body with `max`. `none` wins ties. | 1,112 |
| **second colour** | the prompt is split at `with`/`wearing`/`holding`/`and`; a second `Linear(138→9)` colours the accessory from the words after it. No colour word there → body colour wins via a 0.5 one-hot bonus. | 1,251 |
| **structural shading** | two 3×3 convs with a single tap each: a pixel is *light* if its up-left neighbour is empty, *dark* if its down-right neighbour is. Light from the top-left, for every shape, for 18 parameters. | 18 |
| **size** | `small`/`tiny`/`mini` → nearest-neighbour 0.75× resample of the body map, bottom-centred. | 139 |
The "generative" part is step 3: templates have `?` cells gated by per-seed noise, so the same prompt gives small variations across seeds:
<div align="center">
<img src="seeds.png" alt="blue invader, seeds 0–7" width="80%"/>
<sub><code>"blue invader"</code>, seeds 0 through 7</sub>
</div>
## Numbers
| | v1 | v2 |
|---|---|---|
| Registered parameters | 4,227 | **6,747** |
| Non-trainable buffers (template / shade / ramp / part banks) | 6,171 | 7,333 |
| Total values | 10,398 (≈41.6 KB fp32) | 14,080 (≈56.3 KB fp32) |
| Training steps | **0** | **0** |
| Optimizers, backward passes, `requires_grad=True` in the repo | 0 (grep-verified) | 0 |
| Bodies / accessories / colour ramps | 21 / – / 9 | 21 / 7 / 9 |
| Distinct images at seed 0 | 189 of 189 prompts | 21,866 of 24,192 prompts |
| Output | 16×16 → 128×128 RGB, every pixel ∈ PICO-8 | same |
| `weights/pixelgen.pt` (v1) | 45.7 KB, SHA-256 `4c5dffd1…540eb`, identical across rebuilds | – |
<details>
<summary>Parameter breakdown (v2; v1 is the first six rows)</summary>
| Tensor | Shape | Count | What it is |
|---|---|---|---|
| `word_to_class.weight` | 21 × 138 | 2,898 | vocab → body scores (synonym table) |
| `word_to_class.bias` | 21 | 21 | default body (potion) |
| `word_to_ramp.weight` | 9 × 138 | 1,242 | vocab → colour-ramp scores |
| `word_to_ramp.bias` | 9 | 9 | default ramp (green) |
| `outline.weight` | 1×1×3×3 | 9 | dilation kernel |
| `palette_conv.weight` | 3×16×1×1 | 48 | one-hot colour → RGB (exact PICO-8 ÷ 255) |
| `word_to_acc.weight` / `.bias` | 8 × 138, 8 | 1,112 | vocab → accessory scores, `none` biased |
| `word_to_ramp2.weight` / `.bias` | 9 × 138, 9 | 1,251 | accessory-half vocab → ramp |
| `lit.weight`, `shadow.weight` | 1×1×3×3 ×2 | 18 | up-left tap, down-right tap |
| `word_to_size.weight` / `.bias` | 1 × 138, 1 | 139 | size words → small flag |
| **Parameters** | | **6,747** | |
| `on_bank` / `free_bank` | 21×16×8 ×2 | 5,376 | body templates (`#` solid, `?` noise-controlled) |
| `acc_bank` | 8×16×8 | 1,024 | accessory templates |
| `shade_bank` | 3×16×16 | 768 | row-band base shading |
| `ramp_table` | 9×3 | 27 | palette indices per ramp |
| `acc_mask_a` | 138 | 138 | body-class words never mean a hat in the body half |
| **Buffers** | | **7,333** | |
</details>
## How to use
```bash
git clone https://huggingface.co/DJLougen/pixelgen2 && cd pixelgen2
pip install torch numpy pillow # that's the whole dependency list
python generate.py "red potion" --out out/red_potion.png
python generate.py "green slime with gold crown" --seed 3 --scale 16
python generate.py "small angel skull" --out out/skull.png
python generate.py "blue invader" --v1 # original single-part model
```
Or from Python:
```python
from pixelgen.compose import PixelGen2
model = PixelGen2() # weights are built in __init__, no file needed
img = model.sprite("purple ghost with horns", seed=4, scale=8) # PIL.Image, 128×128
img.save("ghost.png")
```
**Prompt grammar.** `[small] [colour] <body> [with [colour] <accessory>]`. Words are matched by a bag-of-words, so order doesn't matter within each half.
<details>
<summary>Vocabulary</summary>
**Bodies:** potion (bottle, flask, elixir, vial) · sword (blade, dagger) · shield (buckler) · ship (rocket, spaceship, starship) · invader (alien, monster) · slime (blob, jelly) · skull (bone, skeleton) · tree (pine, forest) · key · heart (love, life) · star (sparkle) · mushroom (fungus, toadstool, shroom) · ghost (spirit, phantom, spooky) · bomb (grenade, explosive) · coin (token, money, medal) · gem (diamond, jewel, crystal) · cherry (cherries, berry, fruit) · crown · bat (vampire) · cactus (desert, succulent) · robot (droid, android, mech)
**Accessories:** crown (king, queen, royal, crowned, prince, princess) · halo (angel, angelic, holy, saint) · horns (horned, devil, demon, evil) · helmet (knight, armor, armour, armored) · antenna (antennae, bug, martian) · wings (winged, fly, flying, bird, fairy, butterfly) · torch (torches, fire, flame, burning)
**Colours:** red (crimson, ruby) · blue (azure, sapphire) · green (emerald, lime) · yellow (gold, golden) · purple (violet, amethyst) · orange (amber) · pink (rose) · brown (wood, wooden) · white (grey, gray, silver, bone)
**Size:** small, tiny, mini, little. **Separators:** with, wearing, holding, and, plus.
`"crown"` in the body half is the crown object; after a separator it's a hat. `"bone"` is both a skull synonym and a white synonym, so `"bone"` alone is a white skull. `"alien"` is an invader body and `"martian"` is an antenna accessory, so `"martian alien"` is what you'd hope.
</details>
## How far can you push it?
Four experiments, all in `experiments/` with the scripts that produced them. Full notes in [`experiments/README.md`](experiments/README.md).
**1. Adding a class is a pure append.** Freeze the 11-class model, add `mushroom`, diff. +400 values. Diffed by *index*, 84 old values changed; diffed by *meaning* (class name × word), **zero** — the raw diff was entirely a column permutation from a `sorted()` vocab. 17/17 old prompts bit-identical over 3 seeds. The same check still passes after +10 classes and the whole accessory vocabulary.
**2. Capacity is unbounded; addressing is quadratic.** Synthetic random classes, rebuilt and probed at each size:
| classes | params | dense layer share | isolated |
|---|---|---|---|
| 21 | 2,937 | 22% | ✓ |
| 121 | 51,537 | 57% | ✓ |
| 1,021 | 3.2M | 92% | ✓ |
| 10,021 | 302M | 99.1% | ✓ |
It never breaks — the original 21 classes stay bit-identical at 10k. But vocab grows with classes, so `word_to_class` is ≈3C² and 99.997% zeros. Under 100k params the dense design tops out near **170 classes**.
**3. Compressing the class layer costs semantics.** A binary code (`d = ⌈log₂C⌉`, words emit ±1 codes, score = `d − 2·hamming`) shrinks the class layer **528×** at 10k classes with no loss of isolation — but 61% of two-body-word prompts (`"sword shield"`) now resolve to a *third* class, vs 0% dense. The additive bag-of-words semantics were the thing that got compressed away.
**4. Composition is the lever.** v1 → v2: 1.6× params, **115× distinct outputs**. 781 distinct images per hand-drawn template instead of 9.
<div align="center">
<img src="experiments/shading_before_after.png" alt="row-band shading vs structural rim lighting" width="100%"/>
<sub>Top: v1 row bands. Bottom: v2 structural lighting, 18 params.</sub>
</div>
Quality, honestly: structural lighting helps blobby shapes (potion, heart, mushroom, tree, gem) and **hurts thin ones** (invader, bat, robot) — when every pixel is an edge, rim light becomes speckle. A fixed 3×3 kernel has no way to know a shape is too thin to light. That's the limit of hand-written rules without a learned prior, and exactly the kind of thing a few hundred trained parameters would fix.
## Limitations
- **It only knows what was drawn.** 21 bodies, 7 accessories. Prompt for a dragon and you get a potion (the default). Capacity is unbounded; *content* is a human with a text editor.
- **Every shape is horizontally symmetric.** The mirror step makes 16×8 templates cheap; it also means `"torch"` is two torches and the key is a top-down key. Lighting (v2) is deliberately *not* symmetric.
- **Accessories sit at fixed rows.** A crown floats above short bodies; a helmet merges into tall ones. Fixing that needs per-body anchor points — more drawing, not more mechanism.
- **One accessory per sprite.** `"devil knight"` picks one by index order.
- **Variation is cosmetic.** Seeds only toggle the `?` cells — a few pixels per sprite.
- **Bag of words.** `"sword shield"` is a tie broken by index; `"not red"` is red; within each half, order is ignored.
- **Structural shading speckles thin shapes.** See above.
- **It's PyTorch for the bit, not for the need.** You could rewrite this in NumPy. Using `nn.Linear` and `nn.Conv2d` is the point of the exercise.
- **Art quality is programmer art**, drawn in a text editor by someone who is not a pixel artist.
## Why bother?
Because it's a nice demonstration that a lot of what we call "a model" is just *structure*, and structure can be written directly. A linear layer with the right weights *is* a dictionary. A conv with the right kernel *is* an edge detector. Winner-take-all *is* an `if`. The blog post argues this is a useful lens for understanding what trained networks are doing; this repo is the version where you skip the training and just write the weights — and then push on it to see where hand-writing stops being enough.
Where it stops: addressing (quadratic), compression (breaks additivity), and quality rules at the edges (shading thin shapes). The natural next step is the hybrid — hand-write the ~95% that's structure, train only the small word→part map.
Also it's cool that it worked.
## Repo contents
```
pixelgen/
model.py PixelGen (v1) — all weights set in __init__
compose.py PixelGen2 (v2) — v1 + accessory, second colour, structural shading, size
templates.py 21 body half-templates ('#' solid, '?' noise, '.' empty)
parts.py 7 accessory half-templates
palette.py PICO-8 palette + 9 colour ramps
tokenizer.py vocab, synonyms, prompt splitting
weights.py save/load helpers
build.py writes weights/pixelgen.pt (v1, deterministic)
generate.py CLI (v2 default, --v1 for the original)
tests/ 16 property tests
experiments/ the four experiments above: scripts, frozen baselines, README
showcase2.png the grid at the top
seeds.png seed variation strip
```
Tests: `python -m pytest tests -q` → 16 passed.
## Citation
```bibtex
@misc{pixelgen2_2025,
title = {PixelGen2: a compositional hand-weighted pixel-art generator},
author = {Lougen, D. J.},
year = {2025},
url = {https://huggingface.co/DJLougen/pixelgen2},
note = {Weight-space programming experiment; no parameters were trained.}
}
```
Inspired by *[Weight-Space Programming](https://rikiyat.github.io/blog/weight-space-programming/)*.