pixelgen / README.md
DJLougen's picture
card: drop the obvious disclaimer
18458f1 verified
|
Raw History Blame Contribute Delete
9.22 kB
---
license: mit
library_name: pytorch
pipeline_tag: text-to-image
tags:
- pixel-art
- weight-space-programming
- hand-written-weights
- no-training
- tiny-model
- pico-8
- toy
language:
- en
---
<div align="center">
# πŸ•ΉοΈ PixelGen
**A text-to-pixel-art model with 1,257 parameters. Zero of them were trained.**
Every weight was typed in by hand. It still works.
<img src="showcase.png" alt="11 sprite classes Γ— 9 colour ramps, all generated by PixelGen" width="100%"/>
<sub>Every sprite above: `"<colour> <class>"` β†’ PixelGen β†’ 16Γ—16 PICO-8 sprite. No gradients were harmed.</sub>
<br/>
![params](https://img.shields.io/badge/parameters-1%2C257-ff004d?style=flat-square)
![weights](https://img.shields.io/badge/weights-23.5%20KB-29adff?style=flat-square)
![training](https://img.shields.io/badge/training%20steps-0-00e436?style=flat-square)
![palette](https://img.shields.io/badge/palette-PICO--8-ffec27?style=flat-square)
![license](https://img.shields.io/badge/license-MIT-83769c?style=flat-square)
</div>
---
## What this is
A tiny PyTorch `nn.Module` that turns a text prompt like `"red potion"` or `"blue invader"` into a 16Γ—16 pixel-art sprite in the PICO-8 palette, upscaled to 128Γ—128.
It's an experiment in **weight-space programming**, inspired by [this post](https://rikiyat.github.io/blog/weight-space-programming/): instead of training a network, you write a program and *compile it into weights*. Linear layers become lookup tables, argmax becomes a decision, a 1Γ—1 conv becomes a palette. The model is a real `nn.Module` β€” you can call `.forward()`, save a `state_dict`, load it back β€” but no optimizer ever touched it.
The fun part is that an image generator with the parameter count of a single small MLP layer produces consistently cute, correctly-coloured, correctly-shaped sprites, and you can read the entire "model" as source code.
## Numbers (measured, not estimated)
| | |
|---|---|
| Registered parameters | **1,257** |
| Non-trainable buffers (shape/shade/ramp banks) | 3,611 |
| Total values in the checkpoint | 4,868 (β‰ˆ19.5 KB fp32) |
| `pixelgen.pt` on disk | 23.5 KB |
| Training steps | **0** |
| Optimizers, backward passes, `requires_grad=True` in the repo | 0 (grep-verified) |
| Sprite classes | 11 |
| Colour ramps | 9 |
| Output | 16Γ—16 β†’ 128Γ—128 RGB, every pixel ∈ PICO-8 (16 colours) |
| Build reproducibility | `build.py` twice β†’ identical SHA-256 |
| `pixelgen.pt` SHA-256 | `175cc5249fba8c756ea019f418f4aa0a9633017ffad7f238b863865758c49b80` |
<details>
<summary>Parameter breakdown</summary>
| Tensor | Shape | Count | What it is |
|---|---|---|---|
| `word_to_class.weight` | 11 Γ— 59 | 649 | vocab β†’ class scores (hand-set synonym table) |
| `word_to_class.bias` | 11 | 11 | default class bias (potion) |
| `word_to_ramp.weight` | 9 Γ— 59 | 531 | vocab β†’ colour-ramp scores |
| `word_to_ramp.bias` | 9 | 9 | default ramp bias (green) |
| `outline.weight` | 1Γ—1Γ—3Γ—3 | 9 | 3Γ—3 dilation kernel |
| `palette_conv.weight` | 3Γ—16Γ—1Γ—1 | 48 | one-hot colour β†’ RGB (exact PICO-8 values Γ· 255) |
| **Parameters** | | **1,257** | |
| `on_bank` / `free_bank` | 11Γ—16Γ—8 Γ—2 | 2,816 | half-sprite templates (`#` solid, `?` noise-controlled) |
| `shade_bank` | 3Γ—16Γ—16 | 768 | light / mid / dark row masks |
| `ramp_table` | 9Γ—3 | 27 | palette indices per ramp |
| **Buffers** | | **3,611** | |
</details>
## How it works
The forward pass is nine steps. Each one is a normal tensor op; together they're a program.
```
"red potion"
β”‚
β–Ό
1. bag of words prompt β†’ count vector over a 59-word vocab
2. class WTA Linear(59β†’11) β†’ argmax β†’ one-hot class [potion]
3. shape bank one-hot selects a 16Γ—8 half-template;
'?' cells flip on when seed-noise β‰₯ 0.5
4. mirror half + horizontal flip β†’ 16Γ—16 body (always symmetric)
5. outline 3Γ—3 dilate(body) βˆ’ body β†’ outline pixels (palette index 1)
6. shading rows 0–6 light, 7–10 mid, 11–15 dark
7. ramp WTA Linear(59β†’9) β†’ argmax β†’ colour ramp [red]
8. one-hot palette every pixel β†’ one-hot over 16 PICO-8 colours
9. palette conv Conv2d(16β†’3, 1Γ—1) β†’ RGB
β”‚
β–Ό
16Γ—16 sprite β†’ nearest-neighbour Γ—8 β†’ 128Γ—128 PNG
```
The "generative" part is step 3: templates have `?` cells that are gated by per-seed noise, so the same prompt gives small variations across seeds:
<div align="center">
<img src="seeds.png" alt="blue invader, seeds 0–7" width="80%"/>
<sub><code>"blue invader"</code>, seeds 0 through 7</sub>
</div>
## How to use
```bash
git clone https://huggingface.co/DJLougen/pixelgen && cd pixelgen
pip install torch numpy pillow # that's the whole dependency list
python generate.py "red potion" --seed 0 --out out/red_potion.png
python generate.py "golden key" --seed 3 --scale 16
```
Or from Python:
```python
from pixelgen.model import PixelGen
model = PixelGen() # weights are built in __init__, no file needed
img = model.sprite("purple slime", seed=4, scale=8) # PIL.Image, 128Γ—128
img.save("slime.png")
```
The `pixelgen.pt` checkpoint is included for completeness (`pixelgen.weights.load(path)`), but it's redundant β€” `PixelGen()` reconstructs identical weights from source every time.
**Prompt vocabulary.** Class words (and synonyms):
| class | words |
|---|---|
| potion | potion, bottle, flask, elixir, vial |
| sword | sword, blade, dagger |
| shield | shield, buckler |
| ship | ship, rocket, spaceship, starship |
| invader | invader, alien, monster |
| slime | slime, blob, jelly |
| skull | skull, bone, skeleton |
| tree | tree, pine, forest |
| key | key |
| heart | heart, love, life |
| star | star, sparkle |
Colour words: `red blue green yellow purple orange pink brown white`, plus synonyms (`crimson ruby β†’ red`, `azure sapphire β†’ blue`, `emerald lime β†’ green`, `gold golden β†’ yellow`, `violet amethyst β†’ purple`, `amber β†’ orange`, `rose β†’ pink`, `wood wooden β†’ brown`, `grey gray silver bone β†’ white`). Anything else is ignored. Fun side effect: `"bone"` is both a skull synonym and a white synonym, so `"bone"` alone gives a white skull.
## Limitations
Being upfront, because this is the part that matters:
- **It only knows 11 things.** Prompt for a dragon and you get a potion (the default class). There is no generalisation β€” zero β€” because there is no learned representation. The "model" is a lookup table with a tokenizer in front of it.
- **Every sprite is horizontally symmetric.** The mirror step is what makes 16Γ—8 templates cheap, and it's also why the key had to be drawn as a top-down key rather than a side-view one.
- **Variation is cosmetic.** Seeds only toggle the hand-marked `?` cells (a few pixels per sprite). Two seeds of the same prompt are the same sprite with slightly different speckles.
- **Shading is positional, not structural.** Light/mid/dark bands are fixed rows, so a tall sprite and a short sprite get lit differently for no good reason.
- **Multi-word prompts resolve by word count, not meaning.** `"sword shield"` is a tie broken by index order. `"not red"` is red.
- **Colour choice is global.** One ramp per sprite; no two-tone sprites.
- **It's PyTorch for the bit, not for the need.** You could rewrite this in 100 lines of NumPy. Using `nn.Linear` and `nn.Conv2d` is the point of the exercise, not a requirement.
- **Art quality is "programmer art".** The templates were hand-drawn in a text editor by someone who is not a pixel artist. `invader` and `key` went through a few iterations; the rest are first drafts.
## Why bother?
Because it's a nice demonstration that a lot of what we call "a model" is just *structure*, and structure can be written directly. A linear layer with the right weights *is* a dictionary. A conv with the right kernel *is* an edge detector. Winner-take-all *is* an `if`. The blog post argues this is a useful lens for understanding what trained networks are doing; this repo is the version where you skip the training and just write the weights.
Also it's cool that it worked.
## Repo contents
```
pixelgen/
model.py PixelGen nn.Module β€” all weights set in __init__
templates.py 11 half-sprite templates ('#' solid, '?' noise, '.' empty)
palette.py PICO-8 palette + 9 colour ramps
tokenizer.py vocab + synonyms
weights.py save/load helpers
build.py writes weights/pixelgen.pt (deterministic)
generate.py CLI
tests/ 10 property tests (shape, palette membership, determinism, symmetry, …)
weights/pixelgen.pt
showcase.png the grid at the top
seeds.png seed variation strip
```
Tests: `python -m pytest tests -q` β†’ 10 passed.
## Citation
```bibtex
@misc{pixelgen2025,
title = {PixelGen: a hand-weighted pixel-art generator},
author = {Lougen, D. J.},
year = {2025},
url = {https://huggingface.co/DJLougen/pixelgen},
note = {Weight-space programming experiment; no parameters were trained.}
}
```
Inspired by *[Weight-Space Programming](https://rikiyat.github.io/blog/weight-space-programming/)*.