|
Download README.md from DJLougen/pixelgen2: direct link, hf CLI and curl.
- Browser
- Download file 14.5 kB
-
https://huggingface.co/DJLougen/pixelgen2/resolve/main/README.md
- Command line
-
hf download hf://DJLougen/pixelgen2/README.md
-
curl -L -o README.md https://huggingface.co/DJLougen/pixelgen2/resolve/main/README.md
14.5 kB
| license: mit | |
| library_name: pytorch | |
| pipeline_tag: text-to-image | |
| tags: | |
| - pixel-art | |
| - weight-space-programming | |
| - hand-written-weights | |
| - no-training | |
| - tiny-model | |
| - pico-8 | |
| - toy | |
| language: | |
| - en | |
| <div align="center"> | |
| # 🕹️ PixelGen2 | |
| **A text-to-pixel-art model with 6,747 parameters. Zero of them were trained.** | |
| Every weight was typed in by hand. It draws 21,866 distinct sprites from 28 templates. | |
| <img src="showcase2.png" alt="12 bodies × 8 accessories, two colours each, generated by PixelGen2" width="100%"/> | |
| <sub>Every sprite above is one prompt: <code>"<colour> <body> with <colour> <accessory>"</code>. No gradients were harmed.</sub> | |
| <br/> | |
|  | |
|  | |
|  | |
|  | |
|  | |
| </div> | |
| --- | |
| ## What this is | |
| A tiny PyTorch `nn.Module` that turns a prompt like `"red potion"` or `"purple ghost with gold crown"` into a 16×16 pixel-art sprite in the PICO-8 palette, upscaled to 128×128. | |
| It's an experiment in **weight-space programming**, inspired by [this post](https://rikiyat.github.io/blog/weight-space-programming/): instead of training a network, you write a program and *compile it into weights*. Linear layers become lookup tables, argmax becomes a decision, a 1×1 conv becomes a palette. The model is a real `nn.Module` — you can call `.forward()`, save a `state_dict`, load it back — but no optimizer ever touched it. | |
| There are two models in the repo: | |
| | | `PixelGen` (v1) | `PixelGen2` (v2) | | |
| |---|---|---| | |
| | prompt | `"<colour> <body>"` | `"[small] <colour> <body> [with <colour> <accessory>]"` | | |
| | parameters | 4,227 | 6,747 | | |
| | hand-drawn templates | 21 bodies | 21 bodies + 7 accessories | | |
| | distinct sprites (seed 0) | 189 | **21,866** | | |
| | shading | fixed row bands | structural rim light (two 3×3 convs) | | |
| v1 is the minimal version, published separately as [DJLougen/pixelgen](https://huggingface.co/DJLougen/pixelgen); v2 is the same thing plus four compositional layers, and is what the CLI uses by default. This repo contains both. Everything below was measured, not estimated — scripts in `experiments/`. | |
| ## How it works | |
| Nine steps for v1. Each is an ordinary tensor op; together they're a program. | |
| ``` | |
| "red potion" | |
| │ | |
| ▼ | |
| 1. bag of words prompt → count vector over a 138-word vocab | |
| 2. class WTA Linear(138→21) → argmax → one-hot body [potion] | |
| 3. shape bank one-hot selects a 16×8 half-template; | |
| '?' cells flip on when seed-noise ≥ 0.5 | |
| 4. mirror half + horizontal flip → 16×16 body (always symmetric) | |
| 5. outline 3×3 dilate(body) − body → outline pixels (palette index 1) | |
| 6. shading rows 0–6 light, 7–10 mid, 11–15 dark | |
| 7. ramp WTA Linear(138→9) → argmax → colour ramp [red] | |
| 8. one-hot palette every pixel → one-hot over 16 PICO-8 colours | |
| 9. palette conv Conv2d(16→3, 1×1) → RGB | |
| │ | |
| ▼ | |
| 16×16 sprite → nearest-neighbour ×8 → 128×128 PNG | |
| ``` | |
| v2 inserts four more fixed-weight layers (`pixelgen/compose.py`): | |
| | layer | mechanism | params | | |
| |---|---|---| | |
| | **accessory** | `Linear(138→8)` WTA selects a 16×8 part (crown, halo, horns, helmet, antenna, wings, torch) overlaid on the body with `max`. `none` wins ties. | 1,112 | | |
| | **second colour** | the prompt is split at `with`/`wearing`/`holding`/`and`; a second `Linear(138→9)` colours the accessory from the words after it. No colour word there → body colour wins via a 0.5 one-hot bonus. | 1,251 | | |
| | **structural shading** | two 3×3 convs with a single tap each: a pixel is *light* if its up-left neighbour is empty, *dark* if its down-right neighbour is. Light from the top-left, for every shape, for 18 parameters. | 18 | | |
| | **size** | `small`/`tiny`/`mini` → nearest-neighbour 0.75× resample of the body map, bottom-centred. | 139 | | |
| The "generative" part is step 3: templates have `?` cells gated by per-seed noise, so the same prompt gives small variations across seeds: | |
| <div align="center"> | |
| <img src="seeds.png" alt="blue invader, seeds 0–7" width="80%"/> | |
| <sub><code>"blue invader"</code>, seeds 0 through 7</sub> | |
| </div> | |
| ## Numbers | |
| | | v1 | v2 | | |
| |---|---|---| | |
| | Registered parameters | 4,227 | **6,747** | | |
| | Non-trainable buffers (template / shade / ramp / part banks) | 6,171 | 7,333 | | |
| | Total values | 10,398 (≈41.6 KB fp32) | 14,080 (≈56.3 KB fp32) | | |
| | Training steps | **0** | **0** | | |
| | Optimizers, backward passes, `requires_grad=True` in the repo | 0 (grep-verified) | 0 | | |
| | Bodies / accessories / colour ramps | 21 / – / 9 | 21 / 7 / 9 | | |
| | Distinct images at seed 0 | 189 of 189 prompts | 21,866 of 24,192 prompts | | |
| | Output | 16×16 → 128×128 RGB, every pixel ∈ PICO-8 | same | | |
| | `weights/pixelgen.pt` (v1) | 45.7 KB, SHA-256 `4c5dffd1…540eb`, identical across rebuilds | – | | |
| <details> | |
| <summary>Parameter breakdown (v2; v1 is the first six rows)</summary> | |
| | Tensor | Shape | Count | What it is | | |
| |---|---|---|---| | |
| | `word_to_class.weight` | 21 × 138 | 2,898 | vocab → body scores (synonym table) | | |
| | `word_to_class.bias` | 21 | 21 | default body (potion) | | |
| | `word_to_ramp.weight` | 9 × 138 | 1,242 | vocab → colour-ramp scores | | |
| | `word_to_ramp.bias` | 9 | 9 | default ramp (green) | | |
| | `outline.weight` | 1×1×3×3 | 9 | dilation kernel | | |
| | `palette_conv.weight` | 3×16×1×1 | 48 | one-hot colour → RGB (exact PICO-8 ÷ 255) | | |
| | `word_to_acc.weight` / `.bias` | 8 × 138, 8 | 1,112 | vocab → accessory scores, `none` biased | | |
| | `word_to_ramp2.weight` / `.bias` | 9 × 138, 9 | 1,251 | accessory-half vocab → ramp | | |
| | `lit.weight`, `shadow.weight` | 1×1×3×3 ×2 | 18 | up-left tap, down-right tap | | |
| | `word_to_size.weight` / `.bias` | 1 × 138, 1 | 139 | size words → small flag | | |
| | **Parameters** | | **6,747** | | | |
| | `on_bank` / `free_bank` | 21×16×8 ×2 | 5,376 | body templates (`#` solid, `?` noise-controlled) | | |
| | `acc_bank` | 8×16×8 | 1,024 | accessory templates | | |
| | `shade_bank` | 3×16×16 | 768 | row-band base shading | | |
| | `ramp_table` | 9×3 | 27 | palette indices per ramp | | |
| | `acc_mask_a` | 138 | 138 | body-class words never mean a hat in the body half | | |
| | **Buffers** | | **7,333** | | | |
| </details> | |
| ## How to use | |
| ```bash | |
| git clone https://huggingface.co/DJLougen/pixelgen2 && cd pixelgen2 | |
| pip install torch numpy pillow # that's the whole dependency list | |
| python generate.py "red potion" --out out/red_potion.png | |
| python generate.py "green slime with gold crown" --seed 3 --scale 16 | |
| python generate.py "small angel skull" --out out/skull.png | |
| python generate.py "blue invader" --v1 # original single-part model | |
| ``` | |
| Or from Python: | |
| ```python | |
| from pixelgen.compose import PixelGen2 | |
| model = PixelGen2() # weights are built in __init__, no file needed | |
| img = model.sprite("purple ghost with horns", seed=4, scale=8) # PIL.Image, 128×128 | |
| img.save("ghost.png") | |
| ``` | |
| **Prompt grammar.** `[small] [colour] <body> [with [colour] <accessory>]`. Words are matched by a bag-of-words, so order doesn't matter within each half. | |
| <details> | |
| <summary>Vocabulary</summary> | |
| **Bodies:** potion (bottle, flask, elixir, vial) · sword (blade, dagger) · shield (buckler) · ship (rocket, spaceship, starship) · invader (alien, monster) · slime (blob, jelly) · skull (bone, skeleton) · tree (pine, forest) · key · heart (love, life) · star (sparkle) · mushroom (fungus, toadstool, shroom) · ghost (spirit, phantom, spooky) · bomb (grenade, explosive) · coin (token, money, medal) · gem (diamond, jewel, crystal) · cherry (cherries, berry, fruit) · crown · bat (vampire) · cactus (desert, succulent) · robot (droid, android, mech) | |
| **Accessories:** crown (king, queen, royal, crowned, prince, princess) · halo (angel, angelic, holy, saint) · horns (horned, devil, demon, evil) · helmet (knight, armor, armour, armored) · antenna (antennae, bug, martian) · wings (winged, fly, flying, bird, fairy, butterfly) · torch (torches, fire, flame, burning) | |
| **Colours:** red (crimson, ruby) · blue (azure, sapphire) · green (emerald, lime) · yellow (gold, golden) · purple (violet, amethyst) · orange (amber) · pink (rose) · brown (wood, wooden) · white (grey, gray, silver, bone) | |
| **Size:** small, tiny, mini, little. **Separators:** with, wearing, holding, and, plus. | |
| `"crown"` in the body half is the crown object; after a separator it's a hat. `"bone"` is both a skull synonym and a white synonym, so `"bone"` alone is a white skull. `"alien"` is an invader body and `"martian"` is an antenna accessory, so `"martian alien"` is what you'd hope. | |
| </details> | |
| ## How far can you push it? | |
| Four experiments, all in `experiments/` with the scripts that produced them. Full notes in [`experiments/README.md`](experiments/README.md). | |
| **1. Adding a class is a pure append.** Freeze the 11-class model, add `mushroom`, diff. +400 values. Diffed by *index*, 84 old values changed; diffed by *meaning* (class name × word), **zero** — the raw diff was entirely a column permutation from a `sorted()` vocab. 17/17 old prompts bit-identical over 3 seeds. The same check still passes after +10 classes and the whole accessory vocabulary. | |
| **2. Capacity is unbounded; addressing is quadratic.** Synthetic random classes, rebuilt and probed at each size: | |
| | classes | params | dense layer share | isolated | | |
| |---|---|---|---| | |
| | 21 | 2,937 | 22% | ✓ | | |
| | 121 | 51,537 | 57% | ✓ | | |
| | 1,021 | 3.2M | 92% | ✓ | | |
| | 10,021 | 302M | 99.1% | ✓ | | |
| It never breaks — the original 21 classes stay bit-identical at 10k. But vocab grows with classes, so `word_to_class` is ≈3C² and 99.997% zeros. Under 100k params the dense design tops out near **170 classes**. | |
| **3. Compressing the class layer costs semantics.** A binary code (`d = ⌈log₂C⌉`, words emit ±1 codes, score = `d − 2·hamming`) shrinks the class layer **528×** at 10k classes with no loss of isolation — but 61% of two-body-word prompts (`"sword shield"`) now resolve to a *third* class, vs 0% dense. The additive bag-of-words semantics were the thing that got compressed away. | |
| **4. Composition is the lever.** v1 → v2: 1.6× params, **115× distinct outputs**. 781 distinct images per hand-drawn template instead of 9. | |
| <div align="center"> | |
| <img src="experiments/shading_before_after.png" alt="row-band shading vs structural rim lighting" width="100%"/> | |
| <sub>Top: v1 row bands. Bottom: v2 structural lighting, 18 params.</sub> | |
| </div> | |
| Quality, honestly: structural lighting helps blobby shapes (potion, heart, mushroom, tree, gem) and **hurts thin ones** (invader, bat, robot) — when every pixel is an edge, rim light becomes speckle. A fixed 3×3 kernel has no way to know a shape is too thin to light. That's the limit of hand-written rules without a learned prior, and exactly the kind of thing a few hundred trained parameters would fix. | |
| ## Limitations | |
| - **It only knows what was drawn.** 21 bodies, 7 accessories. Prompt for a dragon and you get a potion (the default). Capacity is unbounded; *content* is a human with a text editor. | |
| - **Every shape is horizontally symmetric.** The mirror step makes 16×8 templates cheap; it also means `"torch"` is two torches and the key is a top-down key. Lighting (v2) is deliberately *not* symmetric. | |
| - **Accessories sit at fixed rows.** A crown floats above short bodies; a helmet merges into tall ones. Fixing that needs per-body anchor points — more drawing, not more mechanism. | |
| - **One accessory per sprite.** `"devil knight"` picks one by index order. | |
| - **Variation is cosmetic.** Seeds only toggle the `?` cells — a few pixels per sprite. | |
| - **Bag of words.** `"sword shield"` is a tie broken by index; `"not red"` is red; within each half, order is ignored. | |
| - **Structural shading speckles thin shapes.** See above. | |
| - **It's PyTorch for the bit, not for the need.** You could rewrite this in NumPy. Using `nn.Linear` and `nn.Conv2d` is the point of the exercise. | |
| - **Art quality is programmer art**, drawn in a text editor by someone who is not a pixel artist. | |
| ## Why bother? | |
| Because it's a nice demonstration that a lot of what we call "a model" is just *structure*, and structure can be written directly. A linear layer with the right weights *is* a dictionary. A conv with the right kernel *is* an edge detector. Winner-take-all *is* an `if`. The blog post argues this is a useful lens for understanding what trained networks are doing; this repo is the version where you skip the training and just write the weights — and then push on it to see where hand-writing stops being enough. | |
| Where it stops: addressing (quadratic), compression (breaks additivity), and quality rules at the edges (shading thin shapes). The natural next step is the hybrid — hand-write the ~95% that's structure, train only the small word→part map. | |
| Also it's cool that it worked. | |
| ## Repo contents | |
| ``` | |
| pixelgen/ | |
| model.py PixelGen (v1) — all weights set in __init__ | |
| compose.py PixelGen2 (v2) — v1 + accessory, second colour, structural shading, size | |
| templates.py 21 body half-templates ('#' solid, '?' noise, '.' empty) | |
| parts.py 7 accessory half-templates | |
| palette.py PICO-8 palette + 9 colour ramps | |
| tokenizer.py vocab, synonyms, prompt splitting | |
| weights.py save/load helpers | |
| build.py writes weights/pixelgen.pt (v1, deterministic) | |
| generate.py CLI (v2 default, --v1 for the original) | |
| tests/ 16 property tests | |
| experiments/ the four experiments above: scripts, frozen baselines, README | |
| showcase2.png the grid at the top | |
| seeds.png seed variation strip | |
| ``` | |
| Tests: `python -m pytest tests -q` → 16 passed. | |
| ## Citation | |
| ```bibtex | |
| @misc{pixelgen2_2025, | |
| title = {PixelGen2: a compositional hand-weighted pixel-art generator}, | |
| author = {Lougen, D. J.}, | |
| year = {2025}, | |
| url = {https://huggingface.co/DJLougen/pixelgen2}, | |
| note = {Weight-space programming experiment; no parameters were trained.} | |
| } | |
| ``` | |
| Inspired by *[Weight-Space Programming](https://rikiyat.github.io/blog/weight-space-programming/)*. | |