Pixelquill-61M
Text β 16Γ16 pixel-art sprite. 61,367,168 parameters, trained from scratch, no VAE and no diffusion.
Every sprite above was generated by this model from the caption alone, seed 0, one sample each. No cherry-picking, no post-editing.
Pixelquill predicts a sprite as 289 discrete tokens: 32 palette colour IDs, then 256 slot
IDs (0 = transparent, 1..32 = index into that sprite's palette). Because the palette comes
first, the model commits to "I will use this red" before deciding which pixels are red. The
output is exact RGBA palette indices, so there is no decoder to drift and no sampling from a
continuous latent.
Results
Held-out subjects only β 500 subjects / 113,550 sprites with zero subject overlap against training. The dataset's own official split is 100% leaky (all 2,598 validation subjects also appear in train), so it is not used anywhere in these numbers.
| Metric | Value | Random baseline |
|---|---|---|
| Pixel accuracy (teacher-forced) | 80.65% | 3.0% |
| Slot cross-entropy | 0.615 | 3.50 |
| Palette cross-entropy | 4.123 | 8.32 |
| Palette RGB error, mean | 115.9 / 441 | β |
| Free-running pixel agreement | 31.6% | β |
| Exact sprites (all 256 px correct) | 0.00% | β |
Teacher-forced accuracy is an upper bound: the model sees the true prefix. 31.6% free-running is the number that matters β whole sprites generated from the caption alone, no ground truth ever shown. This is a capable prototype, not a shippable asset generator.
Independent verification
Measured on an NVIDIA L4 against a freshly downloaded copy of the weights, compared directly with real sprites from the dataset. These checks exist because pixel accuracy alone can look good while the model ignores the prompt.
Colour words bind to actual pixels. For 24 dataset captions beginning with a colour word, the generated sprite's dominant colour was compared with the real sprite's colour:
- 23 / 24 (96%) within 40Β° of hue, median error 4Β°
red violin real (202, 38, 33) -> generated (201, 36, 38) 3Β°
yellow piccolo real (250,216, 79) -> generated (246,212, 80) 0Β°
blue mug real ( 56,121,205) -> generated ( 46,118,195) 3Β°
purple estoc real (158, 98,222) -> generated (216,170,240) 10Β°
Holding the object fixed and swapping only the colour word, 15/16 landed on the requested colour, and the four variants of each object spanned 165β169Β° of hue β the colours are genuinely different, not near-copies. The silhouette stayed put while colour changed (mask IoU: slime 0.94, potion 0.86, hat 0.80, sword 0.46).
Same caption, real vs generated (12 dataset captions):
| Real dataset | Generated | |
|---|---|---|
| Opaque fill | 60.9% | 60.8% |
| Colours per sprite | 24.8 | 21.9 |
| Hue agreement | β | median 4Β°, 83% within 40Β° |
It is not retrieving memorised sprites. Every generated sprite was checked against all 1,405,308 training sprites:
| Check | Result |
|---|---|
| Exact duplicates | 0 / 32 |
| Pixels differing from nearest training sprite | min 53, median 135 of 256 |
The nearest neighbours are also semantically unrelated β reef shark is closest to
"folded red square eyeglasses", not to a shark β which is what a model that generates
rather than looks up should do.
Quick start
Nothing to download by hand β the model fetches itself from the Hub.
pip install huggingface_hub torch safetensors numpy pillow
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("MahdiAlikhah/pixelquill-61m", local_dir="pixelquill")
sys.path.insert(0, path)
from pixelquill import Pixelquill
model = Pixelquill.from_pretrained(path) # CUDA if available
img = model.generate_one("cute teal slime", scale=8) # 16x16 -> 128x128 RGBA
img.save("slime.png")
print(model) # Pixelquill(61,367,168 params, device='cuda', vocab=8643)
Pixelquill.from_pretrained() with no argument also works β it resolves the repo
id itself through the Hub cache:
import sys
from huggingface_hub import snapshot_download
sys.path.insert(0, snapshot_download("MahdiAlikhah/pixelquill-61m"))
from pixelquill import Pixelquill
model = Pixelquill.from_pretrained()
Several prompts in one batched call, plus the raw tokens:
prompts = ["reef shark", "rusty sword", "green potion"]
for prompt, im in zip(prompts, model.generate(prompts, seed=0)):
im.save(prompt.replace(" ", "_") + ".png")
pal, slot = model.tokens(["orb wand"]) # (1, 32) palette ids, (1, 256) slot ids
Straight from the command line:
hf download MahdiAlikhah/pixelquill-61m --local-dir pixelquill
python pixelquill/sample.py "cute teal slime" "reef shark" --repo pixelquill --out sheet.png
Pixelquill.from_pretrained() accepts device, dtype, and local_files_only
(offline, cache only). generate() accepts cfg (guidance scale, default 3.0),
temperature, seed, and scale. Guidance matters: without it, sprites
under-commit to the prompt.
Because the model is a single word-level vocabulary and a 289-token
autoregressive decoder rather than a transformers architecture, it is loaded by
its own small loader (pixelquill.py) instead of AutoModel.
Files
| File | Purpose |
|---|---|
model.safetensors |
weights, fp32, 245.5 MB |
pixelquill.py |
loader + high-level API: Pixelquill.from_pretrained(repo_id) |
model.py |
architecture and load() β no dependency beyond torch |
sample.py |
CLI: prompt β PNG |
config.json |
all architecture and training constants |
word2id.json |
8,643-word closed vocabulary + 4 special tokens |
palette.npy |
(4097, 3) uint8 RGB table, row 0 = transparent |
LICENSE |
Apache-2.0 |
Architecture
| Component | Params |
|---|---|
| Word-level text encoder (3 layers, d=384, learned positions) | 8,675,712 |
| Sprite decoder (12 layers, d=512, cross-attention, 2D axial RoPE, SwiGLU, RMSNorm) | 52,691,456 |
| Total | 61,367,168 |
The tokenizer is a word-level table, not BPE: the vocabulary is closed and only 8,643 words wide, so the entire tokenizer is one embedding matrix.
Why there is no pretrained text tower. The obvious choice, reusing
openai/clip-vit-base-patch32's text encoder, is 63.2M params on its own β which would put the
total at 115.9M and blow the sub-100M budget. A frozen MiniLM tower (22.7M) would also fit
only by shrinking the decoder, and a frozen, dataset-mismatched encoder forces the decoder to
undo the encoder's biases. A small trained-from-scratch encoder on a closed 8.6k vocabulary is
both cheaper and better matched to these short, formulaic captions.
Sequence layout, 289 positions:
0 pal_start
1..32 palette colour IDs (32 tokens)
33 BOS
34..288 pixel slots (256 tokens, raster order)
pal_start and the early palette positions exist so the model has committed to its entire
palette before the first pixel is predicted.
Limitations
- 16Γ16 only. 256 pixels is the whole canvas; there is no upscaling path.
- Object relations fail.
"a cat wearing a wizard hat"produces one merged blob. Two nested objects do not fit in 256 pixels. - Thin objects are inconsistent. Sword silhouettes agree only 0.46 IoU across colour variants, versus 0.94 for blobs. The model is confident about slimes and unsure about blades.
- High sampling variance: 28% pixel agreement across 3 seeds. The same prompt gives noticeably different sprites run to run. Re-roll before concluding a concept fails.
- 13.3% of sprites need more than 32 colours (dataset max is 54). Those lose detail to nearest-colour merging β 0.34% of all pixels affected.
- No exact matches on held-out data. Fine detail and precise hue are the dominant errors.
- Rare and out-of-distribution words degrade sharply. The vocabulary is closed at 8,643
words; anything outside it becomes
<unk>. - Not fast. Sampling uses no KV cache: 289 full trunk passes per sprite, ~2.8 s/sprite on an L4 (a batch of 32 in ~90 s). A cache would cut this by an order of magnitude.
Training
| Data | Angais/pixel-art-atlas-1.4m (CC0-1.0) |
| Split | subject-level: 5,194 train / 500 dev / 500 test subjects, 0% overlap |
| Steps | 24,500 (10.6 epochs), batch 512 |
| Optimiser | AdamW (0.9, 0.95), lr 6e-4 cosine with 1,000-step warmup, floor 0.05, wd 0.05 |
| Precision | bf16 autocast, fp32 master weights |
| Hardware | one RTX PRO 6000 Blackwell, ~5.5 h |
| Seed | 1234 |
Convergence was reached at epoch 8 (dev pixel accuracy 80.1%); epochs 9β10 added +0.3% while palette cross-entropy regressed, so training was stopped at step 24,500.
Licence
Apache-2.0. The training dataset is CC0-1.0, so no licence obligations are inherited from it.
- Downloads last month
- 67