Pixelquill-61M

Text β†’ 16Γ—16 pixel-art sprite. 61,367,168 parameters, trained from scratch, no VAE and no diffusion.

Sixteen sprites generated from text prompts

Every sprite above was generated by this model from the caption alone, seed 0, one sample each. No cherry-picking, no post-editing.

Pixelquill predicts a sprite as 289 discrete tokens: 32 palette colour IDs, then 256 slot IDs (0 = transparent, 1..32 = index into that sprite's palette). Because the palette comes first, the model commits to "I will use this red" before deciding which pixels are red. The output is exact RGBA palette indices, so there is no decoder to drift and no sampling from a continuous latent.


Results

Held-out subjects only β€” 500 subjects / 113,550 sprites with zero subject overlap against training. The dataset's own official split is 100% leaky (all 2,598 validation subjects also appear in train), so it is not used anywhere in these numbers.

Metric Value Random baseline
Pixel accuracy (teacher-forced) 80.65% 3.0%
Slot cross-entropy 0.615 3.50
Palette cross-entropy 4.123 8.32
Palette RGB error, mean 115.9 / 441 β€”
Free-running pixel agreement 31.6% β€”
Exact sprites (all 256 px correct) 0.00% β€”

Teacher-forced accuracy is an upper bound: the model sees the true prefix. 31.6% free-running is the number that matters β€” whole sprites generated from the caption alone, no ground truth ever shown. This is a capable prototype, not a shippable asset generator.


Independent verification

Measured on an NVIDIA L4 against a freshly downloaded copy of the weights, compared directly with real sprites from the dataset. These checks exist because pixel accuracy alone can look good while the model ignores the prompt.

Colour words bind to actual pixels. For 24 dataset captions beginning with a colour word, the generated sprite's dominant colour was compared with the real sprite's colour:

  • 23 / 24 (96%) within 40Β° of hue, median error 4Β°
red violin      real (202, 38, 33)  ->  generated (201, 36, 38)     3Β°
yellow piccolo  real (250,216, 79)  ->  generated (246,212, 80)     0Β°
blue mug        real ( 56,121,205)  ->  generated ( 46,118,195)     3Β°
purple estoc    real (158, 98,222)  ->  generated (216,170,240)    10Β°

Holding the object fixed and swapping only the colour word, 15/16 landed on the requested colour, and the four variants of each object spanned 165–169Β° of hue β€” the colours are genuinely different, not near-copies. The silhouette stayed put while colour changed (mask IoU: slime 0.94, potion 0.86, hat 0.80, sword 0.46).

Same caption, real vs generated (12 dataset captions):

Real dataset Generated
Opaque fill 60.9% 60.8%
Colours per sprite 24.8 21.9
Hue agreement β€” median 4Β°, 83% within 40Β°

It is not retrieving memorised sprites. Every generated sprite was checked against all 1,405,308 training sprites:

Check Result
Exact duplicates 0 / 32
Pixels differing from nearest training sprite min 53, median 135 of 256

The nearest neighbours are also semantically unrelated β€” reef shark is closest to "folded red square eyeglasses", not to a shark β€” which is what a model that generates rather than looks up should do.


Quick start

Nothing to download by hand β€” the model fetches itself from the Hub.

pip install huggingface_hub torch safetensors numpy pillow
import sys
from huggingface_hub import snapshot_download

path = snapshot_download("MahdiAlikhah/pixelquill-61m", local_dir="pixelquill")
sys.path.insert(0, path)
from pixelquill import Pixelquill

model = Pixelquill.from_pretrained(path)              # CUDA if available
img = model.generate_one("cute teal slime", scale=8)  # 16x16 -> 128x128 RGBA
img.save("slime.png")
print(model)   # Pixelquill(61,367,168 params, device='cuda', vocab=8643)

Pixelquill.from_pretrained() with no argument also works β€” it resolves the repo id itself through the Hub cache:

import sys

from huggingface_hub import snapshot_download

sys.path.insert(0, snapshot_download("MahdiAlikhah/pixelquill-61m"))
from pixelquill import Pixelquill

model = Pixelquill.from_pretrained()

Several prompts in one batched call, plus the raw tokens:

prompts = ["reef shark", "rusty sword", "green potion"]
for prompt, im in zip(prompts, model.generate(prompts, seed=0)):
    im.save(prompt.replace(" ", "_") + ".png")

pal, slot = model.tokens(["orb wand"])   # (1, 32) palette ids, (1, 256) slot ids

Straight from the command line:

hf download MahdiAlikhah/pixelquill-61m --local-dir pixelquill
python pixelquill/sample.py "cute teal slime" "reef shark" --repo pixelquill --out sheet.png

Pixelquill.from_pretrained() accepts device, dtype, and local_files_only (offline, cache only). generate() accepts cfg (guidance scale, default 3.0), temperature, seed, and scale. Guidance matters: without it, sprites under-commit to the prompt.

Because the model is a single word-level vocabulary and a 289-token autoregressive decoder rather than a transformers architecture, it is loaded by its own small loader (pixelquill.py) instead of AutoModel.


Files

File Purpose
model.safetensors weights, fp32, 245.5 MB
pixelquill.py loader + high-level API: Pixelquill.from_pretrained(repo_id)
model.py architecture and load() β€” no dependency beyond torch
sample.py CLI: prompt β†’ PNG
config.json all architecture and training constants
word2id.json 8,643-word closed vocabulary + 4 special tokens
palette.npy (4097, 3) uint8 RGB table, row 0 = transparent
LICENSE Apache-2.0

Architecture

Component Params
Word-level text encoder (3 layers, d=384, learned positions) 8,675,712
Sprite decoder (12 layers, d=512, cross-attention, 2D axial RoPE, SwiGLU, RMSNorm) 52,691,456
Total 61,367,168

The tokenizer is a word-level table, not BPE: the vocabulary is closed and only 8,643 words wide, so the entire tokenizer is one embedding matrix.

Why there is no pretrained text tower. The obvious choice, reusing openai/clip-vit-base-patch32's text encoder, is 63.2M params on its own β€” which would put the total at 115.9M and blow the sub-100M budget. A frozen MiniLM tower (22.7M) would also fit only by shrinking the decoder, and a frozen, dataset-mismatched encoder forces the decoder to undo the encoder's biases. A small trained-from-scratch encoder on a closed 8.6k vocabulary is both cheaper and better matched to these short, formulaic captions.

Sequence layout, 289 positions:

0        pal_start
1..32    palette colour IDs          (32 tokens)
33       BOS
34..288  pixel slots                 (256 tokens, raster order)

pal_start and the early palette positions exist so the model has committed to its entire palette before the first pixel is predicted.


Limitations

  • 16Γ—16 only. 256 pixels is the whole canvas; there is no upscaling path.
  • Object relations fail. "a cat wearing a wizard hat" produces one merged blob. Two nested objects do not fit in 256 pixels.
  • Thin objects are inconsistent. Sword silhouettes agree only 0.46 IoU across colour variants, versus 0.94 for blobs. The model is confident about slimes and unsure about blades.
  • High sampling variance: 28% pixel agreement across 3 seeds. The same prompt gives noticeably different sprites run to run. Re-roll before concluding a concept fails.
  • 13.3% of sprites need more than 32 colours (dataset max is 54). Those lose detail to nearest-colour merging β€” 0.34% of all pixels affected.
  • No exact matches on held-out data. Fine detail and precise hue are the dominant errors.
  • Rare and out-of-distribution words degrade sharply. The vocabulary is closed at 8,643 words; anything outside it becomes <unk>.
  • Not fast. Sampling uses no KV cache: 289 full trunk passes per sprite, ~2.8 s/sprite on an L4 (a batch of 32 in ~90 s). A cache would cut this by an order of magnitude.

Training

Data Angais/pixel-art-atlas-1.4m (CC0-1.0)
Split subject-level: 5,194 train / 500 dev / 500 test subjects, 0% overlap
Steps 24,500 (10.6 epochs), batch 512
Optimiser AdamW (0.9, 0.95), lr 6e-4 cosine with 1,000-step warmup, floor 0.05, wd 0.05
Precision bf16 autocast, fp32 master weights
Hardware one RTX PRO 6000 Blackwell, ~5.5 h
Seed 1234

Convergence was reached at epoch 8 (dev pixel accuracy 80.1%); epochs 9–10 added +0.3% while palette cross-entropy regressed, so training was stopped at step 24,500.

Licence

Apache-2.0. The training dataset is CC0-1.0, so no licence obligations are inherited from it.

Downloads last month
67
Safetensors
Model size
61.4M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train MahdiAlikhah/pixelquill-61m