pixelsprite AnimGPT: animate a 16 / 24 / 32 px sprite
Built with Qwen. The training data (the dataset pixelsprite-animated-characters) contains outputs of Qwen-Image 2.1, which is under the Qwen Research License (non-commercial). The weights are therefore for non-commercial use only. See Licence and credits. This is not legal advice and carries no guarantee.
A small autoregressive transformer (39.2M parameters) that takes one pixel-art character sprite and draws it turned to another direction and performing an action (walk, run, punch, fly, ...), frame by frame. It is the text-to-sprite model pixelsprite-canvas-v2 taught to read a sprite instead of a caption.
Columns: 1 = input sprite; 2-4 = the sprite turned to 90 / 180 / 270 degrees; 5-8 = walk (or fly) frames 0-3 at 90 degrees. Rows come in pairs: ground truth (from the 3D farm), then the model. Held-out characters, 32 px, temperature 0.8, last training step.
What it does
| Input | a character sprite (16 or 32 px; 24 px uses a front render), its kind (human / quadruped / bird), an action, a direction |
| Output, "view" | the same character turned to one of 8 directions (45 degree steps) |
| Output, "animation" | frame k of an action at a direction, given frame k-1 (frame 0 follows the view) |
| Actions | 17 for humans, 3 for quadrupeds (idle, walk, run), 2 for birds (fly, glide) |
| Sizes | 16, 24 and 32 px, all trained |
Directions: 0 = front, 90 = profile facing left, 180 = back, 270 = profile facing right.
How it works
- Same transformer as v2: decoder-only, 12 layers, width 512, 8 heads, MLP 2048, pre-norm. It was warm-started from the v2 weights; only the condition embeddings are new (39,178,241 parameters in total).
- Per-pixel drawing: the target frame is written left to right, top to bottom. Each pixel is one of 1,024 palette colours or "transparent" (the same palette as v2,
code/palette.npy). - Sequence:
[v2 prefix: null text, size, S unknown-mask rows] [condition token] [reference S*S] [previous frame S*S] [target S*S](one network row per frame; S is the canvas size). - Condition token: the sum of embeddings for kind, action, azimuth, elevation, frame index, frame count and reference type (0 = a sprite drawn the way v2 draws, 1 = a pixelated front render of the 3D model).
- Two tasks: view (previous = reference; the character turned to an azimuth) and animation (previous = the last frame, or the view at that azimuth for frame 0).
- Palette restriction: when sampling, only the reference sprite's own colours and "transparent" are allowed. This keeps the palette of the character.
- Sampling: temperature 0.8 (the training sheets and the app use it).
Quick start
pip install torch safetensors pillow numpy huggingface_hub
hf download ihor-umanets/pixelsprite-animgpt --local-dir animgpt # the repo is private for now: hf auth login first
cd animgpt
python example.py examples/archer32.png --az 270 --out walk.gif # CPU, ~3 min at 32 px; --size 16 for a quick look
example.py (in the repo) is the whole recipe: it quantises a sprite to the palette, draws the view at the chosen direction, then 8 walk frames, each from the previous one, and saves a GIF and a PNG strip.
The core of it:
import sys, json, numpy as np, torch
from safetensors.torch import load_file
sys.path.insert(0, "code")
from anim_model import AnimGPT, ACTIONS, KINDS, AZIMUTHS
from pixelsprite_model import CanvasConfig
cfg = json.load(open("config.json"))
model = AnimGPT(CanvasConfig.from_dict(cfg["model"])).eval()
model.load_state_dict(load_file("model.safetensors"))
# ref: (1, 32*32) long tensor of palette indices of your sprite, 1024 = transparent
t = lambda v: torch.tensor([v])
cond = dict(kind=t(KINDS.index("human")), action=t(ACTIONS.index("view")), az=t(AZIMUTHS.index(270)),
el=t(0), frame=t(0), nfr=t(1), refk=t(0))
allowed = torch.zeros(1, 1025, dtype=torch.bool); allowed[0, ref[0].unique()] = True; allowed[0, 1024] = True
view = model.sample(32, cond, ref, ref, temperature=0.8, allowed=allowed) # the turned sprite
cond.update(action=t(ACTIONS.index("walk")), nfr=t(8))
frame0 = model.sample(32, cond, ref, view, temperature=0.8, allowed=allowed) # walk frame 0
A desktop app (Rust + ONNX Runtime) that generates sprites with v2 and animates them with this model is coming; it is not published yet.
Files
| File | What |
|---|---|
model.safetensors, config.json |
weights (float32, from the best checkpoint) and the architecture |
best.pt |
the same checkpoint as a PyTorch file (model, meta); no optimizer state |
code/ |
anim_model.py, pixelsprite_model.py, palette.npy (1,024 x 3 sRGB) |
example.py, examples/ |
a tested example, a 32 px input sprite and its output |
onnx/ |
the export used by the app: prefill.onnx (whole context in one call, growing cache), step.onnx (one pixel per call, fixed cache, GPU), step_grow.onnx (growing cache, CPU), anim.data (shared float32 weights), embeddings.bin / .json (input tables, actions, palette, layout). ONNX vs PyTorch: max logit difference about 2e-5, argmax identical on all pixels. |
train_log.json, samples/ |
training log, eval curves, sample sheets |
LICENSE_QWEN.txt, NOTICE.txt |
Qwen Research License and the notice for the data |
Training
- Data: pixelsprite-animated-characters: 20,744 characters (humans, quadrupeds, birds) turned into 3D, rigged, animated and pixelated back, 24.3M sprites in 8 directions at 16, 24 and 32 px. Each training example is one frame or one view of one character. Splits are by character; 463 validation characters are used for the numbers below.
- Reference: the original v2-style sprite 60% of the time when one exists at that size, otherwise a pixelated front render. Views are weighted x8 and animals x2.
- Optimisation: AdamW (betas 0.9 / 0.95, weight decay 0.05), learning rate 3e-4, 1,000 warm-up steps, then cosine decay to 10% over the time budget, gradient clipping 1.0, bf16 autocast, dropout 0.05. About 98k tokens per batch, one canvas size per step (16 / 24 / 32 with probabilities 0.25 / 0.25 / 0.5). Loss on the target pixels only.
- Compute: 600 minutes on one NVIDIA RTX PRO 6000, 143,992 steps. The released weights are the best evaluation (step 136,655, minute 570, score 0.3838; the last step scores 0.3843).
- Selection score: 0.8 x mean animation bits per pixel + 0.2 x mean view bits per pixel on the validation characters.
The spike at minute 60 is a loss spike early in training that recovered by minute 90; both curves flatten after about 400 minutes.
Results
Teacher-forced bits on the validation characters (best checkpoint; lower is better; 463 characters):
| 16 px | 24 px | 32 px | |
|---|---|---|---|
| Animation frame, bits per pixel | 0.328 | 0.324 | 0.306 |
| Animation frame, bits per frame | 84.0 | 186.6 | 313.8 |
| View (turn to a direction), bits per pixel | 0.719 | 0.606 | 0.598 |
| View, bits per frame | 183.9 | 349.1 | 612.2 |
Training bits per pixel at the end are 0.31 / 0.32 / 0.32 (16 / 24 / 32 px), so there is little overfitting. A view is about twice as hard to predict as the next frame, because a view has to invent hidden sides while a frame mostly copies the previous one.
Quality, honestly
- Animation of a given view is good. Walk and fly cycles follow the 3D farm closely (sample sheet, columns 5-8).
- Turning to a new direction is weaker, especially for animals and for sprites the model never saw in the dataset (for example sprites drawn by the app itself). Silhouettes and colours drift: a darker palette, a wrong back view, an odd profile.
- Errors propagate. Every action starts from its drawn view, so a bad view carries into all frames of that action. Draw again with another seed when it happens.
spin_kickis rare in the dataset after QC (14% of humans keep it) and is the messiest action. 24 px uses the front-render reference type.- No human study: we measured bits per pixel and looked at samples.
Speed
Measured on a dev box (RTX 5070 Ti shared with other work, i9-9900K), ONNX Runtime. A walk is the views plus 8 frames per direction, batched:
| 16 px | 32 px | |
|---|---|---|
| CUDA graphs: one frame | 0.4-0.6 s | 3.0-5.3 s |
| CUDA graphs: walk, 1 direction | 3.6-4.0 s | 27.5 s |
| CUDA graphs: walk, 8 directions | 5.8 s | 48.2 s |
| CPU, 8 threads: walk, 1 direction | 19.5 s | 125 s |
The PyTorch example.py on a CPU takes about 30 s (16 px) or 3 minutes (32 px) for one direction of 8 frames.
Limitations and intended use
- For: prototypes, game jams and research on autoregressive pixel models, non-commercially. Characters only: people, four-legged animals and birds in the dataset's styles.
- Not for: props, buildings, other body shapes (snakes, fish, many legs) or sprites far from the data (isometric, top-down); there are no such actions.
- The reference should look like the dataset: one character, facing the camera, 16 or 32 px. Odd poses, tiny details and large accessories (a bow, a staff) are often lost when turning. A 32 px sprite drawn at 16 px is a rough nearest-neighbour shrink and often comes out poor; give a native 16 px sprite instead.
- Sampling is random: results depend on the seed and on the device.
- Style and bias come from the source sprites (two AI-generated pixel-art sets), Qwen-Image restyling and the 3D pipeline, with their errors (some 3D models come out rotated, more often for animals).
Licence and credits
- Weights: CC BY-NC 4.0 (non-commercial; attribution). The training data contains outputs of Qwen-Image 2.1 (Qwen Research License, non-commercial). Commercial use of the weights or of their output needs your own legal review and, where Qwen's licence applies, a separate licence from Qwen. This card gives no legal guarantee.
- Built with Qwen. Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT, Copyright (c) 2026 Hangzhou Tongyi Laboratory Technology Co., Ltd. All Rights Reserved. The text is in
LICENSE_QWEN.txt; seeNOTICE.txtfor third-party material. - Base model: pixelsprite-canvas-v2 (CC BY-NC 4.0), trained on pixel-art-atlas-1.4m and pixelart-172k; the source sprites of the characters come from the same two sets.
- Code (GitHub, later): Apache-2.0.
Links
- Dataset: https://huggingface.co/datasets/ihor-umanets/pixelsprite-animated-characters
- Text-to-sprite model (v2): https://huggingface.co/ihor-umanets/pixelsprite-canvas-v2
- Demo (v2, in the browser): https://umanets.github.io/pixelsprite/
- Code: https://github.com/umanets/pixelsprite
- Article: TODO: link to the article goes here
Citation
@misc{umanets2026animgpt,
author = {Umanets, Ihor},
title = {pixelsprite AnimGPT: turning and animating pixel-art character sprites with an autoregressive transformer},
year = {2026},
howpublished = {\url{https://huggingface.co/ihor-umanets/pixelsprite-animgpt}},
note = {Built with Qwen. Non-commercial (CC BY-NC 4.0).}
}
- Downloads last month
- -
Model tree for ihor-umanets/pixelsprite-animgpt
Base model
ihor-umanets/pixelsprite-canvas-v2