pixmoji — type a phrase, get 16×16 pixel-art emojis

pixmoji generates small 16×16 pixel-art sprites from short text ("lovely sheep", "cat wearing a tiny crown", "three blue apples") fast enough to run as you type. Every image is generated by the model; nothing is retrieved.

sample sheet: 30 prompts × 10 generated emojis

file what it is
pixmoji-v2.2-fast.safetensors the fast model (43.6M params): single pass per step, used for all prompts without a count
pixmoji-v2.1.safetensors the base model (43.6M), used for prompts with a count ≥ 2 ("three blue apples")
palette.npy the 4096-colour palette (token → RGB; token 0 = transparent)
pixmoji/ self-contained inference code (model, sampler, reranker, selection)

Quick start

pip install torch transformers safetensors huggingface_hub pillow numpy
import sys, torch
from huggingface_hub import snapshot_download
from PIL import Image

repo = snapshot_download("model-collapse/pixmoji")     # weights + the small `pixmoji/` inference package
sys.path.insert(0, repo)
import pixmoji

gen = pixmoji.load(repo)                               # GPU if available
# on CPUs with AVX-512 BF16 (e.g. AMD Zen 4, Intel Sapphire Rapids), ~2x faster:
# gen = pixmoji.load(repo, device="cpu", dtype=torch.bfloat16)
emojis = gen.generate_emojis("cat wearing a tiny crown", k=10, seed=0)
for i, e in enumerate(emojis):                         # e.rgba: (16, 16, 4) uint8, e.score: TinyCLIP text-image similarity
    Image.fromarray(e.rgba, "RGBA").resize((128, 128), Image.NEAREST).save(f"emoji_{i}.png")

(To git clone the repo instead, install Git LFS first so the weight files are downloaded.)

generate_emojis returns up to k distinct emojis (near-duplicates are dropped, so a simple prompt can return fewer).

How it works

  • Generator: a 12-layer, 512-wide bidirectional transformer over the 256 pixels of a 16×16 sprite, each pixel a token in a 4096-colour palette (+ transparent). It fills the sprite in over 8–12 parallel decoding steps (MaskGIT-style), starting fully masked and fixing the most confident pixels each step.
  • Conditioning: the frozen text tower of TinyCLIP-39M-PixelArt (pooled embedding + per-word features) and a count token parsed from the prompt (one…four).
  • Selection: 24 candidates are generated and reranked by TinyCLIP image–text similarity; up to 10 diverse ones are returned.
  • Two models, routed by prompt:
    • pixmoji-v2.1 uses classifier-free guidance (two passes per step; guidance 5 for counts). Prompts with a count ≥ 2 use it.
    • pixmoji-v2.2-fast is v2.1 guidance-distilled so one pass per step reproduces v2.1's guided output, trained on states the sampler actually visits (192k recorded v2.1 sampling runs). With 8 steps it needs one third of v2.1's compute at the same measured quality.

Training data

  • Angais/pixel-art-atlas-1.4m (CC0): 1.4M 16×16 sprites with captions, itself generated with Krea 2 and a pixel-art LoRA. Captions were corrected for object counts with Anthropic Claude Haiku 5.5.
  • ~38k additional sprites for counts and combinations, generated with Krea 2 Turbo + the Angais/Krea-2-PixelArt-16x16 LoRA and filtered by Claude Haiku 5.5 (count check, "shows both parts" check) and TinyCLIP.
  • Composites of 2–4 sprites for count training. Evaluation concepts were blocked from the generated training data.

pixmoji is a separately trained model; it does not contain Krea 2 weights.

Evaluation

Judges: Anthropic Claude Sonnet 5.5 and OpenAI GPT-6.1 Terra, both via Amazon Bedrock, each image pair shown in both orders. "Re-seed" = v2.1 run again with other seeds (identical quality by construction), the reference for noise.

v2.1 (base)
Novel combinations depicted (30 prompts × 10) 0.56
Count accuracy, 1–4 objects (Terra / Sonnet) 0.76 / 0.67
Count accuracy, 3–4 objects 0.49–0.65
Judged better than the real dataset sprite (parity = 0.50; Sonnet / Terra) 0.48 / 0.25

pixmoji-v2.2-fast vs v2.1 (300 fresh single-object captions, all 10 shown emojis judged; positive = v2.1 preferred):

re-seed v2.2-fast, 8 steps
Δ share of emojis depicting the caption (Sonnet / Terra) +0.9 / −1.3 pts +1.0 / −1.5 pts
set head-to-head margin (Sonnet / Terra) +4.7 / +9.0 +4.0 / +12.7
novel combinations depicted (Sonnet / Terra; v2.1: 0.53 / 0.62) 0.56 / 0.56 0.53 / 0.61

Also within noise in two independent 720-caption head-to-head rounds. Fewer steps (6), fewer candidates, smaller or pruned students, int4 weights and low-rank layers were all measured to lose quality and are not used.

Speed (10 emojis per request)

hardware prompt without a count (v2.2-fast) count prompt (v2.1)
NVIDIA A100 (shared with another job), fp16 ~0.13 s ~0.25–0.30 s
AMD EPYC 9R14, 64 cores, bf16, threads bound to one socket 0.93 s 3.3 s (with cpu_fast="counts")
same CPU, 8 cores 4.6 s 12.4 s

CPU tips: OMP_PROC_BIND=close OMP_PLACES=cores, keep threads within one socket, dtype=torch.bfloat16 on CPUs with AVX-512 BF16 (the fast model keeps its final colour layer in fp32, which the evaluation showed is needed).

Limitations

  • 16×16 only; small details (an egg next to a hen, a helmet on a penguin) are often missing.
  • Counts of 3–4 objects are right only about half to two thirds of the time.
  • Spelling matters: misspelled words ("helicorpter") can produce unrelated objects.
  • Simple prompts sometimes yield fewer than 10 distinct emojis.
  • Images are AI-generated; the model can reflect biases of its training data and of the teacher models.

Intended use and license

Weights and code are released under CC BY-NC 4.0 (non-commercial use, attribution required). Intended for demos, research and personal projects such as chat emojis. Not intended for commercial use, impersonation, or generating hateful or sexual content. Credit: Tom Aarsen (TinyCLIP-PixelArt, Pixel Atlas), Angais (Pixel Atlas dataset and Krea-2-PixelArt-16x16 LoRA).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train model-collapse/pixmoji