Download README.md from model-collapse/pixmoji: direct link, hf CLI and curl.
- Browser
- Download file 6.58 kB
-
https://huggingface.co/model-collapse/pixmoji/resolve/main/README.md
- Command line
-
hf download hf://model-collapse/pixmoji/README.md
-
curl -L -o README.md https://huggingface.co/model-collapse/pixmoji/resolve/main/README.md
license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: text-to-image
tags:
- pixel-art
- emoji
- 16x16
- maskgit
- text-to-image
datasets:
- Angais/pixel-art-atlas-1.4m
pixmoji — type a phrase, get 16×16 pixel-art emojis
pixmoji generates small 16×16 pixel-art sprites from short text ("lovely sheep", "cat wearing a tiny crown", "three blue apples") fast enough to run as you type. Every image is generated by the model; nothing is retrieved.
| file | what it is |
|---|---|
pixmoji-v2.2-fast.safetensors |
the fast model (43.6M params): single pass per step, used for all prompts without a count |
pixmoji-v2.1.safetensors |
the base model (43.6M), used for prompts with a count ≥ 2 ("three blue apples") |
palette.npy |
the 4096-colour palette (token → RGB; token 0 = transparent) |
pixmoji/ |
self-contained inference code (model, sampler, reranker, selection) |
Quick start
pip install torch transformers safetensors huggingface_hub pillow numpy
import sys, torch
from huggingface_hub import snapshot_download
from PIL import Image
repo = snapshot_download("model-collapse/pixmoji") # weights + the small `pixmoji/` inference package
sys.path.insert(0, repo)
import pixmoji
gen = pixmoji.load(repo) # GPU if available
# on CPUs with AVX-512 BF16 (e.g. AMD Zen 4, Intel Sapphire Rapids), ~2x faster:
# gen = pixmoji.load(repo, device="cpu", dtype=torch.bfloat16)
emojis = gen.generate_emojis("cat wearing a tiny crown", k=10, seed=0)
for i, e in enumerate(emojis): # e.rgba: (16, 16, 4) uint8, e.score: TinyCLIP text-image similarity
Image.fromarray(e.rgba, "RGBA").resize((128, 128), Image.NEAREST).save(f"emoji_{i}.png")
(To git clone the repo instead, install Git LFS first so the weight files are downloaded.)
generate_emojis returns up to k distinct emojis (near-duplicates are dropped, so a simple prompt can return fewer).
How it works
- Generator: a 12-layer, 512-wide bidirectional transformer over the 256 pixels of a 16×16 sprite, each pixel a token in a 4096-colour palette (+ transparent). It fills the sprite in over 8–12 parallel decoding steps (MaskGIT-style), starting fully masked and fixing the most confident pixels each step.
- Conditioning: the frozen text tower of TinyCLIP-39M-PixelArt (pooled embedding + per-word features) and a count token parsed from the prompt (one…four).
- Selection: 24 candidates are generated and reranked by TinyCLIP image–text similarity; up to 10 diverse ones are returned.
- Two models, routed by prompt:
pixmoji-v2.1uses classifier-free guidance (two passes per step; guidance 5 for counts). Prompts with a count ≥ 2 use it.pixmoji-v2.2-fastis v2.1 guidance-distilled so one pass per step reproduces v2.1's guided output, trained on states the sampler actually visits (192k recorded v2.1 sampling runs). With 8 steps it needs one third of v2.1's compute at the same measured quality.
Training data
- Angais/pixel-art-atlas-1.4m (CC0): 1.4M 16×16 sprites with captions, itself generated with Krea 2 and a pixel-art LoRA. Captions were corrected for object counts with Anthropic Claude Haiku 5.5.
- ~38k additional sprites for counts and combinations, generated with Krea 2 Turbo + the Angais/Krea-2-PixelArt-16x16 LoRA and filtered by Claude Haiku 5.5 (count check, "shows both parts" check) and TinyCLIP.
- Composites of 2–4 sprites for count training. Evaluation concepts were blocked from the generated training data.
pixmoji is a separately trained model; it does not contain Krea 2 weights.
Evaluation
Judges: Anthropic Claude Sonnet 5.5 and OpenAI GPT-6.1 Terra, both via Amazon Bedrock, each image pair shown in both orders. "Re-seed" = v2.1 run again with other seeds (identical quality by construction), the reference for noise.
| v2.1 (base) | |
|---|---|
| Novel combinations depicted (30 prompts × 10) | 0.56 |
| Count accuracy, 1–4 objects (Terra / Sonnet) | 0.76 / 0.67 |
| Count accuracy, 3–4 objects | 0.49–0.65 |
| Judged better than the real dataset sprite (parity = 0.50; Sonnet / Terra) | 0.48 / 0.25 |
pixmoji-v2.2-fast vs v2.1 (300 fresh single-object captions, all 10 shown emojis judged; positive = v2.1 preferred):
| re-seed | v2.2-fast, 8 steps | |
|---|---|---|
| Δ share of emojis depicting the caption (Sonnet / Terra) | +0.9 / −1.3 pts | +1.0 / −1.5 pts |
| set head-to-head margin (Sonnet / Terra) | +4.7 / +9.0 | +4.0 / +12.7 |
| novel combinations depicted (Sonnet / Terra; v2.1: 0.53 / 0.62) | 0.56 / 0.56 | 0.53 / 0.61 |
Also within noise in two independent 720-caption head-to-head rounds. Fewer steps (6), fewer candidates, smaller or pruned students, int4 weights and low-rank layers were all measured to lose quality and are not used.
Speed (10 emojis per request)
| hardware | prompt without a count (v2.2-fast) | count prompt (v2.1) |
|---|---|---|
| NVIDIA A100 (shared with another job), fp16 | ~0.13 s | ~0.25–0.30 s |
| AMD EPYC 9R14, 64 cores, bf16, threads bound to one socket | 0.93 s | 3.3 s (with cpu_fast="counts") |
| same CPU, 8 cores | 4.6 s | 12.4 s |
CPU tips: OMP_PROC_BIND=close OMP_PLACES=cores, keep threads within one socket, dtype=torch.bfloat16 on CPUs with
AVX-512 BF16 (the fast model keeps its final colour layer in fp32, which the evaluation showed is needed).
Limitations
- 16×16 only; small details (an egg next to a hen, a helmet on a penguin) are often missing.
- Counts of 3–4 objects are right only about half to two thirds of the time.
- Spelling matters: misspelled words ("helicorpter") can produce unrelated objects.
- Simple prompts sometimes yield fewer than 10 distinct emojis.
- Images are AI-generated; the model can reflect biases of its training data and of the teacher models.
Intended use and license
Weights and code are released under CC BY-NC 4.0 (non-commercial use, attribution required). Intended for demos, research and personal projects such as chat emojis. Not intended for commercial use, impersonation, or generating hateful or sexual content. Credit: Tom Aarsen (TinyCLIP-PixelArt, Pixel Atlas), Angais (Pixel Atlas dataset and Krea-2-PixelArt-16x16 LoRA).
