pixmoji v2.2-fast + v2.1: weights, palette, inference code, model card
Browse files- .gitattributes +1 -0
- README.md +122 -0
- palette.npy +3 -0
- pixmoji-v2.1.json +17 -0
- pixmoji-v2.1.safetensors +3 -0
- pixmoji-v2.2-fast.json +17 -0
- pixmoji-v2.2-fast.safetensors +3 -0
- pixmoji/__init__.py +10 -0
- pixmoji/counts.py +79 -0
- pixmoji/model.py +184 -0
- pixmoji/pipeline.py +155 -0
- pixmoji/sampling.py +90 -0
- pixmoji/select.py +25 -0
- pixmoji/text.py +22 -0
- pixmoji/tinyclip.py +98 -0
- sample_sheet.png +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
sample_sheet.png filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,122 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-4.0
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
pipeline_tag: text-to-image
|
| 5 |
+
tags:
|
| 6 |
+
- pixel-art
|
| 7 |
+
- emoji
|
| 8 |
+
- 16x16
|
| 9 |
+
- maskgit
|
| 10 |
+
- text-to-image
|
| 11 |
+
datasets:
|
| 12 |
+
- Angais/pixel-art-atlas-1.4m
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# pixmoji — type a phrase, get 16×16 pixel-art emojis
|
| 16 |
+
|
| 17 |
+
pixmoji generates small **16×16 pixel-art sprites** from short text ("lovely sheep", "cat wearing a tiny crown",
|
| 18 |
+
"three blue apples") fast enough to run as you type. Every image is **generated** by the model; nothing is retrieved.
|
| 19 |
+
|
| 20 |
+

|
| 21 |
+
|
| 22 |
+
| file | what it is |
|
| 23 |
+
|---|---|
|
| 24 |
+
| `pixmoji-v2.2-fast.safetensors` | the fast model (43.6M params): single pass per step, used for all prompts without a count |
|
| 25 |
+
| `pixmoji-v2.1.safetensors` | the base model (43.6M), used for prompts with a count ≥ 2 ("three blue apples") |
|
| 26 |
+
| `palette.npy` | the 4096-colour palette (token → RGB; token 0 = transparent) |
|
| 27 |
+
| `pixmoji/` | self-contained inference code (model, sampler, reranker, selection) |
|
| 28 |
+
|
| 29 |
+
## Quick start
|
| 30 |
+
|
| 31 |
+
```bash
|
| 32 |
+
pip install torch transformers safetensors huggingface_hub pillow numpy
|
| 33 |
+
git clone https://huggingface.co/model-collapse/pixmoji && cd pixmoji
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
```python
|
| 37 |
+
import torch, pixmoji
|
| 38 |
+
from PIL import Image
|
| 39 |
+
|
| 40 |
+
gen = pixmoji.load(".") # or pixmoji.load() to download from the Hub; GPU if available
|
| 41 |
+
# on CPUs with AVX-512 BF16 (e.g. AMD Zen 4, Intel Sapphire Rapids), ~2x faster:
|
| 42 |
+
# gen = pixmoji.load(".", device="cpu", dtype=torch.bfloat16)
|
| 43 |
+
emojis = gen.generate_emojis("cat wearing a tiny crown", k=10, seed=0)
|
| 44 |
+
for i, e in enumerate(emojis): # e.rgba: (16, 16, 4) uint8, e.score: TinyCLIP text-image similarity
|
| 45 |
+
Image.fromarray(e.rgba, "RGBA").resize((128, 128), Image.NEAREST).save(f"emoji_{i}.png")
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
`generate_emojis` returns up to `k` distinct emojis (near-duplicates are dropped, so a simple prompt can return fewer).
|
| 49 |
+
|
| 50 |
+
## How it works
|
| 51 |
+
|
| 52 |
+
- **Generator:** a 12-layer, 512-wide bidirectional transformer over the 256 pixels of a 16×16 sprite, each pixel a token in a
|
| 53 |
+
4096-colour palette (+ transparent). It fills the sprite in over 8–12 parallel decoding steps (MaskGIT-style), starting fully
|
| 54 |
+
masked and fixing the most confident pixels each step.
|
| 55 |
+
- **Conditioning:** the frozen text tower of [TinyCLIP-39M-PixelArt](https://huggingface.co/tomaarsen/TinyCLIP-39M-PixelArt)
|
| 56 |
+
(pooled embedding + per-word features) and a count token parsed from the prompt (one…four).
|
| 57 |
+
- **Selection:** 24 candidates are generated and reranked by TinyCLIP image–text similarity; up to 10 diverse ones are returned.
|
| 58 |
+
- **Two models, routed by prompt:**
|
| 59 |
+
- `pixmoji-v2.1` uses classifier-free guidance (two passes per step; guidance 5 for counts). Prompts with a count ≥ 2 use it.
|
| 60 |
+
- `pixmoji-v2.2-fast` is v2.1 **guidance-distilled** so one pass per step reproduces v2.1's guided output, trained on
|
| 61 |
+
states the sampler actually visits (192k recorded v2.1 sampling runs). With 8 steps it needs **one third of v2.1's
|
| 62 |
+
compute** at the same measured quality.
|
| 63 |
+
|
| 64 |
+
## Training data
|
| 65 |
+
|
| 66 |
+
- [Angais/pixel-art-atlas-1.4m](https://huggingface.co/datasets/Angais/pixel-art-atlas-1.4m) (CC0): 1.4M 16×16 sprites with
|
| 67 |
+
captions, itself generated with Krea 2 and a pixel-art LoRA. Captions were corrected for object counts with Anthropic Claude
|
| 68 |
+
Haiku 5.5.
|
| 69 |
+
- ~38k additional sprites for counts and combinations, generated with Krea 2 Turbo + the
|
| 70 |
+
[Angais/Krea-2-PixelArt-16x16](https://huggingface.co/Angais/Krea-2-PixelArt-16x16) LoRA and filtered by Claude Haiku 5.5
|
| 71 |
+
(count check, "shows both parts" check) and TinyCLIP.
|
| 72 |
+
- Composites of 2–4 sprites for count training. Evaluation concepts were blocked from the generated training data.
|
| 73 |
+
|
| 74 |
+
pixmoji is a separately trained model; it does not contain Krea 2 weights.
|
| 75 |
+
|
| 76 |
+
## Evaluation
|
| 77 |
+
|
| 78 |
+
Judges: **Anthropic Claude Sonnet 5.5** and **OpenAI GPT-6.1 Terra**, both via Amazon Bedrock, each image pair shown in both
|
| 79 |
+
orders. "Re-seed" = v2.1 run again with other seeds (identical quality by construction), the reference for noise.
|
| 80 |
+
|
| 81 |
+
| | v2.1 (base) |
|
| 82 |
+
|---|---|
|
| 83 |
+
| Novel combinations depicted (30 prompts × 10) | 0.56 |
|
| 84 |
+
| Count accuracy, 1–4 objects (Terra / Sonnet) | 0.76 / 0.67 |
|
| 85 |
+
| Count accuracy, 3–4 objects | 0.49–0.65 |
|
| 86 |
+
| Judged better than the real dataset sprite (parity = 0.50; Sonnet / Terra) | 0.48 / 0.25 |
|
| 87 |
+
|
| 88 |
+
pixmoji-v2.2-fast vs v2.1 (300 fresh single-object captions, all 10 shown emojis judged; positive = v2.1 preferred):
|
| 89 |
+
|
| 90 |
+
| | re-seed | **v2.2-fast, 8 steps** |
|
| 91 |
+
|---|---|---|
|
| 92 |
+
| Δ share of emojis depicting the caption (Sonnet / Terra) | +0.9 / −1.3 pts | +1.0 / −1.5 pts |
|
| 93 |
+
| set head-to-head margin (Sonnet / Terra) | +4.7 / +9.0 | +4.0 / +12.7 |
|
| 94 |
+
| novel combinations depicted (Sonnet / Terra; v2.1: 0.53 / 0.62) | 0.56 / 0.56 | 0.53 / 0.61 |
|
| 95 |
+
|
| 96 |
+
Also within noise in two independent 720-caption head-to-head rounds. Fewer steps (6), fewer candidates, smaller or pruned
|
| 97 |
+
students, int4 weights and low-rank layers were all measured to lose quality and are not used.
|
| 98 |
+
|
| 99 |
+
## Speed (10 emojis per request)
|
| 100 |
+
|
| 101 |
+
| hardware | prompt without a count (v2.2-fast) | count prompt (v2.1) |
|
| 102 |
+
|---|---|---|
|
| 103 |
+
| NVIDIA A100 (shared with another job), fp16 | ~0.13 s | ~0.25–0.30 s |
|
| 104 |
+
| AMD EPYC 9R14, 64 cores, bf16, threads bound to one socket | 0.93 s | 3.3 s (with `cpu_fast="counts"`) |
|
| 105 |
+
| same CPU, 8 cores | 4.6 s | 12.4 s |
|
| 106 |
+
|
| 107 |
+
CPU tips: `OMP_PROC_BIND=close OMP_PLACES=cores`, keep threads within one socket, `dtype=torch.bfloat16` on CPUs with
|
| 108 |
+
AVX-512 BF16 (the fast model keeps its final colour layer in fp32, which the evaluation showed is needed).
|
| 109 |
+
|
| 110 |
+
## Limitations
|
| 111 |
+
|
| 112 |
+
- 16×16 only; small details (an egg next to a hen, a helmet on a penguin) are often missing.
|
| 113 |
+
- Counts of 3–4 objects are right only about half to two thirds of the time.
|
| 114 |
+
- Spelling matters: misspelled words ("helicorpter") can produce unrelated objects.
|
| 115 |
+
- Simple prompts sometimes yield fewer than 10 distinct emojis.
|
| 116 |
+
- Images are AI-generated; the model can reflect biases of its training data and of the teacher models.
|
| 117 |
+
|
| 118 |
+
## Intended use and license
|
| 119 |
+
|
| 120 |
+
Weights and code are released under **CC BY-NC 4.0** (non-commercial use, attribution required). Intended for demos, research
|
| 121 |
+
and personal projects such as chat emojis. Not intended for commercial use, impersonation, or generating hateful or sexual
|
| 122 |
+
content. Credit: Tom Aarsen (TinyCLIP-PixelArt, Pixel Atlas), Angais (Pixel Atlas dataset and Krea-2-PixelArt-16x16 LoRA).
|
palette.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bad6e5a7e34f663c0233119807a9840dd966b6c4f1b847f6fc166bf476fdf6dc
|
| 3 |
+
size 12419
|
pixmoji-v2.1.json
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "pixmoji-v2.1",
|
| 3 |
+
"config": {
|
| 4 |
+
"vocab": 4097,
|
| 5 |
+
"d_model": 512,
|
| 6 |
+
"n_layers": 12,
|
| 7 |
+
"n_heads": 8,
|
| 8 |
+
"mlp_ratio": 4,
|
| 9 |
+
"cond_dim": 512,
|
| 10 |
+
"n_cond_tokens": 4,
|
| 11 |
+
"n_counts": 5,
|
| 12 |
+
"dropout": 0.0,
|
| 13 |
+
"text_tokens": 24,
|
| 14 |
+
"text_dim": 512
|
| 15 |
+
},
|
| 16 |
+
"threshold": -1.0
|
| 17 |
+
}
|
pixmoji-v2.1.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f7f637d5d86304966d004b5d2d907bb27a5e2841ce363ec225e78b52535adced
|
| 3 |
+
size 174571316
|
pixmoji-v2.2-fast.json
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "pixmoji-v2.2-fast",
|
| 3 |
+
"config": {
|
| 4 |
+
"vocab": 4097,
|
| 5 |
+
"d_model": 512,
|
| 6 |
+
"n_layers": 12,
|
| 7 |
+
"n_heads": 8,
|
| 8 |
+
"mlp_ratio": 4,
|
| 9 |
+
"cond_dim": 512,
|
| 10 |
+
"n_cond_tokens": 4,
|
| 11 |
+
"n_counts": 5,
|
| 12 |
+
"dropout": 0.0,
|
| 13 |
+
"text_tokens": 24,
|
| 14 |
+
"text_dim": 512
|
| 15 |
+
},
|
| 16 |
+
"threshold": -1.0
|
| 17 |
+
}
|
pixmoji-v2.2-fast.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:017353adbd9f39cdffa2f07f445e5b4584ed8b18de7a0086f6dfe2e6acb974b3
|
| 3 |
+
size 174571316
|
pixmoji/__init__.py
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""pixmoji: text -> 16x16 pixel-art emojis (masked-token transformer, TinyCLIP-conditioned).
|
| 2 |
+
|
| 3 |
+
import pixmoji
|
| 4 |
+
gen = pixmoji.load() # downloads model-collapse/pixmoji (CPU by default)
|
| 5 |
+
for e in gen.generate_emojis("cat wearing a tiny crown", k=10, seed=0):
|
| 6 |
+
e.rgba # (16, 16, 4) uint8
|
| 7 |
+
"""
|
| 8 |
+
from pixmoji.pipeline import Emoji, EmojiGenerator, load
|
| 9 |
+
|
| 10 |
+
__all__ = ["Emoji", "EmojiGenerator", "load"]
|
pixmoji/counts.py
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Count parsing (0 = unspecified, 1-4) and simple English pluralization."""
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import re
|
| 5 |
+
|
| 6 |
+
NUM_WORDS = {1: "one", 2: "two", 3: "three", 4: "four"}
|
| 7 |
+
_WORD_VALUES = {"one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8,
|
| 8 |
+
"nine": 9, "ten": 10, "eleven": 11, "twelve": 12, "dozen": 12}
|
| 9 |
+
_PAIR = re.compile(r"(?<![\w-])(?:a\s+)?(?:pair|couple)\s+of(?![\w-])", re.I)
|
| 10 |
+
_NUM = re.compile(r"(?<![\w-])(\d+|" + "|".join(_WORD_VALUES) + r")(?![\w-])", re.I)
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
_HEAD_END = re.compile(r"\s+(?:with|in|on|inside|under|holding|wearing|and|for|from)\s+", re.I)
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
def head_phrase(text: str) -> str:
|
| 17 |
+
"""The main noun phrase: text before the first 'with/in/on/...' clause ('crown key with three gems' -> 'crown key')."""
|
| 18 |
+
return _HEAD_END.split(text, maxsplit=1)[0]
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def strip_count(text: str) -> str:
|
| 22 |
+
"""Remove a leading count expression from the head phrase ('three red apples' -> 'red apples')."""
|
| 23 |
+
head = head_phrase(text); rest = text[len(head):]
|
| 24 |
+
head = _PAIR.sub("", head, count=1)
|
| 25 |
+
head = _NUM.sub("", head, count=1)
|
| 26 |
+
return (re.sub(r"\s+", " ", head).strip() + rest).strip()
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def parse_count(text: str) -> int:
|
| 30 |
+
"""Count of the main objects (0 = unspecified), read from the head phrase only."""
|
| 31 |
+
text = head_phrase(text)
|
| 32 |
+
if _PAIR.search(text):
|
| 33 |
+
return 2
|
| 34 |
+
m = _NUM.search(text)
|
| 35 |
+
if not m:
|
| 36 |
+
return 0
|
| 37 |
+
tok = m.group(1).lower()
|
| 38 |
+
n = int(tok) if tok.isdigit() else _WORD_VALUES[tok]
|
| 39 |
+
return n if 1 <= n <= 4 else 0
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
_IRREGULAR = {"man": "men", "woman": "women", "child": "children", "mouse": "mice", "foot": "feet",
|
| 43 |
+
"tooth": "teeth", "knife": "knives", "leaf": "leaves", "wolf": "wolves",
|
| 44 |
+
"sheep": "sheep", "fish": "fish", "deer": "deer"}
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
_NOT_MAN = {"human", "shaman", "talisman", "caiman", "ottoman", "german", "walkman"}
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def _plural_word(w: str) -> str:
|
| 51 |
+
lw = w.lower()
|
| 52 |
+
if lw not in _NOT_MAN:
|
| 53 |
+
for sing, plur in _IRREGULAR.items(): # suffix match covers compounds: snowman, werewolf, goldfish
|
| 54 |
+
if lw.endswith(sing):
|
| 55 |
+
return w[: len(w) - len(sing)] + plur
|
| 56 |
+
if re.search(r"(s|x|z|ch|sh)$", lw):
|
| 57 |
+
return w + "es"
|
| 58 |
+
if re.search(r"[^aeiou]y$", lw):
|
| 59 |
+
return w[:-1] + "ies"
|
| 60 |
+
return w + "s"
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
_HEAD_SPLIT = re.compile(r"(\s+(?:of|with|in|on|for)\s+|-(?:at|of|in)-)")
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def _already_plural(w: str) -> bool:
|
| 67 |
+
lw = w.lower()
|
| 68 |
+
return lw.endswith("s") and not re.search(r"(ss|us|is|os|as)$", lw)
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def pluralize(noun_phrase: str) -> str:
|
| 72 |
+
"""Pluralize the head noun: the last word before 'of/with/in/on/for' or a hyphenated '-at-' phrase."""
|
| 73 |
+
m = _HEAD_SPLIT.search(noun_phrase)
|
| 74 |
+
head, tail = (noun_phrase[: m.start()], noun_phrase[m.start():]) if m else (noun_phrase, "")
|
| 75 |
+
parts = head.rsplit(" ", 1)
|
| 76 |
+
last = parts[-1]
|
| 77 |
+
if not _already_plural(last):
|
| 78 |
+
last = _plural_word(last)
|
| 79 |
+
return (parts[0] + " " if len(parts) == 2 else "") + last + tail
|
pixmoji/model.py
ADDED
|
@@ -0,0 +1,184 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""MaskGIT-style bidirectional transformer over 256 palette-index pixels, conditioned on text embedding + count."""
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import math
|
| 5 |
+
from dataclasses import asdict, dataclass
|
| 6 |
+
|
| 7 |
+
import torch
|
| 8 |
+
import torch.nn as nn
|
| 9 |
+
import torch.nn.functional as F
|
| 10 |
+
|
| 11 |
+
MASK_ID = 4097
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
@dataclass
|
| 15 |
+
class ModelConfig:
|
| 16 |
+
vocab: int = 4097
|
| 17 |
+
d_model: int = 384
|
| 18 |
+
n_layers: int = 8
|
| 19 |
+
n_heads: int = 6
|
| 20 |
+
mlp_ratio: int = 4
|
| 21 |
+
cond_dim: int = 512
|
| 22 |
+
n_cond_tokens: int = 4
|
| 23 |
+
n_counts: int = 5
|
| 24 |
+
dropout: float = 0.0
|
| 25 |
+
text_tokens: int = 0 # >0: word-level conditioning on this many TinyCLIP token features (v2)
|
| 26 |
+
text_dim: int = 512
|
| 27 |
+
|
| 28 |
+
def to_dict(self) -> dict:
|
| 29 |
+
return asdict(self)
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
TINY = ModelConfig(d_model=64, n_layers=2, n_heads=2)
|
| 33 |
+
SMALL = ModelConfig(d_model=320, n_layers=6, n_heads=5)
|
| 34 |
+
V2 = ModelConfig(d_model=512, n_layers=12, n_heads=8, text_tokens=24, text_dim=512)
|
| 35 |
+
V2S = ModelConfig(d_model=512, n_layers=8, n_heads=8, text_tokens=24, text_dim=512) # lighter fallback if V2 misses the latency gate
|
| 36 |
+
V2S6 = ModelConfig(d_model=512, n_layers=6, n_heads=8, text_tokens=24, text_dim=512) # CPU student (layer-drop init)
|
| 37 |
+
V2XS = ModelConfig(d_model=384, n_layers=8, n_heads=6, text_tokens=24, text_dim=512) # CPU student (fresh init), ~2.7x fewer FLOPs
|
| 38 |
+
V3 = ModelConfig(d_model=768, n_layers=12, n_heads=12, text_tokens=24, text_dim=512) # capacity probe (~92M), g7e-only serving
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
class Block(nn.Module):
|
| 42 |
+
def __init__(self, c: ModelConfig):
|
| 43 |
+
super().__init__()
|
| 44 |
+
self.n_heads = c.n_heads
|
| 45 |
+
self.ln1, self.ln2 = nn.LayerNorm(c.d_model), nn.LayerNorm(c.d_model)
|
| 46 |
+
self.qkv = nn.Linear(c.d_model, 3 * c.d_model)
|
| 47 |
+
self.proj = nn.Linear(c.d_model, c.d_model)
|
| 48 |
+
self.mlp = nn.Sequential(nn.Linear(c.d_model, c.mlp_ratio * c.d_model), nn.GELU(), nn.Linear(c.mlp_ratio * c.d_model, c.d_model))
|
| 49 |
+
self.drop = nn.Dropout(c.dropout)
|
| 50 |
+
|
| 51 |
+
def forward(self, x: torch.Tensor, attn_mask: torch.Tensor | None = None) -> torch.Tensor:
|
| 52 |
+
b, t, d = x.shape
|
| 53 |
+
q, k, v = self.qkv(self.ln1(x)).view(b, t, 3, self.n_heads, d // self.n_heads).permute(2, 0, 3, 1, 4)
|
| 54 |
+
a = F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask).transpose(1, 2).reshape(b, t, d)
|
| 55 |
+
x = x + self.drop(self.proj(a))
|
| 56 |
+
return x + self.drop(self.mlp(self.ln2(x)))
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
class PixmojiTransformer(nn.Module):
|
| 60 |
+
def __init__(self, c: ModelConfig = ModelConfig()):
|
| 61 |
+
super().__init__()
|
| 62 |
+
self.cfg = c
|
| 63 |
+
d = c.d_model
|
| 64 |
+
self.cond = nn.Sequential(nn.Linear(c.cond_dim, d), nn.GELU(), nn.Linear(d, c.n_cond_tokens * d))
|
| 65 |
+
self.count_emb = nn.Embedding(c.n_counts, d)
|
| 66 |
+
self.tok_emb = nn.Embedding(c.vocab + 1, d) # + MASK
|
| 67 |
+
self.row_emb, self.col_emb = nn.Parameter(torch.zeros(16, 1, d)), nn.Parameter(torch.zeros(1, 16, d))
|
| 68 |
+
self.n_prefix = c.n_cond_tokens + c.text_tokens + 1
|
| 69 |
+
self.prefix_pos = nn.Parameter(torch.zeros(self.n_prefix, d))
|
| 70 |
+
if c.text_tokens:
|
| 71 |
+
self.word_proj = nn.Linear(c.text_dim, d)
|
| 72 |
+
self.blocks = nn.ModuleList(Block(c) for _ in range(c.n_layers))
|
| 73 |
+
self.ln_f = nn.LayerNorm(d)
|
| 74 |
+
self.head = nn.Linear(d, c.vocab)
|
| 75 |
+
self.apply(self._init)
|
| 76 |
+
for p in (self.row_emb, self.col_emb, self.prefix_pos):
|
| 77 |
+
nn.init.normal_(p, std=0.02)
|
| 78 |
+
|
| 79 |
+
@staticmethod
|
| 80 |
+
def _init(m):
|
| 81 |
+
if isinstance(m, (nn.Linear, nn.Embedding)):
|
| 82 |
+
nn.init.normal_(m.weight, std=0.02)
|
| 83 |
+
if isinstance(m, nn.Linear) and m.bias is not None:
|
| 84 |
+
nn.init.zeros_(m.bias)
|
| 85 |
+
|
| 86 |
+
def forward(self, pixel_tokens: torch.Tensor, text_emb: torch.Tensor, count_ids: torch.Tensor,
|
| 87 |
+
text_feats: torch.Tensor | None = None, text_mask: torch.Tensor | None = None,
|
| 88 |
+
out_pos: torch.Tensor | None = None) -> torch.Tensor:
|
| 89 |
+
"""out_pos (b, m): return logits only for these pixel positions (the sampler only needs still-masked ones)."""
|
| 90 |
+
b = pixel_tokens.shape[0]; c = self.cfg; dt = self.tok_emb.weight.dtype # (Embedding: stays float under int8 Linear quantization)
|
| 91 |
+
cond = self.cond(text_emb.to(dt)).view(b, c.n_cond_tokens, c.d_model)
|
| 92 |
+
parts, attn, pos = [cond], None, self.prefix_pos
|
| 93 |
+
if c.text_tokens:
|
| 94 |
+
L = text_feats.shape[1] # may be < text_tokens when trailing padded slots were trimmed (exact: they are masked anyway)
|
| 95 |
+
parts.append(self.word_proj(text_feats.to(dt)))
|
| 96 |
+
nc = c.n_cond_tokens
|
| 97 |
+
pos = torch.cat([self.prefix_pos[:nc + L], self.prefix_pos[-1:]])
|
| 98 |
+
keep = torch.ones(b, nc + L + 1 + 256, dtype=torch.bool, device=pixel_tokens.device)
|
| 99 |
+
keep[:, nc:nc + L] = text_mask
|
| 100 |
+
attn = keep[:, None, None, :] # key padding mask for SDPA
|
| 101 |
+
parts.append(self.count_emb(count_ids)[:, None])
|
| 102 |
+
prefix = torch.cat(parts, 1) + pos
|
| 103 |
+
pix = self.tok_emb(pixel_tokens) + (self.row_emb + self.col_emb).reshape(1, 256, c.d_model)
|
| 104 |
+
x = torch.cat([prefix, pix], 1)
|
| 105 |
+
for blk in self.blocks:
|
| 106 |
+
x = blk(x, attn)
|
| 107 |
+
h = x[:, prefix.shape[1]:]
|
| 108 |
+
if out_pos is not None:
|
| 109 |
+
h = torch.gather(h, 1, out_pos[..., None].expand(-1, -1, h.shape[-1]))
|
| 110 |
+
return self.head(self.ln_f(h.to(self.ln_f.weight.dtype))) # output layer may run in higher precision (mixed bf16)
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
def null_words(text_feats: torch.Tensor, text_mask: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
|
| 114 |
+
"""CFG / cond-drop null condition for word features: all zeros, only the first token visible."""
|
| 115 |
+
m = torch.zeros_like(text_mask); m[:, 0] = True
|
| 116 |
+
return torch.zeros_like(text_feats), m
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
def masking_loss(model: PixmojiTransformer, tokens: torch.Tensor, text_emb: torch.Tensor, count_ids: torch.Tensor,
|
| 120 |
+
cond_drop: float = 0.1, generator: torch.Generator | None = None, mask: torch.Tensor | None = None,
|
| 121 |
+
text_feats: torch.Tensor | None = None, text_mask: torch.Tensor | None = None) -> torch.Tensor:
|
| 122 |
+
b, n = tokens.shape; dev = tokens.device
|
| 123 |
+
if mask is None:
|
| 124 |
+
u = torch.rand(b, 1, generator=generator).to(dev)
|
| 125 |
+
n_mask = (n * torch.cos(math.pi / 2 * u)).ceil().clamp(1, n) # >= 1 masked position
|
| 126 |
+
rank = torch.rand(b, n, generator=generator).to(dev).argsort(1).argsort(1)
|
| 127 |
+
mask = rank < n_mask
|
| 128 |
+
if cond_drop > 0:
|
| 129 |
+
drop = (torch.rand(b, generator=generator) < cond_drop).to(dev)
|
| 130 |
+
text_emb = text_emb.masked_fill(drop[:, None], 0.0)
|
| 131 |
+
count_ids = count_ids.masked_fill(drop, 0)
|
| 132 |
+
if text_feats is not None:
|
| 133 |
+
nf, nm = null_words(text_feats, text_mask)
|
| 134 |
+
text_feats = torch.where(drop[:, None, None], nf, text_feats); text_mask = torch.where(drop[:, None], nm, text_mask)
|
| 135 |
+
logits = model(tokens.masked_fill(mask, MASK_ID), text_emb, count_ids, text_feats=text_feats, text_mask=text_mask)
|
| 136 |
+
return F.cross_entropy(logits[mask].float(), tokens[mask])
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
@torch.no_grad()
|
| 140 |
+
def mixed_bf16(model: "PixmojiTransformer") -> "PixmojiTransformer":
|
| 141 |
+
"""bf16 everywhere except the final norm + colour head (fp32): fast on AVX-512 BF16 CPUs, keeps logit resolution."""
|
| 142 |
+
model = model.to(torch.bfloat16); model.ln_f.float(); model.head.float(); return model
|
| 143 |
+
|
| 144 |
+
|
| 145 |
+
def guided_logits(teacher: PixmojiTransformer, tokens_in: torch.Tensor, text_emb: torch.Tensor, count_ids: torch.Tensor,
|
| 146 |
+
cfg: float, cfg_count: float, text_feats=None, text_mask=None) -> torch.Tensor:
|
| 147 |
+
"""Classifier-free-guided teacher logits (the serving recipe): l_u + w·(l_c − l_u), w = cfg_count for count prompts (2–4)."""
|
| 148 |
+
b = tokens_in.shape[0]
|
| 149 |
+
words = {}
|
| 150 |
+
if text_feats is not None:
|
| 151 |
+
nf, nm = null_words(text_feats, text_mask)
|
| 152 |
+
words = {"text_feats": torch.cat([text_feats, nf]), "text_mask": torch.cat([text_mask, nm])}
|
| 153 |
+
lg = teacher(torch.cat([tokens_in, tokens_in]), torch.cat([text_emb, torch.zeros_like(text_emb)]),
|
| 154 |
+
torch.cat([count_ids, torch.zeros_like(count_ids)]), **words).float()
|
| 155 |
+
l_c, l_u = lg[:b], lg[b:]
|
| 156 |
+
w = torch.where(count_ids >= 2, torch.full_like(count_ids, 0, dtype=torch.float) + cfg_count, torch.full_like(count_ids, 0, dtype=torch.float) + cfg)
|
| 157 |
+
return l_u + w[:, None, None] * (l_c - l_u)
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
def distill_loss(student: PixmojiTransformer, teacher: PixmojiTransformer, tokens: torch.Tensor, text_emb: torch.Tensor,
|
| 161 |
+
count_ids: torch.Tensor, cfg: float = 2.5, cfg_count: float = 5.0, generator: torch.Generator | None = None,
|
| 162 |
+
text_feats=None, text_mask=None, commit: torch.Tensor | None = None, steps: int = 12,
|
| 163 |
+
final_weight: float = 0.0) -> torch.Tensor:
|
| 164 |
+
"""Guidance distillation: the student's single conditional pass learns the teacher's guided distribution
|
| 165 |
+
(KL on masked positions). Off-policy (commit=None): random masks of real sprites, training schedule.
|
| 166 |
+
On-policy (commit = step each position was fixed in a recorded serving-sampler run of the teacher): the state at a
|
| 167 |
+
random step k is the recorded sample with positions fixed at >= k masked, i.e. exactly what the sampler sees."""
|
| 168 |
+
b, n = tokens.shape; dev = tokens.device
|
| 169 |
+
if commit is not None:
|
| 170 |
+
k = torch.randint(0, steps, (b, 1), generator=generator).to(dev)
|
| 171 |
+
mask = commit.to(dev) >= k
|
| 172 |
+
else:
|
| 173 |
+
u = torch.rand(b, 1, generator=generator).to(dev)
|
| 174 |
+
n_mask = (n * torch.cos(math.pi / 2 * u)).ceil().clamp(1, n)
|
| 175 |
+
mask = torch.rand(b, n, generator=generator).to(dev).argsort(1).argsort(1) < n_mask
|
| 176 |
+
inp = tokens.masked_fill(mask, MASK_ID)
|
| 177 |
+
target = torch.log_softmax(guided_logits(teacher, inp, text_emb, count_ids, cfg, cfg_count, text_feats, text_mask), -1)
|
| 178 |
+
pred = torch.log_softmax(student(inp, text_emb, count_ids, text_feats=text_feats, text_mask=text_mask).float(), -1)
|
| 179 |
+
loss = F.kl_div(pred[mask], target[mask], log_target=True, reduction="batchmean")
|
| 180 |
+
if final_weight and commit is not None:
|
| 181 |
+
# step distillation: also predict the tokens the teacher's sampler actually ended with (from the same on-policy state),
|
| 182 |
+
# so the student can commit more pixels per step coherently
|
| 183 |
+
loss = loss + final_weight * F.nll_loss(pred[mask], tokens.to(dev)[mask])
|
| 184 |
+
return loss
|
pixmoji/pipeline.py
ADDED
|
@@ -0,0 +1,155 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""End-to-end: text -> N sampled sprites -> TinyCLIP scores -> up to k selected Emoji (everything model-generated)."""
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import json
|
| 5 |
+
from dataclasses import dataclass
|
| 6 |
+
from pathlib import Path
|
| 7 |
+
|
| 8 |
+
import numpy as np
|
| 9 |
+
import torch
|
| 10 |
+
|
| 11 |
+
from pixmoji.counts import parse_count
|
| 12 |
+
from pixmoji.model import ModelConfig, PixmojiTransformer
|
| 13 |
+
from pixmoji.sampling import sample
|
| 14 |
+
from pixmoji.select import select
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
@dataclass
|
| 18 |
+
class Emoji:
|
| 19 |
+
rgba: np.ndarray
|
| 20 |
+
score: float
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def tokens_to_rgba_t(tokens: torch.Tensor, palette_t: torch.Tensor) -> torch.Tensor:
|
| 24 |
+
t = tokens.long()
|
| 25 |
+
rgb = palette_t.to(t.device)[t]
|
| 26 |
+
alpha = torch.where(t > 0, 255, 0).to(torch.uint8)[..., None]
|
| 27 |
+
out = torch.cat([rgb.to(torch.uint8), alpha], -1)
|
| 28 |
+
out = out * (t > 0)[..., None].to(torch.uint8)
|
| 29 |
+
return out.reshape(*t.shape[:-1], 16, 16, 4)
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def save_checkpoint(model: PixmojiTransformer, path, threshold: float, meta: dict | None = None) -> None:
|
| 33 |
+
torch.save({"config": model.cfg.to_dict(), "state_dict": model.state_dict(), "threshold": float(threshold), "meta": meta or {}}, path)
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def load_model(path, device: str = "cpu", dtype: torch.dtype = torch.float32) -> tuple[PixmojiTransformer, dict]:
|
| 37 |
+
path = Path(path)
|
| 38 |
+
if path.suffix == ".safetensors": # release format: weights + sibling <name>.json (config, threshold)
|
| 39 |
+
from safetensors.torch import load_file
|
| 40 |
+
meta = json.load(open(path.with_suffix(".json")))
|
| 41 |
+
ck = {"config": meta["config"], "state_dict": load_file(str(path)), "threshold": meta["threshold"]}
|
| 42 |
+
else:
|
| 43 |
+
ck = torch.load(path, map_location="cpu", weights_only=False)
|
| 44 |
+
m = PixmojiTransformer(ModelConfig(**ck["config"])); m.load_state_dict(ck["state_dict"])
|
| 45 |
+
return m.to(device=device, dtype=dtype).eval(), ck
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
class EmojiGenerator:
|
| 49 |
+
# Serving defaults chosen by the 2026-11-14 sampler sweep (blind test vs real, g5 latency): 12 steps, T 0.6,
|
| 50 |
+
# guidance 2.5 (cleanest single objects), guidance 5 when the prompt has a count (low guidance loses counts).
|
| 51 |
+
def __init__(self, model_path, palette_path, device: str = "cuda", n: int = 24, steps: int = 12, cfg: float = 2.5,
|
| 52 |
+
threshold: float | None = None, clip=None, compile: bool = False, temperature: float = 0.6, cfg_count: float = 5.0,
|
| 53 |
+
cfg_steps: int | None = None, fast_model_path=None, fast_n: int = 24, fast_temperature: float = 0.5, fast_steps: int | None = 8,
|
| 54 |
+
fast_counts: bool = False, count_n: int | None = None, count_steps: int | None = None, dtype: torch.dtype | None = None,
|
| 55 |
+
cpu_fast: bool | str = False):
|
| 56 |
+
"""fast_model_path: the on-policy guidance-distilled model (pixmoji-v2.2-fast; single pass per step) used for prompts
|
| 57 |
+
without a count >= 2, with N 24, 8 steps, T 0.5 (1/3 of the base compute; compression study 2026-11-17, rounds 9-16).
|
| 58 |
+
Count prompts keep the guided base model unchanged (distillation and fewer candidates both lost count accuracy)."""
|
| 59 |
+
# dtype: default fp16 on GPU, fp32 on CPU; torch.bfloat16 on CPUs with AVX-512 BF16 is ~2.2x faster (compression study)
|
| 60 |
+
self.device = device; dtype = dtype or (torch.float16 if device.startswith("cuda") else torch.float32)
|
| 61 |
+
self.model, ck = load_model(model_path, device, dtype)
|
| 62 |
+
if compile:
|
| 63 |
+
self.model = torch.compile(self.model)
|
| 64 |
+
self.fast_model, self.fast_n, self.fast_temperature, self.fast_steps = None, fast_n, fast_temperature, fast_steps
|
| 65 |
+
self.fast_counts, self.count_n, self.count_steps = fast_counts, count_n, count_steps # count-route options (study)
|
| 66 |
+
if fast_model_path is not None:
|
| 67 |
+
self.fast_model, _ = load_model(fast_model_path, device, dtype)
|
| 68 |
+
if compile:
|
| 69 |
+
self.fast_model = torch.compile(self.fast_model)
|
| 70 |
+
if dtype == torch.bfloat16 and self.fast_model is not None:
|
| 71 |
+
# measured (rounds 19-21): the fast route needs its final norm + colour head in fp32 (full bf16 cost Terra -4.8
|
| 72 |
+
# depicts points); the guided count route is within noise in full bf16 over 600 count prompts, so it stays full bf16.
|
| 73 |
+
from pixmoji.model import mixed_bf16
|
| 74 |
+
self.fast_model = mixed_bf16(self.fast_model)
|
| 75 |
+
# cpu_fast: torch.compile with Inductor freezing (oneDNN weight prepack, fused bias/GELU/norms), word slots padded to
|
| 76 |
+
# 3 bucket sizes (exact; bounds recompiles) and logits only for still-masked positions (exact in distribution).
|
| 77 |
+
# Measured (rounds 26-27, CPU, judges): within noise on the count route, borderline on the plain route, so
|
| 78 |
+
# cpu_fast="counts" (default when True is not requested) applies it to the count route only; "all" applies it to both.
|
| 79 |
+
# Pair with OMP_PROC_BIND=close OMP_PLACES=cores and threads within one socket.
|
| 80 |
+
self.sample_opts, self.sample_opts_count = {}, {}
|
| 81 |
+
if cpu_fast:
|
| 82 |
+
import torch._inductor.config as ic
|
| 83 |
+
ic.freezing = True
|
| 84 |
+
opts = {"word_buckets": (8, 16, 24), "masked_only": True}
|
| 85 |
+
self.model = torch.compile(self.model); self.sample_opts_count = opts
|
| 86 |
+
if cpu_fast == "all" and self.fast_model is not None:
|
| 87 |
+
self.fast_model = torch.compile(self.fast_model); self.sample_opts = opts
|
| 88 |
+
self.threshold = ck["threshold"] if threshold is None else threshold
|
| 89 |
+
self.n, self.steps, self.cfg, self.temperature, self.cfg_count, self.cfg_steps = n, steps, cfg, temperature, cfg_count, cfg_steps
|
| 90 |
+
self.palette = torch.from_numpy(np.load(palette_path, allow_pickle=False)).to(device)
|
| 91 |
+
if clip is None:
|
| 92 |
+
from pixmoji.tinyclip import TinyClip
|
| 93 |
+
clip = TinyClip(device=device, dtype=dtype)
|
| 94 |
+
self.text_fn, self.image_fn = clip.torch_modules()
|
| 95 |
+
self.clip = clip; self.text_tokens = ck["config"].get("text_tokens", 0) # >0: v2 word-level model
|
| 96 |
+
|
| 97 |
+
def words_for(self, text: str) -> dict:
|
| 98 |
+
"""Word-level features for v2+ models ({} for v1 checkpoints)."""
|
| 99 |
+
if not self.text_tokens:
|
| 100 |
+
return {}
|
| 101 |
+
from pixmoji.text import tokenize_vocab
|
| 102 |
+
ids, lens = tokenize_vocab([text])
|
| 103 |
+
f, m = self.clip.token_features(torch.from_numpy(ids), torch.from_numpy(lens))
|
| 104 |
+
return {"text_feats": f.float(), "text_mask": m}
|
| 105 |
+
|
| 106 |
+
def _encode(self, text: str):
|
| 107 |
+
"""(pooled text embedding (1, 512), word features). One text-tower pass when the prompt fits the word slots."""
|
| 108 |
+
if self.text_tokens and hasattr(self.clip, "text_and_tokens"):
|
| 109 |
+
from pixmoji.text import TEXT_TOKENS, tokenize_vocab
|
| 110 |
+
ids, lens = tokenize_vocab([text])
|
| 111 |
+
if lens[0] < TEXT_TOKENS: # not truncated: EOS state == CLIP pooled output
|
| 112 |
+
te, f, m = self.clip.text_and_tokens(torch.from_numpy(ids), torch.from_numpy(lens))
|
| 113 |
+
return te.float(), {"text_feats": f.float(), "text_mask": m}
|
| 114 |
+
return self.text_fn([text]).float(), self.words_for(text)
|
| 115 |
+
|
| 116 |
+
@torch.inference_mode()
|
| 117 |
+
def generate_emojis(self, text: str, k: int = 10, seed: int | None = None) -> list[Emoji]:
|
| 118 |
+
if not text or not text.strip():
|
| 119 |
+
return []
|
| 120 |
+
gen = torch.Generator(device=self.device).manual_seed(seed) if seed is not None else None
|
| 121 |
+
count = parse_count(text); te, words = self._encode(text)
|
| 122 |
+
if self.fast_model is not None and (count < 2 or self.fast_counts):
|
| 123 |
+
toks = sample(self.fast_model, te.to(self.device), count, self.fast_n, steps=self.fast_steps or self.steps, cfg=self.cfg,
|
| 124 |
+
temperature=self.fast_temperature, generator=gen, cfg_steps=0, **words, **self.sample_opts)
|
| 125 |
+
else:
|
| 126 |
+
cnt_route = count >= 2
|
| 127 |
+
toks = sample(self.model, te.to(self.device), count, (self.count_n or self.n) if cnt_route else self.n,
|
| 128 |
+
steps=(self.count_steps or self.steps) if cnt_route else self.steps, cfg=self.cfg_count if count >= 2 else self.cfg,
|
| 129 |
+
temperature=self.temperature, generator=gen, cfg_steps=self.cfg_steps, **words,
|
| 130 |
+
**(self.sample_opts_count if cnt_route else self.sample_opts))
|
| 131 |
+
rgba = tokens_to_rgba_t(toks, self.palette)
|
| 132 |
+
ie = self.image_fn(rgba).float()
|
| 133 |
+
scores = (ie @ te[0].to(ie.device)).cpu().numpy()
|
| 134 |
+
opaque = (toks > 0).float().mean(1).cpu().numpy()
|
| 135 |
+
idx = select(scores, ie.cpu().numpy(), opaque, k=k, threshold=self.threshold)
|
| 136 |
+
rgba_np = rgba.cpu().numpy()
|
| 137 |
+
return [Emoji(rgba_np[i], float(scores[i])) for i in idx]
|
| 138 |
+
|
| 139 |
+
|
| 140 |
+
def _resolve(repo_or_dir, filename):
|
| 141 |
+
p = Path(str(repo_or_dir)) / filename
|
| 142 |
+
if p.exists():
|
| 143 |
+
return p
|
| 144 |
+
from huggingface_hub import hf_hub_download
|
| 145 |
+
return Path(hf_hub_download(str(repo_or_dir), filename))
|
| 146 |
+
|
| 147 |
+
|
| 148 |
+
def load(repo_or_dir="model-collapse/pixmoji", device: str | None = None, dtype: torch.dtype | None = None, **kw) -> "EmojiGenerator":
|
| 149 |
+
"""Serving setup from the model card: prompts with a count >= 2 use pixmoji-v2.1 (guided); all others pixmoji-v2.2-fast
|
| 150 |
+
(single pass, 8 steps). On CPUs with AVX-512 BF16 pass dtype=torch.bfloat16 for ~2x speed."""
|
| 151 |
+
device = device or ("cuda" if torch.cuda.is_available() else "cpu")
|
| 152 |
+
for f in ("pixmoji-v2.1.json", "pixmoji-v2.2-fast.safetensors", "pixmoji-v2.2-fast.json"):
|
| 153 |
+
_resolve(repo_or_dir, f)
|
| 154 |
+
return EmojiGenerator(_resolve(repo_or_dir, "pixmoji-v2.1.safetensors"), _resolve(repo_or_dir, "palette.npy"), device=device,
|
| 155 |
+
fast_model_path=_resolve(repo_or_dir, "pixmoji-v2.2-fast.safetensors"), dtype=dtype, **kw)
|
pixmoji/sampling.py
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""MaskGIT parallel decoding with classifier-free guidance.
|
| 2 |
+
|
| 3 |
+
Sampling is memory-bound (n x 256 x 4097 logits per step), so by default tokens are drawn among the top_k colours
|
| 4 |
+
per pixel with the Gumbel-max trick in the logits' dtype; top_k=None reproduces full-vocabulary sampling.
|
| 5 |
+
"""
|
| 6 |
+
from __future__ import annotations
|
| 7 |
+
|
| 8 |
+
import math
|
| 9 |
+
|
| 10 |
+
import torch
|
| 11 |
+
|
| 12 |
+
from pixmoji.model import MASK_ID, PixmojiTransformer, null_words
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
def remaining_masked(k: int, steps: int, n: int = 256) -> int:
|
| 16 |
+
return int(math.floor(n * math.cos(math.pi / 2 * (k + 1) / steps)))
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
def _gumbel(shape, generator, device) -> torch.Tensor:
|
| 20 |
+
gen_dev = generator.device if generator is not None else device
|
| 21 |
+
u = torch.rand(shape, generator=generator, device=gen_dev).clamp_(1e-10, 1 - 1e-10)
|
| 22 |
+
return (-torch.log(-torch.log(u))).to(device)
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def _draw(logits: torch.Tensor, temperature: float, top_k: int | None, generator) -> tuple[torch.Tensor, torch.Tensor]:
|
| 26 |
+
"""Sample one token per position; return (tokens, log-prob of the drawn token under the (top-k) distribution)."""
|
| 27 |
+
t = max(temperature, 1e-6)
|
| 28 |
+
if top_k is not None and top_k < logits.shape[-1]:
|
| 29 |
+
vals, idx = logits.topk(top_k, dim=-1)
|
| 30 |
+
vals = vals.float() / t
|
| 31 |
+
else:
|
| 32 |
+
vals, idx = logits.float() / t, None
|
| 33 |
+
logp = torch.log_softmax(vals, -1)
|
| 34 |
+
choice = (logp + _gumbel(logp.shape, generator, logp.device)).argmax(-1, keepdim=True)
|
| 35 |
+
conf = logp.gather(-1, choice)[..., 0]
|
| 36 |
+
drawn = choice[..., 0] if idx is None else idx.gather(-1, choice)[..., 0]
|
| 37 |
+
return drawn, conf
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
@torch.no_grad()
|
| 41 |
+
def sample(model: PixmojiTransformer, text_emb: torch.Tensor, count_id: int, n: int, steps: int = 6, cfg: float = 3.0,
|
| 42 |
+
temperature: float = 1.0, generator: torch.Generator | None = None, top_k: int | None = 64,
|
| 43 |
+
text_feats: torch.Tensor | None = None, text_mask: torch.Tensor | None = None, cfg_steps: int | None = None, return_commit: bool = False,
|
| 44 |
+
word_buckets: tuple[int, ...] | None = None, masked_only: bool = False):
|
| 45 |
+
"""word_buckets: pad the trimmed word slots up to the next bucket size (exact: padded slots are masked) so compiled
|
| 46 |
+
models see only a few shapes. masked_only: compute logits and draws only for still-masked positions (exact in
|
| 47 |
+
distribution; ~30% less head/top-k work over a run, different random stream).
|
| 48 |
+
return_commit: also return (n, 256) the step at which each position was fixed (for on-policy distillation).
|
| 49 |
+
cfg_steps: apply classifier-free guidance only on the first cfg_steps steps (None = all); later steps run the
|
| 50 |
+
conditional pass alone (half the compute for those steps)."""
|
| 51 |
+
dev = next(model.parameters()).device
|
| 52 |
+
e = text_emb.reshape(1, -1).to(dev).expand(n, -1)
|
| 53 |
+
emb = torch.cat([e, torch.zeros_like(e)])
|
| 54 |
+
cnt = torch.cat([torch.full((n,), count_id, device=dev), torch.zeros(n, dtype=torch.long, device=dev)])
|
| 55 |
+
words, words_c = {}, {}
|
| 56 |
+
if text_feats is not None: # word-level conditioning (v2): conditional half gets the words, unconditional half the null words
|
| 57 |
+
mk = text_mask.reshape(1, -1).to(dev)
|
| 58 |
+
L = max(1, int(mk.any(0).nonzero().max()) + 1) if mk.any() else 1 # trim trailing padded slots (exact)
|
| 59 |
+
if word_buckets:
|
| 60 |
+
L = min(next((x for x in word_buckets if x >= L), L), mk.shape[1])
|
| 61 |
+
f = text_feats.reshape(1, *text_feats.shape[-2:])[:, :L].to(dev).expand(n, -1, -1)
|
| 62 |
+
mk = mk[:, :L].expand(n, -1)
|
| 63 |
+
nf, nm = null_words(f, mk)
|
| 64 |
+
words = {"text_feats": torch.cat([f, nf]), "text_mask": torch.cat([mk, nm])}
|
| 65 |
+
words_c = {"text_feats": f, "text_mask": mk}
|
| 66 |
+
tokens = torch.full((n, 256), MASK_ID, dtype=torch.long, device=dev)
|
| 67 |
+
commit = torch.full((n, 256), steps - 1, dtype=torch.long, device=dev)
|
| 68 |
+
for k in range(steps):
|
| 69 |
+
masked = tokens == MASK_ID
|
| 70 |
+
pos = None
|
| 71 |
+
if masked_only: # every row has the same number of masked positions at a given step (deterministic schedule)
|
| 72 |
+
m = int(masked[0].sum()); pos = torch.argsort(masked.to(torch.int8), dim=1, descending=True, stable=True)[:, :m]
|
| 73 |
+
if cfg_steps is None or k < cfg_steps:
|
| 74 |
+
logits = model(torch.cat([tokens, tokens]), emb, cnt, **words, **({"out_pos": torch.cat([pos, pos])} if pos is not None else {}))
|
| 75 |
+
l_c, l_u = logits[:n], logits[n:]
|
| 76 |
+
lg = l_u + cfg * (l_c - l_u) if cfg != 1.0 else l_c
|
| 77 |
+
else:
|
| 78 |
+
lg = model(tokens, emb[:n], cnt[:n], **words_c, **({"out_pos": pos} if pos is not None else {}))
|
| 79 |
+
drawn, conf = _draw(lg, temperature, top_k, generator)
|
| 80 |
+
conf = conf + temperature * (1 - (k + 1) / steps) * _gumbel(conf.shape, generator, dev)
|
| 81 |
+
if pos is not None: # scatter back to all 256 positions; fixed positions keep their token and are never re-masked
|
| 82 |
+
drawn = tokens.scatter(1, pos, drawn); conf = torch.full(tokens.shape, float("inf"), device=dev, dtype=conf.dtype).scatter(1, pos, conf)
|
| 83 |
+
tokens = torch.where(masked, drawn, tokens)
|
| 84 |
+
conf = conf.masked_fill(~masked, float("inf")) # already-fixed positions are never re-masked
|
| 85 |
+
keep_masked = remaining_masked(k, steps)
|
| 86 |
+
if keep_masked > 0:
|
| 87 |
+
tokens.scatter_(1, conf.argsort(1)[:, :keep_masked], MASK_ID)
|
| 88 |
+
if return_commit:
|
| 89 |
+
commit = torch.where(masked & (tokens != MASK_ID), torch.full_like(commit, k), commit)
|
| 90 |
+
return (tokens, commit) if return_commit else tokens
|
pixmoji/select.py
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Pick up to k candidates: relevance first (threshold + score order), diversity only as a near-tie breaker."""
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import numpy as np
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
def select(scores: np.ndarray, img_emb: np.ndarray, opaque_frac: np.ndarray, k: int = 10, threshold: float = 0.0,
|
| 8 |
+
div_weight: float = 0.05, dup_cos: float = 0.97, min_opaque: float = 0.05) -> list[int]:
|
| 9 |
+
"""Greedy MMR with mmr = score - div_weight * max_cos_to_selected.
|
| 10 |
+
|
| 11 |
+
TinyCLIP scores span ~0.2-0.4 while cosine spans 0-1, so a small div_weight keeps relevance dominant
|
| 12 |
+
(spec: representativeness first, diversity second). Never pads: returns fewer than k if few pass.
|
| 13 |
+
"""
|
| 14 |
+
scores = np.asarray(scores, np.float64); emb = np.asarray(img_emb, np.float64)
|
| 15 |
+
cand = [i for i in np.argsort(-scores, kind="stable") if opaque_frac[i] >= min_opaque and scores[i] >= threshold]
|
| 16 |
+
chosen: list[int] = []
|
| 17 |
+
max_cos = np.full(len(scores), -np.inf)
|
| 18 |
+
while cand and len(chosen) < k:
|
| 19 |
+
pen = np.where(np.isfinite(max_cos[cand]), max_cos[cand], 0.0)
|
| 20 |
+
best = cand[int(np.argmax(scores[cand] - div_weight * pen))]
|
| 21 |
+
chosen.append(int(best))
|
| 22 |
+
sims = emb[cand] @ emb[best]
|
| 23 |
+
max_cos[cand] = np.maximum(max_cos[cand], sims)
|
| 24 |
+
cand = [c for c, s in zip(cand, sims) if c != best and s < dup_cos]
|
| 25 |
+
return chosen
|
pixmoji/text.py
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""CLIP tokenisation for pixmoji's word-level conditioning (same tokenizer as TinyCLIP-PixelArt)."""
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import functools
|
| 5 |
+
|
| 6 |
+
import numpy as np
|
| 7 |
+
|
| 8 |
+
TEXT_TOKENS = 24 # word slots (incl. BOS/EOS)
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
@functools.lru_cache(maxsize=1)
|
| 12 |
+
def _tokenizer():
|
| 13 |
+
from transformers import CLIPProcessor
|
| 14 |
+
from pixmoji.tinyclip import MODEL_ID, REVISION
|
| 15 |
+
return CLIPProcessor.from_pretrained(MODEL_ID, revision=REVISION).tokenizer
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def tokenize_vocab(texts: list[str]) -> tuple[np.ndarray, np.ndarray]:
|
| 19 |
+
tok = _tokenizer(); ids = np.zeros((len(texts), TEXT_TOKENS), np.int32); lens = np.zeros(len(texts), np.int32)
|
| 20 |
+
for i, e in enumerate(tok(list(texts), truncation=True, max_length=TEXT_TOKENS)["input_ids"]):
|
| 21 |
+
ids[i, :len(e)] = e; lens[i] = len(e)
|
| 22 |
+
return ids, lens
|
pixmoji/tinyclip.py
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""TinyCLIP-PixelArt wrapper: text and 16x16 sprite embeddings (numpy and torch paths).
|
| 2 |
+
|
| 3 |
+
Uses transformers CLIPModel directly, because the checkpoint was saved with sentence-transformers 6.3.0.dev0
|
| 4 |
+
and the installed 5.x crashes in encode_query (see ledger, pre-flight T4).
|
| 5 |
+
"""
|
| 6 |
+
from __future__ import annotations
|
| 7 |
+
|
| 8 |
+
import numpy as np
|
| 9 |
+
import torch
|
| 10 |
+
import torch.nn.functional as F
|
| 11 |
+
from PIL import Image
|
| 12 |
+
from transformers import CLIPModel, CLIPProcessor
|
| 13 |
+
|
| 14 |
+
MODEL_ID = "tomaarsen/TinyCLIP-39M-PixelArt"
|
| 15 |
+
REVISION = "d25945a2a2f17be4ffd963adf3666053ff57b281" # pinned
|
| 16 |
+
# Model card: "composite the image onto an RGB (128, 128, 128) background before encoding, matching the training preprocessing".
|
| 17 |
+
IMAGE_BG = (128, 128, 128) # measured on 2,000 val rows: hit@1 0.929 / hit@10 0.999 (white 0.924, black 0.885)
|
| 18 |
+
RESIZE_MODE = "bicubic" # processor_config resample=3; 16 -> 224 shortest edge, center-crop 224 (no-op for squares)
|
| 19 |
+
_MEAN = (0.48145466, 0.4578275, 0.40821073)
|
| 20 |
+
_STD = (0.26862954, 0.26130258, 0.27577711)
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def _features(out) -> torch.Tensor:
|
| 24 |
+
"""get_*_features returns a tensor in transformers 4.x and a BaseModelOutputWithPooling (.pooler_output) in 5.x."""
|
| 25 |
+
return out if isinstance(out, torch.Tensor) else out.pooler_output
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def composite(rgba: np.ndarray) -> np.ndarray:
|
| 29 |
+
a = rgba[..., 3:4].astype(np.float32) / 255.0
|
| 30 |
+
bg = np.array(IMAGE_BG, np.float32)
|
| 31 |
+
return (rgba[..., :3].astype(np.float32) * a + bg * (1 - a)).round().astype(np.uint8)
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
class TinyClip:
|
| 35 |
+
def __init__(self, device: str = "cpu", dim: int = 512, dtype: torch.dtype = torch.float32):
|
| 36 |
+
self.device, self.dim = device, dim
|
| 37 |
+
self.model = CLIPModel.from_pretrained(MODEL_ID, revision=REVISION, dtype=dtype).to(device).eval()
|
| 38 |
+
self.processor = CLIPProcessor.from_pretrained(MODEL_ID, revision=REVISION)
|
| 39 |
+
self.size = self.processor.image_processor.crop_size["height"]
|
| 40 |
+
self._mean = torch.tensor(_MEAN, device=device, dtype=dtype).view(1, 3, 1, 1)
|
| 41 |
+
self._std = torch.tensor(_STD, device=device, dtype=dtype).view(1, 3, 1, 1)
|
| 42 |
+
|
| 43 |
+
def _norm(self, e: torch.Tensor) -> torch.Tensor:
|
| 44 |
+
return F.normalize(e[:, : self.dim].float(), dim=-1)
|
| 45 |
+
|
| 46 |
+
@torch.no_grad()
|
| 47 |
+
def _text(self, texts: list[str]) -> torch.Tensor:
|
| 48 |
+
tok = self.processor.tokenizer(list(texts), padding=True, truncation=True, max_length=77, return_tensors="pt").to(self.device)
|
| 49 |
+
return self._norm(_features(self.model.get_text_features(**tok)))
|
| 50 |
+
|
| 51 |
+
def encode_text(self, texts: list[str], batch_size: int = 1024) -> np.ndarray:
|
| 52 |
+
out = [self._text(texts[i:i + batch_size]).cpu().numpy() for i in range(0, len(texts), batch_size)]
|
| 53 |
+
return np.concatenate(out).astype(np.float32) if out else np.zeros((0, self.dim), np.float32)
|
| 54 |
+
|
| 55 |
+
@torch.no_grad()
|
| 56 |
+
def encode_images(self, rgba: np.ndarray, batch_size: int = 256) -> np.ndarray:
|
| 57 |
+
out = []
|
| 58 |
+
for i in range(0, len(rgba), batch_size):
|
| 59 |
+
ims = [Image.fromarray(x) for x in composite(rgba[i:i + batch_size])]
|
| 60 |
+
px = self.processor.image_processor(images=ims, return_tensors="pt")["pixel_values"].to(self.device, self.model.dtype)
|
| 61 |
+
out.append(self._norm(_features(self.model.get_image_features(pixel_values=px))).cpu().numpy())
|
| 62 |
+
return np.concatenate(out).astype(np.float32)
|
| 63 |
+
|
| 64 |
+
@torch.no_grad()
|
| 65 |
+
def token_features(self, ids: torch.Tensor, lengths: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
|
| 66 |
+
"""Per-token features (B, L, 512) from the text tower's final hidden state (after final layer norm) + key mask (B, L).
|
| 67 |
+
|
| 68 |
+
Padded positions are zeroed. The EOS position equals CLIP's pooled output (before projection).
|
| 69 |
+
"""
|
| 70 |
+
ids = ids.to(self.device).long(); L = ids.shape[1]
|
| 71 |
+
mask = torch.arange(L, device=ids.device)[None] < lengths.to(ids.device)[:, None]
|
| 72 |
+
out = self.model.text_model(input_ids=ids, attention_mask=mask.long())
|
| 73 |
+
h = out.last_hidden_state if hasattr(out, "last_hidden_state") else out[0]
|
| 74 |
+
return h.masked_fill(~mask[..., None], 0.0), mask
|
| 75 |
+
|
| 76 |
+
@torch.no_grad()
|
| 77 |
+
def text_and_tokens(self, ids: torch.Tensor, lengths: torch.Tensor):
|
| 78 |
+
"""One text-tower pass for both the pooled, normalised text embedding and the per-token features.
|
| 79 |
+
Matches encode_text only when the text was not truncated to TEXT_TOKENS (caller checks lengths < L)."""
|
| 80 |
+
f, mask = self.token_features(ids, lengths)
|
| 81 |
+
eos = f[torch.arange(f.shape[0], device=f.device), lengths.to(f.device).long() - 1]
|
| 82 |
+
return self._norm(self.model.text_projection(eos)), f, mask
|
| 83 |
+
|
| 84 |
+
def torch_modules(self):
|
| 85 |
+
def text_fn(texts: list[str]) -> torch.Tensor:
|
| 86 |
+
return self._text(texts)
|
| 87 |
+
|
| 88 |
+
@torch.no_grad()
|
| 89 |
+
def image_fn(rgba: torch.Tensor) -> torch.Tensor:
|
| 90 |
+
x = rgba.to(self.device).float()
|
| 91 |
+
a = x[..., 3:4] / 255.0
|
| 92 |
+
bg = torch.tensor(IMAGE_BG, device=x.device, dtype=torch.float32)
|
| 93 |
+
rgb = (x[..., :3] * a + bg * (1 - a)).round().permute(0, 3, 1, 2) # n,3,16,16 in 0..255
|
| 94 |
+
rgb = F.interpolate(rgb, size=(self.size, self.size), mode=RESIZE_MODE, align_corners=False).clamp(0, 255)
|
| 95 |
+
px = ((rgb / 255.0).to(self._mean.dtype) - self._mean) / self._std
|
| 96 |
+
return self._norm(_features(self.model.get_image_features(pixel_values=px)))
|
| 97 |
+
|
| 98 |
+
return text_fn, image_fn
|
sample_sheet.png
ADDED
|
Git LFS Details
|