indepth-10M / README.md
DexagonAI's picture
Upload folder using huggingface_hub
937135c verified
|
Raw History Blame Contribute Delete
8.92 kB
---
library_name: pytorch
pipeline_tag: text-to-image
tags:
- pixel-art
- text-to-image
- sprite
- rectified-flow
- diffusion
- game-assets
- dit
datasets:
- Angais/pixel-art-atlas-1.4m
---
# InDepth (indepth-10M)
**InDepth** is a tiny text-to-sprite model: type a short caption like *"cat wearing green crown"* or *"ghost cat with glowing eyes"* and it paints a **16×16 RGBA pixel sprite** — fully offline, in ~10 seconds on CPU, in any browser. It is the engine behind the single-file [InDepth.html](https://huggingface.co/DexagonAI/indepth-10M/blob/main/InDepth.html) demo page (no build, no libraries, no network).
> InDepth is absolutely not suitable for professional workflows and merely shows how small an image generation model can get.
![card samples](samples/card25_grid.png)
*25 prompts, 16 steps, CFG 1.5, seed 13372000 — straight from the safetensors weights, no cherry-picking.*
## Model details
| | |
|---|---|
| **Output** | 16×16 RGBA (premultiplied handled at decode; transparent background) |
| **Parameters** | 10,040,336 total — 8,052,112 DiT + 1,463,936 text encoder (incl. 524,288 subword embedding) |
| **Architecture** | DiT (patch 2×2 → 64 tokens, width 256, 8 blocks, 4 heads, adaLN-Zero, 2D sincos positions) |
| **Text encoder** | 8,192-word vocab + hashed FNV-1a char-trigram subwords (4,096 buckets), 2-layer transformer width 128, masked-mean pooling |
| **Flow** | Rectified flow (linear interpolation, uniform velocity target), Euler sampling |
| **Conditioning** | Caption embedding into adaLN; caption dropout 10% during training → CFG at inference |
| **Weights (shipped)** | int8 per-row 90th-percentile quantization + sparse fp32 outlier residuals (9.7 MB on disk, ~48 MB in memory as float32) |
| **Training data** | Captions and sprites built on [Angais/pixel-art-atlas-1.4m](https://huggingface.co/datasets/Angais/pixel-art-atlas-1.4m) plus synthetic compositional prompts |
## Training
- **Data**: 3,114,709 image–caption pairs —
- 1,409,404 generated sprites for every caption row of the pixel-art-atlas-1.4m dataset,
- 300,000 synthetic compositional prompts (grammar-built *subject + color + accessory* combos, e.g. "frog holding golden sword") with matching images,
- 1,405,305 original dataset images with their captions.
- **Recipe**: 150,000 steps, batch 256, AdamW (lr 2e-4, 2k warmup, cosine decay), bf16 autocast, EMA 0.9999, grad clip 1.0, caption dropout 10%, seed-pinned.
- **Hardware**: single NVIDIA RTX PRO 6000 (Blackwell), ~6,800 samples/s. Wall time ≈ 1.6 h for train + eval.
## Evaluation
Evaluated against the previous release on 2,000 fresh disjoint prompts and 1,000 held-out compositional prompts (reference images from the training generation pipeline):
| Metric | v2 shipped (500K pairs) | **InDepth v4 (3.11M pairs)** |
|---|---|---|
| Flow loss — fresh prompts | 0.17221 | **0.16428** |
| Flow loss — compositional | 0.20677 | **0.16323** |
| NN-RMSE vs reference — compositional | 92.25 | **87.99** |
| Mask IoU vs reference — compositional | 0.7072 | **0.7584** |
![compositional](samples/compositional_grid.png)
*Held-out compositional prompts — reference / v2 / v4. The 3.1M-pair mix closes most of the gap to the reference.*
![attribute binding](samples/attribute_binding_test.png)
*Attribute binding (v2 left, v4 right): improved, but color→accessory binding is still the weakest skill.*
## Side-by-side at 16 steps vs same-dataset models
The same 15 prompts against three open pixel-art models trained on the same dataset, each at its shipped sampling settings — [tessera-small](https://huggingface.co/TheJagStudio/tessera-small) (47.6M, 16 steps / CFG 3.0), [UbaNova-16](https://huggingface.co/ubanova/UbaNova-16) (42.4M autoregressive, 256 sequential steps, temperature 1.0 / top-p 0.9 / CFG 3.0) and [nyx-image-23m](https://huggingface.co/Bhoidy/nyx-image-23m) (23.2M discrete masked diffusion, 16 steps / CFG 3.0). InDepth runs at 16 steps / CFG 1.5, seed 13372000:
![comparison](samples/comparison_all.png)
InDepth holds its own against all three: it matches their composition on most prompts (crown, potion, rocket ship, cherry blossom tree, spellbook, crystal, ice cream, skull, cheeseburger, dragon), matches or beats them on color-to-object binding (crown gems, potion, crystal), and adds transparent backgrounds where tessera-small paints opaque fills. tessera-small keeps an edge on subtle shading as the 47.6M original; nyx's 513-color palette gives it richer shading on dense objects; UbaNova-16 draws the sharpest single-object edges.
### What that costs in parameters
| Model | Image-generation params | Text encoder | Total |
|---|---:|---|---:|
| tessera-small | 47.6M | ~63M frozen CLIP ViT-B/32 | ~111M |
| UbaNova-16 | 42.4M | ~63M frozen CLIP ViT-B/32 | ~105M |
| nyx-image-23m | 23.2M | ~63M frozen CLIP ViT-B/32 | ~86M |
| **InDepth (this model)** | **8.05M** | **1.46M trained from scratch** | **10.04M** |
Every rival leans on a ~63M-parameter frozen CLIP text tower. InDepth's text encoder is trained from scratch in 1.46M parameters — its whole 10.04M budget is about 8.6× smaller than the next-smallest competitor's total (~86M), and its image-generation stack alone (8.05M) is 2.9× smaller than nyx's generator. InDepth is also the only one of the four that runs as a pure-JS in-browser engine with no install.
## Usage
### Python (PyTorch)
```python
from run_indepth import main # see repo: CLI, loads weights + vocab automatically
main() # or:
```
```bash
pip install torch safetensors pillow huggingface_hub
python run_indepth.py "cat wearing green crown" --steps 4 --cfg 1.5 --seed 13372000 --out sprites/
```
Or programmatically:
```python
import torch
from modeling_indepth import IndepthConfig, IndepthModel, generate, load_indepth, decode_to_rgba
from tokenizer import IndepthTokenizer
model, config, directory = load_indepth("DexagonAI/indepth-10M") # or a local path
tokenizer = IndepthTokenizer(directory)
tokens, subwords = [torch.tensor(r, dtype=torch.long) for r in tokenizer.batch(["cat wearing green crown"])]
image = generate(model, config, tokens, subwords, seed=13372000, steps=4, guidance=1.5, device="cpu")
sprite = decode_to_rgba(image)[0] # (16, 16, 4) uint8 — save with PIL, scale x8 NEAREST for viewing
```
- `steps`: 1–16 (4 is the sweet spot; 1-step is bit-exact with the embedded PyTorch reference).
- `guidance`: 1.0 = off (wild), 1.5 = default, 2.0–2.5 = stronger prompt adherence for two-attribute prompts.
- Prompts: lowercase, short works best; unknown words still paint something (character-trigram fallback).
### Browser (no install)
Download **[InDepth.html](https://huggingface.co/DexagonAI/indepth-10M/blob/main/InDepth.html)** (24 MB, single file) and open it in any modern browser — it ships the same int8 weights, a pure-JS inference engine, a live step-by-step "watch it paint" view, and a PyTorch parity self-test that runs on load (1-step bit-exact, 4-step ≤ 12/255 float32-vs-float64 drift). Works offline; runs in ~48 MB of memory.
### File map
| File | Purpose |
|---|---|
| `model.safetensors` | EMA weights, float32, 127 tensors, 10,040,336 params |
| `config.json` | Architecture hyperparameters |
| `vocab.json` | 8,192-word vocabulary (+ pad/unk) and tokenizer settings |
| `modeling_indepth.py` | Self-contained model definition + CFG sampler |
| `tokenizer.py` | Word + FNV-1a trigram tokenizer (bit-exact with the JS engine) |
| `run_indepth.py` | CLI / library generation script |
| `InDepth.html` | The complete offline browser demo (weights embedded) |
| `samples/` | Sample grids used in this card |
## Limitations
- **Attribute binding**: two-attribute prompts ("X wearing C Y") improved substantially over the previous release but are not solved — colors sometimes bind to the whole subject rather than the accessory.
- **16×16 only**: no resolution flexibility; details are limited by 256 pixels per sprite.
- **Short captions**: the 2-layer text encoder saturates beyond ~7–8 words.
## Credits
Model and InDepth page by **Zwiq**. Caption dataset: [Angais/pixel-art-atlas-1.4m](https://huggingface.co/datasets/Angais/pixel-art-atlas-1.4m).
Models compared on this card (both Apache-2.0, trained on the same dataset):
- [nyx-image-23m](https://huggingface.co/Bhoidy/nyx-image-23m) by Sednium Technologies ([@Bhoidy](https://huggingface.co/Bhoidy))
- [UbaNova-16](https://huggingface.co/ubanova/UbaNova-16) by Muhammad Fahril Irkham ([@UbaNova](https://huggingface.co/ubanova))
## Reproducibility
Default seed 13372000. The `InDepth.html` page self-verifies on load: 1-step no-CFG sampling must be bit-exact (max diff 0/255) against a PyTorch float64 reference embedded at export; the 4-step CFG path documents its float32 drift (≤ 12/255). The browser engine and the PyTorch tokenizer share identical FNV-1a trigram hashing, verified bit-exact.