Image tokenizer 263701: patch VQ, 1024 codes

README.md

@@ -0,0 +1,84 @@
+---
+license: apache-2.0
+tags:
+- image-tokenizer
+- vector-quantization
+- vq
+library_name: numpy
+---
+
+# Image tokenizer — patch VQ, 1024 codes
+
+Turns an image into 1,024 discrete tokens, and back into an approximation of
+the image. Model `263701`.
+
+```python
+from imagetok import PatchVQTokenizer
+from PIL import Image
+
+tok = PatchVQTokenizer.load("imagetok_1024.npz")
+ids = tok.encode(Image.open("photo.jpg"))   # 1024 ints, each 0..1023
+back = tok.decode(ids)                      # a PIL image
+```
+
+## It is lossy, and that is not a bug to be fixed
+
+A text tokenizer is exactly invertible — byte-level BPE returns the bytes it was
+given, and anything less is broken. An image tokenizer cannot do that and
+nothing will make it: a photograph is millions of continuous values and a token
+is one integer out of a thousand. The mapping discards information by design.
+
+So fidelity is reported as a **quantity**, not a pass mark. There is no pass
+mark, and whether coarse is acceptable depends on what the tokens are for.
+
+## Measured on photographs it never saw
+
+| | |
+|---|---|
+| tokens per image | 1,024 (32×32 grid of 8×8 patches, 256×256 input) |
+| bits per token | 10 |
+| size as tokens | 1,280 bytes |
+| size as raw pixels | 196,608 bytes |
+| compression | **154× smaller** |
+| reconstruction | **23.4 dB**, mean absolute error 13.0/255 |
+| codebook used | 970 of 1,024 (95%), usage perplexity 462 |
+
+## What the numbers say about where the loss comes from
+
+Quadrupling the codebook from 256 to 1024 buys **0.9 dB** (22.5 → 23.4). Halving
+the patch size buys **2.3 dB** and costs four times the tokens:
+
+| patch | tokens/image | reconstruction |
+|---|---|---|
+| 16×16 | 256 | 21.6 dB |
+| 8×8 | 1,024 | 23.9 dB |
+| 4×4 | 4,096 | 26.6 dB |
+
+So the loss is dominated by the **patch grid**, not the codebook. An 8×8 patch
+cannot represent an edge crossing it at an angle however many codes exist, which
+is the visible failure — straight edges go blocky while colour and layout
+survive.
+
+That also bounds anything built on top: a model that generates these tokens can
+never produce an image better than this tokenizer can reconstruct. If you are
+using this as the front end of a generative or editing model, 23.4 dB is your
+ceiling, and a learned convolutional VQ-VAE at the same token count would raise
+it — the encoder can place information anywhere in the patch instead of matching
+against a fixed dictionary.
+
+## Method
+
+k-means over patches drawn from the training corpus, k-means++ seeded so that a
+corpus of mostly-sky does not put every code in the sky. Nearest code wins and
+its index is the token. This is what a VQ-VAE does with a neural encoder in
+front; without one it stays inspectable — every token is a patch you can look
+at, so codebook collapse is visible rather than theoretical.
+
+Trained on 272 CC-licensed photographs from Wikimedia Commons, held out 48 for
+the numbers above. A larger and more varied corpus would move these figures;
+they are reported on what was actually used.
+
+## Licence
+
+Apache 2.0. `imagetok.py` and `metric.py` are included so the numbers can be
+reproduced rather than taken on trust.