File size: 2,909 Bytes
fb1af88 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | ---
license: apache-2.0
library_name: pytorch
pipeline_tag: image-to-image
tags:
- image-tokenizer
- binary-tokenizer
- bitdance
- component-extraction
base_model: shallowdream204/BitDance-Tokenizer
---
# BitDance d32c128 binary image tokenizer
Standalone native encoder, sign quantizer and GAN decoder from the official BitDance tokenizer release. The safetensors file is byte-for-byte identical to `ae_d32c128.safetensors`; its original SHA-256 and pinned source revision are recorded in `extraction.json`. No generation backbone is included.
This is the published **BitDance** codec. It is not asserted to be the semantic-distilled UniWeTok checkpoint merely because both describe 128-bit representations.
## Load and reconstruct
```python
from huggingface_hub import snapshot_download
import sys, torch
folder = snapshot_download("gatilin/bitdance_binary_tokenizer_d32c128")
sys.path.insert(0, folder)
from component import load_model
codec = load_model(folder)
# RGB pixels: NCHW, [-1, 1], spatial dimensions divisible by 32.
pixels = torch.linspace(-1, 1, 3*256*256).reshape(1,3,256,256)
with torch.inference_mode():
signs = codec.encode(pixels) # [1,128,8,8], each value -1 or +1
reconstructed = codec.decode(signs)
```
Install `requirements.txt` and run `python validate.py` in the downloaded directory. `component.py` also provides explicitly documented byte-packing helpers for storing 128 sign bits. Their byte convention is a portable bundle helper, **not an inferred mapping to upstream AR vocabulary IDs**.
## Validation
Strict native state-dict loading, official-file SHA-256 equality, 256px encode/decode, binary alphabet, exact bit roundtrip, fixed-seed bit-decoding parity, native stochastic decoding, and rectangular 256x320 encode/decode passed on CPU in float32. Exact results and measured fixture MSE are in `validation.json`. These are component correctness tests on deterministic synthetic inputs, not claims about perceptual quality or text-to-image benchmark scores.
The GAN decoder injects fresh Gaussian noise even in eval mode. Set the same torch RNG seed before decoding if you need reproducible output; this native behavior is preserved.
The 2^128 space is represented implicitly by 128 binary channels, not a materialized embedding table. The published model uses the original convolutional implementation in `autoencoder.py`, not the separate attention-augmented `autoencoder_uni.py`. The only native-file edit changes an absolute filesystem-helper import into a package-relative import; source and code revisions are in the provenance files.
## Source and license
- Weights: https://huggingface.co/shallowdream204/BitDance-Tokenizer
- Code: https://github.com/shallowdream204/BitDance
- Paper: https://arxiv.org/abs/2602.14041
Apache-2.0; original notices are retained. Extraction, tests, loader and documentation added for this standalone distribution.
|