| --- |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: image-to-image |
| tags: |
| - image-tokenizer |
| - binary-tokenizer |
| - bitdance |
| - component-extraction |
| base_model: shallowdream204/BitDance-Tokenizer |
| --- |
| # BitDance d32c128 binary image tokenizer |
|
|
| Standalone native encoder, sign quantizer and GAN decoder from the official BitDance tokenizer release. The safetensors file is byte-for-byte identical to `ae_d32c128.safetensors`; its original SHA-256 and pinned source revision are recorded in `extraction.json`. No generation backbone is included. |
|
|
| This is the published **BitDance** codec. It is not asserted to be the semantic-distilled UniWeTok checkpoint merely because both describe 128-bit representations. |
|
|
| ## Load and reconstruct |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| import sys, torch |
| folder = snapshot_download("gatilin/bitdance_binary_tokenizer_d32c128") |
| sys.path.insert(0, folder) |
| from component import load_model |
| codec = load_model(folder) |
| # RGB pixels: NCHW, [-1, 1], spatial dimensions divisible by 32. |
| pixels = torch.linspace(-1, 1, 3*256*256).reshape(1,3,256,256) |
| with torch.inference_mode(): |
| signs = codec.encode(pixels) # [1,128,8,8], each value -1 or +1 |
| reconstructed = codec.decode(signs) |
| ``` |
|
|
| Install `requirements.txt` and run `python validate.py` in the downloaded directory. `component.py` also provides explicitly documented byte-packing helpers for storing 128 sign bits. Their byte convention is a portable bundle helper, **not an inferred mapping to upstream AR vocabulary IDs**. |
|
|
| ## Validation |
|
|
| Strict native state-dict loading, official-file SHA-256 equality, 256px encode/decode, binary alphabet, exact bit roundtrip, fixed-seed bit-decoding parity, native stochastic decoding, and rectangular 256x320 encode/decode passed on CPU in float32. Exact results and measured fixture MSE are in `validation.json`. These are component correctness tests on deterministic synthetic inputs, not claims about perceptual quality or text-to-image benchmark scores. |
|
|
| The GAN decoder injects fresh Gaussian noise even in eval mode. Set the same torch RNG seed before decoding if you need reproducible output; this native behavior is preserved. |
|
|
| The 2^128 space is represented implicitly by 128 binary channels, not a materialized embedding table. The published model uses the original convolutional implementation in `autoencoder.py`, not the separate attention-augmented `autoencoder_uni.py`. The only native-file edit changes an absolute filesystem-helper import into a package-relative import; source and code revisions are in the provenance files. |
|
|
| ## Source and license |
|
|
| - Weights: https://huggingface.co/shallowdream204/BitDance-Tokenizer |
| - Code: https://github.com/shallowdream204/BitDance |
| - Paper: https://arxiv.org/abs/2602.14041 |
|
|
| Apache-2.0; original notices are retained. Extraction, tests, loader and documentation added for this standalone distribution. |
|
|
