---
license: mit
library_name: pytorch
pipeline_tag: image-to-image
base_model: ByteVisionLab/TokenFlow
tags:
- tokenflow
- image-tokenizer
- unified-multimodal
- component-extraction
---
# TokenFlow CLIP-B 32K with enhanced RGB decoder

The complete official `tokenflow_clipb_32k_enhanced.pt` model mapping: trained CLIP-B/16 semantic encoder, pixel encoder, both codebooks, multi-scale residual quantizer, semantic decoder and enhanced RGB decoder. Source weights: [ByteVisionLab/TokenFlow](https://huggingface.co/ByteVisionLab/TokenFlow/tree/5924ad7f25fe030292de3995912b895ab9fd9321). All 649 tensor entries retain original names, dtypes and values.

```python
from huggingface_hub import snapshot_download
import sys, torch
folder = snapshot_download('gatilin/tokenflow_clipb_32k_visual_codec')
sys.path.insert(0, folder)
from component import load_model
codec = load_model(folder)
x = torch.zeros(1, 3, 256, 256) # RGB NCHW in [-1, 1]
with torch.inference_mode():
    quantized, losses, ids = codec.encode(x)
    semantic_features, reconstructed_rgb = codec.decode_code(ids)
```

This release follows the official enhanced-decoder reconstruction script with `teacher=clipb_224`, `infer_interpolate=True`, and 32768 codes. Scales are `[1,2,4,6,8,10,12,14,16]`; IDs concatenate the square grids in that order. Both semantic and pixel codebook vectors are required to reconstruct the residual representation. The trained CLIP visual weights are already included; the adapter constructs the exact original visual architecture and strictly loads them without an extra CLIP weight download.

Native classes are pinned to source revision `9812da01c0695f4989060cc4eea053715e9e4441`. The two adaptations are documented: parameter device/dtype helper compatibility, and construction of the already-checkpointed CLIP branch without its redundant network download. Original tensor computation classes are retained. See `source_reconstruction.sh`, `code_provenance.json` and `validation.json`.

Validation covers all source tensor payload hashes, strict loading, nine-scale ID geometry and bounds, residual lookup reconstruction, finite semantic/RGB outputs, and exact decode-code parity with canonical lookup decoding. Synthetic reconstruction does not establish image quality or full MLLM/VQA/text-to-image performance. Weight-card MIT terms and source-code Apache-2.0 terms are recorded separately in `LICENSE` and `CODE_LICENSE`; original copyright notices remain in the native files.

`CLIP_LICENSE` separately preserves the original OpenAI MIT license and attribution for the bundled CLIP-derived visual transformer.