| --- |
| library_name: diffusers |
| base_model: black-forest-labs/FLUX.1-dev |
| license: other |
| license_name: flux-1-dev-non-commercial-license |
| license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md |
| tags: |
| - lora |
| - flux |
| - compression |
| - text-to-image |
| - diffusers |
| --- |
| |
| # LITHE adapters for FLUX.1-dev |
|
|
| This repository holds the 110 rank-16 FLUX.1-dev adapters evaluated in the NeurIPS 2026 paper "LITHE: Lattice-Indexed Twin Hadamard Encoding for Diffusion Personalization" (to appear), stored in the LITHE index format. Code: https://github.com/JianJiang1995/LITHE |
|
|
| ## What the files are |
|
|
| Each `.lithe` file is one adapter. It contains no float weights: every 64-value block of every LoRA factor is stored as three 9-bit codeword IDs into a fixed Hadamard codebook that is rebuilt from a closed-form rule at load time (nothing is learned or shipped besides the IDs). The ID stream (uint16) is compressed with LZMA (preset 9, extreme) and preceded by a small compressed JSON header with the module names, factor shapes, codebook parameters, checksums and the training metadata listed in `manifest.csv`. |
|
|
| * Configuration: Kohya sd-scripts FLUX layout, 304 fused linears per adapter (attention, MLP and modulation linears of all 19 double-stream and 38 single-stream blocks; header field `module_set = kohya-304`), rank 16, alpha 16, K = 512 codewords, 3 residual levels, block size 64. This is the configuration used for the quality results of the paper. |
| * Size: 3,589,632 indices per adapter; 3.23 MB per file on average (355.7 MB for all 110). The decoded bf16 LoRA is about 153 MB. |
| * The adapters were trained natively in the discrete space (straight-through estimator), not quantised after training. |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/JianJiang1995/LITHE.git && cd LITHE |
| pip install -e ".[flux]" |
| huggingface-cli download Fanjue/LITHE --local-dir ./lithe_adapters |
| python scripts/decode_to_lora.py --lithe ./lithe_adapters/indices/dreambooth30/db_cat.lithe --out db_cat_lora.safetensors |
| ``` |
|
|
| The decoded file uses Kohya LoRA keys, which the stock diffusers loader converts on load: |
|
|
| ```python |
| import torch |
| from diffusers import FluxPipeline |
| |
| pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16).to("cuda") |
| pipe.load_lora_weights("db_cat_lora.safetensors") |
| image = pipe("a photo of a <v_sub_001> cat on a beach", num_inference_steps=28, guidance_scale=3.5).images[0] |
| ``` |
|
|
| Or decode in memory without writing a file: |
|
|
| ```python |
| import lithe |
| sd = lithe.to_lora_state_dict(lithe.load_lithe("lithe_adapters/indices/dreambooth30/db_cat.lithe")) |
| pipe.load_lora_weights(sd) |
| ``` |
|
|
| `scripts/infer.py --mode peft` and `scripts/serve_demo.py` in the code repository also accept these files. Each file's trigger token and an example prompt are in its header metadata and in `manifest.csv`. The paper evaluated at 1024 x 1024, 28 steps, guidance 3.5. |
|
|
| ## Contents |
|
|
| | Pool | Folder | Adapters | Trigger tokens | Training | Mean size | Notes | |
| |---|---|---|---|---|---|---| |
| | DreamBooth-30 | `indices/dreambooth30/` | 30 | `<v_sub_001>` ... `<v_sub_030>` | 500 steps, 512x512 | 3.31 MB | Subjects of the public DreamBooth dataset (objects and pets), 4 to 6 images each. | |
| | CC101-20 | `indices/cc101_20/` | 20 | `<v_sub_C001>` ... `<v_sub_C020>` | 500 steps, 512x512 | 3.32 MB | 20 subjects sampled from the public CustomConcept101 dataset, one per category, 3 to 10 images each. | |
| | SCIdentity | `indices/scidentity/` | 20 | `<v_id_001>` ... `<v_id_020>` | 500 steps, 512x512 | 3.27 MB | 20 synthetic animal identities (12 generated images each; images not released). | |
| | SC-40 | `indices/sc40/` | 40 | `<v_outfit_001>` ... `<v_outfit_008>`; `<v_light_001>` ... `<v_light_008>`; `<v_pose_001>` ... `<v_pose_008>`; `<v_pstyle_001>` ... `<v_pstyle_008>`; `<v_anime_001>` ... `<v_anime_008>` | 750 steps, 512x512 | 3.11 MB | 40 synthetic concept-transfer targets: 8 outfits, 8 lighting setups, 8 poses, 8 painter styles, 8 anime styles (20 generated images each; images not released). | |
|
|
| Prompt pattern: identity adapters were trained with a single caption such as `a photo of a <v_sub_001> cat`. SC-40 adapters were trained on per-image captions that contain the trigger and the concept label; `manifest.csv` gives the evaluation prompt used for each (SC-40 groups: anime style 8, lighting 8, outfit 8, painter style 8, pose 8). |
|
|
| `manifest.csv` columns: pool, file, concept_id, trigger, description, example_prompt, module_set, n_modules, rank, n_indices, bytes, sha256, train_steps, train_resolution. `SHA256SUMS` lists checksums of all files. |
| |
| ## Data and license |
| |
| * The adapters are derived from FLUX.1-dev and inherit the FLUX.1 [dev] Non-Commercial License. Use them only under its terms. |
| * DreamBooth-30 adapters were trained on the DreamBooth dataset (Ruiz et al., CVPR 2023, https://github.com/google/dreambooth). CC101-20 adapters were trained on CustomConcept101 (Kumari et al., CVPR 2023, https://github.com/adobe-research/custom-diffusion). The images are not redistributed here; see the original datasets for their terms. |
| * SCIdentity and SC-40 adapters were trained on synthetic images generated for the paper. Their training data is not released; the generation protocol is described in the paper appendix. |
| |
| ## Citation |
| |
| ```bibtex |
| @inproceedings{jiang2026lithe, |
| title = {{LITHE}: Lattice-Indexed Twin Hadamard Encoding for Diffusion Personalization}, |
| author = {Jiang, Jian and Celiktutan, Oya and Wang, Yaohui and Ban, Yutong}, |
| booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, |
| year = {2026}, |
| note = {To appear. Code: https://github.com/JianJiang1995/LITHE} |
| } |
| ``` |
| |
