mingyi456 commited on
Commit
f2c3588
·
verified ·
1 Parent(s): 37445ee

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +31 -3
README.md CHANGED
@@ -12,11 +12,13 @@ tags:
12
  - comfyui
13
  ---
14
 
15
- Thanks to @rootey for pointing out that my existing DF11 compressions do not work with ComfyUI.
 
 
16
 
17
  For more information (including how to compress models yourself), check out https://huggingface.co/DFloat11 and https://github.com/LeanModels/DFloat11
18
 
19
- Feel free to request for other models for compression as well, although compressing models that do not use the Flux architecture might be tricky for me.
20
 
21
  This compressed model was made from [rockerBOO/flux.1-dev-SRPO](https://huggingface.co/rockerBOO/flux.1-dev-SRPO)'s BF16 quantization. Thanks to rockerBOO, without which I would not have been able to directly work with. (My PC only has 48 GB of system RAM, too little to work with a 12B model in FP32 precision)
22
 
@@ -28,4 +30,30 @@ Follow the instructions here: https://github.com/LeanModels/ComfyUI-DFloat11. Af
28
  ![](SRPO-DF11-workflow.png)
29
 
30
  #### `diffusers`
31
- Use this [model](https://huggingface.co/mingyi456/SRPO-DF11) instead.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
  - comfyui
13
  ---
14
 
15
+ Thanks to @rootey for pointing out that my existing DF11 compressions do not work with ComfyUI. After I realized the issue was with the different naming schemes of the tensors, I worked on trying to translate and remap each tensor in the `FluxTransformer2D` model (in the `diffusers` library), to the `Flux.diffusion_model` model (which ComfyUI, and I believe other platforms, use), only to realize that the tensors corresponding to the DF11 compressed weights expect to be decompressed into a specific structure at compression time, and trying to adapt for it will be extremely difficult for me, although it should be theoretically possible for an expert to do.
16
+
17
+ Instead, I figured out a way to manually load an instance of the `Flux.diffusion_model` model in ComfyUI, and carry out the compression in ComfyUI instead. Unfortunately, this means that 2 separate, mutrally-incompatible versions of DF11 models need to coexist, which I personally find to be bothersome. Hence, I am currently considering not to release the custom nodes/script for producing DF11 compressed models for ComfyUI, while I research on the feasibility of adapting the existing models. In the meantime, I plan to release ComfyUI compatible DF11 compressions for only a select handful of models, but feel free to request for any of the existing DF11 models to be remade into ComfyUI format.
18
 
19
  For more information (including how to compress models yourself), check out https://huggingface.co/DFloat11 and https://github.com/LeanModels/DFloat11
20
 
21
+ Feel free to request for other models for compression as well, although compressing models that do not use the Flux architecture might be slightly tricky for me.
22
 
23
  This compressed model was made from [rockerBOO/flux.1-dev-SRPO](https://huggingface.co/rockerBOO/flux.1-dev-SRPO)'s BF16 quantization. Thanks to rockerBOO, without which I would not have been able to directly work with. (My PC only has 48 GB of system RAM, too little to work with a 12B model in FP32 precision)
24
 
 
30
  ![](SRPO-DF11-workflow.png)
31
 
32
  #### `diffusers`
33
+ Refer to this [model](https://huggingface.co/mingyi456/SRPO-DF11) instead.
34
+
35
+ ### Compression Details
36
+
37
+ This is the `pattern_dict` for compressing Flux-based models in ComfyUI:
38
+
39
+ ```python
40
+ pattern_dict_comfyui = {
41
+ "double_blocks\.\d+": (
42
+ "img_mod.lin",
43
+ "img_attn.qkv",
44
+ "img_attn.proj",
45
+ "img_mlp.0",
46
+ "img_mlp.2",
47
+ "txt_mod.lin",
48
+ "txt_attn.qkv",
49
+ "txt_attn.proj",
50
+ "txt_mlp.0",
51
+ "txt_mlp.2",
52
+ ),
53
+ "single_blocks\.\d+": (
54
+ "linear1",
55
+ "linear2",
56
+ "modulation.lin",
57
+ ),
58
+ }
59
+ ```