README: split q/k/v MOSS-Image checkpoints

README.md

@@ -0,0 +1,78 @@
+---
+tags:
+- comfyui
+- diffusion-single-file
+base_model:
+- OpenMOSS-Team/MOSS-Image-Base
+- OpenMOSS-Team/MOSS-Image-SFT
+license: apache-2.0
+---
+
+# MOSS-Image — split q/k/v (non-fused) checkpoints
+
+The same MOSS-Image weights as [Comfy-Org/MOSS-Image](https://huggingface.co/Comfy-Org/MOSS-Image),
+packed for ComfyUI with each attention layer's **q, k and v kept as separate tensors** instead of
+fused into one `qkv` tensor. Use them to check whether a LoRA or LoKr behaves differently on the
+fused files, or as the base for LoRAs trained on separate q/k/v projections.
+
+Original model repositories:
+
+- https://huggingface.co/OpenMOSS-Team/MOSS-Image-Base
+- https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT
+
+## Files
+
+```
+📂 ComfyUI/
+├── 📂 models/
+│   ├── 📂 diffusion_models/
+│   │   ├── moss_image_base_bf16_split_qkv.safetensors
+│   │   └── moss_image_sft_bf16_split_qkv.safetensors
+```
+
+| File | SHA-256 |
+| --- | --- |
+| `diffusion_models/moss_image_base_bf16_split_qkv.safetensors` | `65a698df38fbee8307a16059958f6dd1fef0db69c1e3805fe13ec1498cb0499e` |
+| `diffusion_models/moss_image_sft_bf16_split_qkv.safetensors` | `9373f43d0fdad7d6d62b7dc2d96b2a048aab47f9c6e7befe11456b5ec26f70fc` |
+
+Only the diffusion models are here. Get the text encoder (`text_encoders/`) and VAE (`vae/`) from
+[Comfy-Org/MOSS-Image](https://huggingface.co/Comfy-Org/MOSS-Image); they are unchanged. Load these
+files with **Load Diffusion Model** in place of the Comfy-Org ones. Everything else in the workflow
+stays the same.
+
+## How they differ from Comfy-Org/MOSS-Image
+
+|  | Comfy-Org `moss_image_*_bf16` | This repo `moss_image_*_bf16_split_qkv` |
+| --- | --- | --- |
+| Attention weights per layer | `layers.N.proj.qkv.weight` `[9216, 3072]` | `layers.N.proj.q.weight`, `.k.weight`, `.v.weight`, each `[3072, 3072]` |
+| Tensors | 379 | 463 (42 layers × 3 instead of 1) |
+| Everything else | bf16 | identical |
+
+The weights are the same numbers in a different layout: for every layer, `cat(q, k, v)` here
+equals `qkv` in the Comfy-Org file, bit for bit. Spot-checked on layers 0 and 41, plus the
+output projection, for both Base and SFT. ComfyUI loads either layout as is, because the MOSS
+model detects fused or split q/k/v from the keys. Fused is one matmul per layer instead of three,
+so it can be a little faster. Image quality is the same.
+
+## Why this matters for LoRAs and LoKrs
+
+Trainers such as AI Toolkit train adapters against **separate** `q`, `k` and `v` layers. On the
+fused Comfy-Org files, ComfyUI has to map each of those adapters onto a third of the fused `qkv`
+weight:
+
+- **Load LoRA (normal patching):** the adapter is applied to its slice of `qkv`. This works, but
+  only for adapter keys named `diffusion_model.layers.N.proj.{q,k,v}` or
+  `lora_unet_layers_N_proj_{q,k,v}`. Keys under any other name are skipped, with a
+  `lora key not loaded` line in the console.
+- **Load LoRA (Bypass):** adapters that target part of a fused weight are dropped. The console
+  logs `[BypassLoRA] Adapter key not in model state_dict` for each one, so a LoRA or LoKr loses
+  all of its attention q/k/v changes. (ComfyUI `fc04da91`: the patch for a slice is stored under
+  a `(key, slice)` tuple, which the bypass loader looks up as a plain state-dict key.)
+
+With the split checkpoints every q/k/v adapter targets its own tensor directly, with no slicing,
+in either loader. If a LoRA or LoKr looks noticeably different in ComfyUI than in the trainer's
+own samples, try it on these files and check the console for the two messages above.
+
+## Licence
+
+Apache 2.0, as the original MOSS-Image release.