Update README.md
Browse files
README.md
CHANGED
|
@@ -14,57 +14,129 @@ tags:
|
|
| 14 |
|
| 15 |
# MOSS-Image Smaller Encoder β 4B to 9B Adapter
|
| 16 |
|
| 17 |
-
**An experimental, lightweight text-encoder replacement for MOSS-Image.** This
|
| 18 |
|
| 19 |
-
**GitHub β
|
| 20 |
|
| 21 |
-
> **
|
| 22 |
|
| 23 |
## How it works
|
| 24 |
|
| 25 |
-
The 4B and 9B encoders
|
|
|
|
|
|
|
| 26 |
|
| 27 |
| Component | Details |
|
| 28 |
| --- | --- |
|
| 29 |
-
| Student encoder |
|
| 30 |
-
|
|
|
|
|
| 31 |
| Projection | 2,560 β 4,096 dimensions |
|
| 32 |
-
|
|
| 33 |
-
| Adapter size |
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
|
| 36 |
-
|
| 37 |
|
| 38 |
-
|
| 39 |
|
| 40 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |

|
| 43 |
|
| 44 |
-
|
|
|
|
|
|
|
| 45 |
|
| 46 |
## Models and usage
|
| 47 |
|
| 48 |
-
|
| 49 |
|
| 50 |
-
- [MOSS-Image-Base
|
| 51 |
-
- [MOSS-Image-SFT
|
| 52 |
- [ComfyUI-format MOSS-Image files](https://huggingface.co/Comfy-Org/MOSS-Image)
|
| 53 |
|
| 54 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
|
| 56 |
-
-
|
| 57 |
|
| 58 |
-
|
| 59 |
|
| 60 |
-
|
| 61 |
|
| 62 |
-
|
| 63 |
|
| 64 |
## Credits and references
|
| 65 |
|
| 66 |
-
- **[ComfyUI-ClipProj](https://github.com/nicolab28/ComfyUI-ClipProj)** by **nicolab28** β inspiration for replacing a larger text encoder with a learned feature projection. See its [calibration methodology](https://github.com/nicolab28/ComfyUI-ClipProj/blob/main/CALIBRATION.md)
|
| 67 |
- **[MOSS-Image](https://github.com/OpenMOSS/MOSS-Image)** by the OpenMOSS team β the original image-generation model.
|
| 68 |
-
- **[Qwen3.5](https://huggingface.co/Qwen/Qwen3.5-4B)** by the Qwen team β the
|
|
|
|
| 69 |
|
| 70 |
-
This is an independent community experiment and is not affiliated with the OpenMOSS, Qwen, or ComfyUI-ClipProj developers. Refer to the licenses of the original models and software separately.
|
|
|
|
| 14 |
|
| 15 |
# MOSS-Image Smaller Encoder β 4B to 9B Adapter
|
| 16 |
|
| 17 |
+
**An experimental, lightweight text-encoder replacement for MOSS-Image.** This adapter maps features from **Qwen3.5-4B** into the conditioning space of MOSS-Image's original **Qwen3.5-9B** text encoder. The goal is to reduce memory requirements for prompt encoding while retaining useful image-generation quality.
|
| 18 |
|
| 19 |
+
**GitHub β ComfyUI node, source code, and installation instructions:** [bakapotatolord/moss-image-smaller-encoder](https://github.com/bakapotatolord/moss-image-smaller-encoder)
|
| 20 |
|
| 21 |
+
> **Current adapter: fitted on 800 prompts**, with 200 separate validation prompts and 200 held-out test prompts. This is a feature-projection adapter, **not a LoRA or a standalone text-to-image model**. It requires MOSS-Image, a compatible Qwen3.5-4B encoder, and the adapter node. Equivalent quality to the original 9B encoder is not guaranteed.
|
| 22 |
|
| 23 |
## How it works
|
| 24 |
|
| 25 |
+
The 4B and 9B encoders remain **frozen**. The same prompts are encoded using MOSS's prompt template, and token-aligned, final normalized hidden states are collected from both encoders.
|
| 26 |
+
|
| 27 |
+
A standardized affine projection is fitted using **token-weighted ridge regression** to approximate the 9B features from the 4B features. Neither encoder nor the diffusion model is fine-tuned.
|
| 28 |
|
| 29 |
| Component | Details |
|
| 30 |
| --- | --- |
|
| 31 |
+
| Student encoder | Qwen3.5-4B |
|
| 32 |
+
| Target encoder | MOSS-Image's original Qwen3.5-9B |
|
| 33 |
+
| Tested encoder checkpoints | INT8 ConvRot versions of both encoders |
|
| 34 |
| Projection | 2,560 β 4,096 dimensions |
|
| 35 |
+
| Projection parameters | Approximately 10.5 million |
|
| 36 |
+
| Adapter size | Approximately 21 MB / 20 MiB |
|
| 37 |
+
| Stored tensors | FP16 projection matrix and FP32 normalization statistics |
|
| 38 |
+
| Fitting method | Standardized, token-weighted ridge regression |
|
| 39 |
+
| ComfyUI projection | CPU FP32 computation with FP16 output rounding |
|
| 40 |
+
|
| 41 |
+
Only the projection adapter is distributed here. The Qwen encoder and MOSS-Image model weights must be downloaded separately.
|
| 42 |
+
|
| 43 |
+
## Current results β 800 fitting prompts
|
| 44 |
+
|
| 45 |
+
The current adapter uses a **1,200-prompt dataset**, divided into:
|
| 46 |
+
|
| 47 |
+
| Split | SFW prompts | Adult/NSFW prompts | Total |
|
| 48 |
+
| --- | ---: | ---: | ---: |
|
| 49 |
+
| Training | 400 | 400 | 800 |
|
| 50 |
+
| Validation | 100 | 100 | 200 |
|
| 51 |
+
| Held-out test | 100 | 100 | 200 |
|
| 52 |
+
|
| 53 |
+
Only the **800 training prompts** were used to fit the projection. Validation selected the ridge regularization strength, and the separate test set measured feature reconstruction.
|
| 54 |
+
|
| 55 |
+
The mixed corpus includes prompts from [jtatman/stable-diffusion-prompts-stats-full-uncensored](https://huggingface.co/datasets/jtatman/stable-diffusion-prompts-stats-full-uncensored). SFW/adult categories are based on source labels and text screening; they are not independently verified semantic annotations. Equal prompt counts also do not imply equal token contributions to the fit.
|
| 56 |
|
| 57 |
+
### Feature reconstruction
|
| 58 |
|
| 59 |
+
On the 200-prompt held-out test set:
|
| 60 |
|
| 61 |
+
| Metric | Result |
|
| 62 |
+
| --- | ---: |
|
| 63 |
+
| Content-token relative L2 error β | 0.4141 |
|
| 64 |
+
| Content-token cosine similarity β | 0.9082 |
|
| 65 |
+
|
| 66 |
+
These metrics measure how closely projected 4B features approximate the original 9B features. **They are not image-quality scores.**
|
| 67 |
+
|
| 68 |
+
### Image comparisons
|
| 69 |
+
|
| 70 |
+
Eight image pairs compare the current 800-prompt adapter against the original 9B encoder using four SFW prompts and two seeds. Each pair uses matching prompts, seeds, and generation settings:
|
| 71 |
+
|
| 72 |
+
- Resolution: **640 Γ 960**
|
| 73 |
+
- Sampling: **8 steps, Euler / simple**
|
| 74 |
+
- CFG: **1**
|
| 75 |
+
- MOSS Image 8-step DMD LoRA strength: **0.8**
|
| 76 |
|
| 77 |

|
| 78 |
|
| 79 |
+
The examples retain the main subjects and artistic styles, with differences in composition, typography, and incidental details. They do not establish quality parity with the original encoder.
|
| 80 |
+
|
| 81 |
+
Adult prompts were included in fitting and feature evaluation, but **adult image-generation quality has not been visually evaluated**. Quality with other encoder quantizations, longer prompts, image editing, higher CFG, and other generation settings also remains unestablished.
|
| 82 |
|
| 83 |
## Models and usage
|
| 84 |
|
| 85 |
+
### Original MOSS-Image models
|
| 86 |
|
| 87 |
+
- [MOSS-Image-Base](https://huggingface.co/OpenMOSS-Team/MOSS-Image-Base)
|
| 88 |
+
- [MOSS-Image-SFT](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT)
|
| 89 |
- [ComfyUI-format MOSS-Image files](https://huggingface.co/Comfy-Org/MOSS-Image)
|
| 90 |
|
| 91 |
+
### Smaller encoder
|
| 92 |
+
|
| 93 |
+
The tested checkpoint is [Qwen3.5-4B INT8 ConvRot](https://huggingface.co/beycanai/Qwen-LM/resolve/main/Qwen3.5-4B_int8_convrot.safetensors?download=true). Your ComfyUI installation must support its quantization format.
|
| 94 |
+
|
| 95 |
+
Use **Qwen3.5-4B**, not Qwen3-VL-4B. These are different model families.
|
| 96 |
+
|
| 97 |
+
### Download the adapter
|
| 98 |
+
|
| 99 |
+
The current adapter filename is:
|
| 100 |
+
|
| 101 |
+
`moss_qwen35_4b_to_9b_800_train.safetensors`
|
| 102 |
+
|
| 103 |
+
You can download it from the [GitHub release](https://github.com/bakapotatolord/moss-image-smaller-encoder/releases/tag/adapter-800-train). If downloading from this Hugging Face repository, select the same filename and verify the checksum below.
|
| 104 |
+
|
| 105 |
+
You only need one copy of the adapter.
|
| 106 |
+
|
| 107 |
+
<details>
|
| 108 |
+
<summary>Adapter SHA-256 checksum</summary>
|
| 109 |
+
|
| 110 |
+
```text
|
| 111 |
+
948ddeaaf051ced32314a704fa0a37e8fc2faf8440abe8e2c9a07090507ae00a
|
| 112 |
+
```
|
| 113 |
+
|
| 114 |
+
</details>
|
| 115 |
+
|
| 116 |
+
### ComfyUI setup
|
| 117 |
+
|
| 118 |
+
1. Install [moss-image-smaller-encoder](https://github.com/bakapotatolord/moss-image-smaller-encoder) in `ComfyUI/custom_nodes/`.
|
| 119 |
+
2. Place the Qwen3.5-4B checkpoint in `ComfyUI/models/text_encoders/`.
|
| 120 |
+
3. Place the adapter in `ComfyUI/models/moss_encoder_adapters/`.
|
| 121 |
+
4. Restart ComfyUI.
|
| 122 |
+
5. Select Qwen3.5-4B in **Load CLIP**, with **type** set to **moss_image**.
|
| 123 |
+
6. Add **MOSS Smaller Encoder Adapter**, select the adapter file, and connect its CLIP output to the normal **CLIP Text Encode** node.
|
| 124 |
+
|
| 125 |
+
**Load CLIP β Load LoRA (optional) β MOSS Smaller Encoder Adapter β CLIP Text Encode β KSampler**
|
| 126 |
|
| 127 |
+
Keep your usual MOSS diffusion-model and VAE connections. The GitHub repository includes a sample workflow and further compatibility details.
|
| 128 |
|
| 129 |
+
The adapter cannot be loaded through a LoRA loader, and it does not replace the complete text encoder.
|
| 130 |
|
| 131 |
+
## Development approach
|
| 132 |
|
| 133 |
+
This project is **vibe coded-ish**: AI assists with research, implementation, and experiments, while I review the proposed steps, verify the work, and inspect the results. The evaluation limits are documented above.
|
| 134 |
|
| 135 |
## Credits and references
|
| 136 |
|
| 137 |
+
- **[ComfyUI-ClipProj](https://github.com/nicolab28/ComfyUI-ClipProj)** by **nicolab28** β inspiration for replacing a larger text encoder with a smaller encoder and a learned feature projection. See its [calibration methodology](https://github.com/nicolab28/ComfyUI-ClipProj/blob/main/CALIBRATION.md).
|
| 138 |
- **[MOSS-Image](https://github.com/OpenMOSS/MOSS-Image)** by the OpenMOSS team β the original image-generation model.
|
| 139 |
+
- **[Qwen3.5](https://huggingface.co/Qwen/Qwen3.5-4B)** by the Qwen team β the encoder family used in this experiment.
|
| 140 |
+
- **[jtatman/stable-diffusion-prompts-stats-full-uncensored](https://huggingface.co/datasets/jtatman/stable-diffusion-prompts-stats-full-uncensored)** β a source of prompts for the expanded calibration corpus.
|
| 141 |
|
| 142 |
+
This is an independent community experiment and is not affiliated with the OpenMOSS, Qwen, or ComfyUI-ClipProj developers. Refer to the licenses of the original models, datasets, and software separately.
|