BakaPotatoLord commited on
Commit
5c5b3dd
Β·
verified Β·
1 Parent(s): 59cb66f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +96 -24
README.md CHANGED
@@ -14,57 +14,129 @@ tags:
14
 
15
  # MOSS-Image Smaller Encoder β€” 4B to 9B Adapter
16
 
17
- **An experimental, lightweight text-encoder replacement for MOSS-Image.** This repository contains a learned projection that maps features from **Qwen3.5-4B** into the conditioning space of MOSS-Image's original **Qwen3.5-9B** text encoder. The goal is to reduce the memory required for prompt encoding while retaining useful image-generation quality.
18
 
19
- **GitHub β€” loader, source code, and adapter weights:** [bakapotatolord/moss-image-smaller-encoder](https://github.com/bakapotatolord/moss-image-smaller-encoder)
20
 
21
- > **Early proof of concept β€” fitted on just 200 prompts.** This is a projection adapter, **not** a LoRA or a standalone text-to-image model. It requires MOSS-Image, a compatible 4B encoder, and an integration/loader. Equivalent quality to the original 9B encoder is not guaranteed.
22
 
23
  ## How it works
24
 
25
- The 4B and 9B encoders are kept **frozen**. For a corpus of prompts, token-aligned hidden states are collected from both encoders, and **ridge regression** is used to learn a linear mapping from the 4B representation to the 9B representation.
 
 
26
 
27
  | Component | Details |
28
  | --- | --- |
29
- | Student encoder | [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) |
30
- | Original / target encoder | Qwen3.5-9B, used by MOSS-Image |
 
31
  | Projection | 2,560 β†’ 4,096 dimensions |
32
- | Parameters | ~10.5 million |
33
- | Adapter size | ~20 MiB in FP16 |
34
- | Training method | Token-aligned ridge regression; no encoder fine-tuning |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
35
 
36
- Only the small projection is distributed here; the source encoder and MOSS-Image model weights are not included.
37
 
38
- ## Initial results β€” 200 fitting prompts
39
 
40
- The initial adapter was fitted using **just 200 prompts**, with **50 additional prompts for validation and 50 for testing** (300 prompts total). Despite this limited calibration set, early side-by-side generations show encouraging preservation of subjects, artistic styles, and some text-rendering behavior compared with the original 9B encoder.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41
 
42
  ![Original 9B conditioning vs projected 4B conditioning](./comparison.jpeg)
43
 
44
- These results are **qualitative and preliminary**, not evidence of quality parity with the original encoder. Future experiments will expand the calibration corpus, assess generalization across diverse held-out prompts, and investigate improved feature alignment.
 
 
45
 
46
  ## Models and usage
47
 
48
- **Original MOSS-Image models**
49
 
50
- - [MOSS-Image-Base (official)](https://huggingface.co/OpenMOSS-Team/MOSS-Image-Base)
51
- - [MOSS-Image-SFT (official)](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT)
52
  - [ComfyUI-format MOSS-Image files](https://huggingface.co/Comfy-Org/MOSS-Image)
53
 
54
- **Smaller encoder**
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55
 
56
- - [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B)
57
 
58
- **Adapter availability**
59
 
60
- The projection weights are available **both on GitHub and on this Hugging Face model page**. Download from either source; a separate Hugging Face download is **not required** if you already have the adapter from GitHub.
61
 
62
- For installation, loader integration, and supported workflows, consult the [GitHub project](https://github.com/bakapotatolord/moss-image-smaller-encoder). The adapter cannot be loaded directly as a LoRA or used as a replacement for the complete text encoder.
63
 
64
  ## Credits and references
65
 
66
- - **[ComfyUI-ClipProj](https://github.com/nicolab28/ComfyUI-ClipProj)** by **nicolab28** β€” inspiration for replacing a larger text encoder with a learned feature projection. See its [calibration methodology](https://github.com/nicolab28/ComfyUI-ClipProj/blob/main/CALIBRATION.md) for the original token-aligned ridge-regression approach.
67
  - **[MOSS-Image](https://github.com/OpenMOSS/MOSS-Image)** by the OpenMOSS team β€” the original image-generation model.
68
- - **[Qwen3.5](https://huggingface.co/Qwen/Qwen3.5-4B)** by the Qwen team β€” the smaller encoder family used in this experiment.
 
69
 
70
- This is an independent community experiment and is not affiliated with the OpenMOSS, Qwen, or ComfyUI-ClipProj developers. Refer to the licenses of the original models and software separately.
 
14
 
15
  # MOSS-Image Smaller Encoder β€” 4B to 9B Adapter
16
 
17
+ **An experimental, lightweight text-encoder replacement for MOSS-Image.** This adapter maps features from **Qwen3.5-4B** into the conditioning space of MOSS-Image's original **Qwen3.5-9B** text encoder. The goal is to reduce memory requirements for prompt encoding while retaining useful image-generation quality.
18
 
19
+ **GitHub β€” ComfyUI node, source code, and installation instructions:** [bakapotatolord/moss-image-smaller-encoder](https://github.com/bakapotatolord/moss-image-smaller-encoder)
20
 
21
+ > **Current adapter: fitted on 800 prompts**, with 200 separate validation prompts and 200 held-out test prompts. This is a feature-projection adapter, **not a LoRA or a standalone text-to-image model**. It requires MOSS-Image, a compatible Qwen3.5-4B encoder, and the adapter node. Equivalent quality to the original 9B encoder is not guaranteed.
22
 
23
  ## How it works
24
 
25
+ The 4B and 9B encoders remain **frozen**. The same prompts are encoded using MOSS's prompt template, and token-aligned, final normalized hidden states are collected from both encoders.
26
+
27
+ A standardized affine projection is fitted using **token-weighted ridge regression** to approximate the 9B features from the 4B features. Neither encoder nor the diffusion model is fine-tuned.
28
 
29
  | Component | Details |
30
  | --- | --- |
31
+ | Student encoder | Qwen3.5-4B |
32
+ | Target encoder | MOSS-Image's original Qwen3.5-9B |
33
+ | Tested encoder checkpoints | INT8 ConvRot versions of both encoders |
34
  | Projection | 2,560 β†’ 4,096 dimensions |
35
+ | Projection parameters | Approximately 10.5 million |
36
+ | Adapter size | Approximately 21 MB / 20 MiB |
37
+ | Stored tensors | FP16 projection matrix and FP32 normalization statistics |
38
+ | Fitting method | Standardized, token-weighted ridge regression |
39
+ | ComfyUI projection | CPU FP32 computation with FP16 output rounding |
40
+
41
+ Only the projection adapter is distributed here. The Qwen encoder and MOSS-Image model weights must be downloaded separately.
42
+
43
+ ## Current results β€” 800 fitting prompts
44
+
45
+ The current adapter uses a **1,200-prompt dataset**, divided into:
46
+
47
+ | Split | SFW prompts | Adult/NSFW prompts | Total |
48
+ | --- | ---: | ---: | ---: |
49
+ | Training | 400 | 400 | 800 |
50
+ | Validation | 100 | 100 | 200 |
51
+ | Held-out test | 100 | 100 | 200 |
52
+
53
+ Only the **800 training prompts** were used to fit the projection. Validation selected the ridge regularization strength, and the separate test set measured feature reconstruction.
54
+
55
+ The mixed corpus includes prompts from [jtatman/stable-diffusion-prompts-stats-full-uncensored](https://huggingface.co/datasets/jtatman/stable-diffusion-prompts-stats-full-uncensored). SFW/adult categories are based on source labels and text screening; they are not independently verified semantic annotations. Equal prompt counts also do not imply equal token contributions to the fit.
56
 
57
+ ### Feature reconstruction
58
 
59
+ On the 200-prompt held-out test set:
60
 
61
+ | Metric | Result |
62
+ | --- | ---: |
63
+ | Content-token relative L2 error ↓ | 0.4141 |
64
+ | Content-token cosine similarity ↑ | 0.9082 |
65
+
66
+ These metrics measure how closely projected 4B features approximate the original 9B features. **They are not image-quality scores.**
67
+
68
+ ### Image comparisons
69
+
70
+ Eight image pairs compare the current 800-prompt adapter against the original 9B encoder using four SFW prompts and two seeds. Each pair uses matching prompts, seeds, and generation settings:
71
+
72
+ - Resolution: **640 Γ— 960**
73
+ - Sampling: **8 steps, Euler / simple**
74
+ - CFG: **1**
75
+ - MOSS Image 8-step DMD LoRA strength: **0.8**
76
 
77
  ![Original 9B conditioning vs projected 4B conditioning](./comparison.jpeg)
78
 
79
+ The examples retain the main subjects and artistic styles, with differences in composition, typography, and incidental details. They do not establish quality parity with the original encoder.
80
+
81
+ Adult prompts were included in fitting and feature evaluation, but **adult image-generation quality has not been visually evaluated**. Quality with other encoder quantizations, longer prompts, image editing, higher CFG, and other generation settings also remains unestablished.
82
 
83
  ## Models and usage
84
 
85
+ ### Original MOSS-Image models
86
 
87
+ - [MOSS-Image-Base](https://huggingface.co/OpenMOSS-Team/MOSS-Image-Base)
88
+ - [MOSS-Image-SFT](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT)
89
  - [ComfyUI-format MOSS-Image files](https://huggingface.co/Comfy-Org/MOSS-Image)
90
 
91
+ ### Smaller encoder
92
+
93
+ The tested checkpoint is [Qwen3.5-4B INT8 ConvRot](https://huggingface.co/beycanai/Qwen-LM/resolve/main/Qwen3.5-4B_int8_convrot.safetensors?download=true). Your ComfyUI installation must support its quantization format.
94
+
95
+ Use **Qwen3.5-4B**, not Qwen3-VL-4B. These are different model families.
96
+
97
+ ### Download the adapter
98
+
99
+ The current adapter filename is:
100
+
101
+ `moss_qwen35_4b_to_9b_800_train.safetensors`
102
+
103
+ You can download it from the [GitHub release](https://github.com/bakapotatolord/moss-image-smaller-encoder/releases/tag/adapter-800-train). If downloading from this Hugging Face repository, select the same filename and verify the checksum below.
104
+
105
+ You only need one copy of the adapter.
106
+
107
+ <details>
108
+ <summary>Adapter SHA-256 checksum</summary>
109
+
110
+ ```text
111
+ 948ddeaaf051ced32314a704fa0a37e8fc2faf8440abe8e2c9a07090507ae00a
112
+ ```
113
+
114
+ </details>
115
+
116
+ ### ComfyUI setup
117
+
118
+ 1. Install [moss-image-smaller-encoder](https://github.com/bakapotatolord/moss-image-smaller-encoder) in `ComfyUI/custom_nodes/`.
119
+ 2. Place the Qwen3.5-4B checkpoint in `ComfyUI/models/text_encoders/`.
120
+ 3. Place the adapter in `ComfyUI/models/moss_encoder_adapters/`.
121
+ 4. Restart ComfyUI.
122
+ 5. Select Qwen3.5-4B in **Load CLIP**, with **type** set to **moss_image**.
123
+ 6. Add **MOSS Smaller Encoder Adapter**, select the adapter file, and connect its CLIP output to the normal **CLIP Text Encode** node.
124
+
125
+ **Load CLIP β†’ Load LoRA (optional) β†’ MOSS Smaller Encoder Adapter β†’ CLIP Text Encode β†’ KSampler**
126
 
127
+ Keep your usual MOSS diffusion-model and VAE connections. The GitHub repository includes a sample workflow and further compatibility details.
128
 
129
+ The adapter cannot be loaded through a LoRA loader, and it does not replace the complete text encoder.
130
 
131
+ ## Development approach
132
 
133
+ This project is **vibe coded-ish**: AI assists with research, implementation, and experiments, while I review the proposed steps, verify the work, and inspect the results. The evaluation limits are documented above.
134
 
135
  ## Credits and references
136
 
137
+ - **[ComfyUI-ClipProj](https://github.com/nicolab28/ComfyUI-ClipProj)** by **nicolab28** β€” inspiration for replacing a larger text encoder with a smaller encoder and a learned feature projection. See its [calibration methodology](https://github.com/nicolab28/ComfyUI-ClipProj/blob/main/CALIBRATION.md).
138
  - **[MOSS-Image](https://github.com/OpenMOSS/MOSS-Image)** by the OpenMOSS team β€” the original image-generation model.
139
+ - **[Qwen3.5](https://huggingface.co/Qwen/Qwen3.5-4B)** by the Qwen team β€” the encoder family used in this experiment.
140
+ - **[jtatman/stable-diffusion-prompts-stats-full-uncensored](https://huggingface.co/datasets/jtatman/stable-diffusion-prompts-stats-full-uncensored)** β€” a source of prompts for the expanded calibration corpus.
141
 
142
+ This is an independent community experiment and is not affiliated with the OpenMOSS, Qwen, or ComfyUI-ClipProj developers. Refer to the licenses of the original models, datasets, and software separately.