Xabsurd commited on
Commit
f3fe995
·
verified ·
1 Parent(s): 08d4d84

Translate development report into English

Browse files
Files changed (2) hide show
  1. README.md +1 -1
  2. RESEARCH_REPORT.md +103 -42
README.md CHANGED
@@ -192,7 +192,7 @@ different load/apply nodes.
192
  ## Development notes and credits
193
 
194
  See [RESEARCH_REPORT.md](RESEARCH_REPORT.md) for a concise development report in
195
- Chinese, including the architectural choices, color investigation, and checkpoint
196
  selection. [SHA256SUMS.txt](SHA256SUMS.txt) records hashes of the distributed weights
197
  and node ZIP.
198
 
 
192
  ## Development notes and credits
193
 
194
  See [RESEARCH_REPORT.md](RESEARCH_REPORT.md) for a concise development report in
195
+ English, including the architectural choices, color investigation, and checkpoint
196
  selection. [SHA256SUMS.txt](SHA256SUMS.txt) records hashes of the distributed weights
197
  and node ZIP.
198
 
RESEARCH_REPORT.md CHANGED
@@ -1,49 +1,110 @@
1
- # MOSS-Image Tile Control:开发与实验简报
2
 
3
- 本项目探索如何为 MOSS-Image 增加连续可调的图像控制,用于放大后的细化和高清化,同时兼顾原始特征、颜色与推理速度。经过四次方案迭代,最终确定并发布 **V1**。beta2、beta3 保留作为不同取舍的历史方案。
 
 
 
 
4
 
5
- ## 1. 问题从哪里开始
6
 
7
- 最初采用图像条件拼接与主干 LoRA 的方案。它可以强力约束画面,但控制图强度归零时,已经加载的 LoRA 仍会改变主干输出。因此,“关掉图像条件”和“完全撤去控制模块”并不等价。
 
 
 
8
 
9
- 目标随之明确:控制强度应作用于独立的控制路径,强度为零时直接返回输入模型,并跳过控制图编码。这一要求也成为后续实现的验证重点。
 
 
10
 
11
- ## 2. 发布版本与实际取舍
12
 
13
- | 方案 | 结构 | 实际观察与定位 |
14
  | --- | --- | --- |
15
- | beta2 | 图像条件拼接 + 主干 LoRA | 适合高强度控制,原始特征保持最强,画面修改最小;不会修复细节,采样最慢。控制图归零仍不能消除 LoRA 的影响 |
16
- | beta3 | VAE 潜空间条件 + 独立低秩残差分支 | 解决控制归零时的主干污染问题,控制力度较强、速度最快;但倾向于抹平高频细节,目前不作为通用修复首选 |
17
- | V1 | RGB 条件编码器 + 图像 token 的 Q/K/V 输入 LLLite 适配器 | 作者人工比较中清晰度与细节强化最好,同时保留主体特征,颜色保持也较好;综合观感为作者个人首选 |
18
-
19
- 项目共经历四次方案迭代,上表列出保留发布的三个版本及作者在实际工作流中的观察。不同路线还涉及 Base/SFT 底模差异,速度和画质取舍不能全部归因于控制结构;目前没有统一基准测试支持跨模型的严格排名。
20
-
21
- 作者选择 V1 的主要理由是清晰度与细节强化,而不是单纯追求与输入最接近。beta2 对原始特征约束很强,观感更接近保守的放大模型,生成细节的变化空间较小。beta2 的 LoRA 会改变主干扩散层权重,这是实现上的事实;这种影响可能与细节强化较弱有关,但尚未通过消融实验确立因果关系,底模差异、训练与控制强度也可能影响结果。
22
-
23
- ## 3. 偏色排查与保色实验
24
-
25
- 早期实验出现颜色、饱和度和对比度变化。用户在原工作流中确认:VAE 编码再解码没有明显变色,关闭加速 LoRA 也没有明显改善,而关闭控制分支后重新采样仍会改变颜色。这表明底模的重新生成行为本身也是颜色变化的来源,不能把所有偏色都归因于控制分支。
26
-
27
- 项目尝试过从已有残差检查点出发的保色微调和更高学习率实验。改进没有稳定出现在所有检查点上,说明继续训练或加大颜色约束并不必然改善整体观感。最终仍以实际比较结果选模型;Color Match 作为可选后处理保留,而不是把保色问题视为已经彻底解决。
28
-
29
- ## 4. LLLite 适配、训练与模型选择
30
-
31
- LLLite 路线复用了 ComfyUI 原生 Anima LLLite 的条件编码器和适配模块,并适配到 MOSS 的 token 排布、联合注意力与层数。控制作用于图像 token 的 Q/K/V 投影输入。这里对齐的是模块计算与注入位置,并不意味着两个底模或训练结果相同。
32
-
33
- 同权重模块数值核对中,条件编码器输出逐位一致,Q/K/V 输入控制路径的最大绝对误差约为 `1.2e-6`,处于浮点运算误差范围。这项核对不代表端到端生成质量相同。
34
-
35
- LLLite 路线的早期结果更清晰,但颜色、风格和细节表现一度不及预期。继续训练后,实际观感明显改善;结合多轮人工比较,最终确定 V1。后期结果改善支持“早期训练尚不充分”的解释,但还不足以证明普遍欠拟合,也不意味着训练步数越高必然越好。
36
-
37
- ## 5. 推理工程与验证
38
-
39
- 节点通过 ComfyUI 的 ModelPatcher、附加模型加载和 block replacement 接口挂载控制,保留原有显存管理机制。强度零直接绕过控制,不向底模合并控制权重。节点显示名统一为 `Load MOSS-LLLite` / `Apply MOSS-LLLite`,内部 ID 保留,兼容旧工作流。
40
-
41
- 推理优化将融合 QKV 的三次完整投影改为各自只计算所需的输出行,并复用原生权重转换和量化路径。在一次 1024 分辨率、2 步、CFG 3、INT8/ConvRot 的预热后短采样测试中,约从 `1.94 s` 降至 `1.52 s`,最终 latent 逐位一致。该结果仅说明这项局部优化在所测设置中有效,不代表所有工作流的固定加速比。
42
-
43
- 另一个减少 RGB 编码批次计算的尝试因改变 BF16 数值结果而未采用。性能优化以保持生成结果为约束,而不是只比较运行时间。
44
-
45
- ## 6. 结论与边界
46
-
47
- 最终形成三个明确取舍:**beta2 重保留,beta3 重速度,V1 重细化与综合观感**。V1 是经过四次方案迭代及多轮人工测试确定的最终方案,也是作者基于清晰度、细节强化、特征与颜色保持的个人综合首选。
48
-
49
- 目前的证据包括模块数值核对、节点行为测试、局部性能测试及人工出图比较。尚缺少统一测试集上的感知质量、颜色误差、特征一致性与多随机种子统计;也未通过完整文献比较确立方法新颖性。本报告是一份工程探索记录,不把主观评价包装成严格学术结论。
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # MOSS-Image Tile Control: Development and Experimental Report
2
 
3
+ This project explores continuously adjustable image control for MOSS-Image,
4
+ balancing refinement after upscaling, subject-feature preservation, color
5
+ retention, and inference speed. **V1 is the final design established through four
6
+ iterations.** beta2 and beta3 remain available as earlier approaches with different
7
+ trade-offs.
8
 
9
+ ## 1. The initial problem
10
 
11
+ The initial approach combined concatenated image conditioning with a backbone
12
+ LoRA. It provided strong control, but setting image-conditioning strength to zero
13
+ did not undo the LoRA's effect on the backbone. Disabling the image condition and
14
+ removing the controller entirely were therefore different operations.
15
 
16
+ This led to a concrete requirement: strength should scale an independent control
17
+ path. At zero strength, the node should return the input model directly and skip
18
+ control-image encoding. Subsequent implementations explicitly tested this behavior.
19
 
20
+ ## 2. Released approaches and practical trade-offs
21
 
22
+ | Approach | Architecture | Observations and intended use |
23
  | --- | --- | --- |
24
+ | beta2 | Concatenated image conditioning and backbone LoRA | Best suited to high control strength and minimal changes. Preserves original features most strongly, but does not repair fine details; slowest in the tested workflows. Zero conditioning strength does not remove the LoRA's effect |
25
+ | beta3 | VAE-latent conditioning and an independent low-rank residual branch | Resolves the zero-strength backbone-modification issue. Strong structural control and the fastest option, but tends to smooth high-frequency detail; currently not the general restoration recommendation |
26
+ | V1 | RGB conditioning encoder and LLLite adapters at the image-token Q/K/V projection inputs | Best clarity and detail enhancement in the author's visual comparisons, while preserving subject features and retaining colors well. The author's personal choice for overall quality |
27
+
28
+ The project underwent four design iterations; this table summarizes the three
29
+ versions retained for release. These observations come from the author's actual
30
+ workflows. The approaches also use different Base/SFT backbones, so speed and
31
+ quality differences cannot be attributed entirely to the control architecture.
32
+ No standardized cross-model benchmark has been established.
33
+
34
+ The author chose V1 primarily for clarity and detail enhancement, rather than
35
+ maximum similarity to the input alone. beta2's strong preservation behaves more
36
+ like a conservative upscaler, leaving less room for generative refinement. Its
37
+ LoRA changes backbone diffusion-layer weights as an implementation fact. Whether
38
+ this causes its weaker detail enhancement remains a hypothesis: no ablation has
39
+ isolated that relationship, and the backbone, training, and control strength may
40
+ also contribute.
41
+
42
+ ## 3. Investigating color shifts
43
+
44
+ Early experiments showed changes in color, saturation, and contrast. In the
45
+ tested workflow, a VAE encode/decode round trip produced no obvious color shift,
46
+ and disabling the acceleration LoRA did not noticeably reduce it. Resampling
47
+ still changed colors with the control branch disabled. This indicates that the
48
+ backbone's regeneration behavior is also a source of color changes; the control
49
+ branch cannot account for every shift.
50
+
51
+ Experiments included color-preservation fine-tuning from existing residual
52
+ checkpoints and higher-learning-rate trials. Improvements were inconsistent
53
+ across checkpoints. More training or stronger color constraints did not reliably
54
+ improve overall appearance. Selection therefore remained based on actual visual
55
+ comparisons. Color Match is retained as optional post-processing; color
56
+ preservation is not considered fully solved.
57
+
58
+ ## 4. Adapting LLLite and selecting the final design
59
+
60
+ The LLLite approach reuses ComfyUI's native Anima LLLite conditioning encoder and
61
+ adapter modules, adapting them to MOSS token layout, joint attention, and layer
62
+ count. Control acts on image-token inputs to the Q/K/V projections. Alignment
63
+ refers to the module math and injection location, not identical backbones or
64
+ training outcomes.
65
+
66
+ In a module-level numerical check using identical weights, conditioning-encoder
67
+ outputs were bitwise identical. The Q/K/V control-path maximum absolute error
68
+ was approximately `1.2e-6`, within floating-point arithmetic differences. This
69
+ does not establish equivalent end-to-end generation quality.
70
+
71
+ Early LLLite results were clearer, but color, style, and detail quality initially
72
+ fell short of expectations. Further training improved visual results. V1 was
73
+ then chosen through repeated visual comparisons. This improvement supports the
74
+ interpretation that early training was insufficient, but does not conclusively
75
+ establish underfitting or imply that later checkpoints are always better.
76
+
77
+ ## 5. Inference engineering and validation
78
+
79
+ The nodes attach control through ComfyUI's ModelPatcher, additional-model loading,
80
+ and block-replacement interfaces, retaining native memory management. Zero
81
+ strength bypasses control without merging controller weights into the backbone.
82
+ The display names are **Load MOSS-LLLite** and **Apply MOSS-LLLite**; internal IDs
83
+ are preserved for existing workflows.
84
+
85
+ An inference optimization replaced three complete fused-QKV projections with
86
+ the required output rows for each projection, reusing native casting and
87
+ quantization paths. In one warmed-up, 1024-resolution, two-step sampling test
88
+ with CFG 3 and an INT8/ConvRot backbone, time fell from approximately `1.94 s` to
89
+ `1.52 s`, with bitwise-identical final latents. This is evidence for that local
90
+ optimization under the tested settings, not a fixed speedup for every workflow.
91
+
92
+ A separate attempt to reduce RGB encoder batch computation was rejected because
93
+ it changed BF16 numerical results. Preserving generation results constrained
94
+ optimization decisions, rather than runtime alone.
95
+
96
+ ## 6. Conclusions and limits
97
+
98
+ The released options offer distinct trade-offs: **beta2 prioritizes preservation,
99
+ beta3 prioritizes speed, and V1 prioritizes refinement and overall appearance.**
100
+ V1 is the final design established through four iterations and repeated visual
101
+ testing, and the author's personal choice across clarity, detail enhancement,
102
+ subject features, and color retention.
103
+
104
+ Available evidence includes module-level numerical comparisons, node-behavior
105
+ tests, a limited performance benchmark, and visual generation comparisons.
106
+ Standardized evaluations of perceptual quality, color error, feature consistency,
107
+ and variation across random seeds are still missing. Methodological novelty has
108
+ not been established through a comprehensive literature comparison. This report
109
+ documents an engineering exploration; subjective observations are not presented
110
+ as rigorous academic findings.