MOSS-Image-Tile-Control / RESEARCH_REPORT.md
Xabsurd's picture
Translate development report into English
f3fe995 verified
|
Raw History Blame Contribute Delete
6.8 kB

MOSS-Image Tile Control: Development and Experimental Report

This project explores continuously adjustable image control for MOSS-Image, balancing refinement after upscaling, subject-feature preservation, color retention, and inference speed. V1 is the final design established through four iterations. beta2 and beta3 remain available as earlier approaches with different trade-offs.

1. The initial problem

The initial approach combined concatenated image conditioning with a backbone LoRA. It provided strong control, but setting image-conditioning strength to zero did not undo the LoRA's effect on the backbone. Disabling the image condition and removing the controller entirely were therefore different operations.

This led to a concrete requirement: strength should scale an independent control path. At zero strength, the node should return the input model directly and skip control-image encoding. Subsequent implementations explicitly tested this behavior.

2. Released approaches and practical trade-offs

Approach Architecture Observations and intended use
beta2 Concatenated image conditioning and backbone LoRA Best suited to high control strength and minimal changes. Preserves original features most strongly, but does not repair fine details; slowest in the tested workflows. Zero conditioning strength does not remove the LoRA's effect
beta3 VAE-latent conditioning and an independent low-rank residual branch Resolves the zero-strength backbone-modification issue. Strong structural control and the fastest option, but tends to smooth high-frequency detail; currently not the general restoration recommendation
V1 RGB conditioning encoder and LLLite adapters at the image-token Q/K/V projection inputs Best clarity and detail enhancement in the author's visual comparisons, while preserving subject features and retaining colors well. The author's personal choice for overall quality

The project underwent four design iterations; this table summarizes the three versions retained for release. These observations come from the author's actual workflows. The approaches also use different Base/SFT backbones, so speed and quality differences cannot be attributed entirely to the control architecture. No standardized cross-model benchmark has been established.

The author chose V1 primarily for clarity and detail enhancement, rather than maximum similarity to the input alone. beta2's strong preservation behaves more like a conservative upscaler, leaving less room for generative refinement. Its LoRA changes backbone diffusion-layer weights as an implementation fact. Whether this causes its weaker detail enhancement remains a hypothesis: no ablation has isolated that relationship, and the backbone, training, and control strength may also contribute.

3. Investigating color shifts

Early experiments showed changes in color, saturation, and contrast. In the tested workflow, a VAE encode/decode round trip produced no obvious color shift, and disabling the acceleration LoRA did not noticeably reduce it. Resampling still changed colors with the control branch disabled. This indicates that the backbone's regeneration behavior is also a source of color changes; the control branch cannot account for every shift.

Experiments included color-preservation fine-tuning from existing residual checkpoints and higher-learning-rate trials. Improvements were inconsistent across checkpoints. More training or stronger color constraints did not reliably improve overall appearance. Selection therefore remained based on actual visual comparisons. Color Match is retained as optional post-processing; color preservation is not considered fully solved.

4. Adapting LLLite and selecting the final design

The LLLite approach reuses ComfyUI's native Anima LLLite conditioning encoder and adapter modules, adapting them to MOSS token layout, joint attention, and layer count. Control acts on image-token inputs to the Q/K/V projections. Alignment refers to the module math and injection location, not identical backbones or training outcomes.

In a module-level numerical check using identical weights, conditioning-encoder outputs were bitwise identical. The Q/K/V control-path maximum absolute error was approximately 1.2e-6, within floating-point arithmetic differences. This does not establish equivalent end-to-end generation quality.

Early LLLite results were clearer, but color, style, and detail quality initially fell short of expectations. Further training improved visual results. V1 was then chosen through repeated visual comparisons. This improvement supports the interpretation that early training was insufficient, but does not conclusively establish underfitting or imply that later checkpoints are always better.

5. Inference engineering and validation

The nodes attach control through ComfyUI's ModelPatcher, additional-model loading, and block-replacement interfaces, retaining native memory management. Zero strength bypasses control without merging controller weights into the backbone. The display names are Load MOSS-LLLite and Apply MOSS-LLLite; internal IDs are preserved for existing workflows.

An inference optimization replaced three complete fused-QKV projections with the required output rows for each projection, reusing native casting and quantization paths. In one warmed-up, 1024-resolution, two-step sampling test with CFG 3 and an INT8/ConvRot backbone, time fell from approximately 1.94 s to 1.52 s, with bitwise-identical final latents. This is evidence for that local optimization under the tested settings, not a fixed speedup for every workflow.

A separate attempt to reduce RGB encoder batch computation was rejected because it changed BF16 numerical results. Preserving generation results constrained optimization decisions, rather than runtime alone.

6. Conclusions and limits

The released options offer distinct trade-offs: beta2 prioritizes preservation, beta3 prioritizes speed, and V1 prioritizes refinement and overall appearance. V1 is the final design established through four iterations and repeated visual testing, and the author's personal choice across clarity, detail enhancement, subject features, and color retention.

Available evidence includes module-level numerical comparisons, node-behavior tests, a limited performance benchmark, and visual generation comparisons. Standardized evaluations of perceptual quality, color error, feature consistency, and variation across random seeds are still missing. Methodological novelty has not been established through a comprehensive literature comparison. This report documents an engineering exploration; subjective observations are not presented as rigorous academic findings.