darask0/MOSS-Image-color-depth-control

MOSS-Image Color-Depth Control (unofficial)

A control adapter for MOSS-Image that makes generation follow a color region map (where each object is) and a depth map (its 3D shape and front/back order). Same idea and recipe as darask0/Krea-2-color-depth-control, on the MOSS-Image DiT. Made for posing scenes with 3D mannequins and for region-painted layouts, together with per-region prompts.

Unofficial: a community adapter, not made or endorsed by OpenMOSS. Apache-2.0 like the base model. The depth maps used for training were made with Depth Anything V2 Large (CC-BY-NC-4.0) and the color maps with Mask2Former Swin-L; the released weights are the adapter only.

Two people from a 3D mannequin render (region prompts + control). Top row: 5,000 steps with the control on for the whole run, ending at 0.5, ending at 0.3, and the base model without control. Bottom row: the same at 10,000 steps. With the control on for the whole run the mannequin surface shows through; ending it at 0.5 gives real people at the mapped places.

Files

File
moss-control.safetensors	Recommended. 10,000 steps.
moss-control-002500 / 005000 / 007500.safetensors	Intermediate steps. Earlier = looser, looks less like the mannequin; later = tighter to the map.
examples/	Two scenes exported from a three.js mannequin editor: clay.png (shaded render), color.png (color pass), depth.png (depth pass), and regions.json for Hayate (edit the /path/to/ prefixes).
images/	Result sheets for the two scenes (control only and with region prompts).
training/config.toml, training/moss_control.py	Training configuration and code (hayate-train).

What it does

Region prompts alone tell MOSS-Image what goes in each region, but placement and scale drift. Image-to-image from a 3D render keeps the layout but carries the render's flat background and gray color. With this adapter the controls are conditions, not image content: generation stays text-to-image from pure noise, and the output follows the map's placement, scale and pose, with a background described only by the prompt.

Inputs

• Color map: an RGB image at the output size where each object or region has its own flat color. The colors are IDs only. Black is fine for "unspecified". Training maps came from COCO panoptic segmentation.
• Depth map: grayscale at the output size, near = white.

Either input may be omitted: the adapter was trained with control dropout (color only 35%, depth only 35%, both 20%, none 10%).

Recommended settings

• Use it together with region prompts (one prompt per color). With only a scene prompt, a mannequin-shaped map is drawn as a mannequin.
• color_end and depth_end = 0.5 for 3D mannequin or other synthetic inputs: the control acts on the first half of the steps (layout, pose), then the model finishes the surfaces. 0.3 gives almost the same result. Use 1.0 for maps from real images.
• MOSS-Image SFT with the 4-step DMAD adapter (darask0/MOSS-Image_4-step_adapter_lora): about 1.8 s per 1024² image on an RTX PRO 6000.

Format

Weights are BF16 safetensors keyed by the MOSS-Image transformer module names.

• video_embedder.weight [3072, 384] and video_embedder.bias [3072] replace the transformer's video_embedder (originally [3072, 128]). The input is the channel concatenation of three 128-channel latents (FLUX.2 VAE latents, 2x2 packed and batch-norm normalized like the image latent):
  1. the noisy latent (as usual),
  2. the color map latent,
  3. the depth map latent (depth replicated to RGB, pixels scaled to [-1, 1]; posterior mean). A zero latent means "no control" for that input. The output head is unchanged.
• *.lora_A.weight / *.lora_B.weight: rank 64, alpha 64, on the block linears (layers.*.proj.{q,k,v,o,ffn_up_gate,ffn_down}) and the text embedder blocks.
• Metadata: hayate_control=1, base_model=moss-base, control_channels=color,depth, control_preprocess=vae.
• In Hayate, strength scales the LoRA (video_embedder is never scaled). Per-input end fractions (color_end, depth_end) zero a control after that fraction of the steps.

Trained on the MOSS-Image Base transformer; used with MOSS-Image SFT and the 4-step DMAD adapter.

Reference implementation: Hayate

Hayate (Rust/CUDA) runs it through --regions JSON, alone or combined with region prompts, labels and layouts. Put moss-control.safetensors next to the model files (the media root) to use it as the default adapter, or give its absolute path:

{
  "control": {"adapter": "/path/to/moss-control.safetensors", "color_map": "/path/to/color.png", "depth": "/path/to/depth.png",
              "strength": 1.0, "color_end": 0.5, "depth_end": 0.5}
}

hayate run moss --dmd4 --prompt "a photo of a sunny city park, daytime" --regions examples/two-people/regions.json --seed 1 --out out.png

Hayate Studio exposes it in the 3D tab ("control adapter" method) and in the image settings.

Training

• Base: MOSS-Image Base transformer, frozen in FP8; video_embedder widened to 384 inputs, control columns zero-initialized.
• Data: 10,000 Krea 2 Turbo images (1024-class, five aspect ratios) with their captions. Color maps from Mask2Former Swin-L (COCO panoptic), depth from Depth Anything V2 Large (near = white).
• Optimization: AdamW, lr 1e-4, 10x lr on the control columns of video_embedder, batch 1, 10,000 steps (0.8 s/step on one RTX PRO 6000), MOSS timestep distribution (logit-normal), 10% empty captions.

Limitations

• Mannequin look. Fed a 3D mannequin render, the adapter reproduces the mannequin surface (wood or metal) when it controls the whole denoising, because the training depth maps came from real photos. Use color_end/depth_end 0.5 and region prompts (see above). Earlier checkpoints (2,500) follow the map less tightly and look less like mannequins; later ones follow it more.
• Pose precision is coarser than the map: fingers and face details are generated by the model.
• Control alone (no region prompts) on a manga-style color map loses the panel layout and garbles text at 7,500+ steps; use it together with region prompts and layouts.
• Not combinable with img2img / inpaint in Hayate (one or the other).
• Trained on Krea 2 Turbo images, so the style prior is photographic.
• Evaluated on a handful of scenes (two-people, person-desk, manga page) and two seeds; no quantitative benchmark.