circlestone-labs

/

Anima

Like 2.59k

Follow

CircleStone Labs 1.13k

Hi, I'm having an issue where my LoRA sometimes produces images with a bright color shift. It doesn't happen everytime, has other generations with the same LoRA gives correct colors (like 1 out of 5 have wrong colors).

I used the same configs given on the example LoRa with diffusion pipe. Also tried with lower LR but the problem still appears with the same frequency. I also tried using AI Toolkit and an unrelated dataset but the problem still appears.

If possible, let me see the training configs. It might be related to the markup or if you're trying to train the LLM adapter.

I'm not training the LLM, I also tested training only on default resolution (1024x) but nothing changed. Here's the configs used on Diffusion pipe:

epochs = 1000
max_steps = 3000
micro_batch_size_per_gpu = 1
pipeline_stages = 1
gradient_accumulation_steps = 4
gradient_clipping = 1.0
warmup_steps = 100

save_every_n_steps = 100
checkpoint_every_n_minutes = 30
save_dtype = 'bfloat16'

compile = true
activation_checkpointing = true
partition_method = 'parameters'
caching_batch_size = 1
map_num_proc = 8
steps_per_print = 1

[model]
type = 'anima'
dtype = 'bfloat16'
llm_adapter_lr = 0
sigmoid_scale = 1.3

[adapter]
type = 'lora'
rank = 32
dtype = 'bfloat16'

[optimizer]
type = 'adamw_optimi'
lr = 2e-5
betas = [0.9, 0.99]
weight_decay = 0.01
eps = 1e-8

[monitoring]
enable_wandb = false
wandb_api_key = ''
wandb_tracker_name = ''
wandb_run_name = ''

Here's what used on AI Toolkit:

config:
name: "Chihel_copy"
process:
- type: "diffusion_trainer"
sqlite_db_path: "./aitk_db.db"
device: "cuda"
trigger_word: null
performance_log_every: 10
network:
type: "lora"
linear: 32
linear_alpha: 32
lokr_full_rank: true
lokr_factor: -1
network_kwargs:
ignore_if_contains: []
conv: 16
conv_alpha: 16
save:
dtype: "bf16"
save_every: 100
max_step_saves_to_keep: 10
save_format: "diffusers"
push_to_hub: false
datasets:
mask_path: null
mask_min_value: 0.1
default_caption: ""
caption_ext: "txt"
caption_dropout_rate: 0.05
cache_latents_to_disk: true
is_reg: false
network_weight: 1
resolution:
- 1536
- 1024
- 1280
controls: []
shrink_video_to_frames: true
num_frames: 1
flip_x: false
flip_y: false
num_repeats: 1
train:
batch_size: 1
bypass_guidance_embedding: false
steps: 1000
gradient_accumulation: 1
train_unet: true
train_text_encoder: false
gradient_checkpointing: true
noise_scheduler: "flowmatch"
optimizer: "adamw8bit"
timestep_type: "weighted"
content_or_style: "balanced"
optimizer_params:
weight_decay: 0.0001
unload_text_encoder: false
cache_text_embeddings: false
lr: 0.00002
ema_config:
use_ema: false
ema_decay: 0.99
skip_first_sample: true
force_first_sample: false
disable_sampling: false
dtype: "bf16"
diff_output_preservation: false
diff_output_preservation_multiplier: 1
diff_output_preservation_class: "person"
switch_boundary_every: 1
loss_type: "mse"
logging:
log_every: 1
use_ui_logger: true
model:
name_or_path: "circlestone-labs/Anima-Base-v1.0-Diffusers"
quantize: false
qtype: ""
quantize_te: false
qtype_te: ""
arch: "anima"
low_vram: false
model_kwargs: {}
compile: false
layer_offloading: false
layer_offloading_text_encoder_percent: 1
layer_offloading_transformer_percent: 1
meta:
name: "[name]"
version: "1.0"

Well, I noticed a couple of rough questions:

1. There's no Shift setting in the config files; if it's higher than 3, there might be problems.
2. Try training on the base version instead of the diffusion version.
3. weight_decay: 0.0001 <-- IMHO, it seems a bit low; it should be around 0.001, but I don't argue with that since I rarely change this parameter.
4. If the dataset contains a description of the style effect, it's better to remove it and leave only the trigger activator (IMHO).
   For now, that's all I can suggest, since I haven't dealt with it yet.