Trained on immiscible noise.
network_args = ["rank_dropout=0.1", "module_dropout=0.1", "context_mod_dim=0", "context_mlp_dim=0", "x_mod_dim=0", "x_mlp_dim=0", "train_block_indices=4-9"]
I think what made this lora good was: low gradient clipping (0.1), rms learning rate adjustment and clipping (FARMSCropv2 optimizer), and only training attn blocks 4-9.
