Skip to content

Latest commit

 

History

History
50 lines (35 loc) · 7.89 KB

coadapter.md

File metadata and controls

50 lines (35 loc) · 7.89 KB

Overview

We introduce CoAdapter (Composable Adapter) by jointly training T2I-Adapters and an extra fuser. The fuser allows different adapters with various conditions to be aware of each other and synergize to achieve more powerful composability, especially the combination of element-level style and other structural information.

CoAdapter is inspired by Composer. However, instead of training the whole model, it only trains extra light-weight adapters based on T2I-Adapter. But CoAdapter can also show the capability of generating creative images with composibility. Note that the model is still in training and this release is only a preview.

Demos

Sketch Canny Depth Color (Spatial) Style Results
image image image
image image image
image image image image
image image image image

Benefits from CoAdapter

CoAdapter offers two advantages over the original T2I-Adapter. You can try CoAdapter in Huggingface CoAdapter.

  1. CoAdapter has improved composability, especially for the style modality, due to the joint training of multiple adapters.
Input 1 Input 2 Input3 Prompt and seed T2I-Adapter CoAdapter
image
Canny: 1.0		 image
Style: 1.0		 "tower"
seed=42		 image image
image
Skecth: 1.0		 image
Style: 1.0		 image
Color: 1.0		 "motorbike"
seed=993		 image image
image
Skecth: 1.0		 image
Style: 1.0		 "a corgi"
seed=42		 image image
  1. The joint training of CoAdapter can also enhance the generation quality of each individual adapter. TODO

The above results are based on coadapter-sd15v1 and t2iadapter-sd14v1.

Useful Tips

  • Condition weight is important. If the generated image is not well aligned with the condition, increase the corresponding Condition weight. If increasing Condition weight is ineffective or degrades image quality, try decreasing the Condition weight of other conditions.
  • Start with fewer conditions. If you plan to use more than two conditions to control the model, it is recommended to start with only one or two conditions. Once you have found the suitable Condition weight for existing conditions, gradually append the new conditions.
  • It is recommended to use a step size of 0.1 to adjust Condition weight. From experience, Condition weight will not be less than 0.5 or greater than 1.5

Training

python train.py -t --base configs/pl_train/coadapter-v1-train.yaml --gpus 0,1,2,3,4,5,6,7 --scale_lr False --num_nodes 1 --sd_finetune_from models/v1-5-pruned-emaonly.ckpt --name coadapter-v1-train --auto_resume