Fast Steering of Diffusion Models using Inversion-Anchored Concept Activation Vectors
Nicholas Bai, Akshay Kulkarni, Tsui-Wei Weng
Preprint — Trustworthy AI Lab, UC San Diego
Inversion-anchored concept activation vectors that steer diffusion models without retraining.
Text-to-image diffusion models have been shown to have interpretable directions in their latent space, which can be leveraged for controllable image generation.
While prior works require expensive training or optimization at inference-time to obtain these interpretable directions or concept activation vectors (CAVs), we propose a novel and efficient Inversion-Anchored Concept Activation Vector (IA-CAV) method to obtain CAVs for arbitrary concepts using only a few images without any training or optimization.
In quantitative evaluations, we highlight the broad applicability of our IA-CAVs to tasks like image editing and fair generation with significant improvements (11% better) while also being more efficient (24–47× faster) than prior works.