LooseControl: Lifting ControlNet for Generalized Depth Conditioning

This is the official repository for our paper:

LooseControl: Lifting ControlNet for Generalized Depth Conditioning

Shariq Farooq Bhat, Niloy J. Mitra, Peter Wonka

[Project Page] [Paper] [Demo 🤗] [Weights (3D Box Control)]

Usage

git clone https://github.com/shariqfarooq123/LooseControl && cd LooseControl

Start the UI:

gradio app.py

or use via python API:

from loosecontrol import LooseControlNet

lcn = LooseControlNet("shariqfarooq/loose-control-3dbox")

boxy_depth = ...
prompt = "A photo of a snowman in a desert"
negative_prompt = "blurry, text, caption, lowquality,lowresolution, low res, grainy, ugly"


gen_image_1 = lcn(prompt, negative_prompt=negative_prompt, control_image=boxy_depth)

Style preserving edits:

# Fix the 'style' and edit
# Edit 'boxy_depth' -> 'boxy_depth_edited'

lcn.set_cf_attention()

gen_image_edited = lcn.edit(boxy_depth, boxy_depth_edited, prompt, negative_prompt=negative_prompt)

Credits

The Cross Frame attention is adapted from Text2Video-Zero

Citation

@misc{bhat2023loosecontrol,
      title={LooseControl: Lifting ControlNet for Generalized Depth Conditioning}, 
      author={Shariq Farooq Bhat and Niloy J. Mitra and Peter Wonka},
      year={2023},
      eprint={2312.03079},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}