🦙 LaMa: Resolution-robust Large Mask Inpainting with Fourier Convolutions

by Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, Victor Lempitsky.

<p align="center" "font-size:30px;"> 🔥🔥🔥 <br> <b> LaMa generalizes surprisingly well to much higher resolutions (~2k❗️) than it saw during training (256x256), and achieves the excellent performance even in challenging scenarios, e.g. completion of periodic structures.</b> </p>

[Project page] [arXiv] [Supplementary] [BibTeX] [Casual GAN Papers Summary]

<p align="center"> <a href="https://colab.research.google.com/drive/15KTEIScUbVZtUP6w2tCDMVpE-b1r9pkZ?usp=drive_link"> <img src="https://colab.research.google.com/assets/colab-badge.svg"/> </a> <br> Try out in Google Colab <br> All yandex dist links went bad, you can download the model from the https://drive.google.com/drive/folders/1B2x7eQDgecTL0oh3LSIBDGj0fTxs6Ips?usp=sharing </p> <p align="center"> <img src="https://raw.githubusercontent.com/senya-ashukha/senya-ashukha.github.io/master/projects/lama_21/ezgif-4-0db51df695a8.gif" /> </p> <p align="center"> <img src="https://raw.githubusercontent.com/senya-ashukha/senya-ashukha.github.io/master/projects/lama_21/gif_for_lightning_v1_white.gif" /> </p>

LaMa development

(Feel free to share your paper by creating an issue)

• https://github.com/geekyutao/Inpaint-Anything --- Inpaint Anything: Segment Anything Meets Image Inpainting

<p align="center"> <img src="https://raw.githubusercontent.com/geekyutao/Inpaint-Anything/main/example/MainFramework.png" /> </p>

• Feature Refinement to Improve High Resolution Image Inpainting / video / code https://github.com/advimman/lama/pull/112 / by Geomagical Labs (geomagical.com)

<p align="center"> <img src="https://raw.githubusercontent.com/senya-ashukha/senya-ashukha.github.io/master/images/FeatureRefinement.png" /> </p>

Non-official 3rd party apps:

(Feel free to share your app/implementation/demo by creating an issue)

• https://github.com/enesmsahin/simple-lama-inpainting - a simple pip package for LaMa inpainting.
• https://github.com/mallman/CoreMLaMa - Apple's Core ML model format
• https://cleanup.pictures - a simple interactive object removal tool by @cyrildiagne
  • lama-cleaner by @Sanster is a self-host version of https://cleanup.pictures
• Integrated to Huggingface Spaces with Gradio. See demo: by @AK391
• Telegram bot @MagicEraserBot by @Moldoteck, code
• Auto-LaMa = DE:TR object detection + LaMa inpainting by @andy971022
• LAMA-Magic-Eraser-Local = a standalone inpainting application built with PyQt5 by @zhaoyun0071
• Hama - object removal with a smart brush which simplifies mask drawing.
• ModelScope = the largest Model Community in Chinese by @chenbinghui1.
• LaMa with MaskDINO = MaskDINO object detection + LaMa inpainting with refinement by @qwopqwop200.
• CoreMLaMa - a script to convert Lama Cleaner's port of LaMa to Apple's Core ML model format.

Environment setup

❗️❗️❗️ All yandex dist links went bad, you can download the model from the google drive ❗️❗️❗️

Clone the repo: git clone https://github.com/advimman/lama.git

There are three options of an environment:

1. Python virtualenv:

   virtualenv inpenv --python=/usr/bin/python3
   source inpenv/bin/activate
   pip install torch==1.8.0 torchvision==0.9.0
   
   cd lama
   pip install -r requirements.txt 
   

2. Conda

   % Install conda for Linux, for other OS download miniconda at https://docs.conda.io/en/latest/miniconda.html
   wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
   bash Miniconda3-latest-Linux-x86_64.sh -b -p $HOME/miniconda
   $HOME/miniconda/bin/conda init bash
   
   cd lama
   conda env create -f conda_env.yml
   conda activate lama
   conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch -y
   pip install pytorch-lightning==1.2.9
   

3. Docker: No actions are needed 🎉.

Inference <a name="prediction"></a>

Run

cd lama
export TORCH_HOME=$(pwd) && export PYTHONPATH=$(pwd)

1. Download pre-trained models

The best model (Places2, Places Challenge):

curl -LJO https://huggingface.co/smartywu/big-lama/resolve/main/big-lama.zip
unzip big-lama.zip

All models (Places & CelebA-HQ):

download [https://drive.google.com/drive/folders/1B2x7eQDgecTL0oh3LSIBDGj0fTxs6Ips?usp=drive_link]
unzip lama-models.zip

2. Prepare images and masks

Download test images:

unzip LaMa_test_images.zip

<details> <summary>OR prepare your data:</summary> 1) Create masks named as `[images_name]_maskXXX[image_suffix]`, put images and masks in the same folder.

• You can use the script for random masks generation.
• Check the format of the files:

  image1_mask001.png
  image1.png
  image2_mask001.png
  image2.png
  

2. Specify image_suffix, e.g. .png or .jpg or _input.jpg in configs/prediction/default.yaml.

</details>

3. Predict

On the host machine:

python3 bin/predict.py model.path=$(pwd)/big-lama indir=$(pwd)/LaMa_test_images outdir=$(pwd)/output

OR in the docker

The following command will pull the docker image from Docker Hub and execute the prediction script

bash docker/2_predict.sh $(pwd)/big-lama $(pwd)/LaMa_test_images $(pwd)/output device=cpu

Docker cuda:

bash docker/2_predict_with_gpu.sh $(pwd)/big-lama $(pwd)/LaMa_test_images $(pwd)/output

4. Predict with Refinement

On the host machine:

python3 bin/predict.py refine=True model.path=$(pwd)/big-lama indir=$(pwd)/LaMa_test_images outdir=$(pwd)/output

Train and Eval

Make sure you run:

cd lama
export TORCH_HOME=$(pwd) && export PYTHONPATH=$(pwd)

Then download models for perceptual loss:

mkdir -p ade20k/ade20k-resnet50dilated-ppm_deepsup/
wget -P ade20k/ade20k-resnet50dilated-ppm_deepsup/ http://sceneparsing.csail.mit.edu/model/pytorch/ade20k-resnet50dilated-ppm_deepsup/encoder_epoch_20.pth

Places

⚠️ NB: FID/SSIM/LPIPS metric values for Places that we see in LaMa paper are computed on 30000 images that we produce in evaluation section below. For more details on evaluation data check [Section 3. Dataset splits in Supplementary] ⚠️

On the host machine:

# Download data from http://places2.csail.mit.edu/download.html
# Places365-Standard: Train(105GB)/Test(19GB)/Val(2.1GB) from High-resolution images section
wget http://data.csail.mit.edu/places/places365/train_large_places365standard.tar
wget http://data.csail.mit.edu/places/places365/val_large.tar
wget http://data.csail.mit.edu/places/places365/test_large.tar

# Unpack train/test/val data and create .yaml config for it
bash fetch_data/places_standard_train_prepare.sh
bash fetch_data/places_standard_test_val_prepare.sh

# Sample images for test and viz at the end of epoch
bash fetch_data/places_standard_test_val_sample.sh
bash fetch_data/places_standard_test_val_gen_masks.sh

# Run training
python3 bin/train.py -cn lama-fourier location=places_standard

# To evaluate trained model and report metrics as in our paper
# we need to sample previously unseen 30k images and generate masks for them
bash fetch_data/places_standard_evaluation_prepare_data.sh

# Infer model on thick/thin/medium masks in 256 and 512 and run evaluation 
# like this:
python3 bin/predict.py \
model.path=$(pwd)/experiments/<user>_<date:time>_lama-fourier_/ \
indir=$(pwd)/places_standard_dataset/evaluation/random_thick_512/ \
outdir=$(pwd)/inference/random_thick_512 model.checkpoint=last.ckpt

python3 bin/evaluate_predicts.py \
$(pwd)/configs/eval2_gpu.yaml \
$(pwd)/places_standard_dataset/evaluation/random_thick_512/ \
$(pwd)/inference/random_thick_512 \
$(pwd)/inference/random_thick_512_metrics.csv

Docker: TODO

CelebA

On the host machine:

# Make shure you are in lama folder
cd lama
export TORCH_HOME=$(pwd) && export PYTHONPATH=$(pwd)

# Download CelebA-HQ dataset
# Download data256x256.zip from https://drive.google.com/drive/folders/11Vz0fqHS2rXDb5pprgTjpD7S2BAJhi1P

# unzip & split into train/test/visualization & create config for it
bash fetch_data/celebahq_data