GFM
[IJCV 2022] Bridging Composite and Real: Towards End-to-end Deep Image Matting
Install / Use
/learn @JizhiziLi/GFMREADME
<img src="demo/src/homepage/spring.gif" width="25%"><img src="demo/src/homepage/summer.gif" width="25%"><img src="demo/src/homepage/autumn.gif" width="25%"><img src="demo/src/homepage/winter.gif" width="25%">
<h3><strong><i>🚀 News</i></strong></h3>[2021-11-12]: The training code, test code and all the pretrained models are released in this code-base page.
[2021-10-22]: The paper has been accepted by the International Journal of Computer Vision (IJCV)! 🎉
[2021-09-21]: The datasets <a href="#am-2k"><strong>AM-2k</strong></a> and <a href="#bg-20k"><strong>BG-20k</strong></a> can now be <strong>openly accessed</strong> from the links below (both at Google Drive and at Baidu Wangpan) ! Please follow the dataset release agreements to access. Due to some privacy issues, the dataset <Strong>PM-10k</strong> will be published after privacy-preserving from the project Privacy-Preserving Portrait Matting (ACM MM 21). You can refer to this repo for access and updates.
| Dataset | <p>Dataset Link<br>(Google Drive)</p> | <p>Dataset Link<br>(Baidu Wangpan 百度网盘)</p> | Dataset Release Agreement| | :----:| :----: | :----: | :----: | |<strong>AM-2k</strong>|Link|Link (pw: 29r1)|Agreement (MIT License)| |<strong>BG-20k</strong>|Link|Link (pw: dffp)|Agreement (MIT License)|
[2020-11-17]: Create <a href="https://colab.research.google.com/drive/1EaQ5h4u9Q_MmDSFTDmFG0ZOeSsFuRTsJ?usp=sharing"><strong>Google Colab</strong></a> demo to benefit users who want to have a try online.
[2020-11-03]: Publish the <a href="#inference-code-how-to-test-on-your-images">inference code</a> and a pretrained model that can be used to test on your own animal images.
[2020-10-27]: Publish a video demo (YouTube | bilibili | Google drive) contains motivation, network, datasets, and test results on an animal video.
Demo on Google Colab
<p align="justify"> For those who do not have GPUs in their environment or only want to have a simple try online, you can try our <a href="https://colab.research.google.com/drive/1EaQ5h4u9Q_MmDSFTDmFG0ZOeSsFuRTsJ?usp=sharing">Google Colab</a> demo to generate the results for your images easily.</p><a href="https://colab.research.google.com/drive/1EaQ5h4u9Q_MmDSFTDmFG0ZOeSsFuRTsJ?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg"></a>
Introduction
<p align="justify">This repository contains the code, datasets, models, test results and a video demo for the paper <a href="https://arxiv.org/pdf/2010.16188.pdf">Bridging Composite and Real: Towards End-to-end Deep Image Matting</a>. We propose a novel Glance and Focus Matting network (<strong>GFM</strong>), which employs a shared encoder and two separate decoders to learn both tasks in a collaborative manner for end-to-end image matting. We also establish a novel Animal Matting dataset (<strong>AM-2k</strong>) to serve for end-to-end matting task. Furthermore, we investigate the domain gap issue between composition images and natural images systematically, propose a carefully designed composite route <strong>RSSN</strong> and a large-scale high-resolution background dataset (<strong>BG-20k</strong>) to serve as better candidates for composition.</p> <p align="justify">We have released the train code, the test code, the datasets, and the pretrained models in this <a href="https://github.com/JizhiziLi/GFM/tree/master/core">code-base page</a>. We have also prepared a <a href="https://colab.research.google.com/drive/1EaQ5h4u9Q_MmDSFTDmFG0ZOeSsFuRTsJ?usp=sharing">Google Colab</a> demo and <a href="#inference-code-how-to-test-on-your-images"><i>inference code</i></a> for you to test on our pre-trained models on your own sample images. For the datasets <strong>AM-2k</strong> and <strong>BG-20k</strong>, please follow the sections <a href="#am-2k"><i>AM-2k</i></a> and <a href="#bg-20k"><i>BG-20k</i></a> to access. Besides, we prepare a video demo (<a href="https://www.youtube.com/watch?v=FJPm4YQOEyo">YouTube</a> | <a href="https://www.bilibili.com/video/BV1X34y1o7nK">bilibili</a>) to illustrate the motivation, the network, the datasets, and the test results on an animal video</p>GFM
The architecture of our proposed end-to-end method <strong>GFM</strong> is illustrated below. We adopt three kinds of <em>Representation of Semantic and Transition Area</em> (<strong>RoSTa</strong>) -TT, -FT, -BT within our method.
We trained GFM with four backbones, -(r) (ResNet-34), -(d) (DenseNet-121), -(r2b) (ResNet-34 with 2 extra blocks), and -(r') (ResNet-101). The trained model for each backbone can be downloaded via the link listed below.
| Model| GFM(d)-TT | GFM(r)-TT | GFM(r)-FT | GFM(r)-BT |GFM(r2b)-TT | GFM(r')-TT | GFM(d)-RIM | | :----:| :----: | :----: | :----: | :----: | :----: | :----: | :----: | | Google Drive |Link|Link|Link|Link|Link|Link| Link | | <p>Baidu Wangpan<br>(百度网盘)</p> |<p><a href="https://pan.baidu.com/s/1AzuMphkNtt5-fJh-VqPnCA">Link</a><br>(pw: l6bd)</p>|<p><a href="https://pan.baidu.com/s/14TfNWeDGzXm4w91eHWu28w">Link</a><br>(pw: svcv)</p>|<p><a href="https://pan.baidu.com/s/1GmaXfiWbK09X4zhsRgooBg">Link</a><br>(pw: jfli)</p>|<p><a href="https://pan.baidu.com/s/1oaT5R8GnMW-zbbCwie1SlA">Link</a><br>(pw: 80k8)</p>|<p><a href="https://pan.baidu.com/s/1yfRGgI9QFUW9jb878AXHTg">Link</a><br>(pw: 34hf)</p>|<p><a href="https://pan.baidu.com/s/1aKUEB1MYIDbt-8iOHq67zQ">Link</a><br>(pw: 7p8j)</p>| <p><a href="https://pan.baidu.com/s/1V-YjxUsyzUsTRO8m6JoR_Q">Link</a><br>(pw: mrf7)</p>|
AM-2k
Our proposed <strong>AM-2k</strong> contains 2,000 high-resolution natural animal images from 20 categories along with manually labeled alpha mattes. Some examples are shown as below, more can be viewed in the video demo (YouTube | bilibili | Google drive).
<strong>AM-2k</strong> can be accessed from here (Google Drive | [Baidu Wangpan (pw: 29r1)](https://pan.baidu.com/s/1M1uF227-ZrYe
