UCCH

Peng Hu, Hongyuan Zhu, Jie Lin, Dezhong Peng, Yin-Ping Zhao, Xi Peng*,Unsupervised Contrastive Cross-modal Hashing, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 45, no. 3, pp. 3877-3889, 1 March 2023, doi: 10.1109/TPAMI.2022.3177356. (PyTorch Code)

Abstract

In this paper, we study how to make unsupervised cross-modal hashing (CMH) benefit from contrastive learning (CL) by overcoming two challenges. To be exact, i) to address the performance degradation issue caused by binary optimization for hashing, we propose a novel momentum optimizer that performs hashing operation learnable in CL, thus making on-the-shelf deep cross-modal hashing possible. In other words, our method does not involve binary-continuous relaxation like most existing methods, thus enjoying better retrieval performance; ii) to alleviate the influence brought by false-negative pairs (FNPs), we propose a Cross-modal Ranking Learning loss (CRL) which utilizes the discrimination from all instead of only the hard negative pairs, where FNP refers to the within-class pairs that were wrongly treated as negative pairs. Thanks to such a global strategy, CRL endows our method with better performance because CRL will not overuse the FNPs while ignoring the true-negative pairs. To the best of our knowledge, the proposed method could be one of the first successful contrastive hashing methods. To demonstrate the effectiveness of the proposed method, we carry out experiments on five widely-used datasets compared with 13 state-of-the-art methods. The code is available at https://github.com/penghu-cs/UCCH.

Framework

<h4>Figure 1 The pipeline of the proposed method and we take a bimodal case as an example. In the example, two modality-specific networks learn unified binary representations for different modalities. The outputs of networks directly interact with the hash codes to learn the latent discrimination by using instance-level contrast without continuous relaxation, i.e., contrastive hashing learning (𝓛_𝒸). The cross-modal ranking loss 𝓛_𝑟 is utilized to bridge cross-modal hashing learning to cross-modal retrieval. </h4> <img src=paper/UCCH.jpg class='center' \>

Usage

To train a model with 128 bits on MIRFLICKR-25K, just run UCCH.py:

# Features
python UCCH.py --data_name mirflickr25k_fea --bit 128 --alpha 0.7 --num_hiden_layers 3 2 --margin 0.2 --max_epochs 20 --train_batch_size 256 --shift 0.1 --lr 0.0001 --optimizer Adam

# Raw data
python UCCH.py --data_name mirflickr25k --bit 128 --alpha 0.7 --num_hiden_layers 3 2 --margin 0.2 --max_epochs 20 --train_batch_size 256 --shift 0.1 --lr 0.0001 --optimizer Adam --warmup_epoch 5 --pretrain -a vgg11

You can get outputs as follows:

Epoch: 13 / 20
[================= 70/70 ====================>]  Step: 28ms | Tot: 2s18ms | Loss: 13.205 | LR: 0.0001                                                                                                             
Evaluation:	Img2Txt: 0.75797 	 Txt2Img: 0.759172 	 Avg: 0.758571

Epoch: 14 / 20
[================= 70/70 ====================>]  Step: 28ms | Tot: 1s951ms | Loss: 13.193 | LR: 0.0001                                                                                                            
Evaluation:	Img2Txt: 0.759404 	 Txt2Img: 0.759482 	 Avg: 0.759443

Epoch: 15 / 20
[================= 70/70 ====================>]  Step: 28ms | Tot: 1s965ms | Loss: 13.180 | LR: 0.0001                                                                                                            
Evaluation:	Img2Txt: 0.758604 	 Txt2Img: 0.75909 	 Avg: 0.758847

Epoch: 16 / 20
[================= 70/70 ====================>]  Step: 28ms | Tot: 1s973ms | Loss: 13.170 | LR: 0.0001                                                                                                            
Evaluation:	Img2Txt: 0.758019 	 Txt2Img: 0.757934 	 Avg: 0.757976

Epoch: 17 / 20
[================= 70/70 ====================>]  Step: 28ms | Tot: 1s973ms | Loss: 13.160 | LR: 0.0001                                                                                                            
Evaluation:	Img2Txt: 0.757612 	 Txt2Img: 0.758054 	 Avg: 0.757833

Epoch: 18 / 20
[================= 70/70 ====================>]  Step: 29ms | Tot: 1s968ms | Loss: 13.151 | LR: 0.0001                                                                                                            
Evaluation:	Img2Txt: 0.757199 	 Txt2Img: 0.757834 	 Avg: 0.757517

Epoch: 19 / 20
[================= 70/70 ====================>]  Step: 30ms | Tot: 2s43ms | Loss: 13.144 | LR: 0.0001                                                                                                             
Evaluation:	Img2Txt: 0.757373 	 Txt2Img: 0.757289 	 Avg: 0.757331
Test:	Img2Txt: 0.769567 	 Txt2Img: 0.746658 	 Avg: 0.758112

Comparison with the State-of-the-Art

<table> <thead> <h4>TABLE 1: Performance comparison in terms of MAP scores on the MIRFLICKR-25K and IAPR TC-12 datasets. The highest score is shown in <b>boldface</b>.</h4> <tr> <th class="tg-0pky" rowspan="3", align="center">Method</th> <th class="tg-c3ow" colspan="8", align="center">MIRFLICKR-25K</th> <th class="tg-c3ow" colspan="8", align="center">IAPR TC-12</th> </tr> <tr> <td class="tg-c3ow" colspan="4", align="center">Image → Text</td> <td class="tg-c3ow" colspan="4", align="center">Text → Image</td> <td class="tg-c3ow" colspan="4", align="center">Image → Text</td> <td class="tg-c3ow" colspan="4", align="center">Text → Image</td> </tr> <tr> <td class="tg-c3ow">16</td> <td class="tg-c3ow">32</td> <td class="tg-c3ow">64</td> <td class="tg-c3ow">128</td> <td class="tg-c3ow">16</td> <td class="tg-c3ow">32</td> <td class="tg-c3ow">64</td> <td class="tg-c3ow">128</td> <td class="tg-c3ow">16</td> <td class="tg-c3ow">32</td> <td class="tg-c3ow">64</td> <td class="tg-c3ow">128</td> <td class="tg-c3ow">16</td> <td class="tg-c3ow">32</td> <td class="tg-c3ow">64</td> <td class="tg-c3ow">128</td> </tr> </thead> <tbody> <tr> <td class="tg-0pky">CVH[20]</td> <td class="tg-c3ow">0.620</td> <td class="tg-c3ow">0.608</td> <td class="tg-c3ow">0.594</td> <td class="tg-c3ow">0.583</td> <td class="tg-c3ow">0.629</td> <td class="tg-c3ow">0.615</td> <td class="tg-c3ow">0.599</td> <td class="tg-c3ow">0.587</td> <td class="tg-c3ow">0.392</td> <td class="tg-c3ow">0.378</td> <td class="tg-c3ow">0.366</td> <td class="tg-c3ow">0.353</td> <td class="tg-c3ow">0.398</td> <td class="tg-c3ow">0.384</td> <td class="tg-c3ow">0.372</td> <td class="tg-c3ow">0.360</td> </tr> <tr> <td class="tg-0pky">LSSH[59]</td> <td class="tg-c3ow">0.597</td> <td class="tg-c3ow">0.609</td> <td class="tg-c3ow">0.606</td> <td class="tg-c3ow">0.605</td> <td class="tg-c3ow">0.602</td> <td class="tg-c3ow">0.598</td> <td class="tg-c3ow">0.598</td> <td class="tg-c3ow">0.597</td> <td class="tg-c3ow">0.372</td> <td class="tg-c3ow">0.386</td> <td class="tg-c3ow">0.396</td> <td class="tg-c3ow">0.404</td> <td class="tg-c3ow">0.367</td> <td class="tg-c3ow">0.380</td> <td class="tg-c3ow">0.392</td> <td class="tg-c3ow">0.401</td> </tr> <tr> <td class="tg-0pky">CMFH[60]</td> <td class="tg-c3ow">0.557</td> <td class="tg-c3ow">0.557</td> <td class="tg-c3ow">0.556</td> <td class="tg-c3ow">0.557</td> <td class="tg-c3ow">0.553</td> <td class="tg-c3ow">0.553</td> <td class="tg-c3ow">0.553</td> <td class="tg-c3ow">0.553</td> <td class="tg-c3ow">0.312</td> <td class="tg-c3ow">0.314</td> <td class="tg-c3ow">0.314</td> <td class="tg-c3ow">0.315</td> <td class="tg-c3ow">0.306</td> <td class="tg-c3ow">0.306</td> <td class="tg-c3ow">0.306</td> <td class="tg-c3ow">0.306</td> </tr> <tr> <td class="tg-0pky">FSH[18]</td> <td class="tg-c3ow">0.581</td> <td class="tg-c3ow">0.612</td> <td class="tg-c3ow">0.635</td> <td class="tg-c3ow">0.662</td> <td class="tg-c3ow">0.576</td> <td class="tg-c3ow">0.607</td> <td class="tg-c3ow">0.635</td> <td class="tg-c3ow">0.660</td> <td class="tg-c3ow">0.377</td> <td class="tg-c3ow">0.392</td> <td class="tg-c3ow">0.417</td> <td class="tg-c3ow">0.445</td> <td class="tg-c3ow">0.383</td> <td class="tg-c3ow">0.399</td> <td class="tg-c3ow">0.425</td> <td class="tg-c3ow">0.451</td> </tr> <tr> <td class="tg-0pky">DLFH[23]</td> <td class="tg-c3ow">0.638</td> <td class="tg-c3ow">0.658</td> <td class="tg-c3ow">0.677</td> <td class="tg-c3ow">0.684</td> <td class="tg-c3ow">0.675</td> <td class="tg-c3ow">0.700</td> <td class="tg-c3ow">0.718</td> <td class="tg-c3ow">0.725</td> <td class="tg-c3ow">0.342</td> <td class="tg-c3ow">0.358</td> <td class="tg-c3ow">0.374</td> <td class="tg-c3ow">0.395</td> <td class="tg-c3ow">0.358</td> <td class="tg-c3ow">0.380</td> <td class="tg-c3ow">0.403</td> <td class="tg-c3ow">0.434</td> </tr> <tr> <td class="tg-0pky">MTFH[16]</td> <td class="tg-c3ow">0.507</td> <td class="tg-c3ow">0.512</td> <td class="tg-c3ow">0.558</td> <td class="tg-c3ow">0.554</td> <td class="tg-c3ow">0.514</td> <td class="tg-c3ow">0.524</td> <td class="tg-c3ow">0.518</td> <td class="tg-c3ow">0.581</td> <td class="tg-c3ow">0.277</td> <td class="tg-c3ow">0.324</td> <td class="tg-c3ow">0.303</td> <td class="tg-c3ow">0.311</td> <td class="tg-c3ow">0.294</td> <td class="tg-c3ow">0.337</td> <td class="tg-c3