SIHR: a MATLAB/GNU Octave toolbox for single image highlight removal

Citation

@article{Ramos2020,
  doi = {10.21105/joss.01822},
  url = {https://doi.org/10.21105/joss.01822},
  year = {2020},
  month = jan,
  publisher = {The Open Journal},
  volume = {5},
  number = {45},
  pages = {1822},
  author = {V{\'{\i}}tor Ramos},
  title = {{SIHR}: a {MATLAB}/{GNU} {Octave} toolbox for single image highlight removal},
  journal = {Journal of Open Source Software}
}

Summary

An ongoing effort of developing new and implementing established single image highlight removal (SIHR) methods on MATLAB/GNU Octave.

Highlight, specularity, or specular reflection removal (see ¹ for a proper Web of Science expression, see [1] for a reference work entry, see [2], [3] for survey on this problem) concerns the following decomposition.

I welcome and encourage contributions to this project upon review. Please check CONTRIBUTING.md for more details.

Disclaimer 1: this repository is intended for research purposes only.
Disclaimer 2: ~~some~~ most of these methods are based on chromaticity analysis, they fail miserably for grayscale images.

¹ ((remov* NEAR/1 (highlight* OR specular*)) OR (separat* NEAR/1 (reflect* OR specular*)))

Raison d'être

I started out this repository by implementing, translating and collecting code snippets from the rare available^2,3,4,5 codes. Oftentimes papers are cryptical, codes are in C/C++ (requires compilation and major source code modification for general testing), or are just unavailable. See, e.g. this CSDN post⁶ that has no valid links at all.

In this context, this repository aims to be a continous algorithmic aid for ongoing research and development of SIHR methods.

² Tan and Ikeuchi. [Online]. Available: http://tanrobby.github.io/code/highlight.zip
³ Shen et al. [Online]. Available: http://ivlab.org/publications/PR2008_code.zip
⁴ ~~Yang et al. [Online]. Available: http://www6.cityu.edu.hk/stfprofile/qiyang.htm~~
⁵ Shen and Zheng. [Online]. Available: http://ivlab.org/publications/AO2013_code.zip
⁶ ~~https://blog.csdn.net/nvidiacuda/article/details/8078167~~

Usage (API)

Calling this toolbox's functions is very straightforward:

I_d = AuthorYEAR(I); % I is a double-valued input image of dimension
                     % m×n×3 containing linear RGB values and
                     % I_d is the calculated diffuse reflection
                     % using AuthorYEAR method.
                     % The specular component is simply
                     % I_s = I - I_d;

Methods

The following methods are available.

|Year| Method | Function | |:--:|----------------------------|----------------| |2005| Tan and Ikeuchi [4] | Tan2005 | |2006| Yoon et al. [5] | Yoon2006 | |2008| Shen et al. [6] | Shen2008 | |2009| Shen and Cai [7] | Shen2009 | |2010| Yang et al. [8] | Yang2010 | |2013| Shen and Zheng [9] | Shen2013 | |2016| Akashi and Okatani [10] | Akashi2016 |

The following improvement is available.

|Year| Method | Function | |:--:|----------------------------|----------------| |2019| Yamamoto and Nakazawa [11] | Yamamoto2019 |

Environment

The environment this repository is being developed is:

• MATLAB
  • Image Processing Toolbox
• GNU Octave
  • Image package
    • pkg install -forge image

Tested environments

Octave 4.2                                Ubuntu 18.04
Octave 5.1 (latest)    Windows 10 1903
MATLAB 9.1 (R2016b)    Windows 10 1903
MATLAB 9.6 (R2019a)    Windows 10 1903    Ubuntu 16.04 (MATLAB Online)

Installation

1. git clone https://github.com/vitorsr/SIHR.git or download a copy of the repository.
2. Start Octave or MATLAB.
   1. cd('path/to/SIHR'), i.e. change current folder to SIHR root (where SIHR.m is located).
   2. run SIHR.m for session path setup.
   3. help SIHR or doc SIHR provides a summary of the methods available.

Additional Debian/Ubuntu installation

To install the image package from Octave Forge, build-essential and liboctave-dev need to be present. Install them via apt, then proceed with package installation.

sudo apt-get install -qq -y build-essential liboctave-dev
octave --eval "pkg install -forge image"

Performance

This section aims to clarify how well (or not) the methods reproduced in this project were at reproducing results in literature.

Note: Akashi and Okatani's [10] method has highly fluctuating results because of random initialization.

Dataset

In technical literature, there exist two ground truth datasets commonly used right now. One by Shen and Zheng [9] which is distributed alongside their code, and one by Grosse et al. [12] in a dedicated page⁷.

Other test images are included alongside the code for Shen et al. [6] and Yang et al. [8].

Follow the instructions in images in order to download a local copy of these images from the respective authors' pages.

⁷ Grosse et al. [Online]. Available: http://www.cs.toronto.edu/~rgrosse/intrinsic/

Quality

Quantitative results reported are usually regarding the quality of the recovered diffuse component with respect to the ground truth available in the Shen and Zheng [9] test image set.

Automated testing

Reproduced results below are available in the utils/automated_testing.m script.

Note: ssim is not available in Octave Forge image.

Highest (self and peer-reported | reproduced) PSNR results (in dB)

|Year| Method | animals | cups | fruit | masks | Reproduced | animals | cups | fruit | masks | |:--:|--------------------|:---------:|:---------:|:---------:|:---------:|--------------|:---------:|:------:|:-------:|:-------:| |2005| Tan and Ikeuchi | 30.2 | 30.1 | 29.6 | 25.6 | Tan2005 | 30.4 | 31.6 | 30.4 | 25.8 | |2006| Yoon et al. | - | - | - | - | Yoon2006 | 32.9 | 33.3 | 36.6 | 34.1 | |2008| Shen et al. | 34.6 | 37.7 | 37.6 | 31.7 | Shen2008 | 34.2 | 37.5 | 38.0 | 32.1 | |2009| Shen and Cai | 34.8 | 37.6 | 36.9 | 34.0 | Shen2009 | 34.9 | 37.6 | 36.7 | 34.0 | |2010| Yang et al. | 37.2 | 38.0 | 37.6 | 32.2 | Yang2010 | 36.5 | 37.5 | 36.2 | 33.5 | |2013| Shen and Zheng | 37.3 | 39.3 | 38.9 | 34.1 | Shen2013 | 37.5 | 38.3 | 38.2 | 32.7 | |2015| Liu et al. | 33.4 | 37.6 | 35.1 | 34.5 | - | - | - | - | - | |2016| Akashi and Okatani | 26.8 | 35.7 | 30.8 | 32.3 | Akashi2016 | 32.7 | 35.9 | 34.8 | 34.0 | |2016| Suo et al. | - | - | 40.4 | 34.2 | - | - | - | - | - | |2017| Ren et al. | - | 38.0 | 37.7 | 34.5 | - | - | - | - | - | |2018| Guo et al. | 35.7 | 39.1 | 36.4 | 34.4 | - | - | - | - | - |

Highest (self and peer-reported | reproduced) SSIM results

|Year| Method | animals | cups | fruit | masks | Reproduced | animals | cups | fruit | masks | |:--:|--------------------|:---------:|:---------:|:---------:|:---------:|--------------|:---------:|:------:|:-------:|:-------:| |2005| Tan and Ikeuchi | 0.929 | 0.767 | 0.912 | 0.789 | Tan2005 | 0.928 | 0.895 | 0.907 | 0.821 | |2006| Yoon et al. | - | - | - | - | Yoon2006 | 0.980 | 0.961 | 0.961 | 0.953 | |2008| Shen et al. | 0.974 | 0.962 | 0.961 | 0.943 | Shen2008 | 0.975 | 0.962 | 0.961 | 0.943 | |2009| Shen and Cai | - | - | - | - | Shen2009 | 0.985 | 0.970 | 0.962 | 0.961 | |2010| Yang et al. | 0.970 | 0.941 | 0.939 | 0.899 | Yang2010 | 0.952 | 0.937 | 0.916 | 0.896 | |2013| Shen and Zheng | 0.971 | 0.966 | 0.960 | 0.941 | Shen2013 | 0.985 | 0.964 | 0.958 | 0.935 | |2015| Liu et al. | - | - | - | - | - | - | - | - | - | |2016| Akashi and Okatani | 0.802 | 0.937 | 0.765 | 0.657 | Akashi2016 | 0.7340 | 0.9190 | 0.9010 | 0.8710 | |2016| Suo et al.