SIDBench: A Synthetic Image Detection Benchmark

This project is a framework for benchmarking several state-of-the-art synthetic image detection models.

News

<!-- :boom: **29/3/2024** Release traning code -->

:sparkles: 12/3/2024 Integrate DeFake model

Setup

Install dependecies

Create a new environment named pytorch_env:

conda create --name pytorch_env python=3.11
conda activate pytorch_env

and then install pytorch dependecies

conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

Note :sparkles:: This framework has been tested with Python version 3.11.7 and PyTorch 2.1.2. However, it should work with other versions as well. Please ensure that your PyTorch version is greater than 2.0. :warning:

Install additional dependencies:

pip install -r requirements.txt

To run the Dire model, mpi4py is required. You can install it using Conda with the following command:

conda install -c conda-forge mpi4py mpich

Integrated Models

The following models have been integrated. Note that for some of these models, there are multiple pretrained instances, trained on different images generated by various generative models such as ProGAN, StyleGAN, and Latent Diffusion, etc.

|Model Name|Paper Title|Original Code| |:--------:|:------:|:------:| |CNNDetect|CNN-generated images are surprisingly easy to spot...for now | :link:| |DIMD| On the detection of synthetic images generated by diffusion models. | :link:| |FreqDetect|Leveraging Frequency Analysis for Deep Fake Image Recognition | :link:| |Fusing|Fusing global and local features for generalized AI-synthesized image detection| :link:| |GramNet|Global Texture Enhancement for Fake Face Detection In the Wild | :link:| |LGrad|Learning on Gradients: Generalized Artifacts Representation for GAN-Generated Images Detection | :link:| |Dire|DIRE for Diffusion-Generated Image Detection | :link:| |UnivFD|Towards Universal Fake Image Detectors that Generalize Across Generative Models | :link:| |NPR|Rethinking the Up-Sampling Operations in CNN-based Generative Network for Generalizable Deepfake Detection|:link:| |PatchCraft |PatchCraft: Exploring Texture Patch for Efficient AI-generated Image Detection |:link: | |DeFake |DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models | :link: | |Rine |Leveraging Representations from Intermediate Encoder-blocks for Synthetic Image Detection | :link: |

Usage

Run model on multiple images

To run the model on a directory containing images, use the following command:

python test.py --dataPath <root_path_to_images>

This command executes the model using the default selection, which is UnivFD. If you wish to use a different model, you can specify it using the --modelName flag. For example, to use the CNNDetect model, the command would be:

python test.py --dataPath <root_path_to_images> --modelName=CNNDetect

The models supported by this framework are listed in the table above. When selecting a model using the --modelName flag, ensure you use one of the valid names as specified below. These names correspond to the models' implementations and must be used exactly as shown to ensure proper function invocation:

VALID_MODELS = ['CNNDetect', 'FreqDetect', 'Fusing', 'GramNet', 'LGrad', 'UnivFD', 'PatchCraft', 'Rine', 'DIMD', 'NPR', 'Dire', 'DeFake']

You need also to define the path to the pretrained weights with the --cptk flag. Ensure you replace <path_to_pretrained_weights> with the actual file path to your pretrained model weights.

python test.py --dataPath <root_path_to_images> --modelName=CNNDetect --cptk <path_to_pretrained_weights>

Replace <root_path_to_images> with the actual path to your directory of images, and <path_to_pretrained_weights> with the path to the pretrained weights file.

You can download the pretrained weights here: Google Drive

| Model Name | Pretrained Weights File Name | Trained On | |---------------|-------------------------------------------------------------------|-------------| | CNNDetect | weights/cnndetect/blur_jpg_prob0.1.pth | proGAN augmented (recompressed) with 10% probability | | | weights/cnndetect/blur_jpg_prob0.5.pth | proGAN augmented (recompressed) with 50% probability | | DIMD | weights/dimd/corvi22_latent_model.pth | Latent Diffusion | | | weights/dimd/corvi22_progan_model.pth | proGAN | | | weights/dimd/gandetection_resnet50nodown_stylegan2.pth | styleGAN2 images | | Dire | weights/dire/lsun_adm.pth | adm (diffusion) | | | weights/dire/lsun_iddpm.pth | iddpm (Denoising diffusion probabilistic model) | | | weights/dire/lsun_pndm.pth | pndm (Pseudo Numerical Methods for Diffusion Models) | | | weights/dire/lsun_stylegan.pth | stylegan | | FreqDetect| weights/freqdetect/DCTAnalysis.pth | | | UnivFD | weights/univfd/fc_weights..pth | proGAN | | Fusing | weights/fusing/PSM.pth | | | GramNet | weights/gramnet/Gram.pth | | | LGrad | weights/lgrad/LGrad-1class-Trainon-Progan_horse.pth | proGAN with one class images | | | weights/lgrad/LGrad-2class-Trainon-Progan_chair_horse.pth | proGAN with two class images | | | weights/lgrad/LGrad-4class-Trainon-Progan_car_cat_chair_horse.pth | proGAN with four class images | | NPR | weights/npr/NPR.pth | | | DeFake | weights/defake/clip_linear.pth | hybrid detector (image + text) tranined in diffusion images | | Rine | weights/rine/model_1class_trainable.pth | proGAN with one class images | | | weights/rine/model_2class_trainable.pth | proGAN with two class images | | | weights/rine/model_4class_trainable.pth | proGAN with four class images | | | weights/rine/model_ldm_trainable.pth | latent diffusion with one class images | |PatchCraft | weights/rptc/RPTC.pth | proGAN |

Some models require additional parameters to be defined.

FreqDetect requires two additional files, which are to be specified by the flags --dctMean and --dctVar. By default, these are set to ./weights/freqdetect/dct_mean and ./weights/freqdetect/dct_var, respectively. If you have downloaded the weights directory and placed it in the root directory of the framework, these parameters can remain unchanged.

LGrad requires the initialization of a StyleGAN discriminator, which is used to extract image gradients serving as image features. To specify the path to the pretrained discriminator, use the flag --LGradGenerativeModelPath. The default pretrained weights are provided in the file karras2019stylegan-bedrooms-256x256_discriminator.pth, located within the ./weights/preprocessing directory.

Dire requires the initialization of a diffusion generative model that extracts image features. To specify the path to the pretrained discriminator, use the flag --DireGenerativeModelPath. The default pretrained weights are provided in the file lsun_bedroom.pt, located within the ./weights/preprocessing directory.

DeFake requires the initialization of a fine-tuned CLIP encoder (--defakeClipEncoderPath=./weights/defake/finetune_clip.pt) and a BLIP decoder for the generation of images captions (--defakeBlipPath=./weights/defake/model_base_capfilt_large.pth).

Saving the predictions

To save the predictions, specify an output file using the --predictionsFile flag. For example:

python test.py --dataPath <root_path_to_images> --predictionsFile <path_to_output_file>

Additional parameters for input preprocessing

To resize images, use the --resizeSize flag followed by the desired dimension. f no resize size is specified, the default behavior is to apply no resizing. For example, to resize images to 256x256 pixels, you would use:

--resizeSize=256

Important Note on Resizing: The impact of resizing on the results cannot be overstated. For certain models, resizing can significantly improve outcomes, while for others, it may detract from performance. This effect is closely tied to the original resolution of the input images. Specifically, in the case of high-resolution images, resizing becomes a crucial factor to consider. For a deeper insight into how resizing affects model performance, please refer to t