✨MuSc (ICLR 2024)✨

This is an official PyTorch implementation for "MuSc : Zero-Shot Industrial Anomaly Classification and Segmentation with Mutual Scoring of the Unlabeled Images" (MuSc)

Authors: Xurui Li^1* | Ziming Huang^1* | Feng Xue³ | Yu Zhou^1,2

Institutions: ¹Huazhong University of Science and Technology | ²Wuhan JingCe Electronic Group Co.,LTD | ³University of Trento

🧐 Arxiv | OpenReview

📖 Chinese README

<a href='#all_catelogue'>Go to Catalogue</a>

🙈TODO list:

• ⬜️ Using some strategies to reduce the inference time per image from 955.3ms to 249.8ms.
• ⬜️ Compatibility with more industrial datasets.
• ⬜️ Compatibility with more visual backbones, e.g. Vision Mamba.

📣Updates:

04/11/2024

1. The comparisons with the zero/few-shot methods in CVPR 2024 have been added to <a href='#compare_sota'>Compare with SOTA k-shot Methods.</a>
2. Fixed some bugs in models/backbone/_backbones.py.

03/22/2024

1. The supported codes for BTAD dataset are provided.
2. Some codes are modified to support larger batch_size.
3. Some codes are optimized to obtain faster speeds.
4. <a href='#results_backbones'>Results of different backbones</a> in MVTec AD, VisA and BTAD datasets are provided.
5. <a href='#results_datasets'>The detailed results of different datasets</a> are provided.
6. <a href='#inference_time'>The inference time of different backbones</a> is provided.
7. <a href='#compare_sota'>The comparisons with SOTA zero/few-shot methods</a> are provided. This table will be updated continuously.
8. We summarize the <a href='#FAQ'> frequently asked questions </a> from users when using MuSc, and give the answers.
9. We add README in Chinese.

02/01/2024

Initial commits:

1. The complete code of our method MuSc in paper is released.
2. This code is compatible with image encoder (ViT) of CLIP and ViT pre-trained with DINO/DINO_v2.

<span id='compare_sota'/>

🎖️Compare with SOTA k-shot methods <a href='#all_catelogue'>[Go to Catalogue]</a>

We will continuously update the following table to compare our MuSc with the newest zero-shot and few-shot methods. "-" indicates that the authors did not measure this metric in their paper.

MVTec AD

| | | | Classification | | | Segmentation | | | | | :--------------------------------------: | :----------------: | :-----: | :------------: | :--------: | :----: | :----------: | :---------: | :-----: | :------: | | Methods | Venue | Setting | AUROC-cls | F1-max-cls | AP-cls | AUROC-segm | F1-max-segm | AP-segm | PRO-segm | | MuSc(ours) | ICLR 2024 | 0-shot | 97.8 | 97.5 | 99.1 | 97.3 | 62.6 | 62.7 | 93.8 | | RegAD | ECCV 2022 | 4-shot | 89.1 | 92.4 | 94.9 | 96.2 | 51.7 | 48.3 | 88.0 | | GraphCore | ICLR 2023 | 4-shot | 92.9 | - | - | 97.4 | - | - | - | | WinCLIP | CVPR 2023 | 0-shot | 91.8 | 92.9 | 96.5 | 85.1 | 31.7 | - | 64.6 | | WinCLIP | CVPR 2023 | 4-shot | 95.2 | 94.7 | 97.3 | 96.2 | 51.7 | - | 88.0 | | APRIL-GAN | CVPR Workshop 2023 | 0-shot | 86.1 | 90.4 | 93.5 | 87.6 | 43.3 | 40.8 | 44.0 | | APRIL-GAN | CVPR Workshop 2023 | 4-shot | 92.8 | 92.8 | 96.3 | 95.9 | 56.9 | 54.5 | 91.8 | | FastRecon | ICCV 2023 | 4-shot | 94.2 | - | - | 97.0 | - | - | - | | ACR | NeurIPS 2023 | 0-shot | 85.8 | 91.3 | 92.9 | 92.5 | 44.2 | 38.9 | 72.7 | | RegAD+Adversarial Loss | BMVC 2023 | 8-shot | 91.9 | - | - | 96.9 | - | - | - | | PACKD | BMVC 2023 | 8-shot | 95.3 | - | - | 97.3 | - | - | - | | PromptAD | WACV 2024 | 0-shot | 90.8 | - | - | 92.1 | 36.2 | - | 72.8 | | AnomalyCLIP | ICLR 2024 | 0-shot | 91.5 | - | 96.2 | 91.1 | - | - | 81.4 | | InCTRL | CVPR 2024 | 8-shot | 95.3 | - | - | - | - | - | - | | MVFA-AD | CVPR 2024 | 4-shot | 96.2 | - | - | 96.3 | - | - | - | | PromptAD | CVPR 2024 | 4-shot | 96.6 | - | - | 96.5 | - | - | - |

VisA

| | | | Classification | | | Segmentation | | | | | :--------------------------------------: | :----------------: | :-----: | :------------: | :--------: | :----: | :----------: | :---------: | :-----: | :------: | | Methods | Venue | Setting | AUROC-cls | F1-max-cls | AP-cls | AUROC-segm | F1-max-segm | AP-segm | PRO-segm | | MuSc(ours) | ICLR 2024 | 0-shot | 92.8 | 89.5 | 93.5 | 98.8 | 48.8 | 45.1 | 92.7 | | WinCLIP | CVPR 2023 | 0-shot | 78.1 | 79.0 | 81.2 | 79.6 | 14.8 | - | 56.8 | | WinCLIP | CVPR 2023 | 4-shot | 87.3 | 84.2 | 88.8 | 97.2 | 47.0 | - | 87.6 | | APRIL-GAN | CVPR Workshop 2023 | 0-shot | 78.0 | 78.7 | 81.4 | 94.2 | 32.3 | 25.7 | 86.8 | | APRIL-GAN | CVPR Workshop 2023 | 4-shot | 92.6 | 88.4 | 94.5 | 96.2 | 40.0 | 32.2 | 90.2 | | PACKD | BMVC 2023 | 8-shot | 87.5 | - | - | 97.9 | - | - | - | | AnomalyCLIP | ICLR 2024 | 0-shot | 82.1 | - | 85.4 | 95.5 | - | - | 87.0 | | InCTRL | CVPR 2024 | 8-shot | 88.7 | - | - | - | - | - | - | | PromptAD | CVPR 2024 | 4-shot | 89.1 | - | - | 97.4 | - | - | - |

<span id='all_catelogue'/>

📖Catalogue

• <a href='#abstract'>1. Abstract</a>
• <a href='#setup'>2. Environment setup</a>
• <a href='#datasets'>3. Datasets download</a>
  • <a href='#datatets_mvtec_ad'>MVTec AD</a>
  • <a href='#datatets_visa'>VisA</a>
  • <a href='#datatets_btad'>BTAD</a>
• <a href='#run_musc'>4. Run MuSc</a>
• <a href='#rscin'>5. Run RsCIN</a>
• <a href='#results_datasets'>6. Results of different datasets</a>
• <a href='#results_backbones'>7. Results of different backbones</a>
• <a href='#inference_time'>8. Inference time</a>
• <a href='#FAQ'>9. Frequently Asked Questions</a>
• <a href='#citation'>10. Citation</a>
• <a href='#thanks'>11. Thanks</a>
• <a href='#license'>12. License</a>

<span id='abstract'/>

👇Abstract: <a href='#all_catelogue'>[Back to Catalogue]</a>

This paper studies zero-shot anomaly classification (AC) and segmentation (AS) in industrial vision. We reveal that the abundant normal and abnormal cues implicit in unlabeled test images can be exploited for anomaly determination, which is ignored by prior methods. Our key observatio