<p align="center" width="100%"> <img src="assets\NORTON-Logo.png" width="80%" height="80%"> </p> <div> <div align="center"> <a href='https://pageperso.lis-lab.fr/van-tien.pham/' target='_blank'>Van Tien Pham^1,&#x2709</a>&emsp; <a href='https://yzniyed.blogspot.com/p/about-me.html' target='_blank'>Yassine Zniyed¹</a>&emsp; <a href='http://tpnguyen.univ-tln.fr/' target='_blank'>Thanh Phuong Nguyen¹</a>&emsp; </div> <div> <div align="center"> ¹Université de Toulon, Aix Marseille Université, CNRS, LIS, UMR 7020, France&emsp; ^&#x2709 Corresponding Author </div>

---

Enhanced Network Compression Through Tensor Decompositions and Pruning

In this work, we propose NORTON (enhanced Network cOmpRession through TensOr decompositions and pruNing), a novel method for network compression. NORTON introduces the concept of filter decomposition, enabling a more detailed decomposition of the network while preserving the weight's multidimensional properties. Our method incorporates a novel structured pruning approach, effectively integrating the decomposed model. Through extensive experiments on various architectures, benchmark datasets, and representative vision tasks, we demonstrate the usefulness of our method. NORTON achieves superior results compared to state-of-the-art techniques in terms of complexity and accuracy.

<div> <img class="image" src="assets\OverallFramework.png" width="51%" height="100%"> <img class="image" src="assets\OneLayer.png" width="48%" height="100%"> </div> <div align="center "> Left: Graphic illustration of the NORTON approach. Right: The decomposition and then pruning process for one layer. </div> <p align="left"> <img src="assets\CPDBlock.png" width=100%> </p> <div align="center "> Illustration of the CPDBlock structure for conventional deep learning frameworks </div>

🌟 News

• 2024.02.23: Paper accepted by IEEE TNNLS :confetti_ball:. Preprint available here.
• 2024.02.01: Add ablation study :bar_chart: on the impact of rank and pruning ratio selection.
• 2023.11.14: Add qualitative assessment of feature preservation.
• 2023.10.31: :ghost: :jack_o_lantern: Add instance segmentation and keypoint detection visualization.
• 2023.8.23: Throughput acceleration :stars: experiment is released :tada:.
• 2023.8.01: Detail instructions for checkpoint verification are released.
• 2023.7.28: Baseline and compressed checkpoints :gift: are released.
• 2023.7.26: Paper submitted to IEEE TNNLS. The code is released. Stay tuned for more exciting updates!⌛

🚩 Main results

<div align="center "> <img class="image" src="assets\Vgg16-CIFAR10.png" width="40%" height="100%"> </div> <div align="center "> The accuracy-MACs reduction Pareto curves of compressed VGG-16 models are compared on CIFAR-10. </div>

In order to demonstrate the adaptability of NORTON, we assess three representative architectures: VGG-16-BN, ResNet-56/110 with residual blocks, and DenseNet-40 with dense blocks. These models are tested on the CIFAR-10 dataset. Additionally, to validate the scalability of NORTON, experiments are conducted on the challenging ImageNet dataset using the ResNet-50 architecture. Furthermore, the compressed ResNet-50 model is employed as the backbone network for FasterRCNN-FPN, MaskRCNN, and KeypointRCNN on the COCO-2017 dataset.

NORTON is compared with the SOTA in the fields of low-rank decompositions, structured pruning, and hybrid methods.

<details> <summary><strong>1. VGG-16-BN/CIFAR-10</strong></summary> <div align="center">

| Model | Top-1 (%)| MACs (↓%) | Params. (↓%) | |-----------------------------|----------|---------------|---------------| | VGG-16-BN | 93.96 | 313.73M (00) | 14.98M (00) | | HRank-1 | 93.43 | 145.61M (54) | 2.51M (83) | | CHIP | 93.86 | 131.17M (58) | 2.76M (82) | | EZCrop | 93.01 | 131.17M (58) | 2.76M (82) | | DECORE-500 | 94.02 | 203.08M (35) | 5.54M (63) | | AutoBot | 94.19 | 145.61M (54) | 7.53M (50) | | NORTON (Ours) | 94.45 | 126.49M (60) | 2.58M (83) | | HRank-2 | 92.34 | 108.61M (65) | 2.64M (82) | | DECORE-200 | 93.56 | 110.51M (65) | 1.66M (89) | | EZCrop | 93.70 | 104.78M (67) | 2.50M (83) | | CHIP | 93.72 | 104.78M (67) | 2.50M (83) | | AutoBot | 94.01 | 108.71M (65) | 6.44M (57) | | NORTON (Ours) | 94.16 | 101.91M (68) | 2.34M (84) | | WhiteBox | 93.47 | 75.30M (76) | N/A | | AutoBot | 93.62 | 72.60M (77) | 5.51M (63) | | NORTON (Ours) | 94.11 | 74.14M (77) | 3.60M (76) | | QSFM | 92.17 | 79.00M (75) | 3.68M (75) | | DECORE-100 | 92.44 | 51.20M (82) | 0.51M (96) | | FSM | 92.86 | 59.61M (81) | 1.50M (90) | | ALDS | 92.67 | 66.95M (86) | 1.90M (96) | | Lebedev et al. | 93.07 | 68.53M (78) | 3.22M (78) | | EPruner-0.73 | 93.08 | 74.42M (76) | 1.65M (89) | | HALOC | 93.16 | 43.92M (86) | 0.30M (98) | | CHIP | 93.18 | 66.95M (79) | 1.90M (87) | | ASTER | 93.45 | 60.00M (81) | N/A | | FSM | 93.73 | 106.67M (66) | 2.10M (86) | | NORTON (Ours) | 93.84 | 37.68M (88) | 1.94M (87) | | HRank-3 | 91.23 | 73.70M (77) | 1.78M (92) | | DECORE-50 | 91.68 | 36.85M (88) | 0.26M (98) | | NORTON (Ours) | 92.54 | 13.54M (96) | 0.24M (98) | | NORTON (Ours) | 90.32 | 4.58M (99) | 0.14M (99) |

</div> </details> <details> <summary><strong>2. ResNet-56/110/CIFAR-10</strong></summary> <div align="center">

| Model | Top-1(%) | MACs (↓%) | Params. (↓%) | |-------------------------|---------|--------------|--------------| | ResNet-56 | 93.26 | 125.49M (00) | 0.85M (00) | | HRank-1 | 93.52 | 88.72M (29) | 0.71M (17) | | DECORE-450 | 93.34 | 92.48M (26) | 0.64M (24) | | FilterSketch | 93.65 | 88.05M (30) | 0.68M (21) | | TPP | 93.81 | 86.59M (31) | N/A | | WHC | 93.91 | 90.35M (28) | N/A | | NORTON (Ours) | 94.46 | 93.34M (27) | 0.58M (31) | | HRank-2 | 93.17 | 62.72M (50) | 0.49M (42) | | FilterSketch | 93.19 | 73.36M (41) | 0.50M (41) | | DECORE-200 | 93.26 | 62.93M (50) | 0.43M (49) | | TPP | 93.46 | 62.75M (50) | N/A | | MFP | 93.56 | 59.40M (53) | N/A | | FSM | 93.63 | 61.49M (51) | 0.48M (44) | | CC-0.5 | 93.64 | 60.00M (52) | 0.44M (48) | | NORTON (Ours) | 94.00 | 73.22M (42) | 0.44M (48) | | QSFM | 91.88 | 50.62M (60) | 0.25M (71) | | CHIP | 92.05 | 34.79M (72) | 0.24M (72) | | TPP | 92.35 | 36.39M (71) | N/A | | NORTON (Ours) | 93.81 | 37.52M (71) | 0.21M (75) | | HRank-3 | 90.72 | 32.52M (74) | 0.27M (68) | | DECORE-55 | 90.85 | 23.22M (81) | 0.13M (85) | | FilterSketch | 91.20 | 32.47M (74) | 0.24M (72) | | NORTON (Ours) | 91.62 | 14.47M (89) | 0.08M (91) | | ResNet-110 | 93.50 | 256.04M (00) | 1.73M (00) | | DECORE-500 | 93.88 | 163.30M (35) | 1.11M (36) | | NORTON (Ours) | 94.85 | 163.00M (35) | 1.08M (38) | | DECORE-300 | 93.50 | 96.66M (62) | 0.61M (65) | | NORTON (Ours) | 94.11 | 92.99M (64) | 0.59M (65) | | DECORE-175 | 92.71 | 58.37M (77) | 0.35M (80) | | NORTON (Ours) | 92.77 | 47.34M (82) | 0.30M (83) |

</div> </details> <details> <summary><strong>3. DenseNet-40/CIFAR-10</strong></summary> <div align="center">

| Model | Top-1(%) | MACs (↓%) | Params. (↓%) | |---------------------|---------|--------------|--------------| | DenseNet-40 | 94.81 | 282.92M (00) | 1.04M (00) | | DECORE-175 | 94.85 | 228.96M (19) | 0.83M (21) | | NORTON (Ours) | 94.86 | 213.58M (26) | 0.74M (30) | | HRank-1 | 94.24 | 167.41M (41) | 0.66M (37) | | DECORE-115 | 94.59 | 171.36M (39) | 0.56M (46) | | AutoBot | 94.67 | 167.64M (42) | 0.76M (28) | | NORTON (Ours) | 94.67 | 168.23M (42) | 0.58M (45) | | HRank-2 | 93.68 | 110.15M (61) | 0.48M (54) | | EZCrop | 93.76 | 113.08M (60) | 0.39M (62) | | DECORE-70 | 94.04 | 128.13M (55) | 0.37M (65) | | NORTON (Ours) | 94.14 | 123.14M (58)** | 0.40M (62) |

</div> </details> <details> <summary><strong>4. ResNet-50/Imagenet</strong></summary> <div align="center">

| Model