Introduction

This Repo contains the official implementation of the following paper:

|Venue|Method|Paper Title| |----|-----|-----| |ICLR'23|FedDecorr|Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning|

and unofficial implementation of the following papers:

|Venue|Method|Paper Title| |----|-----|-----| |AISTATS'17|FedAvg|Communication-Efficient Learning of Deep Networks from Decentralized Data| |ArXiv'19|FedAvgM|Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification| |MLSys'20|FedProx|Federated Optimization in Heterogeneous Networks| |NeurIPS'20|FedNova|Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization| |CVPR'21|MOON|Model-Contrastive Federated Learning| |ICLR'21|FedAdagrad/Yogi/Adam|Adaptive Federated Optimization| |KDD'21|FedRS|FedRS: Federated Learning with Restricted Softmax for Label Distribution Non-IID Data| |ICML'22|FedLogitCal|Federated Learning with Label Distribution Skew via Logits Calibration| |ICML'22/ECCV'22|FedSAM|Generalized Federated Learning via Sharpness Aware Minimization/Improving Generalization in Federated Learning by Seeking Flat Minima| |ICLR'23|FedExp|FedExP: Speeding up Federated Averaging via Extrapolation|

Dataset preprocessing

TinyImageNet:

1. Download the dataset to "data" directory from this link: http://cs231n.stanford.edu/tiny-imagenet-200.zip
2. Unzip the downloaded file under "data" directory.
3. Lastly, to reformat the validation set, under the folder "data/tiny-imagenet-200", run:

python3 preprocess_tiny_imagenet.py

Running Instructions

Shell scripts to reproduce experimental results in our paper are under "run_scripts" folder. Simply changing the "ALPHA" variable to run under different degree of heterogeneity.

Here are commands that replicate our results:

FedAvg on CIFAR10:

bash run_scripts/cifar10_fedavg.sh

FedAvg + FedDecorr on CIFAR10:

bash run_scripts/cifar10_fedavg_feddecorr.sh

Experiments on other methods (FedAvgM, FedProx, MOON) and other datasets (CIFAR100, TinyImageNet) follow the similar manner.

Citation

If you find our repo/paper helpful, please consider citing our work :)

@article{shi2022towards,
  title={Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning},
  author={Shi, Yujun and Liang, Jian and Zhang, Wenqing and Tan, Vincent YF and Bai, Song},
  journal={arXiv preprint arXiv:2210.00226},
  year={2022}
}

Contact

Yujun Shi (shi.yujun@u.nus.edu)

Acknowledgement

Some of our code is borrowed following projects: MOON, NIID-Bench, SAM(Pytorch)