Our project focuses on implementing and researching various neural network pruning techniques, particularly extending the lottery ticket hypothesis to structured pruning.
An illustration of various structured pruning strategies.
Lottery Ticket Hypothesis
A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.
We aim to reduce model size while maintaining performance, accuracy, and uncertainty, and to decrease training time.
Accuracy vs Pruning Ratio for Different Techniques
Uncertainity
For a better understanding of calibration and the value of ECE, we performed an Out-of-Distribution (OOD) Detection for CIFAR10 trained model on CIFAR100 dataset.
Our analysis demonstrates that neural network pruning reallocates confidence intervals, evidenced by the reduced misclassification of man images in the deer category after intensive pruning, enhancing the model's reliability and robustness.
References
2021
Bayesian Deep Learning via Subnetwork Inference
Proceedings of the 38th International Conference on Machine Learning, 2021
@article{daxberger2021laplace,title={Bayesian Deep Learning via Subnetwork Inference},author={},journal={Proceedings of the 38th International Conference on Machine Learning},year={2021},}
@article{blalock2020state,title={What is the State of Neural Network Pruning?},author={},journal={Proceedings of Machine Learning and Systems},year={2020},}
2019
Well-calibrated Model Uncertainty with Temperature Scaling for Dropout Variational Inference
Proceedings of the 4th Workshop on Bayesian Deep Learning, NeurIPS, 2019
@article{laves2020wellcalibrated,title={Well-calibrated Model Uncertainty with Temperature Scaling for Dropout Variational Inference},author={},journal={Proceedings of the 4th Workshop on Bayesian Deep Learning, NeurIPS},year={2019},}
2018
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Michael Carbin Jonathan Frankle
Proceedings of the International Conference on Learning Representations, 2018
@article{frankle2018the,title={The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks},author={Jonathan Frankle, Michael Carbin},journal={Proceedings of the International Conference on Learning Representations},year={2018},}
