Uncertainty Quantification in Deep Learning

This repo contains literature survey and implementation of baselines for predictive uncertainty estimation in deep learning.

Literature survey

Basic background for uncertainty estimation

• B. Efron and R. Tibshirani. "Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy." Statistical science, 1986. [Link]

• R. Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani. "Predictive inference with the jackknife+." arXiv, 2019. [Link]

• B. Efron. "Jackknife‐after‐bootstrap standard errors and influence functions." Journal of the Royal Statistical Society: Series B (Methodological), 1992. [Link]

• J. Robins and A. Van Der Vaart. "Adaptive nonparametric confidence sets." The Annals of Statistics, 2006. [Link]

• V. Vovk, et al., "Cross-conformal predictive distributions." JMLR, 2018. [Link]

• M. H Quenouille., "Approximate tests of correlation in time-series." Journal of the Royal Statistical Society, 1949. [Link]

• M. H Quenouille. "Notes on bias in estimation." Biometrika, 1956. [Link]

• J. Tukey. "Bias and confidence in not quite large samples." Ann. Math. Statist, 1958.

• R. G. Miller. "The jackknife–a review." Biometrika, 1974. [Link]

• B. Efron. "Bootstrap methods: Another look at the jackknife." Ann. Statist., 1979. [Link]

• R. A Stine. "Bootstrap prediction intervals for regression." Journal of the American Statistical Association, 1985. [Link]

• R. F. Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani. "Conformal prediction under covariate shift." arXiv preprint arXiv:1904.06019, 2019. [Link]

• R. F. Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani. "The limits of distribution-free conditional predictive inference." arXiv preprint arXiv:1903.04684, 2019b. [Link]

• J. Lei, M. G'Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman. "Distribution-free predictive inference for regression." Journal of the American Statistical Association, 2018. [Link]

• R. Giordano, M. I. Jordan, and T. Broderick. "A Higher-Order Swiss Army Infinitesimal Jackknife." arXiv, 2019. [Link]

• P. W. Koh, K. Ang, H. H. K. Teo, and P. Liang. "On the Accuracy of Influence Functions for Measuring Group Effects." arXiv, 2019. [Link]

• D. H. Wolpert. "Stacked generalization." Neural networks, 1992. [Link]

• R. D. Cook, and S. Weisberg. "Residuals and influence in regression." New York: Chapman and Hall, 1982. [Link]

• R. Giordano, W. Stephenson, R. Liu, M. I. Jordan, and T. Broderick. "A Swiss Army Infinitesimal Jackknife." arXiv preprint arXiv:1806.00550, 2018. [Link]

• P. W. Koh, and P. Liang. "Understanding black-box predictions via influence functions." ICML, 2017. [Link]

• S. Wager and S. Athey. "Estimation and inference of heterogeneous treatment effects using random forests." Journal of the American Statistical Association, 2018. [Link]

• J. F. Lawless, and M. Fredette. "Frequentist prediction intervals and predictive distributions." Biometrika, 2005. [Link]

• F. R. Hampel, E. M. Ronchetti, P. J. Rousseeuw, and W. A. Stahel. "Robust statistics: the approach based on influence functions." John Wiley and Sons, 2011. [Link]

• P. J. Huber and E. M. Ronchetti. "Robust Statistics." John Wiley and Sons, 1981.

• Y. Romano, R. F. Barber, C. Sabatti, E. J. Candès. "With Malice Towards None: Assessing Uncertainty via Equalized Coverage." arXiv, 2019. [Link]

• H. R. Kunsch. "The Jackknife and the Bootstrap for General Stationary Observations." The annals of Statistics, 1989. [Link]

Predictive uncertainty for general machine learning models

• A. Malinin, L. Prokhorenkova, A. Ustimenko. "Uncertainty in Gradient Boosting via Ensembles." ICLR, 2021. [Link]

• S. Feldman, S. Bates, Y. Romano. "Improving Conditional Coverage via Orthogonal Quantile Regression." arXiv preprint, 2021. [Link]

• S. Bates, A. Angelopoulos , L. Lei, J. Malik, and M. I. Jordan. "Distribution-Free, Risk-Controlling Prediction Sets." arXiv preprint, 2021. [Link]

• S. Wager, T. Hastie, and B. Efron. "Confidence intervals for random forests: The jackknife and the infinitesimal jackknife." The Journal of Machine Learning Research, 2014. [Link]

• L. Mentch and G. Hooker. "Quantifying uncertainty in random forests via confidence intervals and hypothesis tests." The Journal of Machine Learning Research, 2016. [Link]

• J. Platt. "Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods." Advances in large margin classifiers, 1999. [Link]

• A. Abadie, S. Athey, G. Imbens. "Sampling-based vs. design-based uncertainty in regression analysis." arXiv preprint (arXiv:1706.01778), 2017. [Link]

• T. Duan, A. Avati, D. Y. Ding, S. Basu, Andrew Y. Ng, and A. Schuler. "NGBoost: Natural Gradient Boosting for Probabilistic Prediction." arXiv preprint, 2019. [Link]

• V. Franc, and D. Prusa. "On Discriminative Learning of Prediction Uncertainty." ICML, 2019. [Link]

• Y. Romano, M. Sesia, and E. J. Candès. "Classification with Valid and Adaptive Coverage." arXiv preprint, 2020. [Link]

Predictive uncertainty for deep learning

• I. Osband, Z. Wen, M. Asghari, M. Ibrahimi, X. Lu, and B. Van Roy "Epistemic Neural Networks." arXiv, 2021. [Link]

• Abdar, Moloud, et al. "A review of uncertainty quantification in deep learning: Techniques, applications and challenges." Information Fusion, 2021. [Link]

• Gawlikowski, Jakob, et al. "A Survey of Uncertainty in Deep Neural Networks." arXiv preprint, 2021. [Link]

• P. Morales-Alvarez, D. Hernández-Lobato, R. Molina, J. M. Hernández-Lobato. "Activation-level uncertainty in deep neural networks." ICLR, 2021. [Link]

• A. Angelopoulos, S. Bates, J. Malik, and M. I. Jordan. "Uncertainty Sets for Image Classifiers using Conformal Prediction." ICLR, 2021. [Link]

• K. Patel, W. H. Beluch, B. Yang, M. Pfeiffer, D. Zhang. "Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning." ICLR 2021. [Link]

• B. Adlam, J. Lee, L. Xiao, J. Pennington, J. Snoek. "Exploring the Uncertainty Properties of Neural Networks’ Implicit Priors in the Infinite-Width Limit." ICLR 2021. [Link]

• A. Harakeh, S. L. Waslander. "Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors." ICLR 2021. [Link]

• J. Antoran, U. Bhatt, T. Adel, A. Weller, J. M. Hernández-Lobato. "Getting a CLUE: A Method for Explaining Uncertainty Estimates." ICLR 2021. [Link]

• A.-K. Kopetzki, B. Charpentier, D. Zügner, S. Giri, S. Günnemann. "Evaluating Robustness of Predictive Uncertainty Estimation: Are Dirichlet-based Models Reliable?" ICML 2021. [Link]

• A. Zhou and S. Levine. "Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation." ICML, 2021. [Link]

• M. Havasi, R. Jenatton, S. Fort, J. Z. Liu, J. Snoek, B. Lakshminarayanan, A. M. Dai, and D. Tran. "Training independent subnetworks for robust prediction." ICLR, 2021. [Link]

• B. Adlam, J. Lee, L. Xiao, J. Pennington, J. Snoek. "Exploring the Uncertainty Properties of Neural Ne