Skip to Main content Skip to Navigation
New interface
Preprints, Working Papers, ...

Learning PAC-Bayes Priors for Probabilistic Neural Networks

María Pérez-Ortiz 1, 2 Omar Rivasplata 1 Benjamin Guedj 3, 4, 2, 1 Matthew Gleeson 1 Jingyu Zhang 1 John Shawe-Taylor 1, 2 Miroslaw Bober 5 Josef Kittler 5 
4 MODAL - MOdel for Data Analysis and Learning
LPP - Laboratoire Paul Painlevé - UMR 8524, Université de Lille, Sciences et Technologies, Inria Lille - Nord Europe, METRICS - Evaluation des technologies de santé et des pratiques médicales - ULR 2694, Polytech Lille - École polytechnique universitaire de Lille
Abstract : Recent works have investigated deep learning models trained by optimising PAC-Bayes bounds, with priors that are learnt on subsets of the data. This combination has been shown to lead not only to accurate classifiers, but also to remarkably tight risk certificates, bearing promise towards self-certified learning (i.e. use all the data to learn a predictor and certify its quality). In this work, we empirically investigate the role of the prior. We experiment on 6 datasets with different strategies and amounts of data to learn data-dependent PAC-Bayes priors, and we compare them in terms of their effect on test performance of the learnt predictors and tightness of their risk certificate. We ask what is the optimal amount of data which should be allocated for building the prior and show that the optimum may be dataset dependent. We demonstrate that using a small percentage of the prior-building data for validation of the prior leads to promising results. We include a comparison of underparameterised and overparameterised models, along with an empirical study of different training objectives and regularisation strategies to learn the prior distribution.
Complete list of metadata

https://hal.inria.fr/hal-03351794
Contributor : Benjamin Guedj Connect in order to contact the contributor
Submitted on : Wednesday, September 22, 2021 - 3:41:48 PM
Last modification on : Tuesday, December 6, 2022 - 12:42:13 PM

File

2109.10304.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03351794, version 1
  • ARXIV : 2109.10304

Collections

Citation

María Pérez-Ortiz, Omar Rivasplata, Benjamin Guedj, Matthew Gleeson, Jingyu Zhang, et al.. Learning PAC-Bayes Priors for Probabilistic Neural Networks. 2021. ⟨hal-03351794⟩

Share

Metrics

Record views

15

Files downloads

20