Skip to Main content Skip to Navigation
Conference papers

PAC-Bayesian Contrastive Unsupervised Representation Learning

Kento Nozawa 1, 2 Pascal Germain 3 Benjamin Guedj 4, 5, 6, 3, 7
3 MODAL - MOdel for Data Analysis and Learning
LPP - Laboratoire Paul Painlevé - UMR 8524, Université de Lille, Sciences et Technologies, Inria Lille - Nord Europe, METRICS - Evaluation des technologies de santé et des pratiques médicales - ULR 2694, Polytech Lille - École polytechnique universitaire de Lille
Abstract : Contrastive unsupervised representation learning (CURL) is the state-of-the-art technique to learn representations (as a set of features) from unlabelled data. While CURL has collected several empirical successes recently, theoretical understanding of its performance was still missing. In a recent work, Arora et al. (2019) provide the first generalisation bounds for CURL, relying on a Rademacher complexity. We extend their framework to the flexible PAC-Bayes setting, allowing to deal with the non-iid setting. We present PAC-Bayesian generalisation bounds for CURL, which are then used to derive a new representation learning algorithm. Numerical experiments on real-life datasets illustrate that our algorithm achieves competitive accuracy, and yields generalisation bounds with non-vacuous values.
Complete list of metadata

Cited literature [25 references]  Display  Hide  Download

https://hal.inria.fr/hal-02401282
Contributor : Benjamin Guedj <>
Submitted on : Monday, December 9, 2019 - 9:11:54 PM
Last modification on : Tuesday, February 2, 2021 - 3:31:17 AM
Long-term archiving on: : Tuesday, March 10, 2020 - 10:17:49 PM

File

1910.04464.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02401282, version 1
  • ARXIV : 1910.04464

Collections

Citation

Kento Nozawa, Pascal Germain, Benjamin Guedj. PAC-Bayesian Contrastive Unsupervised Representation Learning. UAI 2020 - Conference on Uncertainty in Artificial Intelligence, Aug 2020, Toronto, Canada. ⟨hal-02401282⟩

Share

Metrics

Record views

75

Files downloads

437