How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets

Abstract : The computational complexity of kernel methods has often been a major barrier for applying them to large-scale learning problems. We argue that this barrier can be effectively overcome. In particular, we develop methods to scale up kernel models to successfully tackle large-scale learning problems that are so far only approachable by deep learning architectures. Based on the seminal work by Rahimi and Recht on approximating kernel functions with features derived from random projections, we advance the state-of-the-art by proposing methods that can efficiently train models with hundreds of millions of parameters, and learn optimal representations from multiple kernels. We conduct extensive empirical studies on problems from image recognition and automatic speech recognition, and show that the performance of our kernel models matches that of well-engineered deep neural nets (DNNs). To the best of our knowledge, this is the first time that a direct comparison between these two methods on large-scale problems is reported. Our kernel methods have several appealing properties: training with convex optimization, cost for training a single model comparable to DNNs, and significantly reduced total cost due to fewer hyperparameters to tune for model selection. Our contrastive study between these two very different but equally competitive models sheds light on fundamental questions such as how to learn good representations.
Type de document :
[Research Report] University of Southern California. 2014
Liste complète des métadonnées

Littérature citée [44 références]  Voir  Masquer  Télécharger
Contributeur : Aurélien Bellet <>
Soumis le : lundi 18 décembre 2017 - 20:58:27
Dernière modification le : mercredi 20 février 2019 - 01:28:51


Fichiers produits par l'(les) auteur(s)


  • HAL Id : hal-01666934, version 1
  • ARXIV : 1411.4000



Zhiyun Lu, Avner May, Kuan Liu, Alireza Bagheri Garakani, Dong Guo, et al.. How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets. [Research Report] University of Southern California. 2014. 〈hal-01666934〉



Consultations de la notice


Téléchargements de fichiers