Skip to Main content Skip to Navigation
Reports

How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets

Abstract : The computational complexity of kernel methods has often been a major barrier for applying them to large-scale learning problems. We argue that this barrier can be effectively overcome. In particular, we develop methods to scale up kernel models to successfully tackle large-scale learning problems that are so far only approachable by deep learning architectures. Based on the seminal work by Rahimi and Recht on approximating kernel functions with features derived from random projections, we advance the state-of-the-art by proposing methods that can efficiently train models with hundreds of millions of parameters, and learn optimal representations from multiple kernels. We conduct extensive empirical studies on problems from image recognition and automatic speech recognition, and show that the performance of our kernel models matches that of well-engineered deep neural nets (DNNs). To the best of our knowledge, this is the first time that a direct comparison between these two methods on large-scale problems is reported. Our kernel methods have several appealing properties: training with convex optimization, cost for training a single model comparable to DNNs, and significantly reduced total cost due to fewer hyperparameters to tune for model selection. Our contrastive study between these two very different but equally competitive models sheds light on fundamental questions such as how to learn good representations.
Complete list of metadata

Cited literature [44 references]  Display  Hide  Download

https://hal.inria.fr/hal-01666934
Contributor : Aurélien Bellet <>
Submitted on : Monday, December 18, 2017 - 8:58:27 PM
Last modification on : Friday, December 18, 2020 - 6:46:05 PM

File

main.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01666934, version 1
  • ARXIV : 1411.4000

Collections

Citation

Zhiyun Lu, Avner May, Kuan Liu, Alireza Bagheri Garakani, Dong Guo, et al.. How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets. [Research Report] University of Southern California. 2014. ⟨hal-01666934⟩

Share

Metrics

Record views

333

Files downloads

878