Fast DNN training based on auxiliary function technique

Dung Tran 1 Nobutaka Ono 2 Emmanuel Vincent 1
1 PAROLE - Analysis, perception and recognition of speech
Inria Nancy - Grand Est, LORIA - NLPKD - Department of Natural Language Processing & Knowledge Discovery
Abstract : Deep neural networks (DNN) are typically optimized with stochastic gradient descent (SGD) using a fixed learning rate or an adaptive learning rate approach (ADAGRAD). In this paper, we introduce a new learning rule for neural networks that is based on an auxiliary function technique without parameter tuning. Instead of minimizing the objective function, a quadratic auxiliary function is recursively introduced layer by layer which has a closed-form optimum. We prove the monotonic decrease of the new learning rule. Our experiments show that the proposed algorithm converges faster and to a better local minimum than SGD. In addition, we propose a combination of the proposed learning rule and ADAGRAD which further accelerates convergence. Experimental evaluation on the MNIST database shows the benefit of the proposed approach in terms of digit recognition accuracy.
Document type :
Conference papers
Complete list of metadatas
Contributor : Dung Tran <>
Submitted on : Friday, January 30, 2015 - 3:50:48 PM
Last modification on : Tuesday, December 18, 2018 - 4:38:02 PM
Long-term archiving on : Saturday, September 12, 2015 - 7:00:22 AM


Files produced by the author(s)


  • HAL Id : hal-01107809, version 3


Dung Tran, Nobutaka Ono, Emmanuel Vincent. Fast DNN training based on auxiliary function technique. 40th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2015, Apr 2015, Brisbane, Queensland, Australia. ⟨hal-01107809v3⟩



Record views


Files downloads