Fast DNN training based on auxiliary function technique

Dung T. Tran 1 Nobutaka Ono 2 Emmanuel Vincent 3
1 PAROLE - Analysis, perception and recognition of speech
Inria Nancy - Grand Est, LORIA - NLPKD - Department of Natural Language Processing & Knowledge Discovery
3 MULTISPEECH - Speech Modeling for Facilitating Oral-Based Communication
Inria Nancy - Grand Est, LORIA - NLPKD - Department of Natural Language Processing & Knowledge Discovery
Abstract : Deep neural networks (DNN) are typically optimized with stochastic gradient descent (SGD) using a fixed learning rate or an adaptive learning rate approach (ADAGRAD). In this paper, we introduce a new learning rule for neural networks that is based on an auxiliary function technique without parameter tuning. Instead of minimizing the objective function, a quadratic auxiliary function is recursively introduced layer by layer which has a closed-form optimum. We prove the monotonic decrease of the new learning rule. Our experiments show that the proposed algorithm converges faster and to a better local minimum than SGD. In addition, we propose a combination of the proposed learning rule and ADAGRAD which further accelerates convergence. Experimental evaluation on the MNIST database shows the benefit of the proposed approach in terms of digit recognition accuracy.
Type de document :
Communication dans un congrès
ICASSP 2015 - 40th IEEE International Conference on Acoustics, Speech and Signal Processing, Apr 2015, Brisbane, Queensland, Australia
Liste complète des métadonnées

Littérature citée [28 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01107809
Contributeur : Dung Tran <>
Soumis le : mercredi 11 février 2015 - 15:26:40
Dernière modification le : jeudi 11 janvier 2018 - 06:27:31
Document(s) archivé(s) le : samedi 12 septembre 2015 - 11:00:25

Fichier

Dung2015ICASSP_v6_final.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01107809, version 4

Collections

Citation

Dung T. Tran, Nobutaka Ono, Emmanuel Vincent. Fast DNN training based on auxiliary function technique. ICASSP 2015 - 40th IEEE International Conference on Acoustics, Speech and Signal Processing, Apr 2015, Brisbane, Queensland, Australia. 〈hal-01107809v4〉

Partager

Métriques

Consultations de la notice

318

Téléchargements de fichiers

303