Skip to Main content Skip to Navigation
Journal articles

Should penalized least squares regression be interpreted as Maximum A Posteriori estimation?

Rémi Gribonval 1
1 METISS - Speech and sound data modeling and processing
IRISA - Institut de Recherche en Informatique et Systèmes Aléatoires, Inria Rennes – Bretagne Atlantique
Abstract : Penalized least squares regression is often used for signal denoising and inverse problems, and is commonly interpreted in a Bayesian framework as a Maximum A Posteriori (MAP) estimator, the penalty function being the negative logarithm of the prior. For example, the widely used quadratic program (with an $\ell^1$ penalty) associated to the LASSO / Basis Pursuit Denoising is very often considered as MAP estimation under a Laplacian prior in the context of additive white Gaussian noise (AWGN) reduction. This paper highlights the fact that, while this is {\em one} possible Bayesian interpretation, there can be other equally acceptable Bayesian interpretations. Therefore, solving a penalized least squares regression problem with penalty $\phi(x)$ need not be interpreted as assuming a prior $C\cdot \exp(-\phi(x))$ and using the MAP estimator. In particular, it is shown that for {\em any} prior $P_X$, the minimum mean square error (MMSE) estimator is the solution of a penalized least square problem with some penalty $\phi(x)$, which can be interpreted as the MAP estimator with the prior $C \cdot \exp(-\phi(x))$. Vice-versa, for {\em certain} penalties $\phi(x)$, the solution of the penalized least squares problem is indeed the MMSE estimator, with a certain prior $P_X$. In general $dP_X(x) \neq C \cdot \exp(-\phi(x))dx$.
Complete list of metadatas

Cited literature [11 references]  Display  Hide  Download
Contributor : Rémi Gribonval <>
Submitted on : Friday, March 11, 2011 - 5:52:30 PM
Last modification on : Friday, November 16, 2018 - 1:21:48 AM
Document(s) archivé(s) le : Sunday, June 12, 2011 - 2:51:48 AM



Rémi Gribonval. Should penalized least squares regression be interpreted as Maximum A Posteriori estimation?. IEEE Transactions on Signal Processing, Institute of Electrical and Electronics Engineers, 2011, 59 (5), pp.2405-2410. ⟨10.1109/TSP.2011.2107908⟩. ⟨inria-00486840v4⟩



Record views


Files downloads