This work provides theoretical arguments to compare dictionary learning algorithms for sparse rep- resentations. Three algorithms are considered: Sparsenet, MOD and K-SVD. The main theoretical result is that the fixed points of the Sparsenet and MOD dictionary update stages are the critical points of the residual error energy function (i.e. points with null gradient, not necessarily local minima), whereas the set of K-SVD fixed points is strictly included in the critical point set of the error energy. An example of a point is also provided where Sparsenet and MOD would stop whereas K-SVD can reach a solution with lower residual error. Further experiments show that the result of Sparsenet is a very good starting point for K-SVD. The combination of Sparsenet followed by K-SVD provides a significant improvement in terms of exact recovery rate and approximation quality.