Measurements of geometric primitives are often noisy in real applications and we need to use statistics either to reduce the uncertainty (estimation), to compare measurements, or to test hypotheses. Unfortunately, geometric primitives often belong to manifolds that are not vector spaces. In previous works , we used invariance requirements to develop some basic probability tools on transformation groups and homogeneous manifolds that avoids paradoxes. In this paper, we consider the Riemannian metric as the basic structure for the manifold. Based on this metric, we develop the notions of mean value and covariance matrix of a random element, normal law, Mahalanobis distance and ^2 test. We provide a simple (but highly non trivial) characterization of Karcher means and an original gradient descent algorithm to efficiently compute them. The notion of Normal law we propose is based on the the minimization of the information knowing the mean and covariance of the distribution. The resulting family of pdfs spans the whole range from uniform (on compact manifolds) to the point mass distribution. Moreover, we were able to provide tractable approximations (with their limits) for small variances which show that we can effectively implement and work with these definitions. To come back to more practical cases, we then reconsider the case of connected Lie groups and homogeneous manifolds. In our Riemannian context, we investigate the use of invariance principles to choose the metric: we show that it can provide the stability of our statistical definitions w.r.t. geometric operations (composition, inversion and action of transformations). However, an invariant metric does not always exists for homogeneous manifolds, nor does a left and right invariant metric for non-compact Lie groups. In this case, we cannot guaranty the full consistency of geometric and statistical operations. Thus, future work will have to concentrate on constraints weaker than invariance.