When performing statistics on elements of sets that possess a particular geometric structure, it is desirable to respect this structure. For instance in a Lie group, it would be judicious to have a notion of a mean which is stable by the group operations (composition and inversion). Such a property is ensured for Riemannian center of mass in Lie groups endowed with a bi-invariant Riemannian metric, like compact Lie groups (e.g. rotations). However, bi-invariant Riemannian metrics do not exist for most non compact and non-commutative Lie groups. This is the case in particular for rigid-body transformations in any dimension greater than one, which form the most simple Lie group involved in biomedical image registration. In this paper, we propose to replace the Riemannian metric by an affine connection structure on the group. We show that the canonical Cartan connections of a connected Lie group provides group geodesics which are completely consistent with the composition and inversion. With such a non-metric structure, the mean cannot be defined by minimizing the variance as in Riemannian Manifolds. However, the characterization of the mean as an exponential barycenter gives us an implicit definition of the mean using a general barycentric equation. Thanks to the properties of the canonical Cartan connection, this mean is naturally bi-invariant. We show the local existence and uniqueness of the invariant mean when the dispersion of the data is small enough. We also propose an iterative fixed point algorithm and demonstrate that the convergence to the invariant mean is at least linear. In the case of rigid-body transformations, we give a simple criterion for the global existence and uniqueness of the bi-invariant mean, which happens to be the same as for rotations. We also give closed forms for the bi-invariant mean in a number of simple but instructive cases, including 2D rigid transformations. For general linear transformations, we show that the bi-invariant mean is a generalization of the (scalar) geometric mean, since the determinant of the bi-invariant mean is the geometric mean of the determinants of the data. Finally, we extend the theory to higher order moments, in particular with the covariance which can be used to define a local bi-invariant Mahalanobis distance.