Modeling and understanding the variability of brain structures is a fundamental problem in neurosciences. Improved mathematical representations of structural brain variation are needed to help detect and understand genetic or disease related sources of abnormality, as well as to improve statistical power when integrating functional brain mapping data across subjects. In this paper, we develop a new mathematical model of normal brain variation based on a large set of cortical sulcal landmarks (72 per brain) delineated in each of 98 healthy human subjects scanned with 3D MRI (age: 51.8+/-6.2 years). We propose an original method to compute an average representation of the sulcal curves, which constitutes the mean anatomy. After affine alignment of the individual data across subjects, the second order moment distribution of the sulcal position is modeled as a sparse field of covariance tensors (symmetric, positive definite matrices). To extrapolate this information to the full brain, one has to overcome the limitations of the standard Euclidean matrix calculus. We propose an affine-invariant Riemannian framework to perform computations with tensors. In particular, we generalize radial basis function (RBF) interpolation and harmonic diffusion partial differential equations (PDEs) to tensor fields. As a result, we obtain a dense 3D variability map that agrees well with prior results on smaller subject samples. Moreover, "leave one (sulcus) out" tests show that our model is globally able to recover the missing information on brain variation when there is a consistent neighboring pattern of variability. Finally, we propose an innovative method to analyze the asymmetry of brain variability. As expected, the greatest asymmetries are found in regions that includes the primary language areas. Interestingly, any such asymmetries in anatomical variance, if it remains after anatomical normalization, could explain why there may be greater power to detect group activation in one hemisphere versus the other in fMRI studies.