Linear scale space methodology uses Gaussian probes at scale s to observe the differential structure. In observing the differential image structure through the Gaussian derivative probes at scale s we implicitly construct the Taylor series expansion of the smoothed image. The Gaussian facet model, as a generalization of the classic Haralick facet model, constructs a polynomial approximation of the unsmoothed image. The measured differential structure therefore is closer to the ‘real' structure than the differential structure measured using Gaussian derivatives. At the points in an image where the differential structure changes abruptly (because of discontinuities in the imaging conditions, e.g. a material change, or a depth discontinuity) both the Gaussian derivatives and the Gaussian facet model diffuse the information from both sides of the discontinuity (smoothing across the edge). Robust estimators that are classically meant to deal with statistical outliers can also be used to deal with these ‘mixed model distributions'. In this paper we introduce the robust estimators of local image structure. Starting with the Gaussian facet model where we replace the quadratic error norm with a robust (Gaussian) error norm leads to a robust Gaussian facet model. We will show examples of using the robust differential structure estimators for luminance and color images, for zero and higher order differential structure. Furthermore we look at a ‘robustified' structure tensor that forms the basis of robust orientation estimation.