Extrinsic factors such as object pose and camera parameters are a main source of shape variability and pose an obstacle to efficiently solving shape matching and inference. Most existing methods address the influence of extrinsic factors by decomposing the transformation of the source shape (model) into two parts: one corresponding to the extrinsic factors and the other accounting for intra-class variability and noise, which are solved in a successive or alternating manner. In this chapter, we consider a methodology to circumvent the influence of extrinsic factors by exploiting shape properties that are invariant to them. Based on higher-order graph-based models, we implement such a methodology to address various important vision problems, such as non-rigid 3D surface matching and knowledge-based 3D segmentation, in a one-shot optimization scheme. Experimental results demonstrate the superior performance and potential of this type of approach.