Abstract: This paper presents a new method for constructing models from a set of positive and negative sample images ; the method requires no manual extraction of significant objects or features. Our model representation is based on two layers. The first one consists of “generic” descriptors which represent sets of similar rotational invariant feature vectors. Rotation invariance allows to group similar, but rotated patterns and makes the method robust to model deformations. The second layer is the joint probability on the frequencies of the “generic” descriptors over neighborhoods. This probability is multi-modal and is represented by a set of “spatial-frequency” clusters. It adds a statistical spatial constraint which is rotationally invariant. Our twolayer representation is novel ; it allows to efficiently capture “texture-like” visual structure. The selection of distinctive structure determines characteristic model features (common to the positive and rare in the negative examples) and increases the performance of the model. Models are retrieved and localized using a probabilistic score. Experimental results for “textured” animals and faces show a very good performance for retrieval as well as localization.