The appearance of non-rigid objects in a video stream is highly variable and therefore makes the identification of similar objects very complex. Furthermore, the indexing process of all detected objects is a very challengin- g problem when all appearances of an object would be stored: The database produced would become so large that searching would be intractable. In this paper we present a framework for object-based indexing which on one side increases the robustness of existing feature detectors used for object recognition and on the other side reduces the size of the database. The temporal variation of features of a tracked object in the video-shot is modeled by a mixture of Gaussians. Given a tracked object, this consists in separating the feature distribution into homogeneous clusters. Each cluster corresponds to a stable view of the tracked object. We put in competitions seven different Gaussian models and the number of Gaussian components varies up to four. The EM algorithm is applied to estimate the parameters of the mixture of Gaussians where the number of its components and the Gaussian model are a priori fixed. The choice of the best structure of the data (model and number of Gaussians) is realized by different criteria: BIC, ICL and NEC. The training of the system is done on a set of different tracked objects and the Gaussian mixture classifier is used to recognize new occurrences of objects. Experiments on a video base of twelve different objects are conducted and eight color features are tested. A comparison in the performance of the proposed system and the temporal feature method is analyzed and reported.