The appearance of non-rigid objects detected and tracked in video streams is highly variable and therefore makes the identification of similar objects very complex. Furthermore, indexing and searching of them represent a very challenging problem in computer vision. This paper presents a framework for object-based matching that increases the robustness of existing feature detectors used for object recognition. The Gaussian mixture densities are used to model intra-shot variations of observed features of tracked objects. This process is achieved by the EM algorithm which separates feature distributions given by a tracked object into homogeneous clusters. We use seven different variants of Gaussian mixtures and the Bayes information criterion to identify the best structure of the data (model and parameters). Experiments are conducted on a video sequence of fifteen different tracked objects and comparison in the performance of the mixture approach and the two key-frame methods is analyzed and reported.