Abstract: In this article, we propose an efficient tracking algorithm to follow 3D objects in image sequences. 3D objects are represented by a collection of reference images. The originality of this method is not to use high-level primitives (points of interest) to follow the movement of the object in the image but rather the difference between the vectors of gray-levels of the tracked reference pattern and the current pattern sampled in an area of interest. The tracking problem is reduced then to the estimate of the parameters representing the possible movements of the object in the image by the determination of interaction matrices learned during an off-line training stage, and that for each reference view. The first one relates the variations of intensity of the 2D current pattern to be tracked to its fronto parallel movement (parallel movement to the image plane). The aspect of the pattern representing the tracked object is not modified by this movement. However, the position, the orientation and the size of the pattern can change. The second matrix relates the variations of appearance of the currently tracked pattern to a change of attitude between the object and the camera (modification of the angular values in rolling and pitching). We show that the on-line use of these interaction matrices for the correction of the predicted position of the object in the image and the estimate of the variations of aspect of the tracked pattern allows a real time implementation of this algorithm (a matrix multiplied by a vector). Moreover, we also show how the problem of occlusions can be managed.