We propose a real time 3D tracking algorithm dedicated to the tracking of human faces in video sequences. A face is represented by a collection of 2D images called reference views. In our approach, a pattern is a region of the image defined in an area of interest and its sampling gives a grey level vector. The tracking technique involves two stages. An off-line learning stage is devoted to the computation of an interaction matrix for every reference view. This matrix relates the grey level difference between the tracked reference pattern and the current pattern sampled inside the area of interest to its 'fronto parallel' movement (which do not modify its aspect in the image). The on-line stage consists in using this matrix to track the reference pattern in the current image. During this stage, appearance changes due to movements in roll are managed by switching between the different reference patterns. The reference pattern, after motion correction, giving the smallest grey level difference is supposed to be the new tracked reference pattern. We present experimental results showing the efficiency and the robustness of our approach.