We describe a method of automatically annotating video sequences, defining a set of corresponding points in every frame. This is an important pre-processing step for many motion analysis systems. Rather than tracking feature points through the sequence, we treat the problem as one of ‘groupwise registration', in which we seek to find the correspondence between every image and an automatically computed model reference, ignoring the ordering of frames. The main contribution of this work is to demonstrate a method of clustering the frames and constructing a shortest path tree over the clusters. This tree defines the order in which frames will be registered with an evolving estimate of the mean. This technique is shown to lead to a more accurate final result than if all frames are registered simultaneously. We describe the method in detail, and demonstrate its application to face sequences used in an experiment to assess the degree of facial motion. The resulting ranking is found to correlate well with that produced by human subjects.