We present a novel method for tracking, in a monocular image sequence, complex objects that can be approximately modeled by a polyhedral shape. The method consists of two stages of global transformation, the first one operating in 2D space and the second one in 3D space. The first stage is able to handle large displacements of the object projection in the image. It utilizes a 2D motion model estimated by a robust statistical method. Then, we refine the localization of the object silhouette by evaluating the 3D parameters related to the object pose by iteratively minimizing a nonlinear cost function. This last step aims at moving the projection of the contours of the object CAD model to the spatial intensity gradients in the image. The proposed tracking method is real-time, reliable, and robust. Real tracking experiments and results embedded in a visual servoing positioning task are reported.