This paper presents results of a model-based approach to visual tracking and pose estimation for a moving polyhedral tool in position-based visual servoing. This enables the control of a robot in look-and-move mode to achieve six degree of freedom goal con?gurations. Robust solutions of the correspondence problem -- known as 'matching' in the static case and 'tracking' in the dynamic one -- are crucial to the feasibility of such an approach in real-world environments. The object's motion along an arbitrary trajectory in space is tracked using visual pose estimates through consecutive images. Subsequent positions are predicted from robot joint angle measurements. To deal with inaccurate models and to relax calibration requirements, adaptive on-line calibration of the kinematic chain is proposed. The kinematic predictions enable unambiguous feature matching by a pessimistic algorithm. The performance of the suggested algorithms and the robustness of the proposed system are evaluated on real image sequences of a moving gripper. The results fulfill the requirements of visual-servoing, and the computational demands are sufficiently low to allow for real-time implementation.