This paper presents an image-based visual servoing for controlling the position and orientation of an unmanned aerial vehicle (UAV) using a fixed downward camera observing landmarks on the level ground. In the proposed method, the visual servoing of the image moments is used to control the vertical motion and rotation around the roll axis. In contrast, an undesired positive feedback arises in visual errors because of the under-actuation of the UAV and this positive feedback makes it difficult to apply the visual servoing to the horizontal motion. Thus, a novel control method using the virtual spring is introduced to control the horizontal motion. The stability of the system is proved based on Lyapunov's direct method. Simulations are presented to validate the proposed method.