Three dimensional reconstruction of ancient building using the theory of computer vision is one of the main ways to protect, promote and disseminate the value of ancient building. Due to there exist some shortcomings in the current digitization process of ancient building, such as large-scale scene reconstruction algorithm is complex and the accuracy is low, we present a three dimensional reconstruction approach to point cloud of ancient building based on multiple view images. The proposed approach uses an ordinary digital camera to capture multiple images of the ancient building to be reconstructed, takes advantage of affine decomposition principle to calculate the depth information of SIFT feature point on the ancient building with no damage and convenient operation. Through three dimensional reconstructing of the shape of the Tengwang Pavilion lying in Nanchang city, one of the three famous pavilions in the south of Yangtze River in China, to verify the validity of the proposed approach.