The point ambiguity owing to the ambiguous local appearances of image points is one of the main causes making the stereo problem difficult. Under the point ambiguity, local similarity measures are easy to be ambiguous and this results in false matches in ambiguous areas. In this paper, we present a new similarity measure to resolve the point ambiguity problem based on the idea that the distinctiveness, not the interest, is an appropriate criterion for feature selection under the point ambiguity. Here, the interest of a point represents how much information a point has for facilitating matching, while the distinctiveness of a point represents how much a point is distinguishable from other points. The proposed similarity measure named the Distinctive Similarity Measure (DSM) is essentially based on the distinctiveness of image points and the dissimilarity between them, which are both closely related to the local appearances of image points; the distinctiveness of an image point is related to the probability of a mismatch while the dissimilarity is related to the probability of a good match. We verify the efficiency of the proposed DSM by using testbed image sets. Experimental results prove that the proposed DSM is very effective for both semi-dense and dense stereo matching and considering the point distinctiveness in both images can improve the performance of stereo methods under the point ambiguity.