Generation of the initial matching hypothesis for a model-based monocular vision system is presented. The primitive shape description of images is a set of line segments and the model is automatically constructed from a sequence of stereo views. The key points of the approach are the use of vanishing points and other perspective invariants such as collinearity, connectivity, and the use of double ratios to get rid of matching ambiguities.