W e present a physics-based deformable model framework for the incremental object shape estimation and tracking in image sequences. The model is estimated by an optimization process that relates image-based cost functions to model motion via the Lagrangian dynamics equations. Although previous approaches have investigated various combinations of cues in the context of deformable model shape and motion estimation, they generally assume a fixed, known, model parameterization, along with a single model discretization in terms of points. Our technique for object shape estimation and tracking is based on the incremental fusing of point information and new line information. Assuming that a deformable model has been initialized to fit part of a complex object (e.g., a bicycle) new line features belonging to the object but excluded from the initial model parameterization are identified during tracking. The identification is based on a set of novel model-based geometric consistency checks relating separate, independent tracking processes at the image feature and model levels, respectively. Identified features are reconstructed and integrated into the model parameterization which results in more accurate object shape estimation and subsequently they are used to increase model coverage in the image, thereby increasing tracking robustness. We derive the forward transfer under the action of the Euclidean group and Jacobian matrices for the underlying line feature mapping. New correspondi- ng image alignment and generalized forces are introduced as soft constraints and combined with the forces derived from model contours to incrementally improve its motion estimation. We demonstrate our approach on image sequences with complex object shape and motion.