Online determination of track loss in the absence of ground truth is an important and challenging problem in visual tracking systems. In this paper, we construct a novel track loss determination strategy using Template Inverse Matching (TIM). The idea of TIM is to inverse the common process of template tracking match. Ground truth is not required to be known. It is proved to be highly efficient and accurate, and is adaptive to all the visual tracking frameworks. The proposed strategy is justified in the theoretical framework of Stable Marriage (SM) problem. In this paper, we prove that the combination between the target position got from the TIM algorithm and the original one can meet the constraints of SM pairs and is irreplaceable when a correct tracking process is performed. This guarantees the stability of object tracking. Compariative experiments show high accuracy of the present method in determining track loss. The performance of tracking systems can be improved based on online track loss determination.