We present a new algorithm for detecting foreground and moving shadows in surveillance videos. For each pixel, we use the Gaussian Mixture Model (GMM) to learn the behavior of cast shadows on background surfaces. The pixelbased model has the advantages over regional or global model for their adaptability to local lighting conditions, particularly for scenes under complex illumination conditions. However, it would take a long time for convergence if motion is rare on that pixel. We hence build a global shadow model that uses global-level information to overcome this drawback. The local shadow models are updated through confidence-rated GMM learning, in which the learning rate depends on the confidence predicted by the global shadow model. For foreground modeling, we use a nonparametric density estimation method to model the complex characteristics of the spatial and color information. Finally, the background, shadow, and foreground models are built into a Markov random field energy function that can be efficiently minimized by the graph cut algorithm. Experimental results on various scene types demonstrate the effectiveness of the proposed method.