Behavior recognition and prediction in public and private areas are still major concerns in video surveillance. Automatic surveillance systems are widely used to extract and analyze the complicated behaviors through logical and mathematical rules. We propose in our work three applications: crowd event detection, human activity recognition and motion pattern extraction. To achieve this goal, we followed an approach based on three levels of analysis. The first level is the detection of low-level features which are retrieved from the pixels of each video frame. The second level is the intermediate level descriptor which is extracted from low-level features and have more semantics. The third level uses the features of the intermediate level in order to produce human readable and useful information. For example, it will be responsible for displaying which event occurred and when. The advantage of this approach is that we can reuse the information of certain levels for different applications. This allowed us to use the optical flow as a low level feature and the direction model as an intermediate level descriptor in all our applications. We have tested our approach using different datasets and different types of video such as day life actions videos and traffic videos.