We propose a generic framework for an active leader-follower surveillance system. The system can ingest inputs from a variety of multi- modal leader sensors: Radars, Automatic Identification Systems (AIS), or cameras. Rule and learning based pattern recognition techniques are performed on the leader sensors to infer unusual behaviors and events. In order to calibrate the follower PTZ cameras to the leader sensors, we break up the calibration process into two independent steps: mapping leader cameras' image coordinates to latitude/longitude values, and mapping the latitude/longitude values to the follower's pan-tilt-zoom settings. This calibration method provides straightforward scalability to complex camera networks since newly added cameras need to be calibrated only with respect to the site map (latitude/longitude), and the Radar or AIS leader sensors also fit in seamlessly. Upon detection of an unusual event by one of the leader sensors, follower PTZ cameras move to the location where the event was triggered, detect the target of interest in the field of view, and then track the target actively by automatically adjusting the PTZ settings. We also propose a system to learn various PTZ camera characteristics which facilitate stable and accurate control of the camera during active tracking. Finally, we evaluate the performance of each of the components individually, as well as that of the entire system.