Abstract: Aiming at the shortcoming of existing video surveillance system, an intelligent surveillance system which can accomplish automatic detection and classification of moving objects by making use of the computer system is presented. Moving object is detected by means of a higher-order statistics performed on a group of inter-frame differences, followed by a motion detection phase, producing a binary moving object. Morphological operator is used to reduce false alarm, moving object is tracked by concept of detecting block. Detecting of moving object and event of interest is used to automatically annotate raw video data, and generate video indexing database. The user can specify object-based queries on the surveillance video sequence and "jump" to events of interest. The experimental results show that the performance proposed method is efficient and robust in the intelligent surveillance system.