Abstract: A visual tracking algorithm was introduced, which was completely based on image information. Several existing algorithms were integrated and improvements were made to achieve the goal of tracking one single moving object when the camera-s movements were small. A target was manually selected. Then feature points were extracted from the first and second frames and feature matching was conducted between the two frames. Using the matched features, affine motion model was established to estimate background movement and target position. With the assumption that target occupies only a few pixels, the separation of optical flow was calculated in a fixed small area around estimated target point and finally the moving object was obtained. The algorithm was applied on two consecutive images of the size of 640×480, and performed well. The experiments on PC show that with proper parameters the algorithm is able to work in the sampling frequency of 12.5Hz or 25Hz.