Abstract: Trajectory analysis is the basis of scene understanding, however noise trajectories causing by barriers in true surveillance scene will influence the result veracity. A trajectory similarity measure and clustering method to segment a scene into semantic regions were proposed to eliminate the effect causing by noise. First, the trajectory was encoded, and then both the object position and its instantaneous velocity were computed by improved similarity measure method to represent the distance between two trajectories. Then, the improved hierarchical clustering algorithm which chooses the longest trajectory as each cluster representation was applied to cluster trajectories according to different spatial and velocity distributions. In each cluster, trajectories were spatially close, had similar velocities of motion, and represented one type of activity pattern. This algorithm does-t need complex pre-process or filter, and because of adding velocity direction, the scene division is more reasonable. Finally, through experiment in true scene, the results show that the method can distinguish different clusters reasonably and improve the effectiveness of clustering.