Abstract: According to Conceptual Metaphor Theory, metaphor is a basic mechanism of human cognition. The cognition of abstract concept such as time relies on the mapping from the concrete concept such as space onto abstract concept such as time, i.e. adopting spatiotemporal metaphor. Being one of the universal spatiotemporal metaphors, "Time Passing is Motion", however, has diversified realization in different cultures due to the influence of language and culture:different language speakers show different preference to the sub-metaphors entailed in it or the positions of "past/future" on the linear temporal axes. By observing and comparing the gestures of native Chinese and English speakers, the present research aims at testing:the psychological reality of the metaphor "Time Passing is Motion"; and the relation between the linguistic and cognitive variables. The research results show that the similarities and differences between native Chinese and English speakers' gesture directions were consistent with those suggested by the two respective languages (including writing direction and metaphorical expressions). In addition, the change in the test language led to a change in the English subjects' gesture direction. Those results help to prove the psychological reality of the above mentioned metaphor and the influence of linguistic variable on cognitive variable.