Abstract: When conventional EKF (extended Kalman filtering) algorithm was used in LEO (low earth orbit satellite) orbit determination by space-borne GPS (global positioning system). It is difficult to know the initial value of system noise variance. Therefore, the novel filter algorithm of orbit determination was proposed. In the new algorithm, the methods of linearization at the prior filtering estimate value was adopted during the linearization process of the nonlinear equations, and the disturbing equations were obtained, and then the disturbing equations were introduced in conventional EKF. Apply the EKF and the modified algorithm in LEO orbit determination based on space-borne GPS respectively, the results show the modified algorithm can restrain filter divergence of conventional EKF caused by the large deflection of system noise variance in a certain extent, and has better robustness to the selecting of initial value of system noise variance.