Abstract: Based on Chinese input-output table and data of labor input and capital input in various industries from 1981 to 2015, this paper calculated the contribution of labor input, capital input, intermediate input and total factor productivity to China's agricultural sector economic growth. Moreover, it examined the growth potential of each factor's input in the agricultural sector depending on their contribution to quality improvement. The results show that the ratio of the contribution of total factor productivity and factor input to the economic growth of China's agricultural sector has changed from "three to seven" to "seven to three" and total factor productivity has become the main factor of sustainable agricultural growth in China. The serious loss of labor force in agricultural sector resulted in the reduction in labor input, which led to the inhibiting effect of labor input on China's agricultural growth from 2001 to 2015. From 1982 to 2000, the overall contribution of factor input to quality improvement was in the shape of inverted "U", and tended to be stable from 2001 to 2015, and from 2006 to 2015, the contribution was almost entirely derived from intermediate inputs. Therefore, to achieve sustainable and high-quality growth of China's agricultural economy, it is necessary to further improve the quality of labor input and capital input, as well as total factor productivity, attach importance to the accumulation of human capital in the agricultural sector, and promote the centralization and industrialization of agricultural production and operation.