A large proportion of the rural labor force in China will continue to transfer to non-agricultural sectors in the near future, which will inevitably lead to the transformation of the agricultural production mode and the structure of the farmers' livelihood. The Chinese government is making great efforts to govern agricultural nonpoint source pollution (ANSP), and farmers' environmental behavior is a key factor that must be considered in the formulation of agricultural environmental policies. Based on a set of micro survey data on farmers in the study area and econometric methods, this study investigates the impact of agricultural labor transfer on ANSP by considering the substitution effect of agricultural factors and the effect of agricultural economies of scale. The results show that the increase of the agricultural labor force will not be conducive to reducing ANSP, while the income increase brought by agricultural labor transfer will improve the input structure of agricultural factors and have a positive impact on ANSP reduction. Government departments should provide subsidies or incentive measures to help agricultural social service organizations to expand their coverage and increase the frequency of socialized agricultural services, in order to guide farmers in the use of environment-friendly agricultural technology to reduce the ANSP caused by agricultural factors at the source. Furthermore, it is necessary to facilitate the development of small-sized agricultural machinery suitable for small-area land cultivation.