Farmland resources in mountainous areas are important for regional food security and ecological security. Studies concerning changes in farmland use in mountainous areas are of considerable significance in China. Here, we analyzed marginalization characteristics of farmland in Renhuai city from 2005 to 2011 and driving factors using land information systems, surveys of farmer households and statistical data. Our results indicate that from 2005 to 2011, 3095.76 hm² of farmland was converted to forest land and natural reserve, accounting for 5.45% of the total farmland area. This suggests significant marginalization of farmland. Marginalization of farmland in mountainous areas was affected by topography, labor forces and effectiveness of land management. Farmland with a greater slope gradient was more likely to be abandoned; among marginalized land in Renhuai, a slope greater than 15° accounted for 62.26%. A high non-agricultural employment rate of rural labor force and annual income per capita of farmer households in mountainous areas were consistent with high speed farmland marginalization. Low land management benefits were the key reason for farmland marginalization. Although farmland marginalization was advantageous for eco-environment protection and sustainable development in mountainous areas, it resulted in inefficient land resource utilization. A win-win model for the exploitation and utilization of sloping farmland should be explored for production development and environmental protection.