Changes in grain production are decomposed and compared among nine major Chinese river basins for the sake of optimal water allocation. The results show that water-deficient northern China, especially the Songliao River Basin and Huai River Basin, contributed the greatest share of the total grain increment from 1995 to 2010. The Songliao River Basin achieved increased grain output largely by expanding multiple cropping, while the Huai River Basin achieved it mainly by improving the yield per unit area. With increased reliance on expanding irrigation and multi-cropping, most northern basins have high levels of agricultural water consumption, despite the rising share of corn, a lower water intensive crop. In contrast, over the same period the warm and humid south, traditionally a major rice-growing area, mostly experienced a sharp decline in rice cropping area and the Southeast Rivers Basin even reduced multiple cropping indexes, contributing to decreased agricultural water consumption. Implications of our findings and the need for tackling the imbalance of agricultural water use in grain production are discussed.