Evolutionary theory predicts that interactions between species such as resource competition or reproductive interference will generate selection for character displacement where similar species co-occur. However, the rate and direction of character displacement will depend not only on the strength of selection for trait divergence, but also on the amount of genetic variation for selected traits and the nature of genetic correlations between them. To assess the importance of genetic constraints for the evolution of character displacement, we examined the genetic architecture of a suite of floral traits previously shown to be under selection in the annual plant Ipomoea hederacea when this species co-occurs with Ipomoea purpurea. We found that the six floral traits we measured are all positively genetically correlated. We also demonstrate, using new statistical approaches, that the predicted response to selection for four of these six traits is substantially constrained by their genetic correlation structure. Most notably, the response to selection for reduced separation of the tallest and shortest anthers, which reduces the degree of detrimental heterospecific pollen flow, is substantially constrained. Our results suggest that the rate of evolution of reproductive character displacement in I. hederacea is limited by the genetic architecture of floral traits.