Sexual dimorphism is one of the most widespread and recognizable patterns of phenotypic variation in the biotic world. Sexual dimorphism in floral display is striking in the dioecious plant Silene latifolia, with males making many, small flowers compared to females. We investigated this dimorphism via artificial selection on two populations to determine whether genetic variation exists within populations for flower size and the extent of the between-sex correlation, whether a flower size and number trade-off exists within each sex, and whether pollen and ovule production vary with flower size. We selected for decreased flower size (calyx width) in females and increased flower size in males and measured the response to selection in size and correlated responses in flower dry mass, flower number, and pollen or ovule number per flower. Four bouts of selection in each of two selection programs were performed, for a total of three selection lines to decrease size, three to increase it, and two control lines. Flower size always significantly responded to selection and we always found a significant correlated response in the sex not under selection. Selection decreased but did not eliminate the sexual dimorphism in flower dry mass and number. A negative relationship between flower size and number within each sex was revealed. Whereas ovule number showed a significant correlated response to selection on flower size, pollen number did not. Our results indicate that although substantial additive genetic variation for flower size exists, the high between-sex genetic correlation would likely constrain flower size from becoming more sexually dimorphic. Furthermore, floral display within each sex is constrained by a flower size and number trade-off. Given this trade-off and lack of variation in pollen production with flower size, we suggest that sexual dimorphism evolved via sexual selection to increase flower number in males but not females.