Many species exhibit sexual dimorphism in a variety of characters, and the underlying genetic architecture of dimorphism potentially involves sex-specific differences in the additive-genetic variance–covariance matrix (G) of dimorphic traits. We investigated the quantitative-genetic structure of dimorphic traits in the dioecious plant Silene latifolia by estimating G (including within-sex matrices, G_m, G_f, and the between-sex variance–covariance matrix, B), and the phenotypic variance–covariance matrix (P) for seven traits. Flower number was the most sexually dimorphic trait, and was significantly genetically correlated with all traits within each sex. Negative genetic correlations between flower size and number suggested a genetic trade-off in investment, but positive environmental correlations between the same traits resulted in no physical evidence for a trade-off in the phenotype. Between-sex genetic covariances for homologous traits were always greater than 0 but smaller than 1, showing that some, but not all, of the variation in traits is caused by genes or alleles with sex-limited expression. Using common principal-components analysis (CPCA), a maximum-likelihood (ML) estimation approach, and element-by-element comparison to compare matrices, we found that G_m and G_f differed significantly in eigenstructure because of dissimilarity in covariances involving leaf traits, suggesting the presence of variation in sex-limited genes with pleiotropic effects and/or linkage between sex-limited loci. The sex-specific structure of G is expected to cause differences in the correlated responses to selection within each sex, promoting the further evolution and maintenance of dimorphism.