Studies of sexual dimorphism often focus on the evolution of elaborate male traits, whereas the evolution of elaborate females has been largely ignored. Yet a phylogenetic perspective suggests that changes in either male or female traits may lead to the evolution of sexual dimorphism. Furthermore, changes in the degree of sexual dichromatism can be caused by gains or losses of elaboration. One common form of elaboration found throughout the animal kingdom is the use of highly saturated and contrasting colors. To investigate further the evolution of female elaboration and sexual dichromatism, we took quantitative measurements of color from New World orioles (Icterus spp.) and then used ancestral-state reconstruction to infer evolutionary changes in male and female elaboration. Our findings suggest that male elaboration is ancestral and strongly conserved but that female elaboration has changed repeatedly, especially through the loss of saturation and contrast. Thus, changes in female—rather than male—color appear to lead to the evolution of sexual dichromatism in orioles. These repeated gains of strong sexual dichromatism through the loss of female elaboration were supported using multiple methods of character coding and reconstruction. Our phylogenetic results suggest that studies of sexual dichromatism cannot assume that color dimorphism arises through increased male elaboration. These findings have important implications for future studies investigating the ultimate causes of sexual dichromatism.