Animal color pattern phenotypes evolve rapidly. What influences their evolution? Because color patterns are used in communication, selection for signal efficacy, relative to the intended receiver's visual system, may explain and predict the direction of evolution. We investigated this in bowerbirds, whose color patterns consist of plumage, bower structure, and ornaments and whose visual displays are presented under predictable visual conditions. We used data on avian vision, environmental conditions, color pattern properties, and an estimate of the bowerbird phylogeny to test hypotheses about evolutionary effects of visual processing. Different components of the color pattern evolve differently. Plumage sexual dimorphism increased and then decreased, while overall (plumage plus bower) visual contrast increased. The use of bowers allows relative crypsis of the bird but increased efficacy of the signal as a whole. Ornaments do not elaborate existing plumage features but instead are innovations (new color schemes) that increase signal efficacy. Isolation between species could be facilitated by plumage but not ornaments, because we observed character displacement only in plumage. Bowerbird color pattern evolution is at least partially predictable from the function of the visual system and from knowledge of different functions of different components of the color patterns. This provides clues to how more constrained visual signaling systems may evolve.