Character displacement has long been considered a major cause of adaptive diversification. When species compete for resources or mates, character displacement minimizes competition by promoting divergence in phenotypes associated with resource use (ecological character displacement) or mate attraction (reproductive character displacement). In this study, we investigated whether character displacement can also have pleiotropic effects that lead to fitness trade-offs between the benefits of avoiding competition and costs accrued in other fitness components. We show that both reproductive and ecological character displacement have caused spadefoot toads to evolve smaller body size in the presence of a heterospecific competitor. Although this shift in size likely arose as a by-product of character displacement acting to promote divergence between species in mating behavior and larval development, it concomitantly reduces offspring survival, female fecundity, and sexual selection on males. Thus, character displacement may represent the “best of a bad situation” in that it lessens competition, but at a cost. Individuals in sympatry with the displaced phenotype will have higher fitness than those without the displaced trait because they experience reduced competition, but they may have reduced fitness relative to individuals in allopatry. Such a fitness trade-off can limit the conditions under which character displacement evolves and may even increase the risk of “Darwinian extinction” in sympatric populations. Consequently, character displacement may not always promote diversification in the manner that is often expected.