Elucidating the forces responsible for genomic variation is critical for understanding evolution. Under standard conditions, X-linked diversity is expected to be three-quarters the level of autosomal diversity. Empirical data often deviate from this prediction, but the reasons for these departures are unclear. We demonstrate that population size changes can greatly alter relative levels of X-linked and autosomal variation: population size reductions lead to particularly low X-linked diversity, whereas growth elevates X-linked relative to autosomal diversity. Genetic variation from a diverse array of taxa supports an important role for this effect in accounting for population differences in the ratio of X-linked to autosomal diversity. Consideration of this effect may improve the inference of population history and other evolutionary processes.