Although most parasites show at least some degree of host specificity, factors governing the evolution of specificity remain poorly understood. Many different groups of host-specific parasites show a striking correlation between parasite and host body size, suggesting that size reinforces specificity. We tested this hypothesis by measuring the relative fitness of host-specific feather lice transferred to pigeons and doves that differ in size by an order of magnitude. To test the general influence of size, we transferred unrelated groups of wing and body lice, which are specialized for different regions of the host. Lice were transferred in both directions, from a large native host species, the rock pigeon (Columba livia), to several progressively smaller hosts, and from a small native host species, the common ground dove (Columbina passerina), to several larger hosts. We measured the relative fitness (population size) of lice transferred to these novel host species after two louse generations. Neither wing lice nor body lice could survive on novel host species that were smaller in size than the native host. However, when host defense (preening behavior) was blocked, both groups survived and reproduced on all novel hosts tested. Thus, host defense interacted with host size to govern the ability of lice to establish on small hosts. Neither wing lice nor body lice could survive on larger hosts, even when preening was blocked. In summary, host size influenced the fitness of both types of feather lice, but through different mechanisms, depending on the direction of the transfer. Our results indicate that host switching is most likely between hosts of similar body size. This finding has important implications for studies of host-parasite coevolution at both the micro- and macroevolutionary scales.