Diversification of insect herbivores is often associated with coevolution between plant toxins and insect countermeasures, resulting in a specificity that restricts host plant shifts. Gall inducers, however, bypass plant toxins and the factors influencing host plant associations in these specialized herbivores remain unclear. We reconstructed the evolution of host plant associations in Western Palaearctic oak gallwasps (Cynipidae: Cynipini), a species-rich lineage of specialist herbivores on oak (Quercus). (1) Bayesian analyses of sequence data for three genes revealed extreme host plant conservatism, with inferred shifts between major oak lineages (sections Cerris and Quercus) closely matching the minimum required to explain observed diversity. It thus appears that the coevolutionary demands of gall induction constrain host plant shifts, both in cases of mutualism (e.g., fig wasps, yucca moths) and parasitism (oak gallwasps). (2) Shifts between oak sections occurred independently in sexual and asexual generations of the gallwasp lifecycle, implying that these can evolve independently. (3) Western Palaearctic gallwasps associated with sections Cerris and Quercus diverged at least 20 million years ago (mya), prior to the arrival of oaks in the Western Palaearctic from Asia 5–7 mya. This implies an Asian origin for Western Palaearctic gallwasps, with independent westwards range expansion by multiple lineages.