House mice offer a powerful system for dissecting the genetic basis of phenotypes that isolate species in the early stages of speciation. We used a series of reciprocal crosses between wild-derived strains of Mus musculus and M. domesticus to examine F₁ hybrid male sterility, one of the primary phenotypes thought to isolate these species. We report four main results. First, we found significantly smaller testes and fewer sperm in hybrid male progeny of most crosses. Second, in some crosses hybrid male sterility was asymmetric and depended on the species origin of the X chromosome. These observations confirm and extend previous findings, underscoring the central role that the M. musculus X chromosome plays in reproductive isolation. Third, comparisons among reciprocal crosses revealed polymorphism at one or more hybrid incompatibilities within M. musculus. Fourth, the spermatogenic phenotype of this polymorphic interaction appears distinct from previously described hybrid incompatibilities between these species. These data build on previous studies of speciation in house mice and show that the genetic basis of hybrid male sterility is fairly complex, even at this early stage of divergence.