Several studies have investigated whether particular Y chromosome haplogroups are associated with spermatogenic failure in Japanese males; however, they produced differing results. In this study, to investigate the association of Y chromosome haplogroup with spermatogenic failure, we recruited 451 infertile patients and 730 fertile men from a Japanese population and typed their Y chromosome haplogroups. The infertile patients were suffering from varicocele, azoospermia, oligozoospermia, asthenozoospermia, obstructive azoospermia, karyotype abnormalities, microdeletions of the long arm of the Y chromosome, or other conditions that affect fertility. The frequency of haplogroup D2* was significantly higher (odds ratio = 2.28, 95% confidence interval = 1.44–3.61, P = 0.00034 using chi-square test) among the men with azoospermia than among the fertile men. None of the other Y haplogroups displayed associations with particular types of infertility. In conclusion, Y chromosome haplogroup D2* is associated with spermatogenic failure in Japanese males, suggesting that the Y chromosome lineage can have significant effects on spermatogenesis.