Several case-control studies have investigated whether Y chromosome haplogroups or deletions are associated with spermatogenic failure. However, the relationships between Y chromosome haplogroups or deletions and semen quality in general population have not been elucidated. In this study, we assessed relationships between Y chromosome haplogroups or deletions and semen parameters in 791 fertile Japanese men and 1221 young men from the general Japanese population. We found that the haplogroup D2 (M55 lineage) was significantly associated with lower semen parameters, especially total motile sperm count (P = 0.00051, beta = −0.097), in men from the general population but not in fertile men. In addition, we found that the gr/gr subdeletion was associated with semen quality and in particular, strongly associated with decreased sperm motility (P = 0.00041, beta = −3.14) and total motile sperm count (P = 0.00031, beta = −0.099) in men from the general population but not in fertile men. The combined analysis of fertile Japanese men and men from the general Japanese population showed that the haplogroup D2 (M55 lineage) and the gr/gr subdeletion were strongly associated with reduced sperm motility (P = 0.00056, beta = −2.71, and P = 7.7 × 10⁻⁵, beta = −3.05, respectively) and that haplogroup O2b1 was strongly associated with elevated sperm motility (P = 0.00089, beta = 2.94). These observations add further support for the view that the gr/gr subdeletion diminishes sperm motility that consequently may result in male infertility.