The human immune system is under great pathogen-mediated selective pressure. Divergent infectious disease pathogenesis across human populations combined with the overrepresentation of “immune genes” in genomic regions with signatures of positive selection suggests that pathogens have significantly altered the human genome. However, important features of the human immune system can confound searches for and interpretations of signatures of pathogen-mediated evolution. Immune system redundancy, immune gene pleiotropy, host ability to acquire immunity and alter the immune repertoire of offspring through “priming,” and host microbiome complicate evolutionary interpretations of host-pathogen interactions. The overall promiscuity and sensitivity of the immune system to local environments can also muddy assumptions about the origins of a selective pressure on a given set of genes. This review addresses (a) how features of the immune system, the primary buffer between a pathogen and the human genome, affect evolutionary signal and (b) the considerations that must be made when assessing how pathogens have contributed to human diversification.