Mildly deleterious mutations are thought to play a major role in the extinction of natural populations, especially those that are small, isolated, or inbred. Self-fertilization should reduce the effective size of populations and simultaneously reduce migration between populations. A history of self-fertilization should therefore cause a population to harbor a substantial “local drift load” caused by the fixation of mildly deleterious mutations. This hypothesis was tested in Leavenworthia alabamica, which contains large, self-incompatible populations and smaller self-compatible populations with adaptations for self-fertilization. The fitness of offspring from within- and between-population crosses was compared to quantify heterosis caused by the masking of deleterious alleles in the heterozygous state. Little heterosis was observed in crosses between five large, self-incompatible populations and two of the three small, self-fertilizing populations of L. alabamica. However, the most geographically isolated and genetically divergent self-fertilizing population (Tuscumbia) exhibited a 110.2% increase in germination and a 73.6% increase in fitness, which is consistent with a sizeable local drift load. The finding of substantial heterosis for fitness supports the idea that small effective size, reproductive isolation, and self-fertilization can make populations particularly vulnerable to mutation accumulation.