How females establish in populations of cosexuals is central to understanding the evolution of gender dimorphism in angiosperms. Inbreeding avoidance hypotheses propose that females can establish and be maintained if cosexual fitness is reduced because they self-fertilize, and their progeny express inbreeding depression. Here we assess the role of inbreeding avoidance in maintaining sexual system variation in Wurmbea biglandulosa. We estimated costs of self-pollination, mating patterns, and inbreeding depression in gender monomorphic (cosexuals only) and dimorphic (males and females) populations. Costs of selfing, estimated from seed set of experimentally self- and cross-pollinated flowers, were severe in both males and cosexuals (inbreeding depression, δ = 0.86). In a field experiment, intact males that could self produced fewer seeds than both emasculated males and females, whereas seed set of intact and emasculated cosexuals did not differ. Thus, pollinator-mediated selfing reduces fitness of males but not cosexuals under natural conditions. Outcrossing rates of males revealed substantial selfing (t = 0.68), whereas females and cosexuals were outcrossed (0.92 and 0.97). For males, progeny inbreeding coefficients exceeded parental coefficients (0.220 vs. 0.009), whereas for females and cosexuals these coefficients did not differ and approached zero. Differences in coefficients between males and their progeny indicate that selfed progeny express severe inbreeding depression (δ = 0.93). Combined with inbreeding depression for seed set, cumulative δ = 0.99, indicating that most or all selfed zygotes fail to reach reproductive maturity. We propose that present sexual system variation in W. biglandulosa is maintained by high inbreeding depression coupled with differences in selfing rates among monomorphic and dimorphic populations.