Fitness costs of defense are often invoked to explain the maintenance of genetic variation in levels of chemical defense compounds in natural plant populations. We investigated fitness costs of iridoid glycosides (IGs), terpenoid compounds that strongly deter generalist insect herbivores, in ribwort plantain (Plantago lanceolata L.) using lines that had been artificially selected for high and low leaf IG concentrations for four generations. Twelve maternal half-sib families from each selection line were grown in four environments, consisting of two nutrient and two competition treatments. We tested whether: (1) in the absence of herbivores and pathogens, plants from lines selected for high IG levels have a lower fitness than plants selected for low IG levels; and (2) costs of chemical defense increase with environmental stress. Vegetative biomass did not differ between selection lines, but plants selected for high IG levels produced fewer inflorescences and had a significantly lower reproductive dry weight than plants selected for low IG levels, indicating a fitness cost of IG production. Line-by-nutrient and line-by-competition interactions were not significant for any of the fitness-related traits. Hence, there was no evidence that fitness costs increased with environmental stress. Two factors may have contributed to the absence of higher costs under environmental stress. First, IGs are carbon-based chemicals. Under nutrient limitation, the relative carbon excess may result in the production of IGs without imposing a further constraint on growth and reproduction. Second, correlated responses to selection on IG levels indicate the existence of a positive genetic association between IG level and cotyledon size. At low nutrient level, a path analysis based on family means revealed that in the presence of competitors, the negative direct effect of a high IG level on aboveground plant dry weight was partly offset by a positive direct effect of the associated larger cotyledon size. This indicates that fitness costs of defense may be modulated by environment-specific fitness effects of genetically associated traits.