Despite abundant empirical evidence that inbreeding depression varies with both the environment and the genotypic context, theoretical predictions about such effects are still rare. Using a quantitative genetics model, we predict amounts of inbreeding depression for fitness emerging from Gaussian stabilizing selection on some phenotypic trait, on which, for simplicity, genetic effects are strictly additive. Given the strength of stabilizing selection, inbreeding depression then varies simply with the genetic variance for the trait under selection and the distance between the mean breeding value and the optimal phenotype. This allows us to relate the expected inbreeding depression to the degree of maladaptation of the population to its environment. We confront analytical predictions with simulations, in well-adapted populations at equilibrium, as well as in maladapted populations undergoing either a transient environmental shift, or gene swamping in heterogeneous habitats. We predict minimal inbreeding depression in situations of extreme maladaptation. Our model provides a new basis for interpreting experiments that measure inbreeding depression for the same set of genotypes in different environments, by demonstrating that the history of adaptation, in addition to environmental harshness per se, may account for differences in inbreeding depression.