Antlers are a costly trait that require skeletal reserves to grow to a large size. Thus, insight into variation in antler size requires understanding the connection between antler and body size, which can be summarized through ontogenetic and static scaling relationships. Both types of scaling relationships are also needed to examine the influence of factors besides body mass on antler growth at different ages and whether the effects of these factors diminish with age. Dietary and maternal effects might decline with age as individuals attempt to mitigate size deficits. We examined the potential effects of diet and maternal attributes of dam (mother) age at birth and litter size on body–antler mass relationships throughout ontogeny and at discrete ages. Data on age, maternal characteristics, body mass, and antler mass were gathered from captive, pen-raised White-tailed Deer (n = 168) that consumed either a low-energy (1.77 kcal/g) or standard-energy diet (2.65 kcal/g) from the time they were weaned until they perished by 5.5 years of age. Both types of scaling relationships were estimated with linear mixed-effects models to account for repeated measurements of males, dams, and sires. Diet affected ontogenetic scaling relationships. Males eating the low-energy diet had faster antler growth when young and lighter in body mass than when they were older and heavier—whereas males fed the standard-energy diet had a similar rate of antler growth across the range of body masses. A Bayesian Information Criterion model selection analysis indicated that diet and litter size (singleton, multiple births), but not dam age at birth influenced static scaling relationships. Static scalar coefficients up to 3.5 years of age were positively allometric (>1.0), but isometric (1.0) in 4.5- and 5.5-year-old males. Furthermore, diet and litter size influenced both intercepts and slopes in only the youngest males (1.5 years) indicating that dietary and maternal effects can diminish with age. Body–antler mass relationships are complex. Young males that are small because of maternal effects, for example, might still be able to mitigate smaller body and antler sizes at older ages.