Large grazing animals can have profound impacts on plant communities and soil properties; however, these impacts are not always uniform across or within regions. The distribution of features such as forage quality, water, or shade within a pasture can change the behavior of grazers and thus, the impact of their grazing. Where managed livestock grazing has been proposed as a conservation tool to enhance or maintain desirable plant communities, understanding how spatial variation between tree and intertree habitats within a savanna landscape affects the response of vegetation and soil properties to grazing will be critical for designing management plans for different sites. In this study, we used a previously established, long-term livestock grazing experiment in California oak [Quercus L.] savannas to investigate how the removal of grazing affected plant communities and soil characteristics underneath and outside of isolated tree canopies. In the oak understory, plant community composition shifted in response to livestock removal, largely due to a 68–400% increase in the relative cover of native species. Overall plant community composition in open grassland neighboring trees changed little in response to livestock grazing removal, yet we did see a decrease in species richness and diversity surrounding deciduous oaks as the dominance of the exotic annual Bromus diandrus Roth increased. The depth of plant litter increased 1–2 cmin both habitat types when livestock grazing was absent, along with minor changes in soil carbon, nitrogen, and bulk density. These results highlight how different habitat patches within savanna landscape can have varying responses to grazing removal and illustrate how challenging it will be to use grazing as a management tool to enhance the diversity of native species. In the oak understory, native species that are tolerant of herbivory may be absent or unable to coexist with non-native annual grasses. The abundance of understory habitat at a particular site may therefore be an important variable predicting the outcome of livestock grazing.