Forest fragmentation continues to increase throughout the United States and many studies show a trend toward taxonomic homogenization in fragments. Characterizing the species composition and diversity of remaining forest fragments and identifying factors influencing composition and diversity can help to inform management practices and aid in setting conservation priorities. Because influencing factors tend to be site-specific, studies of local and regional fragment community composition are needed. We studied forest community composition and diversity in four forest fragments of the Shenandoah Valley, Virginia. We found significant (p < 0.05) differences in tree species diversity (H') as well as abundance of exotic species across forest fragments. In addition, results of nonmetric multidimensional scaling (NMS) showed that there were significant differences in species composition among fragments. Soil properties and stand age, in addition to presence of exotic species, were the most important factors influencing differences in tree species composition. Older fragments on high quality soils, with low abundances of exotic species and low levels of disturbance, were more diverse and had greater abundances of economically and ecologically valuable overstory species such as Quercus spp., Carya spp., Nyssa sylvatica and Liriodendron tulipifera. Younger fragments on clay/sandy soils with higher disturbance levels had higher abundances of exotic invasive species, such as Ailanthus altissima and Microstegium vimineum, and lower abundances of late-successional native hardwood tree species. Based on seedling and sapling species composition, we predict that these forest fragments are likely to change in species composition with mortality of overstory trees.