Fragment size, isolation, and matrix properties have received considerable attention as predictors of species richness, abundance, and composition in habitat patches. However, measurements of habitat attributes or habitat quality are more directly related to the proximate effects of habitat fragmentation and may be more determinant of assemblages than traditional explanatory variables at local scales. We determine how habitat structure in fragments—a measure of habitat quality—compares to fragment size, isolation, and matrix properties as determinants of richness, abundance, and composition of non-volant small mammals in a fragmented landscape of Atlantic Forest. Small mammals were surveyed once in 25 fragments in the Macacu River watershed, Rio de Janeiro, Brazil, from 1999 to 2001 and 2005 to 2009. A total of 83 candidate models were formulated and compared by Akaike Information Criteria. Habitat structure was one of the main determinants of small mammal assemblages in fragments, as important as fragment isolation for species composition and climatic season for species richness. Rodents were more abundant in fragments with increased overstory and understory vegetation density and more fallen logs. The contrary pattern was found for overall species richness and for species of terrestrial habit, which were more abundant in fragments with more open forest: decreased overstory and understory vegetation density and less fallen logs. Habitat quality in fragments may be a more important determinant of assemblages of small mammals and other vertebrates than previously considered in landscape and land use studies.