How do changes in the environment affect the brain? More specifically, how do different physical environments shape areas of the brain that facilitate the contextual perception of animals' surroundings? We evaluated the relationship between rapid habitat change and the volume of the medial, lateral, and dorsal cortices, in two populations of wild male and female Lesser Earless Lizards (Holbrookia maculata). These populations inhabit neighboring, but physically distinct, environments in southern New Mexico: White Sands and dark soil habitats. The White Sands habitat is a geologically recent environment that formed over the last 2,000–7,000 yr. There is a stark physical contrast between White Sands and the surrounding typical Chihuahuan desert habitat that is characterized by dark soil, denser vegetation, and higher competitor and predator species abundances. We observed differences in the volumes of the medial cortex, an important neuroanatomical structure for navigation, but not in those of the dorsal or lateral cortices, between lizards inhabiting the White Sands and dark soil habitats. Male and female lizards in the White Sands habitat had smaller medial cortices (relative to their overall brain size) compared to those in the dark soil environment. We also found sex differences in the volume of the medial cortex; in the dark soil habitat, males had larger medial cortices than females. In summary, we have uncovered differences in neuroanatomy between habitats and between sexes in a wild species of lizard, which may reflect neuroanatomical plasticity and/or rapid evolutionary change in response to a geologically recent environmental change.