Declining populations of grassland breeding birds have led to increased efforts to assess habitat quality, typically by estimating density or relative abundance. Because some grassland habitats may function as ecological traps, a more appropriate metric for determining quality is breeding success, which is challenging to determine for many cryptic-nesting grassland birds. This difficulty led Vickery et al. (1992) to propose a reproductive index based on behavioral observations rather than nest fate. We rigorously evaluated the index for 2 years using a Savannah Sparrow (Passerculus sandwichensis) population in western New York and found a weak correlation in classification of the breeding stages of monitored territories among multiple observers (r = 0.398). We also discovered a large difference between overall territory and nest success rates independently estimated with the index (9.8% over the entire breeding cycle) and with nest searching and monitoring (41.7% of nests successfully fledged young). Most importantly, we made territory-level comparisons of index estimates with actual nest fate and found that the index correctly predicted fates for only 43% of the monitored nests. A Mayfield logistic regression analysis demonstrated that only index rank 4 (eggs hatched, but young failed to fledge) showed a strong positive correlation with nest success. Although the reproductive index may function as a coarse indicator of habitat suitability (e.g., documenting production in potential ecological traps), in our study the index exhibited neither internal consistency nor the ability to predict nest fate at the plot or territory level and functioned poorly as a substitute for nest searching and monitoring.