The negative effects of invasive plant species on native ecosystems, which can be large and long-lasting, are the primary justifications for their research and management. Tremendous effort is focused on quantifying the ecological impacts of invasive plants, though two different methods are primarily used: observational (compare invaded and uninvaded) and removals (compare invaded and invader removal). However, it is unknown whether these methods, which differ in their assumptions and execution, yield similar results, which could affect our ability to draw broad conclusions within and across studies. Therefore, we performed a meta-analysis on 174 studies that described 547 impacts of 72 invasive plants to test the effect of study method, invader cover, and removal period on the direction and magnitude of impact. Overall, by only considering impact magnitude and not direction, both observational and removal methods yielded similar results—invasive plants are changing most aspects of the ecosystem—and the variation among species and study systems was dramatically reduced compared with traditional, directionally focused studies. This is contrary to a similar analysis that did not control for directionality of impacts, which found overall differences in impact depended on methodology. However, even when the effects of study ecosystem, invader life-form, and impact type were accounted for, significant differences occurred between removal and observational studies. Particularly vulnerable systems appear to be those that would be more greatly disturbed by the removal of the target species, such as tree species or invasive plants in riparian areas. Additionally, impact magnitude increased with invader cover and removal time. We confirm that invasive plants impact the systems they invade in a nonuniform manner; however, we suggest some study conditions may be more sensitive to study methodology. Careful consideration should be given as to which methodology is used in the context of the study system.