Researchers interested in describing or understanding agroecological systems have many reasons to consider on-farm research. Yet, despite the inherent realism and pedagogical value of on-farm studies, recruiting cooperators can be difficult and this difficulty can result in so-called “convenience samples” containing a potentially large and unknown bias. There is often no formal justification for claiming that on-farm research results can be extrapolated to farms beyond those participating in the study. In some sufficiently well-understood research areas, models may be able to correct for potential bias; however, no theoretical argument is as persuasive as a direct comparison between a randomized and a convenience sample. In a 30-cooperator on-farm study investigating weed community dynamics across the state of Wisconsin, we distributed a written survey probing farmer weed management behaviors and attitudes. The survey contained 59 questions that overlapped a large, randomized survey of farmer corn pest management behavior. We compared 187 respondents from the larger survey with the 18 respondents from our on-farm study. For dichotomous response questions, we found no difference in response rate for 80% of the questions (α  =  0.2, β > 0.5). Differences between the two groups were logically connected to the selection criteria used to recruit cooperators in the on-farm study. Similarly, comparisons of nondichotomous response questions did not differ for 80% of the questions (α  =  0.05, β > 0.9). Exploratory multivariate analyses failed to reveal differences that might have been hidden from the marginal analyses. We argue that our findings support the notion that the convenience samples often associated with on-farm research may be representative of the more general class of farms, despite lack of bias protection provided by truly randomized designs.