Good quality toxicity data underpins robust hazard and risk assessments in aquatic systems and the derivation of water quality guidelines for ecosystems. Hence, an objective scheme to assess the quality of toxicity data forms an important part of this process. The variation of scores from 2 research papers using the Australasian ecotoxicity database (AED) quality assessment scheme was evaluated by 23 ecotoxicologists. The results showed that the quality class that the assessors gave each paper varied by less than 10% when compared with a quality score agreed a priori between the authors of this study. It was determined that the majority of the variation in each assessment was due to ambiguous or poorly written assessment criteria, information that was difficult to find, or information in the paper that was overlooked by the assessor. This led to refinements of the assessment criteria in the AED, which resulted in a 16% improvement (i.e., reduction) in the mean variation of scores for the 2 papers when compared with the a priori scores. The improvement in consensus among different assessors evaluating the same research papers suggests that the data quality assessment scheme proposed in this article provides a more robust scheme for assessing the quality of aquatic toxicity data than methods currently available.