Peer review is the standard that journals and granting agencies use to ensure the scientific quality of their publications and funded projects. The peer-review process continues to be criticized, but its actual effectiveness at ensuring quality has yet to be fully investigated. Here we use probability theory to model the peer-review process, focusing on two key components: (1) editors' prescreening of submitted manuscripts and (2) the number of referees polled. The model shows that the review process can include a strong “lottery” component, independent of editor and referee integrity. Focusing on journal publications, we use a Bayesian approach and citation data from biological journals to show that top journals successfully publish suitable papers—that is, papers that a large proportion of the scientific community would deem acceptable—by using a prescreening process that involves an editorial board and three referees; even if that process is followed, about a quarter of published papers still may be unsuitable. The element of chance is greater if journals engage only two referees and do no prescreening (or if only one editor prescreens); about half of the papers published in those journals may be unsuitable. Furthermore, authors whose manuscripts were initially rejected can significantly boost their chances of being published by resubmitting their papers to other journals. We make three key recommendations to ensure the integrity of scientific publications in journals: (1) Use an editor or editorial board to prescreen and remove manuscripts of low suitability; (2) use a three-of-three or four-of-four decision rule when deciding on paper acceptance; and (3) use a stricter decision rule for resubmissions.