“Dollo's law” states that, following loss, a complex trait cannot reevolve in an identical manner. Although the law has previously fallen into disrepute, it has only recently been challenged with statistical phylogenetic methods. We employ simulation studies of an irreversible binary character to show that rejections of Dollo's law based on likelihood-ratio tests of transition rate constraints or on reconstructions of ancestral states are frequently incorrect. We identify two major causes of errors: incorrect assignment of root state frequencies, and neglect of the effect of the character state on rates of speciation and extinction. Our findings do not necessarily overturn the conclusions of phylogenetic studies claiming reversals, but we demonstrate devastating flaws in the methods that are the foundation of all such studies. Furthermore, we show that false rejections of Dollo's law can be reduced by the use of appropriate existing models and model selection procedures. More powerful tests of irreversibility require data beyond phylogenies and character states of extant taxa, and we highlight empirical work that incorporates additional information.