To better characterize the resistance development and therefore establish effective pest management strategies, this study was undertaken to investigate the inheritance mode and biochemical mechanisms of malathion resistance in the oriental fruit fly, Bactrocera dorsalis (Hendel), which is one of the most notorious pests in the world. After 22 generations of selection with malathion, the malathion-resistant (MR) strain of B. dorsalis developed a 34-fold resistance compared with a laboratory susceptible strain [malathion-susceptible (MS)]. Bioassay results showed that there was no significant difference between the LD₅₀ values of malathion against the progenies from both reciprocal crosses (F₁-SR and F₁-RS). The degree of dominance values (D) was calculated as 0.39 and 0.32 for F₁-RS and F₁-SR, respectively. The logarithm dosage-probit mortality lines of the F₂ generation and progeny from the backcross showed no clear plateaus of mortality across a range of doses. In addition, Chi-square analysis revealed significant differences between the mortality data and the theoretical expectations. The realized heritability (h²) value was 0.16 in the laboratory-selected resistant strain of B. dorsalis. Enzymatic activities identified significant changes of carboxylesterases, cytochrome P450 (general oxidases), and glutathione S-transferases in MR compared with the MS strain of B. dorsalis. Taken together, this study revealed for the first time that malathion resistance in B. dorsalis follows an autosomal, incompletely dominant, and polygenic mode of inheritance and is closely associated with significantly elevated activities of three major detoxification enzymes.