The brown planthopper, Nilaparvata lugens (Stål), is a destructive insect pest on rice throughout Asia. As a visible genetic marker, red eye mutant colony of brown planthopper is a valuable material. Here, we established the near-isogenic lines, NIL-BB and NIL-rr, through mating red eye females to brown eye brothers for eight successive generations. Biological experiments showed that NIL-BB had big fitness cost; however, NIL-rr had comparable survival and fertility parameters with BB, a normal laboratory brown planthopper strain. Significantly lower number eggs per female and egg hatchability were the key factors resulting in big fitness cost of NIL-BB. The population trend indexes of BB, NIL-rr, and NIL-BB were 52.18, 43.80, and 4.19, respectively. Real-time PCR study suggested that the poorer fertility of NIL-BB was not mediated by the differential expression of genes relating to oogenesis. The stronger fitness of NIL-rr compared with NIL-BB may be caused by the eye mutant gene or its closely linked genes having stronger compensation ability for reproduction. The comparable fitness of NIL-rr with BB indicated that NIL-rr may be used in field research. The NIL-BB strain with significantly declined fecundity and survival ability can be used as study model for the signal pathways relating to fecundity.