Based on the panel data of 30 provinces in China from 2000 to 2018, the mutual relationships and mechanisms of influence between industrial upgrading, total factor energy efficiency and regional carbon emission were investigated. The results show that the sophistication of industrial structure has a significant inhibitory effect on carbon emissions in all regions. The intensity of inhibition in different regions shows a sequence of “western > central > eastern”. The inhibitory effect of the rationalization of industrial structure on carbon emissions varies greatly among the different regions, with a significant restraining influence in the central and western regions, but much less influence in the eastern region. The inhibition of carbon emissions through the improvement of total factor energy efficiency is significant in all regions, and the inhibition intensity shows the sequence of “western > eastern > central”. Furthermore, the mediating effect test shows that the total factor energy efficiency in different regions has either a partial or complete mediating effect on the influence of industrial upgrading on carbon emission, so it can promote and strengthen the inhibitory effect of industrial upgrading on carbon emissions. Therefore, upgrading the industrial structure and improving the total factor energy efficiency are effective means to promote carbon emission reduction. Reducing carbon emissions by relying solely on industrial upgrading is not ideal, and it needs to be combined with improvements in the total factor energy efficiency to effectively promote carbon emission reduction.