Objective Revealing self-thinning mechanism of mixed forests is the basis for achieving forest sustainable management goal.

Methods Based on the permanent plots of mixed forests of Cunninghamia lanceolata and broad-leaved tree species in Fujian, Jiangxi and Anhui provinces between 1991 and 2016, linear regression, random frontier analysis and quantile regression methods were used to study the effects of tree species mixing, latitude and site conditions on the self-thinning trajectory.

Results Compared with pure Chinese fir stands, the self-thinning line of mixed forests of Cunninghamia lanceolata and broad-leaved tree species shifted upward and tended to be flat. Regardless of the fitted methods of self-thinning line, the trajectory of self-thinning line of mixed forests was basically consistent with the pure Chinese fir stand and did not deviate from -1.605. Mixing proportion had a positive effect on maximum size–density relationships, but latitude and site conditions had a negative effect. The linear regression analysis obtained the average stand density of mixed forests, while the quantile regression indicated the maximum forest density line based on quantile threshold. Stochastic frontier analysis can avoid the subjective selection of self-thinning fitting data and is the best fitting approach for the maximum stand density line of mixed forests.

Conclusion Self-thinning trajectory of mixed forests of Cunninghamia lanceolata and broad-leaved tree species is consistent with pure Chinese fir stands, which was mainly affected by mixing proportion, latitude and site conditions. Stochastic frontier analysis is the best fitting approach for the maximum stand density line of mixed forests of Cunninghamia lanceolata and broad-leaved tree species. It can provide practical guidance for density control of mixed forests of Cunninghamia lanceolata and broad-leaved tree species.