The focus of this study is to improve the efficiency of the thermal power cycle from the perspective of photo energy and thermal energy grading conversion. A new concept of combined radiation and heat engine model was proposed to establish a novel flame energy grading conversion system based on photovoltaic conversion and the Rankine cycle. From the perspective of energy utilization, the spectral radiative energy characteristics in oxy-coal combustion were experimentally investigated, and a preliminary thermodynamic model of the new system was established based on the experimental results. Finally, energy and exergy analyses were conducted to determine the general characteristics of the proposed system and assess the improvements to the Rankine cycle. The results indicated that temperature was the main factor affecting the short-waveband energy ratio. Compared with the Rankine cycle, the new flame energy grading conversion system improved the system efficiency by about 3.5%. The exergy analysis indicated that the proposed system reduced the exergy loss factor in the heat transfer of the boiler by about 3% to 6%. This study provides references for improving the thermal power cycle efficiency.