The CO2 transcritical Rankine cycle has considerable potential for the waste heat recovery of engine, but its high operation pressure, harsh condensation condition and low thermodynamic performance are huge obstacle restricting the wide application of it. The purpose of this paper is to investigate the performance improvements of transcritical Rankine cycle using CO2 mixtures for the waste heat recovery of engine. A large number of refrigerants are discussed and then eight of them are selected as the candidate additives into CO2. The Peng-Robinson equations of states are applied to calculate the properties of CO2 mixtures in this study. Afterwards, thermodynamic, economic and miniaturization analyses are implemented. The results show that CO2 mixtures can enlarge the condensation temperature range of the transcritical Rankine cycle and the high operation pressure will also be improved. Moreover, based on transcritical Rankine cycle with optimal power output, CO2/R32 (0.3/0.7) performed best when the condensation temperature is lower than 40 degrees C, else the CO2/R161 (0.45/0.55) is a suitable selection among the CO2 mixtures. The optimal operation pressure will reduce by 36% and 35% respectively for the transcritical Rankine cycles with CO(2/)R161 (0.45/0.55) and CO2/R32 (0.3/0.7) compared with the CO2 transcritical Rankine cycle. Based on transcritical Rankine cycle with optimal total heat transfer area, the CO2/R32 (0.7/0.3) is the best selection. The optimal operation pressure of transcritical Rankine cycle with CO2/R32 (0.7/0.3) decreases by 1.4 MPa and net output power increases by 8.8% compared with CO(2)transcritical Rankine cycle. The optimal total heat transfer area increment of CO2/R32 (0.7/0.3) will decrease by 29.4% compared with that of CO2/R32 (0.3/0.7).