Waste heat recovery is significant to improve energy utilization. Expander is one of the most important components in waste heat recovery. Tesla turbine offers an attractive option for the expander if an efficient design can be well achieved. Faced the theoretical and computational challenges associated with the feasibility of Tesla turbine in a small-scale waste heat recovery system, this paper formulates a systematic design methodology to seek optimal parameters and geometric model of the Tesla turbine which is applied to a coolant waste heat recovery system of an automobile engine. To achieve this goal, a detailed investigation into the performance of Tesla turbine, which incorporates experimental and computational fluid dynamics analysis manifests that Tesla turbine achieves higher performance at lower rotation-speed. Performance analysis on the waste heat recovery system under various operating conditions is conducted based on a comprehensive thermodynamic model. Results show that the total power and overall thermal efficiency of the waste heat recovery system are clearly increased over a low rotation-speed range, furtherly, which can be improved by selecting appropriate viscosity of the working fluid and number of nozzles. It is desirable and significative to effectively improve power output and thermal efficiency for an automobile engine at a lower expense of volume and cost.