An optimized solar dish collector (OPSDC) system was proposed in our previous work, which can achieve excellent the optical efficiency and flux uniformity under ideal optics. On this basis, the impacts of the non-ideal optical factors on the optical performance of OPSDC system with a cylindrical and conical receiver are studied in detail and compared with the conventional solar dish collector (COSDC) system in this paper. Where the non-ideal optical factors considered are relatively comprehensive, including the mirror slope error, tracking error, installation error of the mirror and receiver, and receiver's absorptivity degeneration. An optical model with the non-ideal optical factors is built in detail by the ray tracing method, and the corresponding ray tracing codes are developed and verified by literatures and optical software OptisWorks 2012. The results show that the OPSDC system not only has a significantly smaller peak local concentration ratio (LCR) and non-uniformity factor than the COSDC system under the same non-ideal optical factor, but also has excellent optical performance. This means that OPSDC system can effectively avoid the heat absorber generating high-temperature hot spots, thus significantly improving its working reliability and service lifetime. In addition, the tracking error, installation error of the receiver and mirror all lead to the increase of the peak LCR and non-uniform factor, while the mirror slope error and absorber's absorptivity degeneration are conducive to reducing the peak LCR and non-uniform factor. This work can provide a reference for error control of COSDC system and OPSDC system in manufacturing, installation and operation.