The coupled phenomena of mixed convection and thermal radiation within two vertical coaxial rotating cylinders have been numerically investigated. The outer cylinder is constant, and the inner one is rotating at a constant angular speed around its axis. These two walls are kept at different temperatures, while the top and bottom walls are thermal insulated. The coupled momentum and energy equations are solved by the improved projection algorithm based on collocation spectral method (CSM). Steady state is considered. The discrete ordinates method (DOM) is utilized to solve the radiative transfer equation (RTE) so that the effects of optical parameters can be considered. Each collocation point in CSM acts as the center of the control volume in DOM. The fluid between two coaxial cylinders has characteristics of absorbing, emitting and isotropic scattering. Comparative results for the streamlines, isotherms and Nusselt number profiles are presented with the variation of different optical parameters for three different levels of Richardson number. Based on the results of this study, it is found that the surface and internal thermal radiations play an important role in the fluid flow and heat transfer, especially for the buoyancy dominated flow. (C) 2015 Elsevier Ltd. All rights reserved.