Different illuminations adversely affect color difference evaluation of textile images in dyed fabrics. To address the problem, we propose a rotation forest (RF)-based ensemble particle swarm optimization and sparse least squares support vector regression (RF-PSO-SLSSVR) for building an accurate illumination correction model. In our algorithm, grey-edge is first used to extract the statistics characteristics of the textile image. Second, as the standard LSSVR cannot yield a sparse solution, we develop sparse LSSVR (SLSSVR) by calculating the maximal independent subset in the extracted feature space. Then, SLSSVR is embedded into RF by substituting for the regression tree which is the base learner in the original RF, and the PSO technique is employed to obtain the optimal regularization parameter gamma and kernel parameter sigma. The final model is obtained by fusing the predictions of the different trees through a weighted average method and RF-PSO-SLSSVR is constructed to learn the textile illumination estimation model. To verify the effectiveness of our algorithm, we carry out the experiments on the real dyed fabric images by comparison to several related methods and the performance is measured by the different criterions, including the chromaticity error, the angle error, and the Wilcoxon signed-rank test. Compared with the traditional SVR and ELM algorithm, the results show that the RF-PSO-SLSSVR method reduces similar to 13.6% and 10.6% over the angle RMSE.