Steel slag, an industrial by-product of steel industry, has great potential in asphalt mixture as an alternative aggregate. The use of steel slag can reduce environmental pollution effectively. Meanwhile, rubber asphalt also has good eco-friendly performance. To study the adhesion between steel slag and rubber asphalt, pull-out test and net adsorption test were utilized to evaluate the adhesion between different aggregates and rubber asphalt. The morphology of different aggregates and the microstructure of aggregate-asphalt interface transition zone were observed using scanning electron microscopy (SEM). Moreover, Fourier Transform Infrared (FTIR) Spectrometer was applied to study the chemical reaction mechanism between steel slag and rubber asphalt. The results show that the adhesion property between steel slag aggregate and rubber asphalt is higher than that between diabase and rubber asphalt. The results of net adsorption and SEM test demonstrated the porous structure of steel slag can improve the adhesion between the steel slag and asphalt effectively. In addition, FTIR test results demonstrated there exists chemical reaction between steel slag and asphalt, contributing to the enhancement of the adhesion property between asphalt and steel slag aggregate.