Diamond/aluminum (Dia/Al) composites were fabricated by powder metallurgy using starting Al powders of different size. Effect of matrix-to-reinforcement particle size ratio (PSR) on diamond particle dispersion was revealed, and higher thermal conductivity of Dia/Al composites with a certain volume fraction was achieved by changing PSR. The results indicated that, with PSR increasing from 0.225 to 0.9, diamond particles tended to show a connecting dispersion and the thermal conductivity of 40 and 50 vol.% Dia/Al composites increased by 21% (from 389 to 472 W/mK) and by 42% (from 442 to 628 W/mK), respectively. The underlying cause was discussed from the point of the coordination number of Al powders around diamond particles. This study supplies a new idea to improve thermal conductivity of Dia/Al composites by tuning particle dispersion for the first time, which is also applicable to other metal matrix composites.