Strengthening invar alloy while ensuring its low coefficient of thermal expansion by refining microstructure has been a thorny issue in recent years. In the present work, the effects of Ti, Nb and Ti-Nb compound modifier on rapid solidified microstructure and properties of invar alloy prepared by a mini-type vacuum arc melting furnace were investigated in detail. Invar alloy with uniform and fine structure was obtained by adding 0.2%Ti, 0.2%Nb or 0.2%Ti-Nb compound modifier. And the formation of cellular substructure is attributed to the supercooling condition, leading to segregation of Ni element at the cellular wall and high dislocation density at the intracellular. Results showed that Ti-Nb has the most remarkable refinement and strengthening effect compared with Ti and Nb, who can make the refinement rate of austenite grain and substructure of invar alloy up to 45% and 67.4%, respectively. Moreover, it is revealed that the coefficient of thermal expansion (CTE) of invar alloy with the addition of alloying elements is decreased, which is mainly due to lattice distortion. In addition, the addition of alloying elements is beneficial to the dislocation multiplication of the invar alloy leading the increase of dislocation density, especially for invar alloy with the addition of 0.2%Ti-Nb modifier.