The properties of unsaturated polyester (UP)/montmorillonite (MMT) nanocomposite greatly depend on the preparation procedure because of the chemical reactions and physical interactions involved. To investigate the properties and formation mechanism of UP/MMT nanocomposite, samples were prepared by two different mixing methods. The first method, simultaneous mixing, is similar to the method used for preparing the conventional unsaturated polyester and filler composite. The second method is the sequential mixing, a new approach for preparing unsaturated polyester-layered silicate nanocomposite. In the first step, pre-intercalates of the unsaturated polyester and MMT nanocomposites were prepared. In other words, mixture of the UP and organophillic-treated MMT are prepared in the first step; a styrene monomer was then added to the pre-intercalates of UP/MMT with varying mixing time. The structures of UP/MMT nanocomposite were investigated by X-ray diffraction and transmission electron microscopy. To investigate the formation mechanism of UP/MMT nanocomposite, dynamic mechanical thermal analysis, solution-rheometry and melt-rheometry were performed. The properties and formation processes depending on two methods are compared. These tests enable us to understand the mechanism of UP-silicate nanocomposite formation. Based on this mechanism, we have been able to increase the crosslinking density and the degree of dispersion in UP/MMT nanocomposite.