Organic-inorganic hybrid poly(styrene-co-butyl acrylate)/organically modified montmorillonite (PSBA/organo-MMT) latex particles have been prepared by in situ emulsion polymerization. The effects of modifier variety and the level of organo-MMT have been investigated on the basis of the characteristics and mechanical properties of the resulting hybrid emulsion polymers. Although the more hydrophilic intercalated organic modifiers increased the latex particle size, the hydrophobic ones decreased the particle size. A more heterogeneous copolymer chain intercalation was seen by widespread XRD reflection as the organo-MMT (organoclay) level increases. The tapping mode atomic force microscopy (AFM) and transmission electron microscopy (TEM) were used to determine the dispersion state of organoclay particles inside the nanocomposite copolymer films. Dynamic mechanical thermal analysis (DMTA) showed that adding the organoclay to the copolymer decreased the maximum loss tangent (tan delta) value and caused the shift to a lower temperature. Interestingly, the incorporation of organoclay decreased the glass storage modulus of the copolymer, while increased the rubbery storage modulus to some extent. In addition, a standard indenter for the nanoscratching of copolymer nanocomposite films was used under low applied loads of 150 and 250 mu N. The nanoscratch results showed that incorporation of a 3 wt % hydrophobic organoclay, e. g., Closite15A, in the copolymer matrix enhanced considerably the near-surface hardness and grooving resistance of the nanocomposite film at room temperature. In fact, copolymer nanocomposite films with higher near-surface hardness and tand curve broadening exhibited more nanoscratch resistance through a specific variety of viscoelastic deformation, which did not create a bigger groove. (C) 2011 Wiley Periodicals, Inc. J Appl Polym Sci 123: 2064-2073, 2012