The viscoelastic properties of nanocomposites are influenced by both the nanoparticle distribution and the nanoparticle-polymer affinity. These two parameters are closely coupled, and evaluation of individual contributions to the mechanical properties is a critical requirement for efficient development of nanocomposites. To decouple these two effects, we utilized charge repulsion among nanoparticles so that We could essentially eliminate particle agglomeration. We then investigated how the nanoparticle-polymer affinity relates to the mechanical properties of the nanocomposite by compiling silica and polystyrene nanoparticles. The surface roughness of the particles and the molecular conformation of the interfacial layer between the polymer and the nanoparticles were characterized by synchrotron small-angle X-ray scattering and quartz crystal microbalance, respectively. On polystyrene particles, the Surface roughness was larger, and the polymer adsorbed strongly. Consequently, the mobility of the adsorbed polymer was reduced compared to that oil silica particles. This reduced mobility explains a Smaller viscoelastic loss for the polystyrene-filled nanocomposite compared to the silica-filled nanocomposite.