This research investigates polystyrene mixtures with nanosized surface-modified particles, i.e., molecular silicasol, silsesquioxane, and first-generation carbosilane dendrimer. Three types of similarly sized particles with the same type of surface layer possess different chemical structures and rigidities of the cores. The latter is defined not by the numerical value of the elastic modulus but is based on the concept of molecular mobility of the chemical bonds that form the core. The phase and rheological behaviour as well as the relaxation and dielectric properties of the systems were investigated. The viscosity and the glass transition temperature of the systems decrease with increasing filler content but do not depend on the core rigidity of the particles. However, the molecular structure of the core of a hybrid nanoparticle defines both its own elastic modulus and the elastic modulus of the whole composite as well as affects the rate and activation energy of the segmental motion in the polymer. The characteristics of the composite depend on the core rigidity of the nanoparticles if the correlation radius of the physical characteristic is comparable to or less than the nanoparticle radius. The organic surface layer of nanoparticles serves both as a compatibilizer of the polymer-nanoparticle system and as a peculiar type of a "solvent" in the polymer matrix. (C) 2015 Elsevier B.V. All rights reserved.