We examine the spatial distribution of fluorescent-labeled charged polystyrene (PS) particles (particle volume fraction phi = 0.0001 and 0.001, diameter d = 183 and 333 nm) added to colloidal crystals of charged silica particles (phi = phi(s) = 0.035-0.05, d = 118 nm). At phi(s) = 0.05, the PS particles were almost randomly distributed in the volume-filling polycrystal structures before the grain growth process. Time-resolved confocal laser scanning microscopy observations reveal that the PS particles are swept to the grain boundaries of the colloidal silica crystals owing to grain boundary migration. PS particles with d = 2420 nm are not excluded from the silica crystals. We also examine influences of the impurities on the grain growth laws, such as the power law growth, size distribution, and existence of a time-independent distribution function of the scaled grain size.