We employ fluorescence correlation spectroscopy (FCS) to study the translational mobility of molecular tracers in stimuli-responsive grafted poly(N-isopropylacrylamide) (PNiPAAm) hydrogels, under variable solvency conditions. Tracer-matrix interactions were tuned by selecting three different molecular tracers. In contrast to a non-interacting tracer (Alexa 647), the mobility of a weakly (Alexa 488) and a strongly interacting (Rhodamine 6G) tracer deviates from a simple single Fickian diffusion. In addition to pure crowding effects, the mobility of both Alexa488 and Rhodamine 6G is influenced by tracer-polymer interactions. We interpret the observed trends in tracer mobility in terms of the interplay between Coulombic repulsions and short-range attractions. Although tracer dynamics and hydrogel swelling ratio are interdependent properties, their relation turns out to be nontrivial and does not allow predictions of tracer dynamics on the basis of polymer structural information. Hence, a universal scaling behavior is not possible, due to tracer-polymer interactions.