Responsive nanomaterials composed of gold nanopartides (AuNPs) and temperature-responsive poly(N-isopropylacrylamide) (PNIPAM) hydrogels offer the promise of designing smart materials that can change color in response to varying thermal or photothermal stimuli. Typical PNIPAM/AuNP hybrids are heavily loaded with AuNPs. Here, we demonstrate that hybrids with an average loading of three to five AuNPs per PNIPAM sphere exhibit peak extinction shifts of over 150 nm and color change from red to purple to gray as the temperature increases from 25 to 50 degrees C. We observe that the time scale for spectral shifts is offset from that for hydrophobic collapse of the PNIPAM spheres. Facilitated by the low loading density, we combine kinetic studies of the changes in the extinction spectra with finite-difference time-domain simulations to show that the location of AuNPs relative to the PNIPAM sphere at different stages of collapse is a key variable accounting for the time and temperature dependence of the experimental data.