Gold nanoparticles (AuNPs) coated with a water-soluble polymer displaying a photoswitchable lower critical solution temperature (LCST) were synthesized, and the photoinduced change in the aggregation state of AuNPs was investigated. The photoactivity of AuNPs was conferred by tethering on the surface a random copolymer of N,N-dimethylacrylamide and N-4-phenylazophenyl acrylamide (P(DMA-co-Azo)), whose LCST could be lowered upon the trans-cis photoisomerization of azobenzene groups. It was found that the dispersion of P(DMA-co-Azo)-coated AuNPs in aqueous solution remained stable regardless of temperature and the isomeric form of azobenzene groups. However, with the presence of free P(DMA-co-Azo) chains in the solution, the reversible photoinduced shift of LCST of the polymer could result in a reversible photoswitching between dispersion and aggregation of AuNPs. A stable dispersion could be obtained at T < LCST of the polymer with azobenzene in the trans form (without irradiation), while upon UV irradiation, the lowering of LCST with cis-azobenzene put the solution to above the altered LCST and the insoluble polymer chains led to aggregation of AuNPs; the initial dispersion state could be recovered by visible light irradiation, bringing cis-azobenzene back to the trans form. The use of such a photoresponsive polymer to optically tune the aggregation state of AuNPs allows the surface plasmon resonance (SPR) of AuNPs lobe reversibly switched by light at two wavelengths, the extent of which depends on the amount of free polymer chains in the solution.