Stimuli-responsive assembly of gold nanoparticles (AuNPs) with precise control of the plasmonic properties, assembly size, and stimuli responsivity has shown potential benefits with regard to biosensing devices and drug-delivery systems. Here we present a new pH-responsive coassembly system of oligo(ethylene glycol) (OEG)-coated AuNPs with anionic polymers as an external mediator via hydrogen bonding in water. Hydrogen-bond-driven coassemblies of OEG-AuNPs with poly(acrylic acid) (PAA) were confirmed by the monitoring of plasmonic peaks and hydrodynamic diameters. In this system, the protonation of anionic polymers on change in pH triggered the formation of hydrogen bond between the OEG-AuNPs and polymers, providing sensitive pH responsivity. The plasmonic properties and assembly size are affected by both the ratio of PAA to AuNPs and the molecular weight of PAAs. In addition, the attachment of hydrophobic groups to the surface ligand or anionic polymer changed the responsive pH range. These results demonstrated that the coassembly with an external mediator via hydrogen bonding provides a stimuli-responsive assembly system with tunable plasmonic properties, assembly size, and stimuli responsivity.