Fluorescent gold nanoparticles are important biological labels, in particular for combined optical and electron microscopy. It is reported that density and type of surface ligands have key influence on the dominant UV-vis fluorescence band in positively and negatively charged gold nanoparticles capped with citrate, gold oxide, and cetyltrimethyl ammonium bromide (CTAB). The peak excitation and emission energies and fluorescence intensities vary with nanoparticle size, reflecting changes in surface charge and surface potential as well as a varying density of surface adsorbates. The fluorescence peak shifts, the evolution of zeta potentials, and fluorescence intensity trends are explained by a model of the principal fluorescence transitions that takes into account the nanoparticle surface conditions, such as the adhesion of ligands. Varying surface ligands is a simple strategy to optimize fluorescence intensity and to design spectral properties of gold nanoparticles.