Electron dynamics in gold nanoparticles with an average diameter between 9 and 48 nm have been studied by femtosecond transient absorption spectroscopy. Following the plasmon bleach recovery after low power excitation indicates that a non-Fermi electron distribution thermalizes by electron-electron relaxation on a time scale of 500 fs to a Fermi distribution. This effect is only observed at low excitation power and when the electron distribution is perturbed by mixing with the intraband transitions within the conduction band (i.e., when the excitation wavelength is 630 or 800 nm). However, exciting the interband transitions at 400 nm does not allow following the early electron thermalization process. Electron thermalization with the lattice of the nanoparticle by electron-phonon interactions occurs within 1.7 ps under these conditions, independent of the excitation wavelength. In agreement with the experiments, simulations of the optical response arising from thermalized and nonthermalized electron distributions show that a non-Fermi electron distribution leads to a less intense bleach of the plasmon absorption. Furthermore, the difference between the response from the two electron distributions is greater for small temperature changes of the electron gas (low excitation powers). No size dependence of the electron thermalization dynamics is observed for gold nanoparticles with diameters between 9 and 48 nm. High-resolution transmission electron microscopy (HRTEM) reveals that these gold nanoparticles possess defect structures. The effect of this on the electron-phonon relaxation processes is discussed. 18 nm gold-silver alloy nanoparticles with a gold mole fraction of 0.8 are compared to 15 nm gold nanoparticles. While mixing silver leads to a blue-shift of the plasmon absorption in the ground-state absorption spectrum, no difference is observed in the femtosecond dynamics of the system.