We investigate the desorption of Na atoms from large Na clusters deposited on dielectric surfaces. High-resolution translational energy distributions of the desorbing atoms are determined by three independent methods, two-photon laser-induced fluorescence, as well as single-photon and resonance-enhanced two-photon ionization techniques. Upon variation of surface temperature and for different substrates (mica vs lithium fluoride) clear non-Maxwellian time-of-flight distributions are observed with a cos(2) theta angular dependence and most probable kinetic energies below that expected of atoms desorbing from a surface at thermal equilibrium. The half-width of the time-of-flight distribution decreases with increasing surface temperature. A quantitative description of the energy and angular distributions is presented in terms of a model which assumes that following the initial surface plasmon excitation neutral atoms are scattered by surface vibrations. Recent experiments providing time constants for the decay of the optical excitations in the clusters support this model. The excellent agreement between experiment and theory indicates the importance of both absorption of the laser photons via direct excitation of surface plasmons as well as energy transfer with the substrate during the desorption process.