The dephasing and energy relaxation contributions to the line width in infrared (IR) and sum-frequency generation (SFG) spectra of adsorbates are derived from the generalized master equation approach. Expression for the line shift is also obtained. The anharmonic interaction between the adsorbate and the substrate is expanded in a polynomial in terms of the adsorbate and phonon coordinates, and the dephasing is shown to be mainly due to two-phonon processes, while two-phonon, three-phonon or four-phonon processes can contribute to energy relaxation, depending on the relative values of the adsorbate vibrational and the phonon frequencies. The temperature-dependence data of the IR absorption for C(111):H is found to be consistent with the theory, and the large line width for C(111):D can be accounted for by the efficient two-phonon energy relaxation process which is not available for C(111):H due to the higher adsorbate vibrational frequency for C(111):H.