Water hydrogen-bond (HB) dynamics around amino acids in dilute aqueous solution is investigated through molecular dynamics simulations and analytic modeling. We especially highlight the critical role played by hydrophilic HB acceptors: the strength of the HB formed with water has a pronounced effect on the HB dynamics, in accord with several experimental observations. In contrast, we evidence that hydrophilic HB donors induce a moderate slowdown in the water HB exchange dynamics due to an excluded volume effect, similar to that of hydrophobic groups. We present an analytic model which rationalizes the effect of all examined amino acid sites on the HB dynamics and whose predictions are in excellent agreement with the numerical simulations. This model provides the acceleration or retardation in the HB exchange time with respect to the bulk through the combination of the solute excluded volume factor with the solute-water HB strength factor, both referring to the HB exchange transition state.