Hydrogen bonds (H-bonds) play an important role in determining the structures and dynamics of molecular systems. In this work, we investigated the effect of H-bonds on the vibrational population relaxation and orientational relaxation dynamics of HN3 and N-3(-) in methanol (CH3OH) and N,N-dimethyl sulfoxide (DMSO) using polarization controlled infrared pump-probe spectroscopy and quantum chemical calculations. Our detailed analysis of experimental and computational results reveals that both vibrational population relaxation and orientational relaxation dynamics of HN3 and N-3(-) in CH3OH and DMSO are substantially dependent on the strength of the H-bonds between the probing solute and its surrounding solvent. Especially in the case of N-3(-) in CH3OH, the vibrational population relaxation of N-3(-) is found to occur by a direct intermolecular vibrational energy transfer to CH3OH due to large vibrational coupling strength. The orientational relaxation dynamics of HN3 and N-3(-), which are well fit by a biexponential function, are analyzed by the wobbling-in-a-cone model and extended Debye-Stokes-Einstein equation. Depending on the intermolecular interactions, the slow overall orientational relaxation occurs under slip, stick, and superstick boundary conditions. For HN3 and N-3(-) in CH3OH and DMSO, the vibrational population relaxation becomes faster but the orientational relaxation becomes slower as the H-bond strength is increased. Our current results imply that, H-bonds have significant effects on the vibrational population relaxation and orientational relaxation dynamics of a small solute whose size is comparable to the size of the solvent.