Symmetry plays an essential role in understanding optical activities of a molecule in infrared and Raman vibrational spectroscopy as well as in nonlinear optical vibrational spectroscopy. Each vibrational mode belongs to an irreducible representation of the underlying symmetry group. In this paper, using the alpha-helical polypeptide symmetry as an example, we calculate all the third rank nonzero hyper-Raman tensors as well as the infrared and Raman tensors by applying the projection operators to each irreducible species. We demonstrate that the projection operator method provides selection rules for the infrared, Raman, and hyper-Raman vibrational transitions and also other nonlinear optical spectroscopy such as sum frequency generation and the four-, five-, and six-wave mixing coherent vibrational transitions. Specific expressions for all nonzero elements of the corresponding nonlinear susceptibility tensors in a laboratory-fixed coordinate frame are also deduced.