We have developed a theory of the fifth-order off-resonant spectroscopy to study the effect of anharmonicity of molecular vibrational modes. The anharmonicity, as well as nonlinear dependence of polarizability on nuclear coordinates, can be the origin of the fifth-order Raman signal. A profile of the signal varies depending on the relative importance of the two effects-the anharmonicity and the nonlinearity. The anharmonicity of a potential can be distinguished from the other effects such as the nonlinearity or the inhomogeneity of vibrational modes. In order to carry out calculations analytically, we employ the multimode Brownian oscillator model and treat anharmonicity as perturbation to the harmonic vibrational modes. A simple analytical expression for the fifth-order polarization is obtained through a diagrammatic technique, called Feynman rule on the unified time path. Physical pictures for the analytical expression are given for a single mode system through numerical calculations and through double-sided Feynman diagrams. Applications to CHCl3 and CS are made where the third-order experiments are used to extract parameters. In the CS2 case, the theoretical fifth-order signals are compared with recent experiment, which suggests some sign of anharmonicity.