The holographic grating recording process in thin films of amorphous azobenzene-functionalized polymers has been widely reported in the literature. In spite of the many reports, little is known about the mechanisms responsible for different temporal behaviors of the diffraction efficiency during long recording times. Here, we report on experimental studies of the diffraction efficiency changes during the holographic diffraction grating recording process in photochromic polymer. The gratings were inscribed for four different polarization combinations of the recording beams: s-s, p-p, s-p and right to left circular polarization (RCP-LCP) employing the degenerate two-wave mixing technique. The grating recording process was simultaneously monitored by three different wavelengths: 514.5 nm (writing) and 632.8 and 904 nm (reading). The temporal evolution of the diffraction efficiency (for all polarization configurations and for each wavelengths) was precisely fitted within the model, which assumes simultaneous formation of the absorption grating and three coupling phase gratings shifted by 0 or pi with respect to each other. Two of the phase gratings originate from the refractive index grating changes in the bulk (volume) of a material and the third one from the surface relief modulation. The model enabled us to extract relevant parameters for each grating such as the build-up time constant, its final amplitude, and the phase shifts between phase gratings. Performed studies and the discussion of results revealed the main differences in the diffraction grating formation process for s-s, p-p, s-p, and RCP-LCP polarization configurations.