The interplay of the mechanically and optically induced orientation of azobenzene mesogens as well as the effect of thermal annealing was investigated for both a side-chain liquid crystalline polymer (SCLCP) and a diblock copolymer comprising two SCLCPs bearing azobenzene and biphenyl mesogens, respectively. Typically, the polymer film was first subjected to stretching in either nematic or smectic phase to yield orientation of azobenzene mesogens either parallel or perpendicular to the strain direction, then exposed to unpolarized UV light to erase the mechanically induced orientation upon the trans-cis isomerization, followed by linearly polarized visible light for photoinduced reorientation as a result of the cis-trans backisomerization, and finally heated to different liquid crystalline phases for thermal annealing. The change in orientation was monitored by means of infrared dichroism. The results have unveiled complex and different orientational behaviors for the homopolymer of poly{6-[4-(4-methoxyphenylazo)phenoxy]hexyl methacrylate} (PAzMA) and the diblock copolymer of poly{6-[4-(4-methoxyphenylazo)phenoxy]hexyl methacrylatel}-block-poly{6-[4-(4-cyanophenyl)-phenoxy]hexyl methacrylate} (PAzMA-PBiPh). In particular, the stretching-induced orientation of the homopolymer exerts no memory effect on the photoinduced reorientation, the direction of which is determined by the polarization of the visible light regardless of the mechanically induced orientation direction in the stretched film. Moreover, subsequent thermal annealing in the nematic phase leads to parallel orientation independently of the initial mechanically or photoinduced orientation direction. In contrast, the diblock copolymer displays a strong orientation memory effect. Regardless of the condition used, either for photoinduced reorientation or thermal annealing in the liquid crystalline phase, only the initial stretching-induced perpendicular orientation of azobenzene mesogens can be recovered. The reported findings provide new insight into the different orientation mechanisms, and help understanding the important issue of orientation induction and control in azobenzene-containing SCLCPs.