Radio-frequency magnetron sputtering was used to produce HfO2 films doped with Nd, Si and N. The deposition was carried out in two different atmospheres: i) in pure argon plasma to grow Si-HfO2:Nd films, and ii) in argon-nitrogen mixed plasma to produce Si-N-HfO2:Nd films. The effect of annealing temperature on optical and structural properties of the films was investigated. Annealing was performed at T-A = 800-1100 degrees C for t(A) = 15 min in nitrogen atmosphere. The evolution of film's properties was studied by means of the scanning electronic microscopy (SEM), energy dispersive X-ray spectroscopy (EDS), X-ray diffraction (XRD) and photoluminescence (PL) methods. It was observed that the film morphology depends significantly on the deposition atmosphere. For the Si-HfO2:Nd films, the presence of the grains with the mean size about 100 nm was detected by the SEM method. The tetragonal HfO2 and SiO2 phases have been detected by the XRD method after film annealing at 950 degrees C. PL spectra of these films are complex and demonstrate several PL bands in the visible (400-750 nm) and infrared (800-1430 nm) spectral ranges. Their contribution depends on the annealing temperature and governs the shape of total PL spectrum. In contrary, the Si-N-HfO2:Nd films showed unstructured smooth surface as well as featureless PL spectra. Whatever the annealing temperature, they demonstrate broad unstructured PL band with the peak within 440-480 nm. Peculiarities of PL spectra of both types of the films and the mechanism of phase separation are analyzed and discussed.