Diamond single crystals were grown by a high-pressure high-temperature gradient method using a molten Fe-Ni catalyst in a split-sphere apparatus at a pressure of 5.5 GPa and a temperature of 1473-1873 K. The as-grown crystals, ranging in size from 0.2 to 0.6 g, were generally deep yellow in color because of intake of nitrogen impurity during the process. Two different annealing methods were used to change their color to light and vivid yellow. One method involved annealing at 5.5 GPa and 2193-2473 K within the split-sphere apparatus. The other involved annealing of E-beam irradiated crystals at 1773 K in a high-vacuum furnace at 10(-6) Torr. Distribution of C-center nitrogen, single substitutional nitrogen, and A-center nitrogen defects, a pair of nearest-neighbor nitrogen, were thoroughly studied by FT-IR spectroscopy. Upon annealing, C-center nitrogen defects decreased by 10-60%, while A-center nitrogen defects increased by 5-70%. It was confirmed that the irradiation process and intake of nickel impurity enhanced the nitrogen aggregation rate, which followed the second order kinetics. Measured activation energies were in the range of 2.88-3.01 eV, which were not strongly affected by the irradiation process. The results indicate that the nitrogen aggregation rate was enhanced by vacancies and interstitials introduced by E-beam irradiation.