Deep-etched mesa waveguide in InP/InGaAsP heterostructures were fabricated using different thicknesses of NiCr etching mask in an inductively coupled plasma reactive ion etching system with, Cl-2-based plasmas. An atomic force microscope (AFM) loaded with a carbon nanotube tip was utilized to measure the sidewall roughness of the etched structures. The root-mean-square (rms) roughness values were calculated separately for the horizontal and vertical directions of the waveguide sidewall. As the NiCr mask thickness increased from 10 nm to 150 nm, the rms roughness value in the vertical direction of the waveguide sidewall decreased from 19.5 nm to 2.8 run. On the other hand, the rms roughness value in the horizontal waveguiding direction increased from 2.5 nm to 4.0 nm. The correlation lengths along the horizontal direction for the etched waveguide structures were also extracted from the AFM roughness data. The rms roughness values and correlation lengths were used as parameters in three-dimensional beam propagation method to calculate the optical scattering loss. The calculated optical losses ranged from 1.2 dB/cm to 2.5 dB /cm. (c) 2005 American Vacuum Society.