The initial growth of Si1-x-yGexCy film with Si1-xGex interlayer on Si(100) surface is investigated by scanning tunneling microscopy and reflection high energy electron diffraction. The increase in the Ge fraction of Si1-xGex interlayers raises the critical thickness of the transition from two-dimensional (2D) to three-dimensional (3D) growth of Si1-x-yGexCy layers, and is effective to suppress the surface-roughening of Si1-x-yGexCy layers. The atomic-scale valley structure separating 2D-islands of the Si1-x-yGexCy layers are observed on the surface, which is deduced to be the structure for the compressive-strain relaxation of Si1-x-yGexCy layers on the Si substrate as well as missing dimmer rows. (C) 2003 Elsevier B.V. All rights reserved.