The exact mechanisms occurring during chemical vapor deposition are often not well understood or quantified, and CVD is therefore difficult to optimize efficiently. In this article, we implement a process optimization method based on sequential experimental design. This method simultaneously employs both mechanistic and empirical models, using confidence intervals to predict the region of potential optima. New experiments are constrained to this region, so that model predictions will be improved near the process optimum. After six initial experiments and two sequential experiments, a process temperature of 760 degrees C is computed, to achieve a desired film roughness of 7 +/- 1.65 nm. The experimental data are obtained using a chemical vapor deposition system. The yttrium precursor material is deposited onto a 1 in. silicon wafer in a vacuum chamber. The samples are then analyzed using an atomic force microscope to determine the roughness of the samples. On the basis of these data, we create models that are purely empirical, and also models that are motivated by a more mechanistic understanding of the process. The rationale is that, rather than concentrating our experimental design on creating either empirical or mechanistic models, both types of models should be considered when optimizing a process. Furthermore, by developing both model types simultaneously, mechanistic understanding of the process can also be improved. Models are developed which predict the thin film roughness from the surface temperature and molar flow rate of the precursor. Five empirical models are considered, as well as two additional models which relate nucleation density to final film roughness using mechanistic modeling. The temperature is found to be most important for controlling the thin film roughness. An empirical model with two fitted parameters and a linear temperature dependence predicted performance best, while a hybrid model using nucleation density and two fitted parameters was second best. The confidence interval for the best model was reduced by 20% using our experimental methodology, using only two sequential experiments. The experiments indicate that the hybrid model does capture some of the trends of the experimental data but needs a stronger dependence on temperature to have more statistical significance.