Flame spray pyrolysis (FSP) is an established process to synthesize nanoparticles of various metals and metal oxides. Applying open or enclosed configurations of the FSP reactor is an efficient tool to control the fuel-oxidizer ratio in the reaction zone and, thus, the temperature distribution and the particle formation and growth process. In the present work, geometrical setups representing an open and an enclosed flame reactor are compared and their influence on the temperature, velocity, and particle characteristics is investigated. In addition, several distinct kinetic mechanisms for the combustion reactions are evaluated and their effects on the local reactor temperature and gas composition distribution are analyzed. An Eulerian-Lagrangian approach is adopted to describe the multiphase turbulent gas-droplet flow and a monodisperse approach based on the population balance equation (PBE) model is implemented to predict the particle formation and evolution. From the open reactor results, the air entrainment mass flow rate of gas into the flame is calculated. Several numerical experiments are performed with the enclosed setup. Supplying an appropriate co-flow rate into the enclosed reactor results in similar flame behaviour as found for the open reactor configuration. By reducing the co-flow gas, strong recirculation zones and particle deposition on the enclosure walls are observed. In this situation, the local temperature increases considerably, resulting in larger primary nanoparticle diameters.