Density functional and quantum Monte Carlo calculations are employed to determine the effect of surface passivants on the optical gap of silicon nanoclusters. Our results show that quantum confinement is only one mechanism responsible for visible photoluminescence and that the specific surface chemistry must be taken into account in order to interpret experimental results. Significant changes occur in the optical gap of fully hydrogenated silicon nanoclusters when the surface contains passivants that change the bonding network at the surface. In the case of just one double-bonded oxygen atom, the gap reduction computed as a function of the nanocluster size demonstrates that one contaminant can greatly alter the optical gap. A further significant reduction of the gap occurs with multiple double-bonded oxygen contamination, providing a consistent interpretation of several recent experiments. We predict that other passivants that distort the tetrahedral bonding network at the surface, including other double-bonded groups and in some cases bridged oxygen, will also significantly affect the optical gap. Conversely, single-bonded passivants will have a minimal influence on the optical gap. A discussion of the difference in the strength of the optical transitions for clusters with different passivants is presented.