We have computed absorption and emission energies of silicon nanocrystals as a function of size and of surface passivants, using both density functional theory and quantum Monte Carlo calculations. We have found that the ionic rearrangements and electronic relaxations occurring upon absorption and emission are extremely sensitive to surface chemistry. In particular, nanoclusters with similar sizes and similar absorption gaps can exhibit strikingly different emission energies. Our results provide a unifying interpretation of several recent measurements, which have observed significantly different emission energies from clusters with similar sizes. Our calculations also show that a combination of absorption and emission measurements can provide a powerful tool for identifying both the size and the surface passivants of semiconductor nanocrystals.