Obtaining substantial nonlinear effects at the single-photon level is a considerable challenge that holds great potential for quantum optical measurements and information processing. Of the progress that has been made in recent years1-5 one of the most promising methods is to scatter coherent light from quantum emitters, imprinting quantum correlations onto the photons. We report effective interactions between photons, controlled by a single semiconductor quantum dot that is weakly coupled to a monolithic cavity. We show that the nonlinearity of a transition modifies the counting statistics of a Poissonian beam, sorting the photons in number. This is used to create strong correlations between detection events and to create polarization-correlated photons from an uncorrelated stream using a single spin. These results pave the way for semiconductor optical switches operated by single quanta of light.