In the past decade, our understanding of galaxy evolution has been revolutionized by the discovery that luminous, dusty starburst galaxies were 1,000 times more abundant in the early Universe than at present(1,2). It has, however, been difficult to measure the complete redshift distribution of these objects, especially at the highest redshifts (z>4). Here we report a redshift survey at a wavelength of three millimetres, targeting carbon monoxide line emission from the star-forming molecular gas in the direction of extraordinarily bright millimetre-wave-selected sources. High-resolution imaging demonstrates that these sources are strongly gravitationally lensed by foreground galaxies. We detect spectral lines in 23 out of 26 sources and multiple lines in 12 of those 23 sources, from which we obtain robust, unambiguous redshifts. At least 10 of the sources are found to lie at z > 4, indicating that the fraction of dusty starburst galaxies at high redshifts is greater than previously thought. Models of lens geometries in the sample indicate that the background objects are ultra-luminous infrared galaxies, powered by extreme bursts of star formation.