Carrier relaxation is a key issue in determining the efficiency of semiconductor optoelectronic device operation. Devices incorporating semiconductor quantum dots have the potential to overcome many of the limitations of quantum-well-based devices because of the predicted long quantum-dot excited-state lifetimes. For example, the population inversion required for terahertz laser operation in quantum-well-based devices (quantum-cascade lasers(1,2)) is fundamentally limited by efficient scattering between the laser levels, which form a continuum in the plane of the quantum well. In this context, semiconductor quantum dots are a highly attractive alternative for terahertz devices, because of their intrinsic discrete energy levels. Here, we present the first measurements, and theoretical description, of the intersublevel carrier relaxation in quantum dots for transition energies in the few terahertz range. Long intradot relaxation times (1.5 ns) are found for level separations of 14 meV (3.4 THz), decreasing very strongly to similar to 2 ps at 30 meV (7 THz), in very good agreement with our microscopic theory of the carrier relaxation process. Our studies pave the way for quantum-dot terahertz device development, providing the fundamental knowledge of carrier relaxation times required for optimum device design.