Quantum structures made from epitaxial semiconductor layers have revolutionized our understanding of low-dimensional systems and are used for ultrafast transistors, semiconductor lasers, and detectors. Strain induced by different lattice parameters and thermal properties offers additional degrees of freedom for tailoring materials, but often at the expense of dislocation generation, wafer bowing, and cracks. We eliminated these drawbacks by fast, low-temperature epitaxial growth of Ge and SiGe crystals onto micrometer-scale tall pillars etched into Si(001) substrates. Faceted crystals were shown to be strain-and defect-free by x-ray diffraction, electron microscopy, and defect etching. They formed space-filling arrays up to tens of micrometers in height by a mechanism of self-limited lateral growth. The mechanism is explained by reduced surface diffusion and flux shielding by nearest-neighbor crystals.