Grain boundary sliding is an important process affecting the mechanical properties of polycrystalline materials. To elucidate the microscopic mechanisms that accompany the sliding, density functional theory simulations of twist and tilt boundaries in germanium, a typical brittle semiconductor, and aluminium, a typical ductile metal, have been performed. A variety of sliding behaviours, which depend not only on the covalent versus metallic character of the bonding, but also on the boundary geometry, the local order and the presence of defects, is found. While in germanium sliding is controlled by local stick-slip events involving rebonding of a few atoms at the boundary interface, in aluminium, larger numbers of atoms act in concert over extended areas, ultimately limited by boundary defects.