Alkaline-earth atoms have metastable 'clock' states with minute-long optical lifetimes, high-spin nuclei and SU(N)-symmetric interactions, making them powerful platforms for atomic clocks(1), quantum information processing(2) and quantum simulation(3). Few-particle systems of such atoms provide opportunities to observe the emergence of complex many-body phenomena with increasing system size(4). Multi-body interactions among particles are emergent phenomena, which cannot be broken down into sums over underlying pairwise interactions. They could potentially be used to create exotic states of quantum matter(5,6), but have yet to be explored in ultracold fermions. Here we create arrays of isolated few-body systems in an optical clock based on a three-dimensional lattice of fermionic Sr-87 atoms. We use high-resolution clock spectroscopy to directly observe the onset of elastic and inelastic multi-body interactions among atoms. We measure the frequency shifts of the clock transition for varying numbers of atoms per lattice site, from n = 1 to n = 5, and observe nonlinear interaction shifts characteristic of elastic multi-body effects. These measurements, combined with theory, elucidate an emergence of SU(N)-symmetric multi-body interactions, which are unique to fermionic alkaline-earth atoms. To study inelastic multi-body effects, we use these frequency shifts to isolate n-occupied sites in the lattice and measure the corresponding lifetimes of the clock states. This allows us to access the short-range few-body physics without experiencing the systematic effects that are encountered in a bulk gas. The lifetimes that we measure in the isolated few-body systems agree very well with numerical predictions based on a simple model for the interatomic potential, suggesting a universality in ultracold collisions. By connecting these few-body systems through tunnelling, the favourable energy and timescales of the interactions will allow our system to be used for studies of high-spin quantum magnetism(7,8) and the Kondo effect.