The importance of highly siderophile elements (HSEs; namely, gold, iridium, osmium, palladium, platinum, rhenium, rhodium and ruthenium) in tracking the late accretion stages of planetary formation has long been recognized. However, the precise nature of the Moon's accretional history remains enigmatic. There is a substantial mismatch in the HSE budgets of the Earth and the Moon, with the Earth seeming to have accreted disproportionally more HSEs than the Moon(1). Several scenarios have been proposed to explain this conundrum, including the delivery of HSEs to the Earth by a few big impactors(1), the accretion of pebble-sized objects on dynamically cold orbits that enhanced the Earth's gravitational focusing factor(2), and the 'sawtooth' impact model, with its much reduced impact flux before about 4.10 billion years ago(3). However, most of these models assume a high impactor-retention ratio (the fraction of impactor mass retained on the target) for the Moon. Here we perform a series of impact simulations to quantify the impactor-retention ratio, followed by a Monte Carlo procedure considering a monotonically decaying impact flux(4), to compute the impactor mass accreted into the lunar crust and mantle over their histories. We find that the average impactor-retention ratio for the Moon's entire impact history is about three times lower than previously estimated(1,3). Our results indicate that, to match the HSE budgets of the lunar crust and mantle(5,6), the retention of HSEs should have started 4.35 billion years ago, when most of the lunar magma ocean was solidified(7,8). Mass accreted before this time must have lost its HSEs to the lunar core, presumably during lunar mantle crystallization(9). The combination of a low impactor-retention ratio and a late retention of HSEs in the lunar mantle provides a realistic explanation for the apparent deficit of the Moon's late-accreted mass relative to that of the Earth.