Hexagonal boron nitride is a model lamellar compound where weak, non-local van der Waals interactions ensure the vertical stacking of two-dimensional honeycomb lattices made of strongly bound boron and nitrogen atoms. We study the isotope engineering of lamellar compounds by synthesizing hexagonal boron nitride crystals with nearly pure boron isotopes (B-10 and B-11) compared to those with the natural distribution of boron (20 at% B-10 and 80 at% B-11). On the one hand, as with standard semiconductors, both the phonon energy and electronic bandgap varied with the boron isotope mass, the latter due to the quantum effect of zero-point renormalization. On the other hand, temperature-dependent experiments focusing on the shear and breathing motions of adjacent layers revealed the specificity of isotope engineering in a layered material, with a modification of the van der Waals interactions upon isotope purification. The electron density distribution is more diffuse between adjacent layers in (BN)-B-10 than in (BN)-B-11 crystals. Our results open perspectives in understanding and controlling van der Waals bonding in layered materials.