In the restoration of paintings, solvent diffusion and swelling of polymeric oil paint binding media are important factors to consider. Common cleaning methods with organic solvents or aqueous solutions could lead to undesirable physicochemical changes in the paint in the long term, though the extent of this effect is not yet clear. We used tailored nonporous model systems for aged oil paint to measure paint swelling and solvent diffusion for a wide range of relevant solvents. Using dynamic mechanical analysis (DMA), the glass transition temperature of our model systems was found to be close to room temperature. Subsequently, with a custom sample cell and time-dependent attenuated total reflection Fourier transform infrared (ATR-FTIR) spectroscopy, we were able to accurately track swelling and diffusion processes in the polymer films. To quantify the spectroscopic data, we developed a model that completely describes the solvent migration process, including significant film swelling and non-Fickian solvent diffusion. The relation between solvent properties, the diffusion coefficient, and the swelling capacity proved to be rather complex and could not be explained using a single solvent parameter. However, it was found that strongly swelling solvents generally diffuse faster than weakly swelling solvents and that pigmentation does not significantly influence solvent diffusion. These results contribute to a better understanding of transport phenomena in paintings and support the development of improved paint restoration strategies.