We have performed microwave-infrared double resonance measurements on OCS in pure He-4 and mixed He-4/He-3 clusters. In contrast to our previous study on OCS in pure He-4 cluster, where an asymmetric inhomogeneous linewidth increasing with rotational quantum number j was observed [S. Grebenev, M. Havenith, F. Madeja, J. P. Toennies, and A. F. Vilesov, J. Chem. Phys. 113, 9060 (2000)], we find the lines in the mixed helium cluster to be much narrower, showing a symmetric, probably homogeneous line width. The spectra indicate a significant difference in the rotational relaxation of OCS in mixed clusters as compared to pure He-4 cluster. We have applied the model of Lehmann [K. K. Lehmann, Mol. Phys. 97, 645 (1999)], describing the interaction of a molecule with the helium cluster, and modified it in order to describe particle in a box states of OCS in mixed helium clusters. The calculations indicate while in the mixed cluster the OCS is confined to the inner He-4 core, the energy of the particle in the box states is increased as compared to the pure He-4 cluster. Relaxation mechanisms via these particle in the box states and surface excitations (ripplons) of the cluster are discussed in the text.