The purpose of this study is to optimize the collimator mask design for the rotational modulation collimator (RMC) imaging system. First, we optimized the slat and slit width of the collimator mask which affects the number of particles entering the detector and, thus, alters the morphology of modulation patterns. However, the conventional collimator mask design with bilateral symmetry posed an inherent ambiguity issue in the estimation of the source location in the reconstructed image. Therefore, we propose a new asymmetric collimator mask design which can solve the ambiguity problem for the development of the dual-particle imaging system. Modulation patterns for various types of asymmetric mask designs were calculated with Monte Carlo simulations, and modulation and transmission efficiencies were compared with ones for the conventional RMC mask design to evaluate the nominal performance of the new mask designs. The proposed asymmetric collimator mask removed the ambiguous artifact from the reconstructed image without compromising transmission and modulation efficiencies. In addition, one can even expect to achieve better spatial resolution in the reconstructed image with the proposed new asymmetric mask design, as compared with the asymmetric mask design suggested from the previous study.