In the study, the propagation behaviors of rotating detonation are investigated in an obround slit-channel combustor which has four curvature transition sections by employing a H-2-O-2-N-2 mixture with different oxygen volume fractions (alpha) and equivalence ratios (ERs). High-frequency pressure sensors are circumferentially mounted into the outer wall to obtain pressure temporal variations. Statistical indexes related to wave velocities are adopted to characterize the behaviors of rotating detonation based on which the experiment data are divided into stable and unstable regions. The stable rotating detonation is obtained within wide-range ERs for alpha = 40% and 35% in which the detonation velocity fluctuation ratio is approximately 1%. The pressure histories indicate that the instability of the detonation front is induced by the curvature transition for alpha = 30% and 21%. Complicated unstable phenomena associated with the detonation propagation are observed in which the acoustic modes can be identified. With respect to alpha = 21%, the unstable propagation of rotating detonation and the intensity of the first longitudinal acoustic mode are heightened when the oxidizer flow rate increases and even lead to detonation quenching. It is proposed that the unstable propagation of rotating detonation results from the coupling of acoustic modes and the curvature transition induced instability of detonation front. The stabilization of rotating detonation is affected by the curvature transition and requires smaller equivalent detonation cell size, i.e. higher activity reactants, in the obround combustor when compared to that in annular combustors. (C) 2019 The Combustion Institute. Published by Elsevier Inc. All rights reserved.