The authors experimentally investigated the dynamic behavior of buoyancy-induced flame oscillation under a swirling flow produced by rotating a cylindrical burner tube, focusing on the characterization of the complex dynamics in flame front by a use of nonlinear time series analysis. A Wayland method (R. Wayland et al., 1993), which quantifies the degree of parallelism of trajectories in phase space constructed from time series data of flame front fluctuations, is applied as a sophisticated nonlinear time series analysis in this work. To reveal whether or not the complex dynamics is deterministic chaos, a quantitative method for discussing the null hypothesis that the irregular components of the flame front fluctuations represent a stochastic process (i.e., a surrogate data method; T. Schreiber and A. Schmitz, 1996), is applied in this work. A sophisticated nonlinear time series analysis in combination with a surrogate data method, which has not been widely applied to the study of combustion phenomena, clearly demonstrates that the dynamic behavior undergoes a significant transition from periodic oscillation to low-dimensional deterministic chaos with increasing rotational Reynolds number.