Instability of gas-bearing coal seams is one of the destructive disasters often encountered in underground coal mines. Reasonable analysis of the destabilization risk of gas-bearing coal seam has always been an important research topic. The purpose of the study is to understand the principle of destabilization of gas-bearing coal seams by constructing a rational mathematical model. To this end, we first analyzed and improved the constitutive and equilibrium equations of the elastic zone in the coal seam, introduced the expression formula of nonlinear gas pressure distribution based on the effective stress principle, and constructed the destabilization model of gas-bearing coal seams based on Lippmann's coal seam destabilization theory. By using the above method, we obtain the stress equations in the x-direction and y-direction of the coal seam in the elastic/plastic zone and original rock stress zone. We then analyzed the distribution characteristics of stress in the elastic/plastic zone of an actual coal mine, and the destabilization risk of the coal seam in front of the working face based on the principles of stress wave propagation attenuation and reflection-induced tensile failure. Our results showed that the coal seam stress curve obtained through the model is a continuous, smooth and reasonable one, which is consistent with the stress distribution features in the actual situation, therefore solving the sudden jump problem of the vertical stress during the transition between elastic zone and plastic zone in the Lippmann model and verifying the rationality of our improved model. We further deduced the relation of the coal rib tensile stress to plastic zone width based on the stress wave propagation theory and found that when the plastic zone width is in the range of 3.61-23.14 m, the excavated coal seam is under risk for destabilization, and the higher the gas pressure is, the greater the risk. Overall, our study provides a theoretical basis to rationally evaluate coal seam destabilization.