Site Ordovician limestone karst water plays a key role in the release of heavy metals from abandoned coal gangue or filled-in goafs in closed coal mines. In this study, a series of immersion experiments was performed to evaluate the leaching behavior of heavy metals in coal gangue from the Baizhuang coal mine in Shandong Province. The experiment was conducted with different immersion solutions, including site Ordovician limestone karst water (pH 7.3) that was compared to three solution pH levels (pH 5.3, 7.3, and 9.3), over a maximum duration of 648 h. The surface structure, chemical composition of the original coal gangue and post-leached residue, and multiple water quality parameters [pH, oxidation reduction potential (ORP), electrical conductivity (EC), total dissolved solids (TDS), and dissolved oxygen (DO)] of the immersion solutions were analyzed using scanning electron microscopy-energy-dispersive spectroscopy (SEM-EDS), X-ray fluorescence spectrometry (XRF), and a Hach Hydrolab multiparameter water quality analyzer, respectively. The results indicated that the site karst water presented quite different implications for heavy metal dissolution compared to laboratory solutions having the same and different pH levels as the karst water, which often showed the lowest values of heavy metal concentrations for the four solutions. Although the heavy metal releases did not change in order of increasing solution pH levels, the dissolved quantities of heavy metals in the four solutions showed a generally similar development trend with processing using increasing immersion times and increased almost to the maximum time used, except that Pb exhibited a strong increase followed by a decline. Additionally, the pH, ORP, EC, TDS, and DO values for the coal gangue at different solutions over time showed marked differences. Additionally, the SEM-EDS and XRF results showed that stable heavy metals, including Mn and Pb, released more readily in site karst water or neutral solutions than in acidic and alkaline solutions but unstable heavy metals, including Fe, Zn, and Cu, more easily released under acidic and alkaline solutions. In abandoned coal mines, coal gangues under the effects of site Ordovician limestone karst waters may pose a lesser environmental problem than indicated by the laboratory results.