Different grades of natural manganese ores were used to remove SO2 by dry flue gas desulfurization. The grades of the manganese ores were negatively correlated with the desulfurization performance. The produced MnSO4 enriched on the surface would block the gas diffusion channels during desulfurization. The better desulfurization properties of the low-grade manganese ores are ascribed to the highly active gamma-MnO2, the porous structure, and the high specific surface area as well as the SiO2 bridge-like effects. This study supports the cooperative use of sulfur-containing flue gases and low-grade natural manganese oxides.