High proton conductivity in hydrophobic backbone-based polymers such as Nafion is known to be due to the formation of organized ionic clusters and channels upon hydration. However, a lower proton conductivity in hydrophilic, ionic polymers and the role played by the microstructure are not well understood. In this work, we demonstrate the importance of heterogeneity in crosslinked ionic polymer networks in explaining proton conductivity. Poly(vinyl alcohol) (PVA) crosslinked with sulfosuccinic acid (SSA) is used as the model polymer system for the study. Evolution of the microstructure with hydration and the effect on proton conductivity are analyzed using ATR-FTIR spectroscopy, dielectric spectroscopy, and small-angle neutron scattering. We show that the presence of the two hydrophilic groups in PVA-SSA (hydroxyl and sulfonic acid), as opposed to Nafion, results in competition for water and a lower proton conductivity. The crosslinked polymer-water system contains heterogeneous domains of crosslink nodes which are conductive. These domains (of size 20-35 angstrom) interconnect with each other and form tortuous percolating domains through which proton conduction takes place. The presence of hydroxyl groups results in some of the domains being ineffective for proton transport, resulting in a lower conductivity. (c) 2016 Wiley Periodicals, Inc. J. Polym. Sci., Part B: Polym. Phys. 2016, 54, 1087-1101