In most cases, a three-phase model consisting of crystalline domains surrounded by rigid amorphous areas dispersed in the mobile amorphous phase is required and eventually sufficient to accurately describe the microstructure of semicrystalline polymers. This work shows that the microstructure developed by poly(hydroxybutyrate-co-hydroxyvalerate) by cold crystallization is better described by a complex two-phase model in which the boundaries of the crystalline domains and the surrounding amorphous environment form a "continuum of mobility". This concept can be extended to a wide range of semicrystalline polymers. When the crystalline and the amorphous phases are strongly coupled, the rigid and mobile amorphous fractions are hardly fractionated, and the whole noncrystalline phase should be rather depicted as continuum with a broad distribution of the relaxation times associated with the glass transition. Such a depiction of the amorphous phase allows taking into account any modification of the mobility landscape with time, which can be evidenced as the progressive spreading of the relaxation functions. From a practical point of view, the rigidification of the "continuum of mobility" upon storage in conditions of cold crystallization could be considered as a cause of the progressive embrittlement sometimes observed in semicrystalline materials during physical aging and would be explained by a redistribution of the relaxation times translated in terms of relaxation temperatures.