The softening in ultrafine-grained silver processed by high-pressure torsion (HPT) was studied during annealing using differential scanning calorimetry (DSC). Two separate exothermic peaks were observed in the DSC thermogram of the HPT-processed sample. It is shown that the first and the second peaks are related to the recrystallization of the middle volume and the surface regions of the HPT-processed disk, respectively. Therefore, a very inhomogeneous sandwich-like microstructure develops during annealing with a soft interior and hard surface layers. The lower thermal stability of the middle region appears to be related to the stronger twinning activity since the twinned volumes can act as nuclei for recrystallized grains. The higher twin-fault probability in the interior is attributed to the larger strain due to the outflow of material between the anvils of the HPT facility during quasi-constrained processing.