Re-self-assembly of surfactant molecules must occur at moving contact lines of soluble surfactant solutions. Molecules are transported into and out of the contact line region from four sources: the three interfaces meeting at the contact line and the fluid confined between the solid-liquid and liquid-vapor interfaces. As molecules move among these sources at the contact line, they must rearrange. The dynamics of this re-self-assembly has been shown to have a dominating effect on the structure of advancing contact lines, causing unsteady motion and complex structure of the contact line. It might be assumed that the re-self-assembly for receding contact lines leads to more steady contact line movement. However, in this article we show that for a wide variety of systems this is not true. Quasi-static distortions of the contact line occur as it retreats because of the inability of the surfactant to completely re-self-assemble at localized positions along the contact line.