Molecular simulations and an inherent structure analysis are used to investigate how the response of a glass to stress depends on its thermal history. Although the stress response at low and high strains is independent of the thermal history of the glass, the stress response at intermediate strains depends strongly on the thermal history. Previous experiments and simulations have shown that the peak stress, defined as the maximum stress the system can withstand before flowing, increases as a glass is annealed. The present inherent structure analysis addresses this phenomenon. The present results show that as glasses are strained, more highly annealed glasses remain in their initial metabasin up to larger strains (and thus larger stresses). This greater stress-absorbing capacity of the metabasins that characterize more highly annealed systems allows the more annealed systems to withstand more stress before flowing.