We present a complete, self-consistent set of thermodynamic constitutive equations for viscoelastic solid and fluid materials which can be applied during arbitrary, three-dimensional deformations and thermal processes. Deformational and thermal histories are measured using a fading memory norm in a material time which provides a quantitative indication of the constitutive models’ ability to represent the dynamic response. The free energy constitutive equation is a Frechet expansion about the deformation and temperature histories of arbitrarily large but sufficiently slow departures from equilibrium in material time, The kinetic relationship between the laboratory and material time scales does not depend on equilibrium considerations. This approach greatly extends the applicability of low-order memory expansions to nonequilibrium polymer states. Constitutive equations for stress, internal energy, entropy, enthalpy, and heat capacity are derived. All the required material properties can be evaluated unambiguously by independent experiments via equilibrium and linear thermomechanical tests. Example predictions illustrate (i) the isobaric volume relaxation as a rubber is cooled into the glassy state, (ii) the yielding of a glassy solid polymer in uniaxial extension, (iii) non-Newtonian shear thinning during steady shear of fluids, and (iv) stress overshoot of a polymer fluid in transient shear.