We have developed an ellipsometry method to measure the physical aging rate of polymer films that have been thermally quenched and aged in a free-standing state where the stress imparted to the films is well-defined by the thermal-expansion mismatch between film and rigid support. For free-standing polystyrene films supported by rigid sample holders with circular openings, we demonstrate that the physical aging rate is independent of film thickness between 220 and 1800 nm when the applied stress is the same. In contrast, by comparing free-standing films supported by frames of different materials, the physical aging rate decreases by nearly a factor of 2 when the thermal-expansion mismatch, and hence stress, is reduced. We conclude that stress is key in controlling the resulting physical aging rate of free-standing films, and there is no inherent film-thickness dependence (above 220 nm) to the aging rate when stress during glass formation is held fixed.