We apply the observability rank condition to study the observability of various viscoelastic fluids under imposed shear or extensional flows. In this paper the observability means the ability of determining the viscoelastic stress from the time history of the observations of the first normal stress difference. We consider four viscoelastic models: the upper convected Maxwell (UCM) model, the Phan-Thien-Tanner (PTT) model, the Johnson-Segalman (JS) model and the Giesekus model. Our study reveals that all of the four models have observability for all stress components almost everywhere under shear flow whereas under extensional flow most of the models have no observability for the shear stress component. More specifically, for UCM and JS models under imposed shear flow, the observations of the first normal stress difference allow the reconstruction of all components of viscoelastic stress. For UCM and JS models under extensional flow, the two normal stress components can be determined from the measurements of the first normal stress difference; the shear stress component does not affect the evolution of the normal stress components and consequently it cannot be extracted from the observations. Under shear flow, the PTT and Giesekus models have observability almost everywhere. That is, all components of the viscoelastic stress can be determined from the observations when the vector formed by the components of viscoelastic stress does not lie on a certain surface. Under extensional flow, the PTT model has observability almost everywhere for normal stress components whereas the Giesekus model has observability almost everywhere for all stress components. We also run simulations using the unscented Kalman filter (UKF) to reconstruct the viscoelastic stress from observations without and with noises. The UKF yields accurate and robust estimates for the viscoelastic stress both in the absence and in the presence of observation noises. Published by Elsevier B.V.