In this paper, it is shown that the least squares algorithm with covariance reset, which is originally developed for the purpose of constant parameter identification, can be effectively applied to the observer design for a general linear time-varging system. The new observer successfully avoids many of the disadvantages of other time-varying observers, such as slow convergence rate, heavy computation load, high amplification of measurement noise, and the inapplicability to systems with time-varying observability indexes or discontinuous parameter variations.