A regularized estimator is proposed for regression models in the case where the covariances may be singular. Conditions guaranteeing proximity of a regularized estimator to the optimal estimator are obtained by appropriate choice of regularization parameters by allowing a prescribed level of uncertainty. A simple Monte-Carlo simulation study is reported to highlight some aspects and performance of the proposed approach.