In the last decade, kernel-based learning approaches typically employed for classification and regression have shown outstanding performance also in dynamic system identification. The typical way to compute the solution of this learning problem subsumes the inversion of the kernel matrix. However, due to limited machine precision, this might not be possible in many practical applications. In this article, we analyze the aforementioned problem and show that the typical estimate is just one of the possible infinite solutions that can be leveraged, considering both the supervised and the semisupervised settings. We show under which conditions the infinite solutions are equivalent, and if it is not the case, we provide a bound on the mismatch between two generic solutions. Then, we propose two specific solutions that are particularly suited to boost sparsity or performance.