This paper develops and analyses a novel method for identifying Wiener-Hammerstein models, i.e. models consisting of two linear dynamic parts with a static non-linearity in between. Starting from the best linear model, which is a consistent estimate of the system dynamics for Gaussian excitation, the identification problem includes the partitioning of the poles and zeros between the two linear parts. This partitioning can be formulated as a discrete optimization problem. The fractional approach considers a relaxation of this problem into a continuous one, by parameterizing the partition of each pole and zero in a fractional way, and carrying out the computations in the frequency domain. In this paper it is shown that the fractional approach becomes ill-conditioned for some configurations of the poles and zeros of the linear dynamic parts, causing identifiability issues. A modification of the original fractional approach is then introduced, based on series expansion of the fractional transfer functions. This modification shares most of the properties of the fractional approach. However, it is shown that it provides an implicit regularization of the identification problem. It addresses the ill-conditioning problem while preserving meaningful statistical properties of the estimation. Furthermore, a lifted formulation of the estimation problem is proposed, which improves the algorithmic performance in the framework of Newton-based methods. (C) 2018 Elsevier Ltd. All rights reserved.