A new generalized Douglas-Kroll (DK) approach is proposed for the relativistic many-electron Hamiltonian including the electron-electron interaction. In order to consider the higher-order DK transformation to the two-electron interaction, the present approach adopts the effective one-electron potential in the Dirac-Hartree-Fock/Dirac-Kohn-Sham operator as an expansion parameter in the DK transformation. Its numerical performance is tested for the atomic Hg and molecular HAt and At-2 systems. The third-order DK transformation to both one-electron and two-electron Hamiltonians, which is the highest level of theory treated in this study, gives excellent agreement with the four-component relativistic approach. The first-order DK correction to the two-electron interaction is shown to be satisfactory for both atomic and molecular systems. (C) 2003 American Institute of Physics.