The sequence-domain impedance using harmonic linearization technique can well describe the harmonic behavior of the interconnected system consisting of the grid and converter. However, fast phase-locked loop (PLL) design, unbalanced current controller structure will cause the unneglectable off-diagonal components in impedance matrix of converter. Due to the frequency coupling and sequence coupling effect in the sequence domain, the traditional single input single output transfer function lacks the capability to accurately express this frequency shift relationship. Therefore, in this paper, the harmonic transfer matrix (HTM) is used to model the frequency shift existing in the sequence-domain impedance model. By unifying positive sequence and negative sequence components, the complexity of impedance model derivation can be reduced. The linear time-periodic system is transformed to a linear time-invariant system in harmonic domain. And then, Nyquist criteria is applied to analyze the stability of the interconnected system. Finally, simulation results verify the proposed HTM-based sequence-domain impedance modeling method.