This paper presents a modular approach for the impedance modeling of three-phase voltage source converters (VSC) by representing the VSC dynamics using three-by-three transfer matrix in the dq, sequence, and phasor domains. The transfer matrix form simplifies the modeling process by separately modeling the ac and dc side dynamics, and describing the VSC dynamics independent of the ac and dc side networks. It also explicitly captures coupling among the dominant frequency components of the ac and dc side voltages and currents in the off-diagonal elements. Modeling of the VSC ac and dc side impedances, including the effects of the network on the other side of the VSC, is presented using the transfer matrix models. Transfer matrix based impedance modeling in the three domains and several stability analysis case studies are presented for a VSC-based HVDC station in an offshore wind farm. It is shown that the coupling between the ac and dc networks, and between the positive and negative sequence components of the three-phase quantities, play an important role in the low-frequency stability of the VSC. Impedance models and stability analysis predictions are validated using the wind farm simulations.