D-A conjugated polymers have played critical roles in recently reported nonfullerene acceptors-based polymer solar cells (NF-PSCs) with high performance. Although the molecular design of the D-A polymers is getting more mature, there are still some fundamental unknowns to be unveiled. Here, three new D-A polymers with varied conjugated length for the D-units in their backbones, namely, PDB-1, PDB-2, and PDB-3, were designed, synthesized, and characterized. It was demonstrated that a longer D-unit leads to stronger interchain interaction and higher hole mobility for pristine polymer films. While blending with IT-4F to fabricate photoactive layers in PSCs, it was found that the domain purity, aggregation size, and pi-pi stacking effect of the polymers can be greatly affected by the D-unit size. Compared to polymers with shorter D-units, for the polymer with the largest D-units (PDB-3), hole and electron transport channels can be much more easily formed in the blend films. Interestingly, the highest efficiency was obtained in the PSCs based on a PDB-2:IT-4F blend, in which PDB-2 shows similar D-unit size to the polymers with state-of-the-art high photovoltaic performance. The correlations between the molecular structure and photovoltaic property of PDB-x polymers demonstrate that the modulation of building block size is an important method for designing high-performance D-A conjugated polymers for PSCs.