We report on the low-temperature scanning tunneling microscopy (STM) measurements of the self-assembly of nonplanar 10,10'-dibromo-9,9'bianthryl (DBBA) molecules on Ag(111) combined with density functional theory (DFT) calculations. DBBA molecules have two enantiomorphous adsorption configurations, from which more chiral structures can be formed. At a low coverage [0.4 monolayer (ML)], DBBA forms racemic netlike islands consisting of molecular chains along <<12(3)over bar>>(Ag). Moreover, the gliding between the molecular chains gives rise to chiral windmill-like patterns in the islands. At 0.8 ML, DBBA forms a racemic row phase and a homochiral hexamer phase. The molecular appearance difference between the two coexisted phases and the DFT calculated molecular adsorption configuration reveal a decrease in the molecular dihedral angle of DBBA, which implies an enhancement in the intermolecular interactions-via CH...pi and halogen bonds. The transition from a racemic packing mode to a homochiral one suggests that the suitability of steric configurations is dominant in the close-packing mode under enhanced intermolecular interactions.