Three composition groups in the BaTiO3-CaTiO3-BaHfO3 ternary system, (1-x)Ba(HfyTi1-y)O-3-x(Ba1-zCaz)TiO3 (abbreviated as BHyT-xBC(z)T with x=0-1.0, y=0.16, z=0.20; x=0-1.0, y=0.16, z=0.30; x=0-1.0, y=0.20, z=0.30, respectively), have been designed for investigating the variation of the intermediate O-phase region and its effect on the electrical properties. The temperature-composition phase diagram of each group has been proposed based on the X-ray diffraction patterns and the temperature-dependent dielectric behaviors. The enhanced piezoelectric properties are achieved in the O-T phase boundary compositions of the three groups, which are BH0.16T-0.58BC(0.20)T, BH0.16T-0.48BC(0.30)T and BH0.20T-0.53BC(0.30)T, respectively. Piezoelectric coefficient d(33) of 448pC/N is obtained in BH0.20T-0.53BC(0.30)T, which is higher than those of the other two phase boundary compositions. The O-phase zone in BH0.20T-xBC(0.30)T is narrower than those in the BH0.16T-xBC(0.20)T and BH0.16T-xBC(0.30)T. In spite of its small O-phase volume occupancy, the O-phase plays a key role in the properties of the system. Our work confirms that the enhancement in piezoelectric properties is not only related to the O-T phase boundary near room temperature (RT), but also related to the shift of TR-O toward RT. In addition, a quantitative relation between the phase boundary composition and atomic mole ratio of Hf to Ca (R-m) is proposed, which the R-m value corresponding to the phase boundary is about 0.58. The results clearly demonstrate that in this system high piezoelectric properties are achieved by tuning the specific R-m value. Such a work may provide new clues for designing lead-free piezoelectric materials with enhanced piezoelectric property.