Multiferroics(1,2), in which dielectric and magnetic orders coexist and couple with each other, attract renewed interest for their cross-correlated phenomena(3,4), offering a fundamental platform for novel functionalities. Elementary excitations in such systems are strongly affected by the lattice-spin interaction, as exemplified by the electromagnons(5,6) and the magneto-thermal transport(7-10). Here we report an unprecedented coupling between magnetism and phonons in multiferroics, namely, the giant thermal Hall effect. The thermal transport of insulating polar magnets (ZnxFe1-x)(2)Mo3O8 is dominated by phonons, yet extremely sensitive to the magnetic structure. In particular, large thermal Hall conductivities are observed in the ferrimagnetic phase, indicating unconventional lattice-spin interactions and a new mechanism for the Hall effect in insulators. Our results show that the thermal Hall effect in multiferroic materials can be an effective probe for strong lattice-spin interactions and provide a new tool for magnetic control of thermal currents.