High undercooling of the single-phase decagonal quasicrystal alloy Al72Ni12Co16 was achieved by cyclically superheating the melt in a containerless electromagnetic levitation apparatus. The maximum undercooling of it was up to 368 K. XRD, TEM, SEM and optical microscopy techniques were adopted to investigate the microstructure and identify the phase composition, respectively. A distinct roughening transition was detected in the undercooled samples with the undercooling increasing. The decagonal quasicrystal phase in samples with small undercoolings showed a facet morphology and grew in a lateral growth mode below the roughening transition undercooling point, Delta T = 64 K. Then the decagonal phase took on an equiaxed structure and indicated a continuous growth pattern. The roughening transition was caused by a sudden change of the growth direction from the two-fold axis normal to plane (0 0 0 0 2) to the ten-fold axis in the undercooled melt. Toner's surface dynamics models of,growing three-dimensional quasicrystal and BCT growth model in the undercooled melt (proposed by Boettinger, Coriel and Trivedi) were adopted to explain the roughening transition phenomenon.