Baiyinhua coal was subjected to catalytic pyrolysis over HZSM-5 and Ni-modified HZSM-5 (Ni/HZSM-5) with a novel pyrolysis time-of-flight mass spectrometers (Py-TOF-MS) equipped with vacuum ultraviolet photoionizafion (VUVPI). The soft ionization of VUVPI allowed the detection of volatile organic compounds. The effect of HZSM-5 and Ni/HZSM-5 on distribution of coal pyrolysis products was online investigated. The coal pyrolysis products including alkanes, phenols, aromatics and sulfur containing compounds (H2S and CH3SH) were detected. In the presence of HZSM-5 and Ni/HZSM-5, the products from coal pyrolysis have higher content of substituted benzene and alkenes and lower content of phenols in comparison with that without catalyst. The catalysts promote the conversion of phenols products to aromatic hydrocarbons via dehydroxylation reaction. Ni/HZSM-5 exhibits better catalytic performance for generating products with higher aromatic content and less oxygen species than HZSM-5 alone. Temperature-evolved profiles of pyrolysis products were also measured. It was found that the peak temperature of alkenes and substituted benzene of catalytic pyrolysis shifts to lower temperature region while that of dihydroxybenzenes moves to higher temperature. This work demonstrates the good performance of Py-TOF-MS for the online study of the coal catalytic pyrolysis process.