This study investigated the effects of vanillin on methanogenesis and rumen fermentation, and the responses of ruminal protein-degrading bacteria to vanillin (at concentrations of 0, 0.76 and 1.52 g/L), essential oils (clove oil, 1 g/L; origanum oil, 0.50 g/L, and peppermint oil, 1 g/L), and quillaja saponin (at concentration of 0 and 6 g/L) in vitro. Methane production, degradabilities of feed substrate, and ammonia concentration decreased linearly with increasing doses of vanillin. Concentration of total volatile fatty acids also decreased, whereas proportion of butyrate tended to increase linearly with increasing doses of vanillin. Protozoa population decreased, but abundances of Ruminococcus flavefaciens, Prevotella bryantii, Butyrivibrio fibrisolvens, Prevotella ruminicola, Clostridium aminophilum, and Ruminobacter amylophilus increased with increasing doses of vanillin. Origanum and clove oils resulted in lower ammonia concentrations compared to control and peppermint oil. All the tested essential oils decreased abundances of protozoa, Selenomonas ruminantium, R. amylophilus, P. ruminicola and P. bryantii, with the largest decrease resulted from origanum oil followed by clove oil and peppermint oil. The abundances of Megasphaera elsdenii, C. aminophilum, and Clostridium sticklandii were deceased by origanum oil while that of B. fibrisolvens was lowered by both origanum and clove oils. Saponin decreased ammonia concentration and protozoal population, but increased the abundances of S. ruminantium, R. amylophilus, P. ruminicola, and P. bryantii, though the magnitude was small (less than one log unit). The results suggest that reduction of ammonia production by vanillin and saponin may not be caused by direct inhibition of major known proteolytic bacteria, and essential oils can have different inhibitory effects on different proteolytic bacteria, resulting in varying reduction in ammonia production.