In this study, we found that p-coumaric acid (p-CA), ferulic acid (FA), and condensed tannins were released from rice straw during saccharification. The presence of polyphenols prolonged the lag phase and lowered the productivity of lactic acid. p-CA was identified as a key inhibitor. Tannins had a lower inhibitory effect than p-CA; FA had little inhibitory effect. Acid, alkaline, and ball milling pretreatments elicited different levels of polyphenol release from rice straw. Due to the different levels of polyphenol release in the pretreatment step, the enzymatic hydrolysates contained different concentrations of polyphenols. Compared with fermentation with a synthetic medium, fermentation with the hydrolysates of ball-milled rice straw provided much lower productivity and yield of lactic acid due to the presence of polyphenols. Removal of these compounds played an important role in lactic acid fermentation. When rice straw was alkaline pretreated, the hydrolysates contained few phenolic compounds, resulting in high productivity and yield of lactic acid (1.8 g/L/h and 26.7 g/100 g straw), which were comparable to those in a synthetic medium. This indicates that there is a correlation between removal of phenolic compounds and efficiency in lactic acid fermentation.