To develop a high-throughput screening system to measure the conversion of testosterone to dihydrotestosterone (DHT) in cultured human prostate cancer cells using turbulent flow chromatography liquid chromatography-triple quadrupole mass spectrometry (TFC-LC-TQMS). After optimizing the cell reaction system, this method demonstrated a screening capability of 103 samples, including 78 single compounds and 25 extracts, in less than 12 h without manual sample preparation. Consequently, fucoxanthin, phenethyl caffeate, and Curcuma longa L. extract were validated as bioactive chemicals that inhibited DHT production in cultured DU145 cells. In addition, naringenin boosted DHT production in DU145 cells. The method can facilitate the discovery of bioactive chemicals that modulate the DHT production, and four phytochemicals are potential candidates of nutraceuticals to adjust DHT levels in male hormonal dysfunction.