The beta-carboline alkaloids have been characterized its a group of potential antitumor agents. The underlying mechanisms of harmine and its derivatives were investigated by DNA binding assay and Topoisomerase (Topo) inhibition assay. Meanwhile, the DNA photocleavage potential of these compounds and their cytotoxicity were also examined by DNA photocleavage assay and cytotoxicity assay in vitro. Harmine and its derivatives exhibited remarkable DNA intercalation capacity and significant Topo I inhibition activity but no effect with Topo II. Introducing in appropriate substituent into position-9 of beta-carboline nucleus enhanced the affinity of the drug to DNA resulting in remarkable Topo I inhibition effects. These results Suggested that the ability of these compounds to act as intercalating agents and Topo I inhibitors was related to the antitumor activity. Moreover, these data showing a correlation between cytotoxicity and Topo I inhibition or DNA binding capacity are very important as they strongly suggested that the Topo I-mediated DNA cleavage assay and DNA binding assay Could be used as a guide to design and develop superior analogues for antiturrior activities. (c) 2005 Elsevier Inc. All rights reserved.