Photothermal therapy (PTT) has attracted increasing attention as an effective anticancer strategy, which has minimal side effects and could suppress the tumor growth effectively. Herein, poly (ethylene glycol)-stabilized polydopamine-modified gold nanostars (AuNSs@PDA-PEG) nanoparticles (NPs) were fabricated for PTT of cancer. The AuNSs@PDA-PEG NPs exhibited strong near-infrared (NIR) absorbance and excellent photostability for highly efficient PTT. Cytotoxicity assay indicated that the NPs possess high biocompatibility. Moreover, the AuNSs@PDA-PEG NPs exhibited excellent photothermal killing performance in vitro, in which the cell viability decreased to 10.02% under the 808nm laser irradiation. The apoptosis assay indicated that HeLa cells incubated with AuNSs@PDA-PEG NPs can be seriously destroyed by the NIR laser irradiation. The mechanisms of NIR PTT-induced cancer cell death involved mitochondrial dysfunction, increase in lysosomal membrane permeability, and autophagy. Western blot analysis confirmed AuNSs@PDA-PEG NPs-induced decrease in Bcl-2 expression and elevation of Bax expression. Thus, the biocompatible AuNSs@PDA-PEG NPs exhibit great potential in PTT of cancer.