Semimetals are being explored for their unique advantages in low-energy high-speed photodetection, although they suffer from serious drawbacks such as an intrinsically high dark current. In this Perspective, we envision the exploitation of topological effects in the photoresponse of these materials as a promising route to circumvent these problems. We overview recent studies on photodetection based on graphene and other semimetals, and further discuss the opportunities created by topological effects, along with the additional challenges that they impose on photodetector designs. Semimetal photodetectors provide high-speed and broadband operation but suffer from serious drawbacks such as high dark currents. This Perspective discusses the opportunities offered by topological effects to overcome these issues and improve their performance.