Differences between female and male brains exist across the animal kingdom and extend from molecular to anatomical features. Here we show that sexually dimorphic anatomy, gene expression and function in the nervous system can be modulated by past experiences. In the nematode Caenorhabditis elegans, sexual differentiation entails the sex-specific pruning of synaptic connections between neurons that are shared by both sexes, giving rise to sexually dimorphic circuits in adult animals(1). We discovered that starvation during juvenile stages is memorized in males to suppress the emergence of sexually dimorphic synaptic connectivity. These circuit changes result in increased chemosensory responsiveness in adult males following juvenile starvation. We find that an octopamine-mediated starvation signal dampens the production of serotonin (5-HT) to convey the memory of starvation. Serotonin production is monitored by a 5-HT1A serotonin receptor homologue that acts cell-autonomously to promote the pruning of sexually dimorphic synaptic connectivity under well-fed conditions. Our studies demonstrate how life history shapes neurotransmitter production, synaptic connectivity and behavioural output in a sexually dimorphic circuit.