Whether and how neurons that are present in both sexes of the same species can differentiate in a sexually dimorphic manner is not well understood. A comparison of the connectomes of the Caenorhabditis elegans hermaphrodite and male nervous systems reveals the existence of sexually dimorphic synaptic connections between neurons present in both sexes. Here we demonstrate sex-specific functions of these sex-shared neurons and show that many neurons initially form synapses in a hybrid manner in both the male and hermaphrodite pattern before sexual maturation. Sex-specific synapse pruning then results in the sex-specific maintenance of subsets of these connections. Reversal of the sexual identity of either the pre- or postsynaptic neuron alone transforms the patterns of synaptic connectivity to that of the opposite sex. A dimorphically expressed and phylogenetically conserved transcription factor is both necessary and sufficient to determine sex-specific connectivity patterns. Our studies reveal new insights into sex-specific circuit development.