There are close exchanges between sediment and water in estuaries; however, the patterns of prokaryotic community assembly in these two habitat types are still unclear. This study investigated the bacterial and archaeal abundance, diversity, and community composition in the sediment and the overlying water of the Yellow River estuary. Notably higher prokaryotic abundance and diversity were detected in the sediment than in the water, and bacterial abundance and diversity were remarkably higher than those of archaea. Furthermore, the ratio of bacterial to archaeal 16S rRNA gene abundance was significantly lower in the sediment than in the water. Bacterial communities at different taxonomic levels were apparently distinct between the sediment and water, but archaeal communities were not. The most dominant bacteria were affiliated with Deltaproteobacteria and Gammaproteobacteria in sediment and with Alphaproteobacteria and Betaproteobacteria in water. Euryarchaeota and Thaumarchaeota were the most abundant archaea in both habitats. Although distinct prokaryotic distribution patterns were observed, most of the dominant bacteria and archaea present were related to carbon, nitrogen, and sulfur cycling processes, such as methanogenesis, ammonia oxidation, and sulfate reduction. Unexpectedly, prokaryotes from the water showed a higher sensitivity to environmental factors, while only a few factors affected sediment communities. Additionally, some potential co-occurrence relationships between prokaryotes were also found in this study. These results suggested distinct distribution patterns of bacterial and archaeal communities between sediment and overlying water in this important temperate estuary, which may serve as a useful community model for the further ecological and evolutionary study of prokaryotes in estuarine ecosystems.