The dynamic nature of the first water adlayers on solid surfaces at room temperature has made the direct detection of their microscopic structure challenging. We used graphene as an atomically flat coating for atomic force microscopy to determine the structure of the water adlayers on mica at room temperature as a function of relative humidity. Water adlayers grew epitaxially on the mica substrate in a layer-by-layer fashion. Submonolayers form atomically flat, faceted islands of height 0.37 +/- 0.02 nanometers, in agreement with the height of a monolayer of ice. The second adlayers, observed at higher relative humidity, also appear icelike, and thicker layers appear liquidlike. Our results also indicate nanometer-scale surface defects serve as nucleation centers for the formation of both the first and the second adlayers.