Understanding the effects of graphene-based nanomaterials on lipid membranes is fundamental to determine their environmental impact and the efficiency of their biomedical use. By means of molecular dynamics simulations of simple model lipid bilayers, we analyze in detail the different interaction modes. We have studied bilayers consisting of lipid species (including cholesterol) which display different internal liquid orderings. Nanometric graphene layers can be transiently adsorbed onto the lipid membrane and/or inserted in its hydrophobic region. Once inserted, graphene nanometric flakes display a diffusive dynamics in the membrane plane, they adopt diverse orientations depending on their size and oxidation degree, and they show a particular aversion to be placed close to cholesterol molecules in the membrane. Addition of graphene to phase-segregated ternary membranes is also investigated in the context of the lipid raft model for the lipid organization of biological membranes. Our simulation results show that graphene layers can be inserted indistinctly in the ordered and disordered regions. Once inserted, nanometric flakes migrate to disordered and cholesterol-poor lipid phases.