Two dimensional nanomaterials have provided immense potential for separation applications to relieve the trade-off relationship between selectivity and permeability. Covalent organic frameworks (COFs) are highly compatible with polymers, which can be used to obtain defect free membranes with high flux and enhanced separation performance. The state-of-the-art COF-based membranes are concluded and discussed in this review based on their preparation method and application domains. The structure of nanochannels within the COF-based membranes, the separation performance, and COF design for accurate molecular sieving were discussed. Developing novel COFs with new linkages and binding blocks and COFs with functional groups greatly broadens its application in membrane separation. Although challenges are still in front that need to be conquered, the rapid development of COF forming technologies and its applications in membrane separation can serve as a valuable platform for future applications of such new types of porous materials. The variation of COFs' structure and properties can provide the COF-based separation process numerous possibilities on membrane design, modification, and application, which finally serve for practical applications.