Membrane processes are considered to be a very effective and promising method for drinking water and wastewater treatments. However, particle removal mechanisms have not been fully elucidated due to complex surface interactions between colloids and membranes, especially for very small colloidal particles. In this study, a series of systematic filtration tests for eight different types of membrane filters, having nominal pore sizes from 0.005 to 0.1 mu m, against 1.7 nm ZnS quantum dots (QDs) and 5, 10 and 20 nm Au nanoparticles (NPs) was performed to understand their retention mechanisms, including rejection in front of the filter surface and adsorption inside the filter. By comparing rejection, adsorption and recovery, it was found that the predominant retention mechanisms for retaining small NPs varied from filter to filter. For instance, electrostatic repulsion played a significant role for the rejection of NPs, i.e. impeding them entering the membrane pores in most membranes. In comparison, the Nylon membrane had a significant adsorption retention ability for Au NPs due to electrostatic attraction. Besides, it was found that filtration flow rate, or flux, was also an important parameter for the final retention because the enhanced hydrodynamic drag could trigger the detachment of deposited NPs or press NPs flowing through the superficial entrance leading to penetration. Tests of 10 nm Au NP retention using five different membranes with the same nominal pore size of 0.1 mu m showed large variation of NP retention efficiencies demonstrating that pore size should not be used as the only criterion for rating filter performance, especially for small NPs. Our results provide not only detailed insights into the retention mechanisms of various membranes but also suggestions on how to select membrane filters for different filtration purposes.