The multiscale coarse-graining (MS-CG) approach developed in our previous work is extended here to model solvent-free lipid bilayers. The water (solvent) molecules are completely integrated out of the coarse-grained (CG) effective force field. The MS-CG potential, a sum of pairwise central terms, accurately approximates the many-body potential of mean force in the coarse-grained coordinates. It thus incorporates both energetic and entropic contributions. To improve the stability and elastic properties of the MS-CG simulated bilayer, an additional constraint was adopted: the partial virial associated with CG bilayer sites was matched to its corresponding atomistic value for each configuration of the system. The resulting solvent-free MS-CG model reproduces a liquid-state lipid bilayer with accurate structural and elastic properties. Finally, the solvent-free MS-CG model is used to simulate a very large, flat bilayer and two liposome geometries, demonstrating its greatly enhanced computational efficiency.