Using a simple model of a vesicle and a substrate, we have studied the surface diffusion of an adsorbed vesicle. We show that the experimentally observed but unexplained fact, that a neutral (POPC) vesicle adsorbed to a SiO2 or mica surface does not diffuse but can be moved laterally by an atomic force microscope (AFM) tip, without rupture, can be explained by transient (i.e., temporary) pinning of lipid head groups to surface charges. We studied the surface diffusion for different vesicle adsorption strengths (without any pinning taking place), with the observation that a stronger vesicle-surface attraction leads to slower surface diffusion. However, the surface diffusion was still significant and too high to explain the experimentally observed immobility. When allowing transient lipid pinning between the vesicle and the surface, a 1-2 orders of magnitude decrease in the surface diffusion coefficient was observed. For a lipid adsorption potential of around 20 k(B)T and a lipid pinning potential of about 25 k(B)T, the vesicle is found to be practically immobile on the surface.