We investigate the formation dynamics of self-assembled polyelectrolyte multilayers on glass substrates by in situ and ex situ second harmonic generation (SHG) measurements and atomic force microscopy (AFM). The time dependence of the SHG signal during the adsorption process is attributed to a time dependent surface potential of the polyelectrolyte film. The dynamics can be quantitatively understood using a random sequential adsorption (RSA) model for the buildup of a film consisting of polyelectrolyte disks with polydisperse sizes. Differences between wet and dry films are also investigated.