We calculate potentials of mean force (PMFs) and mean first passage times for a surfactant to escape a micelle, for both ionic sodium dodecyl sulfate (SDS) and nonionic ethoxylated alcohol (C12E5) micelles using both atomistic and coarse-grained molecular dynamics (MD) simulations. The PMFs are obtained by umbrella sampling and used in a Smoluchowski first-passage-time theory to obtain the times for a surfactant to escape a micelle. The calculated mean first passage time for an SDS molecule to break away from a micelle (with an aggregation number of 60) is around 2 mu s, which is consistent with previous experimental measurements of the fast relaxation time for exchange of surfactants between the micellar phase and the bulk solvent. The corresponding escape time calculated for a nonionic ethoxylated alcohol C12E5, with the same tail length as SDS, is 60 mu s, which is significantly longer than for SDS primarily because the PMF for surfactant desorption is about 3kT smaller than for C12E5. We also show that two coarse-grained (CG) force fields, MARTINI and SDK, give predictions similar to the atomistic CHARMM force field for the nonionic C12E5 surfactant, but for the ionic SDS surfactant, the CG simulations give a PMF similar to that obtained with CHARMM only if long-range electrostatic interactions are included in the CG simulations, rather than using a shifted truncated electrostatic interaction. We also calculate that the mean first passage time for an SDS and a C12E5 to escape from a latex binder surface is of the order of milliseconds, which is more than 100 times longer than the time for escape from the micelle, indicating that in latex waterborne coatings, SDS and C12E5 surfactants likely bind preferentially to the latex polymer interface rather than form micelles, at least at low surfactant concentrations.