Robust fluorescent photoswitching molecules, having perylene bisimide as the fluorescent unit and diarylethene as the switching unit, were prepared, and their photochromic reactions were measured at the single-molecule level in various polymer matrices. The histograms of the fluorescent on and off times were found to deviate from normal exponential distribution and showed a peak when the molecules are embedded in rigid polymer matrices, such as Zeonex or poly(methyl methacrylate) (PMMA). In soft polymer matrices, such as poly(n-buthyl methacrylate) (PnBMA), exponential distribution was observed for the on and off times. The abnormal distribution suggests that the quantum yields of the photoreactions are not constant and the molecules undergo the reactions after absorbing a certain number of photons. A multilocal minima model was proposed to explain the environmental effect.