Frequency-resolved, three-pulse photon echo peak shifts of the accessory pigments of the reaction center of Rhodobacter sphaeroides were recorded to study the ultrafast pigment-protein dynamics. For a qualitative understanding of the data, which includes an increase in rephasing capability after 1 ps, it is necessary to take into account the influence of the primary electron donor P on the 800 nm absorption band and direct electron transfer from the excited accessory bacteriochlorohyll B-A. The peak shift data and the absorption spectrum are simulated via the time correlation function of the optical transition frequency. We observe subtle differences in the dynamics on the blue and red side of the absorption band, which may be related to differences in the environment between the BA and Bg pigments, respectively. The bath correlation time is longer, similar to 90 fs, at 810 nm than at 790 nm, where it is similar to 60 fs. Since we conclude from our transient grating measurements that the time of energy transfer is 80 fs at 810 nm and 130 fs at 790 nm, this indicates that around 810 nm no significant nuclear relaxation takes place prior to energy transfer. On a longer time scale we observe coherent oscillations of low frequency, similar to 3-4 and 36 cm(-1), and a small similar to 30 ps decay of rephasing capability.