A site-dependent spectral density system-bath model of the Fenna-Matthews-Olsen (FMO) pigment-protein complex is developed using results from ground-state molecular mechanics simulations together with a partial charge difference model for how the long-range contributions to the chromophore excitation energies fluctuate with environmental configuration. A discussion of how best to consistently process the chromophore excitation energy fluctuation correlation functions calculated in these classical simulations to obtain reliable site-dependent spectral densities is presented. The calculations reveal that chromophores that are close to the protein water interface can experience strongly dissipative environmental interactions characterized by reorganization energies that can be as much as 2-3 times those of chromophores that are buried deep in the hydrophobic protein scaffolding. Using a linearized density matrix quantum propagation method, we demonstrate that the inhomogeneous system bath model obtained from our site-dependent spectral density calculations gives results consistent with experimental dissipation and dephasing rates. Moreover, we show that this model can simultaneously enhance the energy-transfer rate and extend the decoherence time. Finally, we explore the influence of initially exciting different chromophores and mutating local environments on energy transfer through the network. These studies suggest that different pathways, selected by varying initial photoexcitation, can exhibit significantly different relaxation times depending on whether the energy-transfer path involves chromophores at the protein solvent interface or if all chromophores in the pathway are buried in the protein.