We present a new method to extract distance and orientation dependent potentials between amino acid side chains using a database of protein structures and the standard Boltzmann device. The importance of orientation dependent interactions is first established by computing orientational order parameters for proteins with alpha-helical and beta-sheet architecture. Extraction of the anisotropic interactions requires defining local reference frames for each amino acid that uniquely determine the coordinates of the neighboring residues. Using the local reference frames and histograms of the radial and angular correlation functions for a standard set of nonhomologue protein structures, we construct the anisotropic pair potentials. The performance of the orientation dependent potentials was studied using a large database of decoy proteins. The results demonstrate that the new distance and orientation dependent residue-residue potentials present a significantly improved ability to recognize native folds from a set of native and decoy protein structures. (C) 2003 American Institute of Physics.