We consider the class of spatially decaying systems, where the underlying dynamics are spatially decaying and the sensing and controls are spatially distributed. This class of systems arises in various applications where there is a notion of spatial distance with respect to which couplings between the subsystems can be quantified using a class of coupling weights. We exploit the spatial decay property of the underlying dynamics of this class of systems by introducing a class of sparsity and spatial localization measures. We develop a new methodology based on concepts of q-Banach algebras of spatially decaying matrices that enables us to establish a relationship between spatial decay properties of spatially decaying systems and their sparsity and spatial localization features. Moreover, it is shown that the inverse-closedness property of matrix algebras plays a central role in exploiting various structural properties of spatially decaying systems. We characterize conditions for exponential stability of spatially decaying system over q-Banach algebras and prove that the unique solutions of the Lyapunov and Riccati equations over a proper q-Banach algebra also belong to the same q-Banach algebra. It is shown that the quadratically optimal state feedback controllers for spatially decaying systems are sparse and spatially localized in the sense that they have near-optimal sparse information structures.