The demand of electric energy is increasing globally, and the fact remains that the major share of this energy is still being produced from the traditional generation technologies. However, the recent trends, for obvious reasons of environmental concerns, are indicating a paradigm shift towards distributed generation (DG) incorporating renewable energy resources (RERs). But there are associated challenges with high penetration of RERs as these resources are unpredictable and stochastic in nature, and as a result, it becomes difficult to provide immediate response to demand variations. This is where energy storage systems (ESSs) come to the rescue, and they not only can compensate the stochastic nature and sudden deficiencies of RERs but can also enhance the grid stability, reliability, and efficiency by providing services in power quality, bridging power, and energy management. This paper provides an extensive review of different ESSs, which have been in use and also the ones that are currently in developing stage, describing their working principles and giving a comparative analysis of important features and technical as well as economic characteristics. The wide range of storage technologies, with each ESS being different in terms of the scale of power, response time, energy/power density, discharge duration, and cost coupled with the complex characteristics matrices, makes it difficult to select a particular ESS for a specific application. The comparative analysis presented in this paper helps in this regard and provides a clear picture of the suitability of ESSs for different power system applications, categorized appropriately. The paper also brings out the associated challenges and suggests the future research directions.