Recently, broad-band conductivity spectra have been taken in the low-temperature gamma-phase of the archetypal fast ion conductor RbAg4I5. Attempts to reproduce the experimental data in a simple model calculation have led to the conclusion that strictly localized displacive movements of interacting ionic charge carriers should play an important role in the low-temperature phase. However, with no detailed structural study of gamma-RbAg4I5 available, the relevant processes could not be identified within the crystal structure. This state of affairs has triggered the present investigation of the structures of all three phases of rubidium silver iodide. Powder diffraction data of RbAg4I5 have been collected at the high-resolution powder diffractometer at ID31 at the European Synchrotron Radiation Facility (ESRF). The structure of the gamma-phase has been solved by successive Rietveld refinements in combination with difference Fourier analyses. The same structural principle is found to prevail in all three phases, interconnected distorted RbI6 octahedra forming a three-dimensional framework, which undergoes only displacive structural changes during the alpha - beta and beta - gamma phase transitions. With decreasing temperature, the disorder in the silver sublattice is found to decrease, and a clustering of the disordered silver ions is found to develop. In the gamma-phase, "pockets" containing partially occupied silver sites have been identified, and it is suggested that the localized displacive motion detected by conductivity spectroscopy is performed by the silver ions located within these pockets.