Magnesium based metal hydride has been viewed as one of the most commonly-used materials in the practical applications of hydrogen energy systems. The heat and mass transfer processes have significant effects on the hydrogen storage performance of magnesium based metal hydride reactors. Incorporating helical coil heat exchanger into the reactor could be an effective way to improve the performance of heat and mass transfer. In this work, a new three-dimensional model for magnesium based metal hydride reactor with helical coil heat exchanger is proposed and solved using the commercial software package COMSOL Multiphysics V3.5a. The comparison of hydrogen storage behaviors between the reactors incorporating the traditional straight pipe and new helical coil heat exchangers is firstly conducted based on the numerical simulation. The comparison results show that the helical coil heat exchanger has better effect on improving the characteristics of reactor than the straight pipe heat exchanger due to its secondary circulation. The effects of key parameters, including the initial conditions, heat transfer coefficients of heat transfer fluid and helical coil geometry on the characteristics of reactor with the helical coil heat exchanger are also analyzed systematically. It is discovered that larger initial hydrogen pressure and lower initial temperature are beneficial to the improvement of hydrogen absorption kinetics, because of the greater driving force for the hydriding reaction. The results of optimal design suggest that smaller non-dimensional pitch, the ratio of helical pitch to helical diameter, improves the heat and mass transfer performance. The reactor with 0.333 of non-dimensional pitch exhibits the best hydrogen storage behaviors under 3.0 MPa and 523 K, when the heat transfer coefficient of heat transfer fluid is 500 W m(-2) K-1. Approximately 95% of hydrogen absorption process is completed within about 1000 s in this case. (C) 2014 Elsevier Ltd. All rights reserved.