The development of fire retardant for wire and cable sheathing materials has oriented toward low smoke and halogen-free flame retardant technology to achieve better safety for electrical equipment and devices and to satisfy standards. However, many polymer flame resistance materials require a very high proportion of metal hydrate filler within the polymer matrix (60 wt%) to achieve a suitable level of flame resistance, which may lead to inflexibility, poor mechanical properties and problems during compounding and processing. In this study, the alumina trihydrate (ATH) was added to montmorillonite (MMT) as the halogen-free flame retardant of ethylene-vinyl acetate (EVA) copolymer, with various ratios of EVA/ATH/MMT. The prepared nanocomposites were characterized through various techniques of XRD, tensile test, DSC analysis, TGA, LOI evaluation, and FE-SEM to explore the effects of organic modified clay (OMMT) and the layer distance on the mechanical, thermal, and flame resistance properties. In the XRD examinations, the layer-distance of MMT increased from 1.27 to 1.96 nm when polymer was added to the octadecylamine modified MMT. The best tensile strength was obtained at 3 wt% MMT. In addition, the halogen-free flame resistance grade of EVA containing 3 wt% OMMT and 47 wt% ATH revealed the best elongation and fire resistance (LOI = 28). The tensile and flame resistance properties of the nanocomposites were also significantly improved.