With the onset of the warm and cold season, the temperature inside the building goes far away from the human comfort condition, so an optimal energy source is needed to reach the desired temperature. Naturally, the earth is a massive source of heat that is easily accessible under the buildings, yards, and squares of the city; therefore, the geothermal heat exchangers can be used in depth of the ground to drain the heat from the earth. In optimizing heat transfer equipment to achieve higher energy efficiency, the focus is on reducing the size of the heat exchanger on the one hand and increasing the intensity of heat transfer area on the other hand. Helical heat exchangers are compact and can be used in geothermal systems to receive energy. In this paper, a conical coil tube geothermal heat exchanger is simulated according to the environmental conditions of Tehran, the capital city of Iran, and the potential of using such system is examined by changing the geometric parameters and operational requirements. The coil diameter, coil pitch and the angle of conic of the helical tube are varied for the heat exchanger busied at a depth of 3m from the earth surface. The soil temperature profile is obtained from the air temperature data Synoptic Meteorological Station. Design parameters are coil diameter, coil pitch, cone angle, length to width ratio of the fin, number of fins and Reynolds number. Taguchi algorithm is applied to find the best geometric parameters. Finally, different volume fractions of Al2O3 nanoparticles are added to the optimized geometry to enhance the heat transfer rate. Results indicate that the optimized geometry considering thermal- hydraulic performance in the range of this study is theta = 0 degrees, = 1000 mm, P-c = 60 mm, Re = 3000, R = land Fn = 6 for cone angle, coil diameter, coil pitch, Reynolds number, length to width ratio of fin and number of fins respectively. Changing the objective to increase heat transfer (Nitwit number) the optimized values alter to theta = 3.14 degrees, D-c = 1000 mm, P-c = 80 mm, Re = 4993, R = 1.5 and Fn = 6. Heat flux on the surface of the tube increases to 18% for 0.5% volume fraction of nanoparticles.