Hydrothermal characteristics of the water-Al2O3 nanofluid are numerically evaluated in shell-and-tube heat exchanger equipped with helical baffles using the two-phase mixture model. Heat transfer and pressure drop increase by increasing nanopartide concentration and baffle overlapping, and decreasing helix angle. At smaller helix angles, changing the overlapping is more effective on the convective heat transfer coefficient and the pressure drop. Neural network is used for modeling, and based on the test data, the model predicts the convective heat transfer coefficient and the pressure drop with MRE (Mean Relative Error) values of about 0.089% and 0.65%, respectively. In order to obtain conditions of effective parameters which cause maximum heat transfer along with minimum pressure drop, optimization is performed on the neural network model using both two-objective and single-objective approaches. 15 optimal states obtain from two-objective optimization. The results obtained from single-objective optimization indicate that even when a low pressure drop is significantly important for designer, nanofluids with high concentrations can be employed. Meanwhile, when both high heat transfer and low pressure drop are important, a small helix angle can be used. In addition, using large overlapping is recommended only when the heat transfer enhancement is considerably more important than the reduction of the pressure drop. (C) 2015 Elsevier Ltd. All rights reserved.