A genetic algorithm-based neural fuzzy system (GA-NFS) was presented for studying the coagulation process of wastewater treatment in a paper mill. In order to adapt the system to a variety of operating conditions and acquire a more flexible learning ability, the GA-NFS was employed to model the nonlinear relationships between the effluent concentration of pollutants and the chemical dosages, and a hybrid learning algorithm divided into two stages was proposed for parameters learning. During the first learning stage, a genetic algorithm was used to optimize the structure of GA-NFS and the membership function of each fuzzy term due to its capability of parallel and global search. On the basis of an optimized training stage, the back-propagation algorithm (BP algorithm) was chosen to update the parameters of GA-NFS to improve the system precision. The GA-NFS proves to be very effective in modeling coagulation perform and performs better than adaptive-network-based fuzzy inference system (ANFIS). RMSE, MAPE, and R between the predicted and observed values for GA-NFS were only 0.01099, 2.3337, and 0.9375, respectively.