Pumped storage hydropower system has now become a key support for integration of variable renewable energy. Along with the large-scale development of pumped storage plants all over the world, the importance of operation strategy is rapidly increasing with the fast-growing demand of flexibility. How to direct flexible operation and improve the stability of the system has become a key issue. This paper proposes a novel optimization framework to derive optimal operating policies for pumped storage hydropower system, which is divided into three coordinated stages: nonlinear modeling, strategy optimization and decision making. The real-time accurate equivalent circuit model is proposed in first stage, which can reconcile the conflict between simulation efficiency and accuracy, owing to the novel pump-turbine model and space-time discretization. The search corridor which consists of multiple constraints is defined to improve search efficiency. Reference vector guided evolutionary algorithm is performed to handle non-normalization objectives, multiple constraints and irregular Pareto fronts in second stage. Further, fuzzy analytic hierarchy process is innovatively introduced to select compatible solution under extreme conditions. The originality of the framework is embodied in multiple trade-offs, i.e. trade-off between accuracy and efficiency in the nonlinear model, trade-off between convergence and diversity in the strategy optimization, and trade-off in conflicting objectives of the decision making. Compared with the on-site operation, the maximum water pressure of volute and the vacuum at draft tube can be improved by 5.59% and 9.6%, the rotational speed oscillation also decreases with this framework in load rejection. The proposed framework can specify the optimal policy to enhance the system reliability, which can also serve as the basis for smart operation in various conditions.