In this technical note, hierarchical games are investigated for multi-agent systems involving a leader and a large number of followers with infinite horizon tracking-type costs. By jointly analyzing dynamic equations and index functions of all agents, a set of centralized Stackelberg equilibrium strategies is given. Then, by using the mean field approach and the brute force method, a set of distributed strategies is designed. Under mild conditions, it is shown that the closed-loop system is uniformly stable and the set of distributed strategies is an epsilon-Stackelberg equilibrium.