In this article, the controllability issue is addressed for an interconnected system of multiple agents. The network associated with the system is of the leader-follower structure with some agents taking leader role and others being followers interconnected via the neighbour-based rule. Sufficient conditions are derived for the controllability of multi-agent systems with time-delay in state, as well as a graph-based uncontrollability topology structure is revealed. Both single and double integrator dynamics are considered. For switching topology, two algebraic necessary and sufficient conditions are derived for the controllability of multi-agent systems. Several examples are also presented to illustrate how to control the system to shape into the desired configurations.