This note investigates the controllability issues of multi-agent systems, where each node contains generic linear dynamics. First, a neighbor-based control protocol is proposed, under which it is shown that the controllability of a multi-agent system is solely decided by its communication topology structure. We then further consider the effects of communication topology on the controllability from a graph theory point of view by explicitly constructing topologies. The construction exhibits a partition of the designed graph with identified leaders under which the system is shown to be uncontrollable.