This paper focuses on controllability and observability of multi-agent systems with heterogeneous and switching topologies, where the first- and the second-order information interaction topologies are different and switching. First, based on the controllable state set, a controllability criterion is obtained in terms of the controllability matrix corresponding to the switching sequence. Next, by virtue of the subspace sequence, two necessary and sufficient algebraic conditions are established for controllability in terms of the system matrices corresponding to all the possible topologies. Furthermore, controllability is considered from the graphic perspective. It is proved that the system is controllable if the union graph of all the possible topologies is controllable. With respect to observability, two sufficient and necessary conditions are derived by taking advantage of the system matrices and the corresponding invariant subspace, respectively. Finally, some simulation examples are worked out to illustrate the theoretical results.