Using the semi-tensor product of matrices, this paper investigates the maximum (weight) stable set and vertex coloring problems of graphs with application to the group consensus of multi-agent systems, and presents a number of new results and algorithms. Firstly, by defining a characteristic logical vector and using the matrix expression of logical functions, an algebraic description is obtained for the internally stable set problem, based on which a new algorithm to find all the internally stable sets is established for any graph. Secondly, the maximum (weight) stable set problem is considered, and a necessary and sufficient condition is presented, by which an algorithm to find all the maximum (weight) stable sets is obtained. Thirdly, the vertex coloring problem is studied by using the semi-tensor product method, and two necessary and sufficient conditions are proposed for the colorability, based on which a new algorithm to find all the k-coloring schemes and minimum coloring partitions is put forward. Finally, the obtained results are applied to multi-agent systems, and a new protocol design procedure is presented for the group consensus problem. The study of illustrative examples shows that the results/algorithms presented in this paper are very effective. (C) 2012 Elsevier Ltd. All rights reserved.