Inspired by the recent developments in the field of distributed quantum computing, distributed quantum systems are analyzed as networks of quantum systems. This gives rise to the distributed quantum consensus algorithms. Focus of this paper is on optimizing the convergence rate of the continuous-time quantum consensus algorithm over a quantum network with N qudits. It is shown that the optimal convergence rate is independent of the value of d in qudits. By classifying the induced graphs as the Schreier graphs, they are categorized in terms of the partitions of integer N. The intertwining relation is established between one-level dominant partitions in the Hasse Diagram of integer N. Based on this result, the proof of the Aldous' conjecture is extended to all possible induced graphs, and the original optimization problem is reduced to optimizing algebraic connectivity of the smallest induced graph. Utilizing the generalization of Aldous' conjecture, it is shown that the convergence rates of the algorithm to both the consensus state and the reduced quantum state consensus are the same. By providing the analytical solution to semidefinite programming formulation of the obtained problem, closed-form expressions for the optimal results are provided for a range of topologies.