Multi-criteria evaluation methods are often used for sustainability assessment. One such method is the Multi-criteria Spider-gram Cumulative Surface Area (MCSA score) recently used for this purpose (Nzila et al., Multi-criteria sustainability assessment of biogas production in Kenya. Applied Energy 93:496-506, 2012). This paper presents results illustrating that the MCSA method results and rankings might be biased by the arbitrary order of the criteria. This paper also addresses the way the comparison of two alternatives can be biased by the presence of a third (possibly irrelevant) alternative. Such dependence bias is due to the use of an internal normalization operation, a problem shared by some multi-criteria analysis methods. The paper concludes with a few suggestions to avoid such biases. (C) 2013 Elsevier Ltd. All rights reserved.