In visual design, harmony refers to the similarity among objects and the components of these objects that appear as if they belong together. Harmony is related to the body, mind, and emotions as we relate to our living space, which means that the harmony of real objects is an important characteristic. Although the appearance harmony in specific materials has been investigated, harmony among different materials has not received adequate attention. In a previous study, we investigated the appearance harmony among various materials by psychophysical experiments. In this study, we correlated the psychophysical evaluations obtained from the experiments and the physical properties of the materials. By analyzing the relationship between psychophysical evaluations and the physical measurements for each material pairs based on cluster analysis, we found that the property of texture represented by the anisotropy histogram and that of reflectance obtained from the data of the bidirectional reflectance distribution function are important indices that can be used to judge the harmony of materials. In particular, the texture properties were more effective in judging the appearance harmony of material pairs in the fixed condition for illumination and viewing angles, and the reflectance properties were superior for the material pairs in the variable condition for illumination and viewing angles.