A systematic investigation of the factors influencing the notable enhancement of the mechanical and thermal properties of nanodiamonds (NDs)-reinforced crosslinked high density polyethylene (PEX) is presented in this work. The effects of crystal structure and molecular conformation as well as filler dispersion and adhesion with the matrix were found to govern the mechanical properties of the final composites. A considerable increase in the strength, toughness, and elastic modulus of the materials was found for the composites with filler content below 1 wt %. For higher NDs concentrations, the properties degraded. When filler concentration does not exceed 1 wt %, enhanced adhesion with the matrix is achieved, allowing a more successful load transfer between the filler and the matrix, thus enabling an effective reinforcement of the composites. The higher degree of crystallinity along with larger crystal size are also positively influencing the mechanical properties of PEX. Higher filler concentrations, on the other hand, lead to the formation of larger aggregates, which lead to lower adhesion with the matrix, while they also constitute stress concentrators and therefore reduce the positive reinforcement of the matrix. The thermal conductivity of the composites was also found to be significantly increased for low-filler concentrations. This enhancement was less significant for higher NDs concentrations. It is concluded that this reinforcement is due to the heat capacity increase that NDs incorporation causes in PEX. Additionally, a thermal stability enhancement was found for the composite with minimum filler content.