Fractal analysis of randomly rough surface is an interesting tool to establish relationships between surface geometry and properties. Nonetheless, the observation that different methods to determine the fractal dimension D yield different results has raised questions about its physical meaning. This work shows that such variations are caused by the mathematical details of the methods used, particularly by the effect of vertical scaling. For the triangular prism method (TPM), applied to fractional Brownian motion, the effect of vertical scaling on the numerical estimation of D can be addressed through analytic calculations. The analytic approach was compared to simulations of surface topography obtained by the random midpoint algorithm (RMA) using TPM, box count method (BCM), differential box count (DBC) and detrended fluctuation analysis (DFA). The effect of scaling for TPM is considerable and coincides with the mathematical predictions. BCM and DBC show no effect of scaling but provide poor estimates at high D. A small effect was found for DFA. It is concluded that TPM provides a precise estimate of D which is independent of vertical scaling for infinite image resolution. At finite resolutions, the estimation error on D can be minimised by choosing an optimal vertical scaling factor. (C) 2017 Elsevier B.V. All rights reserved.