It is common practice to adjust the maximum step size during Monte Carlo simulations to achieve a target fraction of accepted trial moves. Although simulation experts are well aware that this procedure is technically incorrect, the continuing publication of studies that employ it indicates that the pitfalls are not more widely known, or at least are assumed to be negligible. In this pedagogical Letter, it is shown that dynamic step size adjustment may lead to unwanted correlations and statistical undersampling of flat regions of the potential energy surface, thereby violating balance and producing systematic errors. A variety of examples highlight the conditions where the problem is likely to be worst.