Natural selection drives populations toward higher fitness, but second-order selection for adaptability and mutational robustness can also influence evolution. In many microbial systems, diminishing-returns epistasis contributes to a tendency for more-fit genotypes to be less adaptable, but no analogous patterns for robustness are known. To understand how robustness varies across genotypes, we measure the fitness effects of hundreds of individual insertion mutations in a panel of yeast strains. We find that more-fit strains are less robust: They have distributions of fitness effects with lower mean and higher variance. These differences arise because many mutations have more strongly deleterious effects in faster-growing strains. This negative correlation between fitness and robustness implies that second-order selection for robustness will tend to conflict with first-order selection for fitness.