Designing ligand molecules that bind with high affinity and specificity to a target molecule (or a family of related targets) is a fundamental goal of molecular biophysical research. While it is generally recognized that electrostatic interactions can contribute to binding specificity, it is unclear whether the inclusion of interactions that result in tight binding also necessarily leads to highly specific binding. Here we make use of recently developed charge-optimization techniques to explore the affinity-specificity relation in the context of electrostatic interactions. Using model problems we find that affinity-optimized electrostatic interactions do not necessarily create specificity. Furthermore, we develop several rigorous methods that indicate how best to perturb affinity-optimized ligand-charge distributions to increase specificity with minimal sacrifice in affinity for the target or target set. We provide a theoretical framework for improving specificity against any number of known receptors and/or binding modes as well as against uncharacterized receptors.