Specific ion effects can influence many processes in aqueous solutions: protein folding, enzyme activity, self-assembly, and interface stabilization. Ionic amphiphiles are known to stabilize the oil/water interface, presumably by dipping their hydrophobic tails into the oil phase while sticking their hydrophilic head groups in water. However, we find that anionic and cationic amphiphiles adopt strikingly different structures at liquid hydrophobic/water interfaces, linked to the different specific interactions between water and the amphiphile head groups, both at the interface and in the bulk. Vibrational sum frequency scattering measurements show that dodecylsulfate (DS-) ions do not detectably perturb the oil phase while dodecyltrimethylammonium (DTA(+)) ions do. Raman solvation shell spectroscopy and second harmonic scattering (SHS) show that the respective hydration-shells and the interfacial water structure are also very different. Our work suggests that specific interactions with water play a key role in driving the anionic head group toward the water phase and the cationic head group toward the oil phase, thus also implying a quite different surface stabilization mechanism.