Experiments have repeatedly observed both thermodynamic and dynamic anomalies in aqueous binary mixtures, surprisingly at low solute concentration. Examples of such binary mixtures include water-DMSO, water-ethanol, water-tertiary butyl alcohol (TBA), and water-dioxane, to name a few. The anomalies have often been attributed to the onset of a structural transition, whose nature, however, has been left rather unclear. Here we study the origin of such anomalies using large scale computer simulations and theoretical analysis in water-DMSO binary mixture. At very low DMSO concentration (below 10%), small aggregates of DMSO are solvated by water through the formation of DMSO-(H2O)(2) moieties. As the concentration is increased beyond 10-12% of DMSO, spanning clusters comprising the same moieties appear in the system. Those clusters are formed and stabilized not only through H-bonding but also through the association of CH3 groups of DMSO. We attribute the experimentally observed anomalies to a continuum percolation-like transition at DMSO concentration X-DMSO approximate to 12-15%. The largest cluster size of CH3-CH3 aggregation clearly indicates the formation of such percolating clusters. As a result, a significant slowing down is observed in the decay of associated rotational auto time correlation functions (of the S = O bond vector of DMSO and O-H bond vector of water). Markedly unusual behavior in the mean square fluctuation of total dipole moment again suggests a structural transition around the same concentration range. Furthermore, we map our findings to an interacting lattice model which substantiates the continuum percolation model as the reason for low concentration anomalies in binary mixtures where the solutes involved have both hydrophilic and hydrophobic moieties.