A huge effort is underway to develop semiconductor nanostructures as low-noise qubits. A key source of dephasing for electron spin qubit in GaAs and in naturally occurring Si is the nuclear spin bath. The electron spin is coupled to nuclear spin by the hyperfine interaction. The same interaction also couples two remote nuclear spins via a coupling to the delocalized electron. It has been suggested that this interaction limits both electron and nuclear coherence, but experimental proof is lacking. We show that the nuclear spin decoherence time decreases by two orders magnitude on occupying an empty quantum dot with a single electron, recovering to its original value for two In the case of one electron, agreement with a model calculation verifies the hypothesis of an electron-mediated spin-nuclear spin coupling. The results establish a framework to understand the main features of this complex in semiconductor nanostructures.