We consider networks of passive systems with time delays in the interconnections, and present a stability analysis technique with the help of the integral quadratic constraint (IQC) framework. Unlike the classical passivity approach that fails to characterize delay robustness, and the small-gain approach that conservatively accounts for arbitrarily large delays, the new technique gives sharp stability estimates that depend on the duration of delay. Since the effect of delay depends on its duration relative to the time scales of the system, we make use of a "roll-off" IQC that captures magnitude roll-off at high frequencies, thus, providing the critical time-scale information. We then combine this roll-off IQC with an output strict passivity IQC that incorporates gain and phase information, and demonstrate the benefit of this combined IQC approach on a cyclic interconnection structure with delay. Finally, we develop a technique to verify these IQCs for classes of nonlinear state-space models and present an example from Internet congestion control.