Selectivity and enhanced sensitivity for surface-enhanced Raman scattering (SERS) substrates are highly desirable attributes for their analytical applications. SERS substrate interaction with target molecules plays a key role in determining the enhancement magnitude and spectral profile of the SERS signal as well as their applications. Herein, we demonstrate a SERS active substrate based on reduced graphene oxide/silver nanoparticles (rGO/Ag NPs) composite material synthesized by a facile wet chemical route. The Ag NPs decorated rGO nanosheet has been achieved by reducing AgNO3 with ethylene glycol in the presence of poly vinyl pyrrolidone (PVP) and NaCl over layered graphene oxide. The addition of PVP stimulates the nano-phase growth as well as uniform distribution of the Ag NPs on rGO nanosheet. The well synthesized rGO/Ag composite was confirmed by using various characterization techniques such as FESEM, HRTEM, XRD, FTIR and Raman. Further, the rGO/Ag NPs composite material has been utilized as a platform for SERS based selective detection of Rhodamine-6G, 4-mercapto benzoic acid, and 2,4-dinitrotoluene analytes. The rGO/Ag NPs substrate demonstrates good homogeneity, stability and nanomolar sensitivity with Raman intensity enhancement of the order of 5 for Rh-6G and 4-MBA whereas 3 for 2,4-DNT. The observed enhancement of SERS signal for all the analytes has been attributes to the cumulative effect of rGO and Ag NPs.