For many technological applications of ferrofluids, the magnetorheological properties require being precisely controlled. We study the effect of hydrophilic silica on the magnetorheology of an oil-based ferrofluid containing Fe3O4 nanoparticles of size similar to 10 nm. We observe that the presence of silica nanoparticles lowers the yield stresses, viscoelastic moduli, and shear thinning behavior of the ferrofluid because of the weakening of dipolar interactions, which was evident from the observed lower yield stresses exponent (<2). The ferrofluid containing silica exhibits a dominant elastic behavior, a reduced hysteresis during the forward and reverse magnetic field sweeps, and a longer linear viscoelastic regime under nonlinear deformation. The Mason number plots at low shear rates and magnetic fields show deviations from the master curve in the presence of silica. The magnetic field induced microstructures, visualized using opto-magnetorheometer, showed columnar aggregate structures along the field directions, which are reoriented along the shear flow direction at high shear rates. The image analysis shows that the average thickness of the columnar aggregates in pure ferrofluid is much larger than that of the mixed system, which suggests that the intervening silica matrix hampers the zippering transition of columns at higher magnetic field and shear rates. Our results suggest that optimization of rheological properties of ferrofluids is possible by carefully adding suitable silica nanoparticles, which may find practical applications such as dynamic seals, heat transfer, sensors, and opto-fluidic devices, etc.