Holographic polymer-dispersed liquid crystals (H-PDLCs) filled with layer-structured montmorillonite (MMT) clay were fabricated by wave mixing of two coherent argon-ion laser beams. The effects of MMT doping were investigated by comparing a pristine sodium-type MMT and an organophilic MMT (OMMT) via modification. In order to gain insights into the complex formation of the ternary nematic/polymer/clay nanocomposites, wide-angle X-ray diffraction experiments were carried out on pure liquid crystal, MMT clay, and clay dispersed in the liquid crystal in the mesophase as well as isotropic state. Self-diffraction experiments in the Raman-Nath regime revealed that the first-order diffraction efficiency was enhanced in the H-PDLCs consisting of pristine MMT nanoparticles and yet suppressed by the incorporation of OMMT. Holographic nature of the polymer-matrix nanocomposites was confirmed by optical polarizing microscopy. The results of this study may open up great promise held by such ternary polymer-matrix nanocomposites for photonic applications.