Epoxy-clay ternary nanocomposites were processed by melt blending of hydroxyl terminated poly (ether ether ketone) oligomer with pendant methyl groups (PEEKMOH) with diglycidyl ether of bisphenol A (DGEBA) epoxy resin along with organically modified montmorillonite (OMC-OH) followed by curing with 4,4'-diamino diphenyl sulfone. Small angle X-ray diffraction and transmission electron microscopy revealed an intercalated morphology. Tensile, flexural, storage, and loss moduli were increased whereas the tensile, flexural, and impact strength and glass transition temperature were decreased with increase in clay content for the PEEKMOH toughened epoxy system. Fracture toughness and percentage strain were increased by 66% and 45% respectively whereas the coefficient of the thermal expansion was decreased by 27% with the incorporation of 1 phr OMC-OH to the PEEKMOH toughened epoxy system compared with neat epoxy. The scanning electron microscope pictures of fracture and tensile failed surfaces revealed crack path deflection and ductile fracture with the incorporation of OMC-OH confirming the improvement in toughness. The domain size and the distance between the domains of thermoplastic phase were decreased with the addition of nanoclay into the epoxy matrix indicating the restriction of the growth mechanism by nucleation during phase separation. A marginal increase in thermal stability was observed with increase in clay content. POLYM. ENG. SCI., 49:756-767, 2009. (c) 2009 Society of Plastics Engineers