A hybrid ternary system of thermoplastic/clay/thermoset was synthesized to produce a tougher unsaturated polyester without reducing the glass transition temperature or the elastic modulus. The effects of material characteristics and morphology on fracture toughness were explored. Three synthesis variables control these characteristics: the ratio of methyl methacrylate to styrene used as curing agents, clay loading, and the content of a thermoplastic copolymer of styrene and methyl methacrylate. Specific ranges of these variables were chosen to minimize the effect of the characteristics of each phase on fracture toughness in order to evaluate the influence of morphology. The thermoplastic component contains the vast majority of the silicate layers and forms a second phase, dispersed throughout the continuous thermoset-rich phase. These particles can perturb the propagating crack front causing crack deflection and/or crack pinning where the latter mechanism provides a large increase in fracture toughness of unsaturated polyester. Clay loading is the most important factor on the fracture mechanism by affecting the impenetrability of the thermoplastic-rich particles and the morphology. A non-monotonic correlation between fracture toughness and the ratio of thermoplastic-rich particle radius to interparticle distance is developed allowing for the design of hybrid nanocomposites with desired toughness. (C) 2015 Elsevier Ltd. All rights reserved.