A novel core-shell-structured carboxylated-styrene butadiene rubber (XSBR)-functionalized nanodiamond (ND-XSBR) was synthesized and characterized. Epoxy (EP) nanocomposites toughened by pristine ND and ND-XSBR were investigated and compared. The ND-XSBR-reinforced nanocomposite exhibited mechanical properties superior to those of the one filled by pristine ND. At a low-filler loading, the ND-XSBR exhibited an impressive toughening effect. The maximum flexural strength was shown when the filler loading was as low as 0.1 wt% for the EP/ND-XSBR nanocomposite. Furthermore, enhanced fracture toughness and fracture energy were shown by surface functionalization, representing enhanced compatibility between the ND-XSBR and EP matrix. The glass transition temperature (T-g) and storage modulus of the nanocomposites were studied, and the EP/ND-XSBR0.1 nanocomposite exhibited the highest T-g owing to the stronger interfacial interaction. The EP/ND-XSBR0.2 exhibited higher storage modulus and T-g than the EP/ND0.2, because the higher interfacial interaction can restrict the molecular mobility of the EP by the functionalized ND-XSBR. (c) 2017 Wiley Periodicals, Inc. J. Polym. Sci., Part B: Polym. Phys. 2017, 55, 1890-1898