Octa(maleimidophenyl)silsesquioxane (OMPS) was synthesized, characterized, and employed to modify the BT resin which composed of 4,4'-bismaleimidodiphenylmethane (BMI) and 2,2'-bis(4-cyanatophenyl)propane (BCE). The curing reaction between OMPS and BT resin was first investigated. It was found that OMPS accelerate the curing reaction of BCE, and the onset temperature of the cyclotrimerization was reduced up to 95.5 degrees C (by DSC). As demonstrated by DSC and FTIR, there was no evidence that indicated the coreaction between maleimide and cyanate ester. 2,2'-diallyl bisphenol A (DBA) and diglycidyl ether of bisphenol A (E-51) (Wuxi Resin Factory, Jiangsu Province, China) were also used to enhance the toughness of BT resin, and the formulated BTA (containing DBA) and BTE (containing E-51) resins were obtained. The thermal properties of BT, BTA, and BTE resins incorporated with OMPS were then investigated. The results of DMA and TG showed that the BT, BTA, and BTE resins containing 1 wt% of OMPS exhibit enhanced thermal properties in comparison with their pristine resins respectively, while more contents of OMPS may impair the thermal properties of the polymer matrix, though the effect of OMPS was slight. Finally, the dielectric constant of these hybrid materials were detected, and their dielectric constant were distinctly reduced by the incorporation of OMPS, while overmuch contents of OMPS were disadvantageous for dielectric constant because of the aggregation of OMPS. (c) 2008 Wiley Periodicals, Inc.