Hyperbranched flame retardant containing P/N/Si/B and rigid-flexible structure (PTDOB) was synthesized and used to prepare high-performance epoxy resin with simultaneous improvement in both fire-safety and mechanical properties. The optimal flame-retardant epoxy thermosets show a self-extinguishing behavior, along with a 47.1% decrease in peak heat release rate, 17.3% decrease in total smoke production compared with neat epoxy resin. More importantly, the obtained flame-retardant epoxy thermosets exhibit superior mechanical properties with 55.4% and 37.5% increase in impact strength and flexural strength, respectively, indicating the incorporation of PTDOB has a good effect on flame retardancy, as well as on mechanical properties simultaneously. Flame retardancy mechanisms of both condensed-phase retardance with a barrier char and gas-phase retardance with incombustible gases provided from the hyperbranching architecture of PTDOB have been well demonstrated. The study guarantees PTDOB to be a promising additive for the development of high-performance epoxy resin with attractive potential applications.