A novel modified-unsaturated polyester resin (M-UPR) with phosphorus-containing pendant groups was successfully prepared by employing a bifunctional acid, 2-[10-(9,10-dihydro-9-oxa-10-phosphaphenantrene-10-oxide-10-yl)]-maleic acid (DOPO-MA) as monomer. Characterization of DOPO-MA and M-UPR was performed using element analysis, H-1-NMR, or FT-IR. The curing of M-UPR was measured by differential scanning calorimetry, and the optimal cure temperature was calculated with DSC curves at different heating rates. The thermal stability was investigated with thermogravimetric analysis and comparative FT-IR analysis for char residue before and after heating. The flame-retardant property was determined by limiting oxygen index measurements and UL94 flammability meter. The viscoelastic property was measured with dynamic mechanical analysis. Owing to the incorporation of the rigid structure of pendant phosphorus groups, both thermal stability and flame retardancy of the resultant M-UPR have been improved.