Polymer composites play a significant role in developing flame retardants to prevent fire accidents. The current work aims at investigating the flame retardancy of vinyl ester resins (VER) reinforced with nanotitania (nano-TiO2) nanofiller. The surface functionality of nano-TiO2 was modified by adding Si and N2 to improve its flame retardancy. The chemical structure and thermal stability of nanocomposites were studied using Fourier-transform infrared spectroscopy (FTIR) and Thermogravimetric analysis (TGA). Peaks of Si and N2 in the modified TiO2-VER nanocomposite and weight loss of the modified composite confirmed the addition of Si and N2. The tensile strength results have shown that modified TiO2-VER nanocomposites didn’t make any significant impact on the tensile strength in comparison with pure VER. The flammability and thermal stability behaviors of these nanocomposites are evaluated using microscale combustion calorimetry (MCC). At high percent loading of nanofiller, the normalized heat release capacity (HRC) of modified TiO2-VER nanocomposites was decreased by 27.7%, whereas the HRC of unmodified TiO2-VER nanocomposites was only reduced by 9.8%. Also, the normalized total heat release of modified nanofiller based PNC was found to be 21.4%, whereas the unmodified PNC was 12.4%.