The purpose of thermal protection system (TPS) is to maintain the structural temperature of the hypersonic aircraft within acceptable limits due to intense aerodynamic heating during reentering earth's atmosphere. In the context of hypersonic aircraft design, a major issue is to obtain the optimal thickness of the insulation layers for TPS. In this study, an idea combining a transient heat transfer model and an efficient optimization model is introduced for multilayer insulation of TPS. The TPS geometric dimensions in the thickness direction are particularly considered as the design variables and the objective function is the total mass of the thermal protection structure with the limitation of the extreme temperatures of the hypersonic aircraft structure. In order to decrease the computational complexity, the globally convergent method of moving asymptotes method is specially used to search the optimal solution. The temperature profiles at various surfaces along the thickness direction are presented and analyzed. It is shown that the usage of multilayer insulation materials for the TPS can save more than 17% weight compared with a single-layer TPS. The detailed analysis and comparison indicate the advantages of the presented optimization model.