Herringbone gear transmission system is widely used in high-speed overloaded fields such as ships. Tooth deformation and installation error, which can cause meshing impact, load mutation, and load uneven, will seriously affect dynamic performance of the whole transmission system. Modification technology is the most effective way to achieve damping and noise reduction. In this article, we propose a three-dimensional (3D) modified tooth surface by grinding wheel along axial to the spiral rise and along radial to the parabolic movement based on the grinding principle and a multiobjective ant lion optimizer model for optimizing meshing performance and dynamic characteristics of herringbone gear transmission system and analyze the impact of optimized modifications of tooth profile, axial, and three dimension on meshing performance, loaded transmission error, load distribution coefficient and meshing impact, meshing stiffness, and vibration acceleration by the example. The results show that the 3D modification of optimization can not only eliminate the contact between the tooth side edge and tooth top edge but also eliminate the influence of installation error on contact performance. The root mean square values of the relative comprehensive vibration acceleration of tooth profile modification, axial modification, and 3D modification are reduced by 30.11%, 49.24%, and 61.41% compared with the standard, respectively. The 3D modification can greatly reduce tooth vibration, reduce resonance peak, and achieve the goal of noise reduction.