Special Session 126: Machine Learning and New Framework for Solving Partial Differential Equations

Deep Neural Networks with Rectified Power Units: Efficient Training and Applications in Partial Differential Equations
Haijun Yu
Academy of Mathematics and Systems Science, Chinese Academy of Science
Peoples Rep of China
Co-Author(s):    Haijun Yu
Abstract:
Deep neural networks (DNN) equipped with rectified power units (RePU) have demonstrated superior approximation capabilities compared to those utilizing rectified linear units (ReLU), as highlighted by B. Li, S. Tang, and H. Yu [Commun. Comput. Phys., 2020, 27(2): 379-411]. These units are particularly advantageous for machine learning tasks requiring high-order continuity in the functions represented by neural networks. Despite their theoretical benefits, however, the practical application of RePU DNNs has been limited due to challenges such as gradient vanishing and explosion during training. In this talk, we explore various strategies aimed at facilitating the training of RePU DNNs. Our primary focus lies on the numerical solutions of partial differential equations. We demonstrate that, with appropriate training techniques, RePU DNNs can achieve better results than standard DNNs employing other commonly used activation functions, and do so with a faster training rate.