Noise on Gradient Systems with Forgetting

Abstract

In this paper, we study the effect of noise on a gradient system with forgetting. The noise include multiplicative noise, additive noise and chaotic noise. For multiplicative or additive noise, the noise is a mean zero Gaussian noise. It is added to the state vector of the system. For chaotic noise, it is added to the gradient vector. Let ${\mathbf x}$ be the state vector of a system, $S_b$ be the variance of the Gaussian noise, $\kappa '$ is average noise level of the chaotic noise, $\lambda$ is a positive constant, $V({\mathbf x})$ be the energy function of the original gradient system, $V_{\otimes }({\mathbf x})$, $V_{\oplus }({\mathbf x})$ and $V_{\odot }({\mathbf x})$ be the energy functions of the gradient systems, if multiplicative, additive and chaotic noises are introduced. Suppose $V({\mathbf x}) = F({\mathbf x}) + \lambda \Vert {\mathbf x}\Vert ^2_2$. It is shown that $V_{\otimes }({\mathbf x}) = V({\mathbf x}) + (S_b/2) \sum _{j=1}^n (\partial ^2 F({\mathbf x})/\partial x_j^2) x_j^2 - S_b \sum _{j=1}^n \int x_j (\partial ^2 F({\mathbf x})/\partial x_j^2) dx_j$, $V_{\oplus }({\mathbf x}) = V({\mathbf x}) + (S_b/2) \sum _{j=1}^n \partial ^2 F({\mathbf x})/\partial x_j^2$, and $V_{\odot }({\mathbf x}) = V({\mathbf x}) + \kappa '\sum _{i=1}^n x_i$. The first two results imply that multiplicative or additive noise has no effect on the system if $F({\mathbf x})$ is quadratic. While the third result implies that adding chaotic noise can have no effect on the system if $\kappa '$ is zero. As many learning algorithms are developed based on the method of gradient descent, these results can be applied in analyzing the effect of noise on those algorithms.