A Smoothing Algorithm with Constant Learning Rate for Training Two Kinds of Fuzzy Neural Networks and Its Convergence

Abstract

In this paper, a smoothing algorithm with constant learning rate is presented for training two kinds of fuzzy neural networks (FNNs): max-product and max-min FNNs. Some weak and strong convergence results for the algorithm are provided with the error function monotonically decreasing, its gradient going to zero, and weight sequence tending to a fixed value during the iteration. Furthermore, conditions for the constant learning rate are specified to guarantee the convergence. Finally, three numerical examples are given to illustrate the feasibility and efficiency of the algorithm and to support the theoretical findings.

Appendix

First, let us make an estimation of matrix norm as preparation for the proof of our convergence theorem.

According to (12) and (13), we know that all second partial derivatives of $\widetilde{E}(w)$ exist in $R^n$ for any $t>0$ and its second partial derivative $\frac{\partial ^2 \widetilde{E}(w) }{\partial w_i\partial w_j}$ is given by

$$\begin{aligned} \frac{\partial ^2 \widetilde{E}(w) }{\partial w_i\partial w_j}= & {} \sum \limits _{s=1}^S\left\{ \lambda _j^s(w,t)\lambda _i^s(w,t)x^s_jx^s_i+\frac{1}{t} \left[ T^s-\widetilde{g}^s(w,t)\right] \lambda _j^s(w,t)\lambda _i^s(w,t)x^s_jx^s_i\right\} \nonumber \\= & {} \sum \limits _{s=1}^S \lambda _j^s(w,t)\lambda _i^s(w,t)x^s_jx^s_i\left[ 1+\frac{1}{t} (T^s-\widetilde{g}^s(w,t))\right] \end{aligned}$$

(22)

where $\lambda _i^s(w,t)=\frac{\exp \left( x^s_iw_i/t\right) }{\sum \nolimits _{j=1}^n\exp \left( x^s_jw_j/t\right) }$, $\widetilde{g}^s(w,t)=t\ln \sum \nolimits _{i=1}^n \exp \left( \frac{x_i^sw_i}{t}\right)$.

Then the Hessian matrix H(w) of $\widetilde{E}(w)$ is a square n-by-n matrix and defined as

$$\begin{aligned} H(w)=\left( \frac{\partial ^2 \widetilde{E}(w) }{\partial w_i\partial w_j}\right) _{n\times n} \end{aligned}$$

(23)

It is easy to verify that all second partial derivatives of $\widetilde{E}(w)$ are continuous and the following equation holds for all $i,j=1,2,\ldots ,n$

$$\begin{aligned} \frac{\partial ^2 \widetilde{E}(w) }{\partial w_i\partial w_j}=\frac{\partial ^2 \widetilde{E}(w) }{\partial w_j\partial w_i} \end{aligned}$$

Therefore, the Hessian matrix H(w) is a real symmetric matrix.

Suppose that $\lambda _1(w),\lambda _2(w),\ldots ,\lambda _n(w)$ are n real eigenvalues of matrix H(w) and the norm $\Vert \bullet \Vert _2$ defined for a matrix A by

$$\begin{aligned} \Vert {A}\Vert _2=\max \{\sqrt{\lambda }:\ \lambda \ is\ an\ eigenvalue\ of\ A^TA\} \end{aligned}$$

is taken as the matrix norm considered in the following discussion. We can obtain an estimation of the norm of matrix H(w) as shown in Lemma 2.

Lemma 2

Let H(w) be the Hessian matrix of $\widetilde{E}(w)$ and defined in (22) and (23). There exits a constant $C_1$ such that for all $w\in D$

$$\begin{aligned} \Vert H(w)\Vert _2\leqslant C_1 \end{aligned}$$

where D is a bounded set.

Proof

Since matrix H(w) is a real symmetric matrix and $\lambda _1(w),\lambda _2(w),\ldots ,\lambda _n(w)$ are n real eigenvalues of it, there exits an unitary matrix Q such that

$$\begin{aligned} H(w)=Q^Tdiag\left( \lambda _1(w),\lambda _2(w),\ldots ,\lambda _n(w)\right) Q \end{aligned}$$

Then

$$\begin{aligned} \left( H(w)\right) ^TH(w)=Q^Tdiag\left( \lambda _1^2(w),\lambda _2^2(w),\ldots ,\lambda _n^2(w)\right) Q \end{aligned}$$

It is easy to know that all eigenvalues of $\left( H(w)\right) ^TH(w)$ are

$$\begin{aligned} \lambda _1^2(w),\lambda _2^2(w),\ldots ,\lambda _n^2(w) \end{aligned}$$

Thus, we have

$$\begin{aligned} \Vert H(w)\Vert _2=\max \{|\lambda _i(w)|:i=1,2,\ldots ,n\} \end{aligned}$$

It follows from (22), (23) and the set D is bounded that $\lambda _i(w)$ is bounded for all $i=1,2,\ldots ,n$ and all $w\in D$, namely there exits a constant $C_1$ such that

$$\begin{aligned} |\lambda _i(w)|\leqslant C_1 \end{aligned}$$

for all $i=1,2,\ldots ,n$ and all $w\in D$. Therefore, we have

$$\begin{aligned} \Vert H(w)\Vert _2\leqslant C_1 \end{aligned}$$

This completes the proof of Lemma 2. $\square$

The following lemma is also crucial for the proof of our convergence theorem. This result is almost the same as Lemma 5.3 in [32] and the detail of its proof is omitted.

Lemma 3

Suppose that $F : \mathbb {R}^p\rightarrow \mathbb {R}^q,\ p\ge 1,\ q\ge 1$ is continuous on a bounded closed region $\mathbf{D}\subset \mathbb {R}^n$, and that $\mathbf{D_0}=\{\mathbf {z}\in \mathbf{D}:\ F(\mathbf {z})=0\}$. The projection of $\mathbf{D_0}$ on each coordinate axis does not contain any interior point. If a sequence $\{\mathbf {z}^k\}\subset \mathbf{D}$ satisfies

$$\begin{aligned} \lim \limits _{k\rightarrow \infty } \Vert F(\mathbf {z}^k)\Vert =0,\ \ \lim \limits _{k\rightarrow \infty }\Vert \mathbf {z}^{k+1}-\mathbf {z}^k\Vert =0. \end{aligned}$$

then there exists a unique $\mathbf {z}^*\in \mathbf{D_0}$ such that $\lim \nolimits _{k\rightarrow \infty }\mathbf {z}^k=\mathbf {z}^*$.

Now we are ready to prove the main theorems in terms of the above two lemmas.

Proof to Theorem 1

The proof is divided into four parts, dealing with (18)–(21) respectively.

Proof to (18). Expanding $\widetilde{E}(w^{k+1})$ with Taylor formula, we have for all $k=0,1,2,\ldots$ that

$$\begin{aligned} \widetilde{E}(w^{k+1})-\widetilde{E}(w^{k})= & {} (\nabla \widetilde{E}(w^{k}))^T\Delta w^{k}+\frac{1}{2}(\Delta w^{k})^TH(\xi )(\Delta w^{k})\\\leqslant & {} -\eta \Vert \nabla \widetilde{E}(w^{k}))\Vert ^2+\frac{\Vert H(\xi )\Vert }{2}\eta ^2 \Vert \nabla \widetilde{E}(w^{k}))\Vert ^2\\= & {} -\left( \eta -\frac{\Vert H(\xi )\Vert }{2}\eta ^2\right) \Vert \nabla \widetilde{E}(w^{k}))\Vert ^2 \end{aligned}$$

where $\xi$ lies between $w^k$ and $w^{k+1}$. Write $\alpha =\eta -\frac{\Vert H(\xi )\Vert }{2}\eta ^2$. Then

$$\begin{aligned} \widetilde{E}(w^{k+1})\leqslant \widetilde{E}(w^{k})-\alpha \Vert \nabla \widetilde{E}(w^{k}))\Vert ^2 \end{aligned}$$

(24)

We require the learning rate $\eta$ to satisfy

$$\begin{aligned} 0<\eta \leqslant \frac{1}{C} \end{aligned}$$

(25)

where $C=\frac{C_1}{2}$. By virtue of Lemma 2, we have

$$\begin{aligned} \alpha =\eta -\frac{\Vert H(\xi )\Vert }{2}\eta ^2=\eta \left( 1-\frac{\Vert H(\xi )\Vert }{2}\eta \right) \geqslant 0 \end{aligned}$$

This together with (24) leads to

$$\begin{aligned} \widetilde{E}(w^{k+1})\leqslant \widetilde{E}(w^k),\quad k=0,1,2,\ldots \end{aligned}$$

(26)

This completes the proof of (18).

Proof to (19). By the definition of $\widetilde{E}(w)$ in (12), it is easy to see that $\widetilde{E}(w^k)\ge 0$ for $\ k=0,1,2,\ldots$. Combining with (26), we then conclude by monotone convergence theorem that

$$\begin{aligned} \lim \limits _{k\rightarrow \infty }\widetilde{E}(w^k)=\widetilde{E}^* \end{aligned}$$

(27)

where $\widetilde{E}^*=\inf _{w\in \Omega }\widetilde{E}(w)$. This proves (19).

Proof to (20). According to (24), it is easy to get

$$\begin{aligned} \widetilde{E}(w^{k+1})\leqslant \widetilde{E}(w^{k})-\alpha \Vert \nabla \widetilde{E}(w^{k}))\Vert ^2 \leqslant \cdots \leqslant \widetilde{E}(w^{0})-\alpha \sum \limits _{t=0}^k\Vert \nabla \widetilde{E}(w^{t}))\Vert ^2 \end{aligned}$$

Since $\widetilde{E}(w^{k+1})\geqslant 0$, we have

$$\begin{aligned} \alpha \sum \limits _{t=0}^k\Vert \nabla \widetilde{E}(w^{t}))\Vert ^2\leqslant \widetilde{E}(w^{0}) \end{aligned}$$

Letting $k\rightarrow \infty$ results in

$$\begin{aligned} \sum \limits _{t=0}^\infty \Vert \nabla \widetilde{E}(w^{t}))\Vert ^2\leqslant \frac{1}{\alpha }\widetilde{E}(w^{0})<\infty \end{aligned}$$

This immediately gives

$$\begin{aligned} \lim \limits _{k\rightarrow \infty }\Vert \nabla \widetilde{E}(w^k)\Vert =0 \end{aligned}$$

(28)

(20) is proved.

Proof to (21). By virtue of (14) and (28), we can get that

$$\begin{aligned} \lim \limits _{k\rightarrow \infty }\Vert w^{k+1}-w^{k}\Vert = \lim \limits _{k\rightarrow \infty }\Vert -\eta \nabla \widetilde{E}(w^k)\Vert =0. \end{aligned}$$

(29)

Note that the error function $\widetilde{E}(w)$ defined in (12) is continuous and differentiable. According to (13) and (15), it is easy to verify that $\nabla \widetilde{E}(w)$ is continuous on $w\in \mathbb {R}^n$. As (28), (29) and Assumption (A3) are valid, it follows immediately from Lemma 3 that (21) holds, that is, there exists a unique fixed point $w^*\in \Omega$ such that

$$\begin{aligned} \lim \limits _{k\rightarrow \infty }w^k=w^*. \end{aligned}$$

And this completes the proof of Theorem 1. $\square$

Proof to Theorem 2

Using the Taylor expansion, we can also get for all $k=0,1,2,\ldots$ that

$$\begin{aligned} \overline{E}(w^{k+1})-\overline{E}(w^{k})=(\nabla \overline{E}(w^{k}))^T\Delta w^{k}+\frac{1}{2} (\Delta w^{k})^TH(\xi )(\Delta w^{k}) \end{aligned}$$

where $\xi$ lies between $w^k$ and $w^{k+1}$. Since Hessian matrix H(w) of $\overline{E}(w)$ and $\widetilde{E}(w)$ have the same mathematical properties, we can follow the proof process of Theorem 1 and get the same results as in Theorem 1 for the error function $\overline{E}(w)$ and the sequence $\{w^k\}$ generated by SAMM. This completes the proof of Theorem 2. $\square$