牛顿法、离散零化神经网络模型求解动态非线性优化问题

不念不忘少年蓝@ 2022-12-26 08:17 244阅读 0赞

## 1. 牛顿法求解动态非线性最优化问题 ##

动态非线性最优化问题的定义：![\\large \\text\{min\} ~ f(x(t), t)][large _text_min_ _ f_x_t_ t]

求解优化问题的牛顿法的迭代公式：

![\\large x\_\{n+1\}=x\_n-\\frac\{\\dot f\}\{\\ddot f\}][large x_n_1_x_n-_frac_dot f_ddot f]

目标函数![f(x(t), t)][f_x_t_ t]的梯度：

![g(x(t), t)=\\frac\{\\partial f(x(t), t)\}\{\\partial x(t)\}][g_x_t_ t_frac_partial f_x_t_ t_partial x_t]

目标函数的海塞矩阵：

![\\large H(x(t), t)=\\frac\{\\partial g(x(t), t)\}\{\\partial x^\{\\text\{T\}\}(t)\}=\\frac\{\\partial^2 f(x(t), t)\}\{\\partial x(t) \\partial x^\{\\text\{T\}\}(t)\}][large H_x_t_ t_frac_partial g_x_t_ t_partial x_text_T_t_frac_partial_2 f_x_t_ t_partial x_t_ _partial x_text_T_t]

目标函数对于时间t的导数：

![\\large \\dot g\_t(x(t), t)=\\frac\{\\partial g(x(t), t)\}\{\\partial t\}=\\frac\{\\partial^2f(x(t), t)\}\{\\partial x(t)\\partial t\}][large _dot g_t_x_t_ t_frac_partial g_x_t_ t_partial t_frac_partial_2f_x_t_ t_partial x_t_partial t]

最终根据牛顿法的定义，可以得到用于求解动态非线性最优化问题的牛顿法迭代公式：

![\\large x\_\{n+1\}=x\_n-H^\{-1\}(x\_n, t\_n)g(x\_n, t\_n)][large x_n_1_x_n-H_-1_x_n_ t_n_g_x_n_ t_n]

## 2. 离散零化神经网络求解动态非线性最优化问题 ##

动态非线性最优化问题的定义：![\\large \\text\{min\} ~ f(x(t), t)][large _text_min_ _ f_x_t_ t]

零化神经网络(ZNN)的演化公式：

![\\large \\dot \\epsilon(t)=-\\gamma \\epsilon(t)][large _dot _epsilon_t_-_gamma _epsilon_t]

定义误差函数：![\\large \\epsilon (t)=g(x(t), t)][large _epsilon _t_g_x_t_ t]，其中![\\large g(x(t), t)][large g_x_t_ t]为目标函数![\\large f(x(t), t)][large f_x_t_ t]的梯度：

![\\large g(x(t), t)=\\frac\{\\partial f(x(t), t)\}\{\\partial x(t)\}][large g_x_t_ t_frac_partial f_x_t_ t_partial x_t]

目标函数f的海塞矩阵：

![\\large H(x(t), t)=\\frac\{\\partial g(x(t), t)\}\{\\partial x^\{\\text\{T\}\}(t)\}=\\frac\{\\partial^2 f(x(t), t)\}\{\\partial x(t)\\partial x^\{\\text\{T\}\}(t)\}][large H_x_t_ t_frac_partial g_x_t_ t_partial x_text_T_t_frac_partial_2 f_x_t_ t_partial x_t_partial x_text_T_t]

目标函数的梯度对于时间的导数：

![\\large \\dot g\_t(x(t), t)=\\frac\{\\partial g(x(t), t)\}\{\\partial t\}=\\frac\{\\partial^2 f(x(t), t)\}\{\\partial x(t)\\partial t\}][large _dot g_t_x_t_ t_frac_partial g_x_t_ t_partial t_frac_partial_2 f_x_t_ t_partial x_t_partial t]

误差函数对于时间的导数：

![\\large \\frac\{\\partial \\epsilon (t)\}\{\\partial t\}=\\frac\{\\partial g(x(t), t)\}\{\\partial x(t)\}\\frac\{d x(t)\}\{d t\}-\\frac\{\\partial g(x(t), t)\}\{\\partial t\}][large _frac_partial _epsilon _t_partial t_frac_partial g_x_t_ t_partial x_t_frac_d x_t_d t_-_frac_partial g_x_t_ t_partial t]

![\\large \\dot \\epsilon (t)=H(x(t), t)\\dot x(t)+\\dot g\_t(x(t), t)][large _dot _epsilon _t_H_x_t_ t_dot x_t_dot g_t_x_t_ t]  
根据零化神经网络的演化公式和定义的误差函数可以得到：

![\\large H(x(t), t)\\dot x(t)=-\\gamma g(x(t), t)-\\dot g\_t(x(t), t)][large H_x_t_ t_dot x_t_-_gamma g_x_t_ t_-_dot g_t_x_t_ t]

![\\large \\dot x(t)=-H^\{-1\}(x(t), t)(\\gamma g(x(t), t)+\\frac\{\\partial g(x(t), t)\}\{\\partial t\} )][large _dot x_t_-H_-1_x_t_ t_gamma g_x_t_ t_frac_partial g_x_t_ t_partial t_]

令等式左侧的![\\dot x(t)][dot x_t]为![G(x(t), t)][G_x_t_ t]:

![\\large G(x(t), t)=-H^\{-1\}(x(t), t)(\\gamma \\frac\{\\partial f(x(t), t)\}\{\\partial x(t)\}+\\frac\{\\partial^2 f(x(t), t)\}\{\\partial x(t) \\partial t\})][large G_x_t_ t_-H_-1_x_t_ t_gamma _frac_partial f_x_t_ t_partial x_t_frac_partial_2 f_x_t_ t_partial x_t_ _partial t]

至此，就得到了用于求解动态非线性最优化问题的连续时间零化神经网络模型。

采用欧拉前向差分的方式离散化连续公式：

![\\large \\dot x\_n = \\frac\{x\_\{n+1\} -x\_n\}\{g\}][large _dot x_n _ _frac_x_n_1_ -x_n_g]

于是可以得到：

![\\large x\_\{n+1\}=x\_n+\\zeta G(x\_n,t\_n), ~t\_\{n+1\}=t\_n + \\zeta][large x_n_1_x_n_zeta G_x_n_t_n_ _t_n_1_t_n _ _zeta]

![\\large x\_\{n+1\}=x\_n-H^\{-1\}(x\_n, t\_n)(\\zeta \\gamma g(x\_n, t\_n)+\\zeta \\dot g\_t(x\_n, t\_n))][large x_n_1_x_n-H_-1_x_n_ t_n_zeta _gamma g_x_n_ t_n_zeta _dot g_t_x_n_ t_n]

![\\large x\_\{n+1\}=x\_n-H^\{-1\}\_n(\\zeta \\gamma g\_n+\\zeta \\dot g\_n)][large x_n_1_x_n-H_-1_n_zeta _gamma g_n_zeta _dot g_n]

至此，就得到了用于求解动态非线性最优化问题的离散时间零化神经网络模型。

[large _text_min_ _ f_x_t_ t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20%5Ctext%7Bmin%7D%20~%20f%28x%28t%29%2C%20t%29
[large x_n_1_x_n-_frac_dot f_ddot f]: https://private.codecogs.com/gif.latex?%5Clarge%20x_%7Bn&plus;1%7D%3Dx_n-%5Cfrac%7B%5Cdot%20f%7D%7B%5Cddot%20f%7D
[f_x_t_ t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20f%28x%28t%29%2C%20t%29
[g_x_t_ t_frac_partial f_x_t_ t_partial x_t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20g%28x%28t%29%2C%20t%29%3D%5Cfrac%7B%5Cpartial%20f%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20x%28t%29%7D
[large H_x_t_ t_frac_partial g_x_t_ t_partial x_text_T_t_frac_partial_2 f_x_t_ t_partial x_t_ _partial x_text_T_t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20H%28x%28t%29%2C%20t%29%3D%5Cfrac%7B%5Cpartial%20g%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20x%5E%7B%5Ctext%7BT%7D%7D%28t%29%7D%3D%5Cfrac%7B%5Cpartial%5E2%20f%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20x%28t%29%20%5Cpartial%20x%5E%7B%5Ctext%7BT%7D%7D%28t%29%7D
[large _dot g_t_x_t_ t_frac_partial g_x_t_ t_partial t_frac_partial_2f_x_t_ t_partial x_t_partial t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20%5Cdot%20g_t%28x%28t%29%2C%20t%29%3D%5Cfrac%7B%5Cpartial%20g%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20t%7D%3D%5Cfrac%7B%5Cpartial%5E2f%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20x%28t%29%5Cpartial%20t%7D
[large x_n_1_x_n-H_-1_x_n_ t_n_g_x_n_ t_n]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20x_%7Bn&plus;1%7D%3Dx_n-H%5E%7B-1%7D%28x_n%2C%20t_n%29g%28x_n%2C%20t_n%29
[large _dot _epsilon_t_-_gamma _epsilon_t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20%5Cdot%20%5Cepsilon%28t%29%3D-%5Cgamma%20%5Cepsilon%28t%29
[large _epsilon _t_g_x_t_ t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20%5Cepsilon%20%28t%29%3Dg%28x%28t%29%2C%20t%29
[large g_x_t_ t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20g%28x%28t%29%2C%20t%29
[large f_x_t_ t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20f%28x%28t%29%2C%20t%29
[large g_x_t_ t_frac_partial f_x_t_ t_partial x_t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20g%28x%28t%29%2C%20t%29%3D%5Cfrac%7B%5Cpartial%20f%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20x%28t%29%7D
[large H_x_t_ t_frac_partial g_x_t_ t_partial x_text_T_t_frac_partial_2 f_x_t_ t_partial x_t_partial x_text_T_t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20H%28x%28t%29%2C%20t%29%3D%5Cfrac%7B%5Cpartial%20g%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20x%5E%7B%5Ctext%7BT%7D%7D%28t%29%7D%3D%5Cfrac%7B%5Cpartial%5E2%20f%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20x%28t%29%5Cpartial%20x%5E%7B%5Ctext%7BT%7D%7D%28t%29%7D
[large _dot g_t_x_t_ t_frac_partial g_x_t_ t_partial t_frac_partial_2 f_x_t_ t_partial x_t_partial t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20%5Cdot%20g_t%28x%28t%29%2C%20t%29%3D%5Cfrac%7B%5Cpartial%20g%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20t%7D%3D%5Cfrac%7B%5Cpartial%5E2%20f%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20x%28t%29%5Cpartial%20t%7D
[large _frac_partial _epsilon _t_partial t_frac_partial g_x_t_ t_partial x_t_frac_d x_t_d t_-_frac_partial g_x_t_ t_partial t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20%5Cfrac%7B%5Cpartial%20%5Cepsilon%20%28t%29%7D%7B%5Cpartial%20t%7D%3D%5Cfrac%7B%5Cpartial%20g%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20x%28t%29%7D%5Cfrac%7Bd%20x%28t%29%7D%7Bd%20t%7D-%5Cfrac%7B%5Cpartial%20g%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20t%7D
[large _dot _epsilon _t_H_x_t_ t_dot x_t_dot g_t_x_t_ t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20%5Cdot%20%5Cepsilon%20%28t%29%3DH%28x%28t%29%2C%20t%29%5Cdot%20x%28t%29&plus;%5Cdot%20g_t%28x%28t%29%2C%20t%29
[large H_x_t_ t_dot x_t_-_gamma g_x_t_ t_-_dot g_t_x_t_ t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20H%28x%28t%29%2C%20t%29%5Cdot%20x%28t%29%3D-%5Cgamma%20g%28x%28t%29%2C%20t%29-%5Cdot%20g_t%28x%28t%29%2C%20t%29
[large _dot x_t_-H_-1_x_t_ t_gamma g_x_t_ t_frac_partial g_x_t_ t_partial t_]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20%5Cdot%20x%28t%29%3D-H%5E%7B-1%7D%28x%28t%29%2C%20t%29%28%5Cgamma%20g%28x%28t%29%2C%20t%29&plus;%5Cfrac%7B%5Cpartial%20g%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20t%7D%20%29
[dot x_t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Cdot%20x%28t%29
[G_x_t_ t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20G%28x%28t%29%2C%20t%29
[large G_x_t_ t_-H_-1_x_t_ t_gamma _frac_partial f_x_t_ t_partial x_t_frac_partial_2 f_x_t_ t_partial x_t_ _partial t]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20G%28x%28t%29%2C%20t%29%3D-H%5E%7B-1%7D%28x%28t%29%2C%20t%29%28%5Cgamma%20%5Cfrac%7B%5Cpartial%20f%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20x%28t%29%7D&plus;%5Cfrac%7B%5Cpartial%5E2%20f%28x%28t%29%2C%20t%29%7D%7B%5Cpartial%20x%28t%29%20%5Cpartial%20t%7D%29
[large _dot x_n _ _frac_x_n_1_ -x_n_g]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20%5Cdot%20x_n%20%3D%20%5Cfrac%7Bx_%7Bn&plus;1%7D%20-x_n%7D%7Bg%7D
[large x_n_1_x_n_zeta G_x_n_t_n_ _t_n_1_t_n _ _zeta]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20x_%7Bn&plus;1%7D%3Dx_n&plus;%5Czeta%20G%28x_n%2Ct_n%29%2C%20~t_%7Bn&plus;1%7D%3Dt_n%20&plus;%20%5Czeta
[large x_n_1_x_n-H_-1_x_n_ t_n_zeta _gamma g_x_n_ t_n_zeta _dot g_t_x_n_ t_n]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20x_%7Bn&plus;1%7D%3Dx_n-H%5E%7B-1%7D%28x_n%2C%20t_n%29%28%5Czeta%20%5Cgamma%20g%28x_n%2C%20t_n%29&plus;%5Czeta%20%5Cdot%20g_t%28x_n%2C%20t_n%29%29
[large x_n_1_x_n-H_-1_n_zeta _gamma g_n_zeta _dot g_n]: https://private.codecogs.com/gif.latex?%5Cdpi%7B100%7D%20%5Clarge%20x_%7Bn&plus;1%7D%3Dx_n-H%5E%7B-1%7D_n%28%5Czeta%20%5Cgamma%20g_n&plus;%5Czeta%20%5Cdot%20g_n%29