BP算法公式推导

男娘i 2022-12-25 03:55 339阅读 0赞

令![W^l\_\{ji\}][W_l_ji]表示第![l-1][]层的第![i][]个神经元到第![l][]层的第![j][]个神经元的连接权值，![x^l\_j][x_l_j]表示第![l][]层第![j][]个神经元的输入，![y^l\_j][y_l_j]表示第![l][]层第![j][]个神经元的输出，![\\Theta^l\_j][Theta_l_j]表示![l][]层第![j][]个神经元的偏置，C表示代价函数,

则有：

![x^l\_j = \\sum \_k w^l\_\{ji\}y^\{l-1\}\_i + \\Theta ^l\_j][x_l_j _ _sum _k w_l_ji_y_l-1_i _ _Theta _l_j]

![y^l\_j = f\\left ( x^l\_j \\right )][y_l_j _ f_left _ x_l_j _right]

其中,![f\\left ( \\cdot \\right )][f_left _ _cdot _right]表示激活函数。训练多层网络的目的就是使代价函数C最小化.

定义代价函数为:

![C= \\frac\{1\}\{2\}\\left \\| Y-y^L \\right \\|^2][C_ _frac_1_2_left _ Y-y_L _right _2]

可以看出，这个函数依赖于实际的目标值![Y][]，![y^L][y_L]可以看成权值和偏置的函数,通过不断的修改权值和偏置值来改变神经网络的输出值.

接下来更新权值和偏置:

首先定义误差![\\delta][delta],令![\\delta ^l\_j][delta _l_j]表示第![l][]层第![j][]个神经元上的**误差（？）**，定义为

**![gif.latex][]![\\large \\delta ^l\_j = \\delta x^l\_j = \\frac\{\\partial C\}\{\\partial x^l\_j\}][large _delta _l_j _ _delta x_l_j _ _frac_partial C_partial x_l_j]**

由链式求导法则可得输出层的误差方程为：

![\\large \\delta ^l\_j = \\delta x^l\_j = \\frac\{\\partial C\}\{\\partial y^l\_j\}\{f \\left(x^l\_j\\right)\}'][large _delta _l_j _ _delta x_l_j _ _frac_partial C_partial y_l_j_f _left_x_l_j_right]

因为当前层神经元的输出是上一层神经元输出的线性组合，由链式法则可通过下层神经元的误差来表示当前层的误差：

![\\large \\delta ^l\_j = \\delta x^l\_j = \\frac\{\\partial C\}\{\\partial x^l\_j\} = \\sum \_i \\frac\{\\partial C\}\{\\partial x^\{l+1\}\_i\} \\frac\{\\partial x^\{l+1\}\_i\}\{\\partial x^l\_j\} = \\sum \_i \\frac\{\\partial x^\{l+1\}\_i\}\{\\partial x^l\_j\} \\delta x^\{l+1\}\_i][large _delta _l_j _ _delta x_l_j _ _frac_partial C_partial x_l_j_ _ _sum _i _frac_partial C_partial x_l_1_i_ _frac_partial x_l_1_i_partial x_l_j_ _ _sum _i _frac_partial x_l_1_i_partial x_l_j_ _delta x_l_1_i]   \# 建立了 前一层误差 和 后一层误差的关系

又因为： 建立了 ![\\large x^\{l+1\}\_i][large x_l_1_i]和![\\large x^l\_i][large x_l_i]的联系

![\\large x^\{l+1\}\_i = \\sum \_j w^\{l+1\}\_\{ij\} y^l\_j + \\theta ^\{l+1\}\_i = \\sum \_j w^\{l+1\}\_\{ij\} f\\left(x^l\_j \\right ) + \\theta ^\{l+1\}\_i][large x_l_1_i _ _sum _j w_l_1_ij_ y_l_j _ _theta _l_1_i _ _sum _j w_l_1_ij_ f_left_x_l_j _right _ _ _theta _l_1_i]

对![\\large x^l\_j][large x_l_j]求偏导,可得：

![\\large \\frac\{\\partial x^\{l+1\}\_i\}\{\\partial x^l\_j\} = w^\{l+1\}\_\{ij\} f'\\left(x^l\_j \\right )][large _frac_partial x_l_1_i_partial x_l_j_ _ w_l_1_ij_ f_left_x_l_j _right]

故有:   前一层误差 和 后一层误差的关系推导

![\\large \\delta ^l\_j = \\sum \_i w^\{l+1\}\_\{ij\} f'\\left(x^l\_j \\right ) \\delta^\{l+1\}\_i][large _delta _l_j _ _sum _i w_l_1_ij_ f_left_x_l_j _right _ _delta_l_1_i]

权重和偏置更新公式：

![\\large \\frac\{\\partial C \}\{ \\partial w^l\_\{ji\}\} = \\frac\{\\partial C\}\{\\partial x^l\_j\} \\frac\{\\partial x^l\_j\}\{\\partial w^l\_\{ji\}\} = \\delta ^l\_j y^\{l-1\}\_i][large _frac_partial C _ _partial w_l_ji_ _ _frac_partial C_partial x_l_j_ _frac_partial x_l_j_partial w_l_ji_ _ _delta _l_j y_l-1_i]

![\\large \\frac \{\\partial C\}\{\\partial \\theta ^l\_j\} = \\frac\{\\partial C\}\{\\partial x^l\_j\}\\frac\{\\partial x^l\_j\}\{\\partial \\theta ^l\_j\} = \\delta^l\_j][large _frac _partial C_partial _theta _l_j_ _ _frac_partial C_partial x_l_j_frac_partial x_l_j_partial _theta _l_j_ _ _delta_l_j]

由梯度下降法可得到更新规则为：

![\\large w^l\_\{ji\} = w^l\_\{ji\} - \\alpha \\frac\{\\partial C\}\{\\partial w^l\_\{ji\}\} = w^l\_\{ji\} - \\alpha \\delta^l\_j y^\{l-1\}\_i][large w_l_ji_ _ w_l_ji_ - _alpha _frac_partial C_partial w_l_ji_ _ w_l_ji_ - _alpha _delta_l_j y_l-1_i]

![\\large \\theta ^l\_j = \\theta ^l\_j - \\alpha\\frac\{\\partial C\}\{\\partial \\theta^l\_j\} = \\theta^l\_j - \\alpha \\delta ^l\_j][large _theta _l_j _ _theta _l_j - _alpha_frac_partial C_partial _theta_l_j_ _ _theta_l_j - _alpha _delta _l_j]

BP反向传播的过程；

[W_l_ji]: https://private.codecogs.com/gif.latex?W%5El_%7Bji%7D
[l-1]: https://private.codecogs.com/gif.latex?l-1
[i]: https://private.codecogs.com/gif.latex?i
[l]: https://private.codecogs.com/gif.latex?l
[j]: https://private.codecogs.com/gif.latex?j
[x_l_j]: https://private.codecogs.com/gif.latex?x%5El_j
[y_l_j]: https://private.codecogs.com/gif.latex?y%5El_j
[Theta_l_j]: https://private.codecogs.com/gif.latex?%5CTheta%5El_j
[x_l_j _ _sum _k w_l_ji_y_l-1_i _ _Theta _l_j]: https://private.codecogs.com/gif.latex?x%5El_j%20%3D%20%5Csum%20_k%20w%5El_%7Bji%7Dy%5E%7Bl-1%7D_i%20&plus;%20%5CTheta%20%5El_j
[y_l_j _ f_left _ x_l_j _right]: https://private.codecogs.com/gif.latex?y%5El_j%20%3D%20f%5Cleft%20%28%20x%5El_j%20%5Cright%20%29
[f_left _ _cdot _right]: https://private.codecogs.com/gif.latex?f%5Cleft%20%28%20%5Ccdot%20%5Cright%20%29
[C_ _frac_1_2_left _ Y-y_L _right _2]: https://private.codecogs.com/gif.latex?C%3D%20%5Cfrac%7B1%7D%7B2%7D%5Cleft%20%5C%7C%20Y-y%5EL%20%5Cright%20%5C%7C%5E2
[Y]: https://private.codecogs.com/gif.latex?Y
[y_L]: https://private.codecogs.com/gif.latex?y%5EL
[delta]: https://private.codecogs.com/gif.latex?%5Cdelta
[delta _l_j]: https://private.codecogs.com/gif.latex?%5Cdelta%20%5El_j
[gif.latex]: https://private.codecogs.com/gif.latex?
[large _delta _l_j _ _delta x_l_j _ _frac_partial C_partial x_l_j]: https://private.codecogs.com/gif.latex?%5Clarge%20%5Cdelta%20%5El_j%20%3D%20%5Cdelta%20x%5El_j%20%3D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20x%5El_j%7D
[large _delta _l_j _ _delta x_l_j _ _frac_partial C_partial y_l_j_f _left_x_l_j_right]: https://private.codecogs.com/gif.latex?%5Clarge%20%5Cdelta%20%5El_j%20%3D%20%5Cdelta%20x%5El_j%20%3D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20y%5El_j%7D%7Bf%20%5Cleft%28x%5El_j%5Cright%29%7D%27
[large _delta _l_j _ _delta x_l_j _ _frac_partial C_partial x_l_j_ _ _sum _i _frac_partial C_partial x_l_1_i_ _frac_partial x_l_1_i_partial x_l_j_ _ _sum _i _frac_partial x_l_1_i_partial x_l_j_ _delta x_l_1_i]: https://private.codecogs.com/gif.latex?%5Clarge%20%5Cdelta%20%5El_j%20%3D%20%5Cdelta%20x%5El_j%20%3D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20x%5El_j%7D%20%3D%20%5Csum%20_i%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20x%5E%7Bl&plus;1%7D_i%7D%20%5Cfrac%7B%5Cpartial%20x%5E%7Bl&plus;1%7D_i%7D%7B%5Cpartial%20x%5El_j%7D%20%3D%20%5Csum%20_i%20%5Cfrac%7B%5Cpartial%20x%5E%7Bl&plus;1%7D_i%7D%7B%5Cpartial%20x%5El_j%7D%20%5Cdelta%20x%5E%7Bl&plus;1%7D_i
[large x_l_1_i]: https://private.codecogs.com/gif.latex?%5Clarge%20x%5E%7Bl&plus;1%7D_i
[large x_l_i]: https://private.codecogs.com/gif.latex?%5Clarge%20x%5El_i
[large x_l_1_i _ _sum _j w_l_1_ij_ y_l_j _ _theta _l_1_i _ _sum _j w_l_1_ij_ f_left_x_l_j _right _ _ _theta _l_1_i]: https://private.codecogs.com/gif.latex?%5Clarge%20x%5E%7Bl&plus;1%7D_i%20%3D%20%5Csum%20_j%20w%5E%7Bl&plus;1%7D_%7Bij%7D%20y%5El_j%20&plus;%20%5Ctheta%20%5E%7Bl&plus;1%7D_i%20%3D%20%5Csum%20_j%20w%5E%7Bl&plus;1%7D_%7Bij%7D%20f%5Cleft%28x%5El_j%20%5Cright%20%29%20&plus;%20%5Ctheta%20%5E%7Bl&plus;1%7D_i
[large x_l_j]: https://private.codecogs.com/gif.latex?%5Clarge%20x%5El_j
[large _frac_partial x_l_1_i_partial x_l_j_ _ w_l_1_ij_ f_left_x_l_j _right]: https://private.codecogs.com/gif.latex?%5Clarge%20%5Cfrac%7B%5Cpartial%20x%5E%7Bl&plus;1%7D_i%7D%7B%5Cpartial%20x%5El_j%7D%20%3D%20w%5E%7Bl&plus;1%7D_%7Bij%7D%20f%27%5Cleft%28x%5El_j%20%5Cright%20%29
[large _delta _l_j _ _sum _i w_l_1_ij_ f_left_x_l_j _right _ _delta_l_1_i]: https://private.codecogs.com/gif.latex?%5Clarge%20%5Cdelta%20%5El_j%20%3D%20%5Csum%20_i%20w%5E%7Bl&plus;1%7D_%7Bij%7D%20f%27%5Cleft%28x%5El_j%20%5Cright%20%29%20%5Cdelta%5E%7Bl&plus;1%7D_i
[large _frac_partial C _ _partial w_l_ji_ _ _frac_partial C_partial x_l_j_ _frac_partial x_l_j_partial w_l_ji_ _ _delta _l_j y_l-1_i]: https://private.codecogs.com/gif.latex?%5Clarge%20%5Cfrac%7B%5Cpartial%20C%20%7D%7B%20%5Cpartial%20w%5El_%7Bji%7D%7D%20%3D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20x%5El_j%7D%20%5Cfrac%7B%5Cpartial%20x%5El_j%7D%7B%5Cpartial%20w%5El_%7Bji%7D%7D%20%3D%20%5Cdelta%20%5El_j%20y%5E%7Bl-1%7D_i
[large _frac _partial C_partial _theta _l_j_ _ _frac_partial C_partial x_l_j_frac_partial x_l_j_partial _theta _l_j_ _ _delta_l_j]: https://private.codecogs.com/gif.latex?%5Clarge%20%5Cfrac%20%7B%5Cpartial%20C%7D%7B%5Cpartial%20%5Ctheta%20%5El_j%7D%20%3D%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20x%5El_j%7D%5Cfrac%7B%5Cpartial%20x%5El_j%7D%7B%5Cpartial%20%5Ctheta%20%5El_j%7D%20%3D%20%5Cdelta%5El_j
[large w_l_ji_ _ w_l_ji_ - _alpha _frac_partial C_partial w_l_ji_ _ w_l_ji_ - _alpha _delta_l_j y_l-1_i]: https://private.codecogs.com/gif.latex?%5Clarge%20w%5El_%7Bji%7D%20%3D%20w%5El_%7Bji%7D%20-%20%5Calpha%20%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20w%5El_%7Bji%7D%7D%20%3D%20w%5El_%7Bji%7D%20-%20%5Calpha%20%5Cdelta%5El_j%20y%5E%7Bl-1%7D_i
[large _theta _l_j _ _theta _l_j - _alpha_frac_partial C_partial _theta_l_j_ _ _theta_l_j - _alpha _delta _l_j]: https://private.codecogs.com/gif.latex?%5Clarge%20%5Ctheta%20%5El_j%20%3D%20%5Ctheta%20%5El_j%20-%20%5Calpha%5Cfrac%7B%5Cpartial%20C%7D%7B%5Cpartial%20%5Ctheta%5El_j%7D%20%3D%20%5Ctheta%5El_j%20-%20%5Calpha%20%5Cdelta%20%5El_j