一个循序渐进的反向传播的例子

忘是亡心i 2022-07-14 00:56 178阅读 0赞

A Step by Step Backpropagation Example

## Background ##

Backpropagation is a common method for training a neural network. There is [no shortage of papers][] online that attempt to explain how backpropagation works, but few that include an example with actual numbers. This post is my attempt to explain how it works with a concrete example that folks can compare their own calculations to in order to ensure they understand backpropagation correctly.

If this kind of thing interests you, you should [sign up for my newsletter][] where I post about AI-related projects that I’m working on.

## Backpropagation in Python ##

You can play around with a Python script that I wrote that implements the backpropagation algorithm in [this Github repo][].

## Backpropagation Visualization ##

For an interactive visualization showing a neural network as it learns, check out my[Neural Network visualization][].

## Additional Resources ##

If you find this tutorial useful and want to continue learning about neural networks and their applications, I highly recommend checking out Adrian Rosebrock’s excellent tutorial on [Getting Started with Deep Learning and Python][].

## Overview ##

For this tutorial, we’re going to use a neural network with two inputs, two hidden neurons, two output neurons. Additionally, the hidden and output neurons will include a bias.

Here’s the basic structure:

![neural\_network (7)][neural_network _7]

In order to have some numbers to work with, here are the initial weights, the biases, and training inputs/outputs:

![neural\_network (9)][neural_network _9]

The goal of backpropagation is to optimize the weights so that the neural network can learn how to correctly map arbitrary inputs to outputs.

For the rest of this tutorial we’re going to work with a single training set: given inputs 0.05 and 0.10, we want the neural network to output 0.01 and 0.99.

## The Forward Pass ##

To begin, lets see what the neural network currently predicts given the weights and biases above and inputs of 0.05 and 0.10. To do this we’ll feed those inputs forward though the network.

We figure out the total net input to each hidden layer neuron, squash the total net input using an activation function (here we use the logistic function), then repeat the process with the output layer neurons.

Total net input is also referred to as just  *net input* by  [some sources][].

Here’s how we calculate the total net input for ![h\_1][h_1]:

![net\_\{h1\} = w\_1 \* i\_1 + w\_2 \* i\_2 + b\_1 \* 1][net_h1_ _ w_1 _ i_1 _ w_2 _ i_2 _ b_1 _ 1]

![net\_\{h1\} = 0.15 \* 0.05 + 0.2 \* 0.1 + 0.35 \* 1 = 0.3775][net_h1_ _ 0.15 _ 0.05 _ 0.2 _ 0.1 _ 0.35 _ 1 _ 0.3775]

We then squash it using the logistic function to get the output of ![h\_1][h_1]:

![out\_\{h1\} = \\frac\{1\}\{1+e^\{-net\_\{h1\}\}\} = \\frac\{1\}\{1+e^\{-0.3775\}\} = 0.593269992][out_h1_ _ _frac_1_1_e_-net_h1_ _ _frac_1_1_e_-0.3775_ _ 0.593269992]

Carrying out the same process for ![h\_2][h_2] we get:

![out\_\{h2\} = 0.596884378][out_h2_ _ 0.596884378]

We repeat this process for the output layer neurons, using the output from the hidden layer neurons as inputs.

Here’s the output for ![o\_1][o_1]:

![net\_\{o1\} = w\_5 \* out\_\{h1\} + w\_6 \* out\_\{h2\} + b\_2 \* 1][net_o1_ _ w_5 _ out_h1_ _ w_6 _ out_h2_ _ b_2 _ 1]

![net\_\{o1\} = 0.4 \* 0.593269992 + 0.45 \* 0.596884378 + 0.6 \* 1 = 1.105905967][net_o1_ _ 0.4 _ 0.593269992 _ 0.45 _ 0.596884378 _ 0.6 _ 1 _ 1.105905967]

![out\_\{o1\} = \\frac\{1\}\{1+e^\{-net\_\{o1\}\}\} = \\frac\{1\}\{1+e^\{-1.105905967\}\} = 0.75136507][out_o1_ _ _frac_1_1_e_-net_o1_ _ _frac_1_1_e_-1.105905967_ _ 0.75136507]

And carrying out the same process for ![o\_2][o_2] we get:

![out\_\{o2\} = 0.772928465][out_o2_ _ 0.772928465]

### Calculating the Total Error ###

We can now calculate the error for each output neuron using the [squared error function][] and sum them to get the total error:

![E\_\{total\} = \\sum \\frac\{1\}\{2\}(target - output)^\{2\}][E_total_ _ _sum _frac_1_2_target - output_2]

[Some sources][] refer to the target as the  *ideal* and the output as the  *actual*.

The  ![\\frac\{1\}\{2\}][frac_1_2] is included so that exponent is cancelled when we differentiate later on. The result is eventually multiplied by a learning rate anyway so it doesn’t matter that we introduce a constant here \[ [1][squared error function]\].

For example, the target output for ![o\_1][o_1] is 0.01 but the neural network output 0.75136507, therefore its error is:

![E\_\{o1\} = \\frac\{1\}\{2\}(target\_\{o1\} - out\_\{o1\})^\{2\} = \\frac\{1\}\{2\}(0.01 - 0.75136507)^\{2\} = 0.274811083][E_o1_ _ _frac_1_2_target_o1_ - out_o1_2_ _ _frac_1_2_0.01 - 0.75136507_2_ _ 0.274811083]

Repeating this process for ![o\_2][o_2] (remembering that the target is 0.99) we get:

![E\_\{o2\} = 0.023560026][E_o2_ _ 0.023560026]

The total error for the neural network is the sum of these errors:

![E\_\{total\} = E\_\{o1\} + E\_\{o2\} = 0.274811083 + 0.023560026 = 0.298371109][E_total_ _ E_o1_ _ E_o2_ _ 0.274811083 _ 0.023560026 _ 0.298371109]

## The Backwards Pass ##

Our goal with backpropagation is to update each of the weights in the network so that they cause the actual output to be closer the target output, thereby minimizing the error for each output neuron and the network as a whole.

### Output Layer ###

Consider ![w\_5][w_5]. We want to know how much a change in ![w\_5][w_5] affects the total error, aka ![\\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{5\}\}][frac_partial E_total_partial w_5].

![\\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{5\}\}][frac_partial E_total_partial w_5 1] is read as “the partial derivative of  ![E\_\{total\}][E_total] with respect to  ![w\_\{5\}][w_5 1]“. You can also say “the gradient with respect to  ![w\_\{5\}][w_5 1]“.

By applying the [chain rule][] we know that:

![\\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{5\}\} = \\frac\{\\partial E\_\{total\}\}\{\\partial out\_\{o1\}\} \* \\frac\{\\partial out\_\{o1\}\}\{\\partial net\_\{o1\}\} \* \\frac\{\\partial net\_\{o1\}\}\{\\partial w\_\{5\}\}][frac_partial E_total_partial w_5_ _ _frac_partial E_total_partial out_o1_ _ _frac_partial out_o1_partial net_o1_ _ _frac_partial net_o1_partial w_5]

Visually, here’s what we’re doing:

![output\_1\_backprop (4)][output_1_backprop _4]

We need to figure out each piece in this equation.

First, how much does the total error change with respect to the output?

![E\_\{total\} = \\frac\{1\}\{2\}(target\_\{o1\} - out\_\{o1\})^\{2\} + \\frac\{1\}\{2\}(target\_\{o2\} - out\_\{o2\})^\{2\}][E_total_ _ _frac_1_2_target_o1_ - out_o1_2_ _ _frac_1_2_target_o2_ - out_o2_2]

![\\frac\{\\partial E\_\{total\}\}\{\\partial out\_\{o1\}\} = 2 \* \\frac\{1\}\{2\}(target\_\{o1\} - out\_\{o1\})^\{2 - 1\} \* -1 + 0][frac_partial E_total_partial out_o1_ _ 2 _ _frac_1_2_target_o1_ - out_o1_2 - 1_ _ -1 _ 0]

![\\frac\{\\partial E\_\{total\}\}\{\\partial out\_\{o1\}\} = -(target\_\{o1\} - out\_\{o1\}) = -(0.01 - 0.75136507) = 0.74136507][frac_partial E_total_partial out_o1_ _ -_target_o1_ - out_o1_ _ -_0.01 - 0.75136507_ _ 0.74136507]

![\-(target - out)][-_target - out] is sometimes expressed as  ![out - target][]

When we take the partial derivative of the total error with respect to  ![out\_\{o1\}][out_o1], the quantity  ![\\frac\{1\}\{2\}(target\_\{o2\} - out\_\{o2\})^\{2\}][frac_1_2_target_o2_ - out_o2_2] becomes zero because  ![out\_\{o1\}][out_o1] does not affect it which means we’re taking the derivative of a constant which is zero.

Next, how much does the output of ![o\_1][o_1] change with respect to its total net input?

The partial [derivative of the logistic function][] is the output multiplied by 1 minus the output:

![out\_\{o1\} = \\frac\{1\}\{1+e^\{-net\_\{o1\}\}\}][out_o1_ _ _frac_1_1_e_-net_o1]

![\\frac\{\\partial out\_\{o1\}\}\{\\partial net\_\{o1\}\} = out\_\{o1\}(1 - out\_\{o1\}) = 0.75136507(1 - 0.75136507) = 0.186815602][frac_partial out_o1_partial net_o1_ _ out_o1_1 - out_o1_ _ 0.75136507_1 - 0.75136507_ _ 0.186815602]

Finally, how much does the total net input of ![o1][] change with respect to ![w\_5][w_5]?

![net\_\{o1\} = w\_5 \* out\_\{h1\} + w\_6 \* out\_\{h2\} + b\_2 \* 1][net_o1_ _ w_5 _ out_h1_ _ w_6 _ out_h2_ _ b_2 _ 1]

![\\frac\{\\partial net\_\{o1\}\}\{\\partial w\_\{5\}\} = 1 \* out\_\{h1\} \* w\_5^\{(1 - 1)\} + 0 + 0 = out\_\{h1\} = 0.593269992][frac_partial net_o1_partial w_5_ _ 1 _ out_h1_ _ w_5_1 - 1_ _ 0 _ 0 _ out_h1_ _ 0.593269992]

Putting it all together:

![\\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{5\}\} = 0.74136507 \* 0.186815602 \* 0.593269992 = 0.082167041][frac_partial E_total_partial w_5_ _ 0.74136507 _ 0.186815602 _ 0.593269992 _ 0.082167041]

You’ll often see this calculation combined in the form of the [delta rule][]:

![\\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{5\}\} = -(target\_\{o1\} - out\_\{o1\}) \* out\_\{o1\}(1 - out\_\{o1\}) \* out\_\{h1\}][frac_partial E_total_partial w_5_ _ -_target_o1_ - out_o1_ _ out_o1_1 - out_o1_ _ out_h1]

Alternatively, we have ![\\frac\{\\partial E\_\{total\}\}\{\\partial out\_\{o1\}\}][frac_partial E_total_partial out_o1] and ![\\frac\{\\partial out\_\{o1\}\}\{\\partial net\_\{o1\}\}][frac_partial out_o1_partial net_o1] which can be written as ![\\frac\{\\partial E\_\{total\}\}\{\\partial net\_\{o1\}\}][frac_partial E_total_partial net_o1], aka ![\\delta\_\{o1\}][delta_o1] (the Greek letter delta) aka the *node delta*. We can use this to rewrite the calculation above:

![\\delta\_\{o1\} = \\frac\{\\partial E\_\{total\}\}\{\\partial out\_\{o1\}\} \* \\frac\{\\partial out\_\{o1\}\}\{\\partial net\_\{o1\}\} = \\frac\{\\partial E\_\{total\}\}\{\\partial net\_\{o1\}\}][delta_o1_ _ _frac_partial E_total_partial out_o1_ _ _frac_partial out_o1_partial net_o1_ _ _frac_partial E_total_partial net_o1]

![\\delta\_\{o1\} = -(target\_\{o1\} - out\_\{o1\}) \* out\_\{o1\}(1 - out\_\{o1\})][delta_o1_ _ -_target_o1_ - out_o1_ _ out_o1_1 - out_o1]

Therefore:

![\\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{5\}\} = \\delta\_\{o1\} out\_\{h1\}][frac_partial E_total_partial w_5_ _ _delta_o1_ out_h1]

Some sources extract the negative sign from ![\\delta][delta] so it would be written as:

![\\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{5\}\} = -\\delta\_\{o1\} out\_\{h1\}][frac_partial E_total_partial w_5_ _ -_delta_o1_ out_h1]

To decrease the error, we then subtract this value from the current weight (optionally multiplied by some learning rate, eta, which we’ll set to 0.5):

![w\_5^\{+\} = w\_5 - \\eta \* \\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{5\}\} = 0.4 - 0.5 \* 0.082167041 = 0.35891648][w_5_ _ w_5 - _eta _ _frac_partial E_total_partial w_5_ _ 0.4 - 0.5 _ 0.082167041 _ 0.35891648]

[Some][delta rule]  [sources][] use  ![\\alpha][alpha] (alpha) to represent the learning rate,  [others use][]  ![\\eta][eta](eta), and  [others][some sources] even use  ![\\epsilon][epsilon] (epsilon).

We can repeat this process to get the new weights ![w\_6][w_6], ![w\_7][w_7], and ![w\_8][w_8]:

![w\_6^\{+\} = 0.408666186][w_6_ _ 0.408666186]

![w\_7^\{+\} = 0.511301270][w_7_ _ 0.511301270]

![w\_8^\{+\} = 0.561370121][w_8_ _ 0.561370121]

We perform the actual updates in the neural network *after* we have the new weights leading into the hidden layer neurons (ie, we use the original weights, not the updated weights, when we continue the backpropagation algorithm below).

### Hidden Layer ###

Next, we’ll continue the backwards pass by calculating new values for ![w\_1][w_1], ![w\_2][w_2], ![w\_3][w_3], and ![w\_4][w_4].

Big picture, here’s what we need to figure out:

![\\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{1\}\} = \\frac\{\\partial E\_\{total\}\}\{\\partial out\_\{h1\}\} \* \\frac\{\\partial out\_\{h1\}\}\{\\partial net\_\{h1\}\} \* \\frac\{\\partial net\_\{h1\}\}\{\\partial w\_\{1\}\}][frac_partial E_total_partial w_1_ _ _frac_partial E_total_partial out_h1_ _ _frac_partial out_h1_partial net_h1_ _ _frac_partial net_h1_partial w_1]

Visually:

[![nn-calculation][]][nn-calculation 1]

We’re going to use a similar process as we did for the output layer, but slightly different to account for the fact that the output of each hidden layer neuron contributes to the output (and therefore error) of multiple output neurons. We know that ![out\_\{h1\}][out_h1] affects both ![out\_\{o1\}][out_o1 1] and ![out\_\{o2\}][out_o2] therefore the ![\\frac\{\\partial E\_\{total\}\}\{\\partial out\_\{h1\}\}][frac_partial E_total_partial out_h1] needs to take into consideration its effect on the both output neurons:

![\\frac\{\\partial E\_\{total\}\}\{\\partial out\_\{h1\}\} = \\frac\{\\partial E\_\{o1\}\}\{\\partial out\_\{h1\}\} + \\frac\{\\partial E\_\{o2\}\}\{\\partial out\_\{h1\}\}][frac_partial E_total_partial out_h1_ _ _frac_partial E_o1_partial out_h1_ _ _frac_partial E_o2_partial out_h1]

Starting with ![\\frac\{\\partial E\_\{o1\}\}\{\\partial out\_\{h1\}\}][frac_partial E_o1_partial out_h1]:

![\\frac\{\\partial E\_\{o1\}\}\{\\partial out\_\{h1\}\} = \\frac\{\\partial E\_\{o1\}\}\{\\partial net\_\{o1\}\} \* \\frac\{\\partial net\_\{o1\}\}\{\\partial out\_\{h1\}\}][frac_partial E_o1_partial out_h1_ _ _frac_partial E_o1_partial net_o1_ _ _frac_partial net_o1_partial out_h1]

We can calculate ![\\frac\{\\partial E\_\{o1\}\}\{\\partial net\_\{o1\}\}][frac_partial E_o1_partial net_o1] using values we calculated earlier:

![\\frac\{\\partial E\_\{o1\}\}\{\\partial net\_\{o1\}\} = \\frac\{\\partial E\_\{o1\}\}\{\\partial out\_\{o1\}\} \* \\frac\{\\partial out\_\{o1\}\}\{\\partial net\_\{o1\}\} = 0.74136507 \* 0.186815602 = 0.138498562][frac_partial E_o1_partial net_o1_ _ _frac_partial E_o1_partial out_o1_ _ _frac_partial out_o1_partial net_o1_ _ 0.74136507 _ 0.186815602 _ 0.138498562]

And ![\\frac\{\\partial net\_\{o1\}\}\{\\partial out\_\{h1\}\}][frac_partial net_o1_partial out_h1] is equal to ![w\_5][w_5]:

![net\_\{o1\} = w\_5 \* out\_\{h1\} + w\_6 \* out\_\{h2\} + b\_2 \* 1][net_o1_ _ w_5 _ out_h1_ _ w_6 _ out_h2_ _ b_2 _ 1]

![\\frac\{\\partial net\_\{o1\}\}\{\\partial out\_\{h1\}\} = w\_5 = 0.40][frac_partial net_o1_partial out_h1_ _ w_5 _ 0.40]

Plugging them in:

![\\frac\{\\partial E\_\{o1\}\}\{\\partial out\_\{h1\}\} = \\frac\{\\partial E\_\{o1\}\}\{\\partial net\_\{o1\}\} \* \\frac\{\\partial net\_\{o1\}\}\{\\partial out\_\{h1\}\} = 0.138498562 \* 0.40 = 0.055399425][frac_partial E_o1_partial out_h1_ _ _frac_partial E_o1_partial net_o1_ _ _frac_partial net_o1_partial out_h1_ _ 0.138498562 _ 0.40 _ 0.055399425]

Following the same process for ![\\frac\{\\partial E\_\{o2\}\}\{\\partial out\_\{o1\}\}][frac_partial E_o2_partial out_o1], we get:

![\\frac\{\\partial E\_\{o2\}\}\{\\partial out\_\{h1\}\} = -0.019049119][frac_partial E_o2_partial out_h1_ _ -0.019049119]

Therefore:

![\\frac\{\\partial E\_\{total\}\}\{\\partial out\_\{h1\}\} = \\frac\{\\partial E\_\{o1\}\}\{\\partial out\_\{h1\}\} + \\frac\{\\partial E\_\{o2\}\}\{\\partial out\_\{h1\}\} = 0.055399425 + -0.019049119 = 0.036350306][frac_partial E_total_partial out_h1_ _ _frac_partial E_o1_partial out_h1_ _ _frac_partial E_o2_partial out_h1_ _ 0.055399425 _ -0.019049119 _ 0.036350306]

Now that we have ![\\frac\{\\partial E\_\{total\}\}\{\\partial out\_\{h1\}\}][frac_partial E_total_partial out_h1], we need to figure out ![\\frac\{\\partial out\_\{h1\}\}\{\\partial net\_\{h1\}\}][frac_partial out_h1_partial net_h1] and then ![\\frac\{\\partial net\_\{h1\}\}\{\\partial w\}][frac_partial net_h1_partial w] for each weight:

![out\_\{h1\} = \\frac\{1\}\{1+e^\{-net\_\{h1\}\}\}][out_h1_ _ _frac_1_1_e_-net_h1]

![\\frac\{\\partial out\_\{h1\}\}\{\\partial net\_\{h1\}\} = out\_\{h1\}(1 - out\_\{h1\}) = 0.59326999(1 - 0.59326999 ) = 0.241300709][frac_partial out_h1_partial net_h1_ _ out_h1_1 - out_h1_ _ 0.59326999_1 - 0.59326999 _ _ 0.241300709]

We calculate the partial derivative of the total net input to ![h\_1][h_1] with respect to ![w\_1][w_1] the same as we did for the output neuron:

![net\_\{h1\} = w\_1 \* i\_1 + w\_2 \* i\_2 + b\_1 \* 1][net_h1_ _ w_1 _ i_1 _ w_2 _ i_2 _ b_1 _ 1]

![\\frac\{\\partial net\_\{h1\}\}\{\\partial w\_1\} = i\_1 = 0.05][frac_partial net_h1_partial w_1_ _ i_1 _ 0.05]

Putting it all together:

![\\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{1\}\} = 0.036350306 \* 0.241300709 \* 0.05 = 0.000438568][frac_partial E_total_partial w_1_ _ 0.036350306 _ 0.241300709 _ 0.05 _ 0.000438568]

You might also see this written as:

![\\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{1\}\} = (\\sum\\limits\_\{o\}\{\\frac\{\\partial E\_\{total\}\}\{\\partial out\_\{o\}\} \* \\frac\{\\partial out\_\{o\}\}\{\\partial net\_\{o\}\} \* \\frac\{\\partial net\_\{o\}\}\{\\partial out\_\{h1\}\}\}) \* \\frac\{\\partial out\_\{h1\}\}\{\\partial net\_\{h1\}\} \* \\frac\{\\partial net\_\{h1\}\}\{\\partial w\_\{1\}\}][frac_partial E_total_partial w_1_ _ _sum_limits_o_frac_partial E_total_partial out_o_ _ _frac_partial out_o_partial net_o_ _ _frac_partial net_o_partial out_h1_ _ _frac_partial out_h1_partial net_h1_ _ _frac_partial net_h1_partial w_1]

![\\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{1\}\} = (\\sum\\limits\_\{o\}\{\\delta\_\{o\} \* w\_\{ho\}\}) \* out\_\{h1\}(1 - out\_\{h1\}) \* i\_\{1\}][frac_partial E_total_partial w_1_ _ _sum_limits_o_delta_o_ _ w_ho_ _ out_h1_1 - out_h1_ _ i_1]

![\\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{1\}\} = \\delta\_\{h1\}i\_\{1\}][frac_partial E_total_partial w_1_ _ _delta_h1_i_1]

We can now update ![w\_1][w_1]:

![w\_1^\{+\} = w\_1 - \\eta \* \\frac\{\\partial E\_\{total\}\}\{\\partial w\_\{1\}\} = 0.15 - 0.5 \* 0.000438568 = 0.149780716][w_1_ _ w_1 - _eta _ _frac_partial E_total_partial w_1_ _ 0.15 - 0.5 _ 0.000438568 _ 0.149780716]

Repeating this for ![w\_2][w_2], ![w\_3][w_3], and ![w\_4][w_4]

![w\_2^\{+\} = 0.19956143][w_2_ _ 0.19956143]

![w\_3^\{+\} = 0.24975114][w_3_ _ 0.24975114]

![w\_4^\{+\} = 0.29950229][w_4_ _ 0.29950229]

Finally, we’ve updated all of our weights! When we fed forward the 0.05 and 0.1 inputs originally, the error on the network was 0.298371109. After this first round of backpropagation, the total error is now down to 0.291027924. It might not seem like much, but after repeating this process 10,000 times, for example, the error plummets to 0.000035085. At this point, when we feed forward 0.05 and 0.1, the two outputs neurons generate 0.015912196 (vs 0.01 target) and 0.984065734 (vs 0.99 target).

原文地址：https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/

[no shortage of papers]: https://www.google.com/search?q=backpropagation+algorithm
[sign up for my newsletter]: http://www.emergentmind.com/newsletter
[this Github repo]: https://github.com/mattm/simple-neural-network
[Neural Network visualization]: http://www.emergentmind.com/neural-network
[Getting Started with Deep Learning and Python]: http://www.pyimagesearch.com/2014/09/22/getting-started-deep-learning-python
[neural_network _7]: /images/20220714/f84330ecf8944adda7e696208df005ea.png
[neural_network _9]: /images/20220714/1d92457cf5bc4fa59f44f03f7b6826df.png
[some sources]: http://web.cs.swarthmore.edu/~meeden/cs81/s10/BackPropDeriv.pdf
[h_1]: /images/20220714/dda5195b851f4819943f235eb98694cd.png
[net_h1_ _ w_1 _ i_1 _ w_2 _ i_2 _ b_1 _ 1]: https://s0.wp.com/latex.php?latex=net_%7Bh1%7D+%3D+w_1+%2A+i_1+%2B+w_2+%2A+i_2+%2B+b_1+%2A+1&bg=ffffff&fg=404040&s=0
[net_h1_ _ 0.15 _ 0.05 _ 0.2 _ 0.1 _ 0.35 _ 1 _ 0.3775]: https://s0.wp.com/latex.php?latex=net_%7Bh1%7D+%3D+0.15+%2A+0.05+%2B+0.2+%2A+0.1+%2B+0.35+%2A+1+%3D+0.3775&bg=ffffff&fg=404040&s=0
[out_h1_ _ _frac_1_1_e_-net_h1_ _ _frac_1_1_e_-0.3775_ _ 0.593269992]: https://s0.wp.com/latex.php?latex=out_%7Bh1%7D+%3D+%5Cfrac%7B1%7D%7B1%2Be%5E%7B-net_%7Bh1%7D%7D%7D+%3D+%5Cfrac%7B1%7D%7B1%2Be%5E%7B-0.3775%7D%7D+%3D+0.593269992&bg=ffffff&fg=404040&s=0
[h_2]: /images/20220714/a7fa2c398eba4e8da68628d7f90e6bef.png
[out_h2_ _ 0.596884378]: https://s0.wp.com/latex.php?latex=out_%7Bh2%7D+%3D+0.596884378&bg=ffffff&fg=404040&s=0
[o_1]: /images/20220714/33bbd79eb0584dcfb8273c492c518156.png
[net_o1_ _ w_5 _ out_h1_ _ w_6 _ out_h2_ _ b_2 _ 1]: https://s0.wp.com/latex.php?latex=net_%7Bo1%7D+%3D+w_5+%2A+out_%7Bh1%7D+%2B+w_6+%2A+out_%7Bh2%7D+%2B+b_2+%2A+1&bg=ffffff&fg=404040&s=0
[net_o1_ _ 0.4 _ 0.593269992 _ 0.45 _ 0.596884378 _ 0.6 _ 1 _ 1.105905967]: https://s0.wp.com/latex.php?latex=net_%7Bo1%7D+%3D+0.4+%2A+0.593269992+%2B+0.45+%2A+0.596884378+%2B+0.6+%2A+1+%3D+1.105905967&bg=ffffff&fg=404040&s=0
[out_o1_ _ _frac_1_1_e_-net_o1_ _ _frac_1_1_e_-1.105905967_ _ 0.75136507]: https://s0.wp.com/latex.php?latex=out_%7Bo1%7D+%3D+%5Cfrac%7B1%7D%7B1%2Be%5E%7B-net_%7Bo1%7D%7D%7D+%3D+%5Cfrac%7B1%7D%7B1%2Be%5E%7B-1.105905967%7D%7D+%3D+0.75136507&bg=ffffff&fg=404040&s=0
[o_2]: /images/20220714/03fd46986ed1481f9041f0736ae03d4f.png
[out_o2_ _ 0.772928465]: https://s0.wp.com/latex.php?latex=out_%7Bo2%7D+%3D+0.772928465&bg=ffffff&fg=404040&s=0
[squared error function]: http://en.wikipedia.org/wiki/Backpropagation#Derivation
[E_total_ _ _sum _frac_1_2_target - output_2]: https://s0.wp.com/latex.php?latex=E_%7Btotal%7D+%3D+%5Csum+%5Cfrac%7B1%7D%7B2%7D%28target+-+output%29%5E%7B2%7D&bg=ffffff&fg=404040&s=0
[Some sources]: http://www.amazon.com/Introduction-Math-Neural-Networks-Heaton-ebook/dp/B00845UQL6/ref=sr_1_1?ie=UTF8&qid=1426296804&sr=8-1&keywords=neural+network
[frac_1_2]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B1%7D%7B2%7D&bg=D5E9F6&fg=404040&s=0
[E_o1_ _ _frac_1_2_target_o1_ - out_o1_2_ _ _frac_1_2_0.01 - 0.75136507_2_ _ 0.274811083]: https://s0.wp.com/latex.php?latex=E_%7Bo1%7D+%3D+%5Cfrac%7B1%7D%7B2%7D%28target_%7Bo1%7D+-+out_%7Bo1%7D%29%5E%7B2%7D+%3D+%5Cfrac%7B1%7D%7B2%7D%280.01+-+0.75136507%29%5E%7B2%7D+%3D+0.274811083&bg=ffffff&fg=404040&s=0
[E_o2_ _ 0.023560026]: https://s0.wp.com/latex.php?latex=E_%7Bo2%7D+%3D+0.023560026&bg=ffffff&fg=404040&s=0
[E_total_ _ E_o1_ _ E_o2_ _ 0.274811083 _ 0.023560026 _ 0.298371109]: https://s0.wp.com/latex.php?latex=E_%7Btotal%7D+%3D+E_%7Bo1%7D+%2B+E_%7Bo2%7D+%3D+0.274811083+%2B+0.023560026+%3D+0.298371109&bg=ffffff&fg=404040&s=0
[w_5]: /images/20220714/31b1b98098104755bd1c57093da3808b.png
[frac_partial E_total_partial w_5]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B5%7D%7D&bg=ffffff&fg=404040&s=0
[frac_partial E_total_partial w_5 1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B5%7D%7D&bg=D5E9F6&fg=404040&s=0
[E_total]: https://s0.wp.com/latex.php?latex=E_%7Btotal%7D&bg=D5E9F6&fg=404040&s=0
[w_5 1]: /images/20220714/d7c4a7c66fb84f548d638410e0ae75b8.png
[chain rule]: http://en.wikipedia.org/wiki/Chain_rule
[frac_partial E_total_partial w_5_ _ _frac_partial E_total_partial out_o1_ _ _frac_partial out_o1_partial net_o1_ _ _frac_partial net_o1_partial w_5]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B5%7D%7D+%3D+%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+out_%7Bo1%7D%7D+%2A+%5Cfrac%7B%5Cpartial+out_%7Bo1%7D%7D%7B%5Cpartial+net_%7Bo1%7D%7D+%2A+%5Cfrac%7B%5Cpartial+net_%7Bo1%7D%7D%7B%5Cpartial+w_%7B5%7D%7D&bg=ffffff&fg=404040&s=2
[output_1_backprop _4]: /images/20220714/89cabeb82f644f0bb310830c5594f6ba.png
[E_total_ _ _frac_1_2_target_o1_ - out_o1_2_ _ _frac_1_2_target_o2_ - out_o2_2]: https://s0.wp.com/latex.php?latex=E_%7Btotal%7D+%3D+%5Cfrac%7B1%7D%7B2%7D%28target_%7Bo1%7D+-+out_%7Bo1%7D%29%5E%7B2%7D+%2B+%5Cfrac%7B1%7D%7B2%7D%28target_%7Bo2%7D+-+out_%7Bo2%7D%29%5E%7B2%7D&bg=ffffff&fg=404040&s=0
[frac_partial E_total_partial out_o1_ _ 2 _ _frac_1_2_target_o1_ - out_o1_2 - 1_ _ -1 _ 0]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+out_%7Bo1%7D%7D+%3D+2+%2A+%5Cfrac%7B1%7D%7B2%7D%28target_%7Bo1%7D+-+out_%7Bo1%7D%29%5E%7B2+-+1%7D+%2A+-1+%2B+0&bg=ffffff&fg=404040&s=0
[frac_partial E_total_partial out_o1_ _ -_target_o1_ - out_o1_ _ -_0.01 - 0.75136507_ _ 0.74136507]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+out_%7Bo1%7D%7D+%3D+-%28target_%7Bo1%7D+-+out_%7Bo1%7D%29+%3D+-%280.01+-+0.75136507%29+%3D+0.74136507&bg=ffffff&fg=404040&s=0
[-_target - out]: https://s0.wp.com/latex.php?latex=-%28target+-+out%29&bg=D5E9F6&fg=404040&s=0
[out - target]: https://s0.wp.com/latex.php?latex=out+-+target&bg=D5E9F6&fg=404040&s=0
[out_o1]: /images/20220714/13592e9e417f4d258ccbd1e683bf0101.png
[frac_1_2_target_o2_ - out_o2_2]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B1%7D%7B2%7D%28target_%7Bo2%7D+-+out_%7Bo2%7D%29%5E%7B2%7D&bg=D5E9F6&fg=404040&s=0
[derivative of the logistic function]: http://en.wikipedia.org/wiki/Logistic_function#Derivative
[out_o1_ _ _frac_1_1_e_-net_o1]: https://s0.wp.com/latex.php?latex=out_%7Bo1%7D+%3D+%5Cfrac%7B1%7D%7B1%2Be%5E%7B-net_%7Bo1%7D%7D%7D&bg=ffffff&fg=404040&s=0
[frac_partial out_o1_partial net_o1_ _ out_o1_1 - out_o1_ _ 0.75136507_1 - 0.75136507_ _ 0.186815602]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+out_%7Bo1%7D%7D%7B%5Cpartial+net_%7Bo1%7D%7D+%3D+out_%7Bo1%7D%281+-+out_%7Bo1%7D%29+%3D+0.75136507%281+-+0.75136507%29+%3D+0.186815602&bg=ffffff&fg=404040&s=0
[o1]: https://s0.wp.com/latex.php?latex=o1&bg=ffffff&fg=404040&s=0
[frac_partial net_o1_partial w_5_ _ 1 _ out_h1_ _ w_5_1 - 1_ _ 0 _ 0 _ out_h1_ _ 0.593269992]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+net_%7Bo1%7D%7D%7B%5Cpartial+w_%7B5%7D%7D+%3D+1+%2A+out_%7Bh1%7D+%2A+w_5%5E%7B%281+-+1%29%7D+%2B+0+%2B+0+%3D+out_%7Bh1%7D+%3D+0.593269992&bg=ffffff&fg=404040&s=0
[frac_partial E_total_partial w_5_ _ _frac_partial E_total_partial out_o1_ _ _frac_partial out_o1_partial net_o1_ _ _frac_partial net_o1_partial w_5 1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B5%7D%7D+%3D+%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+out_%7Bo1%7D%7D+%2A+%5Cfrac%7B%5Cpartial+out_%7Bo1%7D%7D%7B%5Cpartial+net_%7Bo1%7D%7D+%2A+%5Cfrac%7B%5Cpartial+net_%7Bo1%7D%7D%7B%5Cpartial+w_%7B5%7D%7D&bg=ffffff&fg=404040&s=0
[frac_partial E_total_partial w_5_ _ 0.74136507 _ 0.186815602 _ 0.593269992 _ 0.082167041]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B5%7D%7D+%3D+0.74136507+%2A+0.186815602+%2A+0.593269992+%3D+0.082167041&bg=ffffff&fg=404040&s=0
[delta rule]: http://en.wikipedia.org/wiki/Delta_rule
[frac_partial E_total_partial w_5_ _ -_target_o1_ - out_o1_ _ out_o1_1 - out_o1_ _ out_h1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B5%7D%7D+%3D+-%28target_%7Bo1%7D+-+out_%7Bo1%7D%29+%2A+out_%7Bo1%7D%281+-+out_%7Bo1%7D%29+%2A+out_%7Bh1%7D&bg=D5E9F6&fg=404040&s=0
[frac_partial E_total_partial out_o1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+out_%7Bo1%7D%7D&bg=D5E9F6&fg=404040&s=0
[frac_partial out_o1_partial net_o1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+out_%7Bo1%7D%7D%7B%5Cpartial+net_%7Bo1%7D%7D&bg=D5E9F6&fg=404040&s=0
[frac_partial E_total_partial net_o1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+net_%7Bo1%7D%7D&bg=D5E9F6&fg=404040&s=0
[delta_o1]: /images/20220714/be838946417c410d89ca8329cee89c2d.png
[delta_o1_ _ _frac_partial E_total_partial out_o1_ _ _frac_partial out_o1_partial net_o1_ _ _frac_partial E_total_partial net_o1]: https://s0.wp.com/latex.php?latex=%5Cdelta_%7Bo1%7D+%3D+%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+out_%7Bo1%7D%7D+%2A+%5Cfrac%7B%5Cpartial+out_%7Bo1%7D%7D%7B%5Cpartial+net_%7Bo1%7D%7D+%3D+%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+net_%7Bo1%7D%7D&bg=D5E9F6&fg=404040&s=0
[delta_o1_ _ -_target_o1_ - out_o1_ _ out_o1_1 - out_o1]: https://s0.wp.com/latex.php?latex=%5Cdelta_%7Bo1%7D+%3D+-%28target_%7Bo1%7D+-+out_%7Bo1%7D%29+%2A+out_%7Bo1%7D%281+-+out_%7Bo1%7D%29&bg=D5E9F6&fg=404040&s=0
[frac_partial E_total_partial w_5_ _ _delta_o1_ out_h1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B5%7D%7D+%3D+%5Cdelta_%7Bo1%7D+out_%7Bh1%7D&bg=D5E9F6&fg=404040&s=0
[delta]: /images/20220714/46d76e6faadb4a7c8bc212f076793488.png
[frac_partial E_total_partial w_5_ _ -_delta_o1_ out_h1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B5%7D%7D+%3D+-%5Cdelta_%7Bo1%7D+out_%7Bh1%7D&bg=D5E9F6&fg=404040&s=0
[w_5_ _ w_5 - _eta _ _frac_partial E_total_partial w_5_ _ 0.4 - 0.5 _ 0.082167041 _ 0.35891648]: https://s0.wp.com/latex.php?latex=w_5%5E%7B%2B%7D+%3D+w_5+-+%5Ceta+%2A+%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B5%7D%7D+%3D+0.4+-+0.5+%2A+0.082167041+%3D+0.35891648&bg=ffffff&fg=404040&s=0
[sources]: http://aima.cs.berkeley.edu/
[alpha]: https://s0.wp.com/latex.php?latex=%5Calpha&bg=D5E9F6&fg=404040&s=0
[others use]: https://www4.rgu.ac.uk/files/chapter3%20-%20bp.pdf
[eta]: https://s0.wp.com/latex.php?latex=%5Ceta&bg=D5E9F6&fg=404040&s=0
[epsilon]: https://s0.wp.com/latex.php?latex=%5Cepsilon&bg=D5E9F6&fg=404040&s=0
[w_6]: https://s0.wp.com/latex.php?latex=w_6&bg=ffffff&fg=404040&s=0
[w_7]: https://s0.wp.com/latex.php?latex=w_7&bg=ffffff&fg=404040&s=0
[w_8]: /images/20220714/9de3d52931d84c7793781a6e2259dbad.png
[w_6_ _ 0.408666186]: https://s0.wp.com/latex.php?latex=w_6%5E%7B%2B%7D+%3D+0.408666186&bg=ffffff&fg=404040&s=0
[w_7_ _ 0.511301270]: https://s0.wp.com/latex.php?latex=w_7%5E%7B%2B%7D+%3D+0.511301270&bg=ffffff&fg=404040&s=0
[w_8_ _ 0.561370121]: https://s0.wp.com/latex.php?latex=w_8%5E%7B%2B%7D+%3D+0.561370121&bg=ffffff&fg=404040&s=0
[w_1]: /images/20220714/c9da233b16344c97903a6b6694426fff.png
[w_2]: https://s0.wp.com/latex.php?latex=w_2&bg=ffffff&fg=404040&s=0
[w_3]: https://s0.wp.com/latex.php?latex=w_3&bg=ffffff&fg=404040&s=0
[w_4]: /images/20220714/0c6ecf48969741b8a3512a406fd0ee7a.png
[frac_partial E_total_partial w_1_ _ _frac_partial E_total_partial out_h1_ _ _frac_partial out_h1_partial net_h1_ _ _frac_partial net_h1_partial w_1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B1%7D%7D+%3D+%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D+%2A+%5Cfrac%7B%5Cpartial+out_%7Bh1%7D%7D%7B%5Cpartial+net_%7Bh1%7D%7D+%2A+%5Cfrac%7B%5Cpartial+net_%7Bh1%7D%7D%7B%5Cpartial+w_%7B1%7D%7D&bg=ffffff&fg=404040&s=0
[nn-calculation]: /images/20220714/5a39ddc4dcb2434bb8e3ec1e971d6ea7.png
[nn-calculation 1]: https://matthewmazur.files.wordpress.com/2015/03/nn-calculation.png
[out_h1]: https://s0.wp.com/latex.php?latex=out_%7Bh1%7D&bg=ffffff&fg=404040&s=0
[out_o1 1]: https://s0.wp.com/latex.php?latex=out_%7Bo1%7D&bg=ffffff&fg=404040&s=0
[out_o2]: https://s0.wp.com/latex.php?latex=out_%7Bo2%7D&bg=ffffff&fg=404040&s=0
[frac_partial E_total_partial out_h1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D&bg=ffffff&fg=404040&s=0
[frac_partial E_total_partial out_h1_ _ _frac_partial E_o1_partial out_h1_ _ _frac_partial E_o2_partial out_h1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D+%3D+%5Cfrac%7B%5Cpartial+E_%7Bo1%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D+%2B+%5Cfrac%7B%5Cpartial+E_%7Bo2%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D&bg=ffffff&fg=404040&s=0
[frac_partial E_o1_partial out_h1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Bo1%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D&bg=ffffff&fg=404040&s=0
[frac_partial E_o1_partial out_h1_ _ _frac_partial E_o1_partial net_o1_ _ _frac_partial net_o1_partial out_h1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Bo1%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D+%3D+%5Cfrac%7B%5Cpartial+E_%7Bo1%7D%7D%7B%5Cpartial+net_%7Bo1%7D%7D+%2A+%5Cfrac%7B%5Cpartial+net_%7Bo1%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D&bg=ffffff&fg=404040&s=0
[frac_partial E_o1_partial net_o1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Bo1%7D%7D%7B%5Cpartial+net_%7Bo1%7D%7D&bg=ffffff&fg=404040&s=0
[frac_partial E_o1_partial net_o1_ _ _frac_partial E_o1_partial out_o1_ _ _frac_partial out_o1_partial net_o1_ _ 0.74136507 _ 0.186815602 _ 0.138498562]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Bo1%7D%7D%7B%5Cpartial+net_%7Bo1%7D%7D+%3D+%5Cfrac%7B%5Cpartial+E_%7Bo1%7D%7D%7B%5Cpartial+out_%7Bo1%7D%7D+%2A+%5Cfrac%7B%5Cpartial+out_%7Bo1%7D%7D%7B%5Cpartial+net_%7Bo1%7D%7D+%3D+0.74136507+%2A+0.186815602+%3D+0.138498562&bg=ffffff&fg=404040&s=0
[frac_partial net_o1_partial out_h1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+net_%7Bo1%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D&bg=ffffff&fg=404040&s=0
[frac_partial net_o1_partial out_h1_ _ w_5 _ 0.40]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+net_%7Bo1%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D+%3D+w_5+%3D+0.40&bg=ffffff&fg=404040&s=0
[frac_partial E_o1_partial out_h1_ _ _frac_partial E_o1_partial net_o1_ _ _frac_partial net_o1_partial out_h1_ _ 0.138498562 _ 0.40 _ 0.055399425]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Bo1%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D+%3D+%5Cfrac%7B%5Cpartial+E_%7Bo1%7D%7D%7B%5Cpartial+net_%7Bo1%7D%7D+%2A+%5Cfrac%7B%5Cpartial+net_%7Bo1%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D+%3D+0.138498562+%2A+0.40+%3D+0.055399425&bg=ffffff&fg=404040&s=0
[frac_partial E_o2_partial out_o1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Bo2%7D%7D%7B%5Cpartial+out_%7Bo1%7D%7D&bg=ffffff&fg=404040&s=0
[frac_partial E_o2_partial out_h1_ _ -0.019049119]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Bo2%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D+%3D+-0.019049119&bg=ffffff&fg=404040&s=0
[frac_partial E_total_partial out_h1_ _ _frac_partial E_o1_partial out_h1_ _ _frac_partial E_o2_partial out_h1_ _ 0.055399425 _ -0.019049119 _ 0.036350306]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D+%3D+%5Cfrac%7B%5Cpartial+E_%7Bo1%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D+%2B+%5Cfrac%7B%5Cpartial+E_%7Bo2%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D+%3D+0.055399425+%2B+-0.019049119+%3D+0.036350306&bg=ffffff&fg=404040&s=0
[frac_partial out_h1_partial net_h1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+out_%7Bh1%7D%7D%7B%5Cpartial+net_%7Bh1%7D%7D&bg=ffffff&fg=404040&s=0
[frac_partial net_h1_partial w]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+net_%7Bh1%7D%7D%7B%5Cpartial+w%7D&bg=ffffff&fg=404040&s=0
[out_h1_ _ _frac_1_1_e_-net_h1]: https://s0.wp.com/latex.php?latex=out_%7Bh1%7D+%3D+%5Cfrac%7B1%7D%7B1%2Be%5E%7B-net_%7Bh1%7D%7D%7D&bg=ffffff&fg=404040&s=0
[frac_partial out_h1_partial net_h1_ _ out_h1_1 - out_h1_ _ 0.59326999_1 - 0.59326999 _ _ 0.241300709]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+out_%7Bh1%7D%7D%7B%5Cpartial+net_%7Bh1%7D%7D+%3D+out_%7Bh1%7D%281+-+out_%7Bh1%7D%29+%3D+0.59326999%281+-+0.59326999+%29+%3D+0.241300709&bg=ffffff&fg=404040&s=0
[frac_partial net_h1_partial w_1_ _ i_1 _ 0.05]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+net_%7Bh1%7D%7D%7B%5Cpartial+w_1%7D+%3D+i_1+%3D+0.05&bg=ffffff&fg=404040&s=0
[frac_partial E_total_partial w_1_ _ 0.036350306 _ 0.241300709 _ 0.05 _ 0.000438568]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B1%7D%7D+%3D+0.036350306+%2A+0.241300709+%2A+0.05+%3D+0.000438568&bg=ffffff&fg=404040&s=0
[frac_partial E_total_partial w_1_ _ _sum_limits_o_frac_partial E_total_partial out_o_ _ _frac_partial out_o_partial net_o_ _ _frac_partial net_o_partial out_h1_ _ _frac_partial out_h1_partial net_h1_ _ _frac_partial net_h1_partial w_1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B1%7D%7D+%3D+%28%5Csum%5Climits_%7Bo%7D%7B%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+out_%7Bo%7D%7D+%2A+%5Cfrac%7B%5Cpartial+out_%7Bo%7D%7D%7B%5Cpartial+net_%7Bo%7D%7D+%2A+%5Cfrac%7B%5Cpartial+net_%7Bo%7D%7D%7B%5Cpartial+out_%7Bh1%7D%7D%7D%29+%2A+%5Cfrac%7B%5Cpartial+out_%7Bh1%7D%7D%7B%5Cpartial+net_%7Bh1%7D%7D+%2A+%5Cfrac%7B%5Cpartial+net_%7Bh1%7D%7D%7B%5Cpartial+w_%7B1%7D%7D&bg=D5E9F6&fg=404040&s=0
[frac_partial E_total_partial w_1_ _ _sum_limits_o_delta_o_ _ w_ho_ _ out_h1_1 - out_h1_ _ i_1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B1%7D%7D+%3D+%28%5Csum%5Climits_%7Bo%7D%7B%5Cdelta_%7Bo%7D+%2A+w_%7Bho%7D%7D%29+%2A+out_%7Bh1%7D%281+-+out_%7Bh1%7D%29+%2A+i_%7B1%7D&bg=D5E9F6&fg=404040&s=0
[frac_partial E_total_partial w_1_ _ _delta_h1_i_1]: https://s0.wp.com/latex.php?latex=%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B1%7D%7D+%3D+%5Cdelta_%7Bh1%7Di_%7B1%7D&bg=D5E9F6&fg=404040&s=0
[w_1_ _ w_1 - _eta _ _frac_partial E_total_partial w_1_ _ 0.15 - 0.5 _ 0.000438568 _ 0.149780716]: https://s0.wp.com/latex.php?latex=w_1%5E%7B%2B%7D+%3D+w_1+-+%5Ceta+%2A+%5Cfrac%7B%5Cpartial+E_%7Btotal%7D%7D%7B%5Cpartial+w_%7B1%7D%7D+%3D+0.15+-+0.5+%2A+0.000438568+%3D+0.149780716&bg=ffffff&fg=404040&s=0
[w_2_ _ 0.19956143]: https://s0.wp.com/latex.php?latex=w_2%5E%7B%2B%7D+%3D+0.19956143&bg=ffffff&fg=404040&s=0
[w_3_ _ 0.24975114]: https://s0.wp.com/latex.php?latex=w_3%5E%7B%2B%7D+%3D+0.24975114&bg=ffffff&fg=404040&s=0
[w_4_ _ 0.29950229]: https://s0.wp.com/latex.php?latex=w_4%5E%7B%2B%7D+%3D+0.29950229&bg=ffffff&fg=404040&s=0