Learning RNN from scratch (ＲＮＮ神经网络参数推导)

柔光的暖阳◎ 2022-09-22 04:56 151阅读 0赞

从上一篇原创的文章到现在，已经有一年多了，目前终于有一些新的总结分享个大家。

本文主要讲了RNN神经网络的基本原理，并给出了RNN神经网络参数更新的详细推导过程（back propagation），对于想知道RNN的参数是如果推导的，可以仔细阅读本文。

因为时间有限，下面的总结难免有疏漏之处，请大家指正。

本文结合了一个非常经典的RNN的例子代码，进行了详细的说明，RNN的代码以及注释请见：[https://github.com/weixsong/min-char-rnn][https_github.com_weixsong_min-char-rnn]

并且，本文给出了验证为什么在output layer采用sigmoid激活函数的时候应该采用cross entropy error作为cost function。

本文目录：

# 1.Introduction #

# 2.Simple RNN Theory #

# 3. Using RNN to predict next character #

# 4. Loss Function #

## 4.1 Sum of Squared error (Quadratic error) ##

## 4.2 Cross Entropy Error ##

# 5. Forward Propagation #

# 6. Quadratic Error VS Cross Entropy Error #

## 6.1 Derivative of error with regard to the output of output layer ##

## 6.2 Derivative of error with regard to the input of output layer ##

# 7. Error Measure of RNN for Character Prediction #

# 8. Back Propagation #

## 8.1 compute the derivative of error with regard to the output of outputlayer ##

## 8.2 compute the derivative of error with regard to the input of outputlayer ##

## 8.3 compute the derivative of error with regard to the weight betweenhidden layer and output layer ##

## 8.4 compute the derivative of error with regard to the output of hiddenlayer ##

## 8.5 Compute the derivative of error with regard to the input of hiddenlayer ##

## 8.6 compute the derivative of error with regard to the weight between inputlayer and hidden layer ##

## 8.7 compute the derivative of error with regard to the weight betweenhidden layer and hidden layer ##

![Center][]

![Center 1][]

![Center 2][]

![Center 3][]

![Center 4][]

![Center 5][]

![Center 6][]

![Center 7][]

![Center 8][]

![Center 9][]

![Center 10][]

![Center 11][]

[https_github.com_weixsong_min-char-rnn]: https://github.com/weixsong/min-char-rnn
[Center]: /images/20220721/16a01ae63e4745679f6e9b93d25c5a3d.png
[Center 1]: /images/20220721/7cb1ca2d0fd44b109f98fa71357cc3db.png
[Center 2]: /images/20220721/1061707e04d24997a56e85e414f9c6f2.png
[Center 3]: /images/20220721/e21fd9901708447ebbfddd6b83724e75.png
[Center 4]: /images/20220721/364aa550bbe2494594a25816ab71a5cd.png
[Center 5]: /images/20220721/679c7b3113e04c458b93815f2a42d851.png
[Center 6]: /images/20220721/2d3faaff09024a9190dc38c06691f848.png
[Center 7]: /images/20220721/7e18385881fe41d1800572082dab6856.png
[Center 8]: /images/20220721/36cabe4eb3184fc780f9c0afb0b35a17.png
[Center 9]: /images/20220721/6992988d35be4fcda5d1f2fffefd8461.png
[Center 10]: /images/20220721/08ed1ab8c537420b95158fe5a038c5cc.png
[Center 11]: /images/20220721/e1ecb938730e4bf98624442bfa4d6155.png