『NLP学习笔记』Pytorch中的DNN训练流程介绍

怼烎@ 2022-08-28 01:48 376阅读 0赞

<table> 
 <tbody> 
  <tr> 
   <td><font>Pytorch中的DNN训练流程介绍！ </font></td> 
  </tr> 
 </tbody> 
</table>

### 文章目录 ###

*  一. What is PyTorch?
 *   *  1.1. Pytorch&TensorFlow
     *  1.2. Overview of the DNN Training Procedure
     *  1.3. Tensor
     *  1.4. PyTorch v.s. NumPy
     *  1.4. How to Calculate Gradient?
     *  1.5. Construct one\_hot
 *  二. Overview of the DNN Training Procedure
 *   *  2.1. Load data(Customize Dataset)
     *  2.2. Define the network structure layer
     *  2.3. Defining loss function
     *  2.4. Defining the optimizer
     *  2.5. Network training and testing
     *  2.6. Network model saving
 *  三. More About PyTorch
 *  四. Practice questions
 *  五. Reference

# 一. What is PyTorch? #

>  *  An open source **machine learning framework.**
>  *  A Python package that provides two high-level features:
>  *  1. Tensor computation (like NumPy) with strong **GPU acceleration**
>  *  2. Deep neural networks built on a **tape-based autograd** system
>  *  Pytorch官网教学：[https://pytorch.org/docs/stable/tensors.html][https_pytorch.org_docs_stable_tensors.html]

## 1.1. Pytorch&TensorFlow ##

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16][]

## 1.2. Overview of the DNN Training Procedure ##

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 1][]

## 1.3. Tensor ##

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 2][]

>  *  **Tensor – Data Type**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 3][]

>  *  **Tensor – Shape of Tensors**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 4][]

>  *  **Tensor – Constructor(构建)**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 5][]

>  *  **Tensor – Operators(tensor.sequeeze操作)**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 6][]

>  *  **Tensor – Operators(tensor.unsequeeze操作)**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 7][]

>  *  **Tensor – Operators(tensor.transpose操作)**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 8][]

>  *  **Tensor – Operators(torch.cat操作)**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 9][]

>  *  **Tensor – Operators(加、减、平方、求和、均值操作)**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 10][]

## 1.4. PyTorch v.s. NumPy ##

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 11][]![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 12][]

>  *  **Tensor – Device**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 13][]![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 14][]

>  *  **Why GPU?**
>  *  Parallel computing：[https://towardsdatascience.com/what-is-a-gpu-and-do-you-need-one-in-deep-learning-718b9597aa0d][https_towardsdatascience.com_what-is-a-gpu-and-do-you-need-one-in-deep-learning-718b9597aa0d]

## 1.4. How to Calculate Gradient? ##

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 15][]

## 1.5. Construct one\_hot ##

>  *  巧妙使用`torch.eye()`方法

torch.eye(n, m=None, out=None)
    # n (int ) – 行数
    # m (int, optional) – 列数.如果为None,则默认为n
    # out (Tensor, optinal) - Output tensor
    # 返回一个二维向量，对角线是1，其它位置都是0。

import torch
    def one_hot(x, class_count):
    	# 第一构造一个[class_count, class_count]的对角线为1的向量
    	# 第二保留label对应的行并返回
    	return torch.eye(class_count)[x,:]
    
    x = [0,2,5,4]
    class_count = 8
    ont_hot(x, class_count)
    # 结果
    # tensor([[1., 0., 0., 0., 0., 0., 0., 0.],
    # [0., 0., 1., 0., 0., 0., 0., 0.],
    # [0., 0., 0., 0., 0., 1., 0., 0.],
    # [0., 0., 0., 0., 1., 0., 0., 0.]])

# 二. Overview of the DNN Training Procedure #

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 16][]

## 2.1. Load data(Customize Dataset) ##

>  *  PyTorch为我们提供的两个Dataset和DataLoader类分别负责可被PyTorch使用的数据集的创建以及向训练传递数据的任务。如果想个性化自己的数据集或者数据传递方式，也可以自己重写子类。

>  *  **读数据第一步创建Dataset对象**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 17][]

>  *  **例子：**

import torch
    from torch.utils.data import Dataset, DataLoader
    
    def one_hot(x, class_count):
    	# 第一构造一个[class_count, class_count]的对角线为1的向量
    	# 第二保留label对应的行并返回
    	return torch.eye(class_count)[x,:]
    
    class MyDataset(torch.utils.data.Dataset):
        def __init__(self, data):
            self.data = data
            self.target = torch.tensor([0, 1, 0, 3, 5, 3, 8, 4, 9, 0, 5, 7])
            self.one_hot = one_hot(self.target, 10)
    #         print(self.one_hot)
            
        def __getitem__(self, index):
            return data[index], self.one_hot[index]
        def __len__(self):
            return len(self.data)
    
    data = "zhangkaifang"
    datasets = MyDataset(data)
    
    # print(list(dataloader))
    for batch_data in dataloader:
        print(batch_data)

[('h', 'a', 'n', 'g'), tensor([[0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
            [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.]])]
    [('z', 'i', 'a', 'a'), tensor([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
            [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
            [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])]
    [('f', 'n', 'g', 'k'), tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
            [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
            [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.]])]

>  *  **把刚才创建的dataset对象丢进DataLoader**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 18][]

自定义类大致是这样的：
    class CustomDataset(data.Dataset):#需要继承data.Dataset
        def __init__(self):
            # TODO
            # 1. Initialize file path or list of file names.
            pass
        def __getitem__(self, index):
            # TODO
            # 1. Read one data from file (e.g. using numpy.fromfile, PIL.Image.open).
            # 2. Preprocess the data (e.g. torchvision.Transform).
            # 3. Return a data pair (e.g. image and label).
            #这里需要注意的是，第一步：read one data，是一个data
            pass
        def __len__(self):
            # You should change 0 to the total size of your dataset.
            return 0

>  *  **例子1：** 自己实验中写的一个例子：这里我们的图片文件储存在`“./data/faces/”`文件夹下，图片的名字并不是从1开始，而是从`final_train_tag_dict.txt`这个文件保存的字典中读取，label信息也是用这个文件中读取。大家可以照着上面的注释阅读这段代码。

from torch.utils import data
    import numpy as np
    from PIL import Image
    
    class face_dataset(data.Dataset):
        def __init__(self):
            self.file_path = './data/faces/'
            f=open("final_train_tag_dict.txt","r")
            self.label_dict=eval(f.read())
            f.close()
    
        def __getitem__(self,index):
            label = list(self.label_dict.values())[index-1]
            img_id = list(self.label_dict.keys())[index-1]
            img_path = self.file_path+str(img_id)+".jpg"
            img = np.array(Image.open(img_path))
            return img,label
    
        def __len__(self):
            return len(self.label_dict)

>  *  **例子2：** **NLP中文本数据的构建。**
>  *  Pytorch-Bert预训练模型的使用（调用transformers：[https://www.cnblogs.com/cxq1126/p/13517394.html][https_www.cnblogs.com_cxq1126_p_13517394.html]
>  *  Transformers包tokenizer.encode()方法源码阅读笔记：[https://blog.csdn.net/qq\_33293040/article/details/105439750][https_blog.csdn.net_qq_33293040_article_details_105439750]

class CustomDataset(Dataset):
    
        def __init__(self, dataframe, tokenizer, max_len):
            self.tokenizer = tokenizer
            self.data = dataframe
            self.comment_text = dataframe.comment_text
            self.targets = self.data.list
            self.max_len = max_len
    
        def __len__(self):
            return len(self.comment_text)
    
        def __getitem__(self, index):
            comment_text = str(self.comment_text[index])[:self.max_len + 100]
            inputs = self.tokenizer.encode_plus(
                comment_text,
                None,
                truncation=True,
                add_special_tokens=True,
                max_length=self.max_len,
                padding='max_length',
                return_token_type_ids=True
            )
            ids = inputs['input_ids']
            mask = inputs['attention_mask']
            token_type_ids = inputs["token_type_ids"]
    
            return {
        
                'ids': torch.tensor(ids, dtype=torch.long),
                'mask': torch.tensor(mask, dtype=torch.long),
                'token_type_ids': torch.tensor(token_type_ids, dtype=torch.long),
                'targets': torch.tensor(self.targets[index], dtype=torch.float)
            }

## 2.2. Define the network structure layer ##

>  *  **Linear Layer (Fully-connected Layer)**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 19][]![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 20][]

>  *  **例如下面的网络结构：**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 21][]

>  *  **torch.nn – Build your own neural network：**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 22][]

>  *  **激活函数：**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 23][]

## 2.3. Defining loss function ##

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 24][]

## 2.4. Defining the optimizer ##

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 25][]

## 2.5. Network training and testing ##

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 26][]

>  *  **Neural Network Evaluation (Testing Set)**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 27][]

>  *  **Neural Network Evaluation (Validation Set)**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 28][]

>  *  **Neural Network Evaluation (Testing Set)**

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 29][]

## 2.6. Network model saving ##

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 30][]

# 三. More About PyTorch #

>  *  **torchaudio**：speech/audio processing
>  *  **torchtext**：natural language processing
>  *  **torchvision**：computer vision
>  *  **skorch**：scikit-learn + pyTorch

>  *  **Useful github repositories using PyTorch**
>  *  [Huggingface Transformers][] (**transformer models: BERT, GPT, …**)
>  *  [Fairseq][](sequence modeling for **NLP & speech**)
>  *  [ESPnet][] (speech recognition, translation, synthesis, …)
>  *  Many implementation of papers

# 四. Practice questions #

![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 31][]![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 32][]![watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 33][]

# 五. Reference #

>  *  HUNG-YI LEE (李宏毅)：[https://speech.ee.ntu.edu.tw/~hylee/ml/2021-spring.html][https_speech.ee.ntu.edu.tw_hylee_ml_2021-spring.html]
>  *  [https://pytorch.org/][https_pytorch.org]
>  *  [https://github.com/pytorch/pytorch][https_github.com_pytorch_pytorch]
>  *  [https://github.com/wkentaro/pytorch-for-numpy-users][https_github.com_wkentaro_pytorch-for-numpy-users]
>  *  [https://blog.udacity.com/2020/05/pytorch-vs-tensorflow-what-you-need-to-know.html][https_blog.udacity.com_2020_05_pytorch-vs-tensorflow-what-you-need-to-know.html]
>  *  [https://www.tensorflow.org/][https_www.tensorflow.org]
>  *  [https://numpy.org/][https_numpy.org]

[https_pytorch.org_docs_stable_tensors.html]: https://pytorch.org/docs/stable/tensors.html
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16]: /images/20220828/4933a1a2958e4f76b1fa2f39d5bc574e.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 1]: /images/20220828/66611edb569f4dfdae00020d6fa6c0dc.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 2]: /images/20220828/138c476b86184cacac6bb09e1ee6a618.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 3]: /images/20220828/afb818e1de334d268f49e2f1914c0858.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 4]: /images/20220828/e95dae7b11db42aa977eef07e7faeabf.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 5]: /images/20220828/495edef81908484c9945c49fb57e25b2.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 6]: /images/20220828/c34d6fc0337249e28ff6d9c44f6004ef.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 7]: /images/20220828/724bca3d01df4af6b10d93603ca22ed7.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 8]: /images/20220828/af33d7399d834d9d8910644be2c97843.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 9]: /images/20220828/ca513abb91af4c918ed7f82c3c33e73d.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 10]: /images/20220828/400dac6ef9084db3b145ce0372c8e9e3.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 11]: https://img-blog.csdnimg.cn/cb23230c69654205b127b854804fad4e.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBA5biD6KGj5bCP5byg,size_20,color_FFFFFF,t_70,g_se,x_16
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 12]: /images/20220828/84ebf7a4d1d04f648b2141dc5c687e2a.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 13]: https://img-blog.csdnimg.cn/1ce08b1beb8f449d8594a7e5f91f3682.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBA5biD6KGj5bCP5byg,size_20,color_FFFFFF,t_70,g_se,x_16
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 14]: /images/20220828/0e3b86dcbfb24fdcb043e8eaf6a3e9e7.png
[https_towardsdatascience.com_what-is-a-gpu-and-do-you-need-one-in-deep-learning-718b9597aa0d]: https://towardsdatascience.com/what-is-a-gpu-and-do-you-need-one-in-deep-learning-718b9597aa0d
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 15]: /images/20220828/e3449488e34b45cb8ec369d2af2206ae.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 16]: /images/20220828/c3b125767c844aa88a95ff5babe67273.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 17]: /images/20220828/8c14af00876b44d3bdd0cb692bb3dd50.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 18]: /images/20220828/4b3593086c2944459f5bcb3f67e92dbb.png
[https_www.cnblogs.com_cxq1126_p_13517394.html]: https://www.cnblogs.com/cxq1126/p/13517394.html
[https_blog.csdn.net_qq_33293040_article_details_105439750]: https://blog.csdn.net/qq_33293040/article/details/105439750
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 19]: https://img-blog.csdnimg.cn/57cb13b7682b4679b7881b1f19641779.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBA5biD6KGj5bCP5byg,size_20,color_FFFFFF,t_70,g_se,x_16
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 20]: /images/20220828/ac4dda3c8fca4d8facb7385949b66e17.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 21]: /images/20220828/b77afdf9a5c44c98935b19a06c0c73b1.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 22]: /images/20220828/c2f56b27a2cb416fa736f6c236820247.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 23]: /images/20220828/a699b1539ba24388862be470b1cd37f2.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 24]: /images/20220828/adf00232b2c34ed5bc63f9d9fc44da54.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 25]: /images/20220828/af5a612c30584ea589e5c879b21b30ad.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 26]: /images/20220828/6f48ba7c12c1495ca861c62a83de3ca0.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 27]: /images/20220828/cefa737a05fb44f5b8d06363c038d25d.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 28]: /images/20220828/ef85a9b128104ba6bdec2fd2ee894c84.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 29]: /images/20220828/f20aa37f17e946a194c73fa93cd4d076.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 30]: /images/20220828/5274df03ed9b45f694c6fbe08ff7dd94.png
[Huggingface Transformers]: https://github.com/huggingface/transformers
[Fairseq]: https://github.com/pytorch/fairseq
[ESPnet]: https://github.com/espnet/espnet
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 31]: https://img-blog.csdnimg.cn/c190776773af4b8bb48d0e09053052c2.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBA5biD6KGj5bCP5byg,size_20,color_FFFFFF,t_70,g_se,x_16
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 32]: https://img-blog.csdnimg.cn/ce48720d43d545509093400f09090604.png?x-oss-process=image/watermark,type_ZHJvaWRzYW5zZmFsbGJhY2s,shadow_50,text_Q1NETiBA5biD6KGj5bCP5byg,size_20,color_FFFFFF,t_70,g_se,x_16
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA5biD6KGj5bCP5byg_size_20_color_FFFFFF_t_70_g_se_x_16 33]: /images/20220828/2bded7b6fdbe4858a8bbc53bee5c8140.png
[https_speech.ee.ntu.edu.tw_hylee_ml_2021-spring.html]: https://speech.ee.ntu.edu.tw/~hylee/ml/2021-spring.html
[https_pytorch.org]: https://pytorch.org/
[https_github.com_pytorch_pytorch]: https://github.com/pytorch/pytorch
[https_github.com_wkentaro_pytorch-for-numpy-users]: https://github.com/wkentaro/pytorch-for-numpy-users
[https_blog.udacity.com_2020_05_pytorch-vs-tensorflow-what-you-need-to-know.html]: https://blog.udacity.com/2020/05/pytorch-vs-tensorflow-what-you-need-to-know.html
[https_www.tensorflow.org]: https://www.tensorflow.org/
[https_numpy.org]: https://numpy.org/