Tacotron2 模型详解

我会带着你远行 2022-12-11 03:27 256阅读 0赞

![b42f4dd4b83fa9a19a2cd5065e11f4cd.png_shadow][]

### 1 概述 ###

![df289e69c1831fe8f7b2ba6236a8561c.png_shadow][]

Tacotron2是由Google Brain在2017年提出来的一个End-to-End语音合成框架。模型从下到上可以看作由两部分组成：

1.  声谱预测网络：一个Encoder-Attention-Decoder网络，用于将输入的字符序列预测为梅尔频谱的帧序列
2.  声码器（vocoder）：一个WaveNet的修订版，用于将预测的梅尔频谱帧序列产生时域波形

### 2 编码器 ###

![7a72dc494a98319d4b692f69259ec01d.png_shadow][]

Encoder的输入是多个句子，每个句子的基本单位是character，例如

*  英文"hello world"就会被拆成"h e l l o w o r l d"作为输入
 *  中文"你好世界"则会先把拼音标识出来得到"ni hao shi jie"，然后进一步按照声韵母的方式来分割成"n i h ao sh i j ie"，或者直接按照类似英文的方式分割成"n i h a o s h i j i e"

Encoder的具体流程为：

1.  输入的数据维度为`[batch_size, char_seq_length]`
2.  使用512维的Character Embedding，把每个character映射为512维的向量，输出维度为`[batch_size, char_seq_length, 512]`
3.  3个一维卷积，每个卷积包括512个kernel，每个kernel的大小是5\*1（即每次看5个characters）。每做完一次卷积，进行一次BatchNorm、ReLU以及Dropout。输出维度为`[batch_size, char_seq_length, 512]`（为了保证每次卷积的维度不变，因此使用了pad）
4.  上面得到的输出，扔给一个**单层BiLSTM**，隐藏层维度是256，由于这是双向的LSTM，因此最终输出维度是`[batch_size, char_seq_length, 512]`

class Encoder(nn.Module):
        def __init__(self, hparams):
            super(Encoder, self).__init__()
    
            convolutions = []
            for _ in range(hparams.encoder_n_convolutions):
                conv_layer = nn.Sequential(
                    ConvNorm(hparams.encoder_embedding_dim,
                             hparams.encoder_embedding_dim,x
                             kernel_size=hparams.encoder_kernel_size, stride=1,
                             padding=int((hparams.encoder_kernel_size - 1) / 2),
                             dilation=1, w_init_gain='relu'),
                    nn.BatchNorm1d(hparams.encoder_embedding_dim))
                convolutions.append(conv_layer)
            self.convolutions = nn.ModuleList(convolutions)
    
            self.lstm = nn.LSTM(hparams.encoder_embedding_dim,
                                int(hparams.encoder_embedding_dim / 2), 1,
                                batch_first=True, bidirectional=True)

class ConvNorm(torch.nn.Module):
        def __init__(self, in_channels, out_channels, kernel_size=1, stride=1,
                     padding=None, dilation=1, bias=True, w_init_gain='linear'):
            super(ConvNorm, self).__init__()
            if padding is None:
                assert(kernel_size % 2 == 1)
                padding = int(dilation * (kernel_size - 1) / 2)
    
            self.conv = torch.nn.Conv1d(in_channels, out_channels,
                                        kernel_size=kernel_size, stride=stride,
                                        padding=padding, dilation=dilation,
                                        bias=bias)
                                        
            torch.nn.init.xavier_uniform_(
                self.conv.weight, gain=torch.nn.init.calculate_gain(w_init_gain))

### 3 注意力机制 ###

![b048499ff38c92690382bf7d187b03dc.png_shadow][]

上图描述了第一次做attention时的输入和输出。其中， y 0 y\_0 y0是PreNet初始输入`<S>`的编码表示， c 0 c\_0 c0是当前的"注意力上下文"。初始第一步时， y 0 y\_0 y0和 c 0 c\_0 c0都被初始化为全0向量，然后将 y 0 y\_0 y0和 c 0 c\_0 c0拼接起来，得到一个768维的向量 y 0 , c y\_\{0,c\} y0,c，将该向量与attention\_hidden和attention\_cell一起作为LSTMcell的输入（attention\_hidden其实就是LSTMcell的hidden\_state，attention\_cell其实就是LSTMcell的cell\_state）。得到的结果是 h 1 h\_1 h1和attention\_cell，这里没有给attention\_cell单独起名字，主要考虑其是"打酱油"的，因为除了attention\_rnn之外，其它地方没有用到attention\_cell

![b6d4da990134640dcd379d67e6146dd1.png_shadow][]

Attention\_Layer一共接受五个输入：

1.   h 1 h\_1 h1是和mel谱相关的变量
2.   m m m来自source character sequence编码的"记忆"
3.   m ′ m' m′是 m m m通过一个Linear后得到的
4.  attention\_weights\_cat是将历史（上一时刻）的attention\_weights和attention\_weights\_cum拼接得到的
5.  mask全false，基本没用

计算细节如下：

![bca04ba8d50a551579e8055d0f4cbf7c.png_shadow][]

其中最核心的部分即get\_alignment\_energies，这个函数内部引入了位置特征，因此是**混合注意力机制**

混合注意力机制实际上是内容注意力机制（常规的Attention）与位置注意力机制的结合：  
 e i j = s c o r e ( s i − 1 , α i − 1 , h j ) e\_\{ij\}=score(s\_\{i-1\},\\alpha\_\{i-1\},h\_j) eij=score(si−1,αi−1,hj)  
其中， s i − 1 s\_\{i-1\} si−1为之前解码器的隐状态， α i − 1 \\alpha\_\{i-1\} αi−1是之前的注意力权重， h j h\_j hj是第 j j j个编码器隐状态。为其添加偏置 b b b，最终的score函数计算如下：  
 e i j = v a T t a n h ( W s i − 1 + V h j + U f i , j + b ) e\_\{ij\}=v\_a^T\\mathop\{tanh\}(Ws\_\{i-1\}+Vh\_j+Uf\_\{i,j\}+b) eij=vaTtanh(Wsi−1\+Vhj\+Ufi,j\+b)  
其中， v a v\_a va、 W W W、 V V V、 U U U和 b b b均为待训练参数， f i , j f\_\{i,j\} fi,j是之前的注意力权重 α i , j \\alpha\_\{i,j\} αi,j经卷积而得的位置特征（location feature）， f i = F ∗ α i − 1 f\_i=F\*\\alpha\_\{i-1\} fi=F∗αi−1

Tancotron2的注意力机制基本上和混合注意力机制差不多，但稍有不同  
 e i , j = s c o r e ( s i , c α i − 1 , h j ) = v a T t a n h ( W s i + V h j + U f i , j + b ) e\_\{i,j\}=score(s\_i,c\\alpha\_\{i-1\},h\_j)=v\_a^T\\mathop\{tanh\}(Ws\_i+Vh\_j+Uf\_\{i,j\}+b) ei,j=score(si,cαi−1,hj)=vaTtanh(Wsi\+Vhj\+Ufi,j\+b)  
其中， s i s\_i si为当前解码器隐状态而非上一步，偏置 b b b初始化为0，位置特征 f i f\_i fi是用累加注意力权重 c α i c\\alpha\_i cαi卷积而来：  
 f i = F ∗ c α i − 1 c α i = ∑ j = 1 i − 1 α j \\begin\{aligned\} f\_i&=F\*c\\alpha\_\{i-1\}\\\\ c\\alpha\_i&=\\sum\_\{j=1\}^\{i-1\}\\alpha\_j \\end\{aligned\} ficαi=F∗cαi−1=j=1∑i−1αj  
get\_alignment\_energies函数图示如下：

![90a63309469dc049707680570056fe3b.png_shadow][]

图中Location\_Layer的代码如下：

class LocationLayer(nn.Module):
        def __init__(self, attention_n_filters, attention_kernel_size, # 32, 31
                     attention_dim): # 128
            super(LocationLayer, self).__init__()
            padding = int((attention_kernel_size - 1) / 2) # padding=15
            self.location_conv = ConvNorm(2, attention_n_filters,
                                          kernel_size=attention_kernel_size,
                                          padding=padding, bias=False, stride=1,
                                          dilation=1)
            self.location_dense = LinearNorm(attention_n_filters, attention_dim,
                                             bias=False, w_init_gain='tanh')
    
        def forward(self, attention_weights_cat): # [1, 2, 151]
            processed_attention = self.location_conv(attention_weights_cat) # [1, 32, 151]
            processed_attention = processed_attention.transpose(1, 2) # [1, 151, 32]
            processed_attention = self.location_dense(processed_attention) # [1, 151, 128]
            return processed_attention

Attention的代码如下：

class Attention(nn.Module):
        def __init__(self, attention_rnn_dim, embedding_dim, attention_dim,
                     attention_location_n_filters, attention_location_kernel_size):
            super(Attention, self).__init__()
            self.query_layer = LinearNorm(attention_rnn_dim, attention_dim,
                                          bias=False, w_init_gain='tanh')
            self.memory_layer = LinearNorm(embedding_dim, attention_dim, bias=False,
                                           w_init_gain='tanh')
            self.v = LinearNorm(attention_dim, 1, bias=False)
            self.location_layer = LocationLayer(attention_location_n_filters,
                                                attention_location_kernel_size,
                                                attention_dim)
            self.score_mask_value = -float("inf")
    
        def get_alignment_energies(self, query, processed_memory,
                                   attention_weights_cat):
            """
            PARAMS
            ------
            query: decoder output (batch, n_mel_channels * n_frames_per_step)
            processed_memory: processed encoder outputs (B, T_in, attention_dim)
            attention_weights_cat: cumulative and prev. att weights (B, 2, max_time)
    
            RETURNS
            -------
            alignment (batch, max_time)
            """
    
            processed_query = self.query_layer(query.unsqueeze(1))
            processed_attention_weights = self.location_layer(attention_weights_cat)
            energies = self.v(torch.tanh(
                processed_query + processed_attention_weights + processed_memory))
    
            energies = energies.squeeze(-1)
            return energies
    
        def forward(self, attention_hidden_state, memory, processed_memory,
                    attention_weights_cat, mask):
            """
            PARAMS
            ------
            attention_hidden_state: attention rnn last output
            memory: encoder outputs
            processed_memory: processed encoder outputs
            attention_weights_cat: previous and cummulative attention weights
            mask: binary mask for padded data
            """
            alignment = self.get_alignment_energies(
                attention_hidden_state, processed_memory, attention_weights_cat)
    
            if mask is not None:
                alignment.data.masked_fill_(mask, self.score_mask_value)
    
            attention_weights = F.softmax(alignment, dim=1)
            attention_context = torch.bmm(attention_weights.unsqueeze(1), memory)
            attention_context = attention_context.squeeze(1)
    
            return attention_context, attention_weights

### 4 解码器 ###

![2cfbf037f083074fcdb2caa80e593898.png_shadow][]

解码器是一个自回归结构，它从编码的输入序列预测出声谱图，一次预测一帧

1.  上一步预测出的频谱首先传入一个PreNet，它包含两层神经网络，PreNet作为一个信息瓶颈层（bottleneck），对于学习注意力是必要的
2.  PreNet的输出和Attention Context向量拼接在一起，传给一个含有1024个单元的两层LSTM。LSTM的输出再次和Attention Context向量拼接在一起，然后经过一个线性投影来预测目标频谱
3.  最后，目标频谱帧经过一个5层卷积的PostNet（后处理网络），再将该输出和Linear Projection的输出相加（残差连接）作为最终的输出
4.  另一边，LSTM的输出和Attention Context向量拼接在一起，投影成标量后传给sigmoid激活函数，来预测输出序列是否已完成预测

PreNet层的图示及代码如下所示：

![0542ddeb0630c7cb812529ae4acc05c4.png_shadow][]

class LinearNorm(torch.nn.Module):
        def __init__(self, in_dim, out_dim, bias=True, w_init_gain='linear'):
            super(LinearNorm, self).__init__()
            self.linear_layer = torch.nn.Linear(in_dim, out_dim, bias=bias)
    
            torch.nn.init.xavier_uniform_(
                self.linear_layer.weight,
                gain=torch.nn.init.calculate_gain(w_init_gain))
    
        def forward(self, x):
            return self.linear_layer(x)
    
    class Prenet(nn.Module):
        def __init__(self, in_dim, sizes):
            super(Prenet, self).__init__()
            in_sizes = [in_dim] + sizes[:-1]
            self.layers = nn.ModuleList(
                [LinearNorm(in_size, out_size, bias=False)
                 for (in_size, out_size) in zip(in_sizes, sizes)])
    
        def forward(self, x):
            for linear in self.layers:
                x = F.dropout(F.relu(linear(x)), p=0.5, training=True)
            return x

PostNet层的图示及代码如下所示：

![1e0d105e9c2b318f5f9cfffc807abe79.png_shadow][]

class Postnet(nn.Module):
        """Postnet
            - Five 1-d convolution with 512 channels and kernel size 5
        """
    
        def __init__(self, hparams):
            super(Postnet, self).__init__()
            self.convolutions = nn.ModuleList()
    
            self.convolutions.append(
                nn.Sequential(
                    ConvNorm(hparams.n_mel_channels, hparams.postnet_embedding_dim,
                             kernel_size=hparams.postnet_kernel_size, stride=1,
                             padding=int((hparams.postnet_kernel_size - 1) / 2),
                             dilation=1, w_init_gain='tanh'),
                    nn.BatchNorm1d(hparams.postnet_embedding_dim))
            )
    
            for i in range(1, hparams.postnet_n_convolutions - 1):
                self.convolutions.append(
                    nn.Sequential(
                        ConvNorm(hparams.postnet_embedding_dim,
                                 hparams.postnet_embedding_dim,
                                 kernel_size=hparams.postnet_kernel_size, stride=1,
                                 padding=int((hparams.postnet_kernel_size - 1) / 2),
                                 dilation=1, w_init_gain='tanh'),
                        nn.BatchNorm1d(hparams.postnet_embedding_dim))
                )
    
            self.convolutions.append(
                nn.Sequential(
                    ConvNorm(hparams.postnet_embedding_dim, hparams.n_mel_channels,
                             kernel_size=hparams.postnet_kernel_size, stride=1,
                             padding=int((hparams.postnet_kernel_size - 1) / 2),
                             dilation=1, w_init_gain='linear'),
                    nn.BatchNorm1d(hparams.n_mel_channels))
                )
    
        def forward(self, x):
            for i in range(len(self.convolutions) - 1):
                x = F.dropout(torch.tanh(self.convolutions[i](x)), 0.5, self.training)
            x = F.dropout(self.convolutions[-1](x), 0.5, self.training)
    
            return x

从下面Decoder初始化部分可以看出Decoder由prenet，attention\_rnn，attention\_layer，decoder\_rnn，linear\_projection，gate\_layer组成

class Decoder(nn.Module):
        def __init__(self, hparams):
            super(Decoder, self).__init__()
            self.n_mel_channels = hparams.n_mel_channels
            self.n_frames_per_step = hparams.n_frames_per_step
            self.encoder_embedding_dim = hparams.encoder_embedding_dim
            self.attention_rnn_dim = hparams.attention_rnn_dim
            self.decoder_rnn_dim = hparams.decoder_rnn_dim
            self.prenet_dim = hparams.prenet_dim
            self.max_decoder_steps = hparams.max_decoder_steps
            self.gate_threshold = hparams.gate_threshold
            self.p_attention_dropout = hparams.p_attention_dropout
            self.p_decoder_dropout = hparams.p_decoder_dropout
    
            self.prenet = Prenet(
                hparams.n_mel_channels * hparams.n_frames_per_step,
                [hparams.prenet_dim, hparams.prenet_dim])
    
            self.attention_rnn = nn.LSTMCell(
                hparams.prenet_dim + hparams.encoder_embedding_dim,
                hparams.attention_rnn_dim)
    
            self.attention_layer = Attention(
                hparams.attention_rnn_dim, hparams.encoder_embedding_dim,
                hparams.attention_dim, hparams.attention_location_n_filters,
                hparams.attention_location_kernel_size)
    
            self.decoder_rnn = nn.LSTMCell(
                hparams.attention_rnn_dim + hparams.encoder_embedding_dim,
                hparams.decoder_rnn_dim, 1)
    
            self.linear_projection = LinearNorm(
                hparams.decoder_rnn_dim + hparams.encoder_embedding_dim,
                hparams.n_mel_channels * hparams.n_frames_per_step)
    
            self.gate_layer = LinearNorm(
                hparams.decoder_rnn_dim + hparams.encoder_embedding_dim, 1,
                bias=True, w_init_gain='sigmoid')

### 5 总结 ###

Tacotron2模型的完整网络结构：

Tacotron2(
      (embedding): Embedding(148, 512)
      (encoder): Encoder(
        (convolutions): ModuleList(
          (0): Sequential(
            (0): ConvNorm(
              (conv): Conv1d(512, 512, kernel_size=(5,), stride=(1,), padding=(2,))
            )
            (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (1): Sequential(
            (0): ConvNorm(
              (conv): Conv1d(512, 512, kernel_size=(5,), stride=(1,), padding=(2,))
            )
            (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (2): Sequential(
            (0): ConvNorm(
              (conv): Conv1d(512, 512, kernel_size=(5,), stride=(1,), padding=(2,))
            )
            (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (lstm): LSTM(512, 256, batch_first=True, bidirectional=True)
      )
      (decoder): Decoder(
        (prenet): Prenet(
          (layers): ModuleList(
            (0): LinearNorm(
              (linear_layer): Linear(in_features=80, out_features=256, bias=False)
            )
            (1): LinearNorm(
              (linear_layer): Linear(in_features=256, out_features=256, bias=False)
            )
          )
        )
        (attention_rnn): LSTMCell(768, 1024)
        (attention_layer): Attention(
          (query_layer): LinearNorm(
            (linear_layer): Linear(in_features=1024, out_features=128, bias=False)
          )
          (memory_layer): LinearNorm(
            (linear_layer): Linear(in_features=512, out_features=128, bias=False)
          )
          (v): LinearNorm(
            (linear_layer): Linear(in_features=128, out_features=1, bias=False)
          )
          (location_layer): LocationLayer(
            (location_conv): ConvNorm(
              (conv): Conv1d(2, 32, kernel_size=(31,), stride=(1,), padding=(15,), bias=False)
            )
            (location_dense): LinearNorm(
              (linear_layer): Linear(in_features=32, out_features=128, bias=False)
            )
          )
        )
        (decoder_rnn): LSTMCell(1536, 1024, bias=1)
        (linear_projection): LinearNorm(
          (linear_layer): Linear(in_features=1536, out_features=80, bias=True)
        )
        (gate_layer): LinearNorm(
          (linear_layer): Linear(in_features=1536, out_features=1, bias=True)
        )
      )
      (postnet): Postnet(
        (convolutions): ModuleList(
          (0): Sequential(
            (0): ConvNorm(
              (conv): Conv1d(80, 512, kernel_size=(5,), stride=(1,), padding=(2,))
            )
            (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (1): Sequential(
            (0): ConvNorm(
              (conv): Conv1d(512, 512, kernel_size=(5,), stride=(1,), padding=(2,))
            )
            (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (2): Sequential(
            (0): ConvNorm(
              (conv): Conv1d(512, 512, kernel_size=(5,), stride=(1,), padding=(2,))
            )
            (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (3): Sequential(
            (0): ConvNorm(
              (conv): Conv1d(512, 512, kernel_size=(5,), stride=(1,), padding=(2,))
            )
            (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (4): Sequential(
            (0): ConvNorm(
              (conv): Conv1d(512, 80, kernel_size=(5,), stride=(1,), padding=(2,))
            )
            (1): BatchNorm1d(80, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
      )
    )

### Reference ###

*  [Tacotron2 论文 + 代码详解][Tacotron2 _ _]
 *  [The Annotated Tacotron2][]
 *  [Tacotron-2 : Implementation and Experiments][Tacotron-2 _ Implementation and Experiments]
 *  [声谱预测网络Tacotron2][Tacotron2]

[b42f4dd4b83fa9a19a2cd5065e11f4cd.png_shadow]: /images/20221123/6b2d523091944ea484aaeb28c45e0ad4.png
[df289e69c1831fe8f7b2ba6236a8561c.png_shadow]: /images/20221123/0682ee3730cb4a518d3ade3b8988bb3e.png
[7a72dc494a98319d4b692f69259ec01d.png_shadow]: /images/20221123/7e0774d40c9141ecb13385770bd26f30.png
[b048499ff38c92690382bf7d187b03dc.png_shadow]: /images/20221123/df9ff3c52e4b44c6922b6b82d2c863f6.png
[b6d4da990134640dcd379d67e6146dd1.png_shadow]: /images/20221123/29a3a856989d43d8af68588c7a30bf95.png
[bca04ba8d50a551579e8055d0f4cbf7c.png_shadow]: /images/20221123/3dd0ed326f5045a1a864ea2ebf4150a3.png
[90a63309469dc049707680570056fe3b.png_shadow]: /images/20221123/fb2d90b330174668a1cd5f661d81c6c9.png
[2cfbf037f083074fcdb2caa80e593898.png_shadow]: /images/20221123/75faadc548a14d12b4796ed2cb8a0dd9.png
[0542ddeb0630c7cb812529ae4acc05c4.png_shadow]: /images/20221123/1f781cc398484123a84f60013957d246.png
[1e0d105e9c2b318f5f9cfffc807abe79.png_shadow]: /images/20221123/bb09e809544d49909c2306414cf5c575.png
[Tacotron2 _ _]: https://blog.csdn.net/whjkm/article/details/89321954
[The Annotated Tacotron2]: https://zhuanlan.zhihu.com/p/115361376
[Tacotron-2 _ Implementation and Experiments]: https://medium.com/@rajanieprabha/tacotron-2-implementation-and-experiments-832695b1c86e
[Tacotron2]: https://www.cnblogs.com/mengnan/p/9527797.html