通俗易懂了解attention机制

落日映苍穹つ 2022-02-09 14:05 366阅读 0赞

**attention其实很简单，比如有翻译:**  
我喜欢游泳->I like swimming  
**那么在翻译的时候可以这样,也就是越靠近相对应的词，我越注意，影响也就越大**  
i = f(0.7(“我”),0.2(“喜欢”)+0.1(“游泳”))  
like = f(0.2(“我”),0.6(“喜欢”)+0.2(“游泳”))  
swimming = f(0.1(“我”),0.2(“喜欢”)+0.7(“游泳”))

**tensorflow又两种attention机制，分别为Bahdanau attention，和LuongAttention**  
**公式如下：**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTEzMTEyOTE_size_16_color_FFFFFF_t_70]  
**代码解释如下：**  
attention部分  
FC = 全连接层dense,期中神经元个数为decode的神经元个数dec\_units  
EO = Encoder output,(batch\_size,states,enc\_hidden)  
H = decoder hidden state\#(batch\_size,state,dec\_hidden),decoder初始化的时候为enc\_state  
X = input to the decoder

**score** = FC(tanh(FC(EO) + FC(H)))\#output shape (batch\_size,states, 1)  
**attention weights** = softmax(score, axis = 1)\#output shape (batch\_size,states, 1),这两步其实就是对EO,H进行加权计算，然后得出每一个encode state(每一个输入的单词)对decoder（输出单词）的影响程度  
**context vector** = sum(attention weights \* EO, axis = 1)\#output shape (batch\_size, enc\_hidden),其中attention weights \* EO的意思就是将states*影响系数，比如state1*0.6,表示输入的第一个单词只有0.6的内容影响当前的输出单词。计算sum后的意思为所有state在算上影响系数以后，总的对当前输出的影响  
**embedding output** = embedding(X)\#output shape (batch\_size, 1, embedding\_dim)  
**merged vector** = concat(embedding output, context vector)\#output shape (batch\_size, 1, embedding\_dim + hidden\_size)将embedding output和context vector拼接在一起

**tensorflow官网keras实现:**  
[https://github.com/tensorflow/tensorflow/blob/r1.13/tensorflow/contrib/eager/python/examples/nmt\_with\_attention/nmt\_with\_attention.ipynb][https_github.com_tensorflow_tensorflow_blob_r1.13_tensorflow_contrib_eager_python_examples_nmt_with_attention_nmt_with_attention.ipynb]

class Decoder(tf.keras.Model):
        def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz):
            super(Decoder, self).__init__()
            self.batch_sz = batch_sz
            self.dec_units = dec_units
            self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
            self.gru = gru(self.dec_units)
            self.fc = tf.keras.layers.Dense(vocab_size)
            
            # used for attention
            self.W1 = tf.keras.layers.Dense(self.dec_units)
            self.W2 = tf.keras.layers.Dense(self.dec_units)
            self.V = tf.keras.layers.Dense(1)
            
        def call(self, x, hidden, enc_output):
            # enc_output shape == (batch_size, max_length, hidden_size)
            
            # hidden shape == (batch_size, hidden size)
            # hidden_with_time_axis shape == (batch_size, 1, hidden size)
            # we are doing this to perform addition to calculate the score
            hidden_with_time_axis = tf.expand_dims(hidden, 1)
            
            # score shape == (batch_size, max_length, 1)
            # we get 1 at the last axis because we are applying tanh(FC(EO) + FC(H)) to self.V
            score = self.V(tf.nn.tanh(self.W1(enc_output) + self.W2(hidden_with_time_axis)))
            
            # attention_weights shape == (batch_size, max_length, 1)
            attention_weights = tf.nn.softmax(score, axis=1)
            
            # context_vector shape after sum == (batch_size, hidden_size)
            context_vector = attention_weights * enc_output
            context_vector = tf.reduce_sum(context_vector, axis=1)
            
            # x shape after passing through embedding == (batch_size, 1, embedding_dim)
            x = self.embedding(x)
            
            # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)
            x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)
            
            # passing the concatenated vector to the GRU
            output, state = self.gru(x)
            
            # output shape == (batch_size * 1, hidden_size)
            output = tf.reshape(output, (-1, output.shape[2]))
            
            # output shape == (batch_size * 1, vocab)
            x = self.fc(output)
            
            return x, state, attention_weights
            
        def initialize_hidden_state(self):
            return tf.zeros((self.batch_sz, self.dec_units))

**训练:**

for epoch in range(EPOCHS):
        start = time.time()
        
        hidden = encoder.initialize_hidden_state()
        total_loss = 0
        
        for (batch, (inp, targ)) in enumerate(dataset):
            loss = 0
            
            with tf.GradientTape() as tape:
                enc_output, enc_hidden = encoder(inp, hidden)
                
                dec_hidden = enc_hidden
                
                dec_input = tf.expand_dims([targ_lang.word2idx['<start>']] * BATCH_SIZE, 1)       
                
                # Teacher forcing - feeding the target as the next input
                for t in range(1, targ.shape[1]):
                    # passing enc_output to the decoder
                    predictions, dec_hidden, _ = decoder(dec_input, dec_hidden, enc_output)
                    
                    loss += loss_function(targ[:, t], predictions)
                    
                    # using teacher forcing
                    dec_input = tf.expand_dims(targ[:, t], 1)
            
            batch_loss = (loss / int(targ.shape[1]))
            
            total_loss += batch_loss
            
            variables = encoder.variables + decoder.variables
            
            gradients = tape.gradient(loss, variables)
            
            optimizer.apply_gradients(zip(gradients, variables))
            
            if batch % 100 == 0:
                print('Epoch {} Batch {} Loss {:.4f}'.format(epoch + 1,
                                                             batch,
                                                             batch_loss.numpy()))

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTEzMTEyOTE_size_16_color_FFFFFF_t_70]: /images/20220206/ca14552fc59e4e3a8d2ca9e87d5d38f2.png
[https_github.com_tensorflow_tensorflow_blob_r1.13_tensorflow_contrib_eager_python_examples_nmt_with_attention_nmt_with_attention.ipynb]: https://github.com/tensorflow/tensorflow/blob/r1.13/tensorflow/contrib/eager/python/examples/nmt_with_attention/nmt_with_attention.ipynb