NLP（三十八）使用keras-bert调用ALBERT模型实现文本分类、文本多标签分类、序列标注任务

布满荆棘的人生 2023-01-02 12:25 170阅读 0赞

在系列文章[NLP（三十四）使用keras-bert实现序列标注任务][NLP_keras-bert]、[NLP（三十五）使用keras-bert实现文本多分类任务][NLP_keras-bert 1]、[NLP（三十六）使用keras-bert实现文本多标签分类任务][NLP_keras-bert 2]中，笔者介绍了如何使用keras-bert模块来调用BERT等模型来实现文本分类、文本多标签分类、序列标注任务。  
  在系列文章[NLP（二十二）利用ALBERT实现文本二分类][NLP_ALBERT]、[NLP（二十五）实现ALBERT+Bi-LSTM+CRF模型][NLP_ALBERT_Bi-LSTM_CRF]、[NLP（二十八）多标签文本分类][NLP]中，笔者将ALBERT模型作为特征向量提取工具，实现了文本分类、文本多标签分类、序列标注任务。  
  本文将会介绍如何使用keras-bert调用ALBERT模型实现文本分类、文本多标签分类、序列标注任务，其模型效果比单纯将ALBERT模型作为特征向量提取工具的效果肯定来得好。  
  使用keras-bert调用ALBERT模型实现文本多分类任务的Github项目网址：[https://github.com/percent4/keras\_albert\_text\_classification][https_github.com_percent4_keras_albert_text_classification] 。  
  使用keras-bert调用ALBERT模型实现文本多标签分类任务的Github项目网址：[https://github.com/percent4/keras\_albert\_multi\_label\_cls][https_github.com_percent4_keras_albert_multi_label_cls] 。  
  使用keras-bert调用ALBERT模型实现序列标注任务的Github项目网址：[https://github.com/percent4/keras\_albert\_sequence\_labeling][https_github.com_percent4_keras_albert_sequence_labeling] 。

### 如何使用keras-bert调用ALBERT模型 ###

keras-bert模块的设计之初是为了支持BERT系列模型，它并不支持ALBERT模型。但在开源世界Github中有个项目名为`keras_albert_model`，其网址为：[https://github.com/TinkerMob/keras\_albert\_model][https_github.com_TinkerMob_keras_albert_model]，利用这个项目，我们可以做到让keras-bert支持ALBERT模型。  
  下载该项目中的albert.py脚本，我们使用如下示例代码来调用ALBERT-tiny模型，代码如下：

# -*- coding: utf-8 -*-
    from albert import load_brightmart_albert_zh_checkpoint
    
    model = load_brightmart_albert_zh_checkpoint('albert_xlarge_zh_183k', training=False)
    model.summary()

输出的albert-tiny模型结构如下：

__________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to                     
    ==================================================================================================
    Input-Token (InputLayer)        (None, 512)          0                                            
    __________________________________________________________________________________________________
    Input-Segment (InputLayer)      (None, 512)          0                                            
    __________________________________________________________________________________________________
    Embed-Token (AdaptiveEmbedding) [(None, 512, 312), ( 2744320     Input-Token[0][0]                
    __________________________________________________________________________________________________
    Embed-Segment (Embedding)       (None, 512, 312)     624         Input-Segment[0][0]              
    __________________________________________________________________________________________________
    Embed-Token-Segment (Add)       (None, 512, 312)     0           Embed-Token[0][0]                
                                                                     Embed-Segment[0][0]              
    __________________________________________________________________________________________________
    Embedding-Position (PositionEmb (None, 512, 312)     159744      Embed-Token-Segment[0][0]        
    __________________________________________________________________________________________________
    Embedding-Norm (LayerNormalizat (None, 512, 312)     624         Embedding-Position[0][0]         
    __________________________________________________________________________________________________
    Attention (MultiHeadAttention)  (None, 512, 312)     390624      Embedding-Norm[0][0]             
                                                                     Feed-Forward-Normal[0][0]        
                                                                     Feed-Forward-Normal[1][0]        
                                                                     Feed-Forward-Normal[2][0]        
    __________________________________________________________________________________________________
    Attention-Add-1 (Add)           (None, 512, 312)     0           Embedding-Norm[0][0]             
                                                                     Attention[0][0]                  
    __________________________________________________________________________________________________
    Attention-Normal (LayerNormaliz (None, 512, 312)     624         Attention-Add-1[0][0]            
                                                                     Attention-Add-2[0][0]            
                                                                     Attention-Add-3[0][0]            
                                                                     Attention-Add-4[0][0]            
    __________________________________________________________________________________________________
    Feed-Forward (FeedForward)      (None, 512, 312)     780312      Attention-Normal[0][0]           
                                                                     Attention-Normal[1][0]           
                                                                     Attention-Normal[2][0]           
                                                                     Attention-Normal[3][0]           
    __________________________________________________________________________________________________
    Feed-Forward-Add-1 (Add)        (None, 512, 312)     0           Attention-Normal[0][0]           
                                                                     Feed-Forward[0][0]               
    __________________________________________________________________________________________________
    Feed-Forward-Normal (LayerNorma (None, 512, 312)     624         Feed-Forward-Add-1[0][0]         
                                                                     Feed-Forward-Add-2[0][0]         
                                                                     Feed-Forward-Add-3[0][0]         
                                                                     Feed-Forward-Add-4[0][0]         
    __________________________________________________________________________________________________
    Attention-Add-2 (Add)           (None, 512, 312)     0           Feed-Forward-Normal[0][0]        
                                                                     Attention[1][0]                  
    __________________________________________________________________________________________________
    Feed-Forward-Add-2 (Add)        (None, 512, 312)     0           Attention-Normal[1][0]           
                                                                     Feed-Forward[1][0]               
    __________________________________________________________________________________________________
    Attention-Add-3 (Add)           (None, 512, 312)     0           Feed-Forward-Normal[1][0]        
                                                                     Attention[2][0]                  
    __________________________________________________________________________________________________
    Feed-Forward-Add-3 (Add)        (None, 512, 312)     0           Attention-Normal[2][0]           
                                                                     Feed-Forward[2][0]               
    __________________________________________________________________________________________________
    Attention-Add-4 (Add)           (None, 512, 312)     0           Feed-Forward-Normal[2][0]        
                                                                     Attention[3][0]                  
    __________________________________________________________________________________________________
    Feed-Forward-Add-4 (Add)        (None, 512, 312)     0           Attention-Normal[3][0]           
                                                                     Feed-Forward[3][0]               
    ==================================================================================================
    Total params: 4,077,496
    Trainable params: 0
    Non-trainable params: 4,077,496
    __________________________________________________________________________________________________

有了它，我们可以在keras-bert模块中轻松愉快地调用ALBERT模型了。以下为keras-bert调用ALBERT模型，实现文本多分类、文本多标签分类以及序列标注任务，其代码与之前的keras-bert调用BERT模型的代码大体一致，只不过在加载预训练模型的时候需要`keras_albert_model`项目的帮助。  
  以下将不再给出具体的项目代码，而只是给出keras-bert在调用ALBERT模型时，在不同NLP任务的表现，即模型评估效果。

### 文本多分类 ###

使用Keras和ALBERT实现文本多分类任务，其中对ALBERT进行微调。数据集为sougou小分类数据集。模型参数为batch\_size = 8, maxlen = 300, epoch=3。

*  albert-tiny的模型评估结果

precision   recall    f1-score   support
    
              体育     0.9700    0.9798    0.9749        99
              健康     0.9278    0.9091    0.9184        99
              军事     0.9899    0.9899    0.9899        99
              教育     0.8585    0.9192    0.8878        99
              汽车     1.0000    0.9394    0.9688        99
    
        accuracy                         0.9475       495
       macro avg     0.9492    0.9475    0.9479       495
    weighted avg     0.9492    0.9475    0.9479       495

*  albert\_base\_zh\_additional\_36k\_steps的模型评估结果

precision    recall  f1-score   support
    
              体育     0.9802    1.0000    0.9900        99
              健康     0.9684    0.9293    0.9485        99
              军事     1.0000    0.9899    0.9949        99
              教育     0.8739    0.9798    0.9238        99
              汽车     1.0000    0.9091    0.9524        99
    
        accuracy                         0.9616       495
       macro avg     0.9645    0.9616    0.9619       495
    weighted avg     0.9645    0.9616    0.9619       495

*  lbert\_xlarge\_zh\_183k的模型评估结果

precision    recall  f1-score   support
    
              体育     0.9898    0.9798    0.9848        99
              健康     0.9412    0.9697    0.9552        99
              军事     0.9706    1.0000    0.9851        99
              教育     0.9300    0.9394    0.9347        99
              汽车     0.9892    0.9293    0.9583        99
    
        accuracy                         0.9636       495
       macro avg     0.9642    0.9636    0.9636       495
    weighted avg     0.9642    0.9636    0.9636       495

### 文本多标签 ###

使用采用Keras和ALBERT实现文本多标签分类任务,其中对ALBERT进行微调。以2020语言与智能技术竞赛：事件抽取任务 中的数据作为多分类标签的样例数据，借助多标签分类模型来解决。模型参数为batch\_size = 16, maxlen = 256, epoch=10。

*  albert-tiny的模型评估结果

micro avg     0.9488    0.8606    0.9025      1657
       macro avg     0.9446    0.8084    0.8589      1657
    weighted avg     0.9460    0.8606    0.8955      1657
     samples avg     0.8932    0.8795    0.8799      1657
    
    accuracy:  0.828437917222964
    hamming loss:  0.0031631919482386773

*  albert\_base\_zh\_additional\_36k\_steps的模型评估结果

micro avg     0.9471    0.9294    0.9382      1657
       macro avg     0.9416    0.9105    0.9208      1657
    weighted avg     0.9477    0.9294    0.9362      1657
     samples avg     0.9436    0.9431    0.9379      1657
    
    accuracy:  0.8931909212283045
    hamming loss:  0.0020848310567936736

### 序列标注 ###

使用本项目采用Keras和ALBERT实现序列标注，其中对ALBERT进行微调。数据集为人民日报命名实体识别数据集、时间识别数据集、CLUENER细粒度实体识别数据集。

*  人民日报命名实体识别数据集

1.1 albert-tiny

模型参数：MAX\_SEQ\_LEN=128, BATCH\_SIZE=32, EPOCH=10