NLP - 基于 BERT 的中文命名实体识别（NER)

淡淡的烟草味﹌ 2022-09-07 06:14 242阅读 0赞

序列标注任务是中文[自然语言处理][Link 1]（NLP）领域在句子层面中的主要任务，在给定的文本序列上预测序列中需要作出标注的标签。常见的子任务有[命名实体识别][Link 2]（NER）、Chunk 提取以及[词性标注][Link 3]（POS）等。

[BERT][] 模型刷新了自然语言处理的 11 项记录，成为 NLP 行业的新标杆。既然 Google 开源这么好的模型架构和预训练的中文模型，那我们就使用它构建一个序列标注模型。

PS: 最近我开源了一个极简文本分类和序列标注框架 [Kashgari][],今天的教程将使用这个框架构建模型。如果想了解文本分类，可以看下面的文章。

*  [NLP - 15 分钟搭建中文文本分类模型][NLP - 15]
 *  **2020.5.2 更新：适配 Kashgari 1.x 版本**
 *  **2019.7.12 更新：适配 Kashgari-tf 版本**

## 搭建环境和数据准备 ##

准备工作，先准备 python 环境，下载 BERT 语言模型。

*  Python 3.6 环境
 *  [BERT-Base, Chinese 中文模型][BERT]

虚拟环境中安装所有需要的依赖

<table> 
 <tbody> 
  <tr> 
   <td> <pre>1
2
</pre> </td> 
   <td> <pre>pip install "kashgari&gt;=1.1,&lt;2.0"
pip install "tensorflow&gt;=1.14,&lt;2.0"
</pre> </td> 
  </tr> 
 </tbody> 
</table>

## 读取数据 ##

<table> 
 <tbody> 
  <tr> 
   <td> <pre>1
2
3
4
5
6
7
8
9
10
11
12
</pre> </td> 
   <td> <pre>from kashgari.corpus import ChineseDailyNerCorpus

train_x, train_y = ChineseDailyNerCorpus.load_data('train')
valid_x, valid_y = ChineseDailyNerCorpus.load_data('validate')
test_x, test_y  = ChineseDailyNerCorpus.load_data('test')

print(f"train data count: {len(train_x)}")
print(f"validate data count: {len(valid_x)}")
print(f"test data count: {len(test_x)}")
train data count: 20864
validate data count: 2318
test data count: 4636
</pre> </td> 
  </tr> 
 </tbody> 
</table>

## 创建 BERT embedding ##

<table> 
 <tbody> 
  <tr> 
   <td> <pre>1
2
3
4
5
6
</pre> </td> 
   <td> <pre>import kashgari
from kashgari.embeddings import BERTEmbedding

bert_embed = BERTEmbedding('&lt;PRE_TRAINED_BERT_MODEL_FOLDER&gt;',
                           task=kashgari.LABELING,
                           sequence_length=100)
</pre> </td> 
  </tr> 
 </tbody> 
</table>

## 创建模型并训练 ##

<table> 
 <tbody> 
  <tr> 
   <td> <pre>1
2
3
4
5
6
7
8
9
10
11
</pre> </td> 
   <td> <pre>from kashgari.tasks.labeling import BiLSTM_CRF_Model

# 还可以选择 `CNN_LSTM_Model`, `BiLSTM_Model`, `BiGRU_Model` 或 `BiGRU_CRF_Model`

model = BiLSTM_CRF_Model(bert_embed)
model.fit(train_x,
          train_y,
          x_validate=valid_x,
          y_validate=valid_y,
          epochs=20,
          batch_size=512)
</pre> </td> 
  </tr> 
 </tbody> 
</table>

## 模型评估结果 ##

### 模型训练 ###

测试环境: V100, 2CPU 40G 感谢 [OpenBayes BayesGear 算力容器][OpenBayes BayesGear] 提供算力支持。  
基础参数:

*  batch\_size: 2317
 *  sequence\_length: 100
 *  epochs: 200

F1 score

<table> 
 <thead> 
  <tr> 
   <th></th> 
   <th>CNN-LSTM</th> 
   <th>B-LSTM</th> 
   <th>B-LSTM-CRF</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>bare embedding</td> 
   <td>0.5275</td> 
   <td>0.6569</td> 
   <td>0.6805</td> 
  </tr> 
  <tr> 
   <td>Word2vec</td> 
   <td>0.5042</td> 
   <td>0.6686</td> 
   <td>0.7341</td> 
  </tr> 
  <tr> 
   <td>BERT</td> 
   <td>0.8212</td> 
   <td>0.9043</td> 
   <td>0.9220</td> 
  </tr> 
 </tbody> 
</table>

每 epoch 耗时

<table> 
 <thead> 
  <tr> 
   <th></th> 
   <th>CNN-LSTM</th> 
   <th>B-LSTM</th> 
   <th>B-LSTM-CRF</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>bare embedding</td> 
   <td>4s</td> 
   <td>7s</td> 
   <td>19s</td> 
  </tr> 
  <tr> 
   <td>Word2vec</td> 
   <td>5s</td> 
   <td>7s</td> 
   <td>20s</td> 
  </tr> 
  <tr> 
   <td>BERT</td> 
   <td>40s</td> 
   <td>46s</td> 
   <td>60s</td> 
  </tr> 
 </tbody> 
</table>

最好成绩是 BERT + B-LSTM-CRF 模型效果最好。详细得分如下：

<table> 
 <tbody> 
  <tr> 
   <td> <pre>1
2
3
4
5
6
7
</pre> </td> 
   <td> <pre>             precision    recall  f1-score   support

LOC     0.9208    0.9324    0.9266      3431
        ORG     0.8728    0.8882    0.8804      2147
        PER     0.9622    0.9633    0.9627      1797

avg / total     0.9169    0.9271    0.9220      7375
</pre> </td> 
  </tr> 
 </tbody> 
</table>

### 模型预测 ###

预测环境: MacBook Pro 13, 2 GHz Intel Core i5, 8G RAM

模型初始化耗时

<table> 
 <thead> 
  <tr> 
   <th></th> 
   <th>CNN-LSTM</th> 
   <th>B-LSTM</th> 
   <th>B-LSTM-CRF</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>bare embedding</td> 
   <td>13.535s</td> 
   <td>9.498s</td> 
   <td>8.739s</td> 
  </tr> 
  <tr> 
   <td>Word2vec</td> 
   <td>20.042s</td> 
   <td>14.942s</td> 
   <td>12.553s</td> 
  </tr> 
  <tr> 
   <td>BERT</td> 
   <td>37.952s</td> 
   <td>21.986s</td> 
   <td>24.435s</td> 
  </tr> 
 </tbody> 
</table>

50个句子一次性预测

<table> 
 <thead> 
  <tr> 
   <th></th> 
   <th>CNN-LSTM</th> 
   <th>B-LSTM</th> 
   <th>B-LSTM-CRF</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>bare embedding</td> 
   <td>1.502s</td> 
   <td>1.395s</td> 
   <td>0.869s</td> 
  </tr> 
  <tr> 
   <td>Word2vec</td> 
   <td>1.034s</td> 
   <td>1.901s</td> 
   <td>0.876s</td> 
  </tr> 
  <tr> 
   <td>BERT</td> 
   <td>36.463s</td> 
   <td>31.252s</td> 
   <td>26.601s</td> 
  </tr> 
 </tbody> 
</table>

50个句子循环预测时每个句子预测时间

<table> 
 <thead> 
  <tr> 
   <th></th> 
   <th>CNN-LSTM</th> 
   <th>B-LSTM</th> 
   <th>B-LSTM-CRF</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>bare embedding</td> 
   <td>0.014s</td> 
   <td>0.019s</td> 
   <td>0.035s</td> 
  </tr> 
  <tr> 
   <td>Word2vec</td> 
   <td>0.015s</td> 
   <td>0.019s</td> 
   <td>0.052s</td> 
  </tr> 
  <tr> 
   <td>BERT</td> 
   <td>0.606s</td> 
   <td>0.641s</td> 
   <td>0.573s</td> 
  </tr> 
 </tbody> 
</table>

可以看得出同样的模型结构，BERT 能大幅度提高，但是也会导致训练时间，模型大小和预测时间大幅度上升。如果想在线实时预测可能性能无法达标，得考虑通过缓存之类的方案解决。

[\# NLP][NLP] [\# Sequence Tagging][Sequence Tagging] [\# NER][NER] [\# Kashgari][Kashgari 1]

[ NLP - 15 分钟搭建中文文本分类模型][NLP - 15 1]

[NLP - BERT/ERNIE 文本分类和部署 ][NLP - BERT_ERNIE _]

[Link 1]: https://zh.wikipedia.org/wiki/%E8%87%AA%E7%84%B6%E8%AF%AD%E8%A8%80%E5%A4%84%E7%90%86
[Link 2]: https://nlp.cs.nyu.edu/sekine/papers/li07.pdf
[Link 3]: https://en.wikipedia.org/wiki/Part-of-speech_tagging
[BERT]: https://github.com/google-research/bert
[Kashgari]: https://github.com/BrikerMan/Kashgari
[NLP - 15]: https://eliyar.biz/nlp_chinese_text_classification_in_15mins
[OpenBayes BayesGear]: https://openbayes.com/
[NLP]: https://eliyar.biz/tags/NLP/
[Sequence Tagging]: https://eliyar.biz/tags/Sequence-Tagging/
[NER]: https://eliyar.biz/tags/NER/
[Kashgari 1]: https://eliyar.biz/tags/Kashgari/
[NLP - 15 1]: https://eliyar.biz/nlp_chinese_text_classification_in_15mins/index.html
[NLP - BERT_ERNIE _]: https://eliyar.biz/nlp_train_and_deploy_bert_text_classification/index.html