命名实体消歧的代码实现

墨蓝 2022-03-02 04:35 139阅读 0赞

将句中识别的实体与知识库中实体进行匹配，解决实体歧义问题。 可利用上下文本相似度进行识别。  
本文准备了两个测试数据集，entity\_list.csv是50个实体，valid\_data.csv是需要消歧的语句。  
结果提交在submit目录中，命名为entity\_disambiguation\_submit.csv。  
格式为：第一列是需要消歧的语句序号，第二列为多个“实体起始位坐标-实体结束位坐标：实体序号”以“|”分隔的字符串。

需要进行实体消歧的语句如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x0MzI2MDMwNDM0_size_16_color_FFFFFF_t_70]  
实体类别如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x0MzI2MDMwNDM0_size_16_color_FFFFFF_t_70 1]

下面讲解具体的实现步骤：  
1.导入数据

import jieba
    import pandas as pd
    
    # TODO：将entity_list.csv中已知实体的名称导入分词词典
    entity_data = pd.read_csv('../data/entity_disambiguation/entity_list.csv', encoding = 'utf-8')
    
    
    # TODO：对每句句子识别并匹配实体 
    valid_data = pd.read_csv('../data/entity_disambiguation/valid_data.csv', encoding = 'gb18030')

1.  建立关键词组  
    将需要进行实体消歧的实体存进keyword\_list

import collections
    
    s = ''
    keyword_list = []
    for i in entity_data['entity_name'].values.tolist():
        s += i + '|'
    for k,v in collections.Counter(s.split('|')).items():
        if v > 1:
            keyword_list.append(k)

1.  生成tfidf矩阵

from sklearn.feature_extraction.text import TfidfVectorizer
    
    train_sentence = []
    for i in entity_data['desc'].values:
        train_sentence.append(' '.join(jieba.cut(i)))
    
    vectorizer = TfidfVectorizer()
    X = vectorizer.fit_transform(train_sentence)

1.  获取包含关键词的句子中关键词所属的entity\_id

import numpy as np
    from sklearn.metrics.pairwise import cosine_similarity
    
    def get_entityid(sentence):
        id_start = 1001 
        a_list = [' '.join(jieba.cut(sentence))]
        res = cosine_similarity(vectorizer.transform(a_list),X)[0]
        top_idx = np.argsort(res)[-1]
        return id_start + top_idx

1.  将计算结果存入文件

row = 0
    result_data = []
    neighbor_sentence = ''
    for sentence in valid_data['sentence']:
        res = [row]
        for keyword in keyword_list:
            if keyword in sentence:
                # 查询关键词在句子中的索引
                k_len = len(keyword)
                ss =''
                for i in range(len(sentence)-k_len+1):
                    if sentence[i:i+k_len] == keyword:
                        s = str(i) + '-' +str(i+k_len) + ':'  # 拿到 x-x:
                        if i > 10 and i + k_len < len(sentence)-9:
                            neighbor_sentence = sentence [i-10:i+k_len+9]
                        elif i < 10:
                            neighbor_sentence = sentence [:20]
                        elif i + k_len > len(sentence)-9:
                            neighbor_sentence = sentence [-20:]
                        s +=  str(get_entityid(neighbor_sentence))   # 拿到 x-x:id
                        ss += s + '|'    # 拿到 x-x:id|x-x:id
                res.append(ss[:-1])  # 拼接成[0, 'x-x:id|x-x:id']
        result_data.append(res)
        row += 1
    pd.DataFrame(result_data).to_csv('../submit/entity_disambiguation_submit.csv', index=False)

7.最后我们可以打印一下保存的结果

pd.read_csv('../submit/entity_disambiguation_submit.csv')

打印结果如下  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x0MzI2MDMwNDM0_size_16_color_FFFFFF_t_70 2]  
格式为： 起始位置-结束位置|实体类别  
我们可以看一下第一条 3-6:1008|109-112:1008|187-190:1008，确实是符合的。  
![在这里插入图片描述][20190319175620453.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x0MzI2MDMwNDM0_size_16_color_FFFFFF_t_70 3]  
在具体的实现过程中，我们可以通过测试数据的准确率去判断所抽取的字符长度是否合理，调整特征的抽取方式来优化准确率。

以上代码及训练数据已上传至github，有兴趣的同学可以[点击这里][Link 1]查看。谢谢！

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x0MzI2MDMwNDM0_size_16_color_FFFFFF_t_70]: /images/20220302/d01f3cbb181147a7940d8f26fda2c11e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x0MzI2MDMwNDM0_size_16_color_FFFFFF_t_70 1]: /images/20220302/7101ea0003b442ffbd09b20e746960a8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x0MzI2MDMwNDM0_size_16_color_FFFFFF_t_70 2]: /images/20220302/e6fe02f196514ee1a03128814a6f214d.png
[20190319175620453.png]: /images/20220302/039aa7c6c1ba42c288613b1dc107cb5e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x0MzI2MDMwNDM0_size_16_color_FFFFFF_t_70 3]: /images/20220302/ca17ae10ab984557b1fe5be546937e99.png
[Link 1]: https://github.com/Maginaaa/NamedEntityDisambiguation