Java基于stanford-corenlp实现英文词形还原

秒速五厘米 2022-04-18 05:23 372阅读 0赞

本文作者：合肥工业大学 管理学院 钱洋 email：1563178220@qq.com 内容可能有不到之处，欢迎交流。

**未经本人允许禁止转载**。

### 文章目录 ###

*  简介
 *  stanford-corenlp
 *  java程序

# 简介 #

在做英文文本数据分析时，第一步便是词形还原。例如，一段文本中了出现‘options’和‘option’，其实这两个单词表示一个意思，那么在预处理时‘options’和‘option’都处理成‘option’。  
例如，下面给定的文本:

jhend925  https://blog.csdn.net/timo1160139211/article/details/77603141. All 2015 GTIs have heated seats, including the S trim level 
    with no options. I used to own a 2013 335i with the news system and now own 
    a 2011 M3 with the stalk system you talked about. The new system on the 2013 was much better. 
    Ok, yeah, you have to turn it on every time, but it's much easier to adjust speed

进行分割单词与标点符号，并进行词形还原。

# stanford-corenlp #

stanford-corenlp是一款非常强大的自然语言处理工具，准确率很高。我们可以利用这款工具，完成上述操作。其需要的jar包包括：  
![在这里插入图片描述][20181108221652828.png]

# java程序 #

如下，为一个java操作程序：

package com.lda.datadeal;
    
    import java.util.ArrayList;
    import java.util.List;
    import java.util.Properties;
    import edu.stanford.nlp.ling.CoreAnnotations;
    import edu.stanford.nlp.ling.CoreLabel;
    import edu.stanford.nlp.pipeline.Annotation;
    import edu.stanford.nlp.pipeline.StanfordCoreNLP;
    import edu.stanford.nlp.util.CoreMap;
    public class Test { 
    
    	public static void main(String[] args)  { 
    		String aString = "jhend925 https://blog.csdn.net/timo1160139211/article/details/77603141. All 2015 GTIs have heated seats, including the S trim level with no options. I used to own a 2013 335i with the news system and now own a 2011 M3 with the stalk system you talked about. The new system on the 2013 was much better. Ok, yeah, you have to turn it on every time, but it's much easier to adjust speed";
    		List<String> word = getlema(aString);
    		for (int i = 0; i < word.size(); i++) { 
    			System.out.println(word.get(i));
    		}
    	}
    	/** * 词形还原 * @param string:字符串 * @return List<String> 分词、提取词形还原后的结果 * */
    	public static List<String> getlema(String text){ 
    		//单词集合
    		List<String> wordslist = new ArrayList<>();;
    		//StanfordCoreNLP词形还原
    		Properties props = new Properties();  // set up pipeline properties
    		props.put("annotators", "tokenize, ssplit, pos, lemma");   //分词、分句、词性标注和次元信息。
    		StanfordCoreNLP pipeline = new StanfordCoreNLP(props);
    		Annotation document = new Annotation(text);
    		pipeline.annotate(document);
    		List<CoreMap> words = document.get(CoreAnnotations.SentencesAnnotation.class);
    		for(CoreMap word_temp: words) { 
    			for (CoreLabel token: word_temp.get(CoreAnnotations.TokensAnnotation.class)) { 
    				String lema = token.get(CoreAnnotations.LemmaAnnotation.class);  // 获取对应上面word的词元信息，即我所需要的词形还原后的单词
    				wordslist.add(lema);
    			}
    		}
    		return wordslist;
    	}
    }

这里写了一个方法getlema，方法的输入是一串文本，输出是分词、词形还原后的结果。为了方便读者直接的理解，以下将控制台输出的结果显示出来：  
**在实际处理中，可根据getlema方法返回的结果进行再次处理，如去除标点符号、去除停用词、去除URL类型的字符。**

jhend925
    https://blog.csdn.net/timo1160139211/article/details/77603141
    .
    all
    2015
    gti
    have
    heat
    seat
    ,
    include
    the
    s
    trim
    level
    with
    no
    option
    .
    I
    use
    to
    own
    a
    2013
    335us
    with
    the
    news
    system
    and
    now
    own
    a
    2011
    m3
    with
    the
    stalk
    system
    you
    talk
    about
    .
    the
    new
    system
    on
    the
    2013
    be
    much
    better
    .
    Ok
    ,
    yeah
    ,
    you
    have
    to
    turn
    it
    on
    every
    time
    ,
    but
    it
    be
    much
    easier
    to
    adjust
    speed

[20181108221652828.png]: /images/20220418/9145f913b7e74fb1aab00fc8b60df4e7.png