NLP相关分词和预处理工具包

不念不忘少年蓝@ 2022-02-13 07:37 271阅读 0赞

### jieba ###

Python下中文分词工具  
地址：[https://github.com/fxsjy/jieba][https_github.com_fxsjy_jieba] , 安装：pip install jieba  
功能：中文分词、词性标注、TF-IDF和TextRank关键词抽取

### ansj ###

Java下中文分词工具  
地址：[https://github.com/NLPchina/ansj\_seg][https_github.com_NLPchina_ansj_seg] ，maven使用：

功能：中文分词、词性标注

### hanlp处理包 ###

支持Python和Java  
Python版：[https://github.com/hankcs/pyhanlp][https_github.com_hankcs_pyhanlp] ， 安装：pip install pyhanlp  
Java版：[https://github.com/hankcs/HanLP][https_github.com_hankcs_HanLP] ，maven使用：

<dependency>
        	<groupId>com.hankcs</groupId>
        	<artifactId>hanlp</artifactId>
        	<version>portable-1.7.3</version>
    	</dependency>

功能：中文分词、词性标注、依存句法分析、命名实体识别、短语提取、自动摘要、文本分类聚类

### NLPIR中科院分词系统 ###

官方网站：[http://ictclas.nlpir.org/][http_ictclas.nlpir.org]  
GitHub：[https://github.com/NLPIR-team/NLPIR][https_github.com_NLPIR-team_NLPIR]  
Python版： [https://github.com/tsroten/pynlpir][https_github.com_tsroten_pynlpir]  
功能：中文分词、词性标注

### LTP语言技术平台 ###

支持C++、Java和Python  
C++版：[https://github.com/HIT-SCIR/ltp][https_github.com_HIT-SCIR_ltp] ， examples中有实例  
Java版：[https://github.com/HIT-SCIR/ltp4j][https_github.com_HIT-SCIR_ltp4j]  
Python版：[https://github.com/HIT-SCIR/pyltp][https_github.com_HIT-SCIR_pyltp]  
需要加载训练好的模型，下载地址：[http://ltp.ai/download.html][http_ltp.ai_download.html]  
功能：中文分词、词性标注、依存句法分析、语义角色标注、命名实体识别

### THULAC清华词法分析工具 ###

支持C++、Java和Python  
C++版：[https://github.com/thunlp/THULAC][https_github.com_thunlp_THULAC]  
Java版：[https://github.com/thunlp/THULAC-Java][https_github.com_thunlp_THULAC-Java]  
Python版：[https://github.com/thunlp/THULAC-Python][https_github.com_thunlp_THULAC-Python]  
功能：中文分词、词性标注

### StanfordNLP ###

支持Java和Python，支持多种语言  
Java版：[https://github.com/stanfordnlp/CoreNLP][https_github.com_stanfordnlp_CoreNLP]  
Python版：[https://github.com/stanfordnlp/stanfordnlp][https_github.com_stanfordnlp_stanfordnlp]  
功能：中文分词、词性标注、依存句法分析、命名实体识别，支持多种语言

### NLTK自然语言处理工具包 ###

NLTK英文支持很好  
官方网站：[http://www.nltk.org/][http_www.nltk.org]  
GitHub：[https://github.com/nltk/nltk][https_github.com_nltk_nltk]  
功能：英文分词，词性标注等预处理

[https_github.com_fxsjy_jieba]: https://github.com/fxsjy/jieba
[https_github.com_NLPchina_ansj_seg]: https://github.com/NLPchina/ansj_seg
[https_github.com_hankcs_pyhanlp]: https://github.com/hankcs/pyhanlp
[https_github.com_hankcs_HanLP]: https://github.com/hankcs/HanLP
[http_ictclas.nlpir.org]: http://ictclas.nlpir.org/
[https_github.com_NLPIR-team_NLPIR]: https://github.com/NLPIR-team/NLPIR
[https_github.com_tsroten_pynlpir]: https://github.com/tsroten/pynlpir
[https_github.com_HIT-SCIR_ltp]: https://github.com/HIT-SCIR/ltp
[https_github.com_HIT-SCIR_ltp4j]: https://github.com/HIT-SCIR/ltp4j
[https_github.com_HIT-SCIR_pyltp]: https://github.com/HIT-SCIR/pyltp
[http_ltp.ai_download.html]: http://ltp.ai/download.html
[https_github.com_thunlp_THULAC]: https://github.com/thunlp/THULAC
[https_github.com_thunlp_THULAC-Java]: https://github.com/thunlp/THULAC-Java
[https_github.com_thunlp_THULAC-Python]: https://github.com/thunlp/THULAC-Python
[https_github.com_stanfordnlp_CoreNLP]: https://github.com/stanfordnlp/CoreNLP
[https_github.com_stanfordnlp_stanfordnlp]: https://github.com/stanfordnlp/stanfordnlp
[http_www.nltk.org]: http://www.nltk.org/
[https_github.com_nltk_nltk]: https://github.com/nltk/nltk