Abstract:

This paper presents an unsupervised method to identify internet new words from the large scale web corpus, which combines with an improved Point-wise Mutual Information (PMI), PMIk algorithm, and some basic rules. This method can recognize internet new words with length from 2 to n (n is any number as needed). Experimented based on 257 MB Baidu Tieba corpus, the precision of proposed system achieves 97.39% when the parameter value of PMIk algorithm is equal to 10, and the precision increases 28.79%, compared to PMI method. The results show that proposed system is significant and efficient for detecting new word from the large scale web corpus. Compiling the results of new word discovery into user dictionary and then loading the user dictionary into ICTCLAS (Institute of Computing Technology, Chinese Lexical Analysis System), experimented with 10 KB Baidu Tieba corpus, the precision, the recall and F-measure were promoted 7.93%, 3.73% and 5.91% respectively, compared with ICTCLAS. The result show that new word discovery could improve the performance of segmentation for web corpus significantly.

Key words: new word recognition, unknown word, PMI, improved PMI algorithm, Chinese word segmentation

摘要：

提出一种非监督的新词识别方法。该方法利用互信息(PMI)的改进算法——PMIk算法与少量基本规则相结合, 从大规模语料中自动识别2~n元网络新词(n为发现的新词最大长度, 可以根据需要指定)。基于257 MB的百度贴吧语料实验, 当PMIk方法的参数为10时, 结果精度达到97.39%, 比PMI方法提高28.79%, 实验结果表明, 该新词发现方法能够有效地从大规模网络语料中发现新词。将新词发现结果编纂成用户词典, 加载到汉语词法分析系统ICTCLAS中, 基于10 KB的百度贴吧语料实验, 比加载用户词典前的分词结果准确率、召回率和F值分别提高7.93%, 3.73%和5.91%。实验表明, 通过进行新词发现能有效改善分词系统对网络文本的处理效果。

关键词: 新词识别, 未登录词, 互信息, PMI 改进算法, 中文分词