Abstract: Conditional random field (CRF) model bears a major drawback of low training efficiency for large-scale data processing. A parallel method of conditional random field model training based on MapReduce is proposed to solve the problem. The method designs parallel algorithm for feature selection and parameters estimation of CRF model to achieve a parallel iterative scaling algorithm. Experiments show that the method improves the efficiency and reduces time cost significantly while guaranteeing the training result correctness.

Key words: part-of-speech (POS) tagging, conditional random field (CRF), MapReduce, parallel

摘要： 针对条件随机场模型面对大规模数据传统训练算法单机处理性能不高的问题, 提出一种基于MapReduce框架的条件随机场模型训练并行化方法, 设计了条件随机场模型特征提取及参数估计的并行算法, 实现了迭代缩放算法的并行。实验表明, 所提出的并行化方法在保证训练结果正确性的同时, 大大减少了训练时间, 效率得到较大提升。

关键词: 词性标注, 条件随机场, MapReduce, 并行