机器翻译自动评价中领域知识复述抽取研究

doi:10.13209/j.0479-8023.2017.030

摘要/Abstract

摘要：

针对通用领域语料中抽取的复述在特定领域机器译文自动评价任务的应用中容易出现复述匹配偏差的问题, 提出采用抽取与测试领域相关的复述来提高机器译文自动评价的方法。首先将通用单语训练语料进行聚类, 并利用改进的M-L方法过滤, 得到特定领域训练语料, 然后在训练语料中利用Markov网络模型, 抽取特定领域复述表, 最后将此复述表应用在机器译文自动评价中, 以提高同义词和近义词的匹配精度。在WMT’14 Metrics task和WMT’15 Metrics task数据集上的实验结果表明, 利用领域知识抽取的复述能够增加自动评价方法METEOR和TER与人工评价的相关性。

关键词: 复述, 机器译文自动评价, 语言模型, Markov网络, 文档聚类

Abstract:

Since the paraphrase extracted from the general domain tends to cause paraphrase match deviation in the specific-domain automatic evaluation of machine translation, this paper proposes an approach exploited specific-domain paraphrase related to the test set to enhance automatic evaluation of machine translation. First, the K-means algorithm is utilized to cluster general-domain monolingual corpus, and the specific-domain training data via improved M-L approach is obtained. Then, the specific-domain paraphrase table is extracted from the training data by Markov network model. Finally, the extracted paraphrase table is applied to automatic MT evaluation metrics to improve word match. The experimental results on the dataset of WMT’14 Metrics task and WMT’15 Metrics task show that the METEOR metric and the TER metric using the specific-domain paraphrase table yield better performance than that using the general-domain paraphrase table.

Key words: paraphrase, automatic evaluation of machine translation, language model, Markov network, document clustering

中图分类号:

TP391

张丽林, 李茂西, 肖文艳, 万剑怡, 王明文. 机器翻译自动评价中领域知识复述抽取研究[J]. 北京大学学报自然科学版, 2017, 53(2): 230-238.

Lilin ZHANG, Maoxi LI, Wenyan XIAO, Jianyi WAN, Mingwen WANG. Improve Automatic Evaluation of Machine Translation Using Specific-Domain Paraphrase[J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2017, 53(2): 230-238.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: https://xbna.pku.edu.cn/CN/10.13209/j.0479-8023.2017.030

https://xbna.pku.edu.cn/CN/Y2017/V53/I2/230

图/表 8

参考文献 24

[1]	Papineni K, Roukos S, Ward T, et al. BLEU: a method for automatic evaluation of machine trans-lation // Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Phila-delphia, 2002: 311-318
[2]	Doddington G.Automatic evaluation of machine translation quality using n-gram co-occurrence statistics // Proceedings of the second international conference on Human Language Technology Re-search (HLT’02). San Diego, 2002: 138-145
[3]	Banerjee S, Lavie A.METEOR: an automatic metric for MT evaluation with improved correlation with human judgments // Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summa-rization. Ann Arbor, 2005: 65-72
[4]	Snover M, Dorr B, Schwartz R, et al.A study of translation edit rate with targeted human annotation // Proceedings of Association for Machine Trans-lation in the Americas. Cambridge, 2006: 223-231
[5]	李茂西, 江爱文, 王明文. 基于ListMLE排序学习方法的机器译文自动评价研究. 中文信息学报, 2013, 27(4): 22-29
[6]	Li M, Wang M, Li H, et al.Modeling monolingual character alignment for automatic evaluation of Chinese translation. ACM Transactions on Asian and Low—Resource Language Information Processing, 2016, 15(3): 1-16
[7]	Denkowski M, Lavie A.Meteor universal: language specific translation evaluation for any target language // Proceedings of the Ninth Workshop on Statistical Machine Translation (WMT). Baltimore, 2014: 376-380
[8]	Snover M, Madnani N, Dorr B, et al.TER-Plus: paraphrase, semantic, and alignment enhancements to translation edit rate. Machine Translation, 2009, 23(2): 117-127
[9]	翁贞, 李茂西, 王明文. 利用Markov 网络抽取复述增强机器译文自动评价方法. 中文信息学报, 2015, 29(5): 136-142
[10]	Moore R C, Lewis W.Intelligent selection of lan-guage model training data // Proceedings of the ACL 2010 Conference. Uppsala, 2010: 220-224
[11]	Axelrod A, He X, Gao J.Domain adaptation via pseudo in-domain data selection // Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Edinburgh, 2011: 355-362
[12]	赵世奇, 刘挺, 李生. 复述技术研究. 软件学报, 2009, 20(8): 2124-2137
[13]	李莉, 刘知远, 孙茂松. 基于中英平行专利语料的短语复述自动抽取研究. 中文信息学报, 2013, 27(6): 151-157
[14]	胡金铭, 史晓东, 苏劲松, 等. 引入复述技术的统计机器翻译研究综述. 智能系统学报, 2013, 8(3): 199-207
[15]	苏晨, 张玉洁, 郭振, 等. 使用源语言复述知识改善统计机器翻译性能. 北京大学学报: 自然科学版, 2015, 51(2): 342-348
[16]	Barzilay R, McKeown K R. Extracting paraphrases from a parallel corpus // Proceedings of 39th Annual Meeting of the Association for Computational Ling-uistics. Toulouse, 2001: 50-57
[17]	Bannard C, Callison-Burch C.Paraphrasing with Bilingual Parallel Corpora // Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, Ann Arbor, 2005: 597-604
[18]	Shinyama Y, Sekine S, Sudo K.Automatic para-phrase acquisition from news articles // Proceedings of the second international conference on Human Language Technology Research. 2002: 313-318
[19]	Barzilay R, Lee L.Learning to paraphrase: an unsu-pervised approach using multiple-sequence alignment// Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, 2003: 16-23
[20]	Pavlick E, Ganitkevitch J, Chan T P, et al.Domain-specific paraphrase extraction // Proceedings of the 53rd Annual Meeting of the Association for Com-putational Linguistics and the 7th International Joint Conference on Natural Language Processing. Beijing, 2015: 57-62
[21]	洪欢, 王明文, 万剑怡, 等. 基于迭代方法的多层Markov 网络信息检索模型. 中文信息学报, 2013, 27(5): 122-128
[22]	Bojar O, Buck C, Federmann C, et al.Findings of the 2014 workshop on statistical machine translation // Proceedings of the Ninth Workshop on Statistical Machine Translation. Baltimore, 2014: 12-58
[23]	Bojar O, Chatterjee R, Federmann C, et al.Findings of the 2015 workshop on statistical machine trans-lation // Proceedings of the Tenth Workshop on Statistical Machine Translation. Lisbon, 2015: 1-46
[24]	Zhang L, Weng Z, Xiao W, et al.Extract domain-specific paraphrase from monolingual corpus for automatic evaluation of machine translation // Pro-ceedings of the First Conference on Machine Translation. Berlin, 2016: 511-517

评价级别	方法	de-en	cs-en	fr-en	fi-en	ru-en	Average
系统级别	TER	0.775	0.989	0.952	0.629	0.809	0.831
	TER-Markov	0.775	0.989	0.952	0.629	0.809	0.831
	TER-SD-Markov	0.784	0.989	0.955	0.629	0.802	0.832
	METEOR	0.885	0.952	0.971	0.515	0.789	0.822
	METEOR-Markov	0.913	0.955	0.971	0.488	0.804	0.826
	METEOR-SD-Makov	0.926	0.951	0.975	0.488	0.804	0.829
句子级别	TER	0.270	0.218	0.384	0.326	0.270	0.294
	TER-Markov	0.270	0.218	0.383	0.326	0.270	0.294
	TER-SD-Makov	0.295	0.233	0.392	0.342	0.281	0.308
	METEOR	0.302	0.253	0.397	0.378	0.297	0.325
	METEOR-Markov	0.325	0.272	0.399	0.400	0.313	0.342
	METEOR-SD-Makov	0.333	0.285	0.406	0.417	0.330	0.354

评价级别	方法	de-en	cs-en	fr-en	fi-en	ru-en	Average
系统级别	TER	0.775	0.989	0.952	0.629	0.809	0.831
	TER-Markov	0.775	0.989	0.952	0.629	0.809	0.831
	TER-SD-Markov	0.784	0.989	0.955	0.629	0.802	0.832
	METEOR	0.885	0.952	0.971	0.515	0.789	0.822
	METEOR-Markov	0.913	0.955	0.971	0.488	0.804	0.826
	METEOR-SD-Makov	0.926	0.951	0.975	0.488	0.804	0.829
句子级别	TER	0.270	0.218	0.384	0.326	0.270	0.294
	TER-Markov	0.270	0.218	0.383	0.326	0.270	0.294
	TER-SD-Makov	0.295	0.233	0.392	0.342	0.281	0.308
	METEOR	0.302	0.253	0.397	0.378	0.297	0.325
	METEOR-Markov	0.325	0.272	0.399	0.400	0.313	0.342
	METEOR-SD-Makov	0.333	0.285	0.406	0.417	0.330	0.354

评价级别	方法	en-de	en-cs	en-fr	en-hi	en-ru	Average
系统级别	TER	0.322	0.979	0.955	0.828	0.934	0.803
	TER-Markov	0.322	0.979	0.955	0.828	0.934	0.803
	TER-SD-Makov	0.337	0.976	0.954	0.828	0.934	0.806
	METEOR	0.240	0.979	0.939	0.924	0.932	0.803
	METEOR-Markov	0.226	0.979	0.939	0.924	0.913	0.796
	METEOR-SD- Makov	0.263	0.976	0.940	0.924	0.923	0.805
句子级别	TER	0.210	0.292	0.261	0.183	0.392	0.268
	TER-Markov	0.210	0.292	0.261	0.183	0.392	0.268
	TER-SD-Makov	0.217	0.292	0.270	0.183	0.392	0.271
	METEOR	0.212	0.310	0.274	0.303	0.407	0.301
	METEOR-Markov	0.222	0.310	0.275	0.303	0.422	0.306
	METEOR-SD- Makov	0.238	0.318	0.277	0.303	0.427	0.313^*

评价级别	方法	en-de	en-cs	en-fr	en-hi	en-ru	Average
系统级别	TER	0.322	0.979	0.955	0.828	0.934	0.803
	TER-Markov	0.322	0.979	0.955	0.828	0.934	0.803
	TER-SD-Makov	0.337	0.976	0.954	0.828	0.934	0.806
	METEOR	0.240	0.979	0.939	0.924	0.932	0.803
	METEOR-Markov	0.226	0.979	0.939	0.924	0.913	0.796
	METEOR-SD- Makov	0.263	0.976	0.940	0.924	0.923	0.805
句子级别	TER	0.210	0.292	0.261	0.183	0.392	0.268
	TER-Markov	0.210	0.292	0.261	0.183	0.392	0.268
	TER-SD-Makov	0.217	0.292	0.270	0.183	0.392	0.271
	METEOR	0.212	0.310	0.274	0.303	0.407	0.301
	METEOR-Markov	0.222	0.310	0.275	0.303	0.422	0.306
	METEOR-SD- Makov	0.238	0.318	0.277	0.303	0.427	0.313^*

评级级别	方法	de-en	cs-en	fr-en	fi-en	ru-en	Average
系统级别	TER	0.890	0.914	0.980	0.878	0.910	0.914
	TER-Markov	0.888	0.926	0.977	0.885	0.912	0.918
	TER-SD- Makov	0.907	0.914	0.977	0.865	0.932	0.919
	METEOR	0.726	0.973	0.979	0.929	0.959	0.953
	METEOR-Markov	0.950	0.974	0.978	0.929	0.965	0.959
	METEOR-SD-Makov	0.959	0.974	0.979	0.939	0.963	0.963
句子级别	TER	0.362	0.391	0.359	0.278	0.330	0.348
	TER-Markov	0.358	0.394	0.357	0.297	0.333	0.348
	TER-SD-Makov	0.375	0.391	0.352	0.315	0.340	0.355
	METEOR	0.389	0.406	0.375	0.385	0.358	0.378
	METEOR-Markov	0.421	0.429	0.386	0.393	0.367	0.400
	METEOR-SD-Makov	0.431	0.434	0.376	0.404	0.383	0.406