Can ChatGPT Be Served as the Sentiment Expert? An Evaluation of ChatGPT on Sentiment and Metaphor Analysis

doi:10.13209/j.0479-8023.2023.075

Acta Scientiarum Naturalium Universitatis Pekinensis ›› 2024, Vol. 60 ›› Issue (1): 43-52.DOI: 10.13209/j.0479-8023.2023.075

Previous Articles Next Articles

Can ChatGPT Be Served as the Sentiment Expert? An Evaluation of ChatGPT on Sentiment and Metaphor Analysis

ZHANG Yazhou^1,2, WANG Mengyao¹, RONG Lu³, YU Yang¹, ZHAO Dongming⁴, QIN Jing^2,†

1. School of Software Engineering, Zhengzhou University of Light Industry, Zhengzhou 450002 2. School of Nursing, The Hong Kong Polytechnic University, Hong Kong 999077 3. Human Resources Office, Zhengzhou University of Light Industry, Zhengzhou 450002 4. Artificial Intelligence Laboratory, China Mobile Communication Group Tianjin Co, Tianjin 300020

Received:2023-05-17 Revised:2023-07-31 Online:2024-01-20 Published:2024-01-20
Contact: QIN Jing, E-mail: harry.qin(at)polyu.edu.hk

ChatGPT可否充当情感专家？——调查其在情感与隐喻分析的潜力

张亚洲^1,2, 王梦遥¹, 戎璐³, 俞洋¹, 赵东明⁴, 秦璟^2,†

1. 郑州轻工业大学软件学院, 郑州 450002 2. 香港理工大学护理学院, 香港 999077 3. 郑州轻工业大学人事处, 郑州 450002 4. 中国移动通信集团天津有限公司人工智能实验室, 天津 3000201

通讯作者: 秦璟, E-mail: harry.qin(at)polyu.edu.hk
基金资助:
国家自然科学基金青年基金(62006212)、中国博士后科学基金(2023M733907)、信息物理社会可信服务计算教育部重点实验室开放基金(CPSDSC202103)和 Project of Strategic Importance Grant of the Hong Kong Polytechnic University (1-ZE2Q)资助

Abstract

Abstract:

To explore the potential for subjective understanding, the subjectivity and metaphorical nature of ChatGPT, this paper evaluates ChatGPT on five sentiment, humor, and metaphor benchmark datasets and discusses its strengths and limitations on different tasks by comparing it with the most cutting-edge models in the field. In addition, this paper also compares the performance of ChatGPT and humans in sentiment analysis, with gaps of 9.52%, 16.64% and 6.69% in human results on sentiment, humor and metaphor tasks. The results suggest that although ChatGPT achieves the best performance in dialogue generation, it still has potential for improvement in sentiment understanding. Finally, this paper investigates ChatGPT’s sensitivity to cueing templates in an emotion understanding scenario by improving the cueing templates.

Key words: ChatGPT, sentiment analysis, humor detection, metaphor recognition

摘要：

为了探索ChatGPT情感分析能力以及对主观性和隐喻性理解的潜力, 将ChatGPT在5个情感、幽默与隐喻基准数据集上展开评估, 通过与领域内最前沿的模型对比, 讨论其在不同任务上的优势与局限。此外, 还通过对比ChatGPT与人类在情感分析中的性能差别, 发现 ChatGPT在情感、幽默与隐喻任务上与人类结果分别相差9.52%, 16.64%和6.69%。实验结果表明, 尽管ChatGPT在对话生成方面获得最佳表现, 但是其在情感理解方面仍具有改进的潜力。最后, 通过改善提示模板, 调查ChatGPT在情感理解场景下对提示模板的敏感性。

关键词: ChatGPT, 情感分析, 幽默检测, 隐喻识别

ZHANG Yazhou, WANG Mengyao, RONG Lu, YU Yang, ZHAO Dongming, QIN Jing.

Can ChatGPT Be Served as the Sentiment Expert? An Evaluation of ChatGPT on Sentiment and Metaphor Analysis

[J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2024, 60(1): 43-52.

张亚洲, 王梦遥, 戎璐, 俞洋, 赵东明, 秦璟. ChatGPT可否充当情感专家？——调查其在情感与隐喻分析的潜力[J]. 北京大学学报自然科学版, 2024, 60(1): 43-52.

Add to citation manager EndNote|Ris|BibTeX

URL: https://xbna.pku.edu.cn/EN/10.13209/j.0479-8023.2023.075

https://xbna.pku.edu.cn/EN/Y2024/V60/I1/43

[1]	HUANG Jin, XU Shi, CAI Ercong, WU Zhijie, GUO Meimei, ZHU Jia. An Emotion-Cause Pair Extraction Model Based on Multichannel Compact Bilinear Pooling [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2022, 58(1): 21-28.
[2]	WU Liangqing, LIU Qiyuan, ZHANG Dong, WANG Jiancheng, LI Shoushan, ZHOU Guodong. Multimodal Emotion Recognition with Auxiliary Sentiment Information [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2020, 56(1): 75-81.
[3]	LI Xiaojun, SHI Hanxiao, CHEN Nannan, LIU Hong, ZOU Yi. Research on Sentiment Analysis Based on Representation Learning [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2019, 55(1): 105-112.
[4]	YAN Leiming, YAN Luqi, WANG Chaozhi, HE Jiahui, WU Hongyu. Sentence Style Meta Learning for Twitter Classification [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2019, 55(1): 98-104.
[5]	LIU Siye, TIAN Yuan, FENG Yuning, ZHUANG Yulong. Comparison of Tourist Thematic Sentiment Analysis Methods Based on Weibo Data [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2018, 54(4): 687-692.
[6]	Jie JIANG, Rui XIA. Microblog Sentiment Classification via Combining Rule-based and Machine Learning Methods [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2017, 53(2): 247-254.
[7]	Li DONG, Zhongqing WANG, Deyi XIONG. Stock Index Prediction Based on Text Information [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2017, 53(2): 273-278.
[8]	LIU Cuijuan, LIU Zhen, CHAI Yanjie, FANG Hao, LIU Liangping. Research on the Visualization Method of Social Crowd Emotion Based on Microblog Text Data Analysis [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2016, 52(1): 178-186.
[9]	ZHU Zhu,WANG Rong,LI Shoushan,ZHOU Guodong. Recognizing the Ellipsis of Opinion Target in Chinese Text [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2015, 51(2): 315-320.
[10]	OUYANG Chunping,YANG Xiaohua,LEI Longyan,XU Qiang,YU Ying,LIU Zhiming. Multi-strategy Approach for Fine-Grained Sentiment Analysis of Chinese Microblog [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2014, 50(1): 67-72.
[11]	CHEN Qiang,HE Yanxiang,LIU Xule,SUN Songtao,PENG Min,LI Fei. Cross-Language Sentiment Analysis Based on Parser [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2014, 50(1): 55-60.
[12]	HE Feiyan,HE Yanxiang,LIU Nan,LIU Jianbo,PENG Min. A Microblog Short Text Oriented Multi-class Feature Extraction Method of Fine-Grained Sentiment Analysis [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2014, 50(1): 48-54.
[13]	SUN Yan,ZHOU Xueguang,FU Wei. Unsupervised Topic and Sentiment Unification Model for Sentiment Analysis [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2013, 49(1): 102-108.

Can ChatGPT Be Served as the Sentiment Expert? An Evaluation of ChatGPT on Sentiment and Metaphor Analysis

ChatGPT可否充当情感专家？——调查其在情感与隐喻分析的潜力

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 13

Recommended Articles

Metrics