Reducing Multi-model Biases for Robust Visual Question Answering

doi:10.13209/j.0479-8023.2023.072

Acta Scientiarum Naturalium Universitatis Pekinensis ›› 2024, Vol. 60 ›› Issue (1): 23-33.DOI: 10.13209/j.0479-8023.2023.072

Previous Articles Next Articles

Reducing Multi-model Biases for Robust Visual Question Answering

ZHANG Fengshuo, LI Yu, LI Xiangqian^†, XU Jin’an, CHEN Yufeng

School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044

Received:2023-05-18 Revised:2023-09-26 Online:2024-01-20 Published:2024-01-20
Contact: LI Xiangqian, E-mail: xqli(at)bjtu.edu.cn

一种消减多模态偏见的鲁棒视觉问答方法

张丰硕, 李豫, 李向前^†, 徐金安, 陈钰枫

北京交通大学计算机与信息技术学院, 北京 100044

通讯作者: 李向前, E-mail: xqli(at)bjtu.edu.cn

Abstract

Abstract:

In order to enhance the robustness of the visual question answering model, a bias reduction method is proposed. Based on this, the influence of language and visual information on bias effect is explored. Furthermore, two bias learning branches are constructed to capture the language bias, and the bias caused by both language and images. Then, more robust prediction results are obtained by using the bias reduction method. Finally, based on the difference in prediction probabilities between standard visual question answering and bias branches, samples are dynamically weighted, allowing the model to adjust learning levels for samples with different levels of bias. Experiments on VQA-CP v2.0 and other data sets demonstrate the effectiveness of the proposed method and alleviate the influence of bias on the model.

Key words: visual question answering, dataset bias, language bias, deep learning

摘要：

为了增强视觉问答模型的鲁棒性, 提出一种偏见消减方法, 并在此基础上探究语言与视觉信息对偏见的影响。进一步地, 构造两个偏见学习分支来分别捕获语言偏见以及语言和图片共同导致的偏见, 利用偏见消减方法, 得到鲁棒性更强的预测结果。最后, 依据标准视觉问答与偏见分支之间的预测概率差异, 对样本进行动态赋权, 使模型针对不同偏见程度的样本动态地调节学习程度。在VQA-CP v2.0等数据集上的实验结果证明了所提方法的有效性, 缓解了偏见对模型的影响。

关键词: 视觉问答, 数据集偏差, 语言偏见, 深度学习

ZHANG Fengshuo, LI Yu, LI Xiangqian, XU Jin’an, CHEN Yufeng. Reducing Multi-model Biases for Robust Visual Question Answering[J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2024, 60(1): 23-33.

张丰硕, 李豫, 李向前, 徐金安, 陈钰枫. 一种消减多模态偏见的鲁棒视觉问答方法[J]. 北京大学学报自然科学版, 2024, 60(1): 23-33.

[1]	LI Dai, WANG Tianmu, ZHANG Si, QIN Yue, XIE Fugui, LIU Xinjun, NIE Zhenguo, HUANG Hongshi. Intelligent Diagnosis on Anterior Cruciate Ligament Deficiency Based on Plantar Pressure and ConvLSTM Neural Network [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2024, 60(1): 109-117.
[2]	YAO Zhaoyuan, MA Lei, WAN Wei, SONG Benqin, WANG Weihong, DENG Jiwei, XIAO Lei, JI Rui, WEI Zhihao, CUI Yaokui. Consistency Assessment of Remote Sensing Dataset Based on Deep Learning [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2023, 59(4): 563-568.
[3]	ZHOU Xiangyu, MAO Shanjun, LI Mei. Bearing Fault Diagnosis Method Based on Down-Sampling in Frequency Domain and CNN [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2023, 59(2): 251-260.
[4]	LI Fang, CAO Jian, LI Pu, XIE Hao, ZHAO Xiongbo, WANG Yuan, ZHANG Xing. Design and Implementation of Object Detection Acceleration Module Based on an ARM+FPGA Heterogeneous Platform [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2022, 58(6): 1035-1041.
[5]	LI Tianran, LIU Mingtong, ZHANG Yujie, XU Jin’an, CHEN Yufeng. A Review of Entity Linking Research Based on Deep Learning [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2021, 57(1): 91-98.
[6]	HUANG Depeng, LI Zhenghua, GONG Chen, ZHANG Min. Neural Network Coupled Model for Conversion and Exploitation of Heterogeneous Lexical Annotations [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2020, 56(1): 97-104.
[7]	HUANG Hongshi, WANG Zhengfei, XU Guoxiong, LI Wenxin, ZHANG Si, ZHANG Dongxia, AO Yingfang. Anterior Cruciate Ligament Deficiency Auxiliary Diagnosis Based on Plantar Pressure Information during Walking [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2019, 55(5): 859-864.
[8]	FANG Haiquan, JIANG Yunzhong, YE Yuntao, CAO Yin. River Extraction from High-Resolution Satellite Images Combining Deep Learning and Multiple Chessboard Segmentation [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2019, 55(4): 692-698.
[9]	CAI Zhenyu, GE Zengxi. Using Artificial Intelligence to Pick P-Wave First-Arrival of the Microseisms: Taking the Aftershock Sequence of Wenchuan Earthquake as an Example [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2019, 55(3): 451-460.
[10]	LI Xiaojun, SHI Hanxiao, CHEN Nannan, LIU Hong, ZOU Yi. Research on Sentiment Analysis Based on Representation Learning [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2019, 55(1): 105-112.
[11]	WANG Qiansheng, YU Nan, ZHANG Meishan, HAN Zijia, FU Guohong. N3LDG: A Lightweight Neural Network Library for Natural Language Processing [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2019, 55(1): 113-119.
[12]	LUO Fan, WANG Houfeng. Chinese Text Sentiment Classification by H-RNN-CNN [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2018, 54(3): 459-465.
[13]	WU Huanqin, ZHANG Hongyang, LI Jingmei, ZHU Junguo, YANG Muyun, LI Sheng. Training Machine Translation Quality Estimation Model Based on Pseudo Data [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2018, 54(2): 279-285.
[14]	LIU Kan,YUAN Yunying. Short Texts Feature Extraction and Clustering Based on Auto-Encoder [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2015, 51(2): 282-288.
[15]	XI Xuefeng,ZHOU Guodong. Pronoun Resolution Based on Deep Learning [J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2014, 50(1): 100-110.

Reducing Multi-model Biases for Robust Visual Question Answering

一种消减多模态偏见的鲁棒视觉问答方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics