Journal Home Online First Current Issue Archive For Authors Journal Information 中文版

2021, Volume 22, Issue 5

Abstract

Keywords

Related Research

Back to Top

Frontiers of Information Technology & Electronic Engineering >> 2021, Volume 22, Issue 5 doi: 10.1631/FITEE.2000722

Visual commonsense reasoning with directional visual connections

Affiliation(s): College of Intelligence and Computing, Tianjin University, Tianjin 300350, China; Tianjin Key Lab of Machine Learning, Tianjin University, Tianjin 300350, China; School of Computer Science, University of Technology Sydney, Sydney 2007, Australia; less

Received: 2020-12-25 Accepted: 2021-05-17 Available online: 2021-05-17
HTML40 PDF 26 Collect 0

Next Previous

Abstract

To boost research into cognition-level visual understanding, i.e., making an accurate inference based on a thorough understanding of visual details, (VCR) has been proposed. Compared with traditional visual question answering which requires models to select correct answers, VCR requires models to select not only the correct answers, but also the correct rationales. Recent research into human cognition has indicated that brain function or cognition can be considered as a global and dynamic integration of local neuron connectivity, which is helpful in solving specific cognition tasks. Inspired by this idea, we propose a to achieve VCR by dynamically reorganizing the that is contextualized using the meaning of questions and answers and leveraging the directional information to enhance the reasoning ability. Specifically, we first develop a GraphVLAD module to capture to fully model visual content correlations. Then, a contextualization process is proposed to fuse sentence representations with visual neuron representations. Finally, based on the output of , we propose to infer answers and rationales, which includes a ReasonVLAD module. Experimental results on the VCR dataset and visualization analysis demonstrate the effectiveness of our method.

Related Research