学习和解释行人意图的群体交互场

Xueyang Wang; Xuecheng Chen; Puhua Jiang; Haozhe Lin; Xiaoyun Yuan; Mengqi Ji; Yuchen Guo; Ruqi Huang; Lu Fang

doi:10.1016/j.eng.2023.05.020

PDF(3630 KB)

工程（英文） ›› 2024, Vol. 34 ›› Issue (3) : 70-82. DOI: 10.1016/j.eng.2023.05.020

研究论文

Article

学习和解释行人意图的群体交互场

Xueyang Wang ^a^,^b^,^c ,
Xuecheng Chen ^c ,
Puhua Jiang ^c ,
Haozhe Lin ^b^,^d ,
Xiaoyun Yuan ^a^,^b ,
Mengqi Ji ^e ,
Yuchen Guo ^b ,
Ruqi Huang ^c ,
Lu Fang ^a^,^b^,^f^,^*

作者信息 +

The Group Interaction Field for Learning and Explaining Pedestrian Anticipation

Xueyang Wang ^a^,^b^,^c ,
Xuecheng Chen ^c ,
Puhua Jiang ^c ,
Haozhe Lin ^b^,^d ,
Xiaoyun Yuan ^a^,^b ,
Mengqi Ji ^e ,
Yuchen Guo ^b ,
Ruqi Huang ^c ,
Lu Fang ^a^,^b^,^f^,^*

Author information +

History +

Abstract

Anticipating others’ actions is innate and essential in order for humans to navigate and interact well with others in dense crowds. This ability is urgently required for unmanned systems such as service robots and self-driving cars. However, existing solutions struggle to predict pedestrian anticipation accurately, because the influence of group-related social behaviors has not been well considered. While group relationships and group interactions are ubiquitous and significantly influence pedestrian anticipation, their influence is diverse and subtle, making it difficult to explicitly quantify. Here, we propose the group interaction field (GIF), a novel group-aware representation that quantifies pedestrian anticipation into a probability field of pedestrians’ future locations and attention orientations. An end-to-end neural network, GIFNet, is tailored to estimate the GIF from explicit multidimensional observations. GIFNet quantifies the influence of group behaviors by formulating a group interaction graph with propagation and graph attention that is adaptive to the group size and dynamic interaction states. The experimental results show that the GIF effectively represents the change in pedestrians’ anticipation under the prominent impact of group behaviors and accurately predicts pedestrians’ future states. Moreover, the GIF contributes to explaining various predictions of pedestrians’ behavior in different social states. The proposed GIF will eventually be able to allow unmanned systems to work in a human-like manner and comply with social norms, thereby promoting harmonious human-machine relationships.

Keywords

Human behavior modeling and prediction / Implicit representation of pedestrian anticipation / Group interaction / Graph neural network

引用本文

EndNote

Ris (Procite)

Bibtex

导出引用

Xueyang Wang, Xuecheng Chen, Puhua Jiang. . Engineering. 2024, 34(3): 70-82 https://doi.org/10.1016/j.eng.2023.05.020

参考文献

原文顺序 | 文献年度倒序 | 文中引用次数倒序

[1]	A. Rasouli, I. Kotseruba, J.K. Tsotsos. Pedestrian action anticipation using contextual feature fusion in stacked RNNs. Proceedings of the 30th British Machine Vision Conference (BMVC 2019) 2019 Sep 9-12, BMVA Press, Cardiff, UK. London (2019).
[2]	Y. Luo, P. Cai, A. Bera, D. Hsu, W.S. Lee, D. Manocha. PORCA: modeling and planning for autonomous driving among many pedestrians. IEEE Robot Autom Lett, 3 (4) (2018), pp. 3418-3425.
[3]	P. Trautman, J. Ma, R.M. Murray, A. Krause. Robot navigation in dense human crowds: the case for cooperation. Proceedings of the IEEE International Conference on Robotics and Automation; 2013 May 6-10; Karlsruhe, Germany, IEEE, New York City (2013), pp. 2153-2160.
[4]	Yao X, Zhang J, Oh J. Following social groups: socially compliant autonomous navigation in dense crowds. 2019. arXiv:1911.12063.
[5]	J. Zhou, Z.K. Shi. A new lattice hydrodynamic model for bidirectional pedestrian flow with the consideration of pedestrian’s anticipation effect. Nonlinear Dyn, 81 (3) (2015), pp. 1247-1262.
[6]	S. Hoogendoorn, P.H.L. Bovy. Simulation of pedestrian flows by optimal control and differential games. Optim Control Appl Methods, 24 (2003), pp. 153-172.
[7]	X. Zheng, Y. Cheng. Conflict game in evacuation process: a study combining Cellular Automata model. Physica A Stat Mech Appl, 390 (2011), pp. 1042-1050.
[8]	S. Bouzat, M.N. Kuperman. Game theory in models of pedestrian room evacuation. Phys Rev E Stat Nonlinear Soft Matter Phys, 89 (2014), 032806.
[9]	Q. Xu, M. Chraibi, A. Seyfried. Anticipation in a velocity-based model for pedestrian dynamics. Transp Res Part C Emerg Technol, 133 (2021), 103464.
[10]	Y. Suma, D. Yanagisawa, K. Nishinari. Anticipation effect in pedestrian dynamics: modeling and experiments. Physica A Stat Mech Appl, 391 (2012), pp. 248-263.
[11]	S. Nowak, A. Schadschneider. Quantitative analysis of pedestrian counterflow in a cellular automaton model. Phys Rev E Stat Nonlin Soft Matter Phys, 85 (6) (2012), 066128.
[12]	R. Bailo, J.A. Carrillo, P. Degond. Pedestrian models based on rational behaviour. L. Gibelli, N. Bellomo (Eds.), Crowd dynamics. Volume 1—modeling and simulation in science, engineering and technology, Springer, Berlin (2018).
[13]	H. Murakami, C. Feliciani, Y. Nishiyama, K. Nishinari. Mutual anticipation can contribute to self-organization in human crowds. Sci Adv, 7 (12) (2021), eabe7758.
[14]	H. Murakami, C. Feliciani, K. Nishinari. Lévy walk process in self-organization of pedestrian crowds. J R Soc Interface, 16 (153) (2019), 20180939.
[15]	R.M. Roe, J.R. Busemeyer, J.T. Townsend. Multialternative decision field theory: a dynamic connectionist model of decision making. Psychol Rev, 108 (2) (2001), pp. 370-392.
[16]	I. Karamouzas, B. Skinner, S.J. Guy. Universal power law governing pedestrian interactions. Phys Rev Lett, 113 (23) (2014), p. 238701.
[17]	F. Zanlungo, T. Ikeda, T. Kanda. Social force model with explicit collision prediction. EPL, 93 (6) (2011), p. 68005.
[18]	V. Kosaraju, A. Sadeghian, R. Martín-Martín, I. Reid, H. Rezatofighi, S. Savarese. Social-BiGAT: multimodal trajectory forecasting using bicycle-GAN and graph attention networks. Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019); 2019 Dec 8-14; Vancouver, BC, Canada (2019).
[19]	A. Mohamed, K. Qian, M. Elhoseiny, C. Claudel. Social-STGCNN: a social spatio-temporal graph convolutional neural network for human trajectory prediction. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020 Jun 14-19; online, IEEE, New York City (2020), pp. 14424-14432.
[20]	A. Rudenko, L. Palmieri, A.J. Lilienthal, K.O. Arras. Human motion prediction under social grouping constraints. Proceedings of the IEEE International Workshop on Intelligent Robots and Systems (IROS 2018) 2018 Oct 1-5;Madrid, Spain, IEEE, New York City (2018), pp. 3358-3364.
[21]	A. Sadeghian, V. Kosaraju, A. Sadeghian, N. Hirose, S.H. Rezatofighi, S. Savarese. SoPhie: an attentive GAN for predicting paths compliant to social and physical constraints. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019 Jun 16-20; Long Beach, CA, USA, IEEE, New York City (2019), pp. 1349-1358.
[22]	J. Sun, Q. Jiang, C. Lu. Recursive social behavior graph for trajectory prediction. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition 2019 Jun 16-20;Long Beach, CA, USA, IEEE, New York City (2019), pp. 660-669.
[23]	K. Mangalam, H. Girase, S. Agarwal, K.H. Lee, E. Adeli, J. Malik, et al. It is not the journey but the destination: endpoint conditioned trajectory prediction. Proceedings of the 2020 European Conference on Computer Vision; 2020 Aug 23-28; Glasgow, UK, Springer, Berlin (2020), pp. 759-776.
[24]	T. Salzmann, B. Ivanovic, P. Chakravarty, M. Pavone. Trajectron++: dynamically-feasible trajectory forecasting with heterogeneous data. Proceedings of the 2020 European Conference on Computer Vision; 2020 Aug 23-28; Glasgow, UK, Springer, Berlin (2020), pp. 683-700.
[25]	C. Zhou, M. Han, Q. Liang, Y.F. Hu, S.G. Kuai. A social interaction field model accurately identifies static and dynamic social groupings. Nat Hum Behav, 3 (8) (2019), pp. 847-855.
[26]	M. Moussaïd, N. Perozo, S. Garnier, D. Helbing, G. Theraulaz. The walking behaviour of pedestrian social groups and its impact on crowd dynamics. PLoS One, 5 (4) (2010), e10047.
[27]	Y. Liu, Q. Yan, A. Alahi. Social NCE: contrastive learning of socially-aware motion representations. Proceedings of the 2020 IEEE/CVF International Conference on Computer Vision; 2020 Jun 13-19; Seattle, WA, USA, IEEE, New York City (2020), pp. 15118-15129.
[28]	H. De Jaegher, E. Di Paolo, S. Gallagher. Can social interaction constitute social cognition?. Trends Cogn Sci, 14 (10) (2010), pp. 441-447.
[29]	L. Cheng, R. Yarlagadda, C.B. Fookes, P.K. Yarlagadda. A review of pedestrian group dynamics and methodologies in modelling pedestrian group behaviours. World J Mech Eng, 1 (2014), pp. 1-13.
[30]	Z. Yücel, F. Zanlungo, M. Shiomi. Modeling the impact of interaction on pedestrian group motion. Adv Robot, 32 (3) (2018), pp. 137-147.
[31]	R. Zhou, H. Zhou, H. Gao, M. Tomizuka, J. Li, Z. Xu. Grouptron: dynamic multi-scale graph convolutional networks for group-aware dense crowd trajectory forecasting. Proceedings of the 2022 International Conference on Robotics and Automation (ICRA 2022); 2022 May 23-27; Philadelphia, PA, USA, IEEE, New York City (2020), pp. 805-811.
[32]	S. Casas, C. Gulino, R. Liao, R. Urtasun. SpAGNN: spatially-aware graph neural networks for relational behavior forecasting from sensor data. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA 2020); 2020 May 31-Aug 31; online, IEEE, New York City (2020), pp. 9491-9497.
[33]	H. Girase, H. Gang, S. Malla, J. Li, A. Kanehara, K. Mangalam, et al. LOKI: long term and key intentions for trajectory prediction. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision; 2021 Oct 11-17; Montreal, BC, Canada, IEEE, New York City (2021), pp. 9803-9812.
[34]	Y. Huang, H. Bi, Z. Li, T. Mao, Z. Wang. STGAT: modeling spatial-temporal interactions for human trajectory prediction. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision; 2019 Oct 27-Nov 2; Seoul, Republic of Korea, IEEE, New York City (2019), pp. 6272-6281.
[35]	A. Gupta, J. Johnson, F.F. Li, S. Savarese, A. Alahi. Social GAN: socially acceptable trajectories with generative adversarial networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2018 Jun 18-23; Salt Lake City, UT, USA, IEEE, New York City (2018), pp. 2255-2264.
[36]	B. Zhang, W. Chen, X. Ma, P. Qiu, F. Liu. Experimental study on pedestrian behavior in a mixed crowd of individuals and groups. Physica A Stat Mech Appl, 556 (2020), 124814.
[37]	A.C. Gallup, J.J. Hale, D.J. Sumpter, S. Garnier, A. Kacelnik, J.R. Krebs, et al. Visual attention and the acquisition of information in human crowds. Proc Natl Acad Sci USA, 109 (19) (2012), pp. 7245-7250.
[38]	X. Wang, X. Zhang, Y. Zhu, Y. Guo, X. Yuan, L. Xiang, et al. PANDA: a gigapixel-level human-centric video dataset. Proceedings of the 2020 IEEE/CVF conference on computer vision and pattern recognition; 2020 Jun 14-19 ; online, IEEE, New York City (2020), pp. 3268-3278.
[39]	P. Raksincharoensak, T. Hasegawa, M. Nagai. Motion planning and control of autonomous driving intelligence system based on risk potential optimization framework. Int J Automot Eng, 7 (AVEC14) (2016), pp. 53-60.
[40]	A. Alahi, V. Ramanathan, F.F. Li. Socially-aware large-scale crowd forecasting. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition 2014 Jun 23-28;Columbus, OH, USA, IEEE, New York City (2014), pp. 2211-2218.
[41]	L. Shi, L. Wang, C. Long, S. Zhou, M. Zhou, Z. Niu, et al. SGCN: sparse graph convolution network for pedestrian trajectory prediction. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021 Jun 19-25; online, IEEE, New York City (2021), pp. 8994-9003.
[42]	A.A.A. Osman, T. Bolkart, M.J. Black. STAR: sparse trained articulated human body regressor. Proceedings of the Computer Vision-ECCV 2020: 16th European Conference; 2020 Aug 23-28; Glasgow, UK, Springer International Publishing, Berlin (2020), pp. 598-613.
[43]	Y. Yuan, X. Weng, Y. Ou, K. Kitani. AgentFormer: agent-aware transformers for socio-temporal multi-agent forecasting. Proceedings of the IEEE/CVF International Conference on Computer Vision; 2021 Oct 10-17; Montreal, QC, Canada, IEEE, New York City (2021), pp. 9813-9823.
[44]	A. Mohamed, D. Zhu, W. Vu, M. Elhoseiny, C. Claudel. Social-Implicit: rethinking trajectory prediction evaluation and the effectiveness of implicit maximum likelihood estimation. Proceedings of the Computer Vision-ECCV 2022: 17th European Conference; 2022 Oct 23-27; Tel Aviv, Israel, Springer, Berlin (2022), pp. 463-479.
[45]	I. Bae, J.H. Park, H.G. Jeon. Learning pedestrian group representations for multi-modal trajectory prediction. Proceedings of the Computer Vision-ECCV 2022: 17th European Conference; 2022 Oct 23-27; Tel Aviv, Israel, Springer, Berlin (2022).
[46]	P. Xu, J.B. Hayet, I. Karamouzas. SocialVAE: human trajectory prediction using timewise latents. Proceedings of the Computer Vision-ECCV 2022: 17th European Conference; 2022 Oct 23-27; Tel Aviv, Israel, Springer, Berlin (2022), pp. 511-528.
[47]	T. Gu, G.Y. Chen, J. Li, C. Lin, Y. Rao, J. Zhou, et al. Stochastic trajectory prediction via motion indeterminacy diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2022 Jun 21-24;New Orleans, LU, USA, IEEE, New York City (2022).
[48]	I. Bae, J.H. Park, H.G. Jeon. Non-probability sampling network for stochastic human trajectory prediction. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition 2022 Jun 21-24;New Orleans, LU, USA, IEEE, New York City (2022).
[49]	Y. Chen, B. Ivanovic, M. Pavone. ScePT: scene-consistent, policy-based trajectory predictions for planning. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2022 Jun 21-24;New Orleans, LU, USA, IEEE, New York City (2022).
[50]	P. Kothari, S. Kreiss, A. Alahi. Human trajectory forecasting in crowds: a deep learning perspective. IEEE Trans Intell Transp Syst, 23 (7) (2021), pp. 7386-7400.
[51]	C. Yu, X. Ma, J. Ren, H. Zhao, S. Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. Proceedings of the 2020 European Conference on Computer Vision 2020 Aug 23-28; online, Springer, Berlin (2020), pp. 507-523.
[52]	J. Qiu, J. Tang, H. Ma, Y. Dong, K. Wang, J. Tang. DeepInf: social influence prediction with deep learning. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining; 2018 Aug 19-23; London, UK, Association for Computing Machinery (ACM), New York City (2018), pp. 2110-2119.
[53]	C. Liu, Y. Chen, M. Liu, B.E. Shi. AVGCN: trajectory prediction using graph convolutional networks guided by human attention. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA); 2021 May 30-Jun 5; Xi’an, China, IEEE, New York City (2021), pp. 14234-14240.
[54]	I. Hasan, F. Setti, T. Tsesmelis, A. Del Bue, M. Cristani, F. Galasso. “Seeing is believing”: pedestrian trajectory forecasting using visual frustum of attention. Proceedings of the 2018 IEEE Workshop on Applications of Computer Vision (WACV 2018); 2018 Mar 12-15;Lake Tahoe, NV, USA, IEEE, New York City (2018), pp. 1178-1185.
[55]	R. Bastien, P. Romanczuk. A model of collective behavior based purely on vision. Sci Adv, 6 (6) (2020), eaay0792.
[56]	F.A. Lavergne, H. Wendehenne, T. Bäuerle, C. Bechinger. Group formation and cohesion of active particles with visual perception-dependent motility. Science, 364 (80) (2019), pp. 70-74.
[57]	J. Li, R. Han, H. Yan, Z. Qian, W. Feng, S. Wang. Self-supervised social relation representation for human group detection. Proceedings of the Computer Vision-ECCV 2022: 17th European Conference; 2022 Oct 23-27; Tel Aviv, Israel, Springer, Berlin (2022).
[58]	F. Solera, S. Calderara, R. Cucchiara. Socially constrained structural learning for groups detection in crowd. IEEE Trans Pattern Anal Mach Intell, 38 (5) (2016), pp. 995-1008.
[59]	T. Kruse, A.K. Pandey, R. Alami, A. Kirsch. Human-aware robot navigation: a survey. Robot Auton Syst, 61 (12) (2013), pp. 1726-1743.
[60]	F. Gul, W. Rahiman, S.S. Nazli Alhady, K. Chen. A comprehensive study for robot navigation techniques. Cogent Eng, 6 (1) (2019), 1632046.