基于人体姿态估计信息的工人施工活动分析

Xuhong Zhou; Shuai Li; Jiepeng Liu; Zhou Wu; Yohchia Frank Chen

doi:10.1016/j.eng.2023.10.004

PDF(4809 KB)

工程（英文） ›› 2024, Vol. 33 ›› Issue (2) : 225-236. DOI: 10.1016/j.eng.2023.10.004

研究论文

Article

基于人体姿态估计信息的工人施工活动分析

Xuhong Zhou ^a ,
Shuai Li ^a^,^* ,
Jiepeng Liu ^a ,
Zhou Wu ^b^,^* ,
Yohchia Frank Chen ^c

作者信息 +

Construction Activity Analysis of Workers Based on Human Posture Estimation Information

Xuhong Zhou ^a ,
Shuai Li ^a^,^* ,
Jiepeng Liu ^a ,
Zhou Wu ^b^,^* ,
Yohchia Frank Chen ^c

Author information +

History +

Abstract

Identifying workers’ construction activities or behaviors can enable managers to better monitor labor efficiency and construction progress. However, current activity analysis methods for construction workers rely solely on manual observations and recordings, which consumes considerable time and has high labor costs. Researchers have focused on monitoring on-site construction activities of workers. However, when multiple workers are working together, current research cannot accurately and automatically identify the construction activity. This research proposes a deep learning framework for the automated analysis of the construction activities of multiple workers. In this framework, multiple deep neural network models are designed and used to complete worker key point extraction, worker tracking, and worker construction activity analysis. The designed framework was tested at an actual construction site, and activity recognition for multiple workers was performed, indicating the feasibility of the framework for the automated monitoring of work efficiency.

Keywords

Pose estimation / Activity analysis / Object tracking / Construction workers / Automatic systems

引用本文

EndNote

Ris (Procite)

Bibtex

导出引用

Xuhong Zhou, Shuai Li, Jiepeng Liu. . Engineering. 2024, 33(2): 225-236 https://doi.org/10.1016/j.eng.2023.10.004

参考文献

原文顺序 | 文献年度倒序 | 文中引用次数倒序

[1]	T. Slaton, C. Hernandez, R. Akhavian. Construction activity recognition with convolutional recurrent networks. Autom Constr, 113 (2020), Article 103138
[2]	A. Kelm, L. Laußat, A. Meins-Becker, D. Platz, M.J. Khazaee, A.M. Costin, et al.. Mobie passive radio frequency identification (RFID) portal for automated and rapid control of personal protective equipment (PPE) on construction sites. Autom Constr, 36 (2013), pp. 38-52
[3]	T. Cheng, J. Teizer, G.C. Migliaccio, U.C. Gatti. Automated task-level activity analysis through fusion of real time location sensors and worker’s thoracic posture data. Autom Constr, 29 (2013), pp. 24-39
[4]	J. Kim, Y. Ham, Y. Chung, S. Chi. Systematic camera placement framework for operation-level visual monitoring on construction jobsites. J Constr Eng Manage, 145 (4) (2019), p. 04019019
[5]	J. Yang, Z. Shi, Z. Wu. Vision-based action recognition of construction workers using dense trajectories. Adv Eng Inf, 30 (3) (2016), pp. 327-336
[6]	A. Khosrowpour, J.C. Niebles, M. Golparvar-Fard. Vision-based workface assessment using depth images for activity analysis of interior construction operations. Autom Constr, 48 (2014), pp. 74-87
[7]	S.U. Han, S.H. Lee. A vision-based motion capture and recognition framework for behavior-based safety management. Autom Constr, 35 (2013), pp. 131-141
[8]	N. Yu, S. Wang. Enhanced autonomous exploration and mapping of an unknown environment with the fusion of dual RGB-D sensors. Engineering, 5 (1) (2019), pp. 164-172
[9]	X. Luo, H. Li, D. Cao, F. Dai, J.O. Seo, S.H. Lee. Recognizing diverse construction activities in site images via relevance networks of construction-related objects detected by convolutional neural networks. J Comput Civ Eng, 32 (3) (2018), p. 04018012
[10]	X. Luo, H. Li, D. Cao, Y. Yu, X. Yang, T. Huang. Towards efficient and objective work sampling: recognizing workers’ activities in site surveillance videos with two-stream convolutional networks. Autom Constr, 94 (2018), pp. 360-370
[11]	N. Pradhananga, J. Teizer. Cell-based construction site simulation model for earthmoving operations using real-time equipment location data. Visualization in Eng, 3 (1) (2015), p. 12
[12]	A. Montaser, O. Moselhi. RFID indoor location identification for construction projects. Autom Constr, 39 (2014), pp. 167-179
[13]	R. Akhavian, A.H. Behzadan. Construction equipment activity recognition for simulation input modeling using mobile sensors and machine learning classifiers. Adv Eng Inf, 29 (4) (2015), pp. 867-877
[14]	J. Zhao, E. Obonyo. Convolutional long short-term memory model for recognizing construction workers’ postures from wearable inertial measurement units. Adv Eng Inf, 46 (2020), Article 101177
[15]	L. Sanhudo, D. Calvetti, J.P. Martins, N.M.M. Ramos, P. Mêda, M.C. Gonçalves, et al.. Activity classification using accelerometers and machine learning for complex construction worker activities. J Build Eng, 35 (2021), Article 102001
[16]	E. Valero, A. Sivanathan, F. Bosché, M. Abdel-Wahab. Musculoskeletal disorders in construction: a review and a novel system for activity tracking with body area network. Appl Ergon, 54 (2016), pp. 120-130
[17]	X. Yan, H. Li, A.R. Li, H. Zhang. Wearable IMU-based real-time motion warning system for construction workers’ musculoskeletal disorders prevention. Autom Constr, 74 (2017), pp. 2-11
[18]	A. Golabchi, S.H. Han, J.O. Seo, S.U. Han, S.H. Lee, M. Al-Hussein. An automated biomechanical simulation approach to ergonomic job analysis for workplace design. J Constr Eng Manage, 141 (8) (2015), p. 04015020
[19]	R. Kanan, O. Elhassan, R. Bensalem. An IoT-based autonomous system for workers’ safety in construction sites with real-time alarming, monitoring, and positioning strategies. Autom Constr, 88 (2018), pp. 73-86
[20]	S. Chi, C.H. Caldas. Automated object identification using optical video cameras on construction sites. Comput Aided Civ Infrastruct Eng, 26 (5) (2011), pp. 368-380
[21]	J.O. Seo, S.H. Lee, J. Seo. Simulation-based assessment of workers’ muscle fatigue and its impact on construction operations. J Constr Eng Manage, 142 (11) (2016), p. 04016063
[22]	Gatt T, Seychell D, Dingli A. Detecting human abnormal behaviour through a video generated model. In:Proceedings of 2019 11th International Symposium on Image and Signal Processing and Analysis; 2019 Sep 23-25; Dubrovnik, Croatia. Piscataway: IEEE; 2019. p. 264-70.
[23]	D. Wang, W. Li, X. Liu, N. Li, C. Zhang. UAV environmental perception and autonomous obstacle avoidance: a deep learning and depth camera combined solution. Comput Electron Agric, 175 (2020), Article 105523
[24]	A. Krizhevsky, I. Sutskever, G.E. Hinton. ImageNet classification with deep convolutional neural networks. Adv Neural Inf Process Syst, 25 (2012), pp. 1097-1105
[25]	S. Ren, K. He, R. Girshick, J. Sun. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell, 39 (6) (2017), pp. 1137-1149
[26]	Donahue J, Hendricks LA, Guadarrama S, Rohrbach M, Venugopalan S, Darrell T, et al. Long-term recurrent convolutional networks for visual recognition and description. In: Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition; 2015 Jun 7-12; Boston, MA, USA. Piscataway: IEEE; 2015. p. 2625-34.
[27]	W. Fang, L. Ding, B. Zhong, P.E.D. Love, H. Luo. Automated detection of workers and heavy equipment on construction sites: a convolutional neural network approach. Adv Eng Inf, 37 (2018), pp. 139-149
[28]	H. Kim, S. Bang, H. Jeong, Y. Ham, H. Kim. Analyzing context and productivity of tunnel earthmoving processes using imaging and simulation. Autom Constr, 92 (2018), pp. 188-198
[29]	Q. Fang, H. Li, X. Luo, L. Ding, H. Luo, T.M. Rose, et al.. Detecting non-hardhat-use by a deep learning method from far-field surveillance videos. Autom Constr, 85 (2018), pp. 1-9
[30]	Q. Zhang, K. Barri, S.K. Babanajad, A.H. Alavi. Real-time detection of cracks on concrete bridge decks using deep learning in the frequency domain. Engineering, 7 (12) (2021), pp. 1786-1796
[31]	L. Ding, W. Fang, H. Luo, P.E.D. Love, B. Zhong, X. Ouyang. A deep hybrid learning model to detect unsafe behavior: integrating convolution neural networks and long short-term memory. Autom Constr, 86 (2018), pp. 118-124
[32]	S. Hochreiter, J. Schmidhuber. Long short-term memory. Neural Comput, 9 (8) (1997), pp. 1735-1780
[33]	X. Luo, H. Li, X. Yang, Y. Yu, D. Cao. Capturing and understanding workers’ activities in far-field surveillance videos with deep action recognition and Bayesian nonparametric learning. Comput Aided Civ Infrastruct Eng, 34 (4) (2019), pp. 333-351
[34]	L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, et al.. Temporal segment networks for action recognition in videos. IEEE Trans Pattern Anal Mach Intell, 41 (11) (2018), pp. 2740-2755
[35]	C. Chen, Z. Zhu, A. Hammad. Automated excavators activity recognition and productivity analysis from construction site surveillance videos. Autom Constr, 110 (2020), Article 103045
[36]	V. Silva, F. Soares, C.P. Leão, J.S. Esteves, G. Vercelli. Skeleton driven action recognition using an image-based spatial-temporal representation and convolution neural network. Sensors, 21 (13) (2021), p. 4342
[37]	Toshev A, Szegedy C. DeepPose:human pose estimation via deep neural networks. In:Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition; 2014 Jun 23-28 ; Columbus, OH, USA. Piscataway: IEEE; 2014. p. 1653-60.
[38]	D. Roberts, W.T. Calderon, S. Tang, M. Golparvar-Fard. Vision-based construction worker activity analysis informed by body posture. J Comput Civ Eng, 34 (4) (2020), p. 04020017
[39]	Cao Z, Simon T, Wei SE, Sheikh Y. Realtime multi-person 2D pose estimation using part affinity fields. In:Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition; 2017 Jul 21- 26 ; Honolulu, HI, USA. Piscataway: IEEE; 2017. p. 1302-10.
[40]	B. Chen, C. Hua, D. Li, Y. He, J. Han. Intelligent human-UAV interaction system with joint cross-validation over action-gesture recognition and scene understanding. Appl Sci, 9 (16) (2019), p. 3277
[41]	Okumura T, Urabe S, Inoue K, Yoshioka M. Cooking activities recognition in egocentric videos using hand shape feature with OpenPose. In: CEA/MADiMa'18: Proceedings of the Joint Workshop on Multimedia for Cooking and Eating Activities and Multimedia Assisted Dietary Management; 2018 Jul 15; Stockholm, Sweden. New York: Association for Computing Machinery; 2018. p. 42-5.
[42]	S. Chen, K. Demachi. Towards on-site hazards identification of improper use of personal protective equipment using deep learning-based geometric relationships and hierarchical scene graph. Autom Constr, 125 (2021), Article 103619
[43]	Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014. arXiv:1409.1556.
[44]	Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC. MobileNetV2:inverted residuals and linear bottlenecks. In:Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2018 Jun 18- 23; Salt Lake City, UT, USA. Piscataway: IEEE; 2018. p. 4510-20.
[45]	Chollet F. Xception:Deep learning with depthwise separable convolutions. In:Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition; 2017 Jul 21-26 ; Honolulu, HI, USA. Piscataway: IEEE; 2017. p. 1800-7.
[46]	Wojke N, Bewley A, Paulus D. Simple online and realtime tracking with a deep association metric. In:Proceedings of 2017 IEEE International Conference on Image Processing; 2017 Sep 17-20: Beijing, China. Piscataway: IEEE; 2017. p. 3645-9.
[47]	Z. Wang, Q. Zhang, B. Yang, T. Wu, K. Lei, B. Zhang, et al.. Vision-based framework for automatic progress monitoring of precast walls by using surveillance videos during the construction phase. J Comput Civ Eng, 35 (1) (2021), p. 04020056