Vision-Based Digital Shadowing to Reveal Hidden Structural Dynamics of a Real Supertall Building

Donglian Gu; Qingrui Yue; Li Li; Chujin Sun; Xinzheng Lu

doi:10.1016/j.eng.2024.10.002

PDF(3949 KB)

Engineering ›› 2024, Vol. 43 ›› Issue (12) : 146-158. DOI: 10.1016/j.eng.2024.10.002

Research

Article

Vision-Based Digital Shadowing to Reveal Hidden Structural Dynamics of a Real Supertall Building

Donglian Gu^a ,
Qingrui Yue^a^,^b ,
Li Li^a ,
Chujin Sun^c^,^d ,
Xinzheng Lu^c^,^*

Author information +

History +

Abstract

Vision-based digital shadowing is a highly efficient way to monitor the health of buildings in use. However, previous studies on digital shadowing have been limited to laboratory experiments. This paper proposes a novel computer-vision-based digital shadow workflow and presents its successful application in a real engineering case. In this case, a 345.8-m supertall building experienced unexpected shaking under normal meteorological conditions. This study established a digital shadow of the building using three-dimensional displacement measurements based on super-resolution monocular vision, revealing the hidden structural dynamics and inherent mechanical reasons for the abnormal shaking. The proposed digital shadowing workflow is a feasible roadmap for developing vision-based digital shadows of real-world structures using low-cost cameras. The abnormal vibration event in the supertall building considered in this study is the first of its type worldwide. The results of this study offer practical strategies and invaluable insights into the prevention and mitigation of this type of global risk, thereby contributing to the lifespan extension of buildings in use worldwide. Furthermore, with the increasing number of general sensing devices, such as surveillance cameras in cities, the proposed method may unleash the immense potential of general sensing devices in achieving the leap from structural health monitoring to city health monitoring.

Graphical abstract

Keywords

Digital shadow / Digital twin / Super-resolution / Monocular vision / Displacement measurement / Finite element simulation / Deep learning / Structural health monitoring

Cite this article

EndNote

Ris (Procite)

Bibtex

Download citation ▾

Donglian Gu, Qingrui Yue, Li Li, Chujin Sun, Xinzheng Lu. Vision-Based Digital Shadowing to Reveal Hidden Structural Dynamics of a Real Supertall Building. Engineering, 2024, 43(12): 146‒158 https://doi.org/10.1016/j.eng.2024.10.002

References

Publishing order | Descend order by publishing year | Descend order by cited within

[1]	Buchholz K. How has the world’s urban population changed from 1950 to today? [Internet] Geneva: World Economic Forum; 2020 Nov 4 [cited2024 Oct 11]. Available from:

[2]	H.P. Chen, Y.Q. Ni. Structural health monitoring of large civil engineering structures. John Wiley & Sons Ltd, West Sussex (2018).

[3]	X. Lu, X.Z. Lu, H. Guan, L.P. Ye. Collapse simulation of reinforced concrete high-rise building induced by extreme earthquakes. Earthquake Eng Struct Dynam, 42 (5) (2013), pp. 705-723.

[4]	X.Z. Lu, L.L. Xie, H. Guan, Y.L. Huang, X. Lu. A shear wall element for nonlinear seismic analysis of super-tall buildings using OpenSees. Finite Elem Anal Des, 98 (2015), pp. 14-25.

[5]	X.Z. Lu, D.L. Gu, Z. Xu, C. Xiong, Y. Tian. CIM-powered multi-hazard simulation framework covering both individual buildings and urban areas. Sustainability, 12 (12) (2020), p. 5059.

[6]	Y. Fujino, D.M. Siringoringo, Y. Ikeda, T. Nagayama, T. Mizutani. Research and implementations of structural monitoring for bridges and buildings in Japan. Engineering, 5 (6) (2019), pp. 1093-1119.

[7]	K.Q. Lin, Y.L. Xu, X.Z. Lu, Z.G. Guan, J.Z. Li. Cluster computing-aided model updating for a high-fidelity finite element model of a long-span cable-stayed bridge. Earthquake Eng Struct Dynam, 49 (9) (2020), pp. 904-923.

[8]	K.Q. Lin, Y.L. Xu, X.Z. Lu, Z.G. Guan, J.Z. Li. Digital twin-based life-cycle seismic performance assessment of a long-span cable-stayed bridge. Bull Earthquake Eng, 21 (2) (2023), pp. 1203-1227.

[9]	Grieves M,. Vickers J. Digital twin:mitigating unpredictable, undesirable emergent behavior in complex systems. In: KahlenJ, FlumerfeltS, AlvesA, editors. Transdisciplinary perspectives on complex systems:new findings and approaches. Cham: Springer; 2017. p. 85-113.

[10]	F. Tao, Q. Qi, L. Wang, A.Y. Nee. Digital twins and cyber-physical systems toward smart manufacturing and Industry 4.0: correlation and comparison. Engineering, 5 (4) (2019), pp. 653-661.

[11]	M. Liebenberg, M. Jarke. Information systems engineering with digital shadows: concept and use cases in the internet of production. Inf Syst, 114 (2023), Article 102182.

[12]	Z. Zheng, W.J. Liao, J.R. Lin, Y.C. Zhou, C. Zhang, X.Z. Lu. Digital twin-based investigation of a building collapse accident. Adv Civ Eng, 2022 (1) (2022), Article 9568967.

[13]	M. Marienkov, I. Kaliukh, O. Trofymchuk. The digital twin use for modeling the multi-storey building response to seismic impacts. Struct Concr, 25 (3) (2024), pp. 2079-2096.

[14]	K.Q. Lin, Y.L. Xu, X.Z. Lu, Z.G. Guan, J.Z. Li. Collapse prognosis of a long-span cable-stayed bridge based on shake table test and nonlinear model updating. Earthquake Eng Struct Dynam, 50 (2) (2021), pp. 455-474.

[15]	K.Q. Lin, Y.L. Xu, X.Z. Lu, Z.G. Guan, J.Z. Li. Digital twin-based collapse fragility assessment of a long-span cable-stayed bridge under strong earthquakes. Autom Construct, 123 (2021), Article 103547.

[16]	D. Feng, M.Q. Feng. Computer vision for SHM of civil infrastructure: from dynamic response measurement to damage detection—a review. Eng Struct, 156 (2018), pp. 105-117.

[17]	A. Güemes, A. Fernandez-Lopez, A.R. Pozo, J. Sierra-Pérez. Structural health monitoring for advanced composite structures: a review. J Compos Sci, 4 (1) (2020), p. 13.

[18]	C. Kralovec, M. Schagerl. Review of structural health monitoring methods regarding a multi-sensor approach for damage assessment of metal and composite structures. Sensors, 20 (3) (2020), p. 826.

[19]	X.W. Ye, T.H. Yi, C.Z. Dong, T. Liu, H. Bai. Multi-point displacement monitoring of bridges using a vision-based approach. Wind Struct, 20 (2) (2015), pp. 315-326.

[20]	C.Z. Dong, F.N. Catbas. A review of computer vision-based structural health monitoring at local and global levels. Struct Health Monit, 20 (2) (2021), pp. 692-743.

[21]	J.W. Park, D.S. Moon, H. Yoon, F. Gomez, B.F. Spencer Jr, J.R. Kim. Visual-inertial displacement sensing using data fusion of vision-based displacement with acceleration. Struct Contr Health Monit, 25 (3) (2018), p. 2122.

[22]	B.F. Spencer Jr, V. Hoskere, Y. Narazaki. Advances in computer vision-based civil infrastructure inspection and monitoring. Engineering, 5 (2) (2019), pp. 199-222.

[23]	D. Feng, M.Q. Feng. Vision-based multipoint displacement measurement for structural health monitoring. Struct Contr Health Monit, 23 (5) (2016), pp. 876-890.

[24]	Z. Dworakowski, P. Kohut, A. Gallina, K. Holak, T. Uhl. Vision-based algorithms for damage detection and localization in structural health monitoring. Struct Contr Health Monit, 23 (1) (2016), pp. 35-50.

[25]	H. Yoon, H. Elanwar, H. Choi, M. Golparvar-Fard, B.F. Spencer Jr. Target-free approach for vision-based structural system identification using consumer-grade cameras. Struct Contr Health Monit, 23 (12) (2016), pp. 1405-1416.

[26]	Y. Yang, C. Dorn, T. Mancini, Z. Talken, G. Kenyon, C. Farrar, et al. Blind identification of full-field vibration modes from video measurements with phase-based video motion magnification. Mech Syst Signal Process, 85 (2017), pp. 567-590.

[27]	C.J. Sun, D.L. Gu, X.Z. Lu. Three-dimensional structural displacement measurement using monocular vision and deep learning based pose estimation. Mech Syst Signal Process, 190 (2023), Article 110141.

[28]	C.J. Sun, D.L. Gu, Y. Zhang, X.Z. Lu. Vision-based displacement measurement enhanced by super-resolution using generative adversarial networks. Struct Contr Health Monit, 29 (10) (2022), p. 3048.

[29]	Z. Wang, J. Chen, S.C. Hoi. Deep learning for image super-resolution: a survey. IEEE Trans Pattern Anal Mach Intell, 43 (10) (2021), pp. 3365-3387.

[30]	S. Hoque, M.Y. Arafat, S. Xu, A. Maiti, Y. Wei. A comprehensive review on 3D object detection and 6D pose estimation with deep learning. IEEE Access, 9 (2021), pp. 143746-143770.

[31]

Ledig

, Theis

, Huszár

, Caballero

, Cunningham

, Acosta

, et al. Photo-realistic single image super-resolution using a generative adversarial network. In:Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition; 2017 Jul 21-26; Honolulu, HI, USA. New York City: IEEE; 2017. p. 4681-90.

[32]	Zhang Z, Wang Z, Lin Z, Qi H. Image super-resolution by neural texture transfer. In:Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019 June 15-20; Long Beach, CA, USA. New York City: IEEE; 2019. p. 7982-91.

[33]	Zakharov S, Shugurov I, Ilic S. DPOD:6D pose object detector and refiner. In:Proceedings of the IEEE/CVF International Conference on Computer Vision; 2019 Oct 27-Nov 2; Seoul, Republic of Korea. New York City: IEEE; 2019. p. 1941-50.

[34]	G. Billings, M. Johnson-Roberson. Silhonet: an RGB method for 6D object pose estimation. IEEE Robot Autom Lett, 4 (4) (2019), pp. 3727-3734.

[35]	I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, et al. Generative adversarial networks. Commun ACM, 63 (11) (2020), pp. 139-144.

[36]

Martin

, Fowlkes

, Tal

, Malik

. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2001 Jul 7-14; Vancouver, BC, Canada. New York City: IEEE; 2001. p. 416-23.

[37]	Kingma DP, Ba J. Adam: a method for stochastic optimization. 2014. arXiv:1412.6980.

[38]	Y.F. Ji, C.C. Chang. Nontarget image-based technique for small cable vibration measurement. J Bridge Eng, 13 (1) (2008), pp. 34-42.

[39]	E. Caetano, S. Silva, J. Bateira. A vision system for vibration monitoring of civil engineering structures. Exp Tech, 35 (4) (2011), pp. 74-82.

[40]	Li XL, Wang H, Yi L, Guibas LJ, Abbott AL, Song S. Category-level articulated object pose estimation. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020 Jun 13-19. Seattle, WA, USA. New York City: IEEE; 2020. p. 3706-15.

[41]

Hinterstoisser

, Lepetit

, Ilic

, Holzer

, Bradski

, Konolige

, et al. Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In: Lee KM, Matsushita Y, Rehg JM, Hu Z, editors. Proceedings of the 11th Asian Conference on Computer Vision; 2012 Nov 5-9; Daejeon, Republic of Korea. Berlin: Springer; 2013. p. 548-62.

[42]

Brachmann

, Krull

, Michel

, Gumhold

, Shotton

, Rother

. Learning 6D object pose estimation using 3D object coordinates. In: Fleet

, Pajdla

, Schiele

, Tuytelaars

, editors. Proceedings of the 13th European Conference on Computer Vision; 2014 Sep 6-12; Zurich, Switzerland. Cham: Springer International Publishing; 2014. p. 536-51.

[43]	He KM, Zhang XY, Ren SQ, Sun J. Deep residual learning for image recognition. In:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016 Jun 27-30; Las Vegas, NV, USA. New York City: IEEE; 2016. p. 770-8.

[44]	J.M. Blain. The complete guide to blender graphics:computer modeling & animation. AK Peters/CRC Press, New York City (2019).

[45]	S. Dev, A. Nautiyal, Y.H. Lee, S. Winkler. CloudSegNet: a deep network for nychthemeron cloud image segmentation. IEEE Geosci Remote Sens Lett, 16 (12) (2019), pp. 1814-2188.

[46]	Brincker R, Andersen P. Understanding stochastic subspace identification. In:Proceedings of IMAC-XXIV: Conference & Exposition on Structural Dynamics Society for Experimental Mechanics; 2006 Jan 30-Feb 2; St Louis, MI, USA. New York City: Curran Associates, Inc; 2006.

[47]	X.Z. Lu, H. Guan. Earthquake disaster simulation of civil infrastructures: from tall buildings to urban areas. (2th ed.), Springer, Singapore (2021).

[48]	Mirjalili S. Genetic algorithm. In: KacprzykJ, editor. Evolutionary algorithms and neural networks-studies in computational intelligence. Berlin: 2019.

[49]	D.L. Gu, P.J. Zhao, W. Chen, Y.L. Huang, X.Z. Lu. Near real-time prediction of wind-induced tree damage at a city scale: simulation framework and case study for Tsinghua University campus. Int J Disaster Risk Reduct, 53 (2021), Article 102003.

[50]	D.L. Gu, W. Chen, X.Z. Lu. Automated assessment of wind damage to windows of buildings at a city scale based on oblique photography, deep learning and CFD. J Build Eng, 52 (2022), Article 104355.

[51]	D.L. Gu, A. Kareem, X.Z. Lu, Q.L. Cheng. A computational framework for the simulation of wind effects on buildings in a cityscape. J Wind Eng Ind Aerodyn, 234 (2023), Article 105347.

[52]	D.L. Gu, Q.W. Shuai, Y. Wang, Y.X. Wang. CIM-powered physics-based assessment of wind damages to building clusters considering trees. Dev Built Environ, 15 (2023), Article 100178.

[53]	Matsson JE. An introduction to Ansys fluent 2023. Mission: SDC Publications; 2023.

[54]	J. Franke, A. Hellsten, H. Schlünzen, B. Carissimo. Best practice guideline for the CFD simulation of flows in the urban environment, cost action 732: quality assurance and improvement of microscale meteorological models. COST Office, Brussels (2007).

[55]	Y. Tominaga, A. Mochida, R. Yoshie, H. Kataoka, T. Nozu, M. Yoshikawa, et al. AIJ guidelines for practical applications of CFD to pedestrian wind environment around buildings. J Wind Eng Ind Aerodyn, 96 (10-11) (2008), pp. 1749-1761.

[56]	Architectural Institute of Japan (AIJ). Guidelines for the evaluation of habitability to building vibration. Tokyo: Architectural Institute of Japan; 2004.