用于采集物理和虚拟抓取的可重构数据手套

Hangxin Liu; Zeyu Zhang; Ziyuan Jiao; Zhenliang Zhang; Minchen Li; Chenfanfu Jiang; Yixin Zhu; Song-Chun Zhu

doi:10.1016/j.eng.2023.01.009

PDF(2672 KB)

工程（英文） ›› 2024, Vol. 32 ›› Issue (1) : 202-216. DOI: 10.1016/j.eng.2023.01.009

研究论文

Article

用于采集物理和虚拟抓取的可重构数据手套

Hangxin Liu ^a^,^* ,
Zeyu Zhang ^a^,^b ,
Ziyuan Jiao ^a^,^b ,
Zhenliang Zhang ^a ,
Minchen Li ^c ,
Chenfanfu Jiang ^c ,
Yixin Zhu ^d^,^* ,
Song-Chun Zhu ^a^,^d^,^e

作者信息 +

A Reconfigurable Data Glove for Reconstructing Physical and Virtual Grasps

Hangxin Liu ^a^,^-^,^* ,
Zeyu Zhang ^a^,^b^,^- ,
Ziyuan Jiao ^a^,^b^,^- ,
Zhenliang Zhang ^a ,
Minchen Li ^c ,
Chenfanfu Jiang ^c ,
Yixin Zhu ^d^,^* ,
Song-Chun Zhu ^a^,^d^,^e

Author information +

History +

Abstract

In this work, we present a reconfigurable data glove design to capture different modes of human hand-object interactions, which are critical in training embodied artificial intelligence (AI) agents for fine manipulation tasks. To achieve various downstream tasks with distinct features, our reconfigurable data glove operates in three modes sharing a unified backbone design that reconstructs hand gestures in real time. In the tactile-sensing mode, the glove system aggregates manipulation force via customized force sensors made from a soft and thin piezoresistive material; this design minimizes interference during complex hand movements. The virtual reality (VR) mode enables real-time interaction in a physically plausible fashion: A caging-based approach is devised to determine stable grasps by detecting collision events. Leveraging a state-of-the-art finite element method, the simulation mode collects data on fine-grained four-dimensional manipulation events comprising hand and object motions in three-dimensional space and how the object’s physical properties (e.g., stress and energy) change in accordance with manipulation over time. Notably, the glove system presented here is the first to use high-fidelity simulation to investigate the unobservable physical and causal factors behind manipulation actions. In a series of experiments, we characterize our data glove in terms of individual sensors and the overall system. More specifically, we evaluate the system’s three modes by ① recording hand gestures and associated forces, ② improving manipulation fluency in VR, and ③ producing realistic simulation effects of various tool uses, respectively. Based on these three modes, our reconfigurable data glove collects and reconstructs fine-grained human grasp data in both physical and virtual environments, thereby opening up new avenues for the learning of manipulation skills for embodied AI agents.

Keywords

Data glove / Tactile sensing / Virtual reality / Physics-based simulation

引用本文

EndNote

Ris (Procite)

Bibtex

导出引用

Hangxin Liu, Zeyu Zhang, Ziyuan Jiao. 用于重建物理和虚拟抓取的可重构数据手套. Engineering. 2024, 32(1): 202-216 https://doi.org/10.1016/j.eng.2023.01.009

参考文献

原文顺序 | 文献年度倒序 | 文中引用次数倒序

[1]	Pinto L, Gupta A. Supersizing self-supervision:learning to grasp from 50K tries and 700 robot hours. In: Proceedingsof IEEE International Conference on Robotics and Automation ( ICRA2016; May 16-21 2016 ; Stockholm, Sweden. New York City: IEEE; 2016.
[2]	J. Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley, et al.. Learning ambidextrous robot grasping policies. Sci Robot, 4 (26) ( 2019), p. eaau4984
[3]	Zeng A, Song S, Yu KT, Donlon E, Hogan FR, Bauza M, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. In: Proceedingsof IEEE International Conference on Robotics and Automation ( ICRA2018; May 21-25 2018 ; Brisbane, QLD, Australia. New York City: IEEE; 2018.
[4]	F. Cini, V. Ortenzi, P. Corke, M. Controzzi. On the choice of grasp type and location when handing over an object. Sci Robot, 4 (27) ( 2019), p. eaau9757
[5]	Yahya A, Li A, Kalakrishnan M, Chebotar Y, Levine S. Collective robot reinforcement learning with distributed asynchronous guided policy search. In: Proceedingsof IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS2017; Sep 24-28 2017. p. 2017 ; Vancouver, BC, Canada. New York City: IEEE; 79-86.
[6]	S. Schaal, A. Ijspeert, A. Billard. Computational approaches to motor learning by imitation. Phil Trans R Soc Lond B, 358 (1431) ( 2003), pp. 537-547
[7]	G. Maeda, M. Ewerton, D. Koert, J. Peters. Acquiring and generalizing the embodiment mapping from human observations to robot skills. IEEE Robot Autom Lett, 1 (2) ( 2016), pp. 784-791
[8]	Nguyen A, Kanoulas D, Caldwell DG, Tsagarakis NG. Detecting object affordances with convolutional neural networks. In: Proceedingsof IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS2016; Oct 9-14 2016. p. 2016 ; Daejeon, Republic of Korea. New York City: IEEE; 2765-70.
[9]	Kokic M, Stork JA, Haustein JA, Kragic D. Affordance detection for task-specific grasping using deep learning. In: Proceedings of 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids); 2017 Nov 15-17; Birmingham, UK. New York City: IEEE; 2017. p. 91-8.
[10]	Mohseni-Kabir A, Rich C, Chernova S, Sidner CL, Miller D. Interactive hierarchical task learning from a single demonstration. In: Proceedings of the 2015 10th Annual ACM/IEEE International Conference on Human-Robot Interaction; 2015 Mar 2-5; Portland, OR, USA. New York City: IEEE; 2015. p. 205-12.
[11]	Xiong C, Shukla N, Xiong W, Zhu SC. Robot learning with a spatial, temporal, and causal and-or graph. In: Proceedings of 2016 IEEE International Conference on Robotics and Automation ( ICRA2016; May 16-21 2016. p. 2016 ; Stockholm, Sweden. New York City: IEEE; 2144-51.
[12]	Liu H, Zhang C, Zhu Y, Jiang C, Zhu SC. Mirroring without overimitation: learning functionally equivalent manipulation actions. In:Proceedings of the AAAI Conference on Artificial Intelligence (AAAI); 2019 Jan 27-Feb 1; Honolulu, HI, USA. 2019. p. 8025-33.
[13]	Abbeel P, Ng AY. Apprenticeship learning via inverse reinforcement learning. In: Proceedings of the 21st International Conference on Machine Learning ( ICML2004; Jul 4-8 Canada. 2004 ; Banff, AB, New York City: Association for Computing Machinery (ACM); 2004.
[14]	Prieur U, Perdereau V, Bernardino A. Modeling and planning high-level in-hand manipulation actions from human knowledge and active learning from demonstration. In: Proceedingsof 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems; Oct 7-12 2012. p. 2012 ; Vilamoura-Algarve, Portugal. New York City: IEEE; 1330-6.
[15]	Ibarz B, Leike J, Pohlen T, Irving G, Legg S, Amodei D.Reward learning from human preferences and demonstrations in Atari. In:Proceedings of the 32nd Conference on Advances in Neural Information Processing Systems (NeurIPS 2018); 2018 Dec 3- 8 ; Montréal, QC, Canada. Red Hook: Curran Associates Inc.; 2018. p.1-13.
[16]	Xie X, Liu H, Zhang Z, Qiu Y, Gao F, Qi S, et al. VRGym:a virtual testbed for physical and interactive AI. In: Proceedingsof the ACM Turing Celebration Conference-China; May 17-19 China. 2019. p. 2019 ; Chengdu, New York City: Association for Computing Machinery; 1-6.
[17]	Li C, Xia F, Martín-Martín R, Lingelbach M, Srivastava S, Shen B, et al. IGibson 2.0:object-centric simulation for robot learning of everyday household tasks. In:Proceedings of the 5th Annual Conference on Robot Learning (CoRL 2021); 2021 Nov 8- 11; online; 2021.
[18]	Szot A, Clegg A, Undersander E, Wijmans E, Zhao Y, Turner J, et al. Habitat 2.0:training home assistants to rearrange their habitat. In:Proceedings of 35th Conference on Neural Information Processing Systems (NeurIPS 2021); 2021 Dec 6- 14; online; 2021.
[19]	M. Li, Z. Ferguson, T. Schneider, T. Langlois, D. Zorin, D. Panozzo, et al.. Incremental potential contact: intersection-and inversion-free, large-deformation dynamics. ACM Trans Graph, 39 (4) ( 2020), p. 49
[20]	Liu H, Xie X, Millar M, Edmonds M, Gao F, Zhu Y, et al. A glove-based system for studying hand-object manipulation via joint pose and force sensing. In: Proceedingsof 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems IROS; Sep 24-28 2017. p. 2019 ; Vancouver, BC, Canada. New York City: IEEE; 6617-24.
[21]	M. Edmonds, F. Gao, H. Liu, X. Xie, S. Qi, B. Rothrock, et al.. A tale of two explanations: enhancing human trust by explaining robot behavior. Sci Robot, 4 (37) ( 2019), p. aay4663
[22]	Brahmbhatt S, Ham C, Kemp CC, Hays J. ContactDB:analyzing and predicting grasp contact via thermal imaging. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition ( CVPR2019; Jun 15-20 2019. p. 2019 ; Long Beach, CA, USA. New York City: IEEE; 8701-11.
[23]	Liu H, Zhang Z, Xie X, Zhu Y, Liu Y, Wang Y, et al. High-fidelity grasping in virtual reality using a glove-based system. In: Proceedings of the 2019 International Conference on Robotics and Automation ( ICRA2019; May 20-24 2019. p. 2019 ; Montreal, QC, Canada. New York City: IEEE; 5180-6.
[24]	Duan K, Parikh D, Crandall D, Grauman K. Discovering localized attributes for fine-grained recognition. In: Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition ( CVPR2012; Jun 16-21 ; Providence RI, USA. 2012. p. 2012 New York City: IEEE; 3474-81.
[25]	Liu Y, Wei P, Zhu SC. Jointly recognizing object fluents and tasks in egocentric videos. In: Proceedingsof 2017 IEEE International Conference on Computer Vision ICCV; Oct 22-29 2017. p. 2017 ; Venice, Italy. New York City: IEEE; 2943-51.
[26]	Nagarajan T, Grauman K. Attributes as operators: factorizing unseen attribute-object compositions. In:Proceedings of European Conference on Computer Vision (ECCV 2018); 2018 Sep 8-14; Munich, Germany. Berlin:Springer; 2018. p. 172-90.
[27]	I. Newton, J. Colson. The method of fluxions and infinite series; with its application to the geometry of curve-lines. Henry Woodfall, London (1736)
[28]	L. Dipietro, A.M. Sabatini, P. Dario. A survey of glove-based systems and their applications. IEEE Trans Syst Man Cybern Part C, 38 (4) ( 2008), pp. 461-482
[29]	Kramer RK, Majidi C, Sahai R, Wood RJ. Soft curvature sensors for joint angle proprioception. In: Proceedings of 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS2011; Sep 25-30 2011. p. 2011 ; San Francisco, CA, USA. New York City: IEEE; 1919-26.
[30]	N.S. Kamel, S. Sayeed, G.A. Ellis. Glove-based approach to online signature verification. IEEE Trans Pattern Anal Mach Intell, 30 (6) ( 2008), pp. 1109-1113
[31]	J. Oh, S. Kim, S. Lee, S. Jeong, S.H. Ko, J. Bae.A liquid metal based multimodal sensor and haptic feedback device for thermal and tactile sensation generation in virtual reality. Adv Funct Mater, 31 (39) ( 2021), p. 2007772
[32]	M. Wang, Z. Yan, T. Wang, P. Cai, S. Gao, Y. Zeng, et al.. Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors. Nat Electron, 3 (9) ( 2020), pp. 563-570
[33]	F. Wen, Z. Sun, T. He, Q. Shi, M. Zhu, Z. Zhang, et al.. Machine learning glove using self-powered conductive superhydrophobic triboelectric textile for gesture recognition in VR/AR applications. Adv Sci, 7 (14) ( 2020), p. 2000261
[34]	Taylor T, Ko S, Mastrangelo C, Bamberg SJM. Forward kinematics using IMU on-body sensor network for mobile analysis of human kinematics. In: Proceedings of 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society ( EMBC2013; Jul 3-7 2013. p. 2013 ; Osaka, Japan. New York City: IEEE; 1230-3.
[35]	H.G. Kortier, V.I. Sluiter, D. Roetenberg, P.H. Veltink.Assessment of hand kinematics using inertial and magnetic sensors. J NeuroEng Rehabil, 11 (1) ( 2014), p. 70
[36]	B. Hu, T. Ding, Y. Peng, L. Liu, X. Wen. Flexible and attachable inertial measurement unit (IMU)-based motion capture instrumentation for the characterization of hand kinematics: a pilot study. Instrum Sci Technol, 49 (2) ( 2020), pp. 125-145
[37]	Santaera G, Luberto E, Serio A, Gabiccini M, Bicchi A. Low-cost, fast and accurate reconstruction of robotic and human postures via IMU measurements. In: Proceedings of 2015 IEEE International Conference on Robotics and Automation ( ICRA2015; May 26-30 ; Seattle WA, USA. 2015. p. 2015 New York City: IEEE; 2728-35.
[38]	G. Ligorio, A.M. Sabatini. Extended Kalman filter-based methods for pose estimation using visual, inertial and magnetic sensors: comparative analysis and performance evaluation. Sensors, 13 (2) ( 2013), pp. 1919-1941
[39]	H.G. Kortier, J. Antonsson, H.M. Schepers, F. Gustafsson, P.H. Veltink. Hand pose estimation by fusion of inertial and magnetic sensing aided by a permanent magnet. IEEE Trans Neural Syst Rehabiln Eng, 23 (5) ( 2015), pp. 796-806
[40]	Hammond FL, Menguč Y, Wood RJ. Toward a modular soft sensor-embedded glove for human hand motion and tactile pressure measurement. In: Proceedings of 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS2014; Sep 14-18 ; Chicago IL, USA. 2014. p. 2014 New York City: IEEE; 4000-7.
[41]	Gu Y, Sheng W, Liu M, Ou Y. Fine manipulative action recognition through sensor fusion. In: Proceedings of 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS2015; 2015. p. 2015 Sep 28-Oct 2; Hamburg, Germany. New York City: IEEE; 886-91.
[42]	Mohammadi M, Baldi TL, Scheggi S, Prattichizzo D. Fingertip force estimation via inertial and magnetic sensors in deformable object manipulation. In: Proceedingsof the International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems ( HAPTICS2016; Apr 8-11 ; Philadelphia PA, USA. 2016. p. 2016 New York City: IEEE; 284-9.
[43]	B.S. Lin, I.J. Lee, J.L. Chen. Novel assembled sensorized glove platform for comprehensive hand function assessment by using inertial sensors and force sensing resistors. IEEE Sensors J, 20 (6) ( 2020), pp. 3379-3389
[44]	E. Battaglia, M. Bianchi, A. Altobelli, G. Grioli, M.G. Catalano, A. Serio, et al.. ThimbleSense: a fingertip-wearable tactile sensor for grasp analysis. IEEE Trans Haptics, 9 (1) ( 2016), pp. 121-133
[45]	Low JH, Khin PM, Yeow CH. A pressure-redistributing insole using soft sensors and actuators. In: Proceedings of 2015 IEEE International Conference on Robotics and Automation ( ICRA2015; May 26-30 ; Seattle WA, USA. 2015. p. 2015 New York City: IEEE; 2926-30.
[46]	Pugach G, Melnyk A, Tolochko O, Pitti A, Gaussier P. Touch-based admittance control of a robotic arm using neural learning of an artificial skin. In: Proceedings of 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS2016; Oct 9-14 2016. p. 2016 ; Daejeon, Republic of Korea. New York City: IEEE; 3374-80.
[47]	Müller S, Schröter C, Gross HM.Smart fur tactile sensor for a socially assistive mobile robot. In:Proceedings of International Conference on Intelligent Robotics and Applications (ICIRA 2015); 2015 Aug 24-27; Portsmouth, UK. Berlin:Springer; 2015. p. 49-60.
[48]	Jeong E, Lee J, Kim D. Finger-gesture recognition glove using Velostat. In: Proceedings of 2011 11th International Conference on Control, Automationand Systems ( ICCAS2011; Oct 26-29 2011. p. 2011 ; Gyeonggi-do, Republic of Korea. New York City: IEEE; 206-10.
[49]	Boulic R, Rezzonico S, Thalmann D. Multi-finger manipulation of virtual objects. In: Proceedingsof the ACM Symposium on Virtual Reality Software and Technology ( VRST1996; Jul 1-4 China. 1996. p. 1996 ; Hong Kong, New York City: Association for Computing Machinery (ACM); 67-74.
[50]	H. Choi, C. Crump, C. Duriez, A. Elmquist, G. Hager, D. Han, et al.. On the use of simulation in robotics: opportunities, challenges, and suggestions for moving forward. Proc Nat Acad Sci USA, 118 (1) ( 2019), Article e1907856118
[51]	Hu Y, Liu J, Spielberg A, Tenenbaum JB, Freeman WT, Wu J, et al. ChainQueen: a real-time differentiable physical simulator for soft robotics. In:Proceedings of 2019 International Conference on Robotics and Automation (ICRA 2019); 2019 Dec 4- 6; Montréal, QC, Canada. 2019. p. 6265-71.
[52]	M. Kennedy, K. Schmeckpeper, D. Thakur, C. Jiang, V. Kumar, K. Daniilidis. Autonomous precision pouring from unknown containers. IEEE Robot Autom Lett, 4 (3) ( 2019), pp. 2317-2324
[53]	Heiden E, Macklin M, Narang Y, Fox D, Garg A, Ramos F. DiSECt:a differentiable simulation engine for autonomous robotic cutting. In: Proceedings of the 2021 Robotics: Science and Systems ( RSS2021; Jul 12-16 2021 ; online. New York City: IEEE; 2021.
[54]	J. Wolper, Y. Fang, M. Li, J. Lu, M. Gao, C. Jiang.CD-MPM: continuum damage material point methods for dynamic fracture animation. ACM Trans Graph, 38 (4) ( 2019), p. 119
[55]	Lin J, Wu Y, Huang TS. Modeling the constraints of human hand motion. In: Proceeding Workshop on Human Motion; 2000 Dec 7-8; Austin, TX, USA. New York City: IEEE; 2000. p. 121-6.
[56]	B.W. Lee, H. Shin. Feasibility study of sitting posture monitoring based on piezoresistive conductive film-based flexible force sensor. IEEE Sensors J, 16 (1) ( 2016), pp. 15-16
[57]	Leap motion controller [Internet]. Mountain View: ultraleap; [cited 2023 Jan 5].
[58]	Intel® RealSense™ Technology [Internet]. Santa Clara: Intel; [cited 2023 Jan 5].
[59]	T. Feix, J. Romero, H.B. Schmiedmayer, A.M. Dollar, D. Kragic. The GRASP Taxonomy of human grasp types. IEEE Trans Hum Mach Syst, 46 (1) ( 2016), pp. 66-77
[60]	T. Liu, Z. Liu, Z. Jiao, Y. Zhu, S.C. Zhu. Synthesizing diverse and physically stable grasps with arbitrary hand structures using differentiable force closure estimator. IEEE Robot Autom Lett, 7 (1) ( 2022), pp. 470-477
[61]	Zienkiewicz OC, Taylor RL. The finite element method, volume 2: solid mechanics. 5th ed. Oxford: Butterworth-Heinemann; 2000.
[62]	M. Li. Robust and accurate simulation of elastodynamics and contact [dissertation]. University of Pennsylvania, Pennsylvania ( 2020)
[63]	M. Li, D.M. Kaufman,C. Jiang. Codimensional incremental potential contact. ACM Trans Graph, 40 (4) ( 2021), p. 170
[64]	Y. Fang, M. Li, C. Jiang, D.M. Kaufman.Guaranteed globally injective 3D deformation processing. ACM Trans Graph, 40 (4) ( 2021), p. 75
[65]	Z. Ferguson, M. Li, T. Schneider, F. Gil-Ureta, T. Langlois, C. Jiang, et al.. Intersection-free rigid body dynamics. ACM Trans Graph, 40 (4) ( 2021), p. 183
[66]	L. Lan, Y. Yang, D.M. Kaufman, J. Yao, M. Li, C. Jiang.Medial IPC: accelerated incremental potential contact with medial elastics. ACM Trans Graph, 40 (4) ( 2021), p. 158
[67]	Zhao Y, Choo J, Jiang Y, Li M, Jiang C, Soga K. A barrier method for frictional contact on embedded interfaces. 2021. arXiv:2107.05814.
[68]	M. Li, M. Gao, T. Langlois, C. Jiang, D.M. Kaufman.Decomposed optimization time integrator for large-step elastodynamics. ACM Trans Graph, 38 (4) ( 2019), p. 70
[69]	X. Wang, M. Li, Y. Fang, X. Zhang, M. Gao, M. Tang, et al.. Hierarchical optimization time integration for CFL-rate MPM stepping. ACM Trans Graph, 39 (3) ( 2020), p. 21
[70]	J. Nocedal, S. Wright. Numerical optimization. Springer Science & Business Media, Berlin ( 2006)
[71]	Hegemann J, Jiang C, Schroeder C, Teran JM. A level set method for ductile fracture. In: Proceedings of ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA); 2013 Jul 19-21; Anaheim, CA, USA. New York City: Association for Computing Machinery (ACM); 2013. p. 193-202.
[72]	M. Bourne. Food texture and viscosity: concept and measurement. Elsevier, Amsterdam ( 2002)
[73]	S.H. Williams, B.W. Wright, V. Truong, C.R. Daubert, C.J. Vinyard. Mechanical properties of foods used in experimental studies of primate masticatory function. Am J Primatol, 67 (3) ( 2005), pp. 329-346
[74]	M. Kiani, H. Maghsoudi, S. Minaei. Determination of Poisson’s ratio and Young’s modulus of red bean grains. J Food Process Eng, 34 (5) ( 2011), pp. 1573-1583
[75]	Edmonds M, Gao F, Xie X, Liu H, Qi S, Zhu Y, et al. Feeling the force:integrating force and pose for fluent discovery through imitation learning to open medicine bottles. In: Proceedings of 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS2017; Sep 24-28 2017. p. 2017 ; Vancouver, BC, Canada. New York City: IEEE; 3530-7.
[76]	Xie X, Li C, Zhang C, Zhu Y, Zhu SC. Learning virtual grasp with failed demonstrations via Bayesian inverse reinforcement learning. In: Proceedings of 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS2019; Nov 3-8 2019. p. 2019 ; Macao, China. New York City: IEEE; 1812-7.
[77]	S.S. Rautaray, A. Agrawal. Vision based hand gesture recognition for human computer interaction: a survey. Artif Intell Rev, 43 (1) ( 2015), pp. 1-54
[78]	K. Dautenhahn, C.L. Nehaniv. Imitation in animals and artifacts. MIT Press, Cambridge ( 2002)
[79]	H.KJ. Kubricht, H. Lu. Intuitive physics: current research and controversies. Trends Cogn Sci, 21 (10) ( 2017), pp. 749-759
[80]	E.S. Spelke. What babies know: core knowledge and composition, volume 1, Oxford University Press, Oxford ( 2022)
[81]	E.S. Spelke, K.D. Kinzler. Core knowledge. Dev Sci, 10 (1) ( 2007), pp. 89-96
[82]	Y. Zhu, T. Gao, L. Fan, S. Huang, M. Edmonds, H. Liu, et al.. Dark, beyond deep: a paradigm shift to cognitive AI with humanlike common sense. Engineering, 6 (3) ( 2020), pp. 310-345
[83]	Z. Zhang, Z. Jiao, W. Wang, Y. Zhu, S.C. Zhu, H. Liu. Understanding physical effects for effective tool-use. IEEE Robot Autom Lett, 7 (4) ( 2022), pp. 9469-9476
[84]	Li P, Liu T, Li Y, Geng Y, Zhu Y, Yang Y, et al. GenDexGrasp: generalizable dexterous grasping. 2022. arXiv:2210.00722.
[85]	Zhu Y, Zhao Y, Zhu SC. Understanding tools:task-oriented object modeling, learning and recognition. In: Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition ( CVPR2015; Jun 7-12 ; Boston MA, USA. 2015. p. 2015 New York City: IEEE; 2855-64.
[86]	M. Han, Z. Zhang, Z. Jiao, X. Xie, Y. Zhu, S.C. Zhu, et al.. Scene reconstruction with functional objects for robot autonomy. Int J Comput Vis, 130 (12) ( 2022), pp. 2940-2961
[87]	Han M, Zhang Z, Jiao Z, Xie X, Zhu Y, Zhu SC, et al. Reconstructing interactive 3D scene by panoptic mapping and cad model alignments. In: Proceedings of 2021 IEEE International Conference on Robotics and Automation ( ICRA2021; 2021. p. 2021 May 30-Jun 5; Xi’an, China. New York City: IEEE; 12199-206.
[88]	Chen Y, Huang S, Yuan T, Zhu Y, Qi S, Zhu SC. Holistic++ scene understanding:single-view 3D holistic scene parsing and human pose estimation with human-object interaction and physical commonsense. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision ( ICCV2019; 2019. p. 2019 Oct 27-Nov 2; Seoul, Republic of Korea. New York City: IEEE; 8647-56.
[89]	Huang S, Qi S, Xiao Y, Zhu Y, Wu YN, Zhu SC.Cooperative holistic scene understanding:unifying 3D object, layout and camera pose estimation. In:Proceedings of Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS 2018); 2018 Dec 3- 8 ; Montréal, QC, Canada. Red Hook: Curran Associates Inc.; 2018. p.206-17.
[90]	Huang S, Qi S, Zhu Y, Xiao Y, Xu Y, Zhu SC.Holistic 3D scene parsing and reconstruction from a single RGB image. In:Proceedings of 2018 15th European Conference on Computer Vision (ECCV 2018); 2018 Sep 14-18; Munich, Germany. Berlin:Springer; 2018. p. 194-211.
[91]	C. Li, W. Liang, C. Quigley, Y. Zhao, L.F. Yu. Earthquake safety training through virtual drills. IEEE Trans Vis Comput Graph, 23 (4) ( 2017), pp. 1275-1284
[92]	Zhu Y, Jiang C, Zhao Y, Terzopoulos D, Zhu SC. Inferring forces and learning human utilities from videos. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition ( CVPR2016; Jun 27-30 2016. p. 2016 ; Las Vegas, NV, USA. New York City: IEEE; 3823-33.
[93]	B. Zheng, Y. Zhao, J. Yu, K. Ikeuchi, S.C. Zhu. Scene understanding by reasoning stability and safety. Int J Comput Vis, 112 (2) ( 2015), pp. 221-238
[94]	Zheng B, Zhao Y, Yu JC, Ikeuchi K, Zhu SC. Beyond point clouds:scene understanding by reasoning geometry and physics. In: Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition ( CVPR2013; Jun 23-28 ; Portland OR, USA. 2013. p. 2013 New York City: IEEE; 3127-34.
[95]	Jiao Z, Zhang Z, Wang W, Han D, Zhu SC, Zhu Y, et al. Efficient task planning for mobile manipulation:a virtual kinematic chain perspective. In: Proceedings of 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS2021; 2021. p. 2021 Sep 27-Oct 1; Prague, Czech Republic. New York City: IEEE; 8288-94.
[96]	Jiao Z, Zhang Z, Jiang X, Han D, Zhu SC, Zhu Y, et al. Consolidating kinematic models to promote coordinated mobile manipulations. In: Proceedings of 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS2021; 2021. p. 2021 Sep 27-Oct 1; Prague, Czech Republic. New York City: IEEE; 979-85.
[97]	Jiao Z, Niu Y, Zhang Z, Zhu SC, Zhu Y, Liu H. Sequential manipulation planning on scene graph. In: Proceedings of 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS2022; Oct 23-27 2022. p. 2022 ; Kyoto, Japan. New York City: IEEE; 8203-10.
[98]	Taheri O, Ghorbani N. Black MJ, Tzionas D. GRAB: a dataset of whole-body human grasping of objects. In:Proceedings of 16th European Conference on Computer Vision (ECCV 2020); 2020 Aug 23-28, Glasgow, UK. Berlin:Springer; 2020. p. 581-600.
[99]	Wang Z, Chen Y, Liu T, Zhu Y, Liang W, Huang S.HUMANISE:language-conditioned human motion generation in 3D scenes. In:Proceedings of 36th Conference on Neural Information Processing Systems (NeurIPS 2022); 2022 Nov 28-Dec 9; New Orleans, LA, USA. Red Hook: Curran Associates Inc.; 2022.
[100]	Jiang N, Liu T, Cao Z, Cui J, Chen Y, Wang H, et al. CHAIRS: towards full-body articulated human-object interaction. 2022. arXiv:2212.10621.
[101]	Jia B, Chen Y, Huang S, Zhu Y, Zhu SC. LEMMA: a multi-view dataset for learning multi-agent multi-task activities. In:Proceedings of European Conference on Computer Vision (ECCV 2020); 2020 Aug 23-28; Glasgow, UK. Berlin:Springer; 2020. p. 1-7.