ECT-VR: Intuitive Teleoperation via Hybrid Virtual–Reality Immersive Interaction with Multi-Sensor Perception for Agricultural Robots

Tao Liu; Yinling Xie; Yang Li; Jun Zhou; Zhonghua Miao; Baohua Zhang; Chengliang Liu

doi:10.1016/j.eng.2026.09.013

Engineering ›› :202609013 DOI: 10.1016/j.eng.2026.09.013

research-article

ECT-VR: Intuitive Teleoperation via Hybrid Virtual–Reality Immersive Interaction with Multi-Sensor Perception for Agricultural Robots

Author information +

History +

PDF (9321KB)

Abstract

Fully autonomous robots are still difficult to deploy in complex, unstructured, and dynamic agricultural environments, whereas conventional teleoperation systems suffer from insufficient environmental information and poor spatial intuitiveness. To overcome these limitations, this paper presents enhanced conventional teleoperation with virtual reality (ECT-VR), an immersive teleoperation system for agricultural mobile robots based on hybrid three- and two-dimensional (3D+2D) perception and interaction. The system employs a collaborative architecture that combines a preconstructed global 3D map with real-time local color-depth feedback. In the 3D subsystem, aerial images are processed using structure-from-motion (SfM) and multi-view stereo reconstruction, together with multi-stage post-processing methods, to generate large-scale static agricultural maps for global environmental perception. In the 2D subsystem, an onboard color-depth camera provides real-time visual feedback, while depth subject comparison between ECT-VR and conventional video-streaming teleoperation will be introduced. Standardized task-level indicators, such as task completion time, path efficiency or trajectory optimality, collision or near-collision events, operation errors, and operator workload metrics, will be recorded. Appropriate paired statistical tests will also be used to quantify performance differences under different teleoperation modes.

Keywords

Teleoperation / Virtual reality / Agricultural mobile robots / 3D reconstruction / Human–robot interaction

Cite this article

Download citation ▾

Tao Liu, Yinling Xie, Yang Li, Jun Zhou, Zhonghua Miao, Baohua Zhang, Chengliang Liu. ECT-VR: Intuitive Teleoperation via Hybrid Virtual–Reality Immersive Interaction with Multi-Sensor Perception for Agricultural Robots. Engineering 202609013 DOI:10.1016/j.eng.2026.09.013

登录浏览全文

4963

注册一个新账户忘记密码

References

Publishing order | Descend order by publishing year | Descend order by cited within

[1]	Ghazal S, Munir A, Qureshi WS . Computer vision in smart agriculture and precision farming: techniques and applications. Artif Intell Agric 2024; 13: 64-83.

[2]	Juwono FH, Wong WK, Verma S, Shekhawat N, Lease BA, Apriono C . Machine learning for weed-plant discrimination in agriculture 5.0: an in-depth review. Artif Intell Agric 2023; 10: 13-25.

[3]	Pandey DK, Mishra R . Towards sustainable agriculture: harnessing AI for global food security. Artif Intell Agric 2024; 12: 72-84.

[4]	Liu T, Zhang B, Tan Q, Zhou J, Yu S, Zhu Q, et al. Immersive human-machine teleoperation framework for precision agriculture: integrating UAV-based digital mapping and virtual reality control. Comput Electron Agric 2024; 226: 109444.

[5]	Wang D, Zhang B, Zhou J, Xiong Y, Liu L, Tan Q . Three-dimensional mapping and immersive human-robot interfacing utilize Kinect-style depth cameras and virtual reality for agricultural mobile robots. J Field Robot 2024; 41(7): 2413-26.

[6]	Li G, Li Q, Yang C, Su Y, Yuan Z, Wu X . The classification and new trends of shared control strategies in telerobotic systems: a survey. IEEE Trans Haptics 2023; 16(2): 118-33.

[7]	Cho SK, Jin HZ, Lee JM, Yao B . Teleoperation of a mobile robot using a force-reflection joystick with sensing mechanism of rotating magnetic field. IEEE/ASME Trans Mechatron 2010; 15(1): 17-26.

[8]	Labonte D, Boissy P, Michaud F . Comparative analysis of 3-D robot teleoperation interfaces with novice users. IEEE Trans Syst Man Cybern B Cybern 2010; 40(5): 1331-42.

[9]	Yanco HA, Drury J . “Where am I?” acquiring situation awareness using a remote robot platform. In: Proceedings of the 2004 IEEE International Conference on Systems, Man and Cybernetics; 2004 Oct 10-13; The Hague, Netherlands. Piscataway: IEEE; 2004. p. 2835-40.

[10]	Uddin Z, Islam M . Search and rescue system for alive human detection by semi-autonomous mobile rescue robot. In: Proceedings of the 2016 International Conference on Innovations in Science, Engineering and Technology; 2016 Dec 15-17; Dhaka, Bangladesh. Piscataway: IEEE; 2016. p. 1-5.

[11]	Sharmin S, Salim SI, Sanim KRI . A low-cost urban search and rescue robot for developing countries. In: Proceedings of the 2019 IEEE International Conference on Robotics, Automation, Artificial-intelligence and Internet-of-Things; 2019 Dec 27-29; Dhaka, Bangladesh. Piscataway: IEEE; 2019. p. 60-4.

[12]	Ismail RM, Muthukumaraswamy S, Sasikala A . Military support and rescue robot. In: Proceedings of the 2020 4th International Conference on Intelligent Computing and Control Systems; 2020 Jun 11-13; Madurai, India. Piscataway: IEEE; 2020. p. 156-62.

[13]	Scholtz JC . Human-robot interactions: creating synergistic cyber forces. In: Schultz AC, Parker LE, editors. Multi-robot systems: from swarms to intelligent automata; 2001; Washington, DC, USA. Dordrecht: Springer Netherlands; 2002. p. 177-84.

[14]	Drury JL, Scholtz J, Yanco HA . Awareness in human-robot interactions. In: Proceedings of the 2003 IEEE International Conference on Systems, Man and Cybernetics; 2003 Oct 5-8; Washington, DC, USA. Piscataway: IEEE; 2003. p. 912-8.

[15]	Jacoff A, Messina E, Evans J. Reference test courses for autonomous mobile robots. In: Unmanned ground vehicle technology III; vol 4364; 2001 Aug 1-3; Orlando, FL, USA. Bellingham: SPIE; 2001. p. 341.

[16]	Kim GH, Kim KD, Lee HS, Choi YS, Mun HS, Oh JH, et al. Development of Wi-Fi-based teleoperation system for forest harvester. J Biosyst Eng 2021; 46(3): 206-16.

[17]	Praveena P, Molina L, Wang Y, Senft E, Mutlu B, Gleicher M . Understanding control frames in multi-camera robot telemanipulation. In: Proceedings of the 2022 17th ACM/IEEE International Conference on Human-Robot Interaction; 2022 Mar 7-10; Tokyo, Japan. New York City: ACM; 2022. p. 432-40.

[18]	Udekwe D, Seyyedhasani H . Human robot interaction for agricultural tele-operation, using virtual reality: a feasibility study. Comput Electron Agric 2025; 228: 109702.

[19]

Shen X, Chong ZJ, Pendleton S, James Fu GM, Qin B, Frazzoli E, et al. Teleoperation of on-road vehicles via immersive telepresence using off-the-shelf components. In: Menegatti E, Michael N, Berns K, Yamaguchi H, editors. Intelligent autonomous systems 13; 2015 Jun 24-26; Padua, Italy. Cham: Springer International Publishing; 2015. p. 1419-33.

[20]	Nakanishi J, Itadera S, Aoyama T, Hasegawa Y . Towards the development of an intuitive teleoperation system for human support robot using a VR device. Adv Robot 2020; 34(19): 1239-53.

[21]	Park S, Jung Y, Bae J . An interactive and intuitive control interface for a tele-operated robot (AVATAR) system. Mechatronics 2018; 55: 54-62.

[22]	Wei D, Huang B, Li Q . Multi-view merging for robot teleoperation with virtual reality. IEEE Robot Autom Lett 2021; 6(4): 8537-44.

[23]	Cai Z, Han J, Liu L, Shao L . RGB-D datasets using microsoft kinect or similar sensors: a survey. Multimed Tools Appl 2017; 76(3): 4313-55.

[24]	Yu S, Liu X, Tan Q, Wang Z, Zhang B . Sensors, systems and algorithms of 3D reconstruction for smart agriculture and precision farming: a review. Comput Electron Agric 2024; 224: 109229.

[25]

Pace FD, Gorjup G, Bai H, Sanna A, Liarokapis M, Billinghurst M . Leveraging enhanced virtual reality methods and environments for efficient, intuitive, and immersive teleoperation of robots. In: Proceedings of the 2021 IEEE International Conference on Robotics and Automation; 2021 May 30-Jun 5; Xi’an, China. Piscataway: IEEE; 2021. p. 12967-73.

[26]	Naceri A, Mazzanti D, Bimbo J, Tefera YT, Prattichizzo D, Caldwell DG, et al. The Vicarios virtual reality interface for remote robotic teleoperation. J Intell Robot Syst 2021; 101(4): 80.

[27]	Zhou T, Zhu Q, Du J . Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction. Adv Eng Inform 2020; 46: 101170.

[28]	Holubek R, Ružarovský R, Delgado Sobrino DR . Using virtual reality as a support tool for the offline robot programming. Res Pap Fac Mater Sci Technol Slovak Univ Technol 2018; 26(42): 85-91.

[29]

Van de Merwe DB, Van Maanen L, Ter Haar FB, Van Dijk RJE, Hoeba N, Van der Stap N . Human-robot interaction during virtual reality mediated teleoperation: how environment information affects spatial task performance and operator situation awareness. In: Chen JYC, Fragomeni G, editors. Virtual, augmented and mixed reality. applications and case studies; 2019 Jul 17-21; Orlando, FL, USA. Cham: Springer International Publishing; 2019. p. 163-77.

[30]	Pérez L, Diez E, Usamentiaga R, García DF . Industrial robot control and operator training using virtual reality interfaces. Comput Ind 2019; 109: 114-20.

[31]	Lesniak K, Tucker CS . Dynamic rendering of remote indoor environments using real-time point cloud data. J Comput Inf Sci Eng 2018; 18(3): 031006.

[32]	Wang C, Cho YK . Smart scanning and near real-time 3D surface modeling of dynamic construction equipment from a point cloud. Autom Constr 2015; 49: 239-49.

[33]	Schönberger JL, Frahm JM . Structure-from-motion revisited. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition; 2016 Jun 27-30; Las Vegas, NV, USA. Piscataway: IEEE; 2016. p. 4104-13.

[34]	Lowe DG . Distinctive image features from scale-invariant keypoints. Int J Comput Vis 2004; 60(2): 91-110.

[35]	Fischler MA, Bolles RC . Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 1981; 24(6): 381-95.

[36]	Triggs B, McLauchlan PF, Hartley RI, Fitzgibbon AW. Bundle adjustment-a modern synthesis. In: Triggs B, Zisserman A, Szeliski R, editors. Vision algorithms: theory and practice; 1999 Sep 13-17; Corfu, Greece. Berlin: Springer; 2000. p. 298-372.

[37]	Yu Z, Gao S . Fast-MVSNet: sparse-to-dense multi-view stereo with learned propagation and Gauss-Newton refinement. In: Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020 Jun 14-19; Seattle, WA, USA. Piscataway: IEEE; 2020. p. 1946-55.

[38]	Yao Y, Luo Z, Li S, Fang T, Quan L . MVSNet: depth inference for unstructured multi-view stereo. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y, editors. Computer vision-ECCV 2018; vol 11212; 2018 Sep 8-14; Munich, Germany. Cham: Springer International Publishing; 2018. p. 785-801.

[39]	Chen R, Han S, Xu J, Su H . Point-based multi-view stereo network. In: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision; 2019 Oct 27-Nov 2; Seoul, Republic of Korea. Piscataway: IEEE; 2019. p. 1538-47.

[40]	Jensen R, Dahl A, Vogiatzis G, Tola E, Aanæs H . Large scale multi-view stereopsis evaluation. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition; 2014 Jun 23-28; Columbus, OH, USA. Piscataway: IEEE; 2014. p. 406-13.

[41]	Yan J, Wei Z, Yi H, Ding M, Zhang R, Chen Y, et al. Dense hybrid recurrent multi-view stereo net with dynamic consistency checking. 2020. arXiv:2007.09728.

[42]	Rusu RB, Cousins S . 3D is here: point cloud library (PCL). In: Proceedings of the 2011 IEEE International Conference on Robotics and Automation; 2011 May 9-13; Shanghai, China. Piscataway: IEEE; 2011. p. 1-4.

[43]	Kazhdan M, Bolitho M, Hoppe H . Poisson surface reconstruction. In: Proceedings of the Fourth Eurographics Symposium on Geometry Processing; 2006 Jun 26-28; Goslar, Germany. Goslar: Eurographics Association; 2006. p. 61-70.

[44]	Taubin G. Curve and surface smoothing without shrinkage. In: Proceedings of the 1995 IEEE International Conference on Computer Vision; 1995 Jun 20-23; Cambridge, MA, USA. Piscataway: IEEE; 1995. p. 852-857.

[45]	Garland M, Heckbert PS . Surface simplification using quadric error metrics. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques; 1997 Aug 4-8; Los Angeles, CA, USA. New York City: ACM Press/Addison-Wesley Publishing Co.; 1997. p. 209-16.

[46]	Shirley P, Ashikhmin M, Marschner S . Fundamentals of computer graphics. 3rd ed. New York City: A K Peters/CRC Press; 2009.

[47]	Heckbert PS . Fundamentals of texture mapping and image warping. Report. Berkeley: University of California, EECS Department; 1989 Jun.

[48]	Labbé M, Michaud F . RTAB-Map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. J Field Robot 2019; 36(2): 416-46.

[49]	Rublee E, Rabaud V, Konolige K, Bradski G . ORB: an efficient alternative to SIFT or SURF. In: Proceedings of the 2011 International Conference on Computer Vision; 2011 Nov 6-13; Barcelona, Spain. Piscataway: IEEE; 2011. p. 2564-71.

[50]	Calonder M, Lepetit V, Strecha C, Fua P . BRIEF: binary robust independent elementary features. In: Daniilidis K, Maragos P, Paragios N, editors. Computer vision-ECCV 2010; vol 6311; 2010 Sep 5-11; Heraklion, Greece. Berlin: Springer; 2010. p. 778-92.