In smart manufacturing, autonomous mobile robots play an indispensable role in conducting inspection and material handling operations, yet they face significant limitations regarding adaptability and resilience within unstructured environments. Vision and language navigation (VLN), a human-guided navigation paradigm, emerges as a compelling solution to these challenges. Nevertheless, VLN’s practical implementation is constrained by limited task generalization capabilities, inadequate response to diverse linguistic commands, and insufficient consideration of sensor-induced noise in environmental perception. This research addresses these limitations by introducing an innovative vision-language model (VLM)-based human-guided mobile robot navigation approach in an unstructured environment for human-centric smart manufacturing (HSM). This approach encompasses three-dimensional (3D) robust scene reconstruction through advanced point cloud techniques, zero-shot semantic segmentation via a VLM, and natural language processing through a large language model (LLM) to interpret instructions and generate control code for navigation. The system’s efficacy is validated through extensive experiments in an unstructured manufacturing setup.
Industry 5.0 represents a paradigm shift toward sustainable and resilient manufacturing systems, emphasizing human-centric smart manufacturing (HSM) that places human well-being at its core [1]. Mobile robotics constitutes a fundamental component within smart manufacturing ecosystems, with widespread applications in inspection protocols, warehouse logistics, and material handling operations [2,3]. While research on mobile robots in manufacturing has focused on localization, tracking, trajectory planning, and motion control, aiming to achieve fully automated and self-organized systems [4], current technological limitations impede the realization of completely self-organized manufacturing environments, particularly regarding system resilience and adaptability. These constraints become especially pronounced in unstructured environments characterized by unpredictable tasks, variable structural configurations, and irregular dimensional parameters, where conventional pre-programmed approaches prove inadequate [5].
Within complex operational environments, intelligent systems must deploy sophisticated methodologies for information acquisition, processing, and environmental cognition to establish reliable situational awareness [6,7]. While structured automation lines represent the manufacturing ideal, their implementation remains impractical in numerous scenarios. The assembly of large-scale aircraft exemplifies this challenge, where the integration of components within and around massive, irregularly shaped fuselages currently necessitates manual intervention. The spatial constraints, coupled with task complexity and variability, preclude workers from maintaining immediate access to all requisite materials, necessitating auxiliary personnel for tool and component logistics, thereby compromising operational efficiency. Similarly, in maintenance operations, despite initial equipment preparation, the dynamic nature of inspection processes frequently reveals unforeseen requirements for additional tools or components. This iterative need for resource retrieval, often extending across multiple maintenance sessions, significantly impacts workflow continuity and operational effectiveness.
In alignment with Industry 5.0 principles, the integration of human-robot interaction (HRI) into mobile robotic systems presents a viable solution to contemporary manufacturing challenges [8,9]. This symbiotic approach harmonizes human cognitive strengths—including creativity and adaptive problem-solving capabilities—with robotic advantages in speed and endurance, facilitating the development of flexible and resilient HSM systems [10], [11], [12], [13]. The recent advancement of large language models (LLMs) and vision-language models (VLMs) has revolutionized vision and language-based HRI, offering unprecedented capabilities [13], [14], [15], [16]. These technologies demonstrate remarkable generalization to novel scenarios, enhance robotic perception through human-like multimodal processing, and establish intuitive communication interfaces [17]. Vision and language-based human-guided navigation emerges as a particularly promising paradigm for unstructured manufacturing environments characterized by dynamic, unpredictable conditions. This approach enables mobile robots to synthesize natural language commands with visual environmental data, facilitating adaptive interaction with both human operators and physical surroundings. Such capabilities prove invaluable in complex assembly and maintenance scenarios, where robots can autonomously respond to engineers’ evolving requirements for tool and component retrieval. This automation of ancillary tasks enables technical personnel to focus on specialized functions, substantially improving operational efficiency and resource utilization within manufacturing processes.
While existing mobile robot systems primarily rely on pre-programmed tasks and autonomous navigation algorithms in structured environments, they lack the flexibility to adapt to unfixed manufacturing scenarios and real-time human needs [3], [18], [19], [20]. In contrast to traditional approaches, vision-language model (VLM)-based methods enable natural HRI and environmental recognition, allowing robots to understand and respond to spontaneous human commands in unstructured manufacturing environments. This integration of VLM with mobile robot navigation demonstrates superior adaptability in addressing unpredictable demands that may arise during manufacturing processes, particularly in complex scenarios such as large aircraft assemblies where traditional automation methods prove inadequate. However, previous research [21], [22], [23], [24] about vision and language-based HRI for robot navigation encounters predominantly two fundamental constraints. Firstly, most endeavors exhibit inadequate generalization capabilities regarding un-seen objects and environments, as well as the diverse and variable natural language expressions of humans. This deficiency severely restricts their application in complex and variable manufacturing environments. Secondly, although most works perform well within simulators, in the real world, sensor noise can lead to a cluttered and confusing reconstruction of top-down maps, aliasing geometric details and impeding accurate segmentation of navigation targets. This limitation prevents their effective empowerment in the industry.
To overcome the challenges, this research proposes a VLM-based human-guided mobile robot navigation approach in an unstructured environment for HSM. This system consists of three modules: three-dimensional (3D) robust scene reconstruction, VLM-based semantic segmentation and integration, and LLM-based spatial navigation. This work aims to facilitate human control over robot goal navigation for completing manufacturing tasks in unstructured settings through a natural interaction mode. Fig. 1 illustrates an example of our proposed method applied to an inspection task. The rest of the paper is organized as follows: Section 2 reviews related work in mobile robots in smart manufacturing as well as vision and language navigation (VLN). Section 3 primarily introduces an overview and the three components that constitute the system (3D robust scene reconstruction, semantic segmentation and integration, and spatial goal navigation). In Section 4, experiments on the three parts of the system are conducted to verify their effectiveness, and the corresponding analysis and discussion of the experimental results are provided. Section 5 gives a conclusion of this paper, as well as discusses the limitations and future perspectives of this work.
2. Related work
2.1. Mobile robots in smart manufacturing
Mobile robots have become integral components of smart manufacturing systems, serving diverse applications including automated inspection of large-scale aviation components [2], material transportation within industrial facilities [25], and workshop logistics distribution [3]. Contemporary manufacturing research predominantly emphasizes autonomous navigation algorithms, encompassing localization, tracking, trajectory planning, and motion control. Yilmaz et al. [3] developed a scan matching-based localization framework to overcome autonomous docking and localization challenges in factory logistics operations. Su et al. [18] implemented magnetic tracking methodology to enhance autonomous guided vehicle (AGV) positioning precision in manufacturing logistics. Goutham et al. [19] introduced a multi-objective optimization approach for path selection utilizing waypoint sequence representation to address material handling complexities. Demesure et al. [20] engineered an innovative hybrid control architecture for shop floor logistics robotics.
Contemporary mobile robots predominantly function within structured production environments or warehouses, requiring human pre-programming of tasks and sequences prior to deployment. However, the manufacturing sector encompasses numerous unstructured environments characterized by dynamic tasks and unpredictable conditions that defy predefinition. Large aircraft assembly, for instance, remains largely manual due to aircraft dimensions, geometric complexity, and component multiplicity, rendering structured automation lines impractical. Spatial constraints and task intricacy necessitate additional personnel for tool and component retrieval, creating operational inefficiencies. Moreover, maintenance operations frequently require technicians to procure additional tools as inspections progress, resulting in multiple retrieval cycles and extended problem-resolution periods. These complex operations in unstructured environments, currently executed manually, exhibit significant inefficiencies. Consequently, there exists a critical demand for mobile robots capable of natural human interaction while seamlessly integrating human directives with environmental perception for collaborative reasoning and task completion. We therefore propose the synthesis of HRI methodologies with mobile robot navigation, enabling technical personnel to concentrate on their core competencies and value-added activities, thereby substantially enhancing overall production efficiency.
2.2. Vision and language navigation
VLN denotes the process by which an AI agent interprets navigation instructions conveyed through natural language interactions with humans and subsequently navigates to the designated target [26]. This paradigm underscores the integration of vision-language interaction, facilitating joint reasoning across 3D environments and human natural language. The incorporation of VLN into mobile robotic systems has the potential to substantially augment the robots’ capacity for human interaction, thereby advancing progress toward HSM.
Previously, most work in VLN utilized traditional learning-based approaches, such as deep learning models [21], [27], [28], [29]. These methods typically rely on large volumes of task-specific data for training and perform well on training sets but often exhibit limited generalization capabilities when confronted with unfamiliar objects or diverse linguistic instructions. This limitation primarily stems from the models’ tendency to capture only the patterns and correlations present in the training data, resulting in constrained handling of new scenarios beyond the training set.
Recently emerged LLMs and VLMs are pre-trained on extensive, diverse datasets, enabling them to learn generalized representations across a broad spectrum of language and visual concepts. These models not only excel on specific task training data, but more importantly, enhance adaptability to new situations through a deeper understanding of broader contexts and concepts [30,31]. Moreover, these large models typically employ more advanced architectures, such as Transformers [32], which make them more effective in processing natural language and visual information, capturing long-range dependencies and complex contextual details better. In this way, LLMs and VLMs can more accurately mimic human multimodal perceptual abilities, offering more natural and flexible interaction and communication methods. This capability is crucial for providing stable and reliable VLN across various environments and scenarios.
A significant number of VLN models combining LLMs and VLMs have been developed, including VLMaps [22], NavGPT [33], and LM-Nav [34]. These models help autonomous agents navigate by processing visual information and natural language instructions, utilizing LLMs for language understanding and VLMs for visual perception. Despite promising results in simulated environments with clean sensor data, these models face challenges in real-world scenarios due to sensor noise from cameras. This noise leads to degraded sensory inputs and loss of fine geometric details in reconstructed maps, impairing the agents’ ability to perceive their surroundings accurately. Therefore, despite the advancements in VLN models leveraging LLMs and VLMs, there remains a significant gap between their performance in controlled simulations and their effectiveness in real-world applications. Addressing these challenges necessitates the development of more robust models and algorithms that can handle sensor noise and maintain high levels of accuracy in perception and navigation under real-world conditions.
3. Methodology
To realize vision-and-language-based robot navigation in unstructured environments pertinent to HSM, we propose a VLM-based human-guided mobile robot navigation system designed to mitigate the impact of sensor noise on reconstructed maps under realistic conditions, as shown in Fig. 2. The input of the system is red-green-blue depth (RGB-D) video frames as well as natural language instructions, while the output of the system is the Python code to control the robot to perform navigation tasks. The system consists of three main parts:
(1) 3D robust scene reconstruction. The first component of the system involves 3D robust scene reconstruction, where a pipeline including several popular point cloud registration methods is employed to reconstruct the scene point cloud based on input RGB-D videos and mitigate the impact of noise on geometric structure reconstruction. Based on the reconstructed scene’s mesh model and the robot’s altitude, an obstacle map is calculated for subsequent robot navigation path planning.
(2) Semantic segmentation and integration. The second part of the system involves semantic segmentation and integration. A VLM is utilized for language-driven semantic segmentation of RGB images in the input video. Subsequently, the semantic information is integrated into the reconstructed 3D scene mesh using the RGB-D integration method to obtain 3D scene semantics.
(3) Spatial goal navigation. The third component of the system involves spatial goal navigation. An LLM is employed to comprehend natural language commands from humans. By designing the prompts and integrating 3D scene semantics, natural language is translated into robot control code to trigger the robot to perform navigation actions.
3.1. 3D robust scene reconstruction
One key element in reconstructing a 3D scene is the accurate estimation of camera pose, which directly impacts the precision of the reconstruction results. In manufacturing environments, where the surroundings are often cluttered and noisy, precise estimation of the camera pose on a mobile robot using sensors such as inertial measurement units (IMUs) and accelerometers becomes exceedingly challenging and fails to meet the required accuracy. Therefore, this research utilized visual odometry calculation based on RGB-D image information to estimate camera pose. Specifically, this research employs a pipeline [35] comprising multiple point cloud registration methods to mitigate noise and visual odometry drift for precise and robust scene reconstruction, as shown in Fig. 3.
In Fig. 3, the entire reconstruction process is divided into two stages to reduce the impact of odometry drift. In the first stage, the input video frames are divided into N segments, each containing L pairs of RGB-D images. The reconstruction process is performed on each segment, resulting in N point cloud fragments. The reconstruction process in the first stage comprises four essential steps: colored point cloud registration [36], fast global registration [37], multiway registration [35], and truncated signed distance function (TSDF) volume integration [38]. The colored point cloud registration finds the camera movement between two consecutive RGB-D image pairs based on color and geometry information. Fast global registration aligns non-consecutive pairs based on geometry information. Multiway registration could prune erroneous constraints created by global registration for robust optimization. And finally, all the frames in this sequence are fused to a fragment mesh model by TSDF volume integration. The second stage entails applying a similar reconstruction process on these N fragments to generate a complete 3D scene mesh, including fast global registration [37], iterative closest points (ICP) registration [39], multiway registration [35], and TSDF volume integration [38]. The fast global registration roughly aligns the fragments as the initialization before ICP registration. Then point-to-plane ICP registration is applied to do the tight alignment, followed by multiway registration to prune erroneous constraints. The ICP registration and multiway registration are applied twice to refine the fragments’ alignment. Finally, based on the optimized pose graph, TSDF volume integration is applied to fuse all the video frames to a whole mesh model.
The obstacle map is a binary map derived from the reconstructed 3D scene and calculations of the robot’s dimensions, serving as the foundation for obstacle avoidance and path planning for subsequent robot navigation. It is crucial to customize the obstacle map based on the robot’s dimensions. For example, a 1.5 m-wide passageway is considered an available area for a robot with a 1 m diameter but considered an obstacle for a robot with a 2 m diameter. The obstacle map calculation method is very mature and can be easily obtained from the simulator, such as the AI Habitat simulator [40], and one may refer to Refs. [41,42] for more details.
3.2. Semantic segmentation and integration
Based on the aforementioned work, one core contribution of this research lies in the fusion of VLM with optimized 3D reconstruction results to achieve a zero-shot understanding of the 3D scene. To achieve this objective, this work employs RGB data to represent the semantic information derived from the VLM segmentation of each RGB image, and integrate the semantic information into the point cloud surface reconstruction phase using the TSDF algorithm, as shown in Fig. 4.
3.2.1. Semantic segmentation
The semantic segmentation part in our work utilizes the LSeg model [43] to perform language-driven semantic segmentation on all RGB frames within the video. LSeg is a VLM, in which the text encoder is the contrastive language-image pretraining (CLIP) text encoder, it is utilized to generate the text embeddings of the input text labels, such as “workstation” and “floor,” and allows for the free-form description of these input labels. In this work, text labels are extracted by an LLM from human natural language descriptions of the environment, pinpointing navigation targets. For instance, when humans describe the environment as “a shopfloor with a cabinet, a table, a shelf, and a workstation,” the LLM identifies “cabinet, table, shelf, and workstation,” facilitating human-centric interaction in smart manufacturing. The image encoder in LSeg is transformer-based, it can generate pixel-level image embeddings. The semantic information can be obtained by calculating the similarity between text embeddings and image embeddings.
In this work, the fine-tuned LSeg’s visual encoder $f(I): R^{H \times W \times 3}: \rightarrow R^{H \times W \times C}$ is applied to RGB image frame Ik with height H and width W to get pixel level visual embedding $F_{k} \in R^{H \times W \times C}$, the feature dimension for each pixel is C. For each pixel Iij, the corresponding visual embedding (qij) is
where $q_{i j} \in R^{1 \times C}$, i and j are the pixel coordinates, and k is the kth frame. Similarly, the pre-trained CLIP text encoder is applied to M text labels to obtain the text embeddings
where $E \in R^{M \times C}$, e0, e1, …, eM means the text embeddings extracted by text encoders. Then the inner product is performed on image and text embeddings to compute the pixel-to-category similarity matrix
where $S_{i j} \in R^{1 \times M} $. In the similarity matrix, the index corresponding to the maximum value represents the label of the pixel, indicating its semantic information. Different RGB values are utilized to represent distinct semantic information. Therefore, the semantic information of the pixel Iij is cij, which is an 8-bit RGB value.
3.2.2. Semantic integration
The transformation between the camera coordinate system and the world coordinate system is a crucial link that connects two-dimensional (2D) pixels with 3D voxels. The transformation from camera frame to world coordinate is
where Rk and Tk means the precise camera pose which has been calculated and optimized in the 3D point cloud robust reconstruction, Pcam is the camera coordinate, and Pwrd is the world coordinate. By employing this transformation, the semantic information of each frame can be integrated with the 3D geometric information.
After that, TSDF integration is applied to integrate all the frames into the global point cloud. For a voxel Vxyz located in (x, y, z) in the global point cloud, its coordinate in world coordinate is $P_{V_{x y z}, \text { wrd }}$ which can be obtained according to voxel length. Based on Eq. (4), the corresponding camera coordinate $ P_{V_{x y z}, \text { cam }} $ can be calculated. Hence, the depth of the voxel Vxyz relative to the camera Camz (Vxyz) can be easily obtained. The principle of camera imaging is
where Kcam means the camera intrinsics matrix. Based on that, the corresponding pixel Iij is found. Thus, the sdf of Vxyz is
where D(Iij) is the depth value of pixel Iij. Therefore, tsdf value of this voxel $ \operatorname{tsdf}_{V_{x y z}} $ is
where t is tsdf_trune = 0.04. Therefore, the tsdf value $ \operatorname{tsdf}_{V_{x y z}} $ and semantic information cij are stored in a voxel Vxyz. By averaging the tsdf value $ \operatorname{tsdf}_{V_{x y z}} $ and semantic value cij of each frame within this voxel Vxyz, the overall information of the entire model can be obtained
The areas where the TSDF value is zero represent the surface of the object. This can be identified using the Marching Cubes algorithm. By assigning colors based on the semantic information values, the final 3D semantic map is generated. In the generated 3D scene semantic mesh, all vertices belonging to the same object have the same RGB value. By averaging the coordinates of these vertices with the same RGB value, the center coordinates of the object can be obtained. In this way, 3D scene semantics are generated, which include the semantic description for each navigation object and its corresponding center coordinate.
3.3. Spatial goal navigation
In spatial goal navigation, the GPT-3.5 model (OpenAI, USA) is utilized to convert instructions into executable Python code, as shown in Fig. 5. To achieve this, a set of initial prompts is established to outline the VLN task. Subsequently, dialog examples are provided to illustrate the mapping between inputs and outputs. User instructions are then incorporated. The prompt is sent to the GPT model via an application programming interface (API) to generate the Python code. The Python function move_to_obj(object) is predefined, including object coordinate indexing, path planning, and controlling the robot to move to the object from its current position. LLM can identify each navigation object from the natural language instruction and generate the code to call the Python function. If there are no compilation errors, the code is executed to carry out the robot navigation actions.
4. Experiments and discussions
In this section, three separate experiments targeting each component of the system are conducted to verify its overall feasibility and effectiveness.
4.1. Evaluation of semantic segmentation
LSeg is selected as the model for zero-shot semantic segmentation to perform semantic segmentation on the scene’s 2D RGB images. The semantic segmentation experiment is divided into two parts: the first part focuses on the conventional semantic segmentation capability assessment, while the second part focuses on the zero-shot capability evaluation.
4.1.1. Experimental setting
A segmentation dataset is collected comprising 175 images: 105 images for training, 35 for validation, and 35 for testing. Notably, the scenes represented in the test set were distinct from those encountered during the training phase. The LSeg image encoder is fine-tuned on the custom dataset using a Nvidia RTX 3090 GPU (Nvidia, USA) to ensure it can accurately segment targets within industrial scenes; meanwhile, the LSeg text encoder remains frozen to preserve the model’s zero-shot capabilities.
4.1.2. Experimental results
To validate the accuracy of semantic segmentation, this work conducts comparative experiments between LSeg and two fully supervised semantic segmentation models, U-Net [44] and DeepLabV3+ [45], all of which are fine-tuned using the same dataset. U-Net [44] has a symmetric U-shaped structure and skip connections of feature maps, which enable the model to effectively combine contextual information and detailed features of images during segmentation tasks. DeepLabv3+ [45] is known for its robust performance in various segmentation challenges, which employs atrous convolution and an encoderdecoder structure to enhance segmentation accuracy. This research evaluates the models using pixel accuracy. The semantic segmentation comparative experiment result is illustrated in Table 1 [43], [44], [45]. UNet and DeepLabv3+, while designed for fully supervised learning, show expectedly strong performance due to their direct training on annotated images. Remarkably, LSeg demonstrates comparable results to the fully supervised models, which is an impressive feat considering its zero-shot learning capability.
The zero-shot capabilities of LSeg are exemplified by its ability to semantically segment images based on textual descriptions that it has not encountered during the training phase. This is largely attributable to the model’s utilization of the CLIP text encoder. The CLIP model maps visual and textual information into a shared embedding space, bringing semantically similar items—such as labels and their corresponding visual representations—closer together. This allows LSeg to directly use text embeddings for segmentation without specialized training for each possible category. Due to the diversity of training data and the model’s strong learning capabilities, CLIP’s text encoder forms a well-structured semantic space within the embedding space. This means that even for labels not seen during training, as long as they are semantically related to labels present in the training data, the model can find similar representations in the embedding space, thereby achieving generalization to these new labels [43]. The labels employed to refine the image encoder encompass the following items: table, chair, workstation, shelf, automated guided vehicle, cabinet, floor, wall, and back-ground. Notably, when diverse terms are utilized to denote the same objects, accurate recognition is achieved, as illustrated in Fig. 6.
Fig. 6(a) displays the results of semantic segmentation indexed by the original training labels, while Figs. 6(b)-(d) illustrate the results of semantic segmentation indexed using labels not encountered during the training process. In Fig. 6(b), the labels “shelving, robot, and floor” are unseen labels, yet the model can accurately segment them. Notably, the segmentation result for the “robot” label is remarkable. In Fig. 6(a), within the original training dataset, the labels “workstation and automated guided vehicle” are employed, both of which encompass elements of a “robot arm,” thereby falling under the category of “robot.” In Fig. 6(b), the “robot” label is solely used, enabling the model to successfully segment both objects as “robot,” demonstrating its robust zero-shot capabilities. In Fig. 6(c), new labels such as “desk, floor, mobile robot, and HRC workstation” have been employed for textual descriptions in semantic segmentation, and the model has successfully and accurately segmented these targets as depicted in the figure. Noteworthy are the labels “HRC workstation and mobile robot,” which significantly differ from the original training labels “workstation and automated guided vehicle,” yet the model has successfully correlated and segmented them accordingly. Particularly, in Fig. 6(b), the segmentation of “mobile robot” does not categorize all objects containing “robot” elements but successfully interprets the meaning of “mobile,” thereby isolating only the “automated guided vehicle.” This demonstrates the model’s formidable comprehension capabilities. In Fig. 6(d), the unseen labels include “desk, floor, AGV, and HRC workstation,” and the model has correctly completed the segmentation. Notably, the label “AGV,” which stands for “automated guided vehicle,” consists of an abbreviation rather than a full word to convey semantic information. Nevertheless, the model has successfully associated the abbreviation “AGV” with its full term, demonstrating the model’s powerful inferential capabilities.
The demonstrated robust generalization ability of LSeg permits humans to refer to objects in an unstructured manufacturing scene according to their own conventions, eliminating the need to predefine and adhere strictly to specific names for each object. This advancement brings the entire system closer to a more natural HRI.
4.2. Evaluation of 3D reconstruction
To mitigate the impact of sensor errors on the reconstruction outcomes, a pipeline comprising several point cloud registration and integration methods is employed in this system. This experiment aims to validate the accuracy of 3D reconstruction under this pipeline, assessing it from both quantitative and qualitative perspectives.
4.2.1. Experimental setting
Data are collected by an Intel RealSense L515 camera (Intel, USA) in real-world settings including three areas, each comprising hundreds of frames of RGB-D video for scene reconstruction. During the reconstruction process, a Nvidia RTX 3080 (Nvidia, USA) is employed to accelerate the computation.
Throughout the reconstruction process, the RGB-D sequence is divided into multiple segments, each containing 100 frames. Every fifth frame is selected as a key frame for odometry estimation and optimization, balancing computational efficiency with registration accuracy. Valid depth values between 0.3 and 3 m are considered to filter out invalid data that are too close or too distant from the camera, thereby reducing noise and errors. A voxel size of 0.05 m is employed for point cloud down sampling and TSDF fusion. During registration, the maximum acceptable distance threshold between corresponding point pairs is set to 0.07 m to reduce mismatches by filtering out pairs with excessive discrepancies. The fitness threshold for registration results is set at 0.3. Only when the fitness exceeds this value is the registration considered valid, ensuring that only high-quality registrations are accepted while unreliable ones are rejected. In SLAC optimization, the maximum number of iterations is limited to five to ensure completion within reasonable computation time and to prevent overfitting and excessive computation. For TSDF fusion, the cubic size is set to 3 m to define the dimensions of the reconstruction space. Additionally, the truncation distance in TSDF fusion is specified as 0.04 m. This truncation threshold affects the level of detail in surface reconstruction and the overall fusion effectiveness.
4.2.2. Experimental results
For quantitative evaluation, iPhone-based light detection and ranging (LiDAR) scanning results are employed as the ground truth, with the Chamfer distance between the reconstructed point clouds and the ground truth calculated to verify the accuracy of the scene reconstruction. The Chamfer distance is a method for measuring the distance between two sets of points, representing the aggregate of distances from one set to another. It is commonly used to compare the similarity between two geometric shapes. The formula for calculating the Chamfer distance $ d_{\mathrm{CD}}(A, B) $ between point cloud $ A=\left\{a_{1}, a_{2}, \ldots, a_{m}\right\} $ and point cloud $ B=\left\{b_{1}, b_{2}, \ldots, b_{n}\right\} $ is
where m ∈ N+ and n ∈ N+.
The smaller the Chamfer distance, the greater the similarity between the two point clouds. This work evaluates the reconstruction effectiveness of the system by calculating the Chamfer distance between the point cloud reconstructed from RGB-D frames and the ground truth. The reconstruction visualization and experiment results are provided in Fig. 7 and Table 2.
The entire scenes are scanned using LiDAR of an iPhone and the reconstructed point clouds are illustrated in Figs. 7(a) and (d). Subsequently, the scenes are divided into three distinct areas, and the reconstruction is performed based on RGB-D for each area; the point clouds of the three reconstructed areas are depicted in Figs. 7(b), (c), and (e). Table 2 presents the Chamfer distance between the reconstruction outcomes of the three areas and their corresponding ground truth, utilized to estimate the accuracy of the 3D reconstructions based on RGB-D frames. The Chamfer distance for each area does not exceed 10 cm, with the average Chamfer distance being 6.9 cm. Given that the two entire scenes encompass more than 10 m2, the reconstruction error is comparatively within an acceptable range. In addition, the reconstruction times for each region are detailed in Table 2, averaging approximately 9.38 min. Consequently, an acceptable trade-off between reconstruction speed and accuracy has been achieved.
For the qualitative comparative experiment, the baseline model is VLMaps as they use a similar framework to ours, but they do not account for sensor noise in real-world industrial scenarios in map reconstruction. Since VLMaps does not execute a complete 3D reconstruction but instead directly produces a reconstructed 2D top-down map, the comparison of reconstruction outcomes is confined to qualitative assessment through images obtained from identical datasets. The qualitative comparative experiment results are provided in Fig. 8.
In the comparative experiment, 3D reconstructions are conducted based on the same dataset by both methods, and then the same fine-tuned LSeg model is applied for semantic segmentation of RGB images within the dataset, finally, the segmentation results are integrated into the reconstructed point clouds. In Fig. 8, the left column displays images of 3D reconstructions incorporating geometric information, while the right column presents images of 3D reconstructions enriched with semantic information. As illustrated in Fig. 8(a), the geometric details in the VLMaps reconstruction results are considerably disordered due to the absence of any camera pose optimization technique, affected by errors in camera pose estimation. Even though LSeg successfully segments navigation targets from RGB images, the reconstructed map is unsuitable for real-world applications. In contrast, Fig. 8(b) shows our reconstruction results, where the geometric details are remarkably clear, and the semantic information is accurately integrated, significantly empowering tasks related to VLN.
4.3. Spatial goal navigation
4.3.1. Experimental setting
The spatial goal navigation experiments are conducted within the AI Habitat simulator [40], which is an open-source simulation platform that offers a highly realistic 3D environment. This platform enables AI agents to test a variety of tasks, including navigation and HRI. In this experiment, GPT-3.5 is linked with the simulator, wherein humans input natural language commands through a dialogue box. GPT-3.5 then generates and executes the code to control the robot’s movement based on the given instructions. Subsequently, the robot performs path planning and navigates to the target location. Upon completing the entire set of instructions, the robot returns a video showcasing the entire movement process from a first-person perspective. The primary focus of this work is on enabling the robot to comprehend language instructions and autonomously identify all navigation target points. Therefore, the process of the robot moving to the subgoals is not concentrated. This work assumes that once the coordinates of the subgoals are identified, the robot will successfully move to those locations. Consequently, the path-planning algorithm is not the emphasis of this work, and this research directly employs the Pathfinding module provided by the simulator to accomplish the path planning.
4.3.2. Experimental results
This experiment tests whether GPT-3.5 could accurately generate the corresponding code and successfully execute it using different natural language instructions. Specifically, the natural language instructions are categorized based on the number of navigation subgoals contained within them, resulting in four categories, encompassing the number of subgoals from 1 to 4. For each category, the tests are conducted with 20 distinct natural language instructions and tallied the success rates. The results are displayed in Table 3.
The data in Table 3 represents the overall navigation success rate of the system. The success rate decreases as the number of subgoals contained within the natural language instructions increases, with an average success rate of 92.5%. During the experiments, it is discovered that errors primarily occur when the sequence of subgoals mentioned in the natural language instructions does not align with the actual intended sequence. For instance, if the input command is “come to the shelf, before that go to check the equipment on the table,” the LLM would direct the robot to first approach the shelf and then the table, which is contrary to the human’s desire for the robot to visit the table before the shelf. However, changing the sequence does not necessarily lead to errors. For example, if the input command emphasizes the order more clearly with “come to the shelf, but before that, firstly you should go to check the equipment on the table,” the LLM would correctly understand the sequence and direct the robot to visit the table before the shelf.
4.4. Case study
Finally, a case study in an industrial setting is presented to validate the proposed VLM-based human-guided mobile robot navigation system in HSM. The experimental scenario, depicted in Fig. 1, takes place in a smart manufacturing laboratory consisting of a workstation, cabinet, shelf, and desk. The workstation serves as a human-robot collaborative assembly bench, while the cabinet, shelf, and desk store various parts. Fig. 9 illustrates the process of the pick-and-place case study.
In Fig. 9, a human gives a natural language command, “take the tape from the cabinet to the shelf, then take the gear from the table to the workstation.” Upon identifying the navigation targets, the LLM generates navigation code and executes it line by line, guiding the mobile robot to the destinations to complete the pick-and-place tasks. For each line of code, the coordinates of the navigation targets are derived from the 3D scene semantics. Traditional localization and path planning are handled by invoking the existing AGV APIs. It is important to note that this task focuses solely on navigation, with the AGV navigation code generated by the LLM based on human verbal commands, while the code related to the robotic arm is pre-configured to provide a comprehensive demonstration of the case study.
This pick-and-place case study validates the effectiveness of the proposed VLM-based human-guided mobile robot navigation system in unstructured HRI and smart manufacturing scenarios. The case study demonstrates the system’s significant potential in real industrial applications, showcasing its ability to effectively enhance the efficiency of HRI and HRC, thereby enabling the realization of HSM for the future of Industry 5.0.
5. Conclusions
To improve the resilience and flexibility of mobile robot navigation systems, and further realize HSM in Industry 5.0, this article presented a VLM-based human-guided mobile robot navigation approach, including 3D robust scene reconstruction, VLM-based semantic segmentation and integration, and LLM-based spatial goal navigation. The main contributions of this work lie in two aspects: A VLM- and LLM-integrated approach was proposed to enable humans to guide mobile robot navigation in unstructured manufacturing environments using free-form natural language; an enhanced VLN framework was explored by integrating a pipeline featuring a series of point cloud registration and fusion methods, which significantly mitigates the impact of sensor noise on 3D reconstruction.
The main limitation of this work lies in the sole focus on the navigation of the mobile robot without considering the operational phase upon reaching the navigation destination. Additionally, there are other limitations, such as reliance on action primitives and the lack of updates for dynamic scenes. In the future, to further empower smart manufacturing with VLMs and LLMs, several directions are highlighted: LLMs can be applied directly to the trajectory planning of robotic arms and end-effectors, enabling robot systems based on LLMs to perform more granular manipulation tasks; future initiatives could explore the incorporation of real-time environmental updates to further optimize VLN systems; integrating augmented reality (AR) into VLN interactions may largely enhance the system’s interactivity in the future; future work focus on enhancing system security, stability, and other related factors to develop more reliable human-centric applications.
XuX, LuY, Vogel-HeuserB, WangL. Industry 4.0 and Industry 5.0-inception, conception and perception. J Manuf Syst 2021; 61:530-5.
[2]
WangJ, TaoB, GongZ, YuS, YinZ. A mobile robotic measurement system for large-scale complex components based on optical scanning and visual tracking. Robot Comput-Integr Manuf2021; 67:102010.
[3]
YilmazA, SumerE, TemeltasH. A precise scan matching based localization method for an autonomously guided vehicle in smart factories. Robot Comput- Integr Manuf2022; 75:102302.
[4]
MengJ, WangS, XieY, LiG, ZhangX, JiangL, et al. A safe and efficient LIDAR- based navigation system for 4WS4WD mobile manipulators in manufacturing plants. Meas Sci Technol2021; 32(4):045203.
[5]
ZhengP, LiC, FanJ, WangL. A vision-language-guided and deep reinforcement learning-enabled approach for unstructured human-robot collaborative manufacturing task fulfilment. CIRP Ann2024; 73(1):341-4.
[6]
Del DottoreE, MondiniA, RoweN, MazzolaiB. A growing soft robot with climbing plant-inspired adaptive behaviors for navigation in unstructured environments. Sci Robot2024; 9(86):eadi5908.
[7]
PeiL, LinJ, HanZ, QuanL, CaoY, XuC, et al. Collaborative planning for catching and transporting objects in unstructured environments. IEEE Robot Autom Lett2024; 9(2):1098-105.
[8]
ZhengP, LiS, FanJ, LiC, WangL. A collaborative intelligence-based approach for handling human-robot collaboration uncertainties. CIRP Ann2023; 72 (1):1-4.
[9]
ZhengP, LiS, XiaL, WangL, NassehiA. A visual reasoning-based approach for mutual-cognitive human-robot collaboration. CIRP Ann2022; 71(1):377-80.
LiS, ZhengP, LiuS, WangZ, WangXV, ZhengL, et al. Proactive human-robot collaboration: mutual-cognitive, predictable, and self-organising perspectives. Robot Comput-Integr Manuf2023; 81:102510.
[12]
FanJ, ZhengP, LiS. Vision-based holistic scene understanding towards proactive human-robot collaboration. Robot Comput-Integr Manuf2022; 75:102304.
[13]
WangT, ZhengP, LiS, WangL. Multimodal human-robot interaction for human-centric smart manufacturing: a survey. Adv Intell Syst2024; 6 (3):2300259.
[14]
RenM, ZhengP. Towards smart product-service systems 2.0: a retrospect and prospect. Adv Eng Inform2024; 61:102466.
[15]
XiaL, LiC, ZhangC, LiuS, ZhengP. Leveraging error-assisted fine-tuning large language models for manufacturing excellence. Robot Comput-Integr Manuf2024; 88:102728.
[16]
RenM, DongL, XiaZ, CongJ, ZhengP. A proactive interaction design method for personalized user context prediction in smart-product service system. Procedia CIRP2023; 119:963-8.
[17]
YinS, FuC, ZhaoS, LiK, SunX, XuT, et al. A survey on multimodal large language models. 2023. arXiv:2306.13549.
[18]
SuS, ZengX, SongS, LinM, DaiH, YangW, et al. Positioning accuracy improvement of automated guided vehicles based on a novel magnetic tracking approach. IEEE Intell Transp Syst Mag2020; 12(4):138-48.
[19]
GouthamM, BoyleS, MenonM, MohanS, GarrowS, StockarS. Optimal path planning through a sequence of waypoints. IEEE Robot Autom Lett2023; 8 (3):1509-14.
[20]
DemesureG, El-HaouziHB, IungB. Mobile-agents based hybrid control architecture—implementation of consensus algorithm in hierarchical control mode. CIRP Ann2021; 70(1):385-8.
[21]
PashevichA, SchmidC, SunC. Episodic transformer for vision-and-language navigation. In:Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); 2021 Oct 10-17; Montreal, QC, Canada. Piscataway: IEEE; 2021. p. 15922-32.
[22]
HuangC, MeesO, ZengA, BurgardW. Visual language maps for robot navigation. In:Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA); 2023 May 29-Jun 02; London, UK. Piscataway: IEEE; 2023. p. 10608-15.
[23]
WangT, FanJ, ZhengP. An LLM-based vision and language cobot navigation approach for human-centric smart manufacturing. J Manuf Syst2024; 75:299-305.
ZachariaPT, XidiasEK. AGV routing and motion planning in a flexible manufacturing system using a fuzzy-based genetic algorithm. Int J Adv Manuf Technol2020; 109:1801-3.
[26]
GuJ, StefaniE, WuQ, ThomasonJ, WangXE. Vision-and-language navigation: a survey of tasks, methods, and future directions. 2022. arXiv:2203.12667.
[27]
KrantzJ, WijmansE, MajumdarA, BatraD, LeeS. Beyond the Nav-Graph:vision-and-language navigation in continuous environments. In: Vedaldi A, Bischof H, Brox T, Frahm JM, editors. Computer vision-ECCV 2020. Cham: Springer; 2020. p. 104-20.
[28]
GuhurPL, TapaswiM, ChenS, LaptevI, SchmidC. Airbert:In-domain pretraining for vision-and-language navigation. In:Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); 2021 Oct 10-17; Montreal, QC, Canada. Piscataway: IEEE; 2021. p. 1614-23.
[29]
KrantzJ, GokaslanA, BatraD, LeeS, MaksymetsO. Waypoint models for instruction-guided navigation in continuous environments. In:Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); 2021 Oct 10-17; Montreal, QC, Canada. Piscataway: IEEE; 2021. p. 15142-51.
[30]
ZhangY, MaZ, LiJ, QiaoY, WangZ, ChaiJ, et al. Vision-and-language navigation today and tomorrow: a survey in the era of foundation models. 2024. arXiv:2407.07035.
[31]
GaoJ, ChenB, ZhaoX, LiuW, LiX, WangY, et al. LLM-enhanced reranking in recommender systems. 2024. arXiv:2406.12433.
[32]
VaswaniA, ShazeerN, ParmarN, UszkoreitJ, JonesL, GomezAN, et al. Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems; 2017 Dec 4-9; Long Beach, CA, USA. Red Hook, NY: Curran Associates Inc. p. 6000-10.
[33]
ZhouG, HongY, WuQ. NavGPT: explicit reasoning in vision-and-language navigation with large language models. Proc Conf AAAI Artif Intell2024; 38 (7):7641-9.
[34]
ShahD, Osin´ skiB, IchterB, LevineS. LM-NAV: robotic navigation with large pre-trained models of language, vision, and action. In: Proceedings of the 6th Conference on Robot Learning (CoRL 2022); 2022 Dec 14-18; Auckland, New Zealand. New York City: PMLR; 2023. p. 492-504.
[35]
ChoiS, ZhouQY, KoltunV. Robust reconstruction of indoor scenes. In: Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2015 Jun 7-12; Boston, MA, USA. Piscataway: IEEE; 2015. p. 5556-65.
[36]
ParkJ, ZhouQY, KoltunV. Colored point cloud registration revisited. In:Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV); 2017 Oct 22-29; Venice, Italy. Piscataway: IEEE; 2017. p. 143-52.
[37]
ZhouQY, ParkJ, KoltunV. Fast global registration. In: Leibe B, Matas J, Sebe N, Welling M, editors. Computer vision-ECCV 2016. Cham: Springer; 2016. p. 766-82.
[38]
NewcombeRA, IzadiS, HilligesO, MolyneauxD, KimD, DavisonAJ, et al. KinectFusion:real-time dense surface mapping and tracking. In:Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality; 2011 Oct 26-29; Basel, Switzerland. Piscataway: IEEE; 2011. p. 127-36.
[39]
RusinkiewiczS, LevoyM. Efficient variants of the ICO algorithm. In:Proceedings of the Third International Conference on 3-D Digital Imaging and Modeling; 2001 May 28-Jun 1; Quebec City, QC, Canada. Piscataway: IEEE; 2001. p. 145-52.
[40]
SavvaM, KadianA, MaksymetsO, ZhaoY, WijmansE, JainB, et al. Habitat:a platform for embodied AI research. In:Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV); 2019 Oct 27-Nov 2; Seoul, Republic of Korea. Piscataway: IEEE; 2019. p. 9338-46.
[41]
PützS, WiemannT, SprickerhofJ, HertzbergJ. 3D navigation mesh generation for path planning in uneven terrain. IFAC-PapersOnLine2016; 49:212-7.
[42]
WennaW, WeiliD, ChangchunH, HengZ, HaibingF, YaoY. A digital twin for 3D path planning of large-span curved-arm gantry robot. Robot Comput-Integr Manuf2022; 76:102330.
[43]
LiB, WeinbergerKQ, BelongieS, KoltunV, RanftlR. Language-driven semantic segmentation. In:Proceedings of the 2022 International Conference on Learning Representations (ICLR2022); 2022 Apr 25-29; Online. ICRL; 2022. p. 1-13.
[44]
RonnebergerO, FischerP, BroxT. U-Net:convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF, editors. Medical image computing and computer-assisted intervention- MICCAI 2015. Cham: Springer; 2015. p. 234-41.
[45]
ChenLC, ZhuY, PapandreouG, SchroffF, AdamH. Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y, editors. Computer vision-ECCV 2018. Cham: Springer; 2018. p. 833-51.
PDF
(4186KB)
