Journal Home Online First Current Issue Archive For Authors Journal Information 中文版

Engineering >> 2024, Volume 32, Issue 1 doi: 10.1016/j.eng.2023.01.009

A Reconfigurable Data Glove for Reconstructing Physical and Virtual Grasps

a National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), Beijing 100080, China
b Center for Vision, Cognition, Learning, and Autonomy, University of California, Los Angeles, CA 90095, USA
c Multi-Physics Lagrangian-Eulerian Simulations Laboratory, Department of Mathematics, University of California, Los Angeles, CA 90095, USA
d Institute for Artificial Intelligence, Peking University, Beijing 100871, China
e Department of Automation, Tsinghua University, Beijing 100084, China

Received: 2022-03-28 Revised: 2022-12-23 Accepted: 2023-01-04 Available online: 2023-03-16

Next Previous

Abstract

In this work, we present a reconfigurable data glove design to capture different modes of human hand–object interactions, which are critical in training embodied artificial intelligence (AI) agents for fine manipulation tasks. To achieve various downstream tasks with distinct features, our reconfigurable data glove operates in three modes sharing a unified backbone design that reconstructs hand gestures in real time. In the tactile-sensing mode, the glove system aggregates manipulation force via customized force sensors made from a soft and thin piezoresistive material; this design minimizes interference during complex hand movements. The virtual reality (VR) mode enables real-time interaction in a physically plausible fashion: A caging-based approach is devised to determine stable grasps by detecting collision events. Leveraging a state-of-the-art finite element method, the simulation mode collects data on fine-grained four-dimensional manipulation events comprising hand and object motions in three-dimensional space and how the object's physical properties (e.g., stress and energy) change in accordance with manipulation over time. Notably, the glove system presented here is the first to use high-fidelity simulation to investigate the unobservable physical and causal factors behind manipulation actions. In a series of experiments, we characterize our data glove in terms of individual sensors and the overall system. More specifically, we evaluate the system's three modes by ① recording hand gestures and associated forces, ② improving manipulation fluency in VR, and ③ producing realistic simulation effects of various tool uses, respectively. Based on these three modes, our reconfigurable data glove collects and reconstructs fine-grained human grasp data in both physical and virtual environments, thereby opening up new avenues for the learning of manipulation skills for embodied AI agents.

SupplementaryMaterials

Related Research